A novel approach towards balanced emotion analysis through synthetic augmentation of facial landmark trajectories

Mental health is essential for well-being, but poorly managed emotions can lead to depression and anxiety. Early emotion recognition systems support timely intervention. While physiological signals like ECG and EEG are accurate, they are invasive for everyday use. This study uses Facial Landmark Trajectories, namely EMO features, from the ASCERTAIN dataset, a non-invasive method tracking facial movements to capture authentic emotions. In contrast to the original dataset, in our work these emotions are classified into four valence-arousal quadrants: High Arousal High Valence (HAHV), High Arousal Low Valence (HALV), Low Arousal High Valence (LAHV), and Low Arousal Low Valence (LALV). To address the inherent class imbalance in this 4-class problem, the SMOTE, GAN, and CTGAN techniques are utilized to generate synthetic data for balanced classes. Four machine learning models, namely, KNN, SVM, Decision Tree, and Random Forest are tested on the augmented datasets, evaluated via confusion matrices and density plots for identifying overfitting. Experimental results demonstrate that CTGAN shows the least overfitting, with synthetic data closely matching the original and Random Forest achieving the highest accuracy (75%) and balanced true positive rates (>40% for all classes), demonstrating the potential utility of using Facial Landmark Trajectories for the development of an efficient automated emotion recognition system.