The MultiMotion Project Developing a Machine Learning Framework for Emotion Recognition from Data Acquisition to Fusion Modeling

Automatic emotion recognition represents a complex challenge in affective computing, hindered by the unreliability of self-reported assessments and the limitations of conventional biosignal processing techniques. This thesis aims to overcome these limitations by developing a robust framework for continuously predicting emotional arousal and valence. The framework is validated on a newly acquired multimodal dataset, collected for this study, which includes photoplethysmographic (PPG), galvanic skin response (GSR), pupillometry, and facial expression (FER) data, recorded concurrently with participant self-ratings during emotional video stimulation. This research presents three key contributions: the Python implementation of the INDSCAL algorithm, used to derive a reliable, individualized ground truth from the collected subjective ratings; an in-depth, comparative analysis of Heart Rate Variability (HRV) using advanced feature extraction; and the development of a multimodal fusion framework for continuous prediction of emotional dimensions. Heart Rate Variability analysis constitutes the methodological core of the study. Using MATLAB’s hctsa toolkit, complex features were extracted from the PPG signal, including fluctuation analysis, wavelet decompositions, state-space representations, autoregressive model parameters, and transition matrix properties. This feature engineering process enabled the identification of optimal descriptors for each emotional dimension. Regression models developed on the HRV features achieved R² values of 0.86 for arousal and 0.82 for valence, demonstrating the effectiveness of the extracted cross-domain features. The multimodal fusion strategy, integrating predictors derived from HRV, GSR, pupillometry, and FER, further improved predictive performance, reaching R² values of 0.93 for arousal and 0.84 for valence, with Pearson correlation values exceeding 0.99. These results confirm that the synergistic combination of heterogeneous signals and the use of non-linear learning algorithms effectively capture the complexity of the relationships between physiological indicators and affective states. This framework's potential applications range from psychological diagnostics to clinical monitoring, from neurorehabilitation to the development of empathetic interfaces, opening promising directions for future research in continuous emotion recognition systems.