Emotional Contagion: An Analysis of Unimodal and Multimodal Approaches Based on Facial Morphometric and Electromyographic Features

Emotional contagion is the mechanism through which individuals synchronize their facial expressions and muscle activations with those of the person they are observing. It can be seen as the preliminary stage of empathy. Investigating how emotional contagion manifests itself and identifying the emotions in which it is most evident represents an effective approach to understanding social dynamics. This study aims to evaluate and recognize emotional contagion by analyzing Euclidean distances between facial landmarks and features extracted from facial electromyographic (EMG) signals using machine learning methods. The research was conducted in collaboration with the Department of Psychology at the University of Turin and focused on three contagious emotions - laughter, mirror pain, and yawning - and a neutral condition. Data were collected following an experimental setup consisting of two phases. Phase 1 involved the creation of a database of contagious emotion stimuli, obtained by recording participants while they were watching carefully selected video clips designed to evoke these emotional reactions. Phase 2 consisted of recording facial responses from participants using both video acquisition and surface facial electromyography. Video recordings were processed by manually selecting the most expressive frames, from which morphometric distances were calculated, and EMG signals were aligned and processed to extract features of three domains: time, frequency and time-frequency. A subject-independent evaluation protocol was applied across data from 17 participants. Subsequently, all data were classified using different machine learning classifiers: K-Nearest Neighbors (KNN), Random Forest, Support Vector Machine (SVM), LogitBoost and RUSBoost. This work evaluates the effect of feature selection on the classifiers’ performance, comparing results obtained using all features with those using only selected features derived from different feature reduction methods. Feature reduction techniques considered are: ReliefF, Random Forest Feature Importance, Between-Within class variance (BWV) and Principal Component Analysis (PCA). The work involved two main approaches: unimodal and multimodal. The unimodal approach involves the use of data from a single source, allowing good performance and lower computational cost. Results obtained with this modality show accuracies ranging from around 55% to 70% for distances and approximately from 55% to 75% for signals, considering all features. Applying feature-reduction methods, performance increased nearly reaching 80% for distances and remained stable at approximately 75% for signals. Overall, the best classifier was SVM. The same analysis was conducted on three classes and performances exceeded 80% for distances and 90% for signals. The multimodal approach, on the other hand, involves the use of data from different sources, allowing complementary aspects and correlations between them to be captured. It was performed using decision-level fusion and feature level fusion. Decision-level fusion achieved the best accuracy across four-class analysis of the overall study exceeding 80% with product rule criterion for final decision-making. On the contrary, the feature-level fusion proved less effective than the unimodal approach with accuracies ranging at about 55%. This analysis showed that multimodal approach can be effective in case of heterogeneous data, providing a basis for future research to a better understanding of the complex dynamics underlying empathy.