Application of Facial Emotion Recognition Techniques to the Study of Contagious Emotions via Global Face Analysis

This work is part of a project in collaboration with the Department of Psychology at the University of Turin that involves creating a scientific database containing video stimuli of spontaneous human facial expressions of yawning, laughter, mirror pain, embarrassment, contempt, and schadenfreude. The purpose behind its creation is to facilitate the study of emotional contagion. Emotional contagion is defined as the tendency to automatically mimic and synchronize movements, facial expressions, postures, and vocalizations among individuals. To create the video stimuli for the database, an experimental setup was devised. The first phase aimed to record the spontaneous reactions of a group of healthy subjects to selected video clips designed to elicit the target emotions. The most intense facial expressions were used as stimuli in the second phase, which focused on validating the stimuli in terms of facial expression recognition and verifying emotional contagion among the subjects from both phases. Facial Emotion Recognition (FER) was used for the analysis of yawning, laughter, and mirror pain emotions. FER uses the face as input, extracted from images or videos, which is crucial in the context of emotional contagion because it is the primary means through which a connection with others can be established. An investigation was conducted into the effect of the primary videos administered to the subjects in Phase 1 in terms of the expected emotion. To do this, three binary SVM models for yawning, laughter, and mirror pain with manually selected HOG features from the faces were trained. Random frames from 10 Phase 1 subjects were tested with the classifiers, and it was found that laughter had the highest number of validating frames, approximately 53%. This was followed by mirror pain, with 36% of identified frames, and finally, yawning, with only 1%. Consistent with these results, visual analysis of the faces and questionnaire responses also showed a stronger reaction to laughter videos and a weaker one to mirror pain and yawning. Subsequently, an analysis of Phase 2 subjects' videos in response to Phase 1 subjects was conducted to verify emotional contagion in laughter, mirror pain, and yawning videos. All frames where the subjects imitated the facial expression of the stimulus were selected. Various multiclass classifiers of SVM and Random Forest with HOG features were trained to validate contagion. The basic idea is to check if faces that are detected with the same facial expression by a human operator during emotional contagion are also labeled in the same class by an emotion detection classifier. Models trained only with Phase 1 subjects show classification bias towards mirror pain and misclassification between mirror pain and neutral expressions, while those trained with subjects from both phases suffer from misclassifications between mirror pain, laughter, and yawning. This is because, as Phase 2 subjects are less expressive, they affect the ability to better distinguish between classes, even with the same accuracy. This is reflected in the classifier's performance. It can be observed that when Phase 1 subjects are added to the test, performance increases from 60% to 87% if the model is trained only with Phase 1 subjects and from 75% to 85-90% if the model is trained with subjects from both phases. Therefore, using FER techniques in psychological analyses can provide an important tool for exploring and studying relatively unknown research areas, such as emotional contagion.