"From Deep Learning to Human Perception: A Comparative Study on Saliency Maps in Facial Expression Recognition"

The facial expression recognition (FER) task is crucial in the field of computer vision, with applications ranging from human-computer interaction to mental health assessment. Thanks to the application of convolutional neural networks, deep learning has advanced significantly in recent years, making advancement in numerous fields, including FER. These achievements have led to high classification accuracy in emotion recognition. However, understanding how the model operates and makes decisions remains a highly complex process. This project provides an overview of 3 different saliency map generation methods including GradCam, Bubbles and External Perturbation performed on YOLOv8 exploring the field of explainable AI in FER. Moreover, through an experimental setup using eye-tracking technology, a comparison was conducted between the saliency maps generated by convolutional models and human attention patterns acquired by using Pupil Lab Invisible glasses. Specifically, the experiment involved recognizing a subject’s emotion based exclusively on their facial expression. A total of 350 images were used, 50 for each of the six basic emotions (anger, disgust, fear, happiness, sadness, surprise) plus the neutral one. 54 participants took part in the study (33 females and 21 males), aged between 19 and 64. Before the experiment, each participant completed a questionnaire to assess fatigue, concentration, and mood, aiming to evaluate their potential impact on decision-making. The goal is to evaluate whether model-generated saliency maps align with human visual attention and, if so, to understand the underlying reasons. The results provide insights into the reliability of saliency maps and their role in improving FER systems.