Understanding and Recognizing Facial Expressions based on Deep Learning

Facial expressions stand out as one of the most powerful non-verbal ways to express human emotions and intentions. These expressions are often deliberate and socially regulated, making them more straightforward to analyze and understand. In addition to these ordinary facial expressions, known as macro-expressions, that we observe daily, there are also spontaneous facial expressions, especially micro-expressions, are involuntary, subtle and occur briefly, often without the awareness of the individual displaying them. These unique characteristics make micro-expressions challenge to detect. To better understand the rapid and involuntary nature of micro-expression, we utilize the optical flow algorithm to calculate facial movements between the onset and apex frame of the micro-expression videos. This method provides a dynamic map of facial movements and the visualization of the subtle facial movement. The analysis of facial expressions has been a long-standing research focus in the field of computer vision, significantly advanced by applications in human-computer interaction. Early methods for facial expression recognition mainly relied on traditional manual feature-based techniques and mapped facial expressions to basic emotion categories. For better understand the complexity of emotions, we project facial expressions into the two-dimension valence-arousal space. We utilized public BU3DFE dataset and CalD3r datasets. Reorganized the data labels based on the discrete emotional categories and the valence-arousal dimension. With the recent success of Deep Learning across numerous tasks, neural networks have gained increasing attention for their potential in spontaneous expression recognition. We utilized pre-trained deep learning models like VGG16, MobileNetV2 and ResNet50，all of which were originally trained on large ImageNet datasets. By applying transfer learning techniques and fine-tuning these models on our small datasets, we effectively enhanced feature extraction and improved expression recognition accuracy. To enhance the robustness of the model, we performed the data augmentation, which can generate more training samples. We use the accuracy measurement like precision, recall and F1 score to evaluate the model’s performance. Additionally, visualization techniques such as Grad-Cam and optical flow were employed to illustrate and compare the dynamics of facial movements, thereby enriching our understanding and the capability to detect and analyze facial expressions.