Use of a multimodal Neural Network to investigate cultural and gender biases in affective stimuli of a novel facial expression database

Artificial Intelligence is about the replication of human intelligence in machines that are programmed to think as humans and imitate their actions. A human characteristic is the one of decoding the facial expression related to a determined emotion of other humans and this is necessary for the coexistence. In order to reproduce this peculiarity, the studies aimed to create empathetic machines able to recognize the facial expression of the facing human. This technique called Facial Expression Recognition (FER), is fundamental to a robot to understand the feelings in order to react consequently considering the perceived emotion. The interest towards this technique is increasing rapidly, so the purpose of the 3D Lab of Polytechnic of Turin was to create a database made up human spontaneous emotions collected with an experiment in collaboration with the Pontif´ıcia Universidade Cat´olica do Rio de Janeiro. The experiment was conducted in Brazil, among brazilian partecipants, showing them a dataset of 48 images carefully chosen between two of the most world??wide known affective database, International Affective Picture System (IAPS) and Geneva Affective PicturE Database (GAPED). It was asked to the partici??pants only to react spontaneously to the images in the meantime of their faces were recorded by means of a RGB + a coded-light depth sensor in order to catch the spontaneous facial expressions of the volunteers. The ambition was to identify the expressed emotion among the 6 basic ones coming from Paul Ekman’s studies (anger, disgust, sadness, happiness, surprise, and fear) plus the neutral expression. After each showed image, a questionnaire was proposed to the participants in such a way to self-asses their feelings in term of 3 parameters: valence, arousal and the name of the emotion itself by means of the Self-Assessment Manikin (SAM) scale. For each of the 48 images, a video of 6 seconds was recorded and subsequently the most representative frame of the emotion was selected. Later, we merged the RGB and Depth data and cropped the image in order to remove the parts not relevant for our purpose (e.g. background, clothes, hair). Furthermore we trained our Neural Network with a 2D + 3D multimodal ap??proach based on a 3D Convolutional Neural Network (CNN) and a 2D Vision Transformer (ViT) encoder in order to classify the expressed emotion from the images combining the advantages of the two approach and improving the recog??nition rate. Since a Caucasian database (CalD3r) was already collected by the Polytech??nic of Turin and Polytechnic of Milan between 2020 and 2021, we performed our analysis to realize if there were some relevant differences between two cul??tures such as the Italian and the Brazilian one. In particular we searched for the discrepancies between the emotion that should have been elicited by a determined image and the real effective perceived one. The ambition was to examine if particular images induce different emotions if showed to Italians or Brazilians. Unifying the two databases we also performed a differentiation between male and female participants looking for some theories already long supported by various studies such as: ’Are women more empathetic than men? Are they more fearful?’ At the end an analysis on the number of neutral expressions was achieved in order to identify among the two cultures (Italians and Brazilians) and the two genres (Male and Female) which is the more expressive.