Towards Automated Facial Mimicry Assessment Using RGB-D Data and a commercial tracking software: preliminary results on healthy and parkinsonian subjects

Hypomimia, the reduction of spontaneous facial movements, is an early and disabling symptom of Parkinson’s disease (PD). Its clinical evaluation is currently subjective and based on qualitative rating scales. Gold standard (GS) methods including manual visual expert inspection and surface electromyography (EMG) provide objective assessments but require expensive equipment, trained personnel, and may interfere with natural facial expressivity. Recently explored markerless (ML) methods using RGB and RGB-D cameras, combined with deep learning-based facial landmark detection techniques, represent a non-invasive alternative. However, their clinical validation remains debated. This thesis aimed to (i) propose a low-cost ML method using a single RGB-D camera to quantify facial muscle activity, (ii) validate it against GS measures (manual measurements and EMG signals), (iii) assess the impact of the depth sensor and RGB image resolution on its performance and (iv) evaluate its applicability in discriminating between young healthy (YH), elderly healthy (EH), and PD subjects during emotions. Participants included 17 YH (25.5±3.7 y.o.), 13 EH (69.7±4.2 y.o.), and 11 PD patients (70.7±8.7 y.o.). Data were acquired using an Azure Kinect RGB-D camera (1280×720, 30Hz), and EMG signals were recorded with a D360 amplifier (5 kHz). Each subject was recorded at rest, during maximum voluntary contraction (MVC) of the depressor anguli oris (DAO) muscle, and while expressing spontaneous happiness and sadness. The MediaPipe Face Mesh algorithm was used to extract 2D DAO landmarks, with depth data extracted from the depth image. To identify any statistical differences between methods (manual vs automatic) and patients' groups, a Mann–Whitney test (α = 0.05) was applied. For validation, the ML-derived DAO length variation (DAO-LV) was compared with manual measures during MVC, resulting in a Mean Absolute Error (MAE) of 1.5±1.6 mm (Mean Absolute Percentage Error = 27.9±27.2) across 41 subjects. No significant differences were found between automatic and manual measures confirming ML protocol validity. Both EMG RMS values and DAO-LV showed no significant differences between YH and EH, while a significant difference was observed between PD and EH, highlighting the ML protocol’s sensitivity in distinguishing between groups during MVC. To evaluate the adding value of the depth sensor with respect to the use of RGB image only and the influence of image resolution on the method's performance, only 2D DAO landmarks were considered and a static calibration phase was introduced to obtain DAO-LV in millimeters. Those values were compared with manual measurements. Removing the depth sensor and reducing image resolution to 640×480 resulted in an increase in MAE of 52% and 79%, respectively, confirming that both factors are critical for accurate facial expression analysis. Emotional expression analysis showed no significant differences between YH and EH for either emotion. Happiness expression revealed significant differences between EH and PD for both DAO-LV and DAO contraction velocity, as happiness elicited greater muscle activation. On the other hand, during sadness, no significant difference between EH and PD were found likely due to high inter-subject variability and low DAO-LV in PD patients. In conclusion, the proposed RGB-D ML method is a valid tool for the objective assessment of hypomimia in PD since it effectively differentiates between subject groups, particularly during MVC and happiness expression.