Multimodal Stress Assessment: From Signal Analysis to Industrial Application

Understanding human emotions is always fascinating, emotions play a vital role in decision-making in our life. An automated way of recognizing human emotions with physiological signals can improve the system’s adaptation to emotive conditions of a person, with applications in working environments, like the optimization of employee well-being in settings based on the repetition of the same task all day. This study analyzes stress and arousal levels using a multimodal approach based on electrodermal activity (EDA), photoplethysmography (PPG) and electroencephalography (EEG) also evaluating the efficacy of two different normalization methods. In the first experiment two different VR environments were developed: one to elicit stress and one to give a less stressful experience. The data have been extracted from 61 participants. At the conclusion of the experiment, subjects were asked to complete two questionnaires, one for each environment, and the answers were used as ground truth for the classifiers. Different features have been extracted from each signal and two different normalization method have been used to compare efficacy: division of the values of the features in the tasks for the values of the features in the baseline (Normalization 1) and subtraction for the values of the features in the baseline (Normalization 2). The features extracted for EDA signals are tonic and phasic component, mean and maximum values of the peaks and the number of peaks; the ones extracted from PPG signals are SDNN, RMSSD, pNN50, LF, HF and LF/HF; for the EEG signal, stress and arousal indicators have been calculated and different features have been extracted from them. These features were statistically analyzed using the Shaphiro-Wilk Test for normality check and the Friedman and Wilcoxon ranked test for analyzing significant statistical differences between the features in the two environments. The Wilcoxon ranked test shows that EDA features and some of the EEG features, for both normalization methods, are the ones that are statistically different between the two environments. At the end, K-nearest neighbors (KNN) and Support Vector Machine (SVM) classifiers are used for binary classification. The results showed that a multimodal approach improves the performance with respect to using only EDA features and that Normalization 1 gives slightly better performances with respect to Normalization 2. Upon completion of the study, an industrial case of study was done in which participants had to inspect Raspberry Pi 5 boards in order to identify soldering defects. The main goal of the study was to apply stress recognition in a working environment characterized by the repetition of the same actions. Two questionnaires were proposed in order to evaluate the state of the subject before and after beginning the experiment. The same features stated before were extracted from the signals and the analysis has been conducted using the same methodology. For the classifiers, the labels are not linked to the answers to the questionnaires; instead, participants were asked, during the experience, to place a marker in order to state that they were starting to feel tired or stressed.