Facial Expression Analysis for Cognitive State Estimation in Aerospace Human-Machine Systems

This thesis is part of a collaboration between the Politecnico di Torino and the Melbourne Institute of Technology RMIT. The developed project was carried out at the Bundoora East campus in collaboration with THALES Australia and Northrop Grumman Corporation. This project aims to analyse the potential integration of Facial Expression monitoring within a sensor network to evaluate real time the cognitive state of ATM and one-to-many UAS operators. ATM operators perform a safety-critical work in which it is essential that the situation awareness is not lacking and the workload is not excessive. Most accidents in the aviation field are due to human error so monitoring the cognitive state of the operators would lead to increase the efficiency of air traffic and the safety of operations. Currently, the Federal Aviation Authority (FAA) has mandated that remote pilots or visual observers are only allowed to operate or command one Unmanned Aerial Vehicle (UAV) at any time (14 CFR 107.35), so current UAS operations require multiple human operators to manage one UAV, known as ‘many-to-one’ operations. This research, therefore, aims to analyze the possibility of managing multiple UAVs with a single operator through a system in which the trusted autonomy is based on a bio-sensing network. These technologies are also fundamental for the development of Single Pilot Operations (SPO), which in recent years have been studied in order to have passenger aircrafts with only one pilot. The sensor network allows to monitor in real-time the operator in order to evaluate the cognitive state and adapt the level of automation of the software according to the latter. It is composed of several sensors that monitor different features in order to give greater reliability. The to date monitored parameters are: Breathing Rate, Blink Rate, Visual Entropy, Heart Rate, EEG which are used to define a parameter: the workload. The objective of this research is to evaluate a potential relationship between workload variation and facial contractions and to form the basis for a potential inclusion of Facial Expression monitoring in the sensor network. FE have been studied for many years to assess emotional state but little research has been done so far to assess cognitive state and workload. Through ATM and OTM experiments the psycho-physiological response of the operators has been evaluated and various types of analysis have been carried out to find correlation between the variation of the workload and the physiological response. These studies have been carried out in aerospace field for ATM, OTM and SPO applications but they can be applied in many other fields such as automotive.