A feasibility study for enhancing student’s engagement in remote lectures

During the COVID-19 pandemic, almost all learning activities, as well as other daily face-to-face activities, were moved to online environments. Although e-learning has many advantages, it also has some disadvantages when compared to traditional face-to-face education. Lack of direct contact between teachers and students and less active participation with respect to face-to-face lectures. This led to the first new difficulties in deciphering common student states, such as boredom, confusion, and frustration. In this regard, technological support that can accurately and efficiently detect the level of attention of their students online can therefore be of great help to remote teachers. In this way, they could change their teaching approach dynamically and adopt techniques of capturing the attention of the students when necessary, increasing the effectiveness of the lessons themselves. So, e-learning platforms could enable the incorporation of novel technologies to estimate various factors such as eye blinking, gaze tracking, and facial expressions, which are important indicators of student engagement. The purpose of this thesis is therefore to carry out a study on the possibility of creating an integrated system to detect the level of involvement and attention of students in real-time during remote lessons. The idea is that starting from the individual student video it is possible to detect input parameters such as gaze detection, blink detection, and facial expression detection. There are several algorithms for detecting eye blinking and gazing. Three of these algorithms, called OpenFace, GazeTracking, and Gaze Controlled Keyboard were used to test videos shot in different conditions, including good and bad lighting and the presence or absence of glasses. These videos are representative of a normal lesson activity and involve viewing the webcam with simple eye movements to the right, left, and center and normal blinking. The accuracy of the blink and gaze tracking models was determined by comparing the output of the models to the ground truth. This was defined by manually labeling the individual video frames. The comparison between the outputs of the models and the ground truth allowed us to select two of the models, which are OpenFace and GazeTracking, as the preferred ones for the realization of the system. The OpenFace library is a comprehensive tool built with C++ for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation. So, starting with the OpenFace GUI example provided with the library, facial expression detection was added to the software. The new code is capable of detecting facial expressions on OpenFace GUI, and it shows the expected emotion for each frame among the following: Angry, Contempt, Disgust, Fear, Happiness, Neutrality, Sadness, and Surprise. GazeTracking is written in Python and it requires the use of libraries such as NumPy, OpenCV, and Dlib. It can detect the user's iris and provide labels such as blink, look right, left, and center. As for OpenFace, also for this library has been added the possibility of detecting facial expressions. The outputs of both programs are provided in real-time but are also saved in a CSV file to make a deferred analysis of the results. We discovered that, in addition to facial expressions, the frequency of blinking and gaze tracking might be important factors in determining whether a student is engaged or distracted during remote lectures.