Detecting human gestures through event camera data using deep neural networks

Human action recognition has diverse applications, particularly in environments where humans and robots interact and collaborate. In any human-robot collaboration (HRC) scenario, a robot’s ability to understand human actions and intentions within the environment is crucial for both operational effectiveness and agent’s safety. Robot situational awareness leverages various innovative technologies. Nowadays, event cameras are a key element of computer vision and represent a promising technology for enabling robots to recognize human actions. These sensors capture movement by detecting changes in pixel brightness asynchronously, providing a higher temporal resolution than traditional cameras and minimizing data redundancy. The event data output can be converted into a frame-like representation, allowing the application of proven frame-based action recognition algorithms. Among these, deep neural networks stand out, leveraging their multi-layer learning capability to identify complex patterns in data. This thesis project aims to build and train a neural network to classify event-based videos of individual hand gestures using an available dataset. Initially, the network will process full video sequences, after which the input length will be shortened to reduce the time and amount of data for a classification. Next, a windowing methodology will be developed, with varying degrees of overlap, to enable offline classification of recorded event data and online classification of event data streams. To elaborate, the first phase of the project involves processing event data and classify them recurring to a 3D Convolutional Neural Network which analyses both the spatial and temporal dimensions of the event videos. This network is trained and tested on data from EDAT24 dataset, which includes event videos of a set of primitive tasks (idle, pick, place, screw) commonly performed in manufacturing assembly scenarios. The network achieves a testing accuracy of 100% when using full video sequences and 97.5% with shortened videos. However, transferring the knowledge gained from individual gestures to more complex assembly tasks involving multiple gestures in succession, along with transitional movements, remains a challenge. Initially, the classification system is used in an “offline” context, processing labelled recorded videos. A sliding classification window with a selected overlap is applied to read the frame sequence, returning a sequence of classifications. The results on test videos are promising, achieving 77.6% accuracy, although transitions between gestures still pose challenges for accurate classification. Misclassifications are analysed, and some strategies are proposed to make the dataset more robust, which in turn can improve the network's accuracy. Next, the system is adapted for an “online” scenario, enabling the continuous acquisition, processing and classification of event data received directly from the event camera. In this case, the primary focus is on reducing computation time to transform continuously received event data into classifications as quickly as possible. The system can classify the streamed data fast enough to achieve a response time comparable to the time interval required to generate a new frame, thereby avoiding delay accumulation. The gesture recognition system presented in this thesis is therefore promising for HRC assembly, providing robots with the necessary information to perform their tasks safely and effectively.