Creation of a neuromorphic dataset for low-power and privacy-aware gesture recognition

In the context of telemedicine and telerehabilitation, activity recognition plays a crucial role, especially in camera-based systems. For example, in Parkinson’s disease, many camera-based systems for automatic hand motor impairment assessment and rehabilitation, require an initial step of gesture recognition. In addition, the privacy of these systems is crucial and traditional RGB videos can be associated with significant privacy issues, especially during data transmissions. A new paradigm of cameras, such as event cameras, are promising solutions for anonymous gesture recognition. Event cameras are neuromorphic devices inspired by the human retina, acquiring only the information deemed useful, with a high temporal resolution (in the order of microseconds), low consumption and a high dynamic range. Combined with neuromorphic architectures such as spiking neural networks, these systems may provide solutions for gesture recognition at the edge, guaranteeing low consumption and compliance with privacy regulations. This thesis aims at collecting a large-scale dataset for gesture recognition using an event camera (DVXplorer Lite), together with some traditional RGB-Depth devices (Azure Kinect) and a motion capture system (OptiTrack), for reference. This dataset can be exploited for training robust spiking neural networks for hand gesture recognition. In further details, 25 subjects were recruited among the student population of Politecnico di Torino and more than 6 hours of hand gestures were recorded, with a 100 kHz sampling rate for events. Gestures were chosen from typical movements assessed in Parkinson’s disease (e.g., finger tapping, hand opening-closing, pronation-supination), to provide a large benchmark of healthy subjects which may be exploited for further research in the field. Finally, to provide an example of how this dataset may be exploited by future researchers, a first attempt at training a spiking neural network using the collected data was performed. Even if preliminary, results are encouraging and will provide indications for future iterations.