EVEgo: Egocentric Event-data for cross-domain analysis in first-person action recognition

Event cameras are novel bio-inspired sensors, which asynchronously capture pixel-level intensity changes in the form of “events". The innovative way they acquire data presents several advantages over standard devices, especially in poor lighting and high-speed motion conditions. In particular, their high pixel bandwidth results in reduced motion blur, and their high dynamic range make them a suitable alternative to traditional cameras when dealing with challenging robotics scenarios. Moreover, the latency and low power consumption of these novel sensors enable their use in real-world applications. Indeed, those peculiarities make them perfect to tackle well-known issues which arise from the use of wearable devices, such as fast camera motion and background clutter. However, their potential in these applications, such as egocentric action recognition, is still underexplored. In this work, we bring to light the potentiality of event sensors in first-person action recognition, showing the advantages that they offer over traditional cameras. Specifically, the latter suffer from egomotion, a phenomenon arising from the rapid and involuntary motion of the wearable device, which inevitably moves around with the user. Indeed, this characteristic enables event cameras to extract continuous information from the video. The recent release of the EPIC-Kitchen large-scale dataset, comprehensive of multiple input modalities, i.e., audio, RGB and optical flow, offers the possibility to show the advantages of the event modality over the traditional ones from the first person viewpoint. In this thesis, we propose an Event version of the large scale EPIC-Kitchens dataset, unlocking the possibility to explore the behavior of event data in First Person Action Recognition scenarios. Extensive experiments have been carried out by repurposing a variety of popular action recognition architectures in conjunction with recent Domain Adaptation methods. Those show the potentiality of event data in both intra- and cross-domain scenarios, establishing a large egocentric action recognition benchmark.