Event-based camera communications: a measurement-based analysis

Address Event Sensing (AES) represents a completely new way of sensing reality. Legacy vision systems rely on sensors that capture intensity images at a constant rate (e.g., $30$~fps) leading to high latency and redundancy in terms of information and computation. This results in wasting precious resources such as energy, CPU, memory access, storage and waiting time for superfluous frames. AES silicon retina instead are bio-inspired sensors that asynchronously emit spikes, i.e., events, whenever they occur. An ON/OFF event is a positive/negative log intensity change that is registered at a pixel whenever a brightness threshold is crossed. Since the concept of frame is completely eliminated, this approach allows to significantly reduce the latency between photodiode illumination change detection and its off-chip transmission down to $\approx 15$~\textmu s. By contrast, the time it takes to build a single frame is $33.3$~ms for a video at $30$~fps resolution. The impact of temporal redundancy removal is such that the low computational requirements make these dynamic vision sensors (also known as event cameras) well suited for real-time feedback vision-based robotics systems. Address-event silicon retina apply to important and trending computer vision and vision-based robotic systems tasks such as multi-object tracking, gesture recognition, autonomous driving and surveillance just to name but a few. \noindent To this date, the vast majority of applications of AES are bounded to local devices. For example, the Istituto Italiano di Tecnologia (IIT)'s iCub humanoid robot features two event cameras as eyes to track ball movements. In this thesis, we aim at going beyond and study the case when these events have to be transmitted to remote devices. The potential of such paradigm change is huge and stems from industrial applications to coordination of drone swarms. Specifically, in this work we assess whether today wireless networks can support event-based communications for specific computer vision tasks such as continuous object tracking. We pay particular attention to the challenges introduced by latency and losses for a distributed event-based video streaming system and compared with legacy vision frame-based systems both over 802.11ac. For the latter, a custom Java platform has been developed to perform the frame-based streaming analysis and object tracking through OpenCV . For the event-based analysis instead, we relied on jAER as rendering support. jAER is an open-source Java-based framework suitable not only for the visualization of real-time events collected through a sensor or recorded datasets, but also for the development of real-time event-based algorithms and applications. The results highlight that a too small buffer size is detrimental for latency and tracking performance because more than the 50\% of events is dropped as the application threads involved are not able to keep up with events generation and packetization. The scene complexity and speed of objects are key features for event generation. Events are generated asynchronously and independently one from each other, but at application level, object tracking is performed over cluster of events because these are correlated and define the same moving object. We show that complex scenes with high speed moving objects generates more events and thus are more sensible to losses.