Memory-constrained On-device Learning for Smart Infrared Sensors

Recent advances in Deep Neural Network (DNN) optimization enable the deployment of these models on resource-constrained embedded devices to perform various tasks. Multiple techniques, such as quantization and pruning, have been successfully applied to reduce the memory, energy, and latency requirements on edge devices. While these improvements enable edge inference, training directly at the edge remains a challenging task, despite its potential benefits for robust and adaptive AI systems. At the same time, low-resolution infrared (IR) cameras have numerous applications as these unobtrusive, privacy-preserving and inexpensive sensors can capture enough information to perform various tasks. By coupling a microprocessor to such IR sensors, it is possible to enable a rich collection of applications that would benefit from DNN inference and training directly on device. This thesis explores the feasibility of deploying and training a convolutional neural network (CNN) on a severely memory-constrained microcontroller (MCU) coupled to an IR array to classify the pose of a human being filmed by the sensor. The classification algorithm is executed on a bare-metal MCU with less than 32 kB of available memory, where just 14.5 kB can be used to store the model weights and all necessary data. Under such constraints, the usual backpropagation and gradient descent algorithms cannot be applied to update the weights and biases of the model. In this work, an alternative solution, called Nearest Class Mean (NCM), is explored. Traditional classification algorithms rely on the features extracted in the last layer to compute probabilities for a fixed number of classes. NCMs use a DNN to encode the infrared image in a low-dimensional embedding space. Then, a sample is classified by comparing the distances with the average encoding (so-called prototypes) of each class. The Convolutional Neural Network (CNN) undergoes a traditional training phase in which the Triplet loss is used to separate the embedding of different classes. After quantization and a few epochs of quantization-aware training, the model is exported and compiled to be executed at the edge. The algorithm can reach an accuracy on par with traditional CNN-based classification approaches when classifying classes it has been trained on, with varying accuracy loss when using prototypes of new classes. With a memory footprint fitting the hardware limit, its execution on the edge device is possible. On top of that, the DNN can be trained at the edge, either by updating the prototypes of selected classes to real-world data distributions or by creating a new prototype for a new class. Both without catastrophic forgetting and minimal memory overhead. This work may lead to the exploration and application of similar algorithms embedded on devices as memory-constrained as 32kB to perform inexpensive, privacy-preserving classifications.