Distributed Inference with Early Exit at Edge Networks

With the increasing prevalence of edge devices and the exponential growth of deep learning applications, there is a pressing need for efficient algorithms and techniques that can be applied to resource-constrained devices. This master thesis presents a novel system that combines distributed computing and early exit strategies to enable deep learning on edge devices. A multi-threaded algorithm is proposed to flexibly manage the load on each device based on both communication and computational requirements. Two solutions are presented, addressing common needs: accuracy constraint and input rate constraint. The primary objective is to investigate the feasibility, performance, and flexibility of the proposed techniques in resource-constrained environments. The evaluation of the framework includes performance benchmarking, analysis of different neural network architectures and network topologies, and assessment of its adaptability. The results demonstrate effective resource exploitation, showcasing superior performance in topologies consisting solely of edge devices compared to traditional approaches or partial offloading to an edge server. The findings of this study highlight the potential of the proposed approach in enhancing deep learning capabilities on resource-constrained edge devices, particularly in server-less topologies.