A Comprehensive Overview of Fault Tolerance Techniques for Convolutional Deep Neural Networks

Deep Neural Networks (DNNs) are being increasingly used in safety-critical applications, from healthcare to autonomous driving. Furthermore, as new frontiers unfold, such as the growing New Space Economy, DNNs are expected to be widely deployed in outer space and other hazardous environments soon. However, their prediction accuracy was shown to degrade in presence of transient hardware faults, leading to unpredictable and potentially catastrophic errors. As these kinds of problems are frequent in radiation-prone domains such as space, it is of the utmost importance to strenghten the DNNs' resistance to computational or parameters errors. In the relevant literature, multiple fault tolerance techniques have been researched, which limit the consequences of potential faults. Nonetheless, most techniques available today mainly rely on hardware redundancy, which can be unsustainable for mass DNNs deployment in out-of-reach scenarios. Consequently, in recent years, several techniques have been proposed to increase the fault tolerance of DNNs by modifying the network structure and/or training procedure, thereby reducing the need for costly additional hardware. These methods, however, have not been extensively tested, if not on few benchmark datasets and models, which often are not representative of real-world applications. Hence, this study aims to categorize existing techniques and further enhance the most suitable for its objectives, an activation clipping strategy called ClipAct. The goal is to propose and evaluate a solid baseline to increase DNNs' fault tolerance, without introducing hardware or runtime overhead. This method is designed to be easily adaptable to a broad variety of use cases and implementations, and able to be complemented with more tailored solutions, depending on the desired application and the available resources. Multiple experiments carried out, utilizing various widely-recognized models and several datasets, including four classification tasks related to Earth Observation. Experimental results emphasize the effectiveness of the investigated fault tolerance techniques. However, experimental results vary across datasets and DNN architectures.