Fault Detection Metrics in Image Segmentation Neural Networks

Faults that occur on weights of semantic segmentation networks can significantly compromise the final result of the output mask. In particular determining the degree of such criticality is not a trivial task and, in some cases, it is also extremely crucial. For this purpose, various techniques have been developed to evaluate the output mask. The state of the art methods consist in the comparison between the faulty and faultless output of the network, based on two main segmentation metrics: mean intersection over union (mIoU) and pixel accuracy (PA). These methods, which turns out to be very effective, actually hides several problems and in particular, the main one is that, in real cases, there is no guarantee to have access to the faultless output. In addition, subjective thresholds used for the classification do not always provide the desired results. Other techniques rely on the usage of an auxiliary neural network, also subject to faults and their latency is not always sufficiently low to guarantee the real time requirement, which is a fundamental constraint in many fields such as the automotive one. This thesis describes the application of state of the art methods on a specific image segmentation network and proposes a new metric, applicable in environments where real time constraint is mandatory, aimed at evaluating and identifying criticalities in segmentation masks, when the network is subject to hardware faults. The advantage introduced by this metric is that it is able to evaluate a single output without having access to the clean one and the temporal coherence across input frames is not required. The latter is a significant detail because masks are evaluated individually, without considering the previous frames, potentially affected by critical faults. The metric involves a statistical approach with an initial overhead that depends on the size of the dataset partition used to produce statistics on the clean network inferences. A labeled dataset is also proposed to evaluate results obtained with the developed metric, compared with the state of the art metrics used for this kind of classification. The network employed is known as Fast-SCNN, an open source semantic segmentation network, developed in python with the PyTorch library and pre-trained and tested on Cityscapes. A fault injector is also used to simulate the faulty network by injecting stack-at faults on a specific bit of the weight. The output masks are then analyzed with the developed metric, composed by different units, each involved in identifying whether a fault is critical or not (binary task). Additionally, a deeper study is performed on each metric unit to investigate the impact on the final results and performances are compared to the state of the art methods, highlighting the fact that identifying a critical fault is a problem that is far from solved and that metrics actually available are not always reliable.