In recent years, the complexity of emerging computing systems has led to the need of developing new algorithms and solutions for both industry and academy. In this scenario, due to their outstanding computational capabilities, artificial neural networks (ANNs) have become attractive solutions in tasks such as image classification, such as radars, flight control, robots, self-driving cars, space applications. However, to adopt this solution safely in human context, it is crucial to assess their reliability. This work presents a methodology to identify the most critical neurons of a convolutional neural networks (CNN) (Resnet-18) trained on two different datasets, i.e. MNIST and CIFAR-10.