Explaining black-box models in the context of image classification.

Neural networks are black-box classification models that can be trained, among other things, to classify the content of an image into different categories. This will be the use case addressed in this work. In some cases, neural networks surpass human precision. Therefore, the interest toward this technology has increased exponentially in the last few years. Although the performances are very high, the interpretability is low. In this thesis, we make contributions to the development of EBANO, an engine that generates transparency reports for black-box models. First, we compared EBANO's feature extraction step with image segmentation techniques, among those examined: flood fill, watershed, grabcuts and meanshift only watershed and meanshift were applicable to our case. From the comparison we concluded that the method adopted by EBANO creates interpretable features, while others create results that are highly dependent on lighting conditions under which the image have been taken and need careful parameter tuning for each case. From this, we decided to keep the EBANO’s feature extraction step, based on the clustering of hypercolumns, as it is. Secondly, we tested how EBANO reacts to adversarial examples, images that have received small perturbations invisible to the human eye, but enough to trick the network to think that the image depicts a different object. In this case, the features extracted are completely unrelated to the image content and highly dependent on the neural network architecture. Changing adversarial image, but keeping the same neural network will produce the same set of features. Then, we trained different VGG16 networks to have unwanted biases, similar to those that might cause discrimination and other unfair practices and used EBANO to explain their prediction. In those cases, the nPIR index, an indication of how much a feature affects the final prediction, proved to be a good indication of how well the network has learned the correct features. The nPIR index is also, very easy to interpret, if a feature contains important characteristics of the predicted class, then it should be high, and vice versa. This makes it easy to assess if the network predictions are based on the correct features. A ‘airplane’ class predicted on the basis of a blue sky background is not what we would like the network to learn. We also, added the Pascal VOC 2012 dataset to the existing list of supported datasets. And finally, we created a website to show the results.