In recent years, Deep Learning models have improved the state of the art in several domains. In particular, Convolutional Neural Networks have arguably become the go-to solution for Computer Vision, thanks to their ability to automatize the feature extraction process in image classification tasks. However, the drawback of these models is that they are black-boxes: that is, the knowledge acquired in training is fully sub-symbolic and thus these models retain no human-understandable representation of their inner decision processes. There is an entire research field working towards describing the rationale behind AI decision-making, called Explainable AI (XAI). However, when it comes to Computer Vision, what all XAI techniques typically produce are heatmaps that measure the correlation of each input-pixel to the classification label.