Active learning limits manual labeling costs by selecting a small number of samples from a large pool of unlabeled data in an iterative fashion. At each step, active learning aims to query the unlabeled samples that maximize the performance gain of the model. Active learning can be applied in deep learning to limit the amount of data required by deep neural networks. This work focuses on a supervised learning problem, namely multi-class image classification, in the context of deep active learning (DAL) using convolutional neural networks (CNNs). Deep neural networks are inherently black-box. Explainable AI (xAI) provides tools to locally explain the reasons behind predictions and understand how the model operates globally.