As speech-based technologies such as virtual assistants become increasingly present in our daily lives, it also increases the need for transparency and interpretability in these systems due to concerns regarding transparency of their decision-making processes. Deep learning has significantly enhanced the performance of Automatic Speech Recognition (ASR), but also turned them into black-boxes, increasing their complexity and opacity. This thesis addresses these challenges by applying Explainable AI (XAI) techniques in the context of ASR systems, aiming to move to a new granularity, shifting from the current word-level explanations to phoneme-level explanations, trying to find the contributions of sub-word units, which has been unexplored until now.