Ultrasound imaging is one of the most commonly used modalities in clinical practice due to its non-invasive nature and the absence of ionizing radiation. Such images are typically produced with well-established algorithms for converting acoustic signals into visual outputs. In recent years, artificial intelligence has experienced remarkable growth in the field of medical imaging, particularly through the application of deep learning techniques. These models have shown high performance and the potential to transform the field with novel methods for image reconstruction from raw data. However, their native lack of explainability is a major limitation, and issues regarding their reliability and clinical uptake are raised.