Diego Chiavetta
AI-Augmented Data Pipeline Observability - A Framework for Anomaly Detection and Automatic Root Cause Analysis on Databricks.
Rel. Paolo Garza. Politecnico di Torino, Corso di laurea magistrale in Ingegneria Informatica (Computer Engineering), 2026
Abstract
Data pipelines compute the indicators on which organizations act, and they fail in two ways. Sometimes they abort, and the failure is visible. Sometimes they complete successfully while writing corrupted data, and nothing signals it until a decision has been taken on the result. In both cases the costly part of an incident is not noticing it but locating its cause: a fault introduced where data enters the system reappears several transformations later as a deviation in an aggregate whose relation to the original fault is no longer apparent. The mechanisms available on a Databricks detect such symptoms and expose the dependencies among tables, but they do not correlate the two into a localized cause, which remains a manual task for an engineer who knows the pipeline.
This thesis presents a framework that automates that step for Medallion pipelines on Databricks
Relatori
Anno Accademico
Tipo di pubblicazione
Numero di pagine
Informazioni aggiuntive
Corso di laurea
Classe di laurea
Aziende collaboratrici
URI
![]() |
Modifica (riservato agli operatori) |
