An end-to-end Data Engineering Architecture in the energy domain

The idea behind this thesis work is to create an end-to-end Data Engineering use case for Bamboo Energy, a tech start-up operating in the energy sector. The development of this work has been made possible firstly doing an internship and then being included in the product team. The overall research and implementation process can be conceptually divided into four different phases. The first step consisted in the study of the high level processes of Bamboo Energy, its main products and its features, its placement in the industry sector and the figures involved in the energy paradigm. After, the focus has been moved into topics related to Data Engineering, covering ETL and aggregation pipelines, Orchestration and Alerting processes. The goal was to make a snapshot of Bamboo Energy situation related to this fields, in order to identify strengths and weaknesses, to have a base from which begin a research and exploration work. The third phase can be identified in the pure exploration of possible new paths, comparing and studying different technologies. Finally, among different solutions, some have been chosen to be implemented, utilizing, as methodology, product and budget constraints, platform and cloud environment compatibility. The outcome is a very practical end-to-end use case covering aspects from data sources ingestion to transformation and aggregation pipelines, from processes Orchestration to Alerting policies.