Modernising Service Assurance: Data migration from a legacy system to the cloud

This thesis, conducted in collaboration with Storm Reply, explores the design of a data model and the implementation of a cloud-based infrastructure for ingesting data from multiple legacy sources into a unified data model. This solution represents the foundational component of a broader initiative undertaken for a major telecommunications enterprise, aimed at modernizing its Service Assurance platform and enhancing system integration. Throughout the project, three principal challenges were addressed: - Development of Connectors for Heterogeneous Legacy Data Sources - Design of a Unified Ingestion and Normalization Module - Implementation of a Configurable Connector Mechanism As will be demonstrated in the following chapters, the aforementioned challenges were addressed through the adoption of advanced cloud technologies to host and support the new data platform. This approach ensures the efficient management of high data volume and throughput using state-of-the-art methodologies. Furthermore, the extensive suite of services offered by AWS was leveraged to facilitate seamless integration with the various legacy data sources, enabling a scalable, resilient, and cost-effective solution. The project was designed with a strong emphasis on \textbf{automation, collaboration, and efficiency}, following DevOps principles and an Agile development framework. By prioritizing continuous integration and delivery, the team ensured a streamlined workflow that allowed for rapid iterations, early issue detection, and consistent software quality. This methodology fostered adaptability to evolving requirements, enabling swift responses to feedback and seamless incorporation of improvements throughout the development lifecycle. This work highlights several key contributions to the field of \textit{data engineering and cloud-based infrastructure}. First, it addresses the complexities of integrating heterogeneous legacy data sources, emphasizing the need for efficient, high-performance data ingestion mechanisms. Second, it underscores the importance of a unified data model and centralized database, detailing the challenges of ensuring consistency, scalability, and fault tolerance in a high-throughput environment. Third, it presents a configurable and future-proof connector framework, demonstrating how flexibility in data integration can be achieved through dynamic configuration rather than rigid, predefined structures. Additionally, this thesis showcases the role of modern cloud technologies in enabling scalable, and maintainable data solutions. By tackling these challenges, this work contributes to the broader discourse on data infrastructure modernization in the telecommunications sector, offering insights that are applicable not only within said sector but also across industries facing similar legacy system constraints.