Empowering Machine Learning Workflows with AWS: Engineering Algorithms for Scalability and Efficiency

The aim of this thesis is to the design and implement a Machine Learning Operation (MLOps) model for large enterprise BigData and Machine Learning algorithms management on cloud. MLOps is the process that moves machine learning models into production environments. It unifies data collection, preprocessing, model training, evaluation, deployment, and retraining in a single process that teams can maintain and continuously monitor. Specifically, this work got started from a customer project in the Energy Sector with the need of using real data and actual machine learning algorithms provided by the customer. The data elaboration and model learning ask for a tailored strategy to exploit cloud services (Amazon Web - AWS) and guarantee security and scalability. The work mainly focused on migrate pre-existing algorithms from an outdated environment to one that is more modern, more efficient, and has more functionality. Another goal of this work is to take advantage of the modularity of this new tool to be able to create multiple flows by rewriting a minimal amount of code. A thorough comparison with pre-existing approaches on the basis of speed, resilience, ease of use metrics showed the advantage of the proposed strategy, that combines the minimum time reduction of the order to the 50% and the resiliency provided by the Endpoints. For the future we are planning to expand the workflows by supporting other types of procedures, and to improve the automatic deploying of the models.