Analytics Ingestion Service: a data ingestion pipeline for MLOps

With machine learning at the core of detection systems, the ability to systemically capture, store and analyze the outputs the model generates has become critical for operational efficacy and model governance. This thesis presents a MLOPS pipeline, the Analytics Ingestion System (AIS), developed to collect and persist rich analytics from the Machine Learning powered cybersecurity workflows within the Sysdig ecosystem. AIS offers a model-driven approach to managing inferences, so that any prediction, batch or real-time, is logged, contextualized, so that it can be used for further analysis. The platform has also been designed for scalable storage, message brokers and orchestration tools to allow for modular, cloud native deployments. Particular attention is paid to observability and traceability of the model outputs, which allows good audit and data collection best practices across the different versions of the model and input sources that it receives. By modelling model inference as a first-class analytical event, the AIS provides a foundation for more elaborate analyses downstream, such as between model comparisons, data and concept drift detection, and A/B testing. In this design, the AIS provides a crucial common carrier for ML-driven security systems at Sysdig that demand with more than just inference and a production quality infrastructure, to get intelligence from those inferences.