MLOps and LLMOps: Development of an LLM-based application with focus on toxicity evaluation and scalable cloud deployment

In this project, the Proof of Concept (POC) of an LLM-based application was designed aiming at accelerating the adoption of conversational AI agents. More and more frequently in this period, companies find themselves wanting to develop conversational AI-based applications, also due to the current boom of LLMs-based technologies, and don’t have the capabilities to develop it from scratch. This POC aims at solving this problem with a highly and easily customizable platform for each use case companies want to cover. Taking inspiration from the paradigms of MLOps and LLMOps and leveraging the Italian open source framework Cheshire Cat AI a conversational AI agent application was developed with a focus on prompt toxicity detection and context precision evaluation. The platform is highly future-proof as the framework it was built on is expected to be easily extended, allowing the current platform to evolve during time and adapt to more specific use cases. The POC also included the deployment on a cloud provider via infrastructure-as-a-code (IAAC) tools, including a framework to deploy multiple versions of the application via the canary deployment paradigm.