Evaluating the impact of few-shot learning approaches on a cloud based agentic generative AI platform for Text to SQL

Abstract: The evolution of Generative Artificial Intelligence (GenAI) has led to a growing demand for scalable and efficient platforms for managing and executing advanced models. This thesis explores the design and implementation of a cloud-based GenAI Platform, consisting of an Agentic system of Large Language Models, highlighting the architectural challenges, technologies used, and best practices proposed by IT consulting firm Target Reply to one of its clients. The purpose of the system will be to transform questions asked in natural language into formal queries in SQL language, in order to facilitate data analysis activities within business processes. This thesis will explore the approach that allows the transition from the specific technical vocabulary used by the client in their requests to the rigid schemas of SQL databases. A key aspect of the research will be to validate the effectiveness of the system within a business environment, where there is a very low margin for error: it will propose several metrics to evaluate the accuracy of the results provided by the agent, and it will study the impact of the “Few-Shot Learning” strategy on the accuracy of the model. “Few-Shot Learning” in the context of LLMs consists of providing the model with examples of the task it will have to perform within its prompt. These examples are extracted from a pre-established collection so that they are as similar as possible to the task assigned at the time. In the context of Text-To-SQL, these examples correspond to pairs of questions and queries based on the same schema that will be queried. It will show how the approach used, combined with the use of Few-Shot Learning, makes an innovative system such as the one presented a suitable tool for business applications, thanks to its accuracy of over 90%. It will also discuss the scalability of the system to broader use cases and databases containing a large number of tables and columns.