Automated Quality Assurance with Multi-modal Large Language Models

Quality Assurance (QA) is a critical aspect for a company whose product is an application directly used by customers. Traditional QA automation often struggles with dynamic interfaces and evolving scenarios, and it cannot effectively handle cases where identifying issues requires reasoning or contextual understanding. This thesis explores the integration of Large Language Models (LLMs) in a multi-modal setup to enhance QA automation, leveraging their contextual understanding of images and ability to extract meaningful information. The research focuses on three key phases: the experimentation phase to assess the current capabilities of state-of-the-art models in image analysis, the implementation of tests using real cases within the company's application, and the exploration of AI Agents to autonomously navigate the application and extract relevant information. The experiments are initially conducted on open-source datasets. Later, custom datasets are created using samples retrieved from the company's application to test the models' abilities in the specific use case of identifying bugs within the internal app. Given the promising results, specific scenarios within the application are selected for the implementation of tests powered by LLMs, with the aim of ensuring the ongoing consistency of both data and layout within the application for cases that cannot be handled by traditional QA. Experiments demonstrate that LLMs can significantly improve QA automation by reducing manual intervention, and the production implementation highlights the feasibility of integrating LLM-driven tests into existing workflows. The AI Agents are directly tested on the implemented tests within the application to assess the feasibility of replacing time-consuming and sometimes complex step implementations with an autonomous agent navigating the app. The Agents demonstrate both surprising abilities and limitations, but these emerging tools show great potential for the future. This work contributes to the AI4AI field, a key topic in the AI world and a central theme in the company's leading innovations, such as Aily. In particular, the thesis demonstrates the potential of LLMs to revolutionize QA testing by enabling more efficient workflows, handling more complex tasks and producing more accurate outcomes.