Application of Large Language Models in Software Testing: An Analysis of Method-Level Bug Detection

Software testing is a crucial phase in the software development lifecycle, essential for delivering secure and reliable software systems. Detection of software bugs is critical in order to deploy robust systems to end users, avoiding unexpected behaviors and maintaining high software quality. However, traditional testing methods frequently rely on manual effort or static rule-based tools, approaches that can be time-consuming and resource-intensive. With the advancement of Artificial Intelligence (AI), Large Language Models (LLMs) have emerged as a breakthrough, demonstrating impressive capabilities in various scenarios, such as machine translation, text summarization and natural language understanding. Motivated by these promising results, researchers have begun exploring the potential of LLMs for a wide range of software engineering tasks, including applications within software testing. This thesis explores the application of LLMs to the task of method-level bug detection in source code. Specifically, it examines the influence of different prompting scenarios, including zero-shot and few-shot approaches, on six state-of-the-art open-source LLMs, and it investigates the impact of enriching model prompts with additional context, such as raised exceptions and method call graphs. The evaluations, conducted on the widely adopted Defects4J dataset, offer valuable insights into how model size, prompting strategy and contextual information affect bug detection performance. Overall, this research provides a comprehensive analysis of LLM-based bug detection, highlighting the potential and limitations of these approaches and identifying directions for future work.