Design, Implementation and Evaluation of LLM-based Agents for Forensic Analysis

Large Language Model (LLM) based agents are increasingly adopted for the automation of complex tasks. In this thesis, I systematically study their capabilities and limitations in cybersecurity forensics. Building upon a publicly available cybersecurity benchmark, I designed and evaluated a modular multi-agent system for forensic analysis. I first addressed two fundamental limitations of LLMs: the lack of long-term memory and the inability to access up-to-date knowledge. To overcome these challenges, I added a semantic memory module for storing and retrieving information and a web search tool (RAG) for external knowledge retrieval. Leveraging these solutions, I then enhanced the agent architecture through iterative refinements. The initial design relied on a single monolithic agent, while later versions added specialized components, including a PCAP Flow Reporter and a Log Reporter for traffic and log analysis. I evaluate the impact of design decisions on tool integration and architecture to provide guidance for practitioners. I benchmark four agent architectures and six LLM backends on 20 incident scenarios of increasing complexity. I also test 10 incidents from 2025, reaching 80% CVE identification accuracy with the best architecture. Finally, a human study with 22 experts rated the agent’s reports as complete, useful, and coherent.