LLM-Based Multi-Agent Recommender System Design and Implementation: A Case Study in Public Tenders

This thesis investigates the practical implications of Large Language Models (LLMs) in the design and implementation of Multi-Agent Systems (MAS), with special attention to their integration into Recommender Systems (RecSys). Specifically, the focus is on the design, development, and evaluation of a Multi-Agent RecSys architecture built upon three core principles: reusability (implementable in different use cases and domains), scalability (ready to serve a large audience), and explainability (transparency at every step). At the core of the pipeline, orchestration policies ensure the asynchronous scheduling of agent tasks across modules: data gathering, data extraction, user profiling, and recommendation generation. LLM instruction prompting adopts a schema-driven (JSON) approach to produce observable and debuggable outputs. These typed interfaces ensure the reuse of modules across different domains, while self-correction loops (e.g., critique-refinement-retry) and self-consistency checks are in place to reduce hallucinations and ensure data consistency. Conceptually, the retrieval stage with a downstream LLM as evaluator is inspired by a Retrieval-Augmented Generation (RAG) approach, where retrieved content is submitted to the agent's judgements and explanations. Candidates (items to recommend) are retrieved using a hybrid methodology. BM25 lexical matching and embedding-based semantic similarity scores are fused using the Mean Reciprocal Rank (MRR). A graph component captures complex, higher-order connections and contextual dependencies between users and content, improving retrieval accuracy and contributing to the final ranking. The retrieval process is further strengthened by an adaptive feedback learning mechanisms: a multi-vector representation of user interests captures heterogeneous and evolving preferences, effectively mitigating overgeneralization. Downstream of this pipeline, an agent compares the user profile with the retrieved items, describing and explaining why a content is classified relevant for the user by the system. End-to-end observability artifacts (logs, reasoning traces) ensure auditable and reproducible steps. Finally, the system was empirically validated on a case study focused on the recommendation of public tenders. Ablation analysis showed that the integration of an LLM-based agent as the final judgment layer significantly improves ranking performance (e.g., higher NDCG, MRR), outperforming baseline hybrid retrieval methods in terms of relevancy. Overall, the schema-driven and reusable multi-agent design ensures straightforward adaptability of architecture to other application domains — such as job-candidate matching or product recommendations. This suggests that LLM-based MAS can deliver transparent, scalable, and domain-agnostic recommendation capabilities, even in sensitive contexts.