Evaluating Large Language Models in Software Design: A Comparative Analysis of UML Class Diagram Generation

This master’s thesis aims to assess the feasibility of utilizing Large Language Models (LLMs) to generate Unified Modeling Language (UML) Class Diagrams. UML is a standardized visual language widely used in software engineering to depict the structure and design of a software system, enabling clear communication and documentation of system components and their relationships. By comparing the LLM-generated diagrams with those produced by humans, particularly during the crucial initial phase of requirement gathering, the research assesses whether AI can support or enhance traditional software modeling practices. The study utilizes a two-part methodology complemented by a statistical analysis. In the first part, the diagrams are examined for syntactic (adherence to UML rules), semantic (accuracy of meaning and concepts), and pragmatic (usefulness and applicability) quality errors, in the second part the semantic distance between the generated diagrams and the given solutions is calculated by using a specific algorithm. The findings suggest that LLMs are capable of generating UML class diagrams with a level of syntactic and pragmatic quality comparable to that of human-produced diagrams, due to no statistically significant differences in these areas. However, the analysis also uncovers a noticeable gap in semantic quality, where human-generated diagrams outperform those generated by LLMs, highlighting the current limitation of LLMs in understanding the semantic nuances required to generate an accurate diagram. Moreover, the study finds that LLM-produced diagrams are typically more distant from the reference solution, emphasizing the challenges that LLMs face in domain-specific knowledge that human experts are able to bring in the creation of the diagram. With those findings the thesis contributes to the ongoing discussion on the role of AI in software engineering, showing that, while LLMs show promise in certain aspects of UML class diagram generation, there is a gap in semantic accuracy, underlining the necessity of further advancements in AI capabilities to reach the same quality level of human-generated diagrams.