Enhancing BPMN Exercise Evaluation: Expanded Solution Spaces and Advanced Validation Frameworks

The recent advancements in Large Language Models (LLMs) have opened new possibilities for the automation of software and process modelling. This thesis investigates the capability of AI systems to generate and interpret Business Process Model and Notation (BPMN) diagrams, with the aim of assessing their sufficiency, syntactic correctness and structural coherence. Although BPMN offers a standardized visual language for describing business workflows, creating and evaluating these diagrams manually is still a demanding and error-prone activity. To tackle this problem, four different AI models (ChatGPT, Copilot, Gemini and DeepSeek) were evaluated through a structured framework inspired by the COPE (Context, Objective, Prompt, Evaluation) methodology. Fifteen BPMN exercises, collected from diversified business and operational sectors, were analysed using two complementary scoring systems: one designed to quantify the relative difficulty of each exercise, and another developed to evaluate and compare the accuracy of AI-generated diagrams against reference solutions. Each exercise was solved, and the resulting diagrams were then evaluated both qualitatively and statistically through the Kruskal-Wallis and Wilcoxon Signed-Rank Tests. The statistical analysis showed that Gemini achieved the highest score and most consistent performance, followed by ChatGPT, while Copilot and DeepSeek showed less reliable results. The qualitative analysis revealed that syntactic precision does not necessarily ensure semantic completeness, highlighting the significance of contextual comprehension when generating BPMN diagrams. In conclusion, this thesis provides an empirical framework for evaluating the AI-generated process models and demonstrates that the reliability and understanding of AI-assisted process modelling can be improved by combining structured scoring and statistical validation.