Evaluating Large Language Models as a diagnostic support tool in the Pediatric Emergency Department

This research focuses on developing a method to assess the use of Large Language Models (LLMs) as a diagnostic support tool for medical professionals. This work has been carried out in collaboration with LINKS Foundation and the Pediatric Emergency Department of the Regina Margherita Hospital in Turin. The Pediatric Emergency Department faces significant challenges, including high patient volumes, time-sensitive decision-making, and the complexity of diagnosing a wide range of conditions. In this demanding environment, the use of artificial intelligence, particularly LLMs, may be a powerful resource thanks to the possibility to analyse large volumes of clinical data and providing timely responses. In fact, LLMs are rapidly emerging as a valuable asset in the medical field and present a significant opportunity to enhance the quality of patient care. Despite the potential of LLMs, it remains uncertain whether these tools can be effective in supporting physicians in the diagnostic process, and what is their ability to evaluate clinical cases. The lack of studies comparing LLMs with existing traditional diagnostic approaches limits the understanding of the role these models could play in pediatric clinical practice. The aim of this study is to evaluate the performance of LLMs on realistic scenarios of a Pediatric Emergency Department, exploring the potential use of chatbots as an assistant for clinical decision-making. Initially, the characteristics, structure, and various types of LLMs currently available are analysed to identify the most adequate models for the study. The selected LLMs include both models accessible through a web interface, and models that can be deployed on-premises. Clinical records from an Emergency Department of a Level II Children's Hospital are reviewed to select 80 real-life cases involving diverse pathologies of varying complexity. These cases are then structured into clinical vignettes, and submitted to a panel of physicians who independently classify them into different levels of difficulty. For each clinical case, both the LLMs and a group of medical specialists are asked to provide a primary diagnosis and two differential diagnoses. The responses are compared with the final diagnoses assigned upon patient discharge from the emergency room or after hospitalization and are independently evaluated by two experts. Our results show that, particularly for easier questions, the chatbots achieve a good performance in suggesting a diagnosis for the clinical cases, demonstrating an understanding of the clinical scenario. For medium- and high-complexity cases, the chatbots' performance generally degrades, especially for smaller-sized models. However, some of the considered models manage to match the majority of the correct answers even for the clinical cases classified as difficult. The promising outcomes achieved provide new insights into the integration of LLMs in pediatric clinical practice, highlighting their potential as diagnostic aids for physicians, while also underscoring the need for further research to optimize the use of these tools for healthcare scenarios.