Conversational Agents for Immersive Virtual Learning Environments: Integrating GPT with VR for Healthcare Education

The integration of Virtual Reality (VR) and Conversational Agents (CAs) has transformed digital learning environments, offering immersive, interactive, and highly adaptable educational experiences. In healthcare education, VR has been widely recognized for its ability to simulate real-world medical procedures, while Conversational Agents (CAs) have emerged as powerful tools for providing personalized, context-aware guidance. This thesis investigates how novice trainees interact with a CA, specifically utilizing OpenAI’s GPT-4o-mini model, within a VR-based training system for intraosseous (IO) access, a critical emergency medical procedure used when traditional intravenous access is not viable. Specifically, the research aims to analyze user interactions, response quality, and usability feedback. To evaluate this integration, the study employs an experimental design in which participants engage with the VR training system and interact with the CA while practicing the IO procedure. Data collection includes conversation logs to analyze the frequency and nature of trainee queries, expert assessments of the CA’s response accuracy and completeness, and participant feedback through standardized assessment metrics. This multifaceted approach examines the CA’s role in guiding trainees, identifying the procedural steps that require the most support and offering valuable insights into the effectiveness of AI-driven assistance in medical simulation training. A key innovation in this work is the transition from a rule-based chatbot framework (Rasa) to OpenAI’s GPT-4o-mini model. This transition should enable, in principle, more natural language interactions, allowing trainees to ask open-ended questions and receive contextually appropriate responses. The CA is integrated into Unity, with OpenAI’s API facilitating real-time conversational interactions. Additionally, Meta’s Voice SDK is incorporated to provide speech-to-text and text-to-speech functionality, enhancing the realism of the training experience. Visual cues within the VR environment dynamically respond to user queries, reinforcing learning through multimodal feedback. Preliminary results suggest that trainees frequently rely on the CA for guidance at critical decision-making points. While the CA’s responses generally align with expert recommendations, limitations in response specificity and occasional latency issues highlight areas for future refinement. As data collection and analysis progress, findings may require further validation to account for additional experimental results. This research contributes to the ongoing development of AI-integrated VR training systems, with implications for broader applications in medical education. Future work will focus on refining response accuracy, optimizing latency, and exploring adaptive learning mechanisms to further enhance the CA’s role.