Augmented reality for surgical planning: integrating HoloLens 2 for 3D visualization and interaction with patient-specific anatomical models

Augmented reality (AR) is an emerging technology with significant potential to improve preoperative surgical planning by optimizing spatial understanding of patient-specific anatomy. This thesis presents the development and initial evaluation of an integrated system leveraging Microsoft HoloLens 2 for immersive 3D visualization and interaction with anatomical models reconstructed from thoracic computed tomography (CT) scans, specifically for planning lung biopsy procedures. The system architecture is based on an integration between the 3D Slicer medical image computing platform and a custom Unity application developed for HoloLens 2. Communication is facilitated by the OpenIGTLink protocol, enabling dynamic streaming of CT slice data and spatial transformations. Anatomical structures, including lungs, nodules, the bronchial tree, vasculature, and ribs, are automatically segmented from CT data using a nnU-Net, TotalSegmentator, within 3D Slicer and subsequently exported as 3D models. These models are imported into Unity and rendered as interactive holograms in the HoloLens 2 environment, controlled via a custom user interface based on the Mixed Reality Toolkit (MRTK). Key functionalities include dynamic loading and manipulation of 3D models, real-time synchronized visualization of CT slices through the holographic models, and simulation of optimal biopsy needle trajectories. This work details the system's architecture, the implementation of the 3D Slicer planning module, the AR application in Unity, and discusses the technical considerations for integrating these tools. Preliminary usability feedback from clinicians is also presented, highlighting the system's potential to improve preoperative planning accuracy and spatial awareness, and potentially reduce reliance on intraoperative imaging.