Hand gesture recognition for user interaction improvement in AR Surgery

Gesture recognition has emerged as a promising technology in the field of surgical support, offering a novel means of enhancing communication and control during the preoperative phase and within the operating room. Surgeons and medical professionals are increasingly adopting gesture recognition systems to streamline their workflows and reduce contamination risks associated with traditional input methods; moreover, this interaction method is widely adopted for Augmented Reality devices such as the Head-Mounted Display (HMD) Hololens 2. This thesis aims to develop and evaluate various neural network models to determine the most effective one for supporting surgery planning and execution. By implementing models that take images and landmarks as inputs, the goal is to find the model with the best compromise between frame rate and accuracy. The requirements for the specific gestures to be adopted were provided by physicians of the Maxillo-facial surgery department of Molinette hospital in Turin and tailored gestures were defined to easily interact with the 3D model of the anatomical district employed to plan and execute the surgery. This project intends to pave the way to address the ease-of-use issues related to the user interaction with HMDs and foster the AR solution adoption in the healthcare industry integrating Deep Learning-based methodologies to fully support the surgeons during the clinical practice.