Deep Learning and Computer Vision methods for Augmented Reality solutions in robot-assisted laparoscopic surgery = Deep Learning and Computer Vision methods for Augmented Reality solutions in robot-assisted laparoscopic surgery

Objective: This thesis focuses on applications of Augmented Reality to support the surgeon during laparoscopic Robot-Assisted procedures. These applications’ goal is to augment the surgeon’s perception of the surgical scene, which is usually more limited than in open procedures. After extensive literature analysis, both in the current methodologies for 6D pose estimation and in the specific context of its application to augment reality in laparoscopic surgery, a novel methodology is here proposed for a real-time pose estimation strategy to superimpose the preoperative anatomical virtual 3D model of the kidney on the endoscopic view. Methods: The proposed methodology is divided into two subsequent steps. The first step is based on a Deep Learning approach and exploits two Convolutional Neural Networks, one to perform the organ detection to track the organ’s position and the other to solve the initial organ’s rotation inferring problem as a classification problem. The second step leverages Computer Vision Optical Flow to perform a pixel-wise tracking of the organ and maintain the superimposed virtual model correctly oriented in real-time. To demonstrate the effectiveness of this methodology, a software application was developed. Its goal is to help the surgeon to easily locate target hidden structures of the organ, as visible on its overlayed 3D model. Several tests of overlay accuracy were performed on real intraoperative videos to assess the application's efficacy, the results of which are described in detail in the text. Results: For the first step, two Convolutional Neural Networks were trained: a) the Segmentation Network was trained and tested on real endoscopic images resulting in good accuracy results. To assess the location tracking on real intraoperative videos, the position of the center of mass of the organ was tagged and compared with the kidney mask center. Results have proven that it is possible to estimate a correct trajectory of the organ when it is fully framed by the camera. b) the Rotational Network was trained on a custom-made synthetic dataset of artificial images of the intra-operative field, rendered in Blender with tagged small angle rotations. This last method showed to be accurate for artificial images, yet, since steady real frames with small angles were not available, it needs to be tested in the operating room. Nevertheless, this rotation prediction was hypothesized to be accurate enough to initialize the virtual model’s orientation in the very first anchoring with the real organ in ideal conditions. After the initial 6D pose is determined, the real-time tracking of the organ exploits Optical Flow algorithms. This step leverages frame-to-frame time information: every pixel displacement between consecutive frames is interpreted as axis rotation or movement, and, thanks to ad-hoc developed heuristics, allows to follow the organ and estimate its rotations thus keeping the augmented model registered on its real counterpart. To assess the registration correctness, the returned rotation vector of 20 frames was compared with the one obtained from a tagging tool that, by manually overlaying the virtual model, returns the ground-truth rotations and translation on the three axes. Testing results show that the heuristics correctly follows the rotation directions, with rotation errors under 10 degrees. This rotation prediction fails when considerable camera movements and wide rotation angles are introduced.