Passive Tomography - Tools for extracting three-dimensional information from monocular video

Animals under study are commonly affected by experimental conditions. Moreover, research often goes towards relatively invasive questions, which leads it to adopt invasive methods. For instance, the vascular structure of the brain is a popular topic in research. If we want to follow the blood flow in order to analyze the brain activity, we are forced to adopt invasive techniques, such as the fMRI, which require the animal to be restrained or anesthetized. It is well known that anesthesia affects the acquisition of physiological data such as the heart rate, the breath rate, and the gut movements, as well as restraining the animal. Investigation into mice is further combined with the use of new imaging techniques. Short-wavelength infrared region imaging (SWIR) is an innovative technique for pre-clinical and clinical imaging. Currently, SWIR researchers are working to produce new detection systems, and new contrast agents to perform imaging of mice which are awake and unrestrained. For example, vascular contrast agents can be used to visualize blood vessels through the skin of an animal, which can allow us to follow the blood flow. Since the mouse is unrestrained during the acquisition, we can obtain physiological parameters much more representative than the animal under anesthesia. Our goal is to track vascular networks in a mouse over time without needing to perform invasive imaging at each time point. By performing SWIR imaging, it may be possible to extract this information as a mouse moving in its cage in which it may show us various parts of its body and we may collect images over time. Is it possible though to correlate those images with a known 3D model of what the mouse looks like? In this thesis project, we focus on methods for extracting information about the mouse movements over time. First, we investigate how photogrammetry works, state-of-the-art of 3D reconstruction, and how far our research can go exploiting its algorithm and computational geometry. We generate a surface mesh and texture of a mouse, using photogrammetry and SWIR images of a mouse with a vascular contrast agent. In this study, we found out that static scenes and non-deformable objects were adequately reconstructed, but the results changed with objects in motion. Therefore, we divide our efforts into two categories: rigid and deformable bodies in motion. The skull of a mouse is a rigid body that, even if it moves, the distance between some selected points is fixed in 3D space. In a video, the points represent the projections of the 3D model in 2D. We perform motion tracking by using a deep learning tool called DeepLabCut to estimate the skull movements over time, a technique known as pose estimation. We found out that the orientation of a rigid body can be easily extracted if we know the 2D projections of its points and how they are related in 3D space. If the object is a deformable body in motion, such as fingers in a moving hand, the distance between its points may change over time. In order to compensate for the body deformation, we perform stereo vision, a process of extracting 3D information from multiple 2D views. We adopt a mirror to have a second view of the object, we track selected points and we exploit the relation between object and its reflection to triangulate those points. We finally obtain a preliminary 3D deformable object, complete in all its parts.