DEEP LEARNING DENOISING OF TIME-OF-FLIGHT DEPTH IMAGES AFFECTED BY RETRO-REFLECTIVE MARKER INTERFERENCE TO ENABLE CONCURRENT MARKER-BASED AND MARKERLESS MOTION CAPTURE

The work presented in this thesis can be divided into four main sections. • The first section focuses on data preprocessing. Irrelevant background in formation was removed from the raw depth images from the Azure Kinect acquisitions to reduce meaningless variability, and a fixed size square region of interest (ROI) containing the subject was extracted to standardize the input for the training of the model in the following steps. • The second section covers the development and validation of a DL framework designed for removing IR interference from depth images. Here, A latent-space generative diffusion model and a companion autoencoder capable to encode depth images into (and reconstruct them from) the same latent space in which the diffusion model operates, were trained on the preprocessed dataset to generate meaningful synthetic images of patients performing motor tasks analogous to those included in the construction set. An inpainting algorithm is implemented to guide the image reconstruction by forcing the model to only overwrite the damaged areas of the images. This section also introduces an algorithm developed to specifically detect and isolate image areas affected by IR marker artifacts. • The third section aims to evaluate the impact of our denoising model on the accuracy of skeletal tracking when integrated into the standard Kinect SDK pipeline as a preprocessing step, exclusively performing preliminary artifact removal on the depth channel. The results showed no improvement, as the Kinect body tracking model (K4ABT) appeared largely indifferent to modifications in the depth channel • The fourth section describes a parallel study conducted in collaboration with a team of researchers at ETRO (VUB’s department of Electronics and Informatics), focused on on getting insights on the azure Kinect skeletal tracking. The investigation found out that when the depthmap is missing, flipped, or corrupted, the Azure Kinect skeletal tracking algorithm mainly relies on the IR stream instead of the depthmap. A simple inpainting method to fill-in the missing pixel values in the IR image based on the value of the surrounding pixels has shown a drastic improvement in Skeletal Tracking Stability