Development and evaluation of a Machine Learning pipeline for the generation of video annotations

The rapid growth of machine learning techniques has promoted innovative advancements across various domains, including health, medicine, and rehabilitation. However, the effectiveness of these methods heavily relies on the availability of large and well-annotated datasets. In the realm of medicine, a common challenge lies in the existence of extensive datasets, which often lack comprehensive labeling. This limitation hampers the progress and deployment of automated algorithms across various medical applications. This thesis addresses the challenge of unlabeled video datasets by proposing a novel approach for generating automatic labels in the context of monitoring the upper limb activity in stroke patients. Leveraging developments in deep learning and computer vision, the proposed framework extracts relevant features from video sequences and inputs them into Snorkel to generate labeled data of hand activity. By utilizing weakly supervised learning techniques, the framework is designed to effectively learn from limited annotated samples and generalize to unlabeled data. The study begins by exploring state-of-the-art machine learning architectures that can learn from scarce data and focus on weakly supervised machine learning along with the generative model used by Snorkel. To evaluate the effectiveness of the proposed approach, a comprehensive dataset of upper limb activity of stroke patient video recordings is analyzed, and quantitative and qualitative assessments are conducted to compare the performance of the automated label generation framework against manual annotations. The metrics include accuracy, Intersection over Union, F1-score, and confusion matrices. Visual comparisons of generated labels and ground truth annotations provide insights into the system’s interpretability. Overall, the pipeline achieved an F1-score of 76%. The results of this study offer an effective solution to the issue of limited labeled data in stroke patient video analysis. The proposed framework showcases the potential of harnessing the growth of machine learning in rehabilitation, even when confronted with large unlabeled datasets. Furthermore, the methodologies developed can serve as a blueprint for addressing similar challenges in other medical domains requiring video data analysis with limited labeled samples.