The advent of cutting-edge medical technologies and telehealth services has resulted in an explosion of health-related data, highlighting the urgent need for efficient data annotation in healthcare research. Manually labeling video footage to identify specific actions or features in medical imaging is both time-consuming and requires specialized expertise, causing significant delays in research progress. This thesis addresses this challenge by focusing on the annotation of upper limb movements in egocentric video data. It introduces an innovative minimally-supervised deep learning system designed to streamline this process. The proposed framework analyzes video recordings from head-mounted cameras capturing individuals performing everyday tasks. Central to the system are two key components: the Hand Object Detector (HOD) and the Snorkel model.