Posture Check: Detecting and Evaluating Compensatory Movements in Robot-based Stroke Rehabilitation through Human Pose Estimation and Machine Learning

Stroke stands as a primary source of disability, often resulting in significant motor control deficits. Motor recovery post-stroke is partly attributable to spontaneous healing and neuroplasticity. Augmenting these natural processes with specialized physical therapies can optimize recovery outcomes. In rehabilitation, the integration of robotic devices is particularly promising, offering substantial enhancements in the efficacy of rehabilitative measures. Notably, a recently investigated innovation in this domain is the use of a robotic end-effector arm. This thesis is about the development of a framework that leverages video-based human pose estimation and machine-learning algorithms to predict compensatory movements performed by post-stroke patients during therapy with a robotic end-effector arm. We collected video recordings from sixty subjects while using a robotic end-effector arm during their therapy. Each participant was instructed to participate in a series of ten video games designed to involve different combinations of movements and various robotic guidance modalities. Six clinical researchers analyzed the video recorded and identified all compensatory movement strategies by the participants. A common compensatory movement strategy is the utilization of non-affected limbs or the overuse of secondary muscles to maintain functionality and perform tasks, often resulting in altered movement patterns and postures that can potentially lead to secondary musculoskeletal issues. The six clinicians recorded the timing and intensity of compensatory movements during the study. High-intensity frames, representing the peak of each compensation, were identified. To precisely analyze these frames, an automatic human pose estimation technique using the OpenPose model was applied, resulting in 25 skeletal joint coordinates (x, y) for each frame. From the extracted coordinates, we derived pertinent features (e.g., angle and distance between right shoulder and neck keypoints), hypothesizing their significance in capturing characteristics essential for identifying compensatory movements. We engineered a hierarchical model comprising multiple random forest classifiers, each trained and assessed for their proficiency in predicting compensatory movement strategies. The initial model discerns compensatory from normative movements, the subsequent one differentiates single from multiple compensatory movements (i.e., when simultaneous motion strategies occur), and the final tier classifies various single compensatory movements. Additionally, we evaluated the applicability of the model trained for single compensatory movements in scenarios involving multiple compensatory movements by analyzing its prediction output scores. Specifically, it is possible to differentiate compensatory movements from normative movements (with an accuracy exceeding 95%) and classify single compensatory movements (with an accuracy exceeding 90%), whereas distinguishing single movements from multiple ones (accuracy exceeding 75%) and classifying the latter presents a more intricate challenge. As a follow-up study, we are already dedicated to further enhancing the prediction of compensatory movements during the motion of a robotic end-effector by harnessing the capabilities of 3D human pose estimation algorithms (e.g., LightBuzz), allowing for more precise predictions and the capacity to anticipate new forms of compensatory movements. Subsequent empirical data will be required for definitive conclusions.