Hand gesture analysis for rehabilitation

This project examines hand micro-gestures performed on the musical keyboard, analyzed through the MediaPipe Hand neural network, to evaluate the potential of the keyboard as a tool for improving manual dexterity. Four simple movements were selected for the experiment to ensure that all participants, regardless of their prior experience with the keyboard, could effectively complete the tasks. The experimental protocol involved 35 participants, of whom only 18 were considered for the study due to the exclusion of outliers. Among these, one hand was designated as the reference, because it executed the movements correctly. The participants performed the selected movements on a 32-key musical keyboard designed to mimic a standard piano layout. An RGB GoPro Hero 7 camera, able to record high-resolution video, was positioned above the keyboard to capture the hand gestures from a top-down perspective, ensuring a clear view of the participants' movements. Before the experiment, participants completed a questionnaire that included details about their experience with the keyboard and other musical instruments, their dominant hand, manual dexterity, and any previous trauma that might have affected their hand movements. This information was crucial for contextualizing the data and understanding the different levels of skill among the participants. The recorded videos were processed using the MediaPipe Hand neural network, which identified the x, y, and z coordinates of 21 landmarks on each hand for every frame of the video. To accommodate variations in execution times, a resampling method was applied to standardize the timing of movements across participants, allowing for a more accurate comparison with the reference hand. In addition, normalized lengths of the five fingers and angles between adjacent fingers, or curvature angles of the fingers, were calculated to evaluate the precision of the movements. A clustering method was applied to classify the hand movements into distinct clusters. The analysis of finger lengths and angles demonstrated variability, leading to the formation of two different clusters to distinguish the precision levels of the movements. In general, the results suggest that the keyboard could serve as an effective tool for improving manual dexterity through accessible hand movements, making it a suitable instrument as support for traditional medical rehabilitation.