Gesture recognition and performance analysis in piano practice using morphometric measures and machine learning

This study aims to answer two research questions related to piano performance. The first investigates the ability of a specific set of morphometric features to represent the performance of participants in the execution of piano gestures, allowing machine learning models to distinguish subjects according to the gesture performed. The second explores the effectiveness of the same set of features in discriminating subjects according to their skill level in performing piano gestures. To this end, an experimental protocol was designed in which 32 participants performed four simple piano gestures, recorded by an Intel RealSense SR305 camera. After a qualitative screening, only the videos of 27 participants were retained for the analysis. In addition, a self-assessment questionnaire was administered to collect personal information and data on piano proficiency, later used for both result interpretation and supervised model training. Analysis of the videos was conducted using the MediaPipe Hand model, which was applied to detect 21 hand landmarks. To obtain a more reliable estimate of the z-coordinate, the spatial information was integrated with depth data. The landmark coordinates were post-elaborated to compensate detection errors, providing a more accurate representation of the hand skeleton during gestures execution. Given the variability in performances duration, each video was subsampled by selecting 10 frames per gesture. Seven morphometric features were computed for each frame to characterize hand contraction, finger curvature and movement stability. The resulting data were aggregated into vectors and used as input for five machine learning techniques. The analysis proceeded in two directions: in the first case, the feature vectors were used to train classification models (Logistic Regression, Support Vector Machine, k-Nearest Neighbour and Random Forest) to distinguish the gestures performed. In the second case, the possibility of classifying participants according to the level of piano performance ability was explored, also using a hierarchical clustering algorithm. The latter analysis was carried out considering two different subdivisions of the self-assessed piano proficiency: a five-level and three-level scale. The results show that the feature set is more effective in classifying the types of gestures performed than in distinguishing participants according to their level of piano-performance ability, where accuracy remains below 50%. Specifically, classification models achieve better performance when the number of gesture classes is reduced, because gesture simplicity and morphological similarity reduce feature space differentiation between classes, making classification more challenging. Regarding the classification of subjects based on piano proficiency, the three-level division produces marginally better results than the division into five, due to the simplicity of the proposed gestures, which limits the models’ ability to discriminate among multiple levels. In conclusion, the study highlights the potential of the proposed feature set in recognizing subjects based on the type of gesture performed, demonstrating the possibility of distinguishing even simple movements. However, discrimination based on piano proficiency is still limited, suggesting the need for refinement of the feature set. Extending the analysis to more complex piano gestures could also be a promising direction to enhance the model’s accuracy to capture differences in performer skill level.