Body pose estimation in sport science based on sensor fusion algorithm of multiple RGB cameras

The computer vision field includes tasks like image classification, object recognition, and feature detection. The focus of this work is body pose estimation, where the human motion is analyzed using human activity recognition algorithms through pattern detection principles. The increasing popularity of cost-effective mobile sensors like Microsoft Kinect has led to the development of various algorithms for activity recognition and tools that enable sport performance analysis and motion rehabilitation at home. These algorithms have the potential to promote a healthy lifestyle, discourage unhealthy habits, and aid in condition tracking, particularly in sports science and healthcare applications. Therefore, in this work we will use data collected with RGB cameras to detect and classify sports movements and exercises involving both the upper and the lower body. The first step of this work is camera calibration, an essential prerequisite in the world of 3D computer vision, performed using the OpenCV library in Python and a checkerboard. The body pose of the subjects will be estimated using the MediaPipe framework offered by Google, obtaining the skeleton of the subject seen from different orienta- tion. Afterwards, a set of ArUco markers will be used to estimate the pose and position of the cameras with respect to a fixed reference system, which will be used to rotate the 3D joints positions of the skeleton into the same reference system, in order then to fuse the data and obtain a more accurate and robust estimation of the body pose. Four different fusion methods will be exploited: mean fusion, Kalman filter fusion, mean fusion using data preprocessed using DBSCAN and Kalman filter fusion using data preprocessed using DBSCAN. The four different fusion methods will be then evaluated and compared using the motion capture system developed by OptiTrack as groundtruth (PrimeX13 cameras and Motive software). Finally, using the data coming from the most accurate fusion algorithm, the joint angles will be computed in order to build a dataset of 24 exercises performed by 4 subjects. The dataset will be then used to train, validate and test a Random forest classifier and a Multi-layer perceptron classifier. The results show a good improvement of the performances when using the data coming from the proposed sensor fusion method instead of the single cameras, resulting in a satisfactory accuracy in the classification of the exercises.