Development of a gym exergame using a low-cost RGB camera for digital health in telerehabilitation

The focus of this thesis is the development of an exergame prototype designed for the rehabilitation and telerehabilitation of the upper limbs. Development focused on creating a game set in a gym environment, using Google's Mediapipe library to detect the user's movements and spatial positioning. The exergame was built using the Unity game engine, based on an existing open source solution that integrates Mediapipe's pose detection via a Python script with a Unity sample scene. From this, the solution was further developed by incorporating a server-side component, consisting of a NodeJS server and a MongoDB database, to store all data collected during the exercises. Real-time data is transmitted via websocket from both the Python code and Unity game engine, while CRUD operations, such as user login and registration, recording successful repetitions and errors, are managed through REST APIs. Additionally, a web user interface (UI) was implemented to enable users to read and download all significant data collected, with filters available for user name, date and time, session, or specific exercises. The architecture of this solution is designed to centralize the server, allowing multiple clients to connect simultaneously. This is especially important for telerehabilitation, enabling the possibility to retrieve data from users performing exercises remotely. The exergame features a range of exercises focused on lifting the upper limbs, both laterally and frontally, in various modes: single lifts, alternating lifts, or simultaneous lifts. Users receive positive reinforcement through audio and visual cues during exercises. Additionally, the game can be used in an assisted mode, where game prompts can be guided by an operator via keyboard input, further allowing customization. To determine if a repetition is performed correctly, the game assesses the angle between the user's arm and the corresponding side of their torso. When the correct angle is achieved, the repetition is marked as correct. The game evaluates also the angle between the arm and the user's shoulder line; if the user fails to maintain the correct plane of movement, the game signals an error. The objective of this study is to demonstrate that a markerless solution, utilizing an integrated webcam or simple RGB camera, can achieve a reliable and sufficient level of precision for the execution of rehabilitation exercises. During the development phase, data was analyzed to verify the accuracy and frequency of data recording. Furthermore, the study highlighted the accuracy of Google's Mediapipe pose detection. The system performed well under optimal lighting conditions, where the subject was evenly exposed and the background had high contrast. Conversely, performance declined in suboptimal lighting conditions, such as when one side of the subject was overexposed or the background contrast was poor.