Monitoring the functional use of the upper extremities (UE) in real-life settings in a 'Smart Kitchen'

For upper limb prosthetic users, monitoring the functional use of the upper extremities (UE) in practical situations is essential to maximize recovery interventions. The precise tracking of neurologic and functional recovery, however, can be hampered by the distance between a patient’s home and the clinical settings. The International Classification of Functioning, Disability and Health’s performance domain, which refers to how people with UE amputations and disabilities carry out activities of daily living (ADLs) and instrumental (iADLs) in their current environment, must therefore be captured by outcome measures of UE function that have been developed and validated. To fill this gap, a Smart Kitchen has recently been developed at EPFL, at the TNE laboratory, in the Campus Biotech (Geneva). The Smart Kitchen is a fully equipped and functional kitchen, whose goal is to simulate the home environment and carry out experiments with healthy volunteers and individuals with UE impairments in a controlled setting. In this context, the goal of the current study is to develop a monitoring system of natural and prosthetics hands during tasks generally performed in a domestic environment: this way, it is possible, with subsequent analysis, to evaluate and investigate how much the prostheses impact on the everyday life of patients. The monitoring is performed using egocentric cameras (Microsoft HoloLens and GoPro) and wearable sensors (IMUs). The frames extracted from the camera are processed and manipulated via a Convolutional Neural Network (CNN), to detect natural and prosthetic hands through the definition of region of interests. The CNN-based model was trained to detect three different prosthetic hands: IH2 Azzurra, Bebionic hand and Bebionic hand covered by a painted glove to mimic the natural hand. The performance evaluation in detecting natural hand-object and prosthetic hand-object interactions was conducted on a dataset collected in the Smart Kitchen (one healthy subject and one UE amputee) and on dataset collected from the Prostheses Centre (INAIL) in Budrio (five UE amputees). Hololens and IMUs were used to detect the cooking activity evaluating different parameters, such as the smoothness and the frequency of cutting. Further investigation was conducted to determine whether the Hololens data could be used alone to characterize the cooking activities. Results showed that the CNN could accurately detect the Bebionic hand covered with the painted glove (F1 score=81.1%) but failed to detect the prostheses in the other two cases, because of the visible metal parts, as well as the difference in the shape and color from natural hand. The original CNN could detect the presence of hand-object interactions with excellent performance on both dataset (F1 score=90%). In conclusion, the integration of this system in a home environment allows to maximize and improve the development of new devices that adapt to everyday life and facilitates the follow up of UE amputees after the surgery.