Deep Computer Vision for Human-Robot Interaction availability evaluation

The introduction of social robots in human-populated environments generates many different challenges to study for a safe and robust collaboration. To support the humans in different collaborative tasks, an autonomous robot should be able to recognize where target objects and persons are and, nonetheless, whether any person is available for interaction. The application of Deep Learning methods in the field of Computer Vision led to relevant improvements in object detection, image segmentation and other meaningful perception tasks. The aim of this thesis is to make use of state-of-the-art convolutional neural networks to build a real-time software system that allows a robot to recognize humans and objects in real-time using a simple RGBD camera, and exploit this information to estimate if the detected humans are available for human-robot interaction based on a scoring system. The system makes use of two neural networks to acquire human poses and object pixel masks. Each person is tracked in the video stream with a unique ID, and objects are associated to persons according to the relative distance and the persons' gaze direction. After data processing, a score is computed for each person detected taking into account position, gaze direction and interaction with objects in the latest camera frames. The robot, basing on these scores, may choose which is the best human to interact with and what action to perform. Tests carried out in diverse indoor situations in an office environment, showed that the proposed method gives promising and coherent results, even though there is room for improvement.