Deep Learning techniques for programming a collaborative robotics system

Collaborative robotics differs from traditional robotics in several aspects. The main difference is that in traditional robotics the robot is located in a "safety cage" to avoid direct contact with users. In collaborative robotics, instead, the robot acts in a complementary way to the user, interacting with him during the performance. This project aims at realizing a robotic system capable of interacting with the user to perform collaborative tasks. In particular, the main goal of the robotic system is to recognize human gestures acquired by the camera, in order to be programmable by demonstration. The main components of the system developed in this thesis are a robotic manipulator, a depth-camera and some auxiliary tools to take care of some implementation aspects and to optimize the overall execution performance. The chosen robot is a 6-joints manipulator (e.DO by Comau), which was born for educational purposes, so it has reduced dimensions compared to an industrial manipulator. The depth-camera (Intel RealSense D435i) has been used to obtain a stereoscopic view of the space around the robot. Specifically, the camera performs two types of acquisitions: a typical RGB acquisition and a depth-acquisition. The last one has been fundamental for the complete implementation of the arm movement in three dimensions. To obtain the desired functionalities, the SDKs pyedo and realsense were used. The project relies also on IKpy (inverse kinematics solver) and Openpose (for object detection). These libraries are available on Github. Openpose is based on two neural networks, and it has the purpose to identify the individual(s) in a capture scene, as well as their key points (e.g. elbow, shoulder, ankle, etc.). There are two types of library versions that can be used, and they depend on the hardware on which execution takes place: CPU version (without a dedicated graphics card) and GPU version (if a dedicated graphics card is present). The environment in which the main tests were carried out did not have a dedicated GPU. The performance is in this case affected by the lack of suitable hardware. Amazon AWS services were used to virtually simulate the features required for optimal performance. A web service was created using Openpose, in order to make the processing independent of the machine on which it is executed. Improvements in terms of processing were obtained, but the use of the web service had a negative impact on the processing time. Thus, it was not possible to obtain point identification in real time, as initially expected. The ultimate improvement could be achieved by processing the images without hardware simulation, but using the physical required hardware, eliminating therefore the web service in order to avoid further waste of time.