Image-Guided Joint Velocity Control of a Meca500 Manipulator through CNNs

Deep learning has become a powerful tool for dealing with various problems. In particular, in recent years, it has been an object of research even in the field of robotics, due to its ability to work easily with millions of parameters, exploiting the possibility of modeling complex functions. The aim of this thesis is to use one of the core architectures of DL, that is, Convolutional Neural Networks (CNNs), in order to build an open control loop to control a Meca500 robotic arm, which is intended to move towards a target. The target object chosen for the experiment is a simple six-faced die, which is placed in the workspace; a small endoscopic camera is mounted on the end-effector, aligned with its approach axis, in such a way that the target is always visible to the camera. The first phase is dedicated to the construction of a proper training set to be used in the successive training step. This is done by capturing the frames of the camera with a certain periodicity and pairing them with the corresponding task velocity vector, which has been normalized. During the offline training phase, frames and velocity labels are used to train a PyTorch CNN that outputs the six predicted normalized components of the task-space velocity vector. In this way, the network is able to pair the correct velocity vector with the position of the target, as visualized by the camera. Then, online, the prediction is denormalized, mapped to joint velocities through the robot Jacobian, and executed in real time. The initial joint positions are then modified, taking care of possible singularities. Finally, from the standpoint of practical application, an admittance-like control branch is added to the main one, and the whole open control loop is tested.