Visual and Tactile Perception to play Jenga with a Robotic Arm

Visual perception and sensory feedback are key elements in any recent robotic system. Human vision and dexterity have been analyzed and adapted to teach the machines how to perform tasks the way people do it. The scope of this Master's Degree thesis is the study of computer vision and tactile perception methodologies to realize a multisensory robotic application able to understand its surroundings and to act with precision on the desired target. More in detail, the goal of this project is to enable the robotic arm e.DO, developed by Comau, to play Jenga tower through a control system based on vision and touch. The constraints given by the rules of the game and the accurate physical interaction with objects constitute a challenging framework for this robotic task. The analysis of the task leads to the development of a 3D model of the tower and a dataset of images to train a segmentation neural network. From its detections a visual tracker estimates the pose of the specific objects in the camera. Finally, a force sensor is implemented for feedback. The tower is modelled in a synthetic environment with the goal of training an instance segmentation neural network to detect and discriminate each block in the camera image. Using Blender, 2D renderings of the tower's 3D model constitute an image dataset to train the Yolact neural network and allows it to understand the tower’s configuration from its pictures. Here the game dictates how the tower is built from blocks and how it changes in time by removing them. The employed vision system is the position-based visual servoing, composed by two steps adapted from the library of ViSP, an open source visual servoing platform. The first one consists of a model-based tracker that, given the generic 3D model of an object, finds the real object in the camera frame, learns and detects its keypoints, then follows its movement in subsequent frames; furthermore, the output is the pose (translation and orientation) of the tracked object in each frame. Instead of a single Jenga block, a group of them is tracked for an increased number of visual features, exploiting the tower's staticity and known geometry. The second step, taking that pose as input, is a visual control law computing a twist command (linear and angular velocities) to move the camera and reach the desired pose for the extraction of the tracked object. This is achieved by mounting a RealSense D435 depth camera on the robotic arm in the so-called eye-in-hand configuration to link the movement of camera to the arm. Physical interaction to extract a block by pushing it is studied from the properties of wood material, weights of blocks and their relative positions. After an estimate of friction values a force sensor is mounted on top of the robot's end effector to take the real measurements. The sensor’s touch feedback allows to abort a potentially disruptive push on the tower's balance, because the extraction of the real blocks is made harder by their irregularities. These avert from the ideal friction behaviour as some blocks are not free to move but they cannot be detected by vision alone. After developing and testing all software components separately, they are connected through ROS interfaces into a final system which ultimately allows to publish e.DO robot specific commands and actuate its joints. The work presents theory and experiments for each unit and, in conclusion, the obtained results achieved by the complete mechatronic system.