Closed-loop Control of Soft-Robots using Reinforcement Learning

The field of soft robotics is a relatively new one which has recently developed thanks to the solutions it offers to the limitations of rigid robots: these include the wider possibilities they have in the area of human-robot interaction since their soft and lightweight bodies are intrinsically safe for humans. The biggest effort of the research community has been on the development of innovative materials and designs to build new soft robots but designing controllers that are employable for them is still an open challenge. This is due to the fact that the control and modelling paradigms used on rigid robots are not applicable on soft robots because of their highly nonlinear dynamics which are difficult to model on account of the small degree of stochasticity they often incorporate. These reasons have opened the door to new unconventional controllers based on model-free neural networks which, while being suboptimal for their rigid counterparts, have proven to be effective on soft robots. The main trend right now is to use static controllers that rely on the kinematic models of these manipulators; these have the advantage to be relatively simple to create but their drawbacks are the limitations on the speed and efficiency of the robot. This thesis is an effort to create both open-loop and closed-loop dynamic controllers, with the closed-loop architecture being the main objective, for tracking tasks on a soft robotic manipulator purely based on data. The first method described is based uniquely on data gotten from the robot and its aim is to design an open-loop controller: initially, static data are collected to approximate its inverse kinematic model that in turn is used to generate trajectories in the task space of the robot at a higher frequency; these trajectories will later serve as input for a new network that includes the error caused by the dynamics effects ignored by the kinematics. The second method is instead thought for making a closed-loop controller which makes the tracking of a moving object resistant to changes of the forward model of the robot: a very important feature considering the high variability of the structure of this kind of manipulators which could experience deformations of their pneumatic actuators when put under prolonged actuation stress. Using a state-of-the-art reinforcement learning algorithm (Trust Region Policy Optimization), the controller is trained on an environment which consists of the soft robot's forward dynamic model, previously derived using a Long-short Term Memory recurrent neural network. The two methods proved to be effective: both the open-loop and the closed-loop controllers applied to the real soft robot are able to follow a 3D trajectory with an average error in the cartesian space of below 6mm; the closed-loop controller also proved to be resistant to moderate changes of the forward model of the robot. The work done in this thesis offers quick and easily applicable ways to generate precise tracking using model-free, neural network-based, controllers. The fact that these rely uniquely on data makes the pipelines to obtain them applicable to virtually any soft robot, and the high accuracy with which they achieve dynamic tracking will be paramount for more complex tasks such as grasping and pick-and-place.