Control System Design with Reinforcement Learning Algorithm for a Space Manipulator

On-Orbit Servicing (OOS) represents the advent of a new approach to space access and promises to be a key element in developing the future space infrastructures. Upcoming robotic spacecraft, mounting a robotic arm, may be able to perform a wider range of operations on a larger number of client spacecraft such as docking, berthing, refueling, repairing, upgrading, transporting, rescuing and orbital debris removal. Space manipulator systems, however, introduce relevant challenges due to the dynamic coupling between the manipulator and the spacecraft, that represents its base, and due to the growing need for autonomy and flexibility to perform new tasks and adapt to environment changes and disturbances. This implies the need for control system design that can reduce the reaction forces exchanged at the mounting point and that is robust to uncertainties on initial conditions. ??The present Master thesis focuses on the modeling and control of a space manipulator system operating in a typical OOS mission environment. In order to obtain end-effector pose expressed as a function of the joint variables of the mechanical structure, the kinematic model of the robotic arm is first derived with a systematic and general approach using Denavit-Hartenberg (DH) convention. In addition, dynamic equations of the manipulator are obtained using Lagrange formulation to simulate the motion, compute the forces exchanged with the base and torques required for the execution of the task and thus design control algorithms. ??Thus, a Reinforcement Learning (RL) controller for the manipulator is implemented exploiting the ability to learn how to complete a task within an unknown environment through repeated trial-and-error interactions with the environment without human involvement. Soft-Actor-Critic (SAC) algorithm, based on the maximum entropy framework, is selected to train the agent. The performance is then compared with a classical Proportional-Integral-Derivative (PID) controller. ??Then a control system design for the spacecraft is proposed. Linear-Quadratic-Regulator (LQR) design, based on a fully controllable quaternion spacecraft model, is implemented for attitude purpose. LQR approach is also adopted for position control, showing pose-keeping ability and robustness of LQR design. ??Last, a typical mission scenario of an OOS mission is presented, describing several phases that characterize this mission and showing the performance of the designed model. ??The simulation scenario, the space manipulator system plant and the proposed controllers are developed using MATLAB R2020b and Simulink exploiting the numerous available toolboxes including Symbolic Toolbox, Deep Learning Toolbox, Reinforcement Learning Toolbox, Robotic System Toolbox and Simulink 3D Animation.