Design and Implementation of an Inverted Pendulum Control System using FPGA and Reinforcement Learning

The inverted pendulum system is a widely recognized control problem that revolves around the task of stabilizing a pendulum connected to a movable base. The primary objective is to keep the pendulum in an upright position, despite its inherent instability, by manipulating the movement of the cart. This system serves as a crucial benchmark for evaluating and testing control algorithms, finding applications in diverse domains such as robotics and industrial automation. Effectively controlling the inverted pendulum has far-reaching implications for the advancement of autonomous vehicles, bipedal robots, and other dynamic systems that rely on maintaining balance and stability. This master's thesis project aims to tackle the inverted pendulum problem from various perspectives, with three main objectives. Firstly, an environment of the inverted pendulum will be constructed in OpenAI Gym, and reinforcement learning techniques, specifically Proximal Policy Optimization (PPO), will be utilized to achieve balance. By training an agent to control the base, the objective is to demonstrate the capability of reinforcement learning algorithms in solving control problems. Secondly, a MATLAB/Simulink mathematical model of the cartpole system, representing the inverted pendulum, will be developed using Lagrangian mechanics. This model will provide insights into the dynamics of the system and serve as a foundation for designing and implementing control strategies. Control strategies such as PID (Proportional-Integral-Derivative) and LQR (Linear Quadratic Regulator) will be designed and implemented in Simulink. The system will be controlled in simulation, and preliminary results will be obtained to determine parameters for the real model. Lastly, the control strategies designed before will be tested on an existing real model. A special HW/SW system will be designed using the PYNQ-Z1 FPGA board. The pendulum position will be observed using a rotary encoder with an optical interrupter, while the cart position will be estimated from the control output applied to the stepper motor by the processor. All input sensor signals will be processed by the programmable logic (PL) before being passed to the processor system (PS). Control algorithms, such as PID and LQR strategies, will be implemented in software using high-level programming languages like C through the Xilinx Vitis application environment. The integration of hardware and software components aims to showcase real-time control of the physical inverted pendulum and highlight the advantages of using FPGA boards in control and sensing applications. By achieving these objectives, this thesis contributes to the understanding and application of control strategies and reinforcement learning algorithms in the context of a problem such as the inverted pendulum system. The results and insights gained from the simulations, mathematical modeling, and physical implementation will enhance the knowledge of control systems and their practical implementation. Furthermore, the successful utilization of reinforcement learning in OpenAI Gym environments demonstrates the potential for solving complex control problems.