\chapter{Conclusions and Future Work}

	Our research findings offer valuable insights into the performance of Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC), and Proximal Policy Optimization (PPO) algorithms in intricate environments. Dynamic environments pose significant challenges for robotics systems, primarily due to their unpredictability and high adaptability required for successful operation. These challenges stem from such environments constantly changing, with the robot needing to react and adapt in real-time to achieve its objective.
	In particular, DDPG's deterministic policy approach showed limitations in these settings. While beneficial in directly approximating the optimal action and possibly hastening convergence, DDPG's exploration strategy could benefit from further refinement to enhance performance in more complex, evolving scenarios.
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.75\linewidth]{images/logo/RSP_DG_PPO_PATH.png}
		\caption{Trajectory obtained with the algorithm PPO in a dynamic environment for increasing episodes}
%		\label{fig:hi}
	\end{figure}
	
	On the other hand, SAC displayed rapid learning, attributed to its unique integration of a stochastic policy and an entropy component within its objective function. By continually seeking to balance reward maximization and action diversification, SAC skillfully negotiates the exploration-exploitation trade-off, fostering speedy learning in complex environments.
	While not the fastest to learn, PPO demonstrated robust and steady performance after adequately approximating the optimal policy and value functions. PPO's stability is derived from its clipped objective function, which regulates policy updates to avoid drastic changes that could lead to performance volatility.
	We also leveraged the Kalman filter to accurately predict the gate's position in space, effectively managing the dynamic nature of the moving gate. The filter uses an iterative process of prediction and correction based on previous states and current measurements, demonstrating its prowess in handling dynamic motion tracking.
	However, these findings are based on simulations, and as we move forward, we need to consider the next steps: transitioning from simulation to hardware testing. Despite the fidelity of our simulations, real-world scenarios can introduce unforeseen variables and challenges that need to be accounted for in the simulation environment. Issues such as sensor noise, mechanical failures, and real-world physics discrepancies may arise and can significantly impact the performance of the robotics systems and the algorithms controlling them.
	To bridge this 'reality gap', future research will focus on hardware-in-the-loop testing, where the algorithms will be integrated with physical systems in controlled environments. These tests will provide valuable data on how these algorithms perform under real-world conditions and how they can be further optimized. Additionally, we anticipate refining our models to handle environmental changes better, incorporating more advanced perception capabilities, and increasing the robustness of the systems. We aim to enhance our robotics systems' practical applicability and performance in dynamic environments by carefully iterating our designs and algorithms based on this new data.
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.65\linewidth]{images/logo/barplots.png}
		\caption{Summary of the Performances for different algorithms metrics}
		\label{fig:barplot}
	\end{figure}
	
	As shown in \ref{fig:barplot}, the \textit{DDPG} is the most dependent on the environment condition, while the PPO is the most stable one from the average reward per step point of view. Also, the \textit{SAC} and \textit{PPO} have higher success rate than the \textit{DDPG}.
	
	Future work will involve improving the corner detection algorithms for the gates in a simulation environment, identifying the center of the gates corresponding to those corners, and integrating the proposed algorithm into this project, including the gate's center detection. As a continuation of the simulation environment, all the algorithms will be implemented in a real environment using a drone.