\chapter{Results}
	
	\section{Experiment Results and Analysis}

	\subsection{Performance Metrics}
	In our research, we employ three main criteria to evaluate the performance of the reinforcement learning algorithms: cumulative score per episode, the count of episodes in which the agent accomplished the objective, and the duration per episode (episode elapsed). These criteria can be further divided into five metrics defined as follows: 
	\begin{itemize}
		\item Cumulative Score Per Episode: The cumulative score per episode directly measures the agent's performance in its assigned task. Superior scores suggest that the agent optimizes its actions to fulfill its objective.
		\item Count of Episodes to Accomplish the Objective: This criterion assesses the agent's learning proficiency. An agent that reliably accomplishes the objective in fewer episodes proves to be quicker in resolving the task and adapting to the environment.
		\item Duration per Episode (episode elapsed): By evaluating the duration spent on each episode, we can gauge the agent's computational efficiency. This becomes crucial when comparing algorithms that garner similar scores, as computational efficiency could be the deciding factor in algorithm choice.
		\item Average Reward per Action: The average reward per action offers a measure of the quality of the decisions made by the agent. An increased average reward per action indicates that the agent is making superior decisions, leading to higher rewards for each action. This becomes particularly noteworthy in scenarios where the agent has a limit on the number of actions it can execute in an episode or when the objective is to garner the highest reward in the least possible time or number of steps. This metric balances the speed (number of steps) and the quality (reward) of the actions, preventing strategies that excessively prioritize one aspect at the detriment of the other.
		\item Success Rate: This represents the ratio of episodes where the agent accomplishes its objective. A higher success rate indicates that the agent can more reliably complete its task.
	\end{itemize}
	
	\noindent Finally, by multiplying Success Rate and Average Rewards together, you get a new metric that can be interpreted as follows:
	\begin{itemize}
		\item High Average Reward per Action, High Success Rate: If the product is high, it suggests that the algorithm is both proficient (it receives a high reward per action) and successful (it frequently accomplishes its goal). This is the optimal situation.
		\item Low Average Reward per Action, High Success Rate: It might imply that the algorithm frequently accomplishes its goal but could be more proficient (it receives a low reward per action).
		\item High Average Reward per Action, Low Success Rate: Conversely, a moderate product might suggest that the algorithm is proficient (it receives a high reward per action) but does not frequently accomplish its goal.
		\item Low Average Reward per Action, Low Success Rate: If the product is low, it suggests that the algorithm needs to be proficient and successful. This would imply that the algorithm's performance is subpar.
	\end{itemize}
	
	These metrics offer a comprehensive perspective of the effectiveness and efficiency of the reinforcement learning algorithms under study.
	
	\subsection{Results and Discussion}
	In reviewing the outcomes of our experiments, several key findings were discernable. The performance of the reinforcement learning algorithms was primarily evaluated through metrics: cumulative score per episode, the number of successful episodes, and the episode duration. The cumulative score per episode served as a crucial indicator of the overall efficiency of the algorithms, offering insights into the ability of the agent to accumulate rewards over the course of an episode. The count of successful episodes measured the agent's reliability in achieving the objectives. Lastly, the duration per episode or episode elapsed, allowed us to gauge the speed of the agent's learning process, with shorter durations reflecting faster convergence to optimal behaviors. These criteria collectively provided a comprehensive understanding of the algorithms' capabilities in various aspects of reinforcement learning.
	
			
			% Please add the following required packages to your document preamble:
			% \usepackage{multirow}
			\begin{table}[]
				\caption{Summary of the results obtained with the fixed starting point (FSP) environment}
				\resizebox{\textwidth}{!}{%
				\begin{tabular}{|c|c|c|c|c|c|c|}
					\hline
					\textbf{Algorithm} & \textbf{Environment}     & \textbf{Total Episodes} & \textbf{Episodes Goal Reached} & \textbf{Average Elapsed Time} & \textbf{Average Reward per Step} & \textbf{Success Rate (\%)} \\ \hline
					FSP\_SG\_DDPG  & \multirow{3}{*}{Static}  & 1000                    & \textbf{432}                             & 6,768                         & \textbf{0,778}                   & 0,432                      \\ \cline{1-1} \cline{3-7} 
					FSP\_SG\_SAC   &                          & 1000                    & 990                                      & \textbf{9,223}                & 0,709                            & \textbf{0,990}             \\ \cline{1-1} \cline{3-7} 
					FSP\_SG\_PPO   &                          & 984                     & \textbf{886}                             & \textbf{0,640}                & \textbf{0,795}                   & 0,900                      \\ \hline
					FSP\_DG\_DDPG  & \multirow{3}{*}{Dynamic} & 1000                    & \textbf{770}                             & 10,801                        & \textbf{0,997}                   & 0,770                      \\ \cline{1-1} \cline{3-7} 
					FSP\_DG\_SAC   &                          & 1000                    & 944                                      & \textbf{20,647}               & 0,694                            & \textbf{0,944}             \\ \cline{1-1} \cline{3-7} 
					FSP\_DG\_PPO   &                          & 984                     & 802                                      & \textbf{1,986}                & \textbf{0,743}                   & 0,815                      \\ \hline
				\end{tabular}%
			}
			\label{tab:table1}
			\end{table}
			
	
	In \ref{tab:table1} a comprehensive summary of the results attained using the three selected algorithms is offered, all originating from a consistent starting point across all episodes. The Soft Actor-Critic (SAC) algorithm stands out for its rapid convergence to high rewards with a minimal episode count in both static (SG) and dynamic (DG) environments. However, despite this swift convergence, the SAC manifests some instability, even at high episode counts. On the other hand, the Proximal Policy Optimization (PPO) algorithm demonstrates a slower convergence rate but eventually reaches similar end-of-training rewards as SAC and exhibits less uncertainty in resolving the environment. The slower convergence of PPO is attributed to its inherent design that encourages more conservative policy updates. This approach, while leading to a slower learning rate, also contributes to the algorithm's superior stability, as it avoids drastic policy changes that could potentially disrupt the learning process and result in increased uncertainty.
	
	% Please add the following required packages to your document preamble:
	% \usepackage{multirow}
	\begin{table}[]
		\caption{Summary of the results obtained with the random starting point (RSP) environment}
		\resizebox{\textwidth}{!}{%
			\begin{tabular}{|c|c|c|c|c|c|c|}
				\hline
				\textbf{Algorithm} & \textbf{Environment}     & \textbf{Total Episodes} & \textbf{Episodes Goal Reached} & \textbf{Average Elapsed Time} & \textbf{Average Reward per Step} & \textbf{Success Rate (\%)} \\ \hline
				RSP\_SG\_DDPG  & \multirow{3}{*}{Static}  & 1000                    & 941                                      & 5,475                         & \textbf{1,038}                   & 0,941                      \\ \cline{1-1} \cline{3-7} 
				RSP\_SG\_SAC   &                          & 1000                    & 994                                      & 9,688                         & 0,752                            & \textbf{0,994}             \\ \cline{1-1} \cline{3-7} 
				RSP\_SG\_PPO   &                          & 984                     & \textbf{822}                             & \textbf{0,721}                & 0,805                            & 0,835                      \\ \hline
				RSP\_DG\_DDPG  & \multirow{3}{*}{Dynamic} & 1000                    & \textbf{414}                             & 14,652                        & 0,743                            & 0,414                      \\ \cline{1-1} \cline{3-7} 
				RSP\_DG\_SAC   &                          & 1000                    & 759                                      & 34,967                        & 0,542                            & \textbf{0,759}             \\ \cline{1-1} \cline{3-7} 
				RSP\_DG\_PPO   &                          & 984                     & 726                                      & \textbf{2,327}                & \textbf{0,746}                   & 0,738                      \\ \hline
			\end{tabular}%
		}
		\label{tab:table2}
	\end{table}

	
	The results of \ref{tab:table2} provide a detailed overview of the outcomes achieved using the three selected algorithms, each initiated from a random starting point defined at the beginning of each episode. The Soft Actor-Critic (SAC) algorithm is noteworthy due to its speedy convergence toward high rewards, requiring only a low number of episodes in both static (SG) and dynamic (DG) settings. Conversely, the Proximal Policy Optimization (PPO) algorithm displays a slower convergence rate but eventually attains comparable end-of-training rewards to SAC and shows less volatility when navigating the environment. 
	
	
	\subsection{Effects of Gate Movement Speed}
	
		In summarizing our research efforts, we scrutinized the performance of various reinforcement learning algorithms under the diverse dynamism of our environment. Specifically, we focused on the gate speeds' variability effect on the reward function. For the sake of this experiment, we parametrized the velocity, setting it to a low limit of $0.45 rad/s$, which denoted the starting point, scaling up to a high end of $0.90 rad/s$. 
		Our observations exposed those algorithms, such as DDPG (Deep Deterministic Policy Gradient) and SAC (Soft Actor-Critic), which appeared to be considerably swayed by the varying speeds of the gates. Consequently, a broad spectrum was evident in the range of rewards during the different trials, indicating high variability and inconsistency in the outcomes. 
		
		
	\begin{figure}[h]
		\centering
		\includegraphics[width=.8\linewidth]{images/logo/DDPG_VAR_Vel.png}
		\caption{Trajectory Planning of DDPG Algorithm with Random Starting Point}
%		\label{fig:hi}
	\end{figure}
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.8\linewidth]{images/logo/SAC_VAR_Vel.png}
		\caption{Trajectory Planning of DDPG Algorithm with Random Starting Point}
%		\label{fig:hi}
	\end{figure}
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.8\linewidth]{images/logo/PPO_VAR_Vel.png}
		\caption{Trajectory Planning of DDPG Algorithm with Random Starting Point}
%		\label{fig:hi}
	\end{figure}