\chapter{A Kalman Filter Reinforcement Learning Approach for Path Planning}
\section{Kalman Filter for State Estimation}
    The Kalman Filter Algorithm employed in this environment is the Standard Kalman Filter, which assumes the problem to be not overly nonlinear. As we are dealing with a simple inverted pendulum, and the characteristics of the pendulum change at each episode, the system can be considered relatively simple. The advantage of using a standard Kalman filter lies in its ability to provide a good trade-off between the accuracy of estimations and the complexity of the model from a computational perspective.

    The Kalman Filter consists of two steps: prediction and update. The input for the prediction step is the time at which the drone is expected to reach the center of the gate. This time can be approximated using the average step time and the distance to the center of the gate. The average step time is dependent on the computational load of the algorithm as well as the processing power of the system. In other words, the average step time represents the average duration it takes for the agent to make a step. By considering these two parameters, the time to reach the gate center can be determined, and the outputs of the prediction step, which include the relative positions and relative velocities of the gate with respect to the drone along the $x$, $y$, and $z$ directions, can be calculated.

    Moreover, the inputs of the update step are the relative positions of the gate with respect to the drone $(x, y, z)$, and the outputs are the estimations of the gate's future position concerning the drone at the time calculated in the previous step. It is important to note that the states of the drone for Reinforcement Learning algorithms are precisely the outputs of the update step at each time step.

    By incorporating the Standard Kalman Filter into the drone's navigation system, it can more accurately estimate its position and velocity relative to the gate, leading to improved performance in various environments.
    
\section{Reinforcement Learning Algorithms with Kalman Filter Integration 2D}

	In this section, different Reinforcement Learning algorithms are trained to evaluate and compare the performance of each algorithms for two different scenarios. The first scenario includes all the initial conditions to be known more specifically the starting point of the drone, whereas in the second scenario, the initial starting point of the drone is selected randomly at each episode. So, the second scenario will consist of more challenges and it will be more generalized rather than the first one.
	
    \subsection{Deep Deterministic Policy Gradient}
    
    	\subsubsection{Training and Hyperparameters}
    		
    		In all the following Training and Hyperparameters sections, most of the parameters are considered to be constant to have a better comparison among the nature of the algorithms by themselves. As you will see, the \textit{\textbf{gamma}} coefficient which is the discount factor that adjusts the rewards importance over time in all the algorithms is considered to be $0.99$. Also, the \textit{\textbf{nS}} factor for the environments including static and dynamic gates is $2$ and $3$, respectively. And \textit{\textbf{nA}} factors in 2D discrete action state environments is $4$ which are the movement directions that are $Up$, $Down$, $Right$, and $Left$.
    		
    		Moreover, the complexity of the networks are determined by \textit{\textbf{$hidden\_dims$}} factors and because of that relative advanced neural network, the \textit{\textbf{$policy\_optimizer\_lr$}} and \textit{\textbf{$value\_optimizer\_lr$}} factors are investigated to be small values. The point should be noticed here is that \textit{\textbf{$policy\_optimizer\_lr$}} factor considered lower than \textit{\textbf{$value\_optimizer\_lr$}} value to have less intervention of the disturbances and noises to the policy network.
    
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for DDPG Algorithm in 2D Environment}]
gamma: 0.99
policy_model_fn = lambda nS, bounds: FCDP(nS, bounds, hidden_dims=(256, 256))
policy_max_grad_norm = float('inf')
policy_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
policy_optimizer_lr = 0.00003
value_model_fn = lambda nS, nA: FCQV(nS, nA, hidden_dims=(256, 256))
value_max_grad_norm = float('inf')
value_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
value_optimizer_lr = 0.0003
training_strategy_fn = lambda bounds: NormalNoiseStrategy(bounds, exploration_noise_ratio=0.1)
replay_buffer_fn = lambda: ReplayBuffer(max_size=100000, batch_size=512)
n_warmup_batches = 1
update_target_every_steps = 1
tau = 0.001
\end{lstlisting}
\end{minipage}
\end{figure}
		
 		\subsubsection{Performance Evaluation}
 			Because of the simplicity of the 2D environments as well as the discret action state environments, the training episodes to get the highest reward at each episode is not too much.
 			
	 		\begin{figure}[h]
	 			\begin{multicols}{2} % Start two-column layout
	 				% Your content here
	 				\includegraphics[width=\linewidth]{images/2d_images/Episode Score DDPG Fixed Starting Point.png}
	 				
	 				\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of DDPG Fixed Starting Point.png}
	 				
	 				\includegraphics[width=\linewidth]{images/2d_images/Episode Score DDPG Random Starting Point.png}
	 				
	 				\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of DDPG Random Starting Point.png}
	 				
	 			\end{multicols}
	 			
	 			\caption{Episode Scores and Trajectories of DDPG Algorithm with Fixed and Random Starting Points}
	
	 		\end{figure}
 		               
            As shown in the Episode score graph and in the plotted trajectories, the agent learns faster to reach the center of the gates in the first scenario which is the fixed starting point. That is the reason why the trajectories for most of the episodes in the case of having static gates are similar to each other. But, as the randomness to the environment increases, the learning would be more challenging, the trajectories would be different, and more episodes is needed learn the desired policy.
            
    \subsection{Dueling Double DQN}
    	
    	\subsubsection{Training and Hyperparameters}
    		
    		In this algorithm, the \textit{\textbf{nS}} and \textit{\textbf{nA}} parameters considered same as the previous algorithm (DDPG), but the network is more complicated by increasing the neurons at each hidden layer. Also the \textit{\textbf{lr}} parameter is considered to be changed based on an Exponential rate which leads to have more stable learning in most of the cases.
    		
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for D3QN Algorithm in 2D Environment}]
gamma: 0.995
value_model_fn = lambda nS, nA: FCDuelingQ(nS, nA, hidden_dims=(512,512))
value_optimizer_fn = lambda net, lr: optim.RMSprop(net.parameters(), lr=lr)
value_optimizer_lr = lambda: LR_ExpStrategy(init_lr= 0.0005, min_lr=0.0002, decay_steps=100000)
max_gradient_norm = float('inf')
training_strategy_fn = lambda: EGreedyExpStrategy(init_epsilon=1.0,
min_epsilon=0.05,decay_steps=100000)
replay_buffer_fn = lambda: ReplayBuffer(max_size=80000, batch_size=512)
n_warmup_batches = 1
update_target_every_steps = 1
tau = 0.01
\end{lstlisting}
\end{minipage}
\end{figure}

		\subsubsection{Performance Evaluation}
			As you can see, this algorithm is not too much dependent on the environment variables and the required episodes to learn the policy for the agent in two scenarios are almost the same. But as you can see, on the up right plot corresponding to Episode Score for Random Starting Point, the reward is oscillating in range of $[450,550]$ which caused by the length of the path the agent should take. This fact is admitted by its lower trajectory that shows the path is taken almost perfectly in different episodes.
			\begin{figure}[h]
				\begin{multicols}{2} % Start two-column layout
					% Your content here
					\includegraphics[width=\linewidth]{images/2d_images/Episode Score D3QN Fixed Starting Point.png}
					
					\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of D3QN Fixed Starting Point.png}
					
					\includegraphics[width=\linewidth]{images/2d_images/Episode Score D3QN Random Starting Point.png}
					
					\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of D3QN Random Starting Point.png}
				\end{multicols}
				
				\caption{Episode Scores and Trajectories of D3QN Algorithm with Fixed and Random Starting Points}
				
			\end{figure}
			
	\subsection{Prioritized Experience Replay}
	
	\subsubsection{Training and Hyperparameters}
		As the previous algorithms, most of the parameters are the same as those ones. in $PrioritizedReplayBuffer$, the $rank\_based$ hyperparameter is considered to be False. If it is set to $True$, it mean the priority order is based on the following equation:
		$$p_{i}=\frac{1}{rank(i)}$$
		Otherwise, if that hyperparameter is set to $False$, the prioritization would be proportional which is the absolute TD error plus a small constant epsilon to avoid zero priorities as the following equations:
		$$p_{i}=\abs{\delta_{i} + \epsilon}$$. Moreover, the $alpha$ coefficient is used in the following formula to get the probabilities from priorities:
		$$p_{i} = \frac{p_{i}^{\alpha}}{\sum_{k}{p_{k}^{\alpha}}}$$.
		Finally, the $beta$ hyperparameter is implemented in the following equation to get the importance=sampling weights using the probabilities.
		$$w_{i}=(NP(i))^{-\beta}$$
		in the above equation, the importance-sampling based is calculated by multiplying each probability by the number of samples in the replay buffer and then powered by $-\beta$. and then the weights are scaled down to have the largest weight be equal to one.
		$$w_{i}=\frac{w_{i}}{max_{j}(w_{i})}$$
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for PER Algorithm in 2D Environment}]
gamma: 0.99
value_model_fn = lambda nS, nA: FCQ(nS, nA, hidden_dims=(512, 512))
value_optimizer_fn = lambda net, lr: optim.RMSprop(net.parameters(), lr=lr)
value_optimizer_lr = lambda: LR_ExpStrategy(init_lr= 0.0005, min_lr=0.0002, decay_steps=100000)
max_gradient_norm = float('inf')
training_strategy_fn = lambda: EGreedyExpStrategy(init_epsilon=1.0,
min_epsilon=0.1,
decay_steps=150000)
replay_buffer_fn = lambda: PrioritizedReplayBuffer(
max_samples=80000, batch_size=6000, rank_based=False,
alpha=0.6, beta0=0.1, beta_rate=0.99997)
n_warmup_batches = 1
update_target_every_steps = 1
tau = 0.01
\end{lstlisting}
\end{minipage}
\end{figure}
	
	\subsubsection{Performance Evaluation}
		As is evident from Figure \ref{fig:PER}, the algorithm successfully reaches the desired results in the Fixed Starting Point case. However, in the case of having a Random Starting Point, it exhibits instability. It appears that this algorithm has the poorest performance among the mentioned algorithms in 2D discrete action environments.
	\begin{figure}[h]
		\begin{multicols}{2} % Start two-column layout
			% Your content here
			\includegraphics[width=\linewidth]{images/2d_images/Episode Score PER Fixed Starting Point.png}
			
			\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of PER Fixed Starting Point.png}
			
			\includegraphics[width=\linewidth]{images/2d_images/Episode Score PER Random Starting Point.png}
			
			\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of PER Random Starting Point.png}
		\end{multicols}
		
		\caption{Episode Scores and Trajectories of PER Algorithm with Fixed and Random Starting Points}
		\label{fig:PER}
	\end{figure}
	
	
    \subsection{Proximal Policy Optimization}
    
    	\subsubsection{Training and Hyperparameters}
    		In the following, the hyperparameters of the PPO algorithm are defined. As it is evident, the PPO algorithm has more hyperparameters than the other discussed algorithms. This leads to the PPO algorithm being more of a hyperparameter tuning problem.
    		
    		Two hyperparameters $policy\_stopping\_kl$ and $value\_stopping\_mse$ are the conditions which determine the policy and value optimization networks to be done or ignored. Also, the $policy\_optimization\_epochs$ and $value\_optimization\_epochs$ are the hyperparameters which determine for how many time the policy and value networks being optimized.
    		Moreover the \textit{\textbf{sd}}, \textit{\textbf{g}}, \textit{\textbf{t}}, \textit{\textbf{me}}, and \textit{\textbf{mes}} hyperparametrs are the state dimension, gamma factor, tau factor, maximum episodes, and maximum steps at each epiode, respectively. And finally, the $policy\_sample\_ratio$ and $value\_sample\_ratio$ are the ones determine the size of the batch whcih are going to be used for policy and value functions at each $policy\_optimization\_epochs$ and $value\_optimization\_epochs$, respectively.
    		
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for PPO Algorithm in 2D Environment}]
gamma: 0.99
policy_model_fn = lambda nS, nA: FCCA(nS, nA, hidden_dims=(256, 256))
policy_model_max_grad_norm = float('inf')
policy_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
policy_optimizer_lr = 0.0003
policy_optimization_epochs = 80
policy_sample_ratio = 0.8
policy_clip_range = 0.1
policy_stopping_kl = 0.02
value_model_fn = lambda nS: FCV(nS, hidden_dims=(256, 256))
value_model_max_grad_norm = float('inf')
value_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
value_optimizer_lr = 0.0005
value_optimization_epochs = 80
value_sample_ratio = 0.8
value_clip_range = float('inf')
value_stopping_mse = 25
episode_buffer_fn = lambda sd, g, t, me, mes: EpisodeBuffer(sd, g, t, me, mes)
max_buffer_episodes = 16
max_buffer_episode_steps = 1000
entropy_loss_weight = 0.01
tau = 0.01
\end{lstlisting}
\end{minipage}
\end{figure}

		\subsubsection{Performance Evaluation}
				
				As is evident in the Episode Reward plots and the corresponding trajectories, the algorithm reaches the desired reward in Fixed Starting Points scenarios. However, it takes more time and more episodes to reach an optimal policy in Random Starting Point scenarios. It should also be considered that through hyperparameter tuning, it might achieve better performance under the same conditions.
			\begin{figure}[h]
				\begin{multicols}{2} % Start two-column layout
					% Your content here
					\includegraphics[width=\linewidth]{images/2d_images/Episode Score PPO Fixed Starting Point.png}
					
					\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of PPO Fixed Starting Point.png}
					
					\includegraphics[width=\linewidth]{images/2d_images/Episode Score PPO Random Starting Point.png}
					
					\includegraphics[width=\linewidth]{images/2d_images/Trajectory Planning of PPO Random Starting Point.png}
					
				\end{multicols}
				\caption{Episode Scores and Trajectories of PPO Algorithm with Fixed and Random Starting Points}
				
			\end{figure}
		
\section{Reinforcement Learning Algorithms with Kalman Filter Integration, 3D}
    In this section, three well-known Reinforcement Learning (RL) algorithms are investigated and analyzed in the context of the environment discussed in the previous section. Each algorithm is examined in detail, focusing on its suitability and performance within the specified environment, taking into account the unique characteristics and challenges presented by the drone navigation task.
    
    \subsection{Deep Deterministic Policy Gradient}

        \subsubsection{Algorithm Integration}
        	It is a deterministic policy algorithm and so it uses off-policy exploration strategy. For example, the Gaussian noise is added to the action-selection process that enhances the exploratory of deterministic policies.
        	
        	\begin{algorithm}
        		\caption{Deep Deterministic Policy Gradient (DDPG)}
        		\begin{algorithmic}[1]
        			\State Initialize actor network $\pi$ and critic network $Q$ with random weights $\theta^\pi$, $\theta^Q$
        			\State Initialize target networks $\pi'$ and $Q'$ with weights $\theta^{\pi'} \gets \theta^\pi$, $\theta^{Q'} \gets \theta^Q$
        			\State Initialize replay buffer $R$
        			\For {episode = 1, M}
        			\State Initialize a random process $N$ for action exploration
        			\State Receive initial observation state $s_1$
        			\For {t = 1, T}
        			\State Select action $a_t = \pi(s_t|\theta^\pi) + N_t$ according to the current policy and exploration noise
        			\State Execute action $a_t$ and observe reward $r_t$ and observe new state $s_{t+1}$
        			\State Store transition $(s_t, a_t, r_t, s_{t+1})$ in $R$
        			\State Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$
        			\State Set $y_i = r_i + \gamma Q'(s_{i+1}, \pi'(s_{i+1}|\theta^{\pi'})|\theta^{Q'})$
        			\State Update critic by minimizing the loss: $L = \frac{1}{N}\sum_i(y_i - Q(s_i, a_i|\theta^Q))^2$
        			\State Update the actor policy using the sampled policy gradient:
        			$$
        				\nabla_{\theta^\pi} J \approx \frac{1}{N} \sum_i \nabla_a Q(s, a|\theta^Q)|_{s=s_i, a=\pi(s_i)} \nabla_{\theta^\pi} \pi(s|\theta^\pi)|_{s_i}
        			$$
        			\State Update the target networks:
        			\begin{align*}
        				\theta^{Q'} \gets \tau\theta^Q + (1 - \tau)\theta^{Q'}\\
        				\theta^{\pi'} \gets \tau\theta^\pi + (1 - \tau)\theta^{\pi'}
        			\end{align*}
        			\EndFor
        			\EndFor
        		\end{algorithmic}
        	\end{algorithm}
        	
        \subsubsection{Training and Hyperparameters}
        	In the following Training parameters, the \textit{FCQV} and \textit{FCDP} are the Q-function network and policy network, respectively. The output of \textit{FCQV} network is just a single value that represents the value of the state-action pair. Also consider the activation function of the last layer of the \textit{FCDP} network is considered as \textit{tanh} to have an action between $[-1,1]$. In the training of the agent with this algorithm \textit{NormalNoiseStrategy} function is considered to add noise to the action which leads to increase the exploration of the agent in the environment. Then the action is clipped to be in the desired action range and finally, the noise ratio is changed based on the designed schedule inside the function. This schedule can be constant, linear, or exponential.
        
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for DDPG Algorithm in 3D Environment}]
gamma: 0.99
policy_model_fn = lambda nS, bounds: FCDP(nS, bounds, hidden_dims=(128, 128))
policy_max_grad_norm = float('inf')
policy_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
policy_optimizer_lr = 0.0003
value_model_fn = lambda nS, nA: FCQV(nS, nA, hidden_dims=(128, 128))
value_max_grad_norm = float('inf')
value_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
value_optimizer_lr = 0.0005
training_strategy_fn = lambda bounds: NormalNoiseStrategy(bounds, exploration_noise_ratio=0.1)
replay_buffer_fn = lambda: ReplayBuffer(max_size=1000000, batch_size=256)
n_warmup_batches = 5
update_target_every_steps = 1
tau = 0.001
\end{lstlisting}
\end{minipage}
\end{figure}
        	
        \subsubsection{Performance Evaluation}
			As shown in Figure \ref{fig:DDPG4}, the algorithm is not as stable and consistent as the other two algorithms (SAC and PPO) for a 3D continuous action state environment. It can be observed that as the randomness of the environment increases, the stability and consistency of the algorithm also increase, and the agent can eventually reach the desired policy after several iterations.
			
			\begin{figure}[h]
				\begin{multicols}{2}

					\includegraphics[width=\linewidth]{images/logo/FSP_SG_DDPG_3D_episode_scores.png}
					
					\includegraphics[width=\linewidth]{images/logo/RSP_SG_DDPG_3D_episode_scores.png}
					
					\includegraphics[width=\linewidth]{images/logo/FSP_DG_DDPG_3D_episode_scores.png}
					
					\includegraphics[width=\linewidth]{images/logo/RSP_DG_DDPG_3D_episode_scores.png}
					
				\end{multicols}
				
				\caption{Episode Scores and Trajectories of PER Algorithm with Fixed Starting Points, and Fixed Vs. Dynamic Gates}
				\label{fig:DDPG4}
			\end{figure}
			
			
			
            \begin{figure}[h]
            	\centering
            	\includegraphics[width=.77\linewidth]{images/logo/RSP_DG_DDPG_PATH.png}
            	\caption{Trajectory obtained with the algorithm DDPG in a dynamic environment for increasing episodes}
            	
            \end{figure}
        
    \subsection{Soft Actor-Critic}

        \subsubsection{Algorithm Integration}
        		This algorithm is known as entropy-maximization algorithm that includes maximizing a mixed objective of the value function and policy entropy which leads to getting the most reward with the most diverse policy. It is an off-policy algorithm in which the agent can reuse the experiences to enhance the policies.
        		
        	\begin{algorithm}
        		\caption{Soft Actor-Critic (SAC)}
        		\begin{algorithmic}[1]
        			\State Initialize actor network $\pi$, critics $Q_1$ and $Q_2$ with random weights $\theta^\pi$, $\theta^{Q1}$, $\theta^{Q2}$
        			\State Initialize target critic networks $Q_1'$ and $Q_2'$ with weights $\theta^{Q1'} \gets \theta^{Q1}$, $\theta^{Q2'} \gets \theta^{Q2}$
        			\State Initialize temperature $\alpha$ and target entropy $\bar{H}$
        			\State Initialize replay buffer $R$
        			\For {episode = 1, M}
        			\State Receive initial observation state $s_1$
        			\For {t = 1, T}
        			\State Select action $a_t \sim \pi(s_t|\theta^\pi)$
        			\State Execute action $a_t$ and observe reward $r_t$ and new state $s_{t+1}$
        			\State Store transition $(s_t, a_t, r_t, s_{t+1})$ in $R$
        			\State Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$
        			\State Compute target \(y_i = r_i + \gamma \min_{j \in \{1,2\}} Q_j'(s_{i+1}, \pi'(s_{i+1}|\theta^{\pi'})) - \alpha \log \pi(a_{i+1}|s_{i+1})\)
        			\State Update critics by minimizing the loss:
        			\begin{align*}
        				L_{Q1} = \frac{1}{N}\sum_i(y_i - Q_1(s_i, a_i|\theta^{Q1}))^2 \\
        				L_{Q2} = \frac{1}{N}\sum_i(y_i - Q_2(s_i, a_i|\theta^{Q2}))^2 \\
        			\end{align*}
        			\State Update actor policy by maximizing:
        			\[ J_{\pi} = \mathbb{E}_{s \sim D, a \sim \pi} [Q_1(s, a) - \alpha \log \pi(a|s)] \]
        			\State Adjust temperature \( \alpha \) to maintain entropy near target:
        			\[ \alpha \gets \alpha - \text{lr}_\alpha (\bar{H} - \mathbb{E}_{a \sim \pi} [-\log \pi(a|s)]) \]
        			\State Update the target critics:
        			\begin{align*}
        				\theta^{Q1'} \gets \tau\theta^{Q1} + (1 - \tau)\theta^{Q1'}\\
        				\theta^{Q2'} \gets \tau\theta^{Q2} + (1 - \tau)\theta^{Q2'}
        			\end{align*}
        			\EndFor
        			\EndFor
        		\end{algorithmic}
        	\end{algorithm}
        	
        	
        \subsubsection{Training and Hyperparameters}
        
        	In SAC algorithm, the \textit{FCGP} is used as a Fully Connected Gaussian Policy network. Because of the nature of the algorithm, the hidden layers in this network are connected to two separate networks to which represent the mean of actions and logarithmic standard deviation. And also consider in the Optimization function, two separate networks are used to get the minimum Q-Value between these models.
        	
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for SAC Algorithm in 3D Environment}]
gamma: 0.99
policy_model_fn = lambda nS, bounds: FCGP(nS, bounds, hidden_dims=(128, 128))
policy_max_grad_norm = float('inf')
policy_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
policy_optimizer_lr = 0.0003
value_model_fn = lambda nS, nA: FCQSA(nS, nA, hidden_dims=(128, 128))
value_max_grad_norm = float('inf')
value_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
value_optimizer_lr = 0.0005
replay_buffer_fn = lambda: ReplayBuffer(max_size=1000000, batch_size=256)
n_warmup_batches = 5
update_target_every_steps = 1
tau = 0.001
\end{lstlisting}
\end{minipage}
\end{figure}


        \subsubsection{Performance Evaluation}
        	
        	\begin{figure}[h]

        		\begin{multicols}{2} % Start two-column layout
        			% Your content here
        			\includegraphics[width=\linewidth]{images/logo/FSP_SG_SAC_3D_episode_scores.png}
        			
        			\includegraphics[width=\linewidth]{images/logo/RSP_SG_SAC_3D_episode_scores.png}
        			
        			\includegraphics[width=\linewidth]{images/logo/FSP_DG_SAC_3D_episode_scores.png}
        			
        			\includegraphics[width=\linewidth]{images/logo/RSP_DG_SAC_3D_episode_scores.png}
        			
        		\end{multicols}
        		
        		\caption{Episode Scores and Trajectories of SAC Algorithm with Fixed Starting Points, and Fixed Vs. Dynamic Gates}
        		\label{fig:SAC4}
			\end{figure}
			
			As shown in Figure \ref{fig:SAC4}, the algorithm outperforms DDPG. It is more stable and consistent compared to the DDPG algorithm. However, in cases where random starting points are employed, the shaded area is wider than that of the PPO algorithm, which will be discussed in the next section. This indicates that, on average, the model performs well, but in some seeds, the agent deviates significantly from the optimal average policy.

            
            \begin{figure}[h]
            	\centering
            	\includegraphics[width=.77\linewidth]{images/logo/RSP_DG_SAC_PATH.png}
            	\caption{Trajectory obtained with the algorithm SAC in a dynamic environment for increasing episodes}
            	
            \end{figure}
        
    \subsection{Proximal Policy Optimization}

        \subsubsection{Algorithm Integration}
        	This is an on-policy algorithm despite the previous algorithms which were off-policy algorithms. Also, this algorithm is more conservative than the previous ones because of clipped objectives strategy it has in optimization process.
        	\begin{algorithm}
        		\caption{Proximal Policy Optimization}
        		\begin{algorithmic}[1]
        			\State Initialize policy parameters $\theta$, value parameters $\phi$, and empty buffer $\mathcal{B}$.
        			\While{training not over}
        			\For{$i = 1, \dots, \text{max\_buffer\_episodes}$}
        			\State Generate episode $i$ using policy $\pi_{\theta}$ and add to buffer $\mathcal{B}$
        			\EndFor
        			\For{$k = 1, \dots, \text{policy\_optimization\_epochs}$}
        			\State Sample minibatch from $\mathcal{B}$
        			\State Calculate surrogate objective $L^{CLIP+\text{VF}}_t(\theta)$ with respect to old policy $\pi_{\theta_{\text{old}}}$
        			\State Update $\theta$ by maximizing $L^{CLIP+\text{VF}}_t(\theta)$
        			\State Calculate policy divergence $KL[\pi_{\theta_{\text{old}}} | \pi_{\theta}]$
        			\If{$KL > \text{policy\_stopping\_kl}$}
        			\State Break
        			\EndIf
        			\EndFor
        			\For{$k = 1, \dots, \text{value\_optimization\_epochs}$}
        			\State Sample minibatch from $\mathcal{B}$
        			\State Calculate value loss $L^{VF}_t(\phi)$
        			\State Update $\phi$ by minimizing $L^{VF}_t(\phi)$
        			\State Calculate mean squared error (MSE) between value estimates
        			\If{$\text{MSE} > \text{value\_stopping\_mse}$}
        			\State Break
        			\EndIf
        			\EndFor
        			\State Empty buffer $\mathcal{B}$
        			\EndWhile
        		\end{algorithmic}
        	\end{algorithm}
        	
        	
        \subsubsection{Training and Hyperparameters}
        	In the optimization of PPO algorithm, the loss is calculated as the negative of the minimum of the objectives.
        	
\begin{figure}[h]
\begin{minipage}{0.95\linewidth}
\begin{lstlisting}[language=Python, caption={Parameter Settings for PPO Algorithm in 3D Environment}]
gamma: 0.99
policy_model_fn = lambda nS, nA: FCCA(nS, nA, hidden_dims=(128,128))
policy_model_max_grad_norm = 0.5
policy_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
policy_optimizer_lr = 0.0003
policy_optimization_epochs = 10
policy_sample_ratio = 0.9
policy_clip_range = 0.1
policy_stopping_kl = 0.02
value_model_fn = lambda nS: FCV(nS, hidden_dims=(128,128))
value_model_max_grad_norm = 0.5
value_optimizer_fn = lambda net, lr: optim.Adam(net.parameters(), lr=lr)
value_optimizer_lr = 0.0005
value_optimization_epochs = 10
value_sample_ratio = 0.9
value_clip_range = 0.1
value_stopping_mse = 25
episode_buffer_fn = lambda sd, g, t, me, mes, seed: EpisodeBuffer(sd, g, t, me, mes, seed)
max_buffer_episodes = 6
max_buffer_episode_steps = 1000
entropy_loss_weight = 0.015
tau = 0.97
policy_lr_scheduler_step_size= 30
policy_lr_scheduler_gamma= 0.96
value_lr_scheduler_step_size= 30
value_lr_scheduler_gamma= 0.96
\end{lstlisting}
\end{minipage}
\end{figure}

        \subsubsection{Performance Evaluation}
			
			\begin{figure}[h]
				\begin{multicols}{2} % Start two-column layout
					% Your content here
					\includegraphics[width=\linewidth]{images/logo/FSP_SG_PPO_3D_episode_scores.png}
					
					\includegraphics[width=\linewidth]{images/logo/RSP_SG_PPO_3D_episode_scores.png}
					
					\includegraphics[width=\linewidth]{images/logo/FSP_DG_PPO_3D_episode_scores.png}
					
					\includegraphics[width=\linewidth]{images/logo/RSP_DG_PPO_3D_episode_scores.png}
					
				\end{multicols}
				
				\caption{Episode Scores and Trajectories of PPO Algorithm with Fixed Starting Points, and Fixed Vs. Dynamic Gates}
				\label{fig:PPO4}
			\end{figure}
			
			As shown in Figure \ref{fig:PPO4}, the algorithm performs significantly better than the two previous ones. It is more stable and consistent when fixed starting points are studied. Additionally, in cases with random starting points, the agent eventually reaches the desired policy, and the shaded area in the static gates is narrower than that of the SAC algorithm. Moreover, in the case of random starting points and dynamic gates, in the final episodes, the shaded area is narrower than the SAC algorithm's, indicating that in the final episodes, the agent performs close to its optimal average policy.			
            
            \begin{figure}[h]
            	\centering
            	\includegraphics[width=.77\linewidth]{images/logo/RSP_DG_PPO_PATH.png}
            	\caption{Trajectory obtained with the algorithm PPO in a dynamic environment for increasing episodes}
            	
            \end{figure}
        





    



