\chapter{Simulation Environment and Experimental Setup}
	
	\section{Introduction}
		
	
	\section{Two Dimensional Drone Environment}
	In the design of this environment, a specific object-oriented programming paradigm was utilized to model the relationships and interactions between various elements in the system. The primary focus was to create a modular and scalable architecture that would allow for easy modification and extension.
	
	To achieve this, a class hierarchy was established, allowing for the separation of concerns and encapsulation of data. The classes were structured in such a way that common functionality and properties were inherited from a base class, and specialized behavior was implemented in derived classes. This structure facilitated code reusability and maintainability.
	
	Additionally, to store and manage the data associated with each object, appropriate attributes were defined within the classes. These attributes allowed for the proper representation of the object's state and properties while maintaining data integrity and minimizing redundancy. Access to these attributes was controlled through carefully designed methods, ensuring that external entities could only interact with the object's data in a safe and consistent manner.
	
	To enable seamless communication and collaboration between objects, well-defined interfaces, and methods were employed. This approach allowed for the efficient exchange of data and execution of tasks, promoting a high degree of cohesion and reducing coupling between objects. This modular design facilitated the ease of testing and debugging the software, ensuring that each component functioned correctly and efficiently within the larger system.
	
	Overall, the careful design and implementation of classes, attributes, and methods within the software contributed to a robust and efficient solution that met the project's requirements and provided a solid foundation for future development and improvement.
	
	The environment is a 2D drone game where the drone navigates through two gates. The environment is created as a class called $DroneGateGame$, which initializes various attributes like the drone's position, gate positions, field of view $(fov)$, and dimensions of the environment (w and h). There are eight scenarios in which the drone and gates have different behaviors.
	
	\begin{itemize}
		\item Fixed starting point of drone and static gates with discrete state-action space
		\item Random starting point of drone and static gates with discrete state-action space
	\end{itemize}
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.7\linewidth]{images/ENV/2d_1.png}
		\caption{Sample of different Starting Points (Orange points) and path of gates at different time steps }
%		\label{fig:hi}
	\end{figure}
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.65\linewidth]{images/ENV/2d.png}
		\caption{Sample of different Starting Points (Orange points) and path of gates at different time steps }
%		\label{fig:hi}
	\end{figure}
	
	\subsection{State Representation}
	The state of the drone is represented by its relative position and velocity with respect to the center of each gate in the environment plan. The state space in the cases of having static gates is two-dimensional $(x, y)$, and in the cases of having dynamic gates is three-dimensional $(x, y, \dot{y})$ in which the relative velocity of the gate with respect to the drone along $y$ direction is included.
	
	\subsection{Action Space}
	The action space in the environment with discrete state-action space consists of four possible actions: moving up, down, left, or right. The actions are defined by the integer values $0$, $1$, $2$, and $3$, representing $right$, $left$, $down$, and $up$, respectively. While in the case of continuous action-state space, the action space consists of one action which is the angle by which the drone should move alongside by a unit magnitude.
	
	\subsection{Reward Function}
	The reward function is designed to encourage the drone to move closer to the gates. It takes into account the magnitude of the distance between the drone and the center of the gate before and after taking a step. If the drone gets closer to the gate center, the reward will be positive; otherwise, it will be negative. Furthermore, if the drone hits the boundaries, which include both the environment's boundaries and locations where the gate centers are out of the camera's field of view, the reward will be a significantly negative value at that step. 
	
	\section{Three Dimensional Drone Environment}
	The environment is a 3D drone game where the drone navigates through two gates. The environment is created as a class called DroneGateGame, which initializes various attributes like the drone’s position, gate positions, field of view $(fov)$, and dimensions of the environment (w and h). There are eight scenarios in which the drone and gates have different behaviors.
	\begin{itemize}
		\item Fixed starting point of drone and static gates with continuous state-action space
		\item Fixed starting point of drone and dynamic gates with continuous state-action space
		\item Random starting point of drone and static gates with continuous state-action space
		\item Random starting point of drone and dynamic gates with continuous state-action space
	\end{itemize}
	\subsection{State Representation}
	In a three-dimensional environment, the investigation is focused on continuous action states. Similar to two-dimensional environments, various behaviors are taken into account for gates and different initial conditions.
	
	In the case of static gates, the state is represented as $(x, y, z)$, where $x$, $y$, and $z$ denote the relative positions of the drone concerning the gate in the respective $x$, $y$, and $z$ directions. This representation provides sufficient information for navigating the drone in a 3D environment with fixed gate locations.
	
	However, when considering an environment with dynamic gates, the state representation must be expanded to include additional information about the relative velocities in the $y$ and $z$ directions. In this case, the state is represented as $(x, y, z, \dot{y}, \dot{z})$. The additional components, $\dot{y}$ and $\dot{z}$, account for the relative velocity in the $y$ and $z$ directions, respectively, enabling the drone to react to moving gates more effectively.
	
	By considering both static and dynamic gates in a three-dimensional environment, the analysis becomes more comprehensive and adaptable to different scenarios. This approach allows for a more robust understanding of drone navigation and control, which can be beneficial for applications such as drone racing, search and rescue missions, and environmental monitoring, where the ability to maneuver through complex environments is crucial.
	
	\subsection{Action Space}
	The action space for both static and dynamic gates encompasses the magnitude of movement in the $x$, $y$, and $z$ directions. Each of these values falls within the range of $[-1, 1]$, representing the displacement in each respective direction. By normalizing the sum of all the movements, the magnitude of the total displacement at each step is equal to one unit.
	
	In this action space, the drone can move in any direction within the three-dimensional environment by combining the appropriate magnitudes for each axis. This flexible approach allows the drone to navigate and adapt to various scenarios, whether it involves static or dynamic gates. By using a normalized range for the magnitudes, the system ensures consistency and comparability across different movement directions and environments.
	
	This action space representation is well-suited for applications in drone navigation and control, as it enables the system to respond effectively to a wide range of situations. The normalization of the movement magnitudes ensures a consistent and comparable scale for evaluating the drone's performance across different scenarios.
	
	\subsection{Reward Function}
	The reward function is designed to encourage the drone to move closer to the center of the gate at each time step. The reward is calculated based on the amount of displacement at each step multiplied by the sign of the displacement. In other words, the relative position concerning the center of the gate before and after taking a step is considered.
	A higher positive reward is granted when the drone moves closer to the center of the gate, while a more significant negative reward is given when the drone moves further away from the center. Moreover, if the drone hits the boundaries, which include both the environment's boundaries and locations where the gate centers are out of the camera's field of view, the reward will be a substantially negative value at that step.
	The formula for the reward function is as follows:
	
	%			$Reward =$
	%            \begin{align}
		%                \text{Relative distance before taking step} - \text{Relative distance after taking step}   
		%            \end{align}
	
	\[
	Reward(t) = 
	\begin{cases} 
		
		-30 & \text{if Collision to Boundaries} \\
		+30 & \text{if Reach Latest Gate Center} \\
		d_{t-1}^{rel}(s_{t-1}, a_{t-1}) - d_t^{rel}(s_t, a_t) & \text{Otherwise} \\
	\end{cases}
	\]
	
	in which,
	
	\begin{itemize}
		\item \(Reward(t)\) represents the reward at time \(t\).
		\item \(d_t^{rel}(s, a)\) is the relative distance at time \(t\) given the state \(s\) and action \(a\).
		\item \(s_t\) and \(a_t\) are the state and action at time \(t\), respectively.
		\item \(s_{t-1}\) and \(a_{t-1}\) are the state and action at the previous time step, \(t-1\).
	\end{itemize}
	
	By utilizing this reward function, the drone's navigation system is encouraged to optimize its path toward the center of the gate, thereby improving its overall performance in various environments. This reward structure is particularly useful in applications such as drone racing, search and rescue missions, and environmental monitoring, where precise navigation and control are essential.
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.8\linewidth]{images/logo/3D_env.png}
		\caption{Sample of Movement of the Drone in 3D environment at different training episode in Fixed Starting Points}
%		\label{fig:hi}
	\end{figure}
	
	\begin{figure}[h]
		\centering
		\includegraphics[width=.8\linewidth]{images/logo/Gate_path.png}
		\caption{Sample of different Starting Points (Orange points) and path of gates at different time steps }
%		\label{fig:hi}
	\end{figure}
	

	
	
		
		