\chapter{Introduction}
	
	\section{Introduction}
		Interest in autonomous vehicles covering aerial, terrestrial, and underwater areas has dramatically increased due to their prospective ability to transform many industries. The development of reliable path-planning methods is crucial for task optimization and the enhancement of the versatility and efficiency of these vehicles. Autonomous drones show substantial potential, able to conduct intricate tasks across a wide array of environments. However, the constantly changing nature of these environments introduces distinct obstacles, making path planning for drones an intriguing field of study. Autonomous drones are utilized in various duties, from surveillance and delivery services to search and rescue missions. To carry out these tasks effectively, they require path planning optimization considering the immediate environment and the inherent dynamism of real-world settings. Several path planning methodologies have been investigated, such as minimum snap path planning \cite{MinimumSnapTraj, MinimumSnapTraj2} and minimum time path planning \cite{MinimumTimeTraj, MinimumTimeTraj2}. Despite their efficiency, these strategies are predominantly designed for static environments. Conversely, drones often operate in dynamic environments where objects and conditions can shift unpredictably. As a result, path planning for dynamic environments is inherently more intricate and demanding. To enable effective path planning and navigation, estimating the status of dynamic objects is made even more challenging by the cluttered nature of these environments. Existing research has started addressing these challenges.
		Path planning and state estimation are central to the operations of autonomous drones. Path planning involves determining the optimal route a drone should take to accomplish its task, which can greatly differ depending on the task and environment specifics. A common method involves defining specific waypoints the drone must reach and then determining the most efficient route. State estimation complements path planning by supplying vital information about the current and future states of the environment. This information facilitates informed decisions about the drone's path. State estimation techniques can be employed to identify significant features in the environment, such as the center of goal objects. Predicting the future location of these objects, especially in dynamic environments, continues to be a field-wide challenge.
		In this study, we put forth a novel methodology to tackle the challenges of state estimation and path planning in dynamic environments. Our strategy harnesses the capabilities of Kalman filter algorithms for estimating and forecasting the states of dynamic objects in the environment. More specifically, we use these algorithms to estimate and forecast the positions of waypoints, which are defined as the centers of gates in our context. The Kalman filter facilitates estimation of the current state of these waypoints, and their future states over a variable time horizon, contingent on the drone's distance from each gate.
		To complement state estimation, we also suggest the employment of advanced actor-critic reinforcement learning (RL) algorithms for optimal path planning. We implement three RL algorithms in a continuous action-state environment to discern the optimal path through the predicted waypoints. The path's optimality is defined based on a reward function provided to the RL algorithms. The combination of state estimation and path planning techniques is intended to significantly enhance the performance of autonomous drones in dynamic environments. In this paper, our primary objective is to explore the effectiveness of a combined approach that uses the Kalman filter to predict future positions of gates over changing time horizons and different Reinforcement Learning (RL) algorithms for optimal path planning in dynamic environments. We aim to assess and compare the performance of three specific RL algorithms under varying environmental conditions, with a particular emphasis on dynamic settings.
		
	\section{Related Work}
		The landscape of available training environments includes several three-dimensional reinforcement learning simulation environments, such as Gym \cite{gym} and AirSim \cite{airsim}, each with their challenges. For instance, despite AirSim's realistic and detailed nature, it demands a high-performance system due to its computational intensity. On the other hand, Gym, which is less computationally intensive, lacks environments specifically dedicated to drone path planning, with some environments supporting only discrete action spaces. Our research aims to surmount these hurdles by formulating a customized continuous action-state environment using the Python Pygame library. This tailor-made environment reduces the computational demand compared to AirSim and is also precisely suited to our specific task. We have integrated a field-of-view feature commonly absent in Pygame environments but vital for authentic drone perception to enhance its realism.
		Various studies have explored autonomous drone path planning, utilizing the Potential Field Method (PFM), Rapidly exploring Random Tree (RRT), and the Voronoi diagram (VD). Each of these techniques presents its limitations. For example, the Potential Field Method is susceptible to local minima, causing a deadlock where the drone gets trapped in a position surrounded by obstacles \cite{potentialfield, potentialfield2}. RRT, although efficient in navigating high-dimensional spaces, may need help finding the optimal path, and its performance can be compromised in densely cluttered environments \cite{RRT, RRT2}. Voronoi Diagrams, while ensuring a safe distance from obstacles, do not necessarily yield the shortest or most efficient path, and their generation can be computationally intense, particularly in complex or dynamic environments \cite{Veroni, Dynamicwild}.
		There has also been considerable effort in applying reinforcement learning to path planning \cite{SACNAV, decisionmaking, yunlung, TOP, DDR}. The Song Et al. study didn't test the algorithm's performance in dynamic environments. It depended on Vicon cameras for gate state prediction, which may not be universally applicable in real-world contexts \cite{yunlung}. Another study applied three reinforcement learning algorithms in a static environment \cite{SACNAV}. Some other researchers have conducted various studies on Model Predictive Control algorithms to address challenges in the field of path planning. \cite{MPCC, RTNMPC, NMPC}.
		
		Furthermore, there is ongoing research on dynamic environments aimed at predicting the desired dynamic goal to navigate towards it. However, the challenge lies in the fact that the implemented approach requires Visual-Inertial Odometry (VIO) cameras for state estimation of the dynamic gates and predicting their future positions. \cite{PSMPC}.
		
		\begin{figure}[h]
			\centering
			\includegraphics[width=.6\linewidth]{images/ENV/PATHETH.png}
			\caption{Policy Search Model Predictive Control proposed by Perception Group at University of Zurich \cite{PSMPC}}

		\end{figure}
		
		Our research strives to bridge these gaps by leveraging Kalman Filters for state estimation and prediction \cite{kalman}. We aim to design a reinforcement learning environment that meets specific needs, such as supporting a continuous action state environment, providing specific state elements, and defining the agent's reward function based on the Kalman filter prediction. Additionally, we incorporate the field of view of real cameras as boundary conditions for the environment to enhance the realism and practicality of our approach.
		
	\section{Methodology}
		We employed a comprehensive methodology with a 3D simulation environment to address the issues related to dynamic environments as well as continuous action space environments. This environment was used to emulate the dynamic conditions that drones may encounter in real-world scenarios. Firstly, some algorithms are implemented to evaluate the results in discrete action space environments, and in the next step the environments and actions considered to be a continuous one. This 3D simulation allowed us to explore all possible outcomes under various scenarios and conditions, thereby providing a robust and extensive platform for the training and testing of our autonomous drone systems. These conditions include the different assumptions for initial position of the drone and gates, velocity and acceleration of the gates, as well as the field of view of the drone.
		Our methodology also involved the use and comparison of different reinforcement learning (RL) algorithms: Deep Deterministic Policy Gradient (DDPG), Dueling Double DQN, Proximal Policy Optimization (PPO), and Prioritized Experience Replay algorithms for discrete action state environments as well as Deep Deterministic Policy Gradient, Soft-Actor Critic (SAC), and Proximal Policy Optimization for continuous action state environments. These algorithms were trained to focus on Path planning and state estimation, two critical factors for successful drone operation in dynamic environments. The algorithms were evaluated on the same scenarios within the 3D simulation to ensure a fair comparison. The performance of each algorithm was then analyzed and compared, with particular attention paid to their respective ability to navigate successfully in the dynamic simulated scenarios with continuous action state environments. Moreover, the Kalman Filter, a well-known technique for state estimation, was integrated with the RL techniques. The goal was to enhance the drones' decision-making process and path planning. Combining these two techniques, we aimed to create an autonomous drone system that could adapt and respond effectively to non-static environments. 
		
	