I. IntroductionThe application of unmanned aerial vehicles (UAVs) to defense and homeland security tasks necessitates the need for robust and efficient methods for tracking ground targets. The purpose of this study is to streamline the tracking capabilities of a UAV in such scenarios.Therefore, a finite horizon optimal control problem is formulated, and reinforcement learning (RL) is used to solve it. The proposed reinforcement learning solution uses a deep single network adaptive critic (DSNAC) framework, allowing the vehicle to utilize minimal real-time computing power to track a trajectory given any set of initial conditions in its domain of training.&nbsp; In doing so, we aim to expand and enrich the use of machine learning to compute the optimal control for ground trajectory tracking by UAVs.II. MethodologyThe fields of machine learning and optimal control currently provide a range of methods for solving tracking problems using UAVs. Among the most useful are position-based visual servoing (PBVS) and image-based visual servoing (IBVS) methods. Other methods include the use of reinforcement learning (RL) algorithms for trajectory planning.Mathematically, optimal control provides control inputs by minimizing a cost function subject to input and state constraints. Dynamic programming (DP) offers a systematic solution to optimal control problems. DP provides a closed loop solution, which is valid for different initial conditions within a domain. DP also handles nonlinear dynamics of systems relatively well. However, DP suffers from the curse of dimensionality: as the order of the system increases, so do the computational resources needed to achieve the desired results. Approximate dynamic programming (ADP) can therefore be introduced as a remedy. To summarize, ADP methods use function approximators, such as neural networks, to approximate the optimal control solution, and then tune the parameters of those function approximators with RL. RL aims to identify a policy that would maximize a future reward to obtain the optimal solution.As an ADP method, single network adaptive critic (SNAC) frameworks offer an appealing solution for trajectory tracking problems. A finite-SNAC architecture&nbsp;provides optimal solutions when there are differing initial conditions, time horizons, and reference trajectories. The method is apt for use in UAVs because it can be used for online feedback control with minimal computational resources after having been trained offline. We aim to expound on this method by incorporating deep RL to the training, thereby allowing the UAV to effectively obtain the optimal solution to the tracking problem.A quadrotor is used for this study. The dynamics of the system are governed by the input thrust force and moments provided by the quadrotor propellers; the thrust and moments are proportional to the squared rotor speeds.&nbsp; Using the Euler and Newton equations for 3D rigid body motion, the dynamic system can be modelled and utlimately described in the input-affine form X = f(x) + g(x)u,&nbsp;where u&nbsp;represents the input vector and X the state vector.&nbsp;Having defined the system dynamics, the SNAC framework can identify the optimal solution to track&nbsp;a trajectory r(t)&nbsp;by minimizing a quadratic cost function J. For this application, the network is first trained offline using RL to optimize a costate vector, which&nbsp;ultimately minimizes J&nbsp;and optimizes the controls. Once that is complete, the UAV can implement feedback control online in real-time in order to track any other trajectory.II. Preliminary ResultsThe finite-SNAC method can track any set of trajectories with unique initial conditions in real-time and online.&nbsp; Unlike in other methods, the solution is valid for any set of initial conditions and trajectories.&nbsp; As such, applying the framework to UAV tracking offers an appealing method for the vehicle to complete its task.III. ConclusionThe use of finite-SNAC and reinforcement learning provides the UAV with a unique method of completing ground tracking tasks. By incorporating the versatility of finite-SNAC and the efficiency of deep reinforcement learning, UAVs will benefit by having reduced computational requirements and better performance in the necessary environments.&nbsp;

Uav Ground Target Trajectory Tracking Using Reinforcement Learning

Paper Proceedings: https://asmedigitalcollection.asme.org/IMECE/IMECE2021/volume/85543

ASME 2021 International Mechanical Engineering Congress and Exposition (IMECE2021)

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.