CN116301007A - A multi-quadrotor UAV assembly task path planning method based on reinforcement learning - Google Patents

Abstract

The invention discloses a multi-quad-rotor unmanned aerial vehicle (MPUAV) gathering task path planning method based on reinforcement learning, which comprises the steps of firstly constructing a PyBullet-based multi-quad-rotor unmanned aerial vehicle Gym environment, setting a reward function mechanism through abstracting a state space and an action space of the MPUAV, then carrying out path planning decision by using an improved deep reinforcement learning algorithm, finally training an improved deep reinforcement learning network, and controlling the MPUAVs through output action information, so that each MPUAV successfully reaches a specified target task in the shortest time. According to the method, N steps of return are utilized in the TD3 algorithm, so that more accurate return estimation, faster learning speed and better generalization performance are obtained, the sampling deviation is reduced by using preferential experience playback, the model deviation caused by unbalanced sampling is reduced, the stability of the algorithm is improved, the TD3 algorithm is more suitable for continuous multidimensional decision-making, and an optimal route can be planned in the shortest time under the condition of achieving specified instantaneity and accuracy.

Description

A multi-quadrotor UAV assembly task path planning method based on reinforcement learning

technical field

The invention belongs to the technical field of path planning for multiple quadrotor UAVs, and in particular relates to a multi-quadrotor UAV assembly task path planning method based on reinforcement learning.

Background technique

The quadrotor UAV balances the gravity of the aircraft through the lift generated by multiple rotors. It can hover, vertically lift, and has low requirements for the take-off site, but its flight speed is relatively slow. Therefore, multi-rotor drones are suitable for complex environments and small-scale application scenarios, such as aerial photography, monitoring, architectural modeling and other tasks. With the continuous development of UAV technology, they have been widely used in civilian fields, and the complexity of the tasks performed is also increasing. Since the load and flight capabilities of a single UAV are limited, the cooperation of multiple UAVs is required to improve the execution capability and scope of the mission.

Since almost all UAV tasks involve the shortest path planning, the shortest path planning problem is the focus and research difficulty of UAV path planning in recent years. In the shortest path planning problem, according to the different characteristics of the task, it can be further divided into assembly task and distribution task. The rendezvous task aims to plan the optimal path for each UAV to reach the same target point from their respective starting points. The goal of such missions is usually to get all drones to the target point at the same time and complete the mission as quickly as possible. In this case, the goal is generally to minimize the total task time or total path length. Compared with distribution tasks, assembly tasks are more versatile.

Compared with existing algorithms based on rules or heuristic search, path planning methods based on reinforcement learning have better adaptability and scalability. Existing methods need to manually design and adjust rules according to the environment, while reinforcement learning methods can adapt to the environment through autonomous learning by agents. Since the agent in reinforcement learning has autonomous decision-making ability, the agent can learn the optimal behavior by interacting with the environment. In addition, the deep learning algorithm has strong perception ability, and the deep reinforcement learning algorithm combined with deep learning and reinforcement learning algorithm can handle higher-dimensional input, which is more suitable for the topic selection of multi-UAV in this paper. Therefore, compared with existing methods, deep reinforcement learning algorithms can better cope with unknown situations and changes, and agents can perform continuous decision-making tasks in complex environments.

At present, the solution technology for multi-UAV to realize the assembly task still faces many challenges, including environment modeling, low learning efficiency, and complex action space and state space. First of all, for the agent project based on the deep reinforcement learning algorithm, the construction of the simulation environment is the basis of the whole experiment, and the design of the UAV system must rely on simulation tools. Therefore, building a proper UAV simulator is crucial for academic research and the development of safety-critical applications. However, many current environments for simulation experiments based on deep reinforcement learning algorithm models lack real-world portability, and many of these reinforcement learning environments sacrifice realism for high sample throughput. In addition, the training efficiency of multi-UAV path planning using deep reinforcement learning algorithms is generally low. In most simulated environments, the path planning task rewards are sparse, the agent can only get the reward signal after the task is over, and due to the difficulty of effective exploration in complex environments, it is difficult to start training at an early stage. Finally, this kind of multi-UAV path planning problem usually involves multi-agents and multi-obstacles, so the state space, action space and reward function of the problem often have high-dimensional and complex characteristics, which increases the modeling and complexity of the problem. Solving difficulty. Since the action space of multi-UAV path planning is very large, an effective search strategy is needed to solve the problem of high-dimensional action space. To sum up, the assembly-type path planning is of great significance to the mission execution of multiple UAVs.

Contents of the invention

In order to solve the above technical problems, the present invention proposes a multi-quadrotor UAV assembly task path planning method based on reinforcement learning, focusing on solving the assembly task in multi-UAV path planning.

The technical solution of the present invention is: a multi-quadrotor unmanned aerial vehicle assembly-type mission path planning method based on reinforcement learning, the specific steps are as follows:

S1. Build a multi-quadrotor drone Gym environment based on PyBullet;

S2. Abstract the state space and action space of the quadrotor UAV, and set the reward function mechanism to make the UAV interact with the environment;

S3. Use the improved deep reinforcement learning algorithm to make path planning decisions, and perform path planning for each quadrotor UAV under the assembly task;

S4. Train the improved deep reinforcement learning network, control the angular velocity and linear velocity of the quadrotor UAV through the output action information, so that each quadrotor UAV can successfully reach the specified target task in the shortest time.

Further, in the step S1, the details are as follows:

S11, building the dynamics simulation model of multi-quadrotor UAV;

The dynamic equations of multi-quadrotor UAVs are composed of the motion equations and aerodynamic effects of quadrotor UAVs, and the construction of the dynamic simulation model of multi-quadrotor UAVs is completed, as follows:

Use PyBullet to model the forces and torques acting on each quadrotor drone in Gym, and use the physics engine to calculate and update the dynamic equations for all quadrotor drones.

Set the arm length of each quadrotor drone as L, the mass as m, the inertial property as J, the physical constants and the convex collision shape are described by a separate URDF file, which is used for the configuration of the "x" type quadrotor drone .

First, set the gravitational acceleration g and the physical step frequency in PyBullet. The force F _i applied to the four motors and the torque T _o around the Z-axis of the drone are proportional to the square of the motor speed P _i , and F _i and The expression of P _i is as follows:

F _i =k _F ·P _i ² (1)

Among them, k _F and k _T represent preset constants.

Set the real-time control of the model, the dynamic equation of the quadrotor UAV is expressed as follows:

J ^T T _o =Ma+h (3)

Among them, J represents the Jacobian matrix, M represents the inertia matrix, a represents the generalized acceleration, h represents Coriolis and gravitational effects, and the superscript T represents the transpose operation.

In practice, flying close to the ground or close to other drones will have additional aerodynamic effects, model them separately in PyBullet and use them jointly, including: propeller drag D, ground effect on a single motor G _i , the downwash effect W acting on the centroid.

螺旋桨的角速度，以及恒定阻力系数矩阵k_D成正比，表达式如下：The rotating propeller of the quadrotor UAV produces resistance D, and the drag D is related to the linear speed of the quadrotor UAV

The angular velocity of the propeller is proportional to the constant drag coefficient matrix k _D , and the expression is as follows:

表示螺旋桨的角速度，60为60s；恒定阻力系数矩k_D具体表达式如下：in,

Indicates the angular velocity of the propeller, 60 is 60s; the specific expression of the constant drag coefficient moment k _D is as follows:

k _D ＝diag(k _⊥ ,k _⊥ ,k _|| ) (5)

Among them, k _⊥ represents the vertical resistance coefficient, k _|| represents the parallel resistance coefficient, and the matrix k _D uses the least square method to fit the data.

When hovering at a very low altitude, there is ground effect, and the influence of ground effect on each motor G _i is proportional to the propeller radius r _P , speed P _i , height h _i and constant k _G as follows:

When two quadrotor UAVs pass the path at the same position at different heights, there is a downwash effect. The influence of the downwash effect is simplified as a single force applied to the center of mass of the UAV, and its magnitude W depends on the two UAVs. The distance (δ _x , δ _{y , δ z ) between human and machine in the coordinate system x, y} , _z and the constants k _D1 , k _D2 , k _D3 , W determined through experiments are expressed as follows:

S12. Construct multi-quadrotor UAV observation space and action space;

In the constructed Gym environment, each action performed by the quadrotor UAV will output an observation vector, and the expression of the observation space of the multi-quadrotor UAV is as follows:

为/>

表示第n个四旋翼无人机的线速度，ω_n为[ω_x,ω_y,ω_z]_n表示第n个四旋翼无人机的角速度；P_n为[P₀,P₁,P₂,P₃]_n表示所有无人机的电机速度。Among them, n∈[0...N] represents the number of quadrotor UAVs; X _n = [x,y,z] _n represents the position of quadrotor UAVs; q _n represents a quaternion, which is used for quadrotors Attitude control of the UAV; r _n , p _n , and y _n respectively represent the roll angle Roll, the pitch angle Pitch, and the yaw angle Yaw, that is, three angles used for attitude estimation;

for />

Indicates the linear velocity of the nth quadrotor UAV, ω _n is [ω _x ,ω _y ,ω _z ] _n indicates the angular velocity of the nth quadrotor UAV; P _n is [P ₀ ,P ₁ ,P ₂ , P ₃ ] _n represents the motor speed of all drones.

In the present invention, the quadrotor UAV uses lidar to detect obstacles, and the quadrotor UAV is set to have k lidars in the model, and the k lidars are used to observe the environment.

Among them, the scanning angle range of the k laser radars is π, and the angle between two lasers is 2π/k; (d ₁ ,...,d _k ) represents the ray length of the k radars on the horizontal plane; d _i represents the The ray length of i radars, the expression is as follows:

Then the environmental information s _E is defined as follows:

s _E =[ρ _i ,d _i ] ^T ,i=1...k (10)

Among them, _ρi represents the one-hot code of the i-th radar, and the expression is as follows:

Then for any quadrotor UAV, the action space expression is as follows:

{n:[v _x ,v _y ,v _z ,v _M ] _n } (12)

Among them, [v _x , v _y , v _z , v _M ] _n represents the speed input to the quadrotor UAV, v _x , v _y , v _z represents the component of the unit vector, and v _M represents the magnitude of the required speed; And the action space can also be represented by the speed of the four motors, the expression is as follows:

{n:[P ₀ ,P ₁ ,P ₂ ,P ₃ ] _n } (13)

Finally, the conversion of the input to pulse width modulation (PWM) and motor speed is delegated to a controller consisting of position and attitude control subroutines.

Further, the step S2 is specifically as follows:

S21. State space and action space of abstract quadrotor UAV;

角速度ω_n、所有无人机的电机速度P_n＝[P₀,P₁,P₂,P₃]，无人机第一视角方向与目标连线间的角度β_n、以及第n个无人机的全局坐标(x,y,z)与目标的全局坐标(x_t,y_t,z_t)之间的差异d_0n。The state of the quadrotor UAV includes: the position of the quadrotor UAV, the quaternion q _n , the roll angle Rollr _n , the pitch angle Pitchp _n , the yaw angle Yawy _n , and the linear velocity

Angular velocity ω _n , motor speed P _n of all UAVs = [P ₀ , P ₁ , P ₂ , P ₃ ], angle β _n between the direction of the UAV’s first viewing angle and the line connecting the target, and the nth UAV’s The difference d _0n between the global coordinates (x, y, z) of the man-machine and the global coordinates (x _t , y _t , z _t ) of the target.

In the state, replace the global position of the UAV with the relative position ΔX _n of the UAV and the target, that is, [Δx, Δy, Δz] _n , then the state s _U of the UAV is as follows:

The state space s of the quadrotor UAV can be obtained from the state s _U of the UAV and the environmental state s _E detected by the lidar, and the expression is as follows:

The action space of the multi-quadrotor UAV environment is composed of the input speed of the quadrotor UAV. Referring to formula (12), for any quadrotor UAV, the expression of the action space is as follows:

a＝[v _x ,v _y ,v _z ,v _M ] ^T (16)

S22. Setting a reward function mechanism to enable the quadrotor UAV to interact with the environment;

The reward function R(s, a) represents the environmental feedback obtained by taking action a in the state s; the reward function composed of three parts is set to make the quadrotor UAV reach the target point of the assembly task as soon as possible, as follows:

First, set the distance reward R _t between the quadrotor UAV and the target point to promote the quadrotor UAV to reach the target, and the expression of R _t is as follows:

表示第n个四旋翼无人机离目标的距离，/>

表示第n个下一时刻离四旋翼无人机离目标的距离。Among them, d ₀ represents the distance from the quadrotor UAV to the target,

Indicates the distance from the nth quadrotor UAV to the target, />

Indicates the distance from the quadrotor UAV to the target at the nth next moment.

Secondly, set the distance reward R _o between the quadrotor UAV and the obstacle to make the UAV far away from the obstacle setting, and the expression of R _o is as follows:

表示第n个四旋翼无人机离目标的距离，设置无人机与障碍物的安全距离d_safe。Among them, d _i represents the ray length of the i-th radar, that is, the detection distance from the quadrotor UAV to obstacles or other quadrotor UAVs,

Indicates the distance between the nth quadrotor UAV and the target, and sets the safe distance d _safe between the UAV and the obstacle.

Finally, let the angle reward R _a between the quadrotor UAV and the target point make the UAV approach the target direction, if the larger the β _n is, the greater the penalty, then the expression of R _a is as follows:

Further, the step S3 is specifically as follows:

The ITD3 algorithm is obtained by improving the TD3 algorithm with N-step rewards and priority experience playback. The ITD3 algorithm consists of four sub-networks, namely two critic networks and two actor networks. The improved deep reinforcement learning algorithm is implemented by the ITD3 algorithm.

First, the N-step return is introduced into the TD3 algorithm. The N-step return adds up the returns of n time steps in the future, providing more comprehensive information than the single-step return;

In the case of sparse rewards, most state transitions p(s′|s,a) have no reward information, and one-step rewards will not be effective; N-step rewards alleviate the problem of reward sparsity by sampling N transitions.

By adding N-step returns, the equation of the TD3 algorithm critic network is modified. In the j-th round of sampling, the modified time difference error function expression δ _j is as follows:

Among them, φ and φ′ represent the parameters of the double-critic network, k represents the return of the k-th step, r _k represents the return of the k-th step, s and a represent the current state and action, s _N and a _N represent the target state and action , Q(s _t , a _t |φ) represents the value function of the critic network, Q′(s _t+N , a _t+N |φ′) represents the value function of the target critic network, and γ represents the discount factor.

Second, using priority experience replay in the original TD3 algorithm, at the beginning of the sample, the sampling probability of the j-th transition is defined as P(j), expressed as follows:

Among them, p _j represents the priority of the jth experience; α represents a constant used to adjust the sampling weight, and α determines how much priority is used. When α is equal to 0, uniform random sampling will be used.

Then, the _sampling weight w _j for each transition used to update the network _is calculated by Normalized:

Finally, using proportional prioritization, the priority of the transfer is updated according to the time difference error, as shown in the following equation:

p _j =|δ _j |+∈ (23)

Among them, δ _j represents the time difference error, and ∈ represents a small value preset to avoid 0 priority.

Further, the step S4 is specifically as follows:

The ITD3 network training network is implemented by a two-part neural network: an actor network consisting of three fully connected layers to complete the state-to-action mapping, and a critic network using four fully connected layers to estimate the Q-value.

In the ITD3 network, for two actor networks, the input is the state and the output is the action. The critic network takes state-action pairs as its input and produces a state-action value function (Q-value). The ITD3 algorithm training process is as follows:

First, a batch of small samples (s, a, s′, r) are preferentially extracted from the experience playback buffer, and s′ is input into the actor target network. Then, a' is obtained in the next pass, and the state-action pair (s', a') is fed into the critic target network.

和/>

)后，选择较小的一个来计算目标值函数y(r,s′)，目标值函数表达式如下：After getting two target Q-values (

and />

), select the smaller one to calculate the objective value function y(r,s′), the expression of the objective value function is as follows:

Among them, r represents the reward, the discount factor γ has the same value as formula (20), and φ _i is the random parameter of the critic network.

On the other hand, feeding (s,a) into the critic network results in two Q-values (Q ₁ (·) and Q ₂ (·)). They are then used to compute the mean squared error of y(r, s′), and the sum of the mean squared errors is backpropagated to update the parameters of the two critic networks, incorporating N-step replay in the time-differenced error update.

Next, the Q-value obtained from the first critic network is input into the actor model network, and the parameters of the actor network are updated along the direction of increasing Q-value (updated every two iterations).

Finally, a soft update method is adopted to update all target networks.

After the training is completed, the angular velocity and linear velocity of the quadrotor UAV are controlled through the output action information, so that each quadrotor UAV can successfully reach the specified target mission in the shortest time and complete the assembly task path planning.

The beneficial effects of the present invention are: the method of the present invention first constructs a multi-quadrotor drone Gym environment based on PyBullet, and sets a reward function mechanism by abstracting the state space and action space of the quadrotor drone, and then uses the improved depth enhancement The learning algorithm makes path planning decisions, and finally trains the improved deep reinforcement learning network to control the quadrotor UAV through the output action information, so that each quadrotor UAV can successfully reach the specified target task in the shortest time. The method of the present invention uses N-step returns in the TD3 algorithm to obtain more accurate return estimates, faster learning speeds, and better generalization performance, and uses priority experience playback to reduce sampling deviations and model deviations caused by unbalanced sampling , improve the stability of the algorithm, make the TD3 algorithm more suitable for continuous multi-dimensional decision-making problems, and can plan the optimal route in the shortest time under the condition of achieving the specified real-time and accuracy.

Description of drawings

FIG. 1 is a flow chart of a multi-quadrotor UAV assembly-type task path planning method based on reinforcement learning of the present invention.

Fig. 2 is a model diagram of "x" type quadrotor UAV in the embodiment of the present invention.

Fig. 3 is a principle diagram of detection of horizontal environment information by laser radar in an embodiment of the present invention.

Fig. 4 is a state diagram of a quadrotor UAV in an embodiment of the present invention.

Fig. 5 is a schematic diagram of the ITD3 algorithm in the embodiment of the present invention.

FIG. 6 is a structural diagram of the neural network in ITD3 in the embodiment of the present invention.

Detailed ways

The method of the present invention will be further described below in conjunction with the accompanying drawings and examples.

As shown in Figure 1, a flow chart of a method for planning a multi-quadrotor unmanned aerial vehicle assembly task path based on reinforcement learning of the present invention, the specific steps are as follows:

S1. Build a multi-quadrotor drone Gym environment based on PyBullet;

S2. Abstract the state space and action space of the quadrotor UAV, and set the reward function mechanism to make the UAV interact with the environment;

S3. Use the improved deep reinforcement learning algorithm to make path planning decisions, and perform path planning for each quadrotor UAV under the assembly task;

S4. Train the improved deep reinforcement learning network, control the angular velocity and linear velocity of the quadrotor UAV through the output action information, so that each quadrotor UAV can successfully reach the specified target task in the shortest time.

In this embodiment, the step S1 is specifically as follows:

S11, building the dynamics simulation model of multi-quadrotor UAV;

Use PyBullet to model the forces and torques acting on each quadrotor drone in Gym, and use the physics engine to calculate and update the dynamic equations for all drones.

As shown in Figure 2, the dynamic model of the simplified "x" quadrotor UAV constructed in this embodiment is set to have the arm length of each UAV as L, the mass as m, the inertial property as J, and the physical constant and convex collision shapes are described by a separate URDF file for the "x" quadcopter configuration.

First, set the gravitational acceleration g and the physics step frequency in PyBullet (which is finer than the Gym step control frequency); in addition to physical properties and constants, URDF information can also be used in PyBullet to load the CAD model of the quadrotor aircraft; The force F _i applied on the four motors and the torque T _o around the Z-axis of the drone are proportional to the square of the motor speed P _i , and the expressions of F _i and P _i are as follows:

F _i =k _F ·P _i ² (1)

Among them, k _F and k _T represent preset constants.

F _i and P _i are linearly related to the input pulse width modulation (Pulse Width Modulation, PWM), and the control of the model is set to be real-time, then the motion equation of the quadrotor UAV is expressed as follows:

J ^T T _o =Ma+h (3)

Among them, J represents the Jacobian matrix, M represents the inertia matrix, a represents the generalized acceleration, h represents Coriolis (Coriolis) and gravitational effects, and the superscript T represents the transpose operation.

In practice, flying close to the ground or close to other UAVs may generate additional aerodynamic effects as shown in Fig. 2 with G _{i = 0, 1, 2,} 3 , the associated forces represented by D and W. They can be modeled separately and used in combination in PyBullet, including: propeller drag D, ground effect G _i acting on a single motor, and downwash effect W acting on the center of mass.

螺旋桨的角速度以及系数矩阵k_D成正比，表达式如下：The spinning propellers of a quadcopter create drag D, a force acting in the opposite direction to the direction of motion. Drag D and linear speed of quadrotor UAV

The angular velocity of the propeller is proportional to the coefficient matrix k _D , the expression is as follows:

表示螺旋桨的角速度，60为60s；为了模拟交叉耦合，需要拟合一个包含9个系数的矩阵k_D。拟合需要一定的对称性：阻力系数以及x和y轴之间的交叉耦合应该相同。由于对称性，z方向的风速在x和y方向上产生相同的力。除此之外，由x方向的速度引起的z方向的阻力应该与由y方向的速度引起的z方向的阻力相同。因此，阻力系数矩阵k_D具体表达式如下：in,

Indicates the angular velocity of the propeller, 60 is 60s; in order to simulate the cross-coupling, it is necessary to fit a matrix k _D containing 9 coefficients. The fit requires a certain symmetry: the drag coefficient and the cross-coupling between the x and y axes should be the same. Due to symmetry, wind speed in the z direction produces the same force in the x and y directions. Besides that, the drag in the z direction caused by the velocity in the x direction should be the same as the drag in the z direction caused by the velocity in the y direction. Therefore, the specific expression of the drag coefficient matrix k _D is as follows:

k _D ＝diag(k _⊥ ,k _⊥ ,k _|| ) (5)

Among them, k _⊥ represents the vertical resistance coefficient, k _|| represents the parallel resistance coefficient, and the matrix k _D uses the least square method to fit the data.

When hovering at a very low altitude, there is a ground effect, that is, the thrust caused by the interaction between the propeller airflow and the ground of the quadrotor UAV will increase, and the influence of the ground effect on each motor G _i and the propeller radius r _P , speed P _i , height h _i and constant k _G are proportional to the equation as follows:

When two quadrotor UAVs pass the path at the same position at different heights, there is a downwash effect, which will cause the lift force of the bottom vehicle to decrease, and the influence of the downwash effect is simplified to a single one applied to the center of mass of the UAV The force, whose magnitude W depends on the distance (δ _x , δ _{y , δ z ) of the two drones in the coordinate system x, y} , _z and the constants k _D1 , k _D2 , k _D3 , W determined through experiments The expression is as follows:

S12. Construct multi-quadrotor UAV observation space and action space;

In the constructed Gym environment, each action performed by the quadrotor UAV will output an observation vector, and the expression of the observation space of the multi-quadrotor UAV is as follows:

为/>

表示第n个四旋翼无人机的线速度，ω_n为[ω_x,ω_y,ω_z]_n表示第n个四旋翼无人机的角速度；P_n为[P₀,P₁,P₂,P₃]_n表示所有无人机的电机速度。Among them, n∈[0...N] represents the number of quadrotor UAVs; X _n = [x,y,z] _n represents the position of quadrotor UAVs; q _n represents a quaternion, which is used for quadrotors Attitude control of the UAV; r _n , p _n , and y _n respectively represent the roll angle Roll, the pitch angle Pitch, and the yaw angle Yaw, that is, three angles used for attitude estimation;

for />

Indicates the linear velocity of the nth quadrotor UAV, ω _n is [ω _x ,ω _y ,ω _z ] _n indicates the angular velocity of the nth quadrotor UAV; P _n is [P ₀ ,P ₁ ,P ₂ , P ₃ ] _n represents the motor speed of all drones.

As shown in Figure 3, in this embodiment, the quadrotor UAV uses lidar to detect obstacles, and the quadrotor UAV is set to have k lidars in the model, and these k lidars are used to detect the environment Make observations.

Wherein, the scanning angle range of the k lidars is π (k=24 in this embodiment), and the angle between the two lasers is 2π/k; (d ₁ ,...,d _k ) represents k on the horizontal plane The ray length of a radar; if a sensor does not detect anything within a finite distance, then the ray length is the maximum detectable distance. Otherwise, the length is the distance between the UAV and the point detected by the radar; d _i represents the ray length of the i-th radar, and the expression is as follows:

Then the environmental information s _E is defined as follows:

s _E =[ρ _i ,d _i ] ^T ,i=1...k (10)

Among them, _ρi represents the one-hot code of the i-th radar. If the radar detects a detectable object within a limited distance, _ρi is 1, otherwise it is 0. The expression is as follows:

Then for any quadrotor UAV, the action space expression is as follows:

{n:[v _x ,v _y ,v _z ,v _M ] _n } (12)

Among them, [v _x , v _y , v _z , v _M ] _n represents the speed input to the quadrotor UAV, v _x , v _y , v _z represents the component of the unit vector, and v _M represents the magnitude of the required speed; And the action space can also be represented by the speed of the four motors, the expression is as follows:

{n:[P ₀ ,P ₁ ,P ₂ ,P ₃ ] _n } (13)

Finally, the conversion of the input to Pulse Width Modulation (PWM) and motor speed is delegated to a controller consisting of position and attitude control subroutines.

In this embodiment, the step S2 is specifically as follows:

S21. State space and action space of abstract quadrotor UAV;

角速度ω_n、所有无人机的电机速度P_n＝[P₀,P₁,P₂,P₃]；如图4所示的无人机n的第一视角方向与目标连线间的角度β_n、以及第n个无人机的全局坐标(x,y,z)与目标的全局坐标(x_t,y_t,z_t)之间的差异d_0n。The state of the quadrotor UAV n includes: the position of the quadrotor UAV, the quaternion (for the attitude control of the quadrotor UAV) q _n , the roll angle (Roll) r _n , the pitch angle (Pitch) p _n , yaw angle (Yaw)y _n , linear velocity

Angular velocity ω _n , motor speed P _n of all UAVs = [P ₀ , P ₁ , P ₂ , P ₃ ]; as shown in Figure 4, the angle between the first viewing angle direction of UAV n and the target line β _n , and the difference d _0n between the global coordinates (x, y, z) of the nth drone and the global coordinates (x _t , y _t , z _t ) of the target.

In order to make the UAV reach the goal faster and improve the convergence speed, in the state, replace the global position of the UAV with the relative position ΔX _n of the UAV and the target, that is, [Δx, Δy, Δz] _n , then there is no The status s _U of the HMI is as follows:

The state space s of the quadrotor UAV can be obtained from the state s _U of the UAV and the environmental state s _E detected by the lidar, and the expression is as follows:

The action space of the multi-quadrotor UAV environment is composed of the input speed of the quadrotor UAV. Referring to formula (12), for any quadrotor UAV, the expression of the action space is as follows:

a＝[v _x ,v _y ,v _z ,v _M ] ^T (16)

S22. Setting a reward function mechanism to enable the quadrotor UAV to interact with the environment;

The setting of the reward function has a great influence on the performance of the deep reinforcement learning model and determines the strategy of the drone. The reward function R(s,a) represents the environmental feedback obtained by taking action a in state s, and is used to evaluate the quality of the action taken in the current state. ; If R(s,a) is large, it means that taking action a in the current state s is beneficial to achieve the goal. In the next policy update, the probability of taking action a in state s will increase, otherwise, the probability will be reduced.

In order to make the quadrotor UAV reach the target point of the assembly task as soon as possible, a reward function composed of three parts is set in this embodiment to make the quadrotor UAV reach the target point of the assembly task as soon as possible, as follows:

First, set the distance reward R _t between the quadrotor UAV and the target point to promote the quadrotor UAV to reach the target, R _t is set as follows: if the UAV is close to the target, the reward is positive, and the reward is the largest when it reaches the target point; If the drone is far away from the target, the reward is negative, and if the drone does not reach the target after a preset time, the penalty is up to -5. The _Rt expression is as follows:

表示第n个四旋翼无人机离目标的距离，/>

表示第n个下一时刻离四旋翼无人机离目标的距离。Among them, d ₀ represents the distance from the quadrotor UAV to the target,

Indicates the distance from the nth quadrotor UAV to the target, />

Indicates the distance from the quadrotor UAV to the target at the nth next moment.

Secondly, set the distance reward R _o between the quadrotor UAV and the obstacle to make the UAV far away from the obstacle setting, and set R _o as follows: if the distance between the UAV and the nearest obstacle is less than d _safe , then no The human-machine will be punished; if the drone collides with an obstacle, the penalty is -3; if the distance between the drone and the nearest obstacle is less than d _safe , the drone is safe and will not be punished . The expression of R _o is as follows:

表示第n个四旋翼无人机离目标的距离，设置无人机与障碍物的安全距离d_safe。Among them, d _i represents the ray length of the i-th radar, that is, the detection distance from the quadrotor UAV to obstacles or other quadrotor UAVs,

Indicates the distance between the nth quadrotor UAV and the target, and sets the safe distance d _safe between the UAV and the obstacle.

Finally, the angle reward R _a between the quadrotor UAV and the target point is set to make the UAV approach the target direction. If β _n (as shown in Figure 4) is larger, the penalty is greater, and the expression of R _a is as follows:

In this embodiment, the step S3 is specifically as follows:

In this embodiment, the ITD3 algorithm is used to realize multi-UAV assembly mission path planning in an unknown environment. The TD3 algorithm solves the problem of overestimation bias in Deep Deterministic Policy Gradient (DDPG). It is a deterministic policy reinforcement learning algorithm suitable for high-dimensional continuous action spaces.

The TD3 algorithm solves the problem of Q-value estimation error and excessive variance by using double Q-network and delayed update strategy. However, when the environment has delayed rewards, the TD3 algorithm may need more experience to learn how to make correct decisions. In order to solve this problem, this embodiment introduces the N-step return into the TD3 algorithm.

The ITD3 algorithm is obtained by improving the TD3 algorithm with N-step rewards and priority experience playback. The ITD3 algorithm consists of four sub-networks, namely two critic networks and two actor networks. The improved deep reinforcement learning algorithm is implemented by the ITD3 algorithm.

First, the N-step return is introduced into the TD3 algorithm. The N-step return adds the returns of n time steps in the future to provide more comprehensive information than the single-step return; so the algorithm can make better use of delayed returns and improve learning efficiency.

In the case of sparse rewards, most of the state transitions (State Transitions) p(s′|s, a) have no reward information, and one-step rewards will not be effective; N-step rewards are sparsely rewarded by sampling N transitions ( Here an instance value N=4 is set).

In this embodiment, the addition of N-step rewards in TD3 increases the chance of finding and learning from rewarded transfers, thus improving learning efficiency. By adding N-step returns, the equation of the TD3 algorithm critic network is modified. In the j-th round of sampling, the modified time difference error (TimeDifference) function expression δ _j is as follows:

Among them, φ and φ′ represent the parameters of the double-critic network, k represents the return of the k-th step, r _k represents the return of the k-th step, s and a represent the current state and action, s _N and a _N represent the target state and action , Q(s _t , a _t |φ) represents the value function of the critic network, Q′(s _t+N , a _t+N |φ′) represents the value function of the target critic network, and γ represents the discount factor.

Secondly, in the original TD3 algorithm, each experience is evenly sampled, but if there is no difference in severity, the learning efficiency will be low, and adding priority can solve this problem. Prioritized experience replay is a technology that enhances DRL performance. Based on experience replay, it prioritizes samples according to the importance of experience, allowing important samples to be sampled more frequently, thereby improving the learning efficiency and performance of the model. . , at the beginning of the sample, the sampling probability of the jth transition is defined as P(j), expressed as follows:

Among them, p _j represents the priority of the jth experience; α represents a constant used to adjust the sampling weight, and α determines how much priority is used. When α is equal to 0, uniform random sampling will be used.

Then, the _sampling weight w _j for each transition used to _update the network is calculated by Weights, used for normalization:

Finally, using proportional prioritization, the priority of the transfer is updated according to the time difference error, as shown in the following equation:

p _j =|δ _j |+∈ (23)

Among them, δ _j represents the time difference error, and ∈ represents a small value preset to avoid 0 priority.

In this implementation, the step S4 is specifically as follows:

The ITD3 network training network is implemented by a two-part neural network: an actor network consisting of three fully connected layers to complete the state-to-action mapping, and a critic network using four fully connected layers to estimate the Q-value.

In the ITD3 network, for two actor networks, the input is the state and the output is the action. The critic network takes state-action pairs as its input and produces a state-action value function (Q-value). As shown in Figure 5, the ITD3 algorithm training process is as follows:

First, a batch of small samples (s, a, s′, r) are preferentially extracted from the experience playback buffer, and s′ is input into the actor target network. Then, a' is obtained in the next pass, and the state-action pair (s', a') is fed into the critic target network.

和/>

)后，选择较小的一个来计算目标值函数y(r,s′)，目标值函数表达式如下：After getting two target Q-values (

and />

), select the smaller one to calculate the objective value function y(r,s′), the expression of the objective value function is as follows:

Among them, r represents the reward, the discount factor γ has the same value as formula (20), and φ _i is the random parameter of the critic network.

On the other hand, feeding (s,a) into the critic network results in two Q-values (Q ₁ (·) and Q ₂ (·)). They are then used to compute the mean squared error of y(r, s′), and the sum of the mean squared errors is backpropagated to update the parameters of the two critic networks, incorporating N-step replay in the time-differenced error update.

Next, the Q-value obtained from the first critic network is input into the actor model network, and the parameters of the actor network are updated along the direction of increasing Q-value (updated every two iterations).

Finally, a soft update method is adopted to update all target networks.

According to empirical risk minimization, the complexity of the neural network is related to the number of samples. Therefore, according to the state space and action space (size of observations) obtained by the above steps in this embodiment, the network structure of ITD3 is designed as shown in FIG. 6 .

In this embodiment, the ITD3 actor network consists of three FC neural network layers with 512, 128 and 4 nodes. The first and second layers are activated by Rectified Linear Unit (ReLU), and the third layer is activated by Hyperbolic Tangent (tanh) to ensure that the output of the actor network is in the range of [-1,1] Inside. After three FCs, the actor maps the input s _t to the drone's action instruction at _t . For the critic network, it consists of four FCs to estimate the Q-value. The input s _t first enters the FC with the ReLU activation function (the number of nodes is 1024), and then combines the vector and the action _at into a 1028-dimensional vector. After two ReLU activation functions and a tanh activation function, the critic network transfers this vector into Q-values.

After the training is completed, the angular velocity and linear velocity of the quadrotor UAV are controlled through the output action information, so that each quadrotor UAV can successfully reach the specified target mission in the shortest time and complete the assembly task path planning.

To sum up, the method of the present invention first performs Gym environment modeling on multi-quadrotor UAVs based on PyBullet to improve the portability of the present invention. Secondly, the N-step reward is used in the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to obtain more accurate reward estimation, faster learning speed, and better generalization performance; at the same time, priority experience playback is used Reduce the sampling deviation, reduce the model deviation caused by unbalanced sampling, and improve the stability of the algorithm, so that the TD3 algorithm is more suitable for continuous multi-dimensional decision-making problems. Finally, the improved TD3 (Improved TD3, ITD3) network is trained, and by controlling the angular velocity and linear velocity of the quadrotor UAV, it can successfully reach the target point of the assembly task in the shortest time when the environment is unknown.

The embodiments described above are intended to help readers understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Researchers in the field can make various other specific modifications and combinations based on the technical revelations disclosed in the present invention without departing from the essence of the present invention, and these modifications and combinations are still within the protection scope of the present invention.

Claims (5)

1. A multi-quad-rotor unmanned helicopter collective task path planning method based on reinforcement learning comprises the following specific steps:

s1, constructing a PyBullet-based multi-quad-rotor unmanned aerial vehicle Gym environment;

s2, abstracting a state space and an action space of the quadrotor unmanned aerial vehicle, and setting a reward function mechanism to enable the unmanned aerial vehicle to interact with the environment;

s3, performing path planning decision by using an improved deep reinforcement learning algorithm, and performing path planning for each four-rotor unmanned aerial vehicle under an aggregation task;

s4, training the improved deep reinforcement learning network, and controlling the angular speed and the linear speed of the four-rotor unmanned aerial vehicle through the output action information, so that each four-rotor unmanned aerial vehicle successfully reaches a specified target task in the shortest time.

2. The method for planning the collective mission path of the multi-quad-rotor unmanned helicopter based on reinforcement learning according to claim 1, wherein in the step S1, the method is specifically as follows:

S11, constructing a dynamics simulation model of the multi-quad-rotor unmanned helicopter;

the four-rotor unmanned aerial vehicle dynamic equation is formed by the motion equation and the aerodynamic effect of the four-rotor unmanned aerial vehicle, so that the construction of the four-rotor unmanned aerial vehicle dynamic simulation model is completed, and the method is as follows:

using PyBullet to establish a force and torque model acting on each quadrotor unmanned aerial vehicle in Gym, and calculating and updating a kinetic equation of all the quadrotor unmanned aerial vehicles by using a physical engine;

setting the arm length of each four-rotor unmanned aerial vehicle to be L, the mass to be m, the inertial property to be J, the physical constant and the convex collision shape to be described through a separate URDF file, and configuring the 'x' -shaped four-rotor unmanned aerial vehicle;

first, set the gravitational acceleration g and the physical step frequency in PyBullet, force F applied to 4 motors _i And a torque T about the Z-axis of the drone _o With motor speed P _i Is proportional to the square of F _i And P _i The expression is as follows:

F _i ＝k _F ·P _i ² (1)

wherein k is _F And k _T Representing a predetermined constant;

setting the real-time control of the model, the kinetic equation of the quadrotor unmanned plane is expressed as follows:

J ^T T _o ＝Ma+h (3)

wherein J represents a jacobian matrix, M represents an inertia matrix, a represents generalized acceleration, h represents Coriolis and the effect of gravity, and the superscript T represents a transposed operation;

In practice, flying near the ground or near other unmanned aerial vehicles creates additional aerodynamic effects, which are modeled separately and used in combination in the pybullets, including: propeller resistance D, ground effect G acting on single motor _i A wash-down effect W on the centroid;

resistance D is produced to four rotor unmanned aerial vehicle's rotatory screw, and resistance D and four rotor unmanned aerial vehicle linear velocity

Angular speed of propeller and constant drag coefficient matrix k _D Proportional, expressed as follows:

wherein,,

indicating the angular velocity of the propeller, 60 is 60s; constant drag coefficient moment k _D The specific expression is as follows:

k _D ＝diag(k _⊥ ,k _⊥ ,k _|| ) (5)

wherein k is _⊥ Represents the vertical drag coefficient, k _|| Represents parallel drag coefficients, and matrix k _D Fitting the data by using a least square method;

when hovering at very low altitudes, there is a ground effect, the effect of which on each motor G _i Radius r to the propeller _P Speed P _i Height h _i And constant k _G The proportional equation is made as follows:

when two four rotor unmannedWhen the plane passes through the path at the same position with different heights, a downward washing effect exists, the influence of the downward washing effect is simplified into a single acting force applied to the mass center of the unmanned plane, and the size W of the downward washing effect depends on the distance (delta _x ,δ _y ,δ _z ) And a constant k determined experimentally _D1 ，k _D2 ，k _D3 The expression of W is as follows:

s12, constructing an observation space and an action space of the multi-four-rotor unmanned aerial vehicle;

in the constructed Gym environment, when the quadrotor unmanned aerial vehicle executes each action and outputs an observation vector, the observation space expression of the quadrotor unmanned aerial vehicle is as follows:

wherein N is ∈ [ 0..N.)]Representing the number of the quadrotors; x is X _n ＝[x,y,z] _n Representing the position of the quadrotor unmanned aerial vehicle; q _n Representing quaternions for attitude control of a quad-rotor unmanned helicopter; r is (r) _n 、p _n 、y _n Respectively representing a Roll angle Roll, a Pitch angle Pitch and a Yaw angle Yaw, namely three angles for attitude estimation;

is->

Represents the linear velocity omega of the nth quad-rotor unmanned helicopter _n Is [ omega ] _x ,ω _y ,ω _z ] _n Representing the angular velocity of the nth quad-rotor unmanned helicopter; p (P) _n Is [ P ] ₀ ,P ₁ ,P ₂ ,P ₃ ] _n Representing motor speeds of all unmanned aerial vehicles;

in the invention, the four-rotor unmanned aerial vehicle uses the laser radar to detect the obstacle, k laser radars are set in the model, and the k laser radars are used for observing the environment;

wherein the k laser radars have a scanning angle range of pi and an angle between two lasers of 2 pi/k; (d) ₁ ,...,d _k ) Representing the ray lengths of k radars on a horizontal plane; d, d _i The ray length of the ith radar is expressed as follows:

Then the environment information s _E The definition is as follows:

s _E ＝[ρ _i ,d _i ] ^T ,i＝1...k (10)

wherein ρ is _i The unique thermal code representing the ith radar is expressed as follows:

then for any one quadrotor unmanned aerial vehicle, the action space expression is as follows:

{n:[v _x ,v _y ,v _z ,v _M ] _n } (12)

wherein [ v ] _x ,v _y ,v _z ,v _M ] _n Representing the speed of input to a quadrotor drone, v _x ，v _y ，v _z Representing the components of a unit vector, v _M Indicating the magnitude of the desired velocity; and the action space can be represented by the rotating speeds of four motors, and the expression is as follows:

{n:[P ₀ ,P ₁ ,P ₂ ,P ₃ ] _n } (13)

finally, the conversion of the input into pulse width modulation PWM and motor speed is delegated to a controller consisting of position and attitude control subroutines.

3. The method for planning the collective mission path of the multi-quad-rotor unmanned helicopter based on reinforcement learning according to claim 1, wherein the step S2 is specifically as follows:

s21, abstracting a state space and an action space of the quadrotor unmanned aerial vehicle;

the states of the quad-rotor unmanned helicopter include: position and quaternion q of four-rotor unmanned aerial vehicle _n Roll angle roller _n Pitch angle pitch _n Yaw angle Yawy _n Linear velocity

Angular velocity omega _n Motor speed P of all unmanned aerial vehicles _n ＝[P ₀ ,P ₁ ,P ₂ ,P ₃ ]Angle beta between first viewing angle direction of unmanned plane and target link _n And the global coordinates (x, y, z) of the nth unmanned aerial vehicle and the global coordinates (x) of the target _t ,y _t ,z _t ) Difference d between _0n ；

Replacing the global position of the unmanned aerial vehicle with the relative position DeltaX of the unmanned aerial vehicle and the target in the state _n I.e. [ Deltax, deltay, deltaz ]] _n Then the status s of the unmanned aerial vehicle _U The following are provided:

from the state s of the unmanned aerial vehicle _U And the laser radar detected environmental state s _E The state space s of the quadrotor unmanned aerial vehicle can be obtained, and the expression is as follows:

the action space of the four-rotor unmanned aerial vehicle environment is composed of the speeds input to the four-rotor unmanned aerial vehicle, and with reference to the reference formula (12), the action space expression for any one four-rotor unmanned aerial vehicle is as follows:

s22, setting a reward function mechanism to enable the quadrotor unmanned aerial vehicle to interact with the environment;

the reward function R (s, a) represents the environmental feedback resulting from taking action a in state s; setting a reward function consisting of three parts to enable the quadrotor unmanned aerial vehicle to reach an aggregation task target point as soon as possible, wherein the reward function is specifically as follows:

first, distance between the quadrotor unmanned plane and the target point is awarded R _t Forcing the quadrotor unmanned aerial vehicle to reach the target, R _t The expression is as follows:

wherein d ₀ Representing the distance of the quadrotor unmanned from the target,

representing the distance of the nth quad-rotor unmanned helicopter from the target,

representing the distance from the quadrotor unmanned plane to the target at the nth next time;

Secondly, distance between the quadrotor unmanned plane and the obstacle is set to be rewarded R _o Make unmanned aerial vehicle keep away from barrier setting, R _o The expression is as follows:

wherein d _i The ray length of the ith radar is represented, namely the detection distance from the quadrotor unmanned aerial vehicle to an obstacle or other quadrotor unmanned aerial vehicle,

indicating the distance between the nth four-rotor unmanned aerial vehicle and the target, and setting the safety distance d between the unmanned aerial vehicle and the obstacle _safe ；

Finally, setting an angle reward R between the quadrotor unmanned aerial vehicle and the target point _a To promote the unmanned plane to approach to the target direction, if beta _n The larger the penalty the larger, R _a The expression is as follows:

4. the method for planning the collective mission path of the multi-quad-rotor unmanned helicopter based on reinforcement learning according to claim 1, wherein the step S3 is specifically as follows:

the method comprises the steps of N-step report and preferential experience playback, improving a TD3 algorithm to obtain an ITD3 algorithm, wherein the ITD3 algorithm consists of four sub-networks, namely two critique networks and two actor networks, and the improved deep reinforcement learning algorithm is realized by the ITD3 algorithm;

firstly, introducing N-step returns into a TD3 algorithm, wherein the N-step returns add the returns of N time steps in the future to provide more comprehensive information than single-step returns;

in the case of sparse rewards, most state transitions p (s' |s, a) have no rewards information, and the one-step rewards will not be valid; n-step rewards are used for relieving the problem of sparse rewards by sampling N transfers;

Modifying the equation of the TD3 algorithm reviewer network by adding N-step returns, and in the j-th round of sampling, modifying the time difference error function expression delta _j The following are provided:

wherein phi and phi' represent parameters of the double commentator network, k represents the kth step return, r _k Representation ofReturn at step k, s and a represent current state and action, s _N And a _N Representing target states and actions, Q(s) _t ,a _t Phi) represents the value function of the critic network, Q'(s) _t+N ,a _t+N Phi') represents the value function of the target critique network, gamma represents the discount factor;

second, using preferential empirical playback in the original TD3 algorithm, at the beginning of the sample, the sampling probability of the jth transition is defined as P (j), expressed as follows:

wherein p is _j Representing the priority of the j-th experience; α represents a constant for adjusting the sampling weight, α determines how much priority to use, and when α equals 0, uniform random sampling will be employed;

then, the sampling weight w for each transition of the update network _j Calculated by the following formula, which represents the importance of each transfer data, M represents the size of the small batch, max _i w _i Representing the maximized sampling weight for normalization:

finally, using proportion prioritization, updating the transferred priority according to the time difference error, wherein the priority is shown in the following formula:

p _j ＝|δ _j |+∈ (23)

Wherein delta _j Representing a time difference error, e represents a small value preset to avoid a 0 priority.

5. The method for planning the collective mission path of the multi-quad-rotor unmanned helicopter based on reinforcement learning according to claim 1, wherein the step S4 is specifically as follows:

the ITD3 network training network is implemented by a two-part neural network: mapping states to actions is accomplished by an actor network consisting of three fully connected layers, and a reviewer network using four fully connected layers to estimate Q-values;

in the ITD3 network, for both actor networks, the input is state and the output is action; the critics network takes the state-action pair as input and generates a state-action value function (Q-value); the ITD3 algorithm training process is specifically as follows:

firstly, preferentially extracting a batch of small samples (s, a, s ', r) from an experience playback buffer zone, and inputting s' into an actor target network; then, a ' is obtained next time, and the state-action pair (s ', a ') is input into the criticizing target network;

after obtaining two target Q-values

And->

) Then, the smaller one is selected to calculate the target value function y (r, s'), the target value function expression is as follows:

wherein r represents the return, and the discount factor gamma is the same as the value of formula (20), phi _i Random parameters of the critics network;

on the other hand, (s, a) is input into the criticizing network to obtain two Q-values (Q ₁ (. Cndot.) and Q ₂ (. Cndot.); then, using them to calculate the mean square error of y (r, s'), and back-propagating the sum of the mean square errors to update the parameters of two critic networks, and adding N-step playback in the time difference error update;

next, inputting the Q-value obtained from the first criticizing home network into the actor model network, and updating parameters of the actor network in a direction in which the Q-value increases (once every two iterations);

finally, updating all target networks by adopting a soft updating method;

after training, the angular speed and the linear speed of the four-rotor unmanned aerial vehicle are controlled through the output action information, so that each four-rotor unmanned aerial vehicle successfully reaches a specified target task in the shortest time, and the collective task path planning is completed.

Images