CN116679719A - Adaptive path planning method for unmanned vehicles based on dynamic window method and proximal strategy - Google Patents

Abstract

The invention relates to an unmanned vehicle self-adaptive path planning method based on a dynamic window method and a near-end strategy. Firstly, constructing an intelligent body-environment interaction model facing an unmanned vehicle, constructing a near-end strategy optimization learning (PPO) model based on an actor-commentator framework, defining a reward function according to a Dynamic Window Algorithm (DWA) principle and main evaluation factors, determining model parameters in an input layer, an output layer, a hidden layer number, a neuron number and the like, and constructing a DWA-PPO deep reinforcement learning model; and then, using the established DWA-PPO deep reinforcement learning model, continuously iterating and training, and finally converging a network model capable of representing the potential relation between surrounding environment information and the evaluation function weight parameter to complete the construction of the self-adaptive PPO-ADWA algorithm. And finally, verifying feasibility and effectiveness of the PPO-ADWA-based unmanned vehicle self-adaptive path planning strategy through a simulation comparison experiment.

Description

Adaptive path planning method for unmanned vehicles based on dynamic window method and proximal strategy

technical field

The invention relates to the technical field of unmanned driving path planning and autonomous navigation, in particular to an adaptive path planning method for unmanned vehicles based on a dynamic window method and a proximal strategy.

Background technique

In recent years, with the rapid development of science and technology, a new round of technological industrial revolution represented by the Internet, artificial intelligence, big data, etc. is redefining all walks of life in society, and the traditional automobile industry is facing profound industrial changes. Traditional cars are developing towards intelligence and unmanned vehicles. Intelligent networked cars and self-driving cars have become the strategic direction of the development of the global auto industry. Intelligent driving technology mainly includes environment perception, navigation and positioning, path planning and control decision-making, etc. Path planning is an important part of intelligent driving, which is of great significance to the development of intelligent driving technology.

Path planning is an important part of autonomous driving smart cars. Path planning technology can be summarized as path planning refers to planning a safe and feasible collision-free path through algorithms in a known environment, and selecting the optimal avoidance path from the starting point to the end point. The obstacle path is essentially the optimal solution under several constraints, and path planning is a key part of the unmanned navigation technology for smart vehicles. Path planning algorithms can be divided into global planning based on complete regional information understanding and local planning based on local regional information understanding. Dynamic Window Approach (DWA), as a local path planning method considering the motion performance of smart cars, is widely used in smart car path navigation. The decision-making function in the DWA algorithm is its evaluation function, which includes three parts: orientation angle function, obstacle function, and speed function. The evaluation function is the weighted sum of these three sub-functions. In the classic DWA algorithm, the three functions correspond to The weight is a fixed value. However, when the smart car is exploring the end point, the surrounding obstacle environment is complex and changeable. Different obstacle distributions require different weights. The fixed weight value method of the classic DWA algorithm is easy to make the smart car fall into a local optimum or Target unreachable. Therefore, with the help of the proximal strategy optimization algorithm in deep reinforcement learning, the classic DWA algorithm is improved.

Contents of the invention

The purpose of the present invention is to solve the problem that the agent cannot find the end point or calculate the optimal path because the weight coefficient in the evaluation function cannot be dynamically adjusted in the face of different obstacle environments. The self-adaptive path planning method for unmanned vehicles based on the terminal strategy is improved on the basis of the classic DWA algorithm. The weight parameters in the improved classic DWA algorithm are combined with the optimization of the proximal strategy in deep reinforcement learning. Through learning and training, different static obstacles are obtained. The model parameters of the object are used to complete the construction of the adaptive PPO-ADWA algorithm.

In order to achieve the above object, the technical solution of the present invention is: a method for self-adaptive path planning of unmanned vehicles based on dynamic window method and proximal strategy, comprising the following steps:

Step 1. Construct the agent-environment interaction model, the unmanned vehicle is used as the agent in deep reinforcement learning, and the obstacle map is used as the environment;

Step 2. Establish the DWA algorithm model, which includes: speed range, angular velocity range, acceleration range, angular acceleration range parameters, and the main elements of the DWA algorithm and evaluation functions according to the Ackerman smart car;

Step 3. Establish a proximal policy optimization learning PPO model based on the actor-critic framework, simulate and establish the actual application scenario of the unmanned vehicle as the learning environment of the model, and determine the state and action in the model according to the application scenario;

Step 4. Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main line reward and the sub-target reward; and determine the DWA-PPO including the input layer, output layer size, number of hidden layers and number of neurons Deep reinforcement learning model parameters, complete the instantiation of DWA-PPO deep reinforcement learning model;

Step 5. Build an adaptive PPO-ADWA algorithm, use the established DWA-PPO deep reinforcement learning model, and simulate the navigation planning of unmanned vehicles in the environment of randomly generated complex static obstacles to collect data for training DWA-PPO The training set of the deep reinforcement learning model, through repeated iterations, converges to a model that can output the corresponding weight parameters according to the changes in the distribution of surrounding obstacles, and completes the construction of the adaptive PPO-ADWA algorithm;

Step 6. Demonstrate the self-adaptive adjustment capability of unmanned vehicle path planning based on the self-adaptive PPO-ADWA algorithm through simulation comparison experiments.

Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention is aimed at the weight coefficient in the evaluation function of the traditional DWA algorithm, and its value will not change with the environment in which the smart car is located and its own motion state. For the problem of dynamic adjustment, the proximal strategy optimization algorithm in deep reinforcement learning is used to construct the DWA-PPO deep reinforcement learning model, and the network model is obtained through continuous iterative training, so as to output the model parameters of the corresponding weight parameters, and complete the adaptive PPO-ADWA Algorithm construction; the method of the present invention solves the problem that the agent cannot find the end point or calculate the optimal path because the weight coefficient in the evaluation function cannot be dynamically adjusted when facing different obstacle environments.

Description of drawings

Figure 1 is a schematic diagram of the agent-environment interaction model.

Figure 2 is a schematic diagram of the principle of the DWA algorithm.

Figure 3 shows the velocity angular velocity window.

Figure 4 is and δ schematic diagram.

Figure 5 is a schematic diagram of the actor-critic framework.

Figure 6 is state s.

Figure 7 shows the policy network structure.

Figure 8 shows the value network structure.

Figure 9 is the DWA-PPO model.

Figure 10 is the change curve of score and arrival rate.

Figure 11 is the simulation environment.

Figure 12 is a classic DWA.

Figure 13 is PPO-ADWA.

Figure 14 is the weight parameter change curve.

Fig. 15 is a flowchart of the method of the present invention.

Detailed ways

The technical solution of the present invention will be specifically described below in conjunction with accompanying drawings 1-15.

As shown in Figure 15, the present invention provides a method for self-adaptive path planning for unmanned vehicles based on dynamic window method and proximal strategy, including the following steps:

Step 1. Construct the agent-environment interaction model, the unmanned vehicle is used as the agent in deep reinforcement learning, and the obstacle map is used as the environment;

Step 2. Establish the DWA algorithm model, which includes: speed range, angular velocity range, acceleration range, angular acceleration range parameters, and the main elements of the DWA algorithm and evaluation functions according to the Ackerman smart car;

Step 3. Establish a proximal policy optimization learning PPO model based on the actor-critic framework, simulate and establish the actual application scenario of the unmanned vehicle as the learning environment of the model, and determine the state and action in the model according to the application scenario;

Step 4. Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main line reward and the sub-target reward; and determine the DWA-PPO including the input layer, output layer size, number of hidden layers and number of neurons Deep reinforcement learning model parameters, complete the instantiation of DWA-PPO deep reinforcement learning model;

Step 5. Build an adaptive PPO-ADWA algorithm, use the established DWA-PPO deep reinforcement learning model, and simulate the navigation planning of unmanned vehicles in the environment of randomly generated complex static obstacles to collect data for training DWA-PPO The training set of the deep reinforcement learning model, through repeated iterations, converges to a model that can output the corresponding weight parameters according to the changes in the distribution of surrounding obstacles, and completes the construction of the adaptive PPO-ADWA algorithm;

Step 6. Demonstrate the self-adaptive adjustment capability of unmanned vehicle path planning based on the self-adaptive PPO-ADWA algorithm through simulation comparison experiments.

The specific implementation of each step is as follows:

Step 1, as shown in Figure 1, build an agent-environment interaction model, the unmanned vehicle is used as the agent in deep reinforcement learning, and the obstacle map is used as the environment;

The agent plays the role of decision-making and learning in the deep reinforcement learning system. It is mainly responsible for the output of action information and receiving rewards and states. The environment is the interaction object of the agent. The interaction process includes the following three steps:

(1) The agent is determined by the environment state Observed information /> is the state space, which is the value set of the environment state; /> is the observation space, and is the value set of agent observations.

(2) The agent makes a corresponding decision based on the known O _t , and decides the action to be imposed on the environment Is the set of action values.

(3) The environment is affected by A _t , its own state S _t is transferred to S _t+1 , and rewards are given to the agent is the value set of rewards. Therefore, the discretized agent-environment interaction model can be represented by the following sequence:

S ₀ ,O ₀ ,A ₀ ,R ₀ ,S 1 ,O ₁ ,A ₁ ,R ₁ ,S ₂ ,O ₂ ,A ₂ ,R _{2 ,} …,S _T =S _termination

When the state of the environment can be completely observed by the agent, then there is S _t =O _t , which can be simplified as:

S ₀ ,A ₀ ,R ₀ ,S ₁ ,A ₁ ,R ₁ ,S ₂ ,A ₂ ,R ₂ ,…,S _T =S _termination

Step 2. Establish the DWA algorithm model, and determine according to the Ackerman smart car, including: speed range, angular velocity range, acceleration range, angular acceleration range parameters and the main element evaluation function of the DWA algorithm;

The DWA algorithm is a local path planning method that intuitively understands the map environment of the unmanned vehicle from the perspective of velocity space. The attainable velocity-angular-velocity window V _win ; discretize it and combine the discretized velocity-angular velocity; the unmanned vehicle traverses all the combinations and simulates forward m _Δt duration according to the given motion model, and obtains the simulated trajectory set τ, That is, a series of point sets; the evaluation function gives the scores of all simulated trajectories in the simulated trajectory set τ, and selects the combination corresponding to the highest-scoring trajectory τ _b ; drives the unmanned vehicle forward with this combination for a length of Δ _t , and reaches the time t+1; Repeat this until the end. m is the number of sampling steps, and _Δt is the sampling interval, as shown in Figure 2.

At time t, the V _win of the unmanned vehicle is constrained by its own hardware conditions and the surrounding environment. Consider the following three constraints:

(1) Limit velocity angular velocity constraint:

V _lim ＝{(v,w)|v∈[v _min ,v _max ]^w∈[w _min ,w _max ]}

(2) Velocity angular velocity constraint of acceleration limitation:

(3) Velocity and angular velocity constraints limited by braking distance:

Above, v _min and v _max are limit linear velocity, w _min and w _max are limit angular velocity. v _cu and w _cu are the current linear velocity and angular velocity, is the limit linear acceleration, /> is the limit angular acceleration. dist(v,w) is the shortest distance from the simulated trajectory corresponding to the combination of velocity and angular velocity (v,w) to the obstacle. V _win at the final time t is expressed as:

V _win = V _lim ∩ V _acc ∩ _{V dis}

Specifically, as shown in Figure 3, the evaluation function includes three sub-functions, which are a comprehensive consideration of the three factors of the driving speed of the unmanned vehicle, the risk of collision with obstacles, and the heading of the unmanned vehicle, as follows:

G(v,w)=σ(αheading(v,w)+ηdist(v,w)+γvel(v,w))

in Indicates the heading angle of the unmanned vehicle, and δ is the angle between the line connecting the unmanned vehicle and the target point and the positive direction of the x-axis, as shown in Figure 4.

dist(v,w) is the Euclidean distance from the simulated trajectory to the nearest obstacle, vel(v,w) represents the linear velocity of the unmanned vehicle, and α, η, γ are three weight coefficients. It can be seen from the above that the evaluation function is composed of sub-functions of different dimensions. The normalization function σ() in the formula is equivalent to dimensionless learning, which can unify data of different dimensions into the same reference system for combination or comparison. In order to avoid evaluation bias due to different scales of data, the details are as follows:

dist(v _i ,w _j ) performs the same normalization operation as vel(v _i ,w _j ).

The unmanned vehicle obtains the simulated trajectory according to the uniform motion model. Under the assumption of the motion model, the linear velocity and angular velocity of the unmanned vehicle remain unchanged, and the change in the direction of the linear velocity has a linear relationship with time. In order to simplify the model and speed up the calculation, It can be considered that the velocity direction remains unchanged in a small time interval, so the uniform motion model can be discretized, x _t and y _t represent the horizontal and vertical coordinates of the smart car at time t, Indicates the heading angle at time t, and v _t and w _t indicate the speed and angular velocity at time t, as shown in the following formula.

Step 3. Establish a proximal policy optimization learning (PPO) model based on the actor-critic framework (as shown in Figure 5), simulate and establish the actual application scenario of the unmanned vehicle as the learning environment of the model, and determine the state in the model according to the application scenario and action;

The Proximal Policy Optimization (PPO) algorithm is based on the objective function Add D _KL (p||q) penalty item in the specific as follows:

In the formula Integrate the parameter θ ₁ to obtain the objective function of policy learning based on importance sampling, θ is the strategy π parameter, when the strategy is better, the objective function /> The larger γ is the parameter introduced by Monte Carlo approximation, U _t is the parameter in the strategy gradient, π(a _t |s _t ; θ ₁ ) is the target strategy, π(a _t |s _t ; θ ₂ ) is Behavior Policy, /> is the mathematical expectation of the policy network, β is a hyperparameter, the greater the difference between the distribution q and p, the greater the D _KL (p||q) item, /> The greater the penalty, otherwise the D _KL (p||q) item is smaller, /> The smaller the penalty received, the goal of reinforcement learning is to maximize So the /> with the penalty term Be able to control the behavior and the target policy within a certain similarity range.

The unmanned vehicle looks for the optimal path that can connect the starting point and the end point in the obstacle environment, so the actual application scene of the unmanned vehicle is used as the learning environment of the model is the obstacle map.

The state s in the model is the environmental information perceived by the unmanned vehicle using the sensor, and can also include its own position and motion state information. The state s is the only source of information for unmanned vehicle action decisions, and it is also an important basis for maximizing returns. Therefore, the quality of the state s directly affects whether the algorithm can converge, the convergence speed and the final performance. The state s can be understood as a high-dimensional vector of surrounding environment information. The ultimate goal of the unmanned vehicle is to reach the end point with the optimal path. Obviously, the position and state of the unmanned vehicle, the distribution of surrounding obstacles, and the location information of the target point will be the actions of the unmanned vehicle. basis for decision-making. In order to better fit the actual application scenario, the laser radar can scan the reflected information for a week at a scanning interval of 2 degrees as the main part of the state s. In addition, the state s should also include the speed v _t of the unmanned vehicle, the angular velocity w _t , and the heading angle and the current target point position information (x ^g _t , y ^g _t ), as shown in Fig. 6 . The specific method is to use the policy network output to replace the fixed weight of the evaluation function to construct an adaptive evaluation function. Obviously, action a corresponds to the weight (α, η, γ) in the evaluation function, so define action a as:

a＝[μ ₁ ,σ ₁ ,μ ₂ ,σ ₂ ,μ ₃ ,σ ₃ ]

Where [μ ₁ ,σ ₁ ] is the mean and variance, used to describe the probability density function of the weight α:

By analogy, [μ ₂ ,σ ₂ ] is the mean and variance, which is used to describe the probability density function of weight η, and [μ ₃ ,σ ₃ ] is the mean and variance, which is used to describe the probability density function of weight γ. Then randomly sample (α, η, γ) according to their respective probability density functions, and map the action to the [-1, 1] interval through the Tanh function.

After the state s and action a are determined, the number of neurons in the input and output layers of the policy network π(a|s; θ) and the value network q(s, a; w) is also determined. The structural diagrams of strategy network and value network are shown in Figures 7 and 8:

Step 4. Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main line reward and the sub-goal reward; and determine the model parameters including the input layer, output layer size, number of hidden layers and number of neurons, etc. Complete the instantiation of the DWA-PPO deep reinforcement learning model;

The reward function is the core content of the learning model. The rewards obtained by unmanned vehicles can be divided into main-line rewards and sub-target rewards according to whether they are triggered by main-line events:

Mainline reward: The so-called mainline reward can be understood as the settlement reward for the agent to reach the terminal state. In this paper, it is the reward R _mian ^goal obtained when the unmanned vehicle navigates to the end point, the penalty reward R _mian ^out when the maximum number of iteration steps is exceeded, and when The penalty reward R _mian ^coll when the unmanned vehicle collides with an obstacle.

Sub-goal rewards: Rewards other than the main line rewards are called auxiliary rewards, and their main form is sub-goal rewards. Combined with the actual application scenario of unmanned vehicle navigation planning in an obstacle environment, the analysis of local key points, environmental status, unmanned vehicle motion state, relative relationship between unmanned vehicle and target points and other factors is crucial for unmanned vehicles to find the optimal path. For the impact of a main task, the following sub-objective rewards are given:

(1) Energy penalty reward R _sub ^step : the existence of R _sub ^step can limit the energy consumption of the unmanned vehicle on the one hand, and at the same time can promote the unmanned vehicle to find the optimal path; E _t is the tth step unmanned vehicle Driving _Δt at speed _vt , the energy consumed in this process, after normalization, defines R _sub ^step as:

(2) Distance change reward R _sub ^dis : During this process, the unmanned vehicle may move away from the end point locally due to avoiding obstacles, but must move closer to the end point globally. Therefore, a reward related to the distance between the position of the unmanned vehicle and the target point can be defined. R _sub ^dis should be a positive reward, and the greater the distance moving towards the end point, the greater the R _sub ^dis .

(3) Obstacle distance reward R _sub ^obs : r _t ^obs is defined as when there are no obstacles within the safe distance of the unmanned vehicle and the unmanned vehicle brakes at the maximum deceleration, the unmanned vehicle will not collide during the planning process It is the first premise to ensure driving safety. After normalization, define R _sub ^obs as:

(4) Azimuth angle reward R _sub ^head : the goal of the unmanned vehicle is to reach the destination, so in the navigation, it is considered that the closer it is to the target point, the better the heading angle of the unmanned vehicle; r ^head is defined as the heading of the unmanned vehicle is very close to the best The positive reward will only be obtained when the azimuth is set. After normalization, define the R _sub ^head as:

To sum up, it can be seen that the reward R _t for the unmanned vehicle at the t-th step is the following formula, Reward modifiers for subgoals.

The AC framework constructs a value network to approximate the action value in the policy gradient, so the network architecture includes at least a value network and a policy network architecture. According to the value network loss function:

The learning objectives of the value network are:

It can be seen that its learning objectives include part of its own predictions Assuming that the value network itself has an overestimation of the action value Q(s,a), then the way the value network uses itself to learn itself will cause the overestimation problem to be continuously amplified, and this overestimation is unevenly overestimated, seriously Affecting network training, this phenomenon is called Bootstrapping. In order to prevent the bootstrapping phenomenon of the value network, use w ^- to construct a target value network as The parameter structure of the network is consistent with the value network but the specific values are different, which is used to calculate the TD error:

The initial parameters of the target value network are consistent with the value network, μ is the parameter, and the sum of coefficients is guaranteed to be 1. For subsequent updates, refer to the following formula:

In summary, the network architecture under the DWA-PPO reinforcement learning model includes three parts: policy network π(a|s;θ), value network q(s,a;w) and target value network q ^T (s,a ;w). The DWA-PPO reinforcement learning model is shown in Figure 9.

To sum up, the construction model includes agent, environment, critic module and actor module. The critic module includes the value network error function L(w), the value network q(s,a;w), and the target value network ^qT (s,a;w ⁻ ). The actor module includes target network π(a|s; θ ₁ ), behavior network π(a|s; θ ₂ ), policy network objective function The beginning stage of training is the collection of training sets, as shown by the black line in the figure: at the initial moment of the 0th round, the unmanned vehicle uses the perception and positioning system to observe the state s ₀ from the environment, and the behavior network π(a|s; θ ₂ ) After receiving s ₀ , output a Gaussian distribution π(A ₀ |s ₀ ; θ ₁ ) about the action A ₀ , and then randomly select the determined action a ₀ from the probability distribution and pass it to the smart car to obtain the evaluation function G of the DWA algorithm at the initial moment ₀ (v, w), complete the evaluation of the simulated trajectory set of the DWA algorithm at the initial moment, and transmit the velocity and angular velocity command of the optimal trajectory to the unmanned vehicle motion control module to drive the unmanned vehicle to move. So far, information such as the position, orientation angle, and distribution of surrounding obstacles of the unmanned vehicle has changed, and the environment has changed to state s ₁ , and the reward function will also feed back the reward r ₀ to the critic module according to the changed information. When s ₁ is not in the termination state s _n , the round enters the next moment; otherwise, the map and unmanned vehicle state are reset, and the trajectory collection of the next round is carried out until the i round is collected, and finally the training set is obtained:

χ=[χ ₀ ,χ ₁ ,…,χ _i ]

χ ₀ ＝[s ₀ ⁰ ,a ₀ ⁰ ,r ₀ ⁰ ,…,s _n-1 ⁰ ,a _n-1 ⁰ ,r _n-1 ⁰ ,s _n ⁰ ]

Step 5. Construct the PPO-ADWA algorithm, use the established DWA-PPO deep reinforcement learning model, and simulate the navigation planning of the unmanned vehicle in the environment of randomly generated complex static obstacles, so as to collect the training set for training the network model , through repeated iterative convergence, the model parameters that can output the corresponding weight parameters according to the changes in the distribution of surrounding obstacles, and complete the construction of the adaptive PPO-ADWA algorithm;

After obtaining the training set, use the error function L(w) to backpropagate to update the value network q(s,a;w); use the error function of the PPO algorithm Backpropagation updates π(a|s; θ ₁ ). Let the current network parameters of the network q(s,a;w), q ^T (s,a;w ^- ), π(a|s;θ ₁ ) be w _now , /> respectively θ _now , repeat the following steps Z times to complete the generation update:

(1) Randomly extract M _I (minimum batch size) states s _N ^I from the shuffled training set χ.

(2) Use q ^T (s, a; w ^- ) to calculate the K-step TD error MTD _N ^I starting from the state s _N ^I :

(3) Use the value network q(s,a;w) to calculate the action value estimate when the state s _N ^I is:

q _N ^I ＝ Q(s _N ^I , a _N ^I ; w _now )

(4) Calculate L(w):

(5) calculation

(6) Update the value network, strategy network, and target value network:

Assume that the parameter before updating is θ _now , and the parameter θ _new is obtained after the importance sampling update. Assume that the parameter before updating is w _now and the parameter w _new is obtained after updating the policy learning. It is assumed that before updating, it is to prevent the value network from bootstrapping. The introduced parameter w ^- _now is updated with μ as a parameter to ensure that the sum of the coefficients is 1 to obtain w ^- _new . After completing Z updates, assign the parameters of the target network π(a|s; θ ₁ ) to the target network π(a|s; θ ₁ ), which is recorded as a generation update, and then clear the training set and re-enter the next generation update until The model converges.

Figure 10 shows the change curve of the average score and arrival rate of each generation of unmanned vehicles in the deep reinforcement learning environment during the network training process. With the iterative convergence of the model, the network model gradually learns to correctly guide the path planning of unmanned vehicles parameter network. Complete the construction of adaptive PPO-ADWA algorithm.

Step 6. Demonstrate the self-adaptive adjustment capability of unmanned vehicle path planning based on PPO-ADWA through simulation comparison experiments;

In order to verify the self-regulation ability of the unmanned vehicle path planning based on the PPO-ADWA algorithm, this section will verify its robustness in the randomly generated complex static obstacle environment. The simulation environment is shown in Figure 11. The size of the map is 60m×60m. The green dot is the starting point, the blue five-pointed star is the end point, and the black geometric figures represent obstacles. The shapes of obstacles include regular polygons and circles, and the size and number of obstacles Randomly generated within a certain range. The performance under 100 maps with different obstacle positions, the simulation results are shown in Table 1.

Table 1 Comparison of simulation results

The arrival rate of the unmanned vehicle path planning results under PPO-ADWA is 84%, which is 6 percentage points higher than that under the classic DWA; the average path length is 93.04m, and the path efficiency is increased by 5.00%; the average number of steps 251.95, a 4.85% reduction in average step cost. The classic DWA planning results are shown in Figure 12, and the PPO-ADWA planning results are shown in Figure 13. During the unmanned vehicle path planning process of the PPO-ADWA fusion strategy, the change curve of the weight parameters is shown in Figure 14. It can be seen that the weight parameters generally maintain the numerical relationship of η>γ>α.

The above are the preferred embodiments of the present invention, and all changes made according to the technical solution of the present invention, when the functional effect produced does not exceed the scope of the technical solution of the present invention, all belong to the protection scope of the present invention.

Claims (6)

1. The unmanned vehicle self-adaptive path planning method based on the dynamic window method and the near-end strategy is characterized by comprising the following steps:

firstly, constructing an intelligent body-environment interaction model, wherein an unmanned vehicle is used as an intelligent body in deep reinforcement learning, and an obstacle map is used as an environment;

establishing a DWA algorithm model, and determining the DWA algorithm model according to the Ackerman intelligent vehicle comprises the following steps: speed range, angular speed range, acceleration range, angular acceleration range parameters, and the main elements and evaluation functions of the DWA algorithm;

thirdly, establishing a near-end strategy optimization learning PPO model based on an actor-critter framework, simulating and establishing an actual application scene of the unmanned vehicle as a learning environment of the model, and determining states and actions in the model according to the application scene;

step four, constructing a DWA-PPO deep reinforcement learning model, and defining a reward function comprising main line rewards and sub-target rewards; determining parameters of a DWA-PPO deep reinforcement learning model including parameters of an input layer, an output layer, the number of hidden layers and the number of neurons, and completing instantiation of the DWA-PPO deep reinforcement learning model;

step five, constructing a self-adaptive PPO-ADWA algorithm, namely simulating navigation planning of the unmanned vehicle in a randomly generated complex static obstacle environment by using the established DWA-PPO deep reinforcement learning model to collect a training set for training the DWA-PPO deep reinforcement learning model, and outputting a model of corresponding weight parameters according to the distribution change of surrounding obstacles by repeated iteration convergence to finish the construction of the self-adaptive PPO-ADWA algorithm;

step six, the self-adaptive adjustment capability of unmanned vehicle path planning based on the self-adaptive PPO-ADWA algorithm is demonstrated through a simulation comparison experiment.

2. The unmanned vehicle self-adaptive path planning method based on the dynamic window method and the near-end strategy according to claim 1, wherein the following steps are specifically implemented:

the intelligent agent is responsible for outputting action information, receiving rewards and states, the environment is an interactive object of the intelligent agent, and the interactive process comprises the following three steps:

(1) The intelligent agent is controlled by the environment stateObserve information +.> The state space is a value set of environmental states; />The system is an observation space and a value set of the observed quantity of the agent;

(2) The agent being derived from known O _t Making a corresponding decision to decide the action to be applied to the environment Is an action value set;

(3) Environmental subject A _t Influence, self state S _t Transition to S _t+1 And awarding rewards to the agent Is a value set of rewards; the discretized agent-environment interaction model is thus represented by the following sequence:

S ₀ ,O ₀ ,A ₀ ,R ₀ ,S ₁ ,O ₁ ,A ₁ ,R ₁ ,S ₂ ,O ₂ ,A ₂ ,R ₂ ,…,S _T ＝S _{termination of}

When the state of the environment can be completely observed by the intelligent agent, S is present _t ＝O _t To be simplified as:

S ₀ ,A ₀ ,R ₀ ,S ₁ ,A ₁ ,R ₁ ,S ₂ ,A ₂ ,R ₂ ,…,S _T ＝S _{termination of} 。

3. The unmanned vehicle self-adaptive path planning method based on the dynamic window method and the near-end strategy according to claim 2, wherein the step two is specifically implemented as follows:

the DWA algorithm is a local path planning method for intuitively understanding the map environment where the unmanned vehicle is located from the perspective of speed space, and the working flow is as follows: taking into consideration the constraint of each condition at the moment t on the speed and angular velocity, obtaining a speed and angular velocity window V which can be reached by the unmanned vehicle at the moment t _win The method comprises the steps of carrying out a first treatment on the surface of the Discretizing the speed and angular velocity, and combining the discretized speed and angular velocity; the drone traverses all combinations and simulates the forward m delta according to a given motion model _t Obtaining a simulation track set tau, namely a series of point sets, according to the duration; the evaluation function gives the scores of all the simulated tracks in the simulated track set tau, and the track tau with the highest score is selected _b Corresponding combinations; driving the unmanned aerial vehicle with the combination for a forward time delta _t Reaching the time t+1; taking the cycle to the end point, m is the sampling step number, delta _t Is the sampling interval;

at time t, speed angular velocity window V of unmanned vehicle _win Constrained by its hardware conditions and the surrounding environment, consider the following three-point constraint:

(1) Limit speed angular speed constraint:

V _lim ＝{(v,w)|v∈[v _min ,v _max ]∧w∈[w _min ,w _max ]}

(2) Acceleration limited speed angular velocity constraint:

(3) Speed angular velocity constraint for braking distance limitation:

above, v _min 、v _max Is the limit linear velocity, w _min 、w _max For the limit angular velocity v _cu 、w _cu For the current linear velocity and the angular velocity,for extreme linear acceleration, +.>For the limit angular acceleration, dist (v, w) is the nearest distance from the obstacle of the simulated track corresponding to the velocity angular velocity combination (v, w); finally, speed angular velocity window V of unmanned vehicle at t moment _win Expressed as:

V _win ＝V _lim ∩V _acc ∩V _dis

the evaluation function comprises three subfunctions, which are comprehensive consideration of three factors of the unmanned vehicle running speed, the obstacle collision risk and the unmanned vehicle heading, and specifically comprises the following steps:

G(v,w)＝σ(αheading(v,w)+ηdist(v,w)+γvel(v,w))

wherein the method comprises the steps of The course angle of the unmanned vehicle is represented, delta is the included angle between the connecting line of the unmanned vehicle and the target point and the positive direction of the x-axis; dist (v, w) is the Euclidean distance from the simulated track to the nearest obstacle, vel (v, w) represents the linear speed of the unmanned vehicle, and alpha, eta and gamma are three weight coefficients; from the above, the evaluation function is notThe homonymy subfunction is formed, the normalization function sigma () in the formula is equivalent to dimensionless learning, and can unify the data of different dimensions under the same reference system for combination or comparison, so as to avoid evaluation deviation caused by different dimensions of the data, and the method is concretely as follows:

dist(v _i ,w _j ) And vel (v) _i ,w _j ) Performing the same normalization operation;

the unmanned vehicle obtains a simulation track according to a uniform motion model, under the assumption condition of the uniform motion model, the linear speed and the angular speed of the unmanned vehicle are kept unchanged, the linear speed direction change amount and time are in linear relation, the speed direction is kept unchanged in a tiny time interval for simplifying the model to accelerate the operation, and therefore the uniform motion model is discretized, and x is calculated _t 、y _t Represents the abscissa of the intelligent vehicle at the time t,indicating the course angle at time t, v _t 、w _t The speed and angular velocity at time t are represented by the following formulas:

4. the unmanned vehicle self-adaptive path planning method based on the dynamic window method and the near-end strategy according to claim 3, wherein the third implementation is as follows:

the approach of the near-end policy optimization algorithm is that the approach is based on the objective functionIncrease D _KL The (p||q) penalty term is specifically as follows:

in the middle ofTo the parameter theta ₁ The integration of (2) to obtain the target function of strategy learning based on importance sampling, theta is the strategy pi parameter, when the strategy is better, the target function +.>The larger, γ is the parameter introduced for the Monte Carlo approximation, U _t Pi (a _t |s _t ；θ ₁ ) For the target policy, pi (a _t |s _t ；θ ₂ ) For behavioural policy->For mathematical expectation of policy network, beta is super parameter, and D is larger as the difference between the distribution q and p is larger _KL The larger the (p||q) term is, ++>The greater the penalty received, the conversely D _KL The smaller the (p||q) term is, ++>The smaller the penalty is, the goal of reinforcement learning is to maximize +.>Thus have penalty term->The behavior and the target strategy can be controlled within a preset similarity range;

the unmanned vehicle searches an optimal path capable of connecting the starting point and the end point in the obstacle environment, so that the learning environment taking the actual application scene of the unmanned vehicle as a model is an obstacle map;

the state s in the model is environmental information perceived by the unmanned vehicle by using a sensor and comprises self position and motion state information; the laser radar scans the information reflected by a circle at a scanning interval of 2 degrees to form a main part of a state s, and the state s also comprises the speed v of the unmanned vehicle _t Angular velocity w _t Course angleCurrent target point position information (x ^g _t ,y ^g _t ) The specific method is that a strategy network is utilized to output fixed weights for replacing an evaluation function, a self-adaptive evaluation function is constructed, and obviously, an action a corresponds to the weights (alpha, eta, gamma) in the evaluation function, so that the action a is defined as follows:

a＝[μ ₁ ,σ ₁ ,μ ₂ ,σ ₂ ,μ ₃ ,σ ₃ ]

wherein [ mu ] ₁ ,σ ₁ ]For the mean and variance, the probability density function used to describe the weight α, the same applies:

[μ ₂ ,σ ₂ ]for mean and variance, probability density function for describing weight η, [ mu ] ₃ ,σ ₃ ]The mean value and the variance are probability density functions for describing the weight gamma; then randomly sampling and determining (alpha, eta, gamma) according to the respective probability density function, and mapping the actions to [ -1,1 through the Tanh function]The interval is within;

after the state s and the action a are determined, the number of neurons of the output layer of the input and output layers of the strategy network and the value network are also determined.

5. The unmanned vehicle self-adaptive path planning method based on the dynamic window method and the near-end strategy according to claim 4, wherein the fourth implementation is as follows:

the rewarding function is to learn core content in a DWA-PPO deep reinforcement learning model, and rewards obtained by unmanned vehicles are divided into main line rewards and sub-target rewards according to whether the rewards obtained by triggering main line events are:

main line rewards: the main line rewards are settlement rewards for the agent reaching the ending state, namely rewards R obtained by the navigation of the unmanned vehicle to the ending point _mian ^goal Punishment rewards R when the maximum number of iterative steps is exceeded _mian ^out And punishment rewards R when unmanned vehicle collides with obstacle _mian ^coll ；

Sub-target rewards: rewards other than the main rewards are called auxiliary rewards, and the main form of the rewards is sub-target rewards; and analyzing the influence of factors such as local key points, environment states, movement states of the unmanned aerial vehicle, relative relation between the unmanned aerial vehicle and target points and the like on a main line task of the unmanned aerial vehicle for finding an optimal path by combining with the actual application scene of navigation planning of the unmanned aerial vehicle in an obstacle environment, and giving the following sub-target rewards:

(1) Energy penalty prize R _sub ^step ：R _sub ^step On one hand, the energy consumption of the unmanned vehicle can be limited, and meanwhile, the unmanned vehicle can be promoted to find an optimal path; e (E) _t At speed v for the t step unmanned vehicle _t Running delta _t The energy consumed by the process, normalized, defines R _sub ^step The method comprises the following steps:

(2) Distance change reward R _sub ^dis : defining a prize related to the position-target point distance of the unmanned vehicle, R _sub ^dis Should be a positive prize, and R is the greater the distance moved in the direction of the end point _sub ^dis The larger;

(3) Obstacle distance rewards R _sub ^obs ：r _t ^obs Defining that when no obstacle exists in the safety distance of the unmanned vehicle and the unmanned vehicle brakes at the maximum deceleration, the unmanned vehicle does not collide in the planning process, which is the primary premise of ensuring the driving safety, and defining R after normalization _sub ^obs The method comprises the following steps:

(4) Azimuth prize R _sub ^head : the target of the unmanned vehicle reaches the destination, so that the more the unmanned vehicle is considered to face the destination in navigation, the better the heading angle of the unmanned vehicle is; r is (r) ^head The method is defined as that forward rewards are obtained when the heading of the unmanned vehicle is very close to the optimal azimuth angle, and R is defined after normalization _sub ^head The method comprises the following steps:

rewards R at t-th step of unmanned vehicle _t Is a compound of the formula (I),awarding the adjustment factor for the sub-target;

the network architecture under the DWA-PPO reinforcement learning model at least comprises a value network and a strategy network architecture; according to the value network loss function:

the learning targets of the value network are:

learning objectives for value networks include a portion of predictions themselvesTo prevent bootstrap phenomenon of value network, w is used ^- Constructing a target value network q ^T (s,a；w ^- ) The parameter structure of the target value network is consistent with the value network but different in specific value, and is used for calculating TD error:

the initial parameters of the target value network are consistent with the value network, mu is a parameter, the sum of coefficients is ensured to be 1, and the following updating reference is carried out:

w ^- _new ←μw _new +(1-μ)w ^- _now

the network architecture under the DWA-PPO reinforcement learning model includes three major parts: policy network pi (a|s; θ), value network q (s, a; w) and target value network q ^T (s,a；w)；

The DWA-PPO reinforcement learning model is built in a comprehensive way and comprises an agent, an environment, a commentator module and an actor module; the commentator module comprises a value network error function L (w), a value network q (s, a; w) and a target value network q ^T (s,a；w ^- ) The method comprises the steps of carrying out a first treatment on the surface of the Actor modules include a target network pi (a|s; θ ₁ ) A behavioral network pi (a|s; θ ₂ ) Policy network objective functionThe training beginning stage is collection of training sets: the 0 th round of initial time unmanned vehicle observes state s from the environment using the sensing and positioning system ₀ Behavior network pi (a|s; θ ₂ ) Receiving s ₀ Post-outputting a message about action A ₀ Is of Gaussian distribution pi (A) ₀ |s ₀ ；θ ₁ ) Then randomly extracting the determination action a from the probability distribution ₀ Transmitting to an intelligent vehicle to obtain an evaluation function G of the DWA algorithm at the initial moment ₀ (v, w) completing evaluation of a simulated track set of the DWA algorithm at the initial moment, and transmitting a speed angular speed instruction of the optimal track to an unmanned vehicle motion control module to drive the unmanned vehicle to move; to this end, information including the position, the angle of orientation, the surrounding obstacle distribution of the drone is changed, the environment is switched to state s ₁ The rewarding function also feeds back the rewarding r to the commentator module according to the changed information ₀ The method comprises the steps of carrying out a first treatment on the surface of the When s is ₁ Not in the termination state s _n And (3) entering the next time in the round, otherwise resetting the map and the unmanned vehicle state, and collecting the track of the next round until the round i is collected, so as to finally obtain a training set:

χ＝[χ ₀ ,χ ₁ ,…,χ _i ]

χ ₀ ＝[s ₀ ⁰ ,a ₀ ⁰ ,r ₀ ⁰ ,…,s _n-1 ⁰ ,a _n-1 ⁰ ,r _n-1 ⁰ ,s _n ⁰ ]。

6. the unmanned vehicle self-adaptive path planning method based on the dynamic window method and the near-end strategy according to claim 5, wherein the fifth implementation is as follows:

after obtaining the training set, the value network q (s, a; w) is updated by back propagation of an error function L (w); error function using near-end strategy optimization algorithmBack propagation update pi (a|s; θ) ₁ ) The method comprises the steps of carrying out a first treatment on the surface of the Let q (s, a; w), q ^T (s,a；w ^- )、π(a|s；θ ₁ ) The current network parameters are w respectively _now 、w ^- _now 、θ _now The following steps are repeated for Z times to finish the generation of updating:

(1) Randomly extracting the minimum lot size M from the shuffled training set χ _I Individual states s _N ^I ；

(2) With q ^T (s,a；w ^- ) Calculating the state s _N ^I K-step TD error MTD as starting point _N ^I ：

(3) Calculating the state s using the value network q (s, a; w) _N ^I Action value estimation at the time:

q _N ^I ＝Q(s _N ^I ,a _N ^I ；w _now )

(4) Calculation L (w):

(5) Calculation of

(6) Update value network, policy network, target value network:

w ^- _new ←(1-μ)w _new +μw ^- _now

assume that the pre-update parameter is θ _now Obtaining parameter theta after importance sampling update _new Let the pre-update parameter be w _now Obtaining a parameter w after strategy learning and updating _new Let us assume the parameters w introduced before the update to prevent the bootstrap phenomenon of the value network ^- _now After mu is taken as a parameter, the w is obtained after the update of the coefficient sum of 1 ^- _new The method comprises the steps of carrying out a first treatment on the surface of the After Z times of updating, the target network pi (a|s; theta ₁ ) Parameters are assigned to the target network pi (a|s; θ ₁ ) Recording as one generation of update, then emptying the training set, and re-entering the next generation of update until the model converges;

the average score and arrival rate change curve of each generation of the unmanned vehicle in the deep reinforcement learning environment gradually learns a parameter network capable of guiding the unmanned vehicle path planning correctly along with the iterative convergence of the model; and completing the construction of the self-adaptive PPO-ADWA algorithm.