CN118034355A - Network training method, unmanned aerial vehicle obstacle avoidance method and device - Google Patents

Abstract

The present invention provides a network training method, a method and device for avoiding obstacles of unmanned aerial vehicles, and relates to the technical field of unmanned aerial vehicle control. The method comprises: updating an experience replay pool with sample data of the target moment constructed by the target moment environmental situation of the sample unmanned aerial vehicle, the next moment environmental situation, the optimal heading angle at the target moment, and the target moment reward value; when the number of sample data in the updated experience replay pool reaches a preset number, extracting a plurality of sample data to be processed from the pool for multi-step prediction to obtain the optimal heading angle prediction value, environmental situation prediction value, and reward prediction value at a plurality of future moments; training a target strategy network according to the environmental situation, the optimal heading angle, the reward value, the environmental situation prediction value, the reward prediction value, and the optimal heading angle prediction value in each sample data to be processed, and obtaining an optimized strategy network. The present invention effectively improves the learning efficiency and sample utilization rate in obstacle avoidance of unmanned aerial vehicles.

Description

Network training method, unmanned aerial vehicle obstacle avoidance method and device

Technical Field

The present invention relates to the technical field of unmanned aerial vehicle control, and in particular to a network training method, a unmanned aerial vehicle obstacle avoidance method and a device.

Background Art

The obstacle avoidance problem of drones can be described as the task of a drone navigating in a space with obstacles. The task usually follows some optimization criteria, such as minimum work cost, shortest flight distance, shortest flight time, etc. Common traditional obstacle avoidance methods include: dynamic programming algorithm, artificial potential field method, sampling-based method and graph theory-based method, but these methods require different models to be established according to different situations. However, in the actual drone flight environment, the working environment is complex and unpredictable, and drones are often required to detect in unknown environments and make real-time decisions.

With the advancement of artificial intelligence technology, reinforcement learning has been increasingly used in games, robots, the Internet and other fields, and has attracted widespread attention. Model-free reinforcement learning is a commonly used method to solve decision-making in unknown environments and has been widely used in the obstacle avoidance problem of drones. However, due to the limited interaction between drones and the environment, the sample utilization rate of model-free reinforcement learning is low and the autonomous learning efficiency is low, which in turn leads to poor obstacle avoidance performance of drones.

Summary of the invention

The present invention provides a network training method, a UAV obstacle avoidance method and a device, which are used to solve the defects in the prior art that the interaction between the UAV and the environment is limited, resulting in low sample utilization rate and low autonomous learning efficiency of model-free reinforcement learning, and further resulting in poor obstacle avoidance performance of the UAV, so as to improve the autonomous learning efficiency and sample utilization rate in the obstacle avoidance of the UAV, so as to improve the obstacle avoidance performance of the UAV.

The present invention provides a network training method, comprising:

According to the environmental situation of the sample drone at the target moment, the optimal heading angle at the target moment, the environmental situation at the next moment, and the reward value at the target moment, the sample data at the target moment is constructed;

The sample data at the target moment is updated to the experience replay pool, and when the number of sample data in the updated experience replay pool reaches a preset number, the sample data to be processed at multiple different moments within the target prediction interval are extracted from the updated experience replay pool;

Inputting the environmental situation in each of the sample data to be processed into the target strategy network to obtain the optimal heading angle prediction values at multiple different future moments, and inputting the environmental situation in each of the sample data to be processed and the optimal heading angle prediction values at multiple different future moments into the target prediction network for multi-step prediction to obtain the environmental situation prediction values and reward prediction values at multiple different future moments;

According to the environmental situation, the optimal heading angle and the reward value in each of the sample data to be processed, as well as the environmental situation prediction value, the reward prediction value and the optimal heading angle prediction value, the target policy network is subjected to reinforcement learning training, and an optimized policy network is obtained according to the training results;

The optimized strategy network is used to predict the optimal heading angle of the current drone at the current moment based on the current environmental situation of the current drone at the current moment, so that the current drone can perform an obstacle avoidance task according to the optimal heading angle at the current moment.

According to a network training method provided by the present invention, the environmental situation at the target moment and the optimal heading angle at the target moment are obtained based on the following steps:

Determine the target time environmental situation according to the target time position, radius and destination position of the sample UAV, the target time position, target time speed and radius of the obstacle, and the target time distance between the sample UAV and the obstacle;

The environmental situation at the target moment is input into the target strategy network to obtain the optimal heading angle at the target moment.

According to a network training method provided by the present invention, the next moment environment situation is obtained based on the following steps:

The next moment position of the sample UAV is calculated based on the UAV dynamic constraint model, kinematic constraints and disturbance flow field method;

The environmental situation at the next moment is determined according to the next moment position, radius and destination position of the sample drone, the next moment position, next moment speed and radius of the obstacle, and the next moment distance between the sample drone and the obstacle.

According to a network training method provided by the present invention, the target moment reward value is obtained based on the following steps:

When the target time distance between the sample drone and the obstacle is less than the first distance value, determining the target time reward value according to the target time distance between the sample drone and the obstacle, the radius of the sample drone, the radius of the obstacle, and the first reward value;

When the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, and the target time distance between the sample drone and the destination position is less than the second distance value, the target time reward value is determined according to the target time distance between the sample drone and the destination position, the distance between the starting position of the sample drone and the destination position, and the second reward value and the third reward value;

When the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, and the target time distance between the sample drone and the destination position is greater than or equal to the second distance value, the target time reward value is determined according to the target time distance between the sample drone and the destination position, the distance between the starting position of the sample drone and the destination position, and the third reward value;

Among them, the first reward value is a constant reward value; the second reward value is a threat reward for limiting the sample drone to stay away from the obstacle; and the third reward value is an additional reward value corresponding to task completion.

According to a network training method provided by the present invention, the second reward value is determined based on the following steps:

When the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value and less than the third distance value, the second reward value is determined based on the target time distance between the sample drone and the obstacle, the radius of the sample drone, the radius of the obstacle, the preset threat radius and the fourth reward value;

When the target time distance between the sample UAV and the obstacle is less than the first distance value, or greater than or equal to the third distance value, the second reward value is determined based on a preset constant value.

According to a network training method provided by the present invention, the environmental situation in each of the sample data to be processed and the optimal heading angle prediction values at multiple different future moments are input into the target prediction network for multi-step prediction to obtain environmental situation prediction values and reward prediction values at multiple different future moments, including:

The environmental situation in each of the sample data to be processed and the predicted value of the optimal heading angle are input into the reward function network of the target prediction network, and the environmental situation in each of the sample data to be processed and the predicted value of the optimal heading angle are input into the situation transfer function network of the target prediction network, and multi-step prediction is performed to obtain the reward prediction value output by the reward function network and the environmental situation prediction value output by the situation transfer function network.

According to a network training method provided by the present invention, the target strategy network is subjected to reinforcement learning training according to the environmental situation, the optimal heading angle and the reward value in each of the sample data to be processed, as well as the environmental situation prediction value, the reward prediction value and the optimal heading angle prediction value, including:

Obtaining a value function cost function according to the optimal heading angle in each of the sample data to be processed, as well as the optimal heading angle prediction value, the environmental situation prediction value and the reward prediction value;

Acquire a strategy cost function according to the optimal heading angle prediction value and the environmental situation prediction value;

Obtaining a situation transfer cost function according to the environmental situation in each of the sample data to be processed and the environmental situation prediction value;

Obtaining a reward cost function according to the reward value in each of the sample data to be processed and the reward prediction value;

Reinforcement learning is performed on the target policy network according to the value function cost function, the strategy cost function, the situation transfer cost function and the reward cost function.

The present invention also provides a method for avoiding obstacles in a drone, comprising:

Get the current environmental situation of the current drone;

Inputting the current environmental situation into the optimized strategy network to obtain the optimal heading angle of the current drone at the current moment;

According to the optimal heading angle at the current moment, controlling the current drone to perform an obstacle avoidance task;

Wherein, the optimized policy network is trained based on any of the network training methods described above.

The present invention also provides a network training device, comprising:

A construction unit, used to construct sample data at the target moment according to the target moment environmental situation of the sample UAV, the optimal heading angle at the target moment, the next moment environmental situation, and the target moment reward value;

An extraction unit, used to update the sample data at the target moment to the experience replay pool, and when the number of sample data in the updated experience replay pool reaches a preset number, extract the sample data to be processed at multiple different moments within the target prediction interval from the updated experience replay pool;

A prediction unit, used for inputting the environmental situation in each of the sample data to be processed into a target strategy network to obtain optimal heading angle prediction values at multiple different future moments, and inputting the environmental situation in each of the sample data to be processed and the optimal heading angle prediction values at multiple different future moments into a target prediction network for multi-step prediction to obtain environmental situation prediction values and reward prediction values at multiple different future moments;

An optimization unit, configured to perform reinforcement learning training on the target policy network according to the environmental situation, the optimal heading angle and the reward value in each of the sample data to be processed, as well as the environmental situation prediction value, the reward prediction value and the optimal heading angle prediction value, and obtain an optimized policy network according to the training results;

The optimized strategy network is used to predict the optimal heading angle of the current drone at the current moment based on the current environmental situation of the current drone at the current moment, so that the current drone can perform an obstacle avoidance task according to the optimal heading angle at the current moment.

The present invention also provides a drone obstacle avoidance device, comprising:

The acquisition unit is used to obtain the current environmental situation of the current drone;

A decision-making unit, used for inputting the current environmental situation into the optimized strategy network to obtain the optimal heading angle of the current UAV at the current moment;

An obstacle avoidance control unit, used to control the current UAV to perform an obstacle avoidance task according to the optimal heading angle at the current moment;

Wherein, the optimized policy network is trained based on any of the network training methods described above.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the network training method described in any one of the above, or the drone obstacle avoidance method described in any one of the above, is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the network training method described in any one of the above or the drone obstacle avoidance method described in any one of the above is implemented.

The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the network training methods described above, or any of the drone obstacle avoidance methods described above.

The network training method, unmanned aerial vehicle obstacle avoidance method and device provided by the present invention extract a plurality of to-be-processed sample data at different moments from an experience replay pool, and predict each to-be-processed sample data based on a target policy network and a target prediction network, so as to obtain the optimal heading angle prediction values, environmental situation prediction values and reward prediction values at a plurality of different future moments, and perform rolling optimization on the target policy network based on the environmental situation, optimal heading angle and reward value in each to-be-processed sample data, and the environmental situation prediction value, reward prediction value and optimal heading angle prediction value at the future moment, so as to realize not only the expansion of the number of sample data by generating virtual environmental data of the target prediction network, so as to reduce the number of interactions with the real environment, improve the sample utilization rate, accelerate the training speed, and improve the learning efficiency, so as to enable the target policy network to converge to the optimum quickly when the number of interactions with the environment of the unmanned aerial vehicle is less; and also enable the optimized decision model to have both current decision experience and future decision experience, so as to make a more optimized obstacle avoidance decision for the unmanned aerial vehicle, thereby improving the obstacle avoidance performance of the unmanned aerial vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a network training method provided by the present invention;

FIG2 is a second flow chart of the network training method provided by the present invention;

FIG3 is a third flow chart of the network training method provided by the present invention;

FIG4 is a schematic diagram of a flow chart of a method for avoiding obstacles in a drone provided by the present invention;

FIG5 is a schematic diagram of the structure of a network training device provided by the present invention;

FIG6 is a schematic structural diagram of a drone obstacle avoidance device provided by the present invention;

FIG. 7 is a schematic diagram of the structure of an electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The obstacle avoidance problem of drones can be described as the task of a drone navigating in a space with obstacles. The task usually follows some optimization criteria, such as minimum work cost, shortest flight distance, shortest flight time, etc. Common traditional obstacle avoidance methods include: dynamic programming algorithm, artificial potential field method, sampling-based method and graph theory-based method, but these methods require different models to be established according to different situations. However, in the actual drone flight environment, the working environment is complex and unpredictable, and drones are often required to have self-learning, adaptive and robust capabilities to detect and make real-time decisions in unknown environments.

In order to overcome the weaknesses of these methods, relevant personnel have explored various solutions, such as reinforcement learning, which can learn appropriate behaviors from environmental states. The advantage is that it has the ability to learn online and can generate corresponding rewards or penalties in different environments. It is a commonly used method to solve unknown environmental decision-making and has been widely used in the obstacle avoidance problem of drones. The end-to-end motion planning method of reinforcement learning allows the system to be considered as a whole, making it more robust. However, due to the limited number of interactions required between drones and the environment, that is, the limited interaction between drones and the environment, the sample utilization rate of model-free reinforcement learning is low and the autonomous learning efficiency is low. The application of model-free reinforcement learning in the real world is also limited, which leads to poor obstacle avoidance performance of drones.

In response to the problems of low training efficiency and low sample utilization in model-free reinforcement learning in drone obstacle avoidance, an embodiment of the present application provides a network training method. This method obtains a target prediction network prediction simulated environmental situation and reward model through a data-driven method, and uses a rolling optimization interval to perform optimization training of the policy network, so that the drone can avoid obstacles with higher learning efficiency, faster convergence speed of the strategy to the optimal value, and less sample capacity space required for the experience replay buffer, so that the drone can learn the optimal strategy through only a small amount of real environment interaction, which promotes the application of reinforcement learning methods in obstacle avoidance problems and improves the obstacle avoidance performance of drones.

FIG. 1 is a flow chart of a network training method provided by the present invention. As shown in FIG. 1 , the method includes:

Step 110, constructing sample data at the target moment according to the target moment environmental situation of the sample drone, the optimal heading angle at the target moment, the next moment environmental situation, and the target moment reward value;

Here, the target moment may be the current moment or any historical moment, and this implementation does not specifically limit this; the next moment is the moment after the target moment.

The target moment environmental situation is the environmental situation at the target moment, which can be determined based on the target moment operating parameters of the sample drone and the target moment operating parameters of the obstacles in the environment where the sample drone is located. The target moment optimal heading angle is the optimal heading angle at the target moment. The optimal heading angle here can be obtained by inputting the target moment environmental situation into the target strategy network for decision output. The next moment environmental situation is the environmental situation at the next moment of the target moment, which can be determined based on the next moment operating parameters of the sample drone and the next moment operating parameters of the obstacles in the environment where the sample drone is located. The target moment reward value is the reward value formed by performing the obstacle avoidance operation based on the target moment optimal heading angle.

In some implementations, the target moment environmental situation and the target moment optimal heading angle are obtained based on the following steps:

The target moment environmental situation is determined according to the target moment position, radius and destination position of the sample UAV, the target moment position, target moment speed and radius of the obstacle, and the target moment distance between the sample UAV and the obstacle; the target moment environmental situation is input into the target strategy network to obtain the optimal heading angle at the target moment.

Here, the obstacle may be a cylinder or a sphere, etc., which is not specifically limited in this embodiment.

Optionally, the step of obtaining the environmental situation at the target moment specifically includes:

Based on the difference between the target moment position of the obstacle and the target moment position of the sample UAV, the sum of the radius of the sample UAV and the radius of the obstacle, and the target moment distance between the sample UAV and the obstacle, the first parameter value of the target moment environmental situation is determined; based on the difference between the destination position and the target moment position of the obstacle, the second parameter value of the target moment environmental situation is determined; based on the target moment speed of the obstacle, the third parameter value of the target moment environmental situation is determined, and the target moment environmental situation is constructed based on the first parameter value, the second parameter value and the third parameter value. The specific calculation formula is as follows:

;

in, for The current environmental situation; and They are sample drones, obstacles, Time location, is the destination location; is the distance between the sample drone and the obstacle. Time distance; and are the radii of the sample drone and obstacles, respectively; for obstacles Time speed.

Optionally, the step of obtaining the optimal heading angle at the target moment specifically includes:

The environmental situation at the target moment is input into the target policy network, so that the target policy network can flexibly make corresponding decisions according to the environmental situation at the target moment, thereby outputting the optimal heading angle at the target moment, so that the target policy network based on the subsequent optimization training can flexibly adjust the heading angle of the UAV according to the environmental situation, so as to avoid obstacles and reach the destination in time to the greatest extent, and can reduce reliance on pre-established maps or static path planning, thereby adapting to obstacle avoidance scenarios in various complex environments and improving the generalization and robustness of UAV obstacle avoidance.

Here, the target strategy network can be trained based on the environmental situation, optimal heading angle, and environmental situation prediction value and optimal heading angle prediction value at the corresponding time in the to-be-processed sample data at multiple different times extracted in the previous prediction interval of the target prediction interval.

Here, the optimal heading angle includes but is not limited to the optimal roll angle, the optimal pitch angle, and the optimal yaw angle. The specific calculation formula is as follows:

;

in, For sample drones Optimal heading angle at the moment; They represent the optimal roll angle, optimal pitch angle, and optimal yaw angle of the sample UAV respectively; is the target policy network to be trained. In the DDPG (Deep Deterministic Policy Gradient, deep reinforcement learning) algorithm, the main target policy network in the target policy network It is the executor (also called Actor), the main value function network in the value function network It is the evaluator (also called Critic). What the Actor needs to do is to interact with the environment and learn a better strategy using policy gradient under the guidance of the Critic value function. What the Critic needs to do is to learn a value function through the data collected by the Actor's interaction with the environment. This value function will be used to determine what actions are good and what actions are not good in the current state, and then help the Actor to update its strategy. For the main value function network By inputting the environment situation and optimal heading angle at the target time, the network will output a value function, which represents the size of the future cumulative reward. By inputting the environmental situation at the target time, the optimal heading angle can be calculated. In addition, in DDPG, the target policy network can also be attached with a policy network And the value function network can also contain additional value function network , which is used to help the reinforcement learning training process become more stable.

Optionally, after obtaining the target moment environmental situation and the optimal heading angle of the sample drone based on the above embodiments, environmental interaction can be performed based on the target moment environmental situation and the optimal heading angle to obtain the next moment environmental situation and the target moment reward value, which can be specifically expressed as:

;

in, for The current environmental situation; for Moment reward value; Representing the environmental interaction function, the next moment position of the sample UAV can be obtained according to the UAV dynamic constraint model, kinematic constraints and perturbation flow field method, and the next moment position of the obstacle and the destination position can be combined to obtain the next moment environmental situation and the current moment reward.

In some embodiments, the next moment environmental situation is obtained based on the following steps:

According to the UAV dynamic constraint model, kinematic constraints and disturbance flow field method, the next moment position of the sample UAV is calculated; according to the next moment position, radius and destination position of the sample UAV, the next moment position, next moment speed and radius of the obstacle, and the next moment distance between the sample UAV and the obstacle, the next moment environmental situation is determined.

Optionally, the step of obtaining the environmental situation at the next moment specifically includes:

First, based on the UAV dynamic constraint model, kinematic constraints and perturbation flow field method, the next moment position of the sample UAV is calculated according to the target moment position, target moment speed and destination position of the sample UAV, the target moment position and target moment speed of the obstacle. This calculation method can achieve accurate motion planning of the UAV by considering the dynamic constraints and kinematic constraints of the UAV in the calculation process and using the perturbation flow field method to correct environmental factors, making the calculation of the next moment position of the UAV more accurate and reliable.

Here, the UAV dynamic constraint model includes but is not limited to constraints such as the UAV's yaw angle, climb angle, trajectory segment length, flight altitude and total trajectory length; the kinematic constraints include the restriction conditions for the UAV's movement; the perturbation flow field method can be based on the next moment position, next moment speed and radius of the obstacle, and the next moment distance between the sample UAV and the obstacle for operation trajectory planning, and based on the UAV dynamic constraint model and the kinematic constraint constraints, the operation trajectory planning is corrected to obtain the next moment position of the sample UAV.

Next, according to the difference between the next moment position of the obstacle and the next moment position of the sample UAV, the sum of the radius of the sample UAV and the radius of the obstacle, and the next moment distance between the sample UAV and the obstacle, determine the first parameter value of the environmental situation at the next moment; based on the difference between the destination position and the next moment position of the obstacle, determine the second parameter value of the environmental situation at the next moment; based on the next moment speed of the obstacle, determine the third parameter value of the environmental situation at the next moment, and construct the environmental situation at the next moment based on the first parameter value, the second parameter value and the third parameter value. For details, please refer to the steps for obtaining the environmental situation at the target moment, which will not be repeated here.

In some embodiments, the target moment reward value is obtained based on the following steps:

In the case where the target time distance between the sample drone and the obstacle is less than the first distance value, the target time reward value is determined according to the target time distance between the sample drone and the obstacle, the radius of the sample drone, the radius of the obstacle, and the first reward value; in the case where the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, and the target time distance between the sample drone and the destination position is less than the second distance value, the target time reward value is determined according to the target time distance between the sample drone and the destination position, the distance between the starting position of the sample drone and the destination position, and the second reward value and the third reward value; in the case where the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, and the target time distance between the sample drone and the destination position is greater than or equal to the second distance value, the target time reward value is determined according to the target time distance between the sample drone and the destination position, the distance between the starting position of the sample drone and the destination position, and the third reward value; wherein the first reward value is a constant reward value; the second reward value is a threat reward for limiting the sample drone to stay away from the obstacle; and the third reward value is an additional reward value corresponding to task completion.

Here, the first distance value is determined according to the sum of the radius of the sample UAV and the radius of the obstacle; the second distance is determined according to the preset distance value.

Optionally, the specific steps of obtaining the target moment reward value include:

The target time distance between the sample drone and the obstacle is compared with the first distance value. If the target time distance between the sample drone and the obstacle is less than the first distance value, the target time reward value is calculated based on the target time distance between the sample drone and the obstacle, the sum of the radius of the sample drone and the radius of the obstacle, and the first reward value; if the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, the target time distance between the sample drone and the destination position is further compared with the second distance value. If on this basis, the target time distance between the sample drone and the destination position is less than the second distance value, the target time reward value is calculated according to the ratio between the target time distance between the sample drone and the destination position and the distance between the starting position of the sample drone and the destination position, as well as the second reward value and the third reward value; if on the basis that the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, the target time reward value is calculated according to the ratio between the target time distance between the sample drone and the destination position and the distance between the starting position of the sample drone and the destination position, as well as the third reward value.

in, The specific calculation formula of the moment reward value is as follows:

;

in, is the distance between the sample drone and the obstacle. Time distance; and are the radii of the sample drone and obstacles, respectively; , and are the first reward value, the second reward value and the third reward value respectively; is a second distance value, which is preset; is the distance between the sample UAV and the destination location Time distance; is the distance between the starting position and the destination position of the sample UAV.

Here, the first reward value is a pre-set constant reward value; the third reward value is the corresponding additional reward value set for task completion; and the second reward value is a threat reward set to restrict the sample drone from staying away from obstacles.

The method provided in this embodiment dynamically adjusts the target time reward value through the target time distance between the drone and the obstacle and the target time distance between the drone and the destination, so that when the drone approaches the obstacle, it will be restricted by the first reward value to avoid collision with the obstacle as much as possible. When it is far away from the obstacle and close to the destination, it will be encouraged by the second reward value and the third reward value to guide the drone to maintain a safe distance as much as possible while encouraging the drone to reach the destination as soon as possible to complete the task; when it is far away from the destination, it will be encouraged by the third reward value to encourage the drone to reach the destination as soon as possible to complete the task, thereby improving the dynamic adaptability of the drone's flight obstacle avoidance, obstacle avoidance ability and task completion ability, thereby improving the drone's obstacle avoidance performance.

In some embodiments, the second reward value is determined based on the following steps:

When the target moment distance between the sample UAV and the obstacle is greater than or equal to the first distance value and less than the third distance value, the second reward value is determined based on the target moment distance between the sample UAV and the obstacle, the radius of the sample UAV, the radius of the obstacle, the preset threat radius and the fourth reward value; when the target moment distance between the sample UAV and the obstacle is less than the first distance value, or greater than or equal to the third distance value, the second reward value is determined based on a preset constant value.

Here, the third distance value is determined based on the sum of the radius of the sample UAV, the radius of the obstacle and the preset threat radius.

Optionally, the specific steps of obtaining the second reward value include:

The target time distance between the sample drone and the obstacle is compared with the first distance value. When the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value, the target time distance between the sample drone and the obstacle is further compared with the third distance value. If it is determined that the target time distance between the sample drone and the obstacle is greater than or equal to the first distance value and less than the third distance value, the second reward value is calculated based on the sum of the radius of the sample drone, the radius of the obstacle and the preset threat radius, the target time distance between the sample drone and the obstacle, and the preset threat radius. If it is determined that the target time distance between the sample drone and the obstacle is less than the first distance value, or greater than or equal to the third distance value, the second reward value is determined based on the preset constant value. The preset constant value here can be 0 or a constant value infinitely close to 0.

Among them, the second reward value The calculation formula is:

;

in, is the preset threat radius; is the fourth reward value; here, the fourth reward value may also be a preset constant reward value.

The method provided in this embodiment dynamically adjusts the second reward value according to the target moment distance between the drone and the obstacle to more accurately reflect the relationship between the drone and the obstacle, provide a more effective reward signal, improve the drone's obstacle avoidance capability and flight safety, and thus help improve the drone's obstacle avoidance performance and ability to cope with diverse scenarios.

Optionally, after obtaining the target moment environmental situation, next moment environmental situation, target moment optimal heading angle and target moment reward value of the sample drone based on the above embodiments, the target moment environmental situation, next moment environmental situation, target moment optimal heading angle and target moment reward value can be constructed into a 4-tuple to form sample data at the target moment.

Step 120, updating the sample data at the target moment to the experience replay pool, and when the number of sample data in the updated experience replay pool reaches a preset number, extracting to-be-processed sample data at multiple different moments within the target prediction interval from the updated experience replay pool;

Here, the experience replay pool is a buffer for storing historical sample data, which is used to randomly extract samples for training during the training process.

Optionally, after obtaining the sample data at the target time, the sample data at the target time may be updated to the experience replay pool; the specific expression formula is as follows:

;

in, It is the experience replay pool; for Sample data at the moment; and They are The current environmental situation and The current environmental situation; and They are The optimal heading angle at the moment Moment reward value.

Next, the number of sample data in the updated experience replay pool is compared with the preset number. If the number of sample data in the updated experience replay pool does not reach the preset number, the sample data of the next moment is collected and updated to the experience replay pool until the number of sample data in the updated experience replay pool reaches the preset number. If the number of sample data in the updated experience replay pool reaches the preset number, the sample data to be processed at multiple different moments in the target prediction interval are sampled from it, so that the multi-step prediction method based on model predictive control can be used to predict the future environmental situation of each sample data, the optimal heading angle of the drone, and the reward. The rolling optimization method is used to maximize the cumulative return of each prediction interval, thereby improving the learning efficiency and sample utilization of the drone in reinforcement learning, and thus improving the obstacle avoidance performance of the drone.

The formula for extracting the sample data to be processed at multiple different times from the updated experience replay pool is:

;

in, The total number of sample data that needs to be extracted.

Step 130, inputting the environmental situation in each of the sample data to be processed into the target strategy network to obtain the optimal heading angle prediction values at multiple different future moments, and inputting the environmental situation in each of the sample data to be processed and the optimal heading angle prediction values at multiple different future moments into the target prediction network for multi-step prediction to obtain the environmental situation prediction values and reward prediction values at multiple different future moments;

Optionally, after extracting a plurality of sample data to be processed at different times, each sample data to be processed may be subjected to Step prediction operation is performed to obtain the environmental situation prediction values and reward prediction values at multiple different future moments corresponding to each sample data to be processed. is the pre-set prediction step length, N is less than .

Here, for each sample data to be processed, iterative execution The specific steps of the step prediction operation are as follows:

First, based on the target strategy network and the environmental situation at different times of the sample data to be processed at different times, the prediction steps are calculated. The optimal heading angle at the corresponding moment (that is, multiple different future moments) under different prediction steps is input into the target strategy network to obtain the optimal heading angle prediction value at the corresponding moment under different prediction steps. The specific calculation formula is as follows:

;

in, For the Sample data in the prediction step The optimal heading angle prediction value at the corresponding moment; As the main strategy network; For the Sample data in the prediction step The environmental situation at the corresponding moment.

Next, it interacts with the trained target prediction network to obtain the environmental situation at the next moment corresponding to different prediction steps and the reward value at the corresponding moment of different prediction steps; specifically, the environmental situation at the corresponding moment under different prediction steps and the optimal heading angle prediction value are input into the target prediction network, and the target prediction network performs multi-step prediction to predict the environmental situation prediction value at the next moment corresponding to multiple different prediction steps and the reward prediction value at the corresponding moment of multiple different prediction steps.

The target prediction network here is a model that can perform environmental situation prediction and reward value prediction; it uses the environmental situation values and optimal heading angles of sample drones at each moment in the previous prediction interval of the target prediction interval as samples, the environmental situation values of the next moment and the reward values at each moment as labels, and/or the environmental situation prediction values and optimal heading angle prediction values of sample drones at each moment in the previous prediction interval, as well as the environmental situation prediction values and reward prediction values at the next moment to iteratively train the target prediction network, so that the trained target prediction network has a good approximation effect in a low-dimensional environment.

The method provided in this embodiment uses sample data extracted from the experience replay pool to update the target prediction network and the subsequent policy model update, and the target prediction network promotes the subsequent update of the policy model, so that the drone has higher learning efficiency, faster optimal policy convergence speed and less sample capacity space required for the experience replay buffer in obstacle avoidance problems.

Step 140, according to the environmental situation, optimal heading angle and reward value in each of the sample data to be processed, as well as the environmental situation prediction value, the reward prediction value and the optimal heading angle prediction value, the target policy network is subjected to reinforcement learning training, and an optimized policy network is obtained according to the training results; wherein the optimized policy network is used to predict the optimal heading angle of the current drone at the current moment based on the current environmental situation of the current drone at the current moment, so that the current drone can perform an obstacle avoidance task according to the optimal heading angle at the current moment.

Optionally, after obtaining the environmental situation prediction values, reward prediction values and optimal heading angle prediction values at multiple different future moments corresponding to each sample data to be processed, at least one cost function can be obtained based on the environmental situation prediction values, reward prediction values and optimal heading angle prediction values at multiple different future moments, as well as the environmental situation, reward value and optimal heading angle contained in the sample data to be processed, so as to perform gradient update on the network parameters of the target policy network based on the cost function, thereby achieving continuous updating and optimization of the target policy network by interacting with the environment and utilizing prediction information, so as to obtain an optimized policy network that can accurately decide the optimal heading angle of the drone according to the environmental situation of the drone.

Subsequently, after obtaining the optimized strategy network, the current environmental situation of the current UAV can be input into the optimized strategy network, so that the optimized strategy network can make decisions based on the current environmental situation to predict the current optimal heading angle of the current UAV, thereby facilitating guiding the current UAV to perform obstacle avoidance tasks according to the current optimal heading angle, so that it can obtain the maximum reward in the current environment and achieve effective obstacle avoidance behavior.

Compared to some related technologies that generate additional virtual data through model-based reinforcement learning, which requires long-term prediction and makes learning expensive, the method provided in this embodiment optimizes the trajectory within a limited short range within each prediction interval through model predictive control, provides the current local optimal solution, avoids the high cost of long-term planning, and achieves a balance between input cost and convergence benefit. In addition, compared to some related technologies that achieve environmental situation prediction and reward prediction through probabilistic modeling, the method provided in this embodiment uses deterministic modeling, that is, a data-driven method to model the state transition and reward model of the environmental model, that is, to approximate the environmental model through a neural network, which is conducive to reducing the number of neural networks, improving the calculation speed of the algorithm, and thus improving the learning efficiency of the target policy network.

The network training method provided in this embodiment extracts sample data to be processed at multiple different moments from the experience replay pool, and predicts each sample data to be processed based on the target policy network and the target prediction network to obtain the optimal heading angle prediction values, environmental situation prediction values and reward prediction values at multiple different future moments, so as to perform rolling optimization on the target policy network based on the environmental situation, optimal heading angle and reward value in each sample data to be processed, as well as the environmental situation prediction value, reward prediction value and optimal heading angle prediction value at the future moment. This can not only realize the expansion of the number of sample data by generating virtual environment data through the target prediction network to reduce the number of interactions with the real environment, improve sample utilization, speed up training, and improve learning efficiency, so that the target policy network can quickly converge to the optimum when the number of interactions between the drone and the environment is less; it can also enable the optimized decision model to have both current decision-making experience and future decision-making experience, so as to make more optimized drone obstacle avoidance decisions, thereby improving the drone obstacle avoidance performance.

In some embodiments, in step 130, the environmental situation in each of the sample data to be processed and the optimal heading angle prediction values at multiple different future moments are input into the target prediction network for multi-step prediction to obtain environmental situation prediction values and reward prediction values at multiple different future moments, including:

The environmental situation in each of the sample data to be processed and the predicted value of the optimal heading angle are input into the reward function network of the target prediction network, and the environmental situation in each of the sample data to be processed and the predicted value of the optimal heading angle are input into the situation transfer function network of the target prediction network, and multi-step prediction is performed to obtain the reward prediction value output by the reward function network and the environmental situation prediction value output by the situation transfer function network.

Here, the target prediction network may include a situation transfer function network capable of predicting environmental situations and a reward function network capable of predicting reward values.

Optionally, the multi-step prediction acquisition steps of the environmental situation prediction value and the reward prediction value specifically include:

Input the environmental situation in each sample data to be processed and the optimal heading angle prediction value of the environmental situation at the corresponding time into the reward function network to obtain the reward prediction value at the corresponding time; input the environmental situation in each sample data to be processed and the optimal heading angle prediction value of the environmental situation at the corresponding time into the situation transfer function network to obtain the environmental situation prediction value at the next moment of the corresponding time;

The predicted value of the environmental situation at the next moment of the corresponding moment and the predicted value of the optimal heading angle at the next moment of the corresponding moment are input into the reward function network to obtain the predicted value of the reward at the next moment of the corresponding moment. The predicted value of the environmental situation at the next moment of the corresponding moment and the predicted value of the optimal heading angle at the next moment of the corresponding moment are input into the situation transfer function network to obtain the predicted value of the environmental situation at the next future moment. The above prediction steps are iteratively performed for each sample data to be processed until the prediction step length reaches the preset prediction step length. .

Among them, the multi-step prediction acquisition formula of the environmental situation prediction value and the reward prediction value can be expressed as:

;

in, is the situation transfer function network, is the reward function network, and are the model parameters of the situation transfer function network and the reward function network respectively. and They respectively represent the environmental situation prediction value obtained by the situation transfer function network and the reward prediction value obtained by the reward function network.

The method provided in this embodiment performs multi-step prediction on the sample data to be processed through the reward function network and the situation transfer function network of the target prediction network, so as to approximate the environmental model through the method of the deterministic model, that is, to use the data-driven method to learn the environmental situation prediction value and the reward prediction value of the environment, thereby improving the sample utilization rate of the UAV in reinforcement learning and further improving the obstacle avoidance performance of the UAV.

In some embodiments, in step 140, reinforcement learning training is performed on the target policy network according to the environmental situation, optimal heading angle and reward value in each of the sample data to be processed, as well as the environmental situation prediction value, the reward prediction value and the optimal heading angle prediction value, including: obtaining a value function cost function according to the optimal heading angle in each of the sample data to be processed, as well as the optimal heading angle prediction value, the environmental situation prediction value and the reward prediction value; obtaining a strategy cost function according to the optimal heading angle prediction value and the environmental situation prediction value; obtaining a situation transfer cost function according to the environmental situation in each of the sample data to be processed and the environmental situation prediction value; obtaining a reward cost function according to the reward value in each of the sample data to be processed and the reward prediction value; and performing reinforcement learning on the target policy network according to the value function cost function, the strategy cost function, the situation transfer cost function and the reward cost function.

Optionally, the reinforcement learning training steps of the target policy network specifically include:

The optimal heading angle prediction value or optimal heading angle at the corresponding moment of each prediction step, as well as the environmental situation prediction value and the reward prediction value are input into the additional value function network of the target value function network. According to the output of the additional value function network and the reward prediction value at the corresponding moment of each prediction step, the target value at the corresponding moment of each prediction step is obtained. The specific calculation formula is as follows:

;

in, For Sample data in the prediction step The target value at the corresponding moment; is the pre-set prediction step size; and is the coefficient; , and They are respectively the predicted value of reward, the predicted value of environmental situation and the predicted value of optimal heading angle; is the value-added function network.

The optimal heading angle prediction value and environmental situation prediction value at the corresponding time of each prediction step are input into the main value function network of the target value function network, so as to calculate the value function cost function according to the difference between the output of the main value function network and the target value at the corresponding time of each prediction step; wherein, the value function cost function The specific calculation formula is:

;

in, is the number of sample data required to be extracted in each prediction interval; , and are the target value, environmental situation prediction value and optimal heading angle prediction value at the corresponding moment of each prediction step respectively; is the main value function network; is the preset prediction step size.

The optimal heading angle prediction value and environmental situation prediction value at the corresponding moment of each prediction step are input into the principal value function network in the target value function network, and the strategy cost function is calculated based on the output of the principal value function network; wherein, the strategy cost function The specific calculation formula is:

;

According to the difference between the environmental situation at the corresponding moment and the environmental situation prediction value, the situation transfer cost function is calculated; among them, the situation transfer cost function The specific calculation formula is:

;

and Respectively The environmental situation prediction value and environmental situation at the corresponding moment of each sample data.

The reward cost function is calculated based on the difference between the reward value at the corresponding moment and the predicted reward value; where the reward cost function The specific calculation formula is:

;

and Respectively The reward prediction value and reward value at the corresponding moment of the sample data.

Subsequently, after obtaining the above-mentioned cost function, the target value function network, the target policy network and the target prediction network can be optimized based on the value function cost function, and the policy network trained this time is used as the target policy network corresponding to the next prediction interval, the value function network trained this time is used as the target value function network corresponding to the next prediction interval, and the prediction network trained this time is used as the target prediction network corresponding to the next prediction interval. On this basis, based on the to-be-processed sample data at multiple different times in the next prediction interval and the predicted optimal heading angle prediction value, environmental situation prediction value and reward prediction value, the target value function network corresponding to the next prediction interval, the target policy network corresponding to the next prediction interval and the target prediction network corresponding to the next prediction interval are iteratively optimized until the number of iterative training rounds reaches the set training rounds, thereby obtaining the optimized policy network.

The method provided in this embodiment maximizes the cumulative return of each prediction interval by applying a rolling optimization method, and integrates multiple cost functions to optimize the policy network, which helps to improve the learning efficiency and sample utilization of the policy, thereby enabling the optimized policy network to have higher obstacle avoidance performance.

FIG. 2 is a second flow chart of the network training method provided by the present invention; as shown in FIG. 2 , the complete flow of the method includes:

Step 210, calculating the optimal heading angle of the sample UAV at the target time based on the target strategy network and the environmental situation at the target time;

Step 220, the next moment position of the sample UAV is obtained according to the UAV dynamic constraint model, kinematic constraint and disturbance flow field method, and the next moment environmental situation and target moment reward value of the sample UAV are obtained in combination with the next moment position of the obstacle and the destination position, and the sample data at the target moment is constructed according to the optimal heading angle at the target moment, the environmental situation at the target moment, the next moment environmental situation and the target moment reward value, and then the sample data is stored in the experience replay pool;

Step 230, determining whether the number of sample data in the experience playback pool in step 220 exceeds a certain value; if it exceeds the certain value, executing step 240; if it does not exceed the certain value, jumping to step 210;

Step 240, sampling part of the sample data to be processed from the experience replay pool, and predicting the environmental situation, the optimal heading angle of the UAV and the reward value at multiple future moments corresponding to each sample data to be processed based on the multi-step prediction method of the prediction network predictive control;

Step 250, calculating at least one cost function by using the environmental situation at the corresponding moment and future moment of the sample data to be processed in step 240, the optimal heading angle of the drone, and the reward value, and then performing gradient update on the target policy network and the target prediction network according to the cost function;

Step 260, repeat steps 210-250 until a specified training round is reached to obtain an optimized policy network.

FIG. 3 is a third flow chart of the network training method provided by the present invention; as shown in FIG. 3 , the specific execution steps of step 240 include:

Step 310, sampling a plurality of sample data to be processed at different times from the experience replay pool;

Step 320, based on the environmental situation prediction value at the corresponding moment of the current prediction step and the target strategy network, calculate the optimal heading angle prediction value at the corresponding moment of the current prediction step; it should be noted that when the current prediction step is the initial prediction step, the environmental situation prediction value at the corresponding moment of the current prediction step is the environmental situation in each sample data to be processed;

Step 330, interacting with the prediction network to predict the environmental situation prediction value at the corresponding moment of the next prediction step and the reward prediction value at the corresponding moment of the current prediction step;

Step 340, calculating an updated target value of the target policy network;

Step 350, determine whether the current prediction step is greater than the preset prediction step length; if it is greater than the preset prediction step length, end step 240 and jump to step 250; if it is not greater than the preset prediction step length, jump to step 320.

In summary, in view of the problem of low training efficiency of reinforcement learning in drone obstacle avoidance, the obstacle avoidance method provided in this embodiment learns the environmental situation and reward model through a data-driven method, and uses a rolling optimization interval to train the drone's strategy. This method has higher learning efficiency, faster convergence speed of the strategy to the optimal value, and less sample capacity space required for the experience replay buffer, so that the drone can learn the best strategy with only a small number of attempts, which promotes the application of reinforcement learning methods in obstacle avoidance problems and improves the obstacle avoidance performance of drones.

Figure 4 is a flow chart of the obstacle avoidance method for unmanned aerial vehicles provided by the present invention; as shown in Figure 4, the method includes: step 410, obtaining the current environmental situation of the current unmanned aerial vehicle; step 420, inputting the current environmental situation into the optimized strategy network to obtain the current optimal heading angle of the current unmanned aerial vehicle; step 430, controlling the current unmanned aerial vehicle to perform the obstacle avoidance task according to the current optimal heading angle; wherein the optimized strategy network is trained based on the network training method provided in the above-mentioned embodiments.

Optionally, after obtaining the optimized policy network based on steps 110-140, the current environmental situation of the current drone is input into the optimized policy network, so that the optimized policy network makes a decision based on the current environmental situation to predict the optimal heading angle of the current drone at the current moment, thereby facilitating the current drone to perform the obstacle avoidance task according to the optimal heading angle at the current moment, so that it can obtain the maximum reward in the current environment and achieve effective obstacle avoidance behavior. This method realizes the expansion of the number of sample data by generating virtual environment data of the target prediction network, so as to reduce the number of interactions with the real environment, improve sample utilization, speed up training, and improve learning efficiency, so that the target policy network can quickly converge to the optimal when the drone interacts with the environment less times; it can also make the optimized decision model have both current decision experience and future decision experience, so as to make more optimized drone obstacle avoidance decisions, thereby improving the drone obstacle avoidance performance.

The network training device provided by the present invention is described below. The network training device described below and the network training method described above can be referenced to each other.

Figure 5 is a schematic diagram of the structure of the network training device provided by the present invention; as shown in Figure 5, the device includes: a construction unit 510 is used to construct sample data at the target moment according to the environmental situation of the sample drone at the target moment, the optimal heading angle at the target moment, the environmental situation at the next moment, and the reward value at the target moment; an extraction unit 520 is used to update the sample data at the target moment to the experience replay pool, and when the number of sample data in the updated experience replay pool reaches a preset number, extract the sample data to be processed at multiple different moments within the target prediction interval from the updated experience replay pool; a prediction unit 530 is used to input the environmental situation in each of the sample data to be processed into the target strategy network, obtain the optimal heading angle prediction value at multiple different future moments, and each of the The environmental situation in the sample data to be processed and the optimal heading angle prediction values at multiple different future moments are input into the target prediction network for multi-step prediction to obtain environmental situation prediction values and reward prediction values at multiple different future moments; the optimization unit 540 is used to perform reinforcement learning training on the target policy network according to the environmental situation, optimal heading angle and reward value in each of the sample data to be processed, as well as the environmental situation prediction value, the reward prediction value and the optimal heading angle prediction value, and obtain an optimized policy network according to the training results; wherein the optimized policy network is used to predict the optimal heading angle of the current drone at the current moment based on the current environmental situation of the current drone, so that the current drone can perform an obstacle avoidance task according to the optimal heading angle at the current moment. The network training device provided in this embodiment expands the amount of sample data by generating virtual environment data of the target prediction network, so as to reduce the number of interactions with the real environment, improve sample utilization, accelerate training speed, and improve learning efficiency, so that the target strategy network of the drone can converge to the optimal quickly when the number of interactions with the environment is reduced; it can also make the optimized decision model have both current decision-making experience and future decision-making experience, so as to make more optimized drone obstacle avoidance decisions, thereby improving the drone obstacle avoidance performance.

FIG6 is a schematic diagram of the structure of the obstacle avoidance device for unmanned aerial vehicles provided by the present invention; as shown in FIG6 , the device includes: an acquisition unit 610 for acquiring the current environmental situation of the current unmanned aerial vehicle; a decision unit 620 for inputting the current environmental situation into the optimized strategy network to obtain the optimal heading angle of the current unmanned aerial vehicle at the current moment; an obstacle avoidance control unit 630 for controlling the current unmanned aerial vehicle to perform the obstacle avoidance task according to the optimal heading angle at the current moment; wherein the optimized strategy network is obtained by training based on the network training method provided in the above embodiments. This device realizes the expansion of the number of sample data by generating virtual environment data of the target prediction network, so as to reduce the number of interactions with the real environment, improve sample utilization, speed up training speed, and improve learning efficiency, so that the target strategy network can quickly converge to the optimal when the number of interactions with the environment is less; it can also make the optimized decision model have both current decision experience and future decision experience, so as to make a more optimized obstacle avoidance decision for the unmanned aerial vehicle, thereby improving the obstacle avoidance performance of the unmanned aerial vehicle.

FIG7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG7 , the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the network training method or the drone obstacle avoidance method provided by the above methods.

In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the network training method or drone obstacle avoidance method provided by the above methods.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the network training method or drone obstacle avoidance method provided by the above methods.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of network training, comprising:

According to the environmental situation at the target moment, the optimal course angle at the target moment, the environmental situation at the next moment and the rewarding value at the target moment of the sample unmanned aerial vehicle, constructing sample data at the target moment;

Updating the sample data at the target moment to an experience playback pool, and extracting a plurality of sample data to be processed at different moments in a target prediction interval from the updated experience playback pool under the condition that the number of the sample data in the updated experience playback pool reaches a preset number;

Inputting the environmental situation in each sample data to be processed into a target strategy network to obtain a plurality of optimal heading angle predicted values at different future times, and inputting the environmental situation in each sample data to be processed and the optimal heading angle predicted values at the different future times into the target strategy network to perform multi-step prediction to obtain a plurality of environmental situation predicted values and rewarding predicted values at the different future times;

Performing reinforcement learning training on the target strategy network according to the environmental situation, the optimal course angle and the rewarding value in each sample data to be processed, and the environmental situation predicted value, the rewarding predicted value and the optimal course angle predicted value, and acquiring an optimized strategy network according to a training result;

the optimized strategy network is used for predicting the current time optimal course angle of the current unmanned aerial vehicle based on the current time environment situation of the current unmanned aerial vehicle so that the current unmanned aerial vehicle can execute obstacle avoidance tasks according to the current time optimal course angle.

2. The network training method according to claim 1, wherein the target time environment situation and the target time optimal heading angle are obtained based on the steps of:

Determining the target moment environment situation according to the target moment position, the radius and the destination position of the sample unmanned aerial vehicle, the target moment position, the target moment speed and the radius of the obstacle and the target moment distance between the sample unmanned aerial vehicle and the obstacle;

And inputting the environment situation at the target moment into the target strategy network to obtain the optimal course angle at the target moment.

3. The network training method of claim 1, wherein the next time environmental situation is obtained based on the steps of:

calculating to obtain the next moment position of the sample unmanned aerial vehicle according to an unmanned aerial vehicle dynamics constraint model, a kinematic constraint and a disturbance flow field method;

And determining the environmental situation of the next moment according to the position, the radius and the destination position of the sample unmanned aerial vehicle at the next moment, the position, the speed and the radius of the obstacle at the next moment and the distance between the sample unmanned aerial vehicle and the obstacle at the next moment.

4. A network training method as claimed in any one of claims 1 to 3, wherein the target time prize value is obtained on the basis of:

Determining a target moment rewarding value according to the target moment distance between the sample unmanned aerial vehicle and the obstacle, the radius of the sample unmanned aerial vehicle, the radius of the obstacle and a first rewarding value under the condition that the target moment distance between the sample unmanned aerial vehicle and the obstacle is smaller than a first distance value;

Determining a target time rewarding value according to the target time distance between the sample unmanned aerial vehicle and the destination position, the distance between the starting point position of the sample unmanned aerial vehicle and the destination position, the second rewarding value and the third rewarding value when the target time distance between the sample unmanned aerial vehicle and the obstacle is larger than or equal to the first distance value and the target time distance between the sample unmanned aerial vehicle and the destination position is smaller than the second distance value;

Determining a target time rewarding value according to the target time distance between the sample unmanned aerial vehicle and the destination position, the distance between the starting point position of the sample unmanned aerial vehicle and the destination position, and the third rewarding value when the target time distance between the sample unmanned aerial vehicle and the obstacle is larger than or equal to the first distance value and the target time distance between the sample unmanned aerial vehicle and the destination position is larger than or equal to the second distance value;

wherein the first prize value is a constant prize value; the second prize value is a threat prize for limiting the sample drone from moving away from the obstacle; and the third prize value is an additional prize value corresponding to the completion of the task.

5. The network training method of claim 4, wherein the second prize value is determined based on the steps of:

Determining the second rewarding value based on the target moment distance between the sample unmanned aerial vehicle and the obstacle, the radius of the sample unmanned aerial vehicle, the radius of the obstacle, a preset threat radius and a fourth rewarding value under the condition that the target moment distance between the sample unmanned aerial vehicle and the obstacle is larger than or equal to the first distance value and smaller than a third distance value;

And determining the second rewarding value based on a preset constant value under the condition that the target moment distance between the sample unmanned aerial vehicle and the obstacle is smaller than the first distance value or larger than or equal to the third distance value.

6. A network training method according to any one of claims 1 to 3, wherein the inputting the environmental situation in each of the sample data to be processed and the optimal heading angle predicted values at the different future times into the target prediction network to perform multi-step prediction, to obtain environmental situation predicted values and rewards predicted values at the different future times, includes:

Inputting the environmental situation and the optimal course angle predicted value in each sample data to be processed into a reward function network of the target prediction network, and inputting the environmental situation and the optimal course angle predicted value in each sample data to be processed into a situation transfer function network of the target prediction network to perform multi-step prediction, so as to obtain the reward predicted value and the environmental situation predicted value output by the reward function network.

7. A network training method according to any one of claims 1 to 3, wherein said reinforcement learning training of said target strategy network according to the environmental situation, the optimal course angle and the prize value in each of said sample data to be processed, and said environmental situation predicted value, said prize predicted value and said optimal course angle predicted value comprises:

Obtaining a value function cost function according to the optimal course angle in each piece of sample data to be processed, the optimal course angle predicted value, the environment situation predicted value and the rewarding predicted value;

acquiring a strategy cost function according to the optimal course angle predicted value and the environment situation predicted value;

acquiring a situation transfer cost function according to the environmental situation and the environmental situation predicted value in each sample data to be processed;

Obtaining a reward cost function according to the reward value and the reward predicted value in each sample data to be processed;

And performing reinforcement learning on the target strategy network according to the value function cost function, the strategy cost function, the situation transfer cost function and the rewarding cost function.

8. An unmanned aerial vehicle obstacle avoidance method, comprising:

Acquiring the current moment environmental situation of the current unmanned aerial vehicle;

inputting the current environmental situation to an optimized strategy network to obtain the current optimal course angle of the current unmanned aerial vehicle;

according to the optimal course angle at the current moment, controlling the current unmanned aerial vehicle to execute an obstacle avoidance task;

Wherein the optimized policy network is trained based on the network training method according to any one of claims 1 to 7.

9. A network training device, comprising:

The construction unit is used for constructing sample data of the target moment according to the environmental situation of the target moment, the optimal course angle of the target moment, the environmental situation of the next moment and the rewarding value of the target moment of the sample unmanned plane;

the extraction unit is used for updating the sample data at the target moment to the experience playback pool, and extracting the sample data to be processed at a plurality of different moments in the target prediction interval from the updated experience playback pool under the condition that the quantity of the sample data in the updated experience playback pool reaches the preset quantity;

The prediction unit is used for inputting the environmental situation in each sample data to be processed into a target strategy network to obtain a plurality of optimal heading angle predicted values at different future moments, inputting the environmental situation in each sample data to be processed and the optimal heading angle predicted values at the different future moments into the target prediction network to perform multi-step prediction to obtain a plurality of environmental situation predicted values and rewarding predicted values at the different future moments;

The optimizing unit is used for performing reinforcement learning training on the target strategy network according to the environment situation, the optimal course angle and the rewarding value in each sample data to be processed, the environment situation predicted value, the rewarding predicted value and the optimal course angle predicted value, and obtaining an optimized strategy network according to training results;

the optimized strategy network is used for predicting the current time optimal course angle of the current unmanned aerial vehicle based on the current time environment situation of the current unmanned aerial vehicle so that the current unmanned aerial vehicle can execute obstacle avoidance tasks according to the current time optimal course angle.

10. An unmanned aerial vehicle keeps away barrier device, characterized in that includes:

The acquisition unit is used for acquiring the current environmental situation of the current unmanned aerial vehicle at the current moment;

The decision unit is used for inputting the current time environment situation into an optimized strategy network to obtain the current time optimal course angle of the current unmanned aerial vehicle;

The obstacle avoidance control unit is used for controlling the current unmanned aerial vehicle to execute an obstacle avoidance task according to the current time optimal course angle;

Wherein the optimized policy network is trained based on the network training method according to any one of claims 1 to 7.