CN115509233A - Robot path planning method and system based on prior experience playback mechanism - Google Patents

Abstract

The invention discloses a robot path planning method and a system based on a priority experience playback mechanism; the method comprises the following steps: acquiring the current state and the target position of the path planning robot; inputting the current state and the target position of the path planning robot into the trained depth certainty strategy gradient network to obtain the action of the robot; the path planning robot finishes the path planning of the robot according to the obtained robot action; in the training process of the trained depth deterministic strategy gradient network, experience generated by the robot is stored in an experience pool, and the storage mode adopts an experience sample priority sequence for storage; the construction process of the experience sample priority sequence is as follows: calculating the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; and determining the weights of the three by using the information entropy, calculating the priority of the empirical sample by adopting a weighted summation mode, and constructing the priority sequence of the empirical sample.

Description

Robot path planning method and system based on priority experience playback mechanism

technical field

The invention relates to the technical field of robot path planning, in particular to a robot path planning method and system based on a priority experience playback mechanism.

Background technique

The statements in this section merely mention the background technology related to the present invention and do not necessarily constitute the prior art.

With the in-depth research of robots and artificial intelligence technology, the types of intelligent robots are becoming more and more abundant, and they are also playing an increasingly important role in various industries. Path planning enables intelligent robots to find a collision-free safe path from the starting point to the end point in a designated area, which is the basis of intelligent robot movement and is also a hot spot of current research. The process is to perceive the surrounding environment information of the intelligent robot through the sensor, determine its own pose, and then find an optimal path from the current position to the specified position in the environment.

In recent years, deep reinforcement learning (DRL) has been widely used in many fields, and the path planning algorithm combined with deep reinforcement learning has gradually become the focus of research. Deep reinforcement learning does not require the robot to know the environment in advance, but predicts the next action by perceiving the state of the surrounding environment, and obtains the reward of environmental feedback after performing the action, so that the robot can migrate from the current state to the next state. Repeat the loop until the robot reaches the target point or reaches the set maximum number of steps. DeepMind proposed the DDPG (Deep Deterministic Policy Gradient) algorithm, which uses a deterministic policy gradient algorithm, combines the Actor-critic framework with DQN, uses the convolutional neural network to simulate the policy function and Q function, and makes the output result a definite action value, which solves the problem of Deep reinforcement learning is currently an effective path planning algorithm for the problem that it cannot be applied or performs extremely poorly on high-dimensional or continuous action tasks. However, due to the insufficient utilization of empirical samples, the DDPG algorithm has poor environmental adaptability to robot path planning, and has problems such as low success rate and slow convergence speed.

The traditional DDPG adopts a random experience playback mechanism. The experience generated by the robot [s _t , a _t , r _t , s _t+1 ] is stored in the experience pool, and the experience samples are randomly selected to train the neural network. By breaking the temporal correlation between experiences, it solves the problem that experiences cannot be reused, and accelerates the learning process of robots. However, ER uses a unified random sampling strategy, which does not consider the importance of different experiences to robot learning, and cannot make full use of experiences with high importance, which affects the training efficiency of neural networks.

Contents of the invention

In order to solve the deficiencies of the prior art, the present invention provides a robot path planning method and system based on a priority experience playback mechanism; the present invention proposes a priority experience playback mechanism with dynamic sample priority, taking TD-error and Actor network losses into consideration Immediate rewards for functions and experience, and the weighted sum of the three to set the priority of experience. When sampling experience, give higher priority to the experience (positive experience) whose reward is greater than zero, and use these experiences to update the network parameters first. After positive experience samples are selected for training, in the next round of training, the priority of these experience samples will be decayed exponentially until it drops to the average value of the priority sequence. Increase the diversity of experience samples, improve the utilization rate of experience samples, and solve the problems of low success rate and slow convergence speed of DDPG algorithm path planning.

In the first aspect, the present invention provides a robot path planning method based on a priority experience playback mechanism;

A robot path planning method based on the priority experience playback mechanism, including:

Obtain the current state and target position of the path planning robot; input the current state and target position of the path planning robot into the trained deep deterministic policy gradient network to obtain the robot action; the path planning robot completes the path planning of the robot based on the obtained robot action ;

Among them, the deep deterministic policy gradient network after training, during the training process, the experience generated by the robot is stored in the experience pool, and the storage method is stored in the priority sequence of experience samples;

Among them, the experience sample priority sequence, its construction process is:

Calculate the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; use the information entropy to determine the weight of the three, use the weighted sum to calculate the priority of the experience sample, and construct the priority sequence of the experience sample ;

When sampling experience, judge whether the reward is greater than zero, if so, increase the priority of the experience sample, if not, keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then Update network parameters.

Further, the method also includes:

After the experience is selected to participate in the training, in the next round of training, the priority of the experience that has participated in the training is attenuated, and it is judged whether the average value of all the priorities after the attenuation is less than the set threshold, and if so, the experience is raised The priority of the sample, if not, continue to decay the priority until it is reduced to the average value of the priority sequence.

In the second aspect, the present invention provides a robot path planning system based on a priority experience playback mechanism;

Robot path planning system based on priority experience playback mechanism, including:

An acquisition module configured to: acquire the current state and target position of the path planning robot;

The path planning module is configured to: input the current state and target position of the path planning robot into the trained deep deterministic policy gradient network to obtain the robot action; the path planning robot completes the path planning of the robot according to the obtained robot action;

Among them, the deep deterministic policy gradient network after training, during the training process, the experience generated by the robot is stored in the experience pool, and the storage method is stored in the priority sequence of experience samples;

Among them, the experience sample priority sequence, its construction process is: calculate the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; use the information entropy to determine the weight of the three, and use the weighted summation method to calculate the priority of experience samples, and construct the priority sequence of experience samples;

When sampling experience, judge whether the reward is greater than zero, if so, increase the priority of the experience sample, if not, keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then Update network parameters.

In a third aspect, the present invention also provides an electronic device, comprising:

memory for non-transitory storage of computer readable instructions; and

a processor for executing said computer readable instructions,

Wherein, when the computer-readable instructions are executed by the processor, the method described in the first aspect above is performed.

In a fourth aspect, the present invention also provides a storage medium that non-transitorily stores computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions for executing the method described in the first aspect .

In a fifth aspect, the present invention also provides a computer program product, including a computer program, which is used to implement the method described in the first aspect when running on one or more processors.

Compared with prior art, the beneficial effect of the present invention is:

Aiming at the problems of single sample and low sample utilization rate in the experience playback mechanism based on DDPG algorithm, the present invention proposes a priority experience playback mechanism of dynamic sample priority sequence, which integrates TD-error, Actor network loss function and immediate reward, Use information entropy to determine the weights of the three, calculate the priority of experience samples through weighted summation, and construct a priority sequence of experience samples; give positive experience a higher priority, make it preferentially sampled, and accelerate network convergence; fully consider the diversity of experience samples , when the positive experience is trained, its priority decays exponentially in each subsequent round until it reaches the average value of the priority sequence.

Description of drawings

The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention.

FIG. 1 is a schematic diagram of a flow chart for adjusting experience sample priorities in Embodiment 1 of the present invention;

FIG. 2 is a flow chart of priority experience playback in Embodiment 1 of the present invention.

detailed description

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the present invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used here is only for describing specific embodiments, and is not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive Comprising, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include steps or units not explicitly listed or for these processes, methods, Other steps or units inherent in a product or equipment.

In the case of no conflict, the embodiments and the features in the embodiments of the present invention can be combined with each other.

The acquisition of all data in this embodiment is based on compliance with laws and regulations and user consent, and the legal application of data.

Explanation of terms:

Robot path planning: let the intelligent robot find a collision-free safe path from the starting point to the ending point in the designated area.

Deep Reinforcement Learning (DRL): Combining the perception ability of deep learning and the decision-making ability of reinforcement learning, it is an artificial intelligence method that is closer to the way of human thinking.

DDPG: an algorithm for robot path planning

Experience Replay (ER): Part of the robot path planning training process, a technique that stabilizes the probability distribution of experience, which can improve the stability of training.

Prioritized Experience Replay (PER): Improve on the basis of experience replay, and use non-uniform sampling instead of uniform sampling of experience replay to extract experience.

Information entropy: Describes the uncertainty of each possible event of information.

DeepMind proposes the concept of Priority Experience Replay (PER), which measures the priority of experience by the absolute value of time difference error (TD-error). The larger the TD-error, the more important the experience is for the robot to learn. On the contrary, the smaller the TD-error, the lower the importance of the experience, so that the robot can focus on the experience with high importance and further improve the experience utilization rate. Accelerate the learning process of robots. However, the learning process of PER often ignores the effects of immediate reward and small TD-error experience, resulting in the problem of a single sample.

Embodiment one

This embodiment provides a robot path planning method based on a priority experience playback mechanism;

As shown in Figure 1, the robot path planning method based on the priority experience playback mechanism includes:

S101: Obtain the current state and target position of the path planning robot;

S102: Input the current state and target position of the path planning robot into the trained deep deterministic policy gradient network to obtain the robot action; the path planning robot completes the path planning of the robot according to the obtained robot action;

Among them, the deep deterministic policy gradient network after training, during the training process, the experience generated by the robot is stored in the experience pool, and the storage method is stored in the priority sequence of experience samples;

Among them, the experience sample priority sequence, its construction process is:

Calculate the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; use the information entropy to determine the weight of the three, use the weighted sum to calculate the priority of the experience sample, and construct the priority sequence of the experience sample ;

When sampling experience, judge whether the reward is greater than zero, if so, increase the priority of the experience sample, if not, keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then Update network parameters.

Further, the method also includes:

After the experience is selected to participate in the training, in the next round of training, the priority of the experience that has participated in the training is attenuated, and it is judged whether the average value of all the priorities after the attenuation is less than the set threshold, and if so, the experience is raised The priority of the sample, if not, continue to decay the priority until it is reduced to the average value of the priority sequence.

Further, the network structure of the trained deep deterministic policy gradient network includes:

Actor module, experience pool and Critic module connected in sequence;

Among them, the Actor module includes: the current Actor network and the target Actor network connected in sequence;

Among them, the Critic module includes: the current Critic network and the target Critic network connected in sequence;

The current Actor network is connected to the current Critic network.

Further, the trained deep deterministic policy gradient network, during the training process, the interaction process between the robot and the environment is as follows:

At each moment t, the current Actor network of the robot gets the action a _t according to the environment state _st , acts on the environment to obtain the immediate reward r _t and the environment state _st+1 at the next moment, and the current Critic network according to the environment state _st and the action a _t gets the Q value Q(s _t , a _t ), and evaluates the action a _t ;

Q表示当前Critic网络产生的Q值，s_i表示从经验池采样的状态，a_i表示从经验池中采样的动作，θ^Q表示当前Actor网络的参数，Q(s_t,a_t)表示状态s_t和动作a_t的价值。Sampling the i-th experience [s _i ,a _i ,ri ,s _{i +1} ] from the experience pool, the current Actor network adjusts the action strategy according to the Q value Q(s _t ,at ₎ , the loss function of the current Actor network is

Q represents the Q value generated by the current Critic network, s _i represents the state sampled from the experience pool, a _i represents the action sampled from the experience pool, θ ^Q represents the parameters of the current Actor network, and Q(s _t , a _t ) represents the state s _t and the value of action a _t .

The target Actor network obtains the estimated action a' according to the environment state s _t+1 at the next moment;

The target critic network obtains the Q' value Q'(s _t+1 ,a') according to the next moment environment state s _t+1 and the estimated action a';Q'(s _t+1 ,a') represents the state s _{t+ 1} and the value of action a';

Calculate the difference between the Q value and the Q' value to obtain the time differential error TD-error.

It should be understood that information entropy represents the probability of occurrence of a discrete random event, and measures the amount of information in order to eliminate the uncertainty of the event. The more information introduced to eliminate the uncertainty of events, the higher the information entropy, and vice versa.

The calculation of information entropy H(X) is shown in formula (1):

Among them, X represents the unknown event, p _i represents the probability of the unknown event.

Considering the impact of immediate reward, TD-error and Actor network loss function on the robot training process, it is introduced into the construction process of the priority sequence of experience samples. However, the simple accumulation and summation of the three cannot determine the degree of their influence on the training, which may cause a certain factor to account for too much, resulting in inaccurate priority of the calculated experience samples. In order to eliminate the uncertainty of the three kinds of information, information entropy is introduced to calculate its weight factor.

Further, the priority of the calculation time difference error, the priority of the current Actor network loss function and the priority of the immediate reward specifically include:

p _im-reward ＝|r _i | (2)

p _TD-error = |δ _i | (3)

表示Actor网络损失函数，Q表示当前Critic网络产生的Q值，s_i表示从经验池采样的状态，a_i表示从经验池中采样的动作，θ^Q表示当前Actor网络的参数，p_im-reward表示立即奖励的优先级，p_TD-error表示TD-error的优先级，p_Actor-loss表示当前Actor网络损失函数的优先级。Among them, ò represents a very small constant with a value of 0.05, r _i represents immediate reward, δ _i represents TD-error,

Represents the Actor network loss function, Q represents the Q value generated by the current Critic network, s _i represents the state sampled from the experience pool, a _i represents the action sampled from the experience pool, θ ^Q represents the parameters of the current Actor network, p _im-reward Indicates the priority of the immediate reward, p _TD-error indicates the priority of TD-error, and p _Actor-loss indicates the priority of the current Actor network loss function.

Further, the use of information entropy to determine the weights of the three, using weighted summation to calculate the priority of experience samples, and constructing a priority sequence of experience samples, specifically includes:

H(r _i )＝-p _{reward＞Ravg} log ₂ (p _{reward＞Ravg} )-p _{reward＜Ravg} log ₂ (p _{reward＜Ravg} ) (7)

为积极经验训练中TD-error 的概率，

为消极经验训练中TD-error的概率；

为积极经验训练中Actor 网络损失函数的概率，

为消极训练中Actor网络损失函数的概率；p_{reward＞Ravg}为立即奖励大于奖励平均值的概率，p_{reward＜Ravg}为立即奖励小于奖励平均值的概率。During a training process of the network model, if the reward obtained by the robot is greater than 0, this training is called active training, and the obtained experience samples are called positive experience samples; otherwise, it is called negative training; formula (5)- In (7), H(TD) represents the TD-error information entropy, H(TA) represents the information entropy of the current Actor network loss function, H(r _i ) represents the immediate reward information entropy,

is the probability of TD-error in positive experience training,

is the probability of TD-error in negative experience training;

is the probability of the Actor network loss function in positive experience training,

is the probability of the Actor network loss function in negative training; p _reward>Ravg is the probability that the immediate reward is greater than the average reward, and p _reward<Ravg is the probability that the immediate reward is smaller than the average reward.

Calculate the information entropy to determine the weight a of the immediate reward priority, the weight β of the time difference error priority and the weight υ of the Actor network loss function priority:

υ＝1-a-β (10)

In formulas (8)-(10), H(TD) represents the TD-error information entropy, H(TA) represents the information entropy of the current Actor network loss function, and H(r _i ) represents the immediate reward information entropy.

Finally, according to the calculated weight coefficients and p _im-reward , p _TD-error and p _Actor-loss , use formula (11) to calculate the priority of each empirical sample, where ò represents a very small constant. As shown in formula (11):

p _i ＝(a×p _im-reward +β×p _TD-error +υ×p _Actor-loss )+ò (11)

Further, as shown in Figure 2, during experience sampling, it is judged whether the reward is greater than zero, if so, the priority of the experience sample is raised, and if not, the priority of the experience sample is kept unchanged; according to the priority from high to The low order samples the experience, and then updates the network parameters; specifically includes:

作为积极经验(奖励大于零的经验) 每一轮训练开始时的优先级权重，根据经验样本优先级p_i，对经验样本的优先级进行调整，令积极经验样本(奖励大于零的经验样本)的优先级为：During experience sampling, it is judged whether the reward is greater than zero. If so, the priority of the experience sample is raised. During the process of raising the priority of the experience sample, the parameter

As the priority weight of positive experience (experience with reward greater than zero) at the beginning of each round of training, adjust the priority of experience samples according to the priority p _i of experience samples, so that positive experience samples (experience samples with reward greater than zero) The priority is:

where p′ _i >p _i ,

The priority of experience samples whose reward is less than or equal to zero is not adjusted, and it is still p _i ;

Calculate the sampling probability P _i according to the priority of the experience samples, and sample the experience samples from the experience pool according to the sampling probability P _i for training;

The sampling probability calculation process is shown in formula (12):

Among them, α is a constant, and the value range is [0,1]. When α=0, it means uniform sampling, and n is the total number of empirical samples.

It should be understood that in the experience pool, an experience sample with a high absolute value of TD-error usually means that the Q value of the current critic network is far from the Q value of the target critic network, and has high learning potential. Prioritizing the replay of such experience samples may lead to a substantial improvement in the learning ability of the robot.

However, if only the factor of TD-error is considered and the importance of rewards in the training process is ignored, it is easy to use marginal experience excessively, resulting in network overfitting.

Positive experience such as successful experience or high reward experience is also the most important experience that robots need to learn. If more positive experience is sampled, it can not only accelerate the convergence of the algorithm, but also effectively alleviate overfitting. Therefore, positive experience samples will be given higher priority for experience replay.

Further, after the experience is selected to participate in the training, in the next round of training, the priority of the experience that has participated in the training is attenuated, and it is judged whether the average value of all the priorities after the attenuation is less than the set threshold, and if so, Just increase the priority of the experience sample, if not, continue to decay the priority until it is reduced to the average value of the priority sequence, including:

Suppose the positive experience sample is j, the priority is p _j , and the priority sequence of batch sampling of experience samples is p=(p ₁ ,p ₂ ,···p _j ···,p _n ).

Then, the priority p _j of the next round j undergoes exponential decay according to the decay factor σ:

When the priority of the experience sample is greater than the set threshold Z, the priority decay is allowed, otherwise the decay is stopped;

The calculation formula of the threshold Z:

In order to quickly learn the latest experience of robot-environment interaction in a limited-capacity experience pool, the priority of experience samples needs to be updated in time.

The method proposed in the present invention is a DDPG algorithm based on priority sequence sampling. Make full use of the actor-critic framework of the DDPG algorithm, combine the actor network loss function, TD-error and immediate reward, determine the weighting coefficient according to the information entropy, and construct the priority sequence of experience samples. The immediate reward of the robot is reasonably called, and the experience samples with immediate reward greater than zero are regarded as positive experience, and the rest are negative experience. Giving higher priority to positive experiences allows them to be sampled more frequently to speed up the training process of the DDPG algorithm. At the same time, in order to take into account the diversity of experience samples, experience samples with low priority can also be sampled. Whenever a positive experience is sampled, its priority will decay exponentially in the next round of training until it is less than or equal to the set up to the threshold. The specific flow of the algorithm is shown in Figure 2.

The present invention conducts experiments evaluating the DDPG algorithm based on priority sequence sampling. The experiment uses Gazebo physical simulation software for simulation experiments, chooses python language and pytorch framework, and uses Ros operating system for information transmission. An important indicator for evaluating the quality of an algorithm is the size of the robot’s rewards for interacting with the environment in a round of training according to the algorithm. If the rewards obtained are high and relatively stable, we believe that the algorithm performs well in this round of the environment.

The robot was trained in four different environments, the first of which had no obstacles, and the remaining environments had obstacles of different sizes and arrangements. The results of robot training show that in an environment without obstacles, the PER method is not stable enough, and the average reward value is low, mainly floating in the range of 0-0.5. The value is relatively high, mainly floating in the range of 0.75 to 1.75.

In an environment with obstacles, the training curve of the PER method fluctuates greatly, and the anti-risk ability is poor. 1.0 interval, and achieved good results.

In order to verify the success rate and effectiveness of the proposed method in robot path planning in an unknown environment, after the robot training is completed, it is tested for 200 tasks in a new simulation environment. The test results show that in terms of success rate, the PER method has a success rate of 86%, while the success rate of the proposed algorithm of the present invention can reach 90.5%, which is obviously higher than the PER method; in terms of time-consuming, the algorithm of the present invention is 13% faster than the PER algorithm, The robot reaches the target point more quickly. The algorithm proposed by the invention has a low trial-and-error rate during path planning and fewer collisions with obstacles.

The above experimental results show that in the task of robot path planning, the average reward value of the PER method fluctuates greatly, and the training process is unstable, because the method for evaluating the priority of experience samples has only one TD-error index, ignoring the immediate reward in network training. role, resulting in excessive use of marginal experience samples, prone to local optima. The present invention proposes to construct a compound priority to comprehensively consider the immediate reward, TD-error and the feedback of the Actor network, use the information entropy to determine the dynamic weighting coefficient, give the experience sample a reasonable priority, the training process is gradually stable, and the average reward value is improved compared with PER , indicating that the dynamically adjusted priority guarantees the demand for diversity of experience samples during robot training. On this basis, the priority sequence sampling mechanism is superimposed to distinguish between positive experience samples and negative experience samples, and adjust the priority of positive experience samples. After sampling training, the priority is attenuated. In the comparative experiments of the number of training rounds, higher average reward values are obtained, which proves the effectiveness of the improved algorithm of the present invention.

Embodiment two

This embodiment provides a robot path planning system based on a priority experience playback mechanism;

Robot path planning system based on priority experience playback mechanism, including:

An acquisition module configured to: acquire the current state and target position of the path planning robot;

The path planning module is configured to: input the current state and target position of the path planning robot into the trained deep deterministic policy gradient network to obtain the robot action; the path planning robot completes the path planning of the robot according to the obtained robot action;

Among them, the deep deterministic policy gradient network after training, during the training process, the experience generated by the robot is stored in the experience pool, and the storage method is stored in the priority sequence of experience samples;

Among them, the experience sample priority sequence, its construction process is: calculate the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; use the information entropy to determine the weight of the three, and use the weighted summation method to calculate the priority of experience samples, and construct the priority sequence of experience samples;

When sampling experience, judge whether the reward is greater than zero, if so, increase the priority of the experience sample, if not, keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then Update network parameters.

What needs to be explained here is that the above acquisition module and path planning module correspond to steps S101 to S102 in the first embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to those disclosed in the first embodiment above Content. It should be noted that, as a part of the system, the above-mentioned modules can be executed in a computer system such as a set of computer-executable instructions.

The description of each embodiment in the foregoing embodiments has its own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The proposed system can be implemented in other ways. For example, the above-described system embodiments are only illustrative. For example, the division of the above modules is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules can be combined or integrated into another A system, or some feature, can be ignored, or not implemented.

Embodiment Three

This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are programmed Stored in the memory, when the electronic device is running, the processor executes one or more computer programs stored in the memory, so that the electronic device executes the method described in Embodiment 1 above.

It should be understood that in this embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate array FPGA or other programmable logic devices , discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory may include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software module may be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in this embodiment can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

Embodiment Four

This embodiment also provides a computer-readable storage medium for storing computer instructions, and when the computer instructions are executed by a processor, the method described in the first embodiment is completed.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A robot path planning method based on a priority experience playback mechanism, characterized in that it includes: Obtain the current state and target position of the path planning robot; input the current state and target position of the path planning robot into the trained deep deterministic policy gradient network to obtain the robot action; the path planning robot completes the path planning of the robot based on the obtained robot action ; Among them, the deep deterministic policy gradient network after training, during the training process, the experience generated by the robot is stored in the experience pool, and the storage method is stored in the priority sequence of experience samples; Among them, the experience sample priority sequence, its construction process is: Calculate the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; use the information entropy to determine the weight of the three, use the weighted sum to calculate the priority of the experience sample, and construct the priority sequence of the experience sample ; When sampling experience, judge whether the reward is greater than zero, if so, increase the priority of the experience sample, if not, keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then Update network parameters. 2. The robot path planning method based on the priority experience playback mechanism as claimed in claim 1, wherein said experience is sampled in order of priority from high to low, and then network parameters are updated, and then further comprising: After the experience is selected to participate in the training, in the next round of training, the priority of the experience that has participated in the training is attenuated, and it is judged whether the average value of all the priorities after the attenuation is less than the set threshold, and if so, the experience is raised The priority of the sample, if not, continue to decay the priority until it is reduced to the average value of the priority sequence. 3. the robot path planning method based on priority experience replay mechanism as claimed in claim 1, it is characterized in that, the deep deterministic policy gradient network after the training, in the process of training, the interactive process of robot and environment is as follows: At each moment t, the current Actor network of the robot gets the action a _t according to the environment state _st , acts on the environment to obtain the immediate reward r _t and the environment state _st+1 at the next moment, and the current Critic network according to the environment state _st and the action a _t gets the Q value Q(s _t , a _t ), and evaluates the action a _t ; Sampling the i-th experience [s _i ,a _i ,ri ,s _{i +1} ] from the experience pool, the current Actor network adjusts the action strategy according to the Q value Q(s _t ,at ₎ , and the loss function of the current Actor network is ▽ _a Q(s _i , a _i |θ ^Q ); Q represents the Q value generated by the current Critic network, s _i represents the state sampled from the experience pool, a _i represents the action sampled from the experience pool, and θ ^Q represents the current Actor network The parameters of , Q(s _t , a _t ) represent the value of state s _t and action a _t ; The target Actor network obtains the estimated action a' according to the environment state s _t+1 at the next moment; The target critic network obtains the Q' value Q'(s _t+1 ,a') according to the next moment environment state s _t+1 and the estimated action a';Q'(s _t+1 ,a') represents the state s _{t+ 1} and the value of action a'; Calculate the difference between the Q value and the Q' value to obtain the time differential error TD-error. 4. The robot path planning method based on priority experience playback mechanism as claimed in claim 1, characterized in that, the priority of the calculation time difference error, the priority of the current Actor network loss function and the priority of immediate reward, specifically include: p _im-reward ＝|r _i | (2) p _TD-error = |δ _i | (3) p _Actor-loss ＝|▽ _a Q(s _i ,a _i |θ ^Q )| (4) Among them, r _i represents the immediate reward, δ _i represents TD-error, ▽ _a Q(s _i , a _i |θ ^Q ) represents the loss function of the Actor network, Q represents the Q value generated by the current critic network, and s _i represents the value from the experience pool The state of sampling, a _i represents the action sampled from the experience pool, θ ^Q represents the parameters of the current Actor network, p _im-reward represents the priority of immediate reward, p _TD-error represents the priority of TD-error, p _{Actor- loss} represents the priority of the current Actor network loss function. 5. The robot path planning method based on the priority experience replay mechanism as claimed in claim 4, characterized in that, the use of information entropy to determine the weight of the three, the weighted summation method is used to calculate the experience sample priority, and the experience sample is constructed Priority order, specifically including:

H(r _i )＝-p _{reward＞Ravg} log ₂ (p _{reward＞Ravg} )-p _{reward＜Ravg} log ₂ (p _{reward＜Ravg} ) (7) During a training process of the network model, if the reward obtained by the robot is greater than 0, this training is called active training, and the obtained experience samples are called positive experience samples; otherwise, it is called negative training; formula (5)- In (7), H(TD) represents the TD-error information entropy, H(TA) represents the information entropy of the current Actor network loss function, H(r _i ) represents the immediate reward information entropy,

is the probability of TD-error in positive experience training,

is the probability of TD-error in negative experience training;

is the probability of the Actor network loss function in positive experience training,

is the probability of the Actor network loss function in passive training; p _reward>Ravg is the probability that the immediate reward is greater than the average reward, and p _reward<Ravg is the probability that the immediate reward is smaller than the average reward; Calculate the information entropy to determine the weight a of the immediate reward priority, the weight β of the time difference error priority and the weight υ of the Actor network loss function priority:

υ＝1-a-β (10) In formulas (8)-(10), H(TD) represents TD-error information entropy, H(TA) represents the information entropy of the current Actor network loss function, and H(r _i ) represents immediate reward information entropy; Finally, according to the calculated weight coefficient and p _im-reward , p _TD-error and p _Actor-loss , use formula (11) to calculate the priority of each experience sample, where ∈ represents a very small constant; such as formula (11) Shown: p _i =(a×p _im-reward +β×p _TD-error +υ×p _Actor-loss )+∈ (11). 6. The robot path planning method based on priority experience replay mechanism as claimed in claim 1, characterized in that, during experience sampling, it is judged whether the reward is greater than zero, if yes, the priority of experience sample is raised, if not, then Keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then update the network parameters; specifically include: During experience sampling, it is judged whether the reward is greater than zero. If so, the priority of the experience sample is raised. During the process of raising the priority of the experience sample, the parameter

As the priority weight of the experience with a reward greater than zero at the beginning of each round of training, the priority of the experience sample is adjusted according to the priority p _i of the experience sample, so that the priority of the experience sample with a reward greater than zero is:

Where p _i ' represents the adjusted priority; The priority of experience samples whose reward is less than or equal to zero is not adjusted, and it is still p _i ; Calculate the sampling probability P _i according to the priority of the experience samples, and sample the experience samples from the experience pool according to the sampling probability P _i for training; The sampling probability calculation process is shown in formula (12):

Among them, α is a constant, and the value range is [0,1]. When α=0, it means uniform sampling, and n is the total number of empirical samples. 7. The robot path planning method based on the priority experience playback mechanism as claimed in claim 2, characterized in that, after the experience is selected to participate in training, in the next round of training process, the priority of the experience that has participated in the training is carried out. Attenuation, judging whether the average value of all priorities after the attenuation is less than the set threshold, if yes, increase the priority of the experience sample, if not, continue to attenuate the priority until it is reduced to the average value of the priority sequence, specifically include: Suppose the positive experience sample is j, the priority is p _j , and the priority sequence of batch sampling of experience samples is p=(p ₁ ,p ₂ ,…p _j …,p _n ); Then, the priority p _j of the next round j undergoes exponential decay according to the decay factor σ:

When the priority of the experience sample is greater than the set threshold Z, the priority decay is allowed, otherwise the decay is stopped; The calculation formula of the threshold Z:

8. Robot path planning system based on priority experience playback mechanism, including: An acquisition module configured to: acquire the current state and target position of the path planning robot; The path planning module is configured to: input the current state and target position of the path planning robot into the trained deep deterministic policy gradient network to obtain the robot action; the path planning robot completes the path planning of the robot according to the obtained robot action; Among them, the deep deterministic policy gradient network after training, during the training process, the experience generated by the robot is stored in the experience pool, and the storage method is stored in the priority sequence of experience samples; Among them, the experience sample priority sequence, its construction process is: calculate the priority of the time difference error, the priority of the current Actor network loss function and the priority of the immediate reward; use the information entropy to determine the weight of the three, and use the weighted summation method to calculate the priority of experience samples, and construct the priority sequence of experience samples; When sampling experience, judge whether the reward is greater than zero, if so, increase the priority of the experience sample, if not, keep the priority of the experience sample unchanged; sample the experience in order of priority from high to low, and then Update network parameters. 9. An electronic device comprising: memory for non-transitory storage of computer readable instructions; and a processor for executing said computer readable instructions, Wherein, when the computer-readable instructions are executed by the processor, the method described in any one of claims 1-7 is performed. 10. A storage medium storing computer-readable instructions in a non-transitory manner, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-7 are executed.

Images