CN114815834A - A dynamic path planning method for mobile agents in a stage environment - Google Patents

Abstract

The invention discloses a dynamic path planning method for a mobile intelligent agent in a stage environment, belonging to the technical field of path planning of intelligent robots; the method comprises the steps of firstly, obtaining obstacle information around a mobile intelligent body by constructing a global map, classifying obstacles into dynamic obstacles and static obstacles, then establishing a local map, coding the dynamic obstacle information through an LSTM network, and calculating the importance of each dynamic obstacle through a social attention mechanism to achieve better obstacle avoidance. Different avoiding conditions of the dynamic and static obstacles are responded by constructing a new reward function, so that the path planning problem of the mobile intelligent body under the complex stage environment is realized. A new experience pool updating method is provided to improve the convergence speed of network training, and meanwhile, simulation experiments are carried out on the method provided by the invention to prove the superiority of the algorithm, so that the method has very high practical value.

Description

A dynamic path planning method for mobile agents in a stage environment

technical field

The invention relates to the technical field of intelligent robot path planning, in particular to a dynamic path planning method for a mobile intelligent body in a stage environment.

Background technique

In order to meet the diverse service needs of grassroots culture, it is necessary to carry out cultural service functions such as cultural performances, conference discussions, exhibition reading and folk activities in the activity space of grassroots small cultural service complexes. Cultural facilities with classified configuration cannot meet residents' expectations for comprehensive cultural needs. Small cultural service complexes can better solve this problem. Its focus is to promote the construction of multi-functional activity spaces. It can integrate folk activities, exhibitions, conferences, reading and other functions to form an integrated venue service carrier, which not only meets the needs of rural areas. The diversity and self-organization needs of grassroots culture have also explored a new model to meet the requirements of public cultural services in my country's new rural areas.

In order to reduce the waste of land resources and improve the efficiency of space utilization, various intelligent mobile bodies are required to assist in the rapid combination and switching of various functional spaces. To meet the requirements of "multi-purpose in one hall" in small cultural complexes, it is often necessary to use intelligent mobile bodies to assist in the rapid combination and switching of multiple functional spaces, so as to realize dynamic path planning in a crowded environment in small complex spaces, so as to meet the needs of a single space. A variety of cultural service needs.

The small cultural complex space is a typical coexistence environment of people, machines and objects. In the process of functional space switching, multiple objects and equipment in the space need to have other equipment to carry out trajectory planning motion to realize the switching service of space functions, so How to quickly avoid dynamic and static obstacles and reach the target point in the switching process requires us to design a dynamic path planning algorithm to control the intelligent moving body, and the environment in the cultural complex space is crowded, so the dynamic path planning algorithm has high requirements.

The traditional dynamic path planning algorithm relies on the rapid refresh of the sensor to perceive the information of the surrounding obstacles, and the planned path will also change with the dynamic obstacles, resulting in the problem of detours or unnatural trajectories. It is unable to predict the movement trend of surrounding dynamic obstacles and lacks adaptability. For the above problems, it is urgent to propose a dynamic path planning method that can distinguish between dynamic and static obstacles and predict the trend of dynamic obstacles.

SUMMARY OF THE INVENTION

Aiming at the problems existing in the prior art, the purpose of the present invention is to provide a dynamic path planning method for a mobile agent in a stage environment. The method designs a new Markov decision process and Network structure, add attention score to dynamic obstacles by introducing social attention mechanism, use long short-term memory neural network to solve the problem of unfixed dimension of feedforward network, and build a new reward function to deal with different avoidance of dynamic and static obstacles Therefore, a new experience pool update method is proposed to improve the convergence speed of network training, so that the mobile agent can distinguish between dynamic and static obstacles and predict the motion trend of dynamic obstacles. In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

A dynamic path planning method for a mobile agent in a stage environment, comprising the following steps:

1) Establish a simulation environment model of mobile agents and dynamic and static obstacles based on the gym library;

2) Design the Maldives decision-making process, design the state space S, the action space A, the transition probability P, the reward R and the discount factor γ;

3) Design the neural network structure;

4) Use the optimal reciprocal collision avoidance algorithm (ORCA) to initialize the network parameters through imitation learning pre-training; after the imitation learning is over, the network parameters are optimized by training the actual interaction of the mobile agent in the simulation environment;

5) The optimal value function is obtained by training the neural network through the adaptive time estimation method (Adam):

V ^* (u _t )=∑γΔt ^·Vpref ·P( _{u t} ,at )

6) Set the optimal strategy by maximizing the cumulative return:

表示下一时刻的联合状态where _ut represents the joint state of the current mobile agent and the obstacle, at represents the set of action spaces, γ represents the decay factor, _Δt represents the time interval between two actions, Vpref represents the preferred speed, and V ^* represents the Merit function, P is the state transition function, R is the reward function;

Represents the joint state at the next moment

7) Select the action at the current moment according to the optimal _strategy until the mobile agent reaches the target.

Further, in the step 1), the mobile agent and the dynamic obstacle are set as a circle with a radius of 0.3 meters, and the static obstacle is defined as a circle with a radius of 0.5 meters to 1 meter or an area of Quadrilaterals between 1 square meter and 1.5 square meters.

Further, in the step 2), the state space S is set, wherein the state of the dynamic obstacle is S _D =[P _x ,P _y ,V _x ,V _y ,r,V _pref ], the state of the static obstacle is S _S =[P _x ,P _y ,r], the state of the mobile agent is S _T =[P _x ,P _y ,G _x ,G _y ,V _x ,V _y ,θ,r,V _pref ] joint State u _t =[S _T , S _S , S _D ]. Among them (P _x , P _y ) are the current positions of the mobile agent and static and dynamic obstacles, and (G _x , G _y ) are the set target points , θ is the heading angle of the mobile agent, r is the radius of the mobile agent and the dynamic and static obstacles, V _pref is the preferred speed of the mobile agent, (V _x , V _y ) is the mobile agent and the dynamic obstacle the moving speed of the object;

x取1，2，3，4，5可获得5个变化平滑的线速度，动作空间共有90中动作组合；The action space A is the linear velocity and the angular velocity. In order to meet the dynamic constraints, the angular velocity is divided into 18 equal parts in the interval [-π/4, π/4], and the linear velocity follows the function

Take 1, 2, 3, 4, and 5 for x to obtain 5 linear velocities with smooth changes. There are 90 action combinations in the action space;

The transition probability P is approximated by the trajectory prediction model;

The reward R is set as:

where G _{x, y} is the position information of the target point, P _{x, y} is the current position information of the mobile agent, d _s is the distance between the mobile agent and the static obstacle, and d _d is the mobile agent and the dynamic obstacle The distance between; the discount factor γ is taken as 0.9.

Further, the network structure in the step 3) is composed of the following modules: 1. Input layer: the input layer is the joint state u _t =[ST , _{S S ,} S _D ] in the above steps. 2. Long Short-Term Memory Neural Network Module (LSTM): Through the LSTM module, the obstacles around the mobile agent can be sorted, and the output parameters of the network layer can be fixed. 3. Social attention mechanism: Through the social attention mechanism module, the probability of collision between the mobile agent and the surrounding dynamic obstacles can be analyzed and displayed in the form of scores. 4. Output layer: The output layer outputs the optimal value function V ^* (u _t ) through a weighted linear combination of network parameters.

Further, the network operation process in the step 3) is as follows: first, input the state information of the mobile agent and the obstacle into the network, and then divide the obstacles into dynamic obstacles and static obstacles according to the state information, and then divide the mobile agent into a dynamic obstacle and a static obstacle. The state of the dynamic obstacle and the state of the dynamic obstacle are input into the LSTM module, and then into the social attention mechanism module, and then the processed state, the obtained interaction feature and the state of the static obstacle are input into the two-layer fully-connected layer, and finally the activation function is used to adjust it. Perform normalization to get the optimal value function.

Further, in the step 4), when the mobile agent interacts in the simulation environment, the current state information, action information and reward information are stored in the experience pool as an experience, and when the experience reaches the maximum capacity, the new experience replaces the reward. Low old experience storage, thereby increasing the probability of selecting excellent experience and improving the convergence speed of the network. During each episode of interaction, the current episode ends when the mobile agent encounters an obstacle or exceeds the maximum time for a single episode run. The experience is then propagated through the gradient direction to update the network parameters.

The beneficial effects of the invention are as follows: a new Markov decision-making process is designed to adapt to the complex situation of obstacles in the complex space, and a new network structure is designed to realize the classification and processing of dynamic and static obstacles, and realize the detection of dynamic obstacles. Prediction, a new reward function is designed to deal with different obstacles, a new experience pool update method is proposed to improve the training efficiency of the neural network, imitation learning is used to pre-train the network, and the convergence speed of the network is improved; The agent implements an efficient dynamic programming method in the complex space.

Description of drawings

Fig. 1 is the method realization flow chart of the embodiment of the present invention;

Fig. 2 is a network structure diagram in an embodiment of the present invention;

Fig. 3 is the simulation result diagram in the embodiment of the present invention;

FIG. 4 is a graph of the total reward of network training in an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

The purpose of the present invention is to provide a dynamic path planning method for a mobile agent in a stage environment. The method designs a new Markov decision process and network structure on the basis of a deep reinforcement learning method, and introduces a social attention mechanism. Add attention scores to dynamic obstacles, use long short-term memory neural network to solve the problem that the dimension of feedforward network is not fixed, build a new reward function to deal with different avoidance situations of dynamic and static obstacles, and propose a new experience pool update method Improve the convergence speed of network training, so that the mobile agent can distinguish between dynamic and static obstacles and predict the movement trend of dynamic obstacles.

In this embodiment, the simulation environment is shown in Figure 3. The range of the planning map is set to 10*10, the starting point of the path planning is (0,-10), the target point is (0,7.5), and the static obstacle position Random distribution, rectangular or square, dynamic obstacles are circles with a radius of 0.5.

A dynamic path planning algorithm for mobile agents in a stage environment, the specific steps are as follows:

1) Establish a simulation environment model of mobile agents and dynamic and static obstacles based on the gym library;

2) Design the Maldives decision-making process, design the state space S, the action space A, the transition probability P, the reward R and the discount factor γ;

3) Design the neural network structure;

4) Using the Optimal Reciprocal Collision Avoidance Algorithm (ORCA), the network parameters are initialized by imitating the learning pre-training 3000 sets; then the network parameters are optimized by training the actual interaction of mobile agents in a simulated environment.

5) The optimal value function is obtained by training the neural network through the adaptive time estimation method (Adam):

V ^* (u _t )=∑γΔt ^·Vpref ·P( _{u t} ,at )

6) Set the optimal strategy by maximizing the cumulative return:

表示下一时刻的联合状态where _ut represents the joint state of the current mobile agent and obstacles, a represents the set of action spaces, γ represents the decay factor, Δt represents the time interval between two actions, Vpref represents the preferred speed, and V ^* represents the optimal Value function, P is the state transition function, R is the reward function;

Represents the joint state at the next moment

7) Select the action at the current moment according to the optimal _strategy until the mobile agent reaches the target.

The above are the preferred embodiments of the present invention. Any changes made according to the technical solutions of the present invention, when the resulting functional effects do not exceed the scope of the technical solutions of the present invention, belong to the protection scope of the present invention.

Claims (6)

1. a mobile agent dynamic path planning method under a stage environment, is characterized in that, comprises the following steps: 1) Establish a simulation environment model of mobile agents and dynamic and static obstacles based on the gym library; 2) Design Markov decision process, Markov decision process is represented by quintuple <S, A, P, R, γ>, design state space S, action space A, transition probability P, reward R and discount factor γ ; 3) Design the neural network structure; 4) Using the best reciprocal collision avoidance algorithm ORCA, the network parameters are initialized by imitation learning pre-training; after the imitation learning is over, the network parameters are optimized by training the actual interaction of the mobile agent in the simulation environment; 5) The optimal value function is obtained by training the neural network through the adaptive time estimation method Adam: V ^* (u _t )=∑γΔt ^·Vpref ·P( _{u t} ,at ) 6) Set the optimal strategy by maximizing cumulative returns:

where _ut represents the joint state of the current mobile agent and the obstacle, at represents the set of action spaces, γ represents the decay factor, _Δt represents the time interval between two actions, Vpref represents the preferred speed, and V ^* represents the Merit function, P is the state transition function, R is the reward function;

Represents the joint state at the next moment; 7) Select the action at the current moment according to the optimal _strategy until the mobile agent reaches the target. 2. the mobile agent dynamic path planning method under a kind of stage environment according to claim 1, is characterized in that, in described step 1), establishes simulation environment model based on gym library, and mobile agent and dynamic obstacle are set up. A circle with a radius of 0.3 m is defined, while static obstacles are defined as a circle with a radius of 0.5 m to 1 m or a quadrilateral with an area of 1 m to 1.5 m. 3. The mobile agent dynamic path planning method under a stage environment according to claim 1, wherein in the step 2), a state space S is set, and the state of the dynamic obstacle is S _D = [P _x ,P _y ,V _x ,V _y ,r,V _pref ], the state of the static obstacle is S _S =[P _x ,P _y ,r], the state of the mobile agent is S _T =[P _x ,P _y ,G _x ,G _y ,V _x ,V _y ,θ,r,V _pref ], joint state u _t =[S _T ,S _S ,S _D ]; where (P _x ,P _y ) is the movement The current position of the agent and the dynamic and static obstacles, (G _x , G _y ) is the position of the set target point, θ is the heading angle of the mobile agent, r is the radius of the mobile agent and the dynamic and static obstacles , V _pref is the preferred speed of the mobile agent, (V _x , V _y ) is the moving speed of the mobile agent and dynamic obstacles; The action space A is the linear velocity and the angular velocity. In order to comply with the dynamic constraints, the angular velocity is divided into 18 equal parts in the interval [-π/4, π/4], and the linear velocity follows an exponential function.

Take 1, 2, 3, 4, and 5 for x to obtain 5 smooth linear velocities; there are 90 action combinations in the action space; The transition probability P transfers the state through the actual interaction of the mobile agent in the simulation environment; the reward R is set as:

where G _{x, y} is the position information of the target point, P _{x, y} is the current position information of the mobile agent, d _s is the distance between the mobile agent and the static obstacle, and d _d is the mobile agent and the dynamic obstacle the distance between. 4. the mobile agent dynamic path planning method under a kind of stage environment according to claim 1, is characterized in that, the neural network structure in described step 3) comprises: input layer, long short term memory neural network module LSTM, Social attention mechanism and output layer; The input layer: the input layer is the joint state u _t = [S _T , S _S , S _D ] in the above steps; the long short-term memory neural network module LSTM: the obstacles around the mobile agent are sorted through the LSTM module, And the output parameters of the network layer are fixed; social attention mechanism: the probability of collision between the mobile agent and surrounding dynamic obstacles is analyzed through the social attention mechanism module, and displayed in the form of scores; output layer: the output layer is based on the network parameters. The weighted linear combination of outputs the optimal value function V ^* (u _t ). 5. the mobile agent dynamic path planning method under a kind of stage environment according to claim 1, is characterized in that, in described step 3), the network operation process is as follows: at first the state information of mobile agent and obstacle is input Enter the neural network structure, and then divide the obstacles into dynamic obstacles and static obstacles according to the state information, input the state of the mobile agent and the state of the dynamic obstacles into the LSTM module, and then into the social attention mechanism module, and then pass The processed state, the obtained interaction feature and the static obstacle state are input into two fully connected layers, and finally they are normalized by the activation function to obtain the optimal value function. 6. the mobile agent dynamic path planning method under a kind of stage environment according to claim 1, is characterized in that, in described step 4), when mobile agent interacts in simulation environment, current state information, action information and The reward information is stored in the experience pool as an experience, and the importance is added to each experience through TD-error. TD-error is a difference between the value function of the action at a certain moment and the optimal value function of the current network. Larger values indicate poorer current experience; by definition: P _t =(|δ _t |+ε) ^α Among them, P _t is the probability of selecting the current experience, α and ε are constants, δ _t is TD-error, and ε is to prevent the experience from being played back after the TD-error is 0; Give different probabilities to experience according to different experiences, thereby increasing the probability of selecting excellent experience and improving the convergence speed of the network; in each episode of interaction, when the mobile agent encounters an obstacle or exceeds the maximum time of a single episode operation, it ends The current set; the experience is then back-propagated through the gradient to update the network parameters.

Images