CN102929281A - Robot k-nearest-neighbor (kNN) path planning method under incomplete perception environment - Google Patents

Abstract

Robot path planning technology under unknown dynamic environment has an important application value. Withal, the invention discloses a robot k-nearest-neighbor (kNN) path planning method under incomplete perception environment. The robot kNN path planning method under the incomplete perception environment comprises partially observable markov decision process (POMDP) model building, POMDP model solution and construction of an iterative learning model. Due to the fact that the iterative model is utilized, the learning capability and the adapting capability to the environment of a robot in the process of path planning are improved, and the path planning performance can be improved.

Description

The kNN of robot paths planning method under a kind of incomplete perception environment

Technical field

The present invention is the robot path planning method under a kind of Unknown Dynamic Environment, relates to robot navigation's technical field, refers more particularly to the Robot Path Planning Algorithm aspect.

Background technology

Along with the development of Robotics, the machine Man's Power improves constantly, and the robot application field also constantly enlarges, and especially in some danger, special or application that the people should not go to, such as nuclear emergency disposal, space operation etc., all needs the machine human intervention.Path planning is the important step of robot navigation's technology, robot path planning's problem is commonly defined as: the starting point of given robot and impact point, in the environment that fixing or moving disorder are arranged, plan a nothing path that bump, that satisfy certain optiaml ciriterion, make robot move to impact point according to this path.Wherein, optiaml ciriterion generally has: minimum, used shortest time, the path of the energy that consumes is the shortest etc.Therefore, a nothing is bumped to seeking, optimal path plays vital effect in the research of paths planning method.

Robot will be in Unknown Dynamic Environment safety, finish path planning reliably, need to possess the ability that can process various uncertain conditions, to improve the adaptability to environment.Therefore, the robot path planning who has an intelligence learning ability seems particularly important.The intensified learning algorithm is used for the robot path planning, it is advantageous that this algorithm is a kind of non-supervisory on-line study method, and does not need the accurate model of environment, and therefore the mobile robot path planning under Dynamic Unknown Environment just comes into one's own in using.As: Reinforcementbased mobile robot navigation in dynamic environment one literary composition of MohammadAbdel Kareem Jaradat is compared with Artificial Potential Field Method to intensified learning, and experimental result shows that the robot path planning method based on the intensified learning algorithm has better applicability.Extended Dyna-QAlgorithm for Path Planning of Mobile Robots one literary composition of Hoang-huu VIET is on Dyna-Q intensified learning algorithm basis, utilize Maximum Likelihood Model to select action and renewal Q value function, improved convergence of algorithm speed.

In these methods, robot finishes path planning under complete observable environment, Wu Feng in " based on the research of the multi-agent system planning problem of decision theory " literary composition from the angle of decision theory DEC-POMDP model, to solve the multiple agent planning problem under the large state space, the method has been considered the incomplete observability of environmental information, but model and the algorithm set up have higher complicacy.

For these problems, the present invention proposes the kNN of the robot paths planning method under a kind of incomplete perception environment.The method adopts the local value iterative learning model based on k arest neighbors classificating thought, the uncertainty of moving under the consideration circumstances not known and the imperfection of environment information acquisition, the adaptability of Robot Path Planning Algorithm in the raising actual environment.

Summary of the invention

The object of the invention is to solve under the Unknown Dynamic Environment, the robot path planning exists the incomplete observability of environmental information, large state space to find the solution difficult problem, with the applicable ability of Effective Raise path planning algorithm.The method adopts the local point value iteration based on the k nearest neighbor classification, replaces the value iterative computation to whole states, finds the solution dimension disaster problem in the POMDP model with effective alleviation, improves simultaneously intensified learning convergence in the path planning process.

In order to achieve the above object, the technical scheme that the present invention takes is: the kNN of the robot paths planning method under a kind of incomplete perception environment may further comprise the steps:

One, the POMDP model is set up:

Adopt grating map that the robot planning environment is divided into little grid.Utilize Grid Method to set up environment map, each little grid cell is corresponding to a state s among the POMDP model state collection S.Behavior aggregate A has east (East), west (West), south (South), north (North) four actions, and robot can be in constantly at next one of 4 adjacent accessible grid cells; Robot arrives dbjective state can obtain return value 0, and other situation return value is-1.In the continuous reciprocal process of machine human and environment, because that the execution of action exists is uncertain, so transition probability is set to correctly carry out the action that optimal strategy is selected with greater probability, to slide to the left and right sides of this action than small probability.

Two, POMDP model solution:

In the POMDP of the robot path planning, robot sensor can not be observed all environmental informations fully.In order to find the solution optimal strategy, action and the complete sequence of observer state, i.e. historical information that robot need to experience.Historical information can utilize conviction state (Belief State) to replace, and conviction state b (s) is a probability distribution on the state set S, and all conviction states form one | and S| ties up matrix.Replace state with the conviction state when finding the solution, the POMDP problem just is converted into the MDP problem based on the conviction state, Action Selection strategy π is converted into by the mapping of conviction state to action: π (b) → a, under optimal strategy π *, the accumulation of discount reward value of all conviction states forms optimal value function Q (b, a), thus can utilize method kNN-Sarsa (λ) the Algorithm for Solving POMDP problem of finding the solution the MDP problem.

Three, the structure of iterative learning model:

After robot arranges reference position and target location, utilize the robot path planning method based on the intensified learning algorithm, a nothing from the reference position to the target location is bumped, optimal path for robot is sought.In the process of seeking optimal path, the grid cell that the present invention may arrive robot is defined as the state s of iterative learning model, action a is defined as concrete direction of action: East, West, South, North, the purpose of Action Selection are the paths of farthest shortening from the reference position to the target location.The iterative model of intensified learning algorithm is given each (s, a) defined a state-working value function Q, it is the conversion accumulation return value that robot obtains when current state selects a certain action to be updated to NextState, the Action Selection strategy is selected optimum action according to this Q value, so that the accumulation return value is maximum.

Four, the list structure used of iterative learning model:

In order to realize the inventive method, need to make up lower list structure:

(1) QTable table

Based on the robot path planning of iterative learning model, at first need to set up state-working value function table Q Table.This Q Table shows | and S| is capable | the two-dimensional matrix of A| row (| S| is the number of elements of state set S, | A| is the number of elements of behavior aggregate A), it has stored accumulation return value corresponding to each state-action, i.e. Q (s, the cumulative maximum return value when a) being updated to state s for the optimum action of selection a.

(2) transfer function table T

Transfer function T:S * A → ∏ (S), the variation of action is described on the impact of ambient condition, robot elemental motion has four of the four corners of the world, need set up four transfer function table: T_E, T_W, T_S, T_N, be respectively and select after east (East), west (West), south (South), north (North) action by state s _tBe converted to state s _T+1Probability.

(3) observation function table O

Robot makes a strategic decision according to the information that self-contained sensor is surveyed, and observation function O:S * A → ∏ (Z) expression robot is at current state s _tBe transformed into new state s behind the execution action a _T+1Obtain the probability of observer state z, namely set up one based on the probability distribution table of observed reading.

(4) return value table R

By judging whether robot arrives the target location and come the detection machine people whether to finish route searching one time, when robot arrives the target location, provide the repayment of 0 value, otherwise provide the repayment of-1 value.Repayment function R:S * A → R can obtain the higher return value of environment if describe certain action, and the trend that produces this action in the searching path process so afterwards will be strengthened, otherwise the trend that produces this action will weaken.

(c2) iterative learning process:

The iterative learning process is comprised of following steps:

Step1: initialization

Init state-working value function table Q Table, make Q (s, a)=0, qualification mark e (s, a)=0, initial conviction state b (s)=0.001076, parameter k=5 (5 neighbours are selected in expression), study factor-alpha=0.95, discount factor γ=0.99, λ=0.95, wherein γ λ represents qualification mark e according to probability γ λ decay, the probable value of random action is selected in ε=0.001 expression.

Step2: obtain current state s _tAnd the conviction state set B of k arest neighbors state

1) with the reference position of robot as current state s _t

2) calculate the state set knn that k state of Euclidean distance minimum consists of among st and the state set S;

3) the conviction state value b of each state among the computing mode collection knn _t(s): b _t(s)=1/ (| S|).

Step3: obtain conviction state value function

Conviction state b _t(s) corresponding value function is calculated by following formula:

$Q(b,a)=\underset{i\∈\mathrm{knn}}{\mathrm{\Σ}}Q(i,a)b\left(i\right)$

Be Q (s, a) current state s in the table _tK arest neighbors collection knn in all state value function Q (i, a) with conviction state b (i) sum of products.

Step4: select action

Move according to ε-greedy Action Selection policy selection:

$\mathrm{\&pi;}\left(a\right)=\left\{\begin{array}{cc}\arg {\max }_a\underset{s\&Element;S}{\mathrm{\&Sigma;}}Q(s,a)b\left(s\right)& (U\&GreaterEqual;\mathrm{\&epsiv;})\\ \mathrm{rand}\left(a\right)& (U<\mathrm{\&epsiv;})\end{array}\right.,$

Wherein, U is equally distributed random number between (0,1), and probable value ε speed with 0.99 times in each learning cycle (Episode) decays, namely starting stage of learning cycle with larger probability selection random action, avoid algorithm to be absorbed in local optimum; Along with the increase of Q value effective information, ε reduces gradually, has guaranteed Algorithm Convergence.

Step5: execution action

Execution action s _tAfter be transformed into new state s _T+1, obtain simultaneously observer state z and return value R.

Step6: calculate return value R

Robot has carried out action a _tRear arrival reposition judges whether this position is the target location, if not, then obtain return value-1, carry out Step7; Otherwise, obtain return value 0, carry out Step10.

Step7: obtain NextState s _T+1Corresponding conviction state set B '

1) calculates s _T+1The state set knn ' that consists of with k state of Euclidean distance minimum among the state set s;

2) the conviction state value b of each state among the computing mode collection knn ' _T+1(s '):

$b_{t+1}\left(s^{\&prime;}\right)=\frac{O(s^{\&prime;},a,z)\underset{s\&Element;S}{\mathrm{\&Sigma;}}T(s,a,s^{\&prime;})b_t\left(s\right)}{\underset{s^{\&prime;}\&Element;S}{\mathrm{\&Sigma;}}O(s^{\&prime;},a,z)\underset{s\&Element;S}{\mathrm{\&Sigma;}}T(s,a,s^{\&prime;})b_t\left(s\right)}.$

3) repeat Step3-Step4.

Step8: upgrade

1) the qualification mark is pressed the following formula definition:

$e(s,j)=\left\{\begin{array}{cc}b\left(s\right)& j=a,\\ 0& j\&NotEqual;a.\end{array}\right.(s\&Element;\mathrm{knn}).$

2) to the state of all k arest neighbors states of robot status-working value function Q (i, a) upgrade:

${\mathrm{\&Delta;q}}_a\left(s\right)=\mathrm{\&alpha;}(r+\mathrm{\&gamma;}\underset{a^{\&prime;}}{\max }Q(b^{\&prime;},a^{\&prime;})-Q(s,a)b\left(s\right))e(s,a),(s\&Element;\mathrm{knn})$

Q _t+1(s，a)＝Q _t(s，a)+Δq _a(s)(s∈knn)

3)s _t＝s _t+1，a _t＝a _t+1，knn＝knn′，e _t+1＝γλe _t，b _t(s)＝b _t+1(s′)。

Step9: turn to Step5.

Step10: one time the iterative learning process finishes, and forwards Step2 to and enters next iterative learning process, until the Q value converges to optimum or near-optimization.

Claims (1)

1. the kNN of robot paths planning method that is applicable under the incomplete perception environment is characterized in that: the POMDP model is set up, and POMDP model solution, iterative model make up three steps:

(a) the POMDP model is set up: adopt grating map that the robot planning environment is divided into little grid, a state s among the corresponding POMDP model state of each the little grid cell collection S, behavior aggregate A has east (East), west (West), south (South), north (North) four actions, robot can be in one of adjacent 4 accessible grid cells constantly at next, robot arrives dbjective state can obtain return value 0, other situation return value is-1, in robot and environmental interaction, transition probability is set to correctly carry out the action that optimal strategy is selected with greater probability, to slide to the left and right sides of this action than small probability;

(b) POMDP model solution: optimal strategy is found the solution by robot needs the action experience and the historical information of observer state, historical information can utilize conviction state (Belief State) to replace, conviction state b (s) is a probability distribution on the state set S, replace state with the conviction state when finding the solution, the POMDP problem is converted into the MDP problem based on the conviction state, Action Selection strategy π is converted into by the mapping of conviction state to action: π (b) → a, under optimal strategy π *, the accumulation of discount reward value composition optimal value function Q of all conviction states (b, a);

(c) iterative model makes up: after robot arranges reference position and target location, utilization is based on the robot path planning method of intensified learning algorithm, the intensified learning algorithm is given each (s, a) defined a state-working value function Q, it is the conversion accumulation return value that robot obtains when current state selects a certain action to be updated to NextState, the Action Selection strategy is selected optimum action according to this Q value, so that the accumulation return value is maximum, the concrete steps of Iterative Algorithm are as follows:

Step1: initialization

Init state-working value function table Q Table, to Q (s, a), qualification mark e (s, a), initial conviction state b (s), parameter k, study factor-alpha, and random action selects probable value ε to compose initial value,

Step 2: obtain current state s _tAnd the conviction state set B of k arest neighbors state

1) with the reference position of robot as current state s _t

2) calculate s _tThe state set knn that consists of with k state of Euclidean distance minimum among the state set S;

3) the conviction state value b of each state among the computing mode collection knn _t(s): b _t(s)=1/ (| S|),

Step3: obtain conviction state value function

Conviction state b _t(s) corresponding value function is calculated by following formula:

$Q(b,a)=\underset{i\&Element;\mathrm{knn}}{\mathrm{\&Sigma;}}Q(i,a)b\left(i\right)$

Be Q (s, a) current state s in the table _tK arest neighbors collection knn in all state value function Q (i, a) with conviction state b (i) sum of products,

Step4: select action

Move according to ε-greedy Action Selection policy selection:

$\mathrm{\&pi;}\left(a\right)=\left\{\begin{array}{cc}\arg {\max }_a\underset{s\&Element;S}{\mathrm{\&Sigma;}}Q(s,a)b\left(s\right)& (U\&GreaterEqual;\mathrm{\&epsiv;})\\ \mathrm{rand}\left(a\right)& (U<\mathrm{\&epsiv;})\end{array}\right.$

Wherein, U is equally distributed random number between (0,1), and probable value ε speed with 0.99 times in each learning cycle (Episode) decays, namely starting stage of learning cycle with larger probability selection random action, avoid algorithm to be absorbed in local optimum; Along with the increase of Q value effective information, ε reduces gradually, has guaranteed Algorithm Convergence,

Step5: execution action

Execution action a _tAfter be transformed into new state s _T+1, obtain simultaneously observer state z and return value R,

Step6: calculate return value R

Robot has carried out action a _tRear arrival reposition judges whether this position is the target location, if not, then obtain return value-1, carry out Step7; Otherwise, obtain return value 0, carry out Step10,

Step 7: obtain NextState s _T+1Corresponding conviction state set B '

1) calculates s _T+1The state set knn ' that consists of with k state of Euclidean distance minimum among the state set S,

2) the conviction state value b of each state among the computing mode collection knn ' _T+1(s '):

$b_{t+1}\left(s^{\&prime;}\right)=\frac{O(s^{\&prime;},a,z)\underset{s\&Element;S}{\mathrm{\&Sigma;}}T(s,a,s^{\&prime;})b_t\left(s\right)}{\underset{s^{\&prime;}\&Element;S}{\mathrm{\&Sigma;}}O(s^{\&prime;},a,z)\underset{s\&Element;S}{\mathrm{\&Sigma;}}T(s,a,s^{\&prime;})b_t\left(s\right)}.$

3) repeat Step3-Step4,

Step8: upgrade

1) the qualification mark is pressed the following formula definition:

$e(s,j)=\left\{\begin{array}{cc}b\left(s\right)& j=a,\\ 0& j\&NotEqual;a.\end{array}\right.(s\&Element;\mathrm{knn}).$

2) to the state of all k arest neighbors states of robot status-working value function Q (i, a) upgrade:

${\mathrm{\&Delta;q}}_a\left(s\right)=\mathrm{\&alpha;}(r+\mathrm{\&gamma;}\underset{a^{\&prime;}}{\max }Q(b^{\&prime;},a^{\&prime;})-Q(s,a)b\left(s\right))e(s,a),(s\&Element;\mathrm{knn})$

Q _t+1(s，a)＝Q _t(s，a)+Δq _a(s)(s∈knn)

3)s _t＝s _t+1，a _t＝a _t+1，knn＝knn′，e _t+1＝γλe _t，b _t(s)＝b _t+1(s′)，

Step9: turn to Step5,

Step10: one time the iterative learning process finishes, and forwards Step 2 to and enters next iterative learning process, until the Q value converges to optimum or near-optimization.