CN102929281A - Robot k-nearest-neighbor (kNN) path planning method under incomplete perception environment - Google Patents

Abstract

Robot path planning technology in an unknown dynamic environment has important application value. For this, the present invention discloses a robot kNN path planning method in an incomplete perception environment, which mainly includes: POMDP model establishment, POMDP model solution, iterative learning model Construct. The invention utilizes the iterative model to improve the learning and adaptability of the robot to the environment during path planning, and can improve the path planning performance.

Description

A robot kNN path planning method in incomplete perception environment

technical field

The invention relates to a robot path planning method in an unknown dynamic environment, relates to the technical field of robot navigation, and in particular relates to a robot path planning algorithm.

Background technique

With the development of robot technology, the ability of robots is constantly improving, and the application fields of robots are also expanding, especially in some dangerous, special or unsuitable application fields, such as nuclear emergency response, space operations, etc., all require the intervention of robots. Path planning is an important part of robot navigation technology. The problem of robot path planning is generally defined as: given the starting point and target point of the robot, in an environment with fixed or moving obstacles, plan a non-collision path that satisfies certain optimal criteria. Path, so that the robot moves to the target point according to the path. Among them, the optimal criteria generally include: the least energy consumed, the shortest time used, the shortest path length, and so on. Therefore, the research on path planning methods plays a vital role in finding a collision-free and optimal path.

To complete path planning safely and reliably in an unknown dynamic environment, a robot needs to be able to handle various uncertain situations to improve its adaptability to the environment. Therefore, robot path planning with intelligent learning ability is particularly important. The reinforcement learning algorithm is used for robot path planning. Its advantage is that the algorithm is an unsupervised online learning method and does not require an accurate model of the environment. Therefore, it is being paid attention to in the application of mobile robot path planning in dynamic unknown environments. For example: Reinforcement based mobile robot navigation in dynamic environment by Mohammad Abdel Kareem Jaradat compares reinforcement learning with artificial potential field method. The experimental results show that the robot path planning method based on reinforcement learning algorithm has better applicability. Hoang-huu VIET's Extended Dyna-QAlgorithm for Path Planning of Mobile Robots is based on the Dyna-Q reinforcement learning algorithm, using the maximum likelihood model to select actions and update the Q value function, which improves the convergence speed of the algorithm.

In these methods, the robot completes path planning in a fully observable environment. Wu Feng used the DEC-POMDP model from the perspective of decision theory in his article "Research on Multi-Agent System Planning Problems Based on Decision Theory" to solve large-scale For the multi-agent programming problem in the state space, this method takes into account the incomplete observability of environmental information, but the established model and algorithm have high complexity.

In view of these problems, the present invention proposes a robot kNN path planning method in an incomplete perception environment. This method adopts the local value iterative learning model based on the idea of k-nearest neighbor classification, considers the uncertainty of actions in unknown environments and the incompleteness of environmental information acquisition, and improves the adaptability of robot path planning algorithms in actual environments.

Contents of the invention

The purpose of the present invention is to solve the problem of incomplete observability of environmental information and difficult solution in large state space in robot path planning in an unknown dynamic environment, so as to effectively improve the applicability of path planning algorithms. This method uses local point value iteration based on the k-nearest neighbor classification method to replace the iterative calculation of the value of all states to effectively alleviate the dimensionality disaster problem in solving the POMDP model, and at the same time improve the convergence of the reinforcement learning algorithm in the path planning process.

In order to achieve the above object, the technical solution adopted by the present invention is: a robot kNN path planning method under an incomplete perception environment, comprising the following steps:

1. Establishment of POMDP model:

A grid map is used to divide the robot planning environment into small grids. The environment graph is established by using the grid method, and each small grid unit corresponds to a state s in the state set S of the POMDP model. Action set A has four actions: East (East), West (West), South (South), and North (North). The robot can be in one of the four adjacent barrier-free grid cells at the next moment; the robot reaches the target state The return value can be 0, and the return value in other cases is -1. During the continuous interaction between the robot and the environment, due to the uncertainty in the execution of the action, the transition probability is set to correctly execute the action selected by the optimal strategy with a high probability, and slide to the left and right sides of the action with a small probability.

2. POMDP model solution:

In robot POMDP path planning, robot sensors cannot fully observe all environmental information. In order to solve the optimal policy, the robot needs the complete sequence of experienced actions and observed states, that is, historical information. Historical information can be replaced by a belief state (Belief State). The belief state b(s) is a probability distribution on the state set S, and all belief states form a |S|-dimensional matrix. When solving the problem, the belief state is used instead of the state, and the POMDP problem is transformed into an MDP problem based on the belief state. The action selection strategy π is transformed into a mapping from the belief state to the action: π(b)→a. Under the optimal strategy π*, all The discounted cumulative reward value of the belief state constitutes the optimal value function Q(b, a), so that the kNN-Sarsa(λ) algorithm can be used to solve the MDP problem to solve the POMDP problem.

3. Construction of iterative learning model:

After the robot sets the starting position and the target position, the robot path planning method based on the reinforcement learning algorithm is used to find a collision-free and optimal path for the robot from the starting position to the target position. In the process of finding the optimal path, the present invention defines the grid unit that the robot may reach as the state s of the iterative learning model, and the action a is defined as the specific action direction: east, west, south, north, and the purpose of action selection is to maximize The degree of shortening the path from the starting position to the target position. The iterative model of the reinforcement learning algorithm defines a state-action value function Q for each (s, a), that is, the converted cumulative return value obtained when the robot selects an action in the current state to update to the next state, and the action selection strategy is based on This Q-value selects the optimal action so that the cumulative reward value is maximized.

4. The table structure used in the iterative learning model:

In order to realize the method of the present invention, it is necessary to construct the following table structure:

(1) QTable table

Robot path planning based on the iterative learning model first needs to establish a state-action value function table Q Table. The Q Table is a two-dimensional matrix of |S|rows|A|columns (|S| is the number of elements of the state set S, |A| is the number of elements of the action set A), which stores each state-action correspondence The cumulative reward value of , that is, Q(s, a) is the maximum cumulative reward value when the optimal action a is updated to state s.

(2) Conversion function table T

The conversion function T: S×A→∏(S), describes the impact of the change of action on the state of the environment. There are four basic movements of the robot: east, west, south, north, and four conversion function tables need to be established: T_E, T_W, T_S, T_N, respectively The probability of transitioning from state s _t to state s _t+1 after choosing East, West, South, and North actions.

(3) Observation function table O

The robot makes decisions based on the information detected by its own sensors. The observation function O: S×A→∏(Z) represents the probability that the robot transitions to the new state st ₊₁ to obtain the observed state z after executing action a in the current state _st . That is, a table of probability distributions based on observations is established.

(4) Return value table R

By judging whether the robot has reached the target position, it is detected whether the robot has completed a path search. When the robot reaches the target position, a return value of 0 is given, otherwise a return value of -1 is given. Reward function R: S×A→R, describing that if an action can obtain a higher reward value from the environment, then the trend of generating this action will be strengthened in the process of finding the path in the future, otherwise the trend of generating this action will be weakened .

(c2) Iterative learning process:

The iterative learning process consists of the following steps:

Step1: Initialize

Initialize the state-action value function table Q Table, set Q(s, a) = 0, qualification trace e(s, a) = 0, initial belief state b(s) = 0.001076, parameter k = 5 (indicates that 5 neighbors are selected ), learning factor α=0.95, discount factor γ=0.99, λ=0.95, where γλ indicates that eligibility trace e decays according to probability γλ, and ε=0.001 indicates the probability value of choosing a random action.

Step2: Obtain the belief state set B of the current state s _t and its k nearest neighbor states

1) Take the initial position of the robot as the current state s _t ;

2) Calculate the state set knn consisting of k states with the smallest Euclidean distance between st and the state set S;

3) Calculate the belief state value b _t (s) of each state in the state set knn: b _t (s)=1/(|S|).

Step3: Get the belief state value function

The value function corresponding to the belief state b _t (s) is calculated by the following formula:

$\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; knn</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>$

That is, the sum of the products of all state value functions Q(i, a) and belief state b(i) in the k-nearest neighbor set knn of the current state s _t in the Q(s, a) table.

Step4: Select an action

Actions are selected according to the ε-greedy action selection strategy:

$\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; arg</span>}{\mathrm{<span\; class="notranslate">\; max</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; rand</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; )</span>\end{array}\right.<span\; class="notranslate">\; ,</span>$

Among them, U is a uniformly distributed random number between (0, 1), and the probability value ε decays at a rate of 0.99 times in each learning cycle (Episode), that is, in the initial stage of the learning cycle, a random number is selected with a greater probability. action to prevent the algorithm from falling into local optimum; as the effective information of the Q value increases, ε gradually decreases, which ensures the convergence of the algorithm.

Step5: Execute the action

After executing the action s _t , switch to the new state s _t+1 , and obtain the observed state z and the reward value R at the same time.

Step6: Calculate the return value R

After the robot executes the action a _t and arrives at a new position, judge whether the position is the target position, if not, get a reward value of -1, and execute Step7; otherwise, get a reward value of 0, and execute Step10.

Step7: Obtain the belief state set B′ corresponding to the next state s _t+1

1) Calculate the state set knn′ consisting of k states with the smallest Euclidean distance between s _t+1 and the state set s;

2) Calculate the belief state value b _t+1 (s') of each state in the state set knn':

${\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>}{\underset{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; .</span>$

3) Repeat Step3-Step4.

Step8: update

1) The qualification trace is defined as follows:

$\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; .</span>\end{array}\right.<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; knn</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

2) Update the state-action value function Q(i, a) of all k-nearest neighbor states in the state of the robot:

${\mathrm{<span\; class="notranslate">\; \&Delta;q</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}\underset{{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; knn</span>}<span\; class="notranslate">\; )</span>$

Q _t+1 (s, a)=Q _t (s, a)+Δq _a (s)(s∈knn)

3) s _t =s _t+1 , a _t = _at+1 , knn=knn', e _t+1 =γλe _t , b _t (s)=b _t+1 (s').

Step9: Turn to Step5.

Step10: An iterative learning process ends, go to Step2 to enter the next iterative learning process, until the Q value converges to the optimal or nearly optimal.

Claims (1)

1. A robot kNN path planning method applicable to an incomplete perception environment is characterized in that: the POMDP model is set up, the POMDP model is solved, and the iterative model is constructed in three steps: (a) POMDP model establishment: use the grid map to divide the robot planning environment into small grids, each small grid unit corresponds to a state s in the state set S of the POMDP model, and the action set A includes East (East), West ( West), South, North four actions, the robot can be in one of the four adjacent barrier-free grid units at the next moment, the robot can get a return value of 0 when it reaches the target state, and the return value in other cases is -1, in the interaction between the robot and the environment, the transition probability is set to correctly execute the action selected by the optimal strategy with a high probability, and slide to the left and right sides of the action with a low probability; (b) Solving the POMDP model: The robot needs the historical information of the actions experienced and the observed state to solve the optimal strategy. The historical information can be replaced by the belief state (Belief State). The belief state b(s) is a state set S Probability distribution, the belief state is used to replace the state when solving, the POMDP problem is transformed into an MDP problem based on the belief state, and the action selection strategy π is transformed into a mapping from the belief state to the action: π(b)→a, under the optimal strategy π* , the discounted cumulative reward values of all belief states form the optimal value function Q(b, a); (c) Iterative model construction: After the robot sets the starting position and the target position, the robot path planning method based on the reinforcement learning algorithm is used. The reinforcement learning algorithm defines a state-action value function Q for each (s, a), namely The converted cumulative return value obtained when the robot selects an action in the current state to update to the next state. The action selection strategy selects the optimal action based on the Q value to maximize the cumulative return value. The specific steps of the iterative learning algorithm are as follows: Step1: Initialize Initialize the state-action value function table Q Table, assign Q(s, a), qualification trace e(s, a), initial belief state b(s), parameter k, learning factor α, and random action selection probability value ε initial value, Step 2: Obtain the belief state set B of the current state s _t and its k nearest neighbor states 1) Take the initial position of the robot as the current state s _t ; 2) Calculate the state set knn consisting of k states with the smallest Euclidean distance between s _t and the state set S; 3) Calculate the belief state value b _t (s) of each state in the state set knn: b _t (s) = 1/(|S|), Step3: Get the belief state value function The value function corresponding to the belief state b _t (s) is calculated by the following formula: $\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; knn</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>$ That is, the sum of the products of all state value functions Q(i, a) and belief state b(i) in the k-nearest neighbor set knn of the current state s _t in the Q(s, a) table, Step4: Select an action Actions are selected according to the ε-greedy action selection strategy: $\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; arg</span>}{\mathrm{<span\; class="notranslate">\; max</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; rand</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; )</span>\end{array}\right.$ Among them, U is a uniformly distributed random number between (0, 1), and the probability value ε decays at a rate of 0.99 times in each learning cycle (Episode), that is, in the initial stage of the learning cycle, a random number is selected with a greater probability. action, to avoid the algorithm from falling into local optimum; with the increase of the effective information of Q value, ε gradually decreases, which ensures the convergence of the algorithm. Step5: Execute the action After executing the action a _t, switch to the new state s _t+1 , and obtain the observed state z and the return value R at the same time, Step6: Calculate the return value R After the robot performs the action a _t and arrives at a new position, judge whether the position is the target position, if not, get a reward value of -1, and execute Step7; otherwise, get a reward value of 0, and execute Step10, Step 7: Obtain the belief state set B′ corresponding to the next state s _t+1 1) Calculate the state set knn′ consisting of k states with the smallest Euclidean distance between _st+1 and the state set S, 2) Calculate the belief state value b _t+1 (s') of each state in the state set knn': ${\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>}{\underset{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; .</span>$ 3) Repeat Step3-Step4, Step8: update 1) The qualification trace is defined as follows: $\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; .</span>\end{array}\right.<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; knn</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$ 2) Update the state-action value function Q(i, a) of all k-nearest neighbor states in the state of the robot: ${\mathrm{<span\; class="notranslate">\; \&Delta;q</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}\underset{{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; knn</span>}<span\; class="notranslate">\; )</span>$ Q _t+1 (s, a)=Q _t (s, a)+Δq _a (s)(s∈knn) 3) s _t =s _t+1 , a _t =a _t+1 , knn=knn', e _t+1 =γλe _t , b _t (s)=b _t+1 (s'), Step9: Turn to Step5, Step10: After an iterative learning process ends, go to Step 2 to enter the next iterative learning process until the Q value converges to the optimal or nearly optimal.