CN102402712A - Robot reinforced learning initialization method based on neural network - Google Patents

Abstract

The invention provides a robot reinforced learning initialization method based on a neural network. The neural network has the same topological structure as a robot working space, and each neuron corresponds to a discrete state of a state space. The method comprises the following steps of: evolving the neural network according to the known partial environmental information till reaching a balance state, wherein at the moment, the output value of each neuron represents maximum cumulative return acquired when the corresponding state follows the optimal strategy; defining the initial value of a Q function as the sum of the immediate return of the current state and the maximum converted cumulative return acquired when the subsequent state follows the optimal strategy; and the mapping the known environmental information into the initial value of the Q function by the neural network. Therefore, the prior knowledge is fused into a robot learning system, and the learning capacity of the robot at the initial stage of reinforced learning is improved; and compared with the conventional Q learning algorithm, the method has the advantages of effectively improving the learning efficiency of the initial stage and increasing the algorithm convergence speed.

Description

Robot intensified learning initial method based on neural network

Technical field

The present invention relates to priori is dissolved in mobile robot's learning system, the method to Q value initialization in the robot intensified learning process belongs to the machine learning techniques field.

Background technology

Continuous expansion along with the robot application field; The task that robot faced also becomes increasingly complex; Although the researchist can carry out pre-programmed to the repetition behavior that robot possibly carry out under a lot of situation; But for realizing that whole expected behavior carries out the behavior design more and more difficult that becomes, the designer often can not make rational prediction to all behaviors of robot in advance.Therefore, autonomous robot that can the perception environment must obtain new behavior through the mutual on-line study with environment, makes robot can select to reach the optimum action of target based on specific task.

The intensified learning utilization is similar to the method for the trial and error (trial-and-error) among the human thinking and finds the optimum behavior strategy, is showing the better learning performance aspect the robot behavior study at present.The Q learning algorithm is a kind of reinforcement Learning Method of finding the solution INFORMATION OF INCOMPLETE Markov decision problem; Repayment immediately according to ambient condition and last step study acquisition; The mapping policy of modification from state to action; So that the accumulation return value that behavior obtains is maximum, thereby obtain the optimum behavior strategy from environment.Standard Q learning algorithm is Q or random number with Q value initialization generally, and to the priori of environment, the starting stage of study can only not selected action randomly in robot, and therefore, algorithm the convergence speed is slower in complex environment.In order to improve algorithm the convergence speed, the researchist has proposed the method for many improvement Q study, improves the algorithm learning efficiency, improves learning performance.

Generally, quicken Q studying convergence method of velocity and mainly comprise two aspects: a kind of method is the suitable repayment function of design, and another kind of method is reasonable initialization Q function.At present, the researchist has proposed many improved Q learning algorithms, makes robot in the process of intensified learning, can obtain more effectively to repay, and mainly comprises: related Q learning algorithm, inertia Q learning algorithm, Bayes Q learning algorithm etc.Its fundamental purpose will be dissolved in the repayment function for the valuable implicit information of robot exactly, thus the accelerating algorithm speed of convergence.Related Q study compares current repayment and constantly repayment immediately in the past, selects the bigger action of return value, can improve the learning ability of system through related method for returning, reduces obtaining the needed iteration step number of optimal value.The target of inertia Q study provides the method that a kind of predicted state is repaid immediately; Utilize the message delay principle in the learning process; Under the situation of necessity, new target is predicted that the expectation repayment of each situation of action comparer inspection selects the maximum action of expectation repayment to carry out then.Bayes Q study utilizes probability distribution to describe the uncertainty estimation of robotary-action to the Q value; Need to consider the distribution of previous moment Q value in the learning process; And utilize robot learning to experience previous distribution is upgraded; Utilize Bayes's variable to represent the cumulative maximum repayment of current state, bayes method has improved the performance of Q study from having improved the exploration strategy of Q study in essence.

Because enhanced signal all is the scalar value that is obtained by the state value function calculation in the standard intensified learning, can't people's knowledge form and behavior pattern be dissolved in the learning system.And in the robot learning process, the people often has the experience and knowledge of association area, and therefore, the cognition with the people in learning process feeds back to robot with intelligent form with enhanced signal, can reduce the state space dimension, accelerates algorithm the convergence speed.To the problem that the standard intensified learning exists in the man-machine interaction process, Thomaz etc. provide the external strengthening signal by the people in real time in robot intensified learning process, and the people is according to self experience adjustments training behavior, and guided robot carries out perspective exploration.Arsenio has proposed a kind of training data to be carried out learning strategy online, automatic mark, in the man-machine interaction process, obtains training data through triggering specific incident, thereby will the person of teaching be embedded into the backfeed loop of intensified learning.Mirza etc. have proposed the architecture based on interactive history, and robot can utilize and carry out social mutual historical experience with the people and carry out intensified learning, make robot in the easy game that carries out with the people, obtain suitable behavior gradually.

Another kind improves the method for Q learning algorithm performance to be dissolved into priori in the learning system exactly, and the Q value is carried out initialization.At present, the Q value is carried out initialized method and mainly comprise approximate function method, fuzzy rule method, potential function method etc.The approximate function method utilizes intelligence systems such as neural network to approach optimal value function, and priori is become the repayment functional value, robot is learnt on the subclass of whole state space, thereby can be accelerated algorithm the convergence speed.The fuzzy rule method is set up fuzzy rule base according to initial environment information, utilizes fuzzy logic that the Q value is carried out initialization then.The fuzzy rule that utilizes this method to set up is all set according to environmental information is artificial, often can not reflect the ambient condition of robot objectively, causes algorithm unstable.Potential function method defines corresponding state potential function at whole state space; Every bit potential energy value is corresponding to a certain discrete state value in the state space; Utilize the state potential function that the Q value is carried out initialization then, the Q value of learning system can be expressed as the change amount that initial value adds each iteration.In the middle of the various actions of robot; A series of code of conduct must be observed by robot; Robot emerges corresponding behavior and intelligence through cognitive with reciprocation, and intensified learning Q value initialization of robot will become corresponding robot behavior with priori exactly.Therefore; How to obtain the regularization expression-form of priori; Particularly realize the machine inference of domain experts'experiences and general knowledge, people's cognition and intelligence are converted into the calculating of machine and human-machine intelligence's integration technology of reasoning is that robot behavior is learnt urgent problem.

Summary of the invention

Deficiency to existing robot intensified learning Study on Technology present situation and existence; The present invention proposes a kind of learning efficiency that can effectively improve the starting stage, accelerates the robot intensified learning initial method based on neural network of speed of convergence; This method can be dissolved into priori in the learning system through Q value initialization; Study to the robot initial stage is optimized, thereby a learning foundation preferably is provided for robot.

Robot intensified learning initial method based on neural network of the present invention, neural network has identical topological structure with the robot working space, and each neuron is corresponding to a discrete state in the state space.At first neural network is developed according to known component environment information; Up to reaching equilibrium state; At this moment each neuron output value is just represented the cumulative maximum repayment that its corresponding states can obtain, and then the repayment immediately of current state is added that follow-up state follows the maximum conversion accumulation repayment (commutation factor is multiply by in the cumulative maximum repayment) that optimal strategy obtains, Q (s that can be right to all state-actions; A) set rational initial value; Can priori be dissolved in the learning system through Q value initialization, the study in robot initial stage is optimized, thereby a learning foundation preferably is provided for robot; Specifically may further comprise the steps:

(1) sets up neural network model

Neutral net has identical topological structure with the structure space of robot work, each neuron only with its local neighborhood in neuron be connected, type of attachment is all identical; Connecting power all equates; The propagation of information is two-way between the neuron, and neutral net has the architecture of highly-parallel, and each neuron is corresponding to one of the robot working space discrete state; Whole neutral net is formed two dimensional topology by N * N neuron; The state in its neighborhood is upgraded in neutral net input based on each discrete state in evolutionary process, reaches poised state up to neutral net, when reaching poised state; Neuron output value just forms a single-peaked curved surface in the neutral net, and the value of every bit is just represented the cumulative maximum repayment that institute's corresponding states can obtain on the curved surface;

(2) design repayment function

In learning process; Robot can move on 4 directions; Select 4 actions up and down in free position, action is selected based on current state by robot, and the repayment immediately that obtains if this action makes robot arrive target is 1; If bumping, robot and barrier or other machines people obtain immediately that repayment is-0.2, if robot moves then the repayment immediately that obtains is-0.1 at free space;

(3) calculate cumulative maximum repayment initial value

When neural network reaches equilibrium state, define the cumulative maximum repayment V of the corresponding state of each neuron ^* _Init(s _i) equal this neuron output value x _i, its relation formula is following:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\←x_i,$

In the formula, x _iI neuronic output valve when reaching equilibrium state for neural network, V ^* _Init(s _i) be from state s _iSet out and follow the obtainable cumulative maximum repayment of optimal strategy institute;

(4) Q value initialization

Q (s _i, initial value a) is defined as at state s _iFollowing select action a obtained repay the maximum conversion accumulation repayment that r adds follow-up state immediately:

$Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\γ}{V}_{\mathrm{Init}}^{*}\left(s_j\right)$

In the formula, s _jFor robot at state s _iThe following action follow-up state that a produced, the Q of selecting _Init(s _i, be that state-action is to (s a) _i, initial Q value a); γ is a commutation factor, selects γ=0.95;

(5) based on the robot intensified learning step of neural network

(a) neural network develops according to initialization context information, up to reaching equilibrium state;

(b) state s _iThe initial value of the cumulative maximum repayment that obtains is defined as neuron output value x _i, relation formula is following:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\←x_i$

(c) according to following regular initialization Q value: $Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\γ}{V}_{\mathrm{Init}}^{*}\left(s_j\right),$

(d) observe current state s _t

(e) continue in complex environment, to explore, at current state s _tUnder select one the action a _tAnd carry out, ambient condition is updated to new state s ' _t, and r is repaid in reception immediately _t

(f) observe new state s ' _t

(g) upgrade list item Q (s according to following formula _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\α}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\α}}_t(r_t+\mathrm{\γ}\underset{a_t^{\′}}{\arg }\max Q_{t-1}(s_t^{\′},a_t^{\′}))$

In the formula, α _tBe learning rate, span is (0,1), and value is 0.5 usually, and decays with learning process; Q _T-1(s _t, a _t) and Q _T-1(s ' _t, a ' _t) state of being respectively-action is to (s _t, a _t) and (s ' _t, a ' _t) at t-1 value constantly, a ' _tFor at new state s ' _tThe action of following selection;

(h) judge whether robot has arrived the maximum study number of times that target or learning system have reached setting; The maximum study number of times of setting should guarantee that learning system restrains in maximum study number of times; If both satisfy one of which, then study finishes, and continues study otherwise turn back to step (d).

The present invention becomes Q function initial value through neural network with the known environment information mapping; Thereby priori is dissolved in the robotics learning system; Improved the learning ability of robot in intensified learning starting stage; Compare with the traditional Q-learning algorithm, can effectively improve the learning efficiency of starting stage, accelerate algorithm the convergence speed.

Description of drawings

Fig. 1 is an i neuron neighbour structure synoptic diagram.

Fig. 2 is a neuron output value synoptic diagram in the robot target vertex neighborhood.

Fig. 3 is cumulative maximum repayment initial value V* _Init(s, a) synoptic diagram.

Fig. 4 is neural network neuron output value synoptic diagram when reaching equilibrium state.

Fig. 5 is the robot planning path synoptic diagram that existing Q study obtains.

Fig. 6 is existing Q learning algorithm convergence process synoptic diagram.

Fig. 7 is a robot planning of the present invention path synoptic diagram.

Fig. 8 is a studying convergence process synoptic diagram of the present invention.

Embodiment

The present invention is based on neural network the robot intensified learning is carried out initialization; Neural network has identical topological structure with the robot working space; When neural network reaches equilibrium state; Neuron output value is represented the cumulative maximum repayment of corresponding states, utilizes the repayment immediately of current state and the maximum conversion of follow-up state to accumulate the initial value that repayment obtains the Q function.Can priori be dissolved in the learning system through Q value initialization, the study in robot initial stage is optimized, thereby a learning foundation preferably is provided for robot; Specifically may further comprise the steps:

1 neural network model

Neural network has identical topological structure with the robot working space, and each neuron is corresponding to robot working space's a discrete state.All neurons all only with its local neighborhood in neuron be connected, and its type of attachment is all identical, whole neural network is formed two dimensional topology by N * N neuron.Neural network has the architecture of highly-parallel, and all connect power and equate that all the propagation of information is two-way between the neuron.The state in its neighborhood is upgraded in neural network input according to each discrete state in evolutionary process, and whole neural network can be regarded a discrete time dynamical system as.

Neural network is in evolutionary process; Outside input according to impact point and the mapping generation neural network of barrier positional information in neural network topology structure; The neuron that barrier region is corresponding has negative outside input, and the impact point neuron has positive outside input.Neural network develops according to the outside input, and the positive neuron output value in impact point position can propagate into whole state space gradually damply through neuronic local the connection, up to reaching equilibrium state.S type activation function has guaranteed that impact point position neuron has the maximum positive neuron output value of the overall situation, and it is zero that the neuronic output valve of barrier region then is suppressed.Neural network reaches after the balance, and all neuron output values have just constituted a single-peaked curved surface, and the value of each point is just represented the obtainable cumulative maximum repayment of its corresponding states on the curved surface.

Suppose that the machine manual work does the space and be made up of 20 * 20 grids, neural network has identical topological structure with the robot working space, also comprises 20 * 20 neurons, and each neuron is corresponding to a discrete state of work space.Each neuron all only with its local neighborhood in neuron be connected, wherein neuronic type of attachment is as shown in Figure 1 in i neuron and its neighborhood.Whole neural network is formed two dimensional topology by 20 * 20 neurons.Neural network has the architecture of highly-parallel, and all connect power and all equate.In the neural network evolutionary process all neurons be input neuron be again output neuron, the propagation of information is two-way between the neuron, whole neural network can be regarded a discrete time dynamical system as.

I neuron of neural network is corresponding to i discrete state of structure space, and then i neuron discrete time kinetics equation is:

$x_i(t+1)=\left\{\begin{array}{cc}1& \mathrm{ifi}=i_{*}\\ f(\underset{j=1}{\overset{N}{\mathrm{\Σ}}}{w}_{\mathrm{ij}}{x}_j\left(t\right)+I_i\left(t\right))& \mathrm{otherwise}\end{array}\right.$

In the formula, i _*Be target nerve unit index value, x _i(t) be i neuron in t output valve constantly, N is the interior neuron number of i neuron neighborhood, I _i(t) be that i neuron imported in t outside constantly, f is an activation function, w _IjBe that j neuron weighed to i neuronic connection, computing formula is shown below:

$w_{\mathrm{ij}}=\left\{\begin{array}{cc}{e}^{-\mathrm{\&eta;}{|i-j|}^2}& \mathrm{if}|i-j|\&le;\mathrm{<figure-callout\; id="0"\; label="r"\; filenames="HDA0000088055920000013.png,HDA0000088055920000021.png"\; state="\{\{state\}\}">r</figure-callout>}\\ 0& \mathrm{if}|i-j|>r\end{array}\right.$

In the formula, | i-j| is a vector x in the structure space _iAnd x _jBetween the Euclidian distance since each neuron only with its local neighborhood in neuron is connected, the r value is 1, neuron output forms single-peaked curved surface when guaranteeing that neural network reaches equilibrium state, the η span is (1,2), obvious w _Ij=w _Ji, i.e. w _IjFor symmetry; The neuronal activation function is selected the S type function, defines as follows:

$f\left(x\right)=\left\{\begin{array}{cc}0& \mathrm{ifx}\&le;0\\ \mathrm{kx}& \mathrm{if}0<x<1\\ 1& \mathrm{ifx}\&GreaterEqual;1\end{array}\right.$

In the formula; K is the slope of linear model; Span is (0,1), and f (x) has guaranteed that the positive neuron output value in impact point position can propagate into whole state space gradually damply; And impact point has the maximum positive neuron output value of the overall situation, and it is zero that the neuronic output valve of barrier region then is suppressed; The individual neuronic outside input of i is produced by impact point and the mapping of barrier positional information in neural network topology structure, is defined as follows:

$I_i\left(t\right)=\left\{\begin{array}{cc}V& \mathrm{if}{x}_i\left(t\right)=t\arg \mathrm{et}\\ -V& \mathrm{if}{x}_i\left(t\right)=\mathrm{<figure-callout\; id="0"\; label="obstacle"\; filenames="HDA0000088055920000013.png,HDA0000088055920000021.png"\; state="\{\{state\}\}">obstacle</figure-callout>}\\ 0& \mathrm{otherwise}\end{array}\right.$

In the formula; V is bigger constant, have the maximum neuron output value of the overall situation in order to guarantee the impact point neuron, and the barrier region neuron has the minimum neuron output value of the overall situation; The value of V should be greater than neuronic input summation, and span is the real number greater than 4.

2 repayment function designs

In learning process; Robot can move on 4 directions; Can select 4 actions up and down in free position, action is selected according to current state by robot, and the repayment immediately that obtains if this action makes robot arrive target is 1; If bumping, robot and barrier or other machines people obtain immediately that repayment is-0.2, if robot moves then the repayment immediately that obtains is-0.1 at free space.

3 calculate cumulative maximum repayment initial value

The mapping in neural network topology structure of based target point and barrier positional information produces the outside input of neutral net, and the neuron that barrier region is corresponding has negative outside input, and the impact point neuron has positive outside input.Neural network develops according to the outside input, and the positive neuron output value in impact point position can propagate into whole state space gradually damply through neuronic local the connection, up to reaching equilibrium state.S type activation function has guaranteed that impact point position neuron has the maximum positive neuron output value of the overall situation, and it is zero that the neuronic output valve of barrier region then is suppressed.Neural network reaches after the equilibrium state, and all neuron output values have just constituted a single-peaked curved surface, and are as shown in Figure 2, and the value of each point is just represented the obtainable cumulative maximum repayment of its corresponding states on the curved surface.

Robot is from arbitrary initial state s _tThe accumulation repayment that obtains of setting out defines as follows:

$V^{\mathrm{\&pi;}}\left(s_t\right)=r_t+\mathrm{\&gamma;}{r}_{t+1}+{\mathrm{\&gamma;}}^2{r}_{t+2}+\mathrm{\&Lambda;}=\underset{i=0}{\overset{\&infin;}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}^i{r}_{t+i}$

In the following formula, π is a control strategy, the immediately repayment sequence of r for obtaining, and γ is a commutation factor, span is (0,1), selects γ=0.95 here; Then robot follows the cumulative maximum repayment V that optimal strategy obtains from state s ^*(s) calculate as follows:

$V^{*}\left(s\right)=\underset{\mathrm{\&pi;}}{\arg \max }{V}^{\mathrm{\&pi;}}\left(s\right),(\&ForAll;s)$

When neural network reaches equilibrium state, define the cumulative maximum repayment V of the corresponding state of each neuron ^* _Init(s _i) equal this neuron output value x _i, its relation formula is following:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i.$

In the formula, x _iI neuronic output valve when reaching equilibrium state for neural network, V ^* _Init(s _i) be from state s _iSet out and follow the obtainable cumulative maximum repayment of optimal strategy institute.

4 robot intensified learnings based on neural network

4.1 traditional Q-learning algorithm

In the Markovian decision process, current state is known through the sensor senses surrounding environment in robot, and selects the current action that will carry out, and this action of environmental response also provides repayment immediately, and produce follow-up state.The task of robot intensified learning is exactly to obtain an optimal strategy to make robot obtain maximum conversion accumulation repayment from current state.The accumulation repayment that robot follows any tactful π acquisition from the arbitrary initial state is defined as:

$V^{\mathrm{\&pi;}}\left(s_t\right)\&equiv;r_t+\mathrm{\&gamma;}{r}_{t+1}+{\mathrm{\&gamma;}}^2{r}_{t+2}+\mathrm{\&Lambda;}\&equiv;\underset{i=0}{\overset{\&infin;}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}^i{r}_{t+i}$

In the formula, r _tBe t repayment immediately constantly, γ is a commutation factor, and span is (0,1), selects γ=0.95 here.

Robot defines as follows from the optimal strategy π * that state s can obtain the cumulative maximum repayment:

${\mathrm{\&pi;}}^{*}\&equiv;\underset{\mathrm{\&pi;}}{\arg \max }{V}^{\mathrm{\&pi;}}\left(s\right),(\&ForAll;s)$

Robot from state s follow optimal strategy π * the cumulative maximum repayment that can obtain be defined as V* (s), then the value of Q function is the maximum conversion accumulation repayment that the repayment immediately of current state adds follow-up state, computing formula is following:

Q(s，a)≡(1-α _t)Q(s，a)+α _t(r(s，a)+γV*(s′))

In the formula, α _tBe learning rate, span is (0,1), selects α usually _tInitial value is 0.5, and with the decay of study number of times; V* (s ') and Q (s ', a ') relational expression is following:

$V*\left(s^{\&prime;}\right)=\underset{a^{\&prime;}}{\max Q(s^{\&prime;},a^{\&prime;})}$

Q (s then _t, a _t) according to following Policy Updates:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\underset{a_t^{\&prime;}}{\mathrm{\&gamma;}\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

In the formula, Q _T-1(s _t, a _t) and Q _T-1(s ' _t, a ' _t) state of being respectively-action is to (s _t, a _t) and (s ' _t, a ' _t) at t-1 value constantly, a ' _tFor at new state s ' _tThe action of following selection.

4.2Q value initialization

According to known environment information neural network is developed,, at this moment define the obtainable cumulative maximum repayment of each discrete state and equal its corresponding neuronic output valve up to reaching equilibrium state.To carry out repayment immediately that selected action obtains from current state then and add that follow-up state follows the maximum conversion accumulation repayment that optimal strategy obtains, Q (s that can be right to all state-actions _i, a) rational initial value is set.Q (s _i, initial value computing formula a) is following:

$Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\&gamma;}{V}_{\mathrm{Init}}^{*}\left(s_j\right)$

In the formula, r is at state s _iThe following repayment immediately of selecting action a to obtain, γ is a commutation factor, span is (0,1), selects γ=0.95 here; s _jFor robot at state s _iThe following action follow-up state that a produced, the Q of selecting _Init(s _i, be that state-action is to (s a) _i, initial Q value a);

4.3 the Q learning algorithm based on neural network of the present invention

(1) neural network develops according to initialization context information, up to reaching equilibrium state.

(2) utilize neuron output value x _iTo state s _iInitialization is carried out in obtainable cumulative maximum repayment, and relation formula is following:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i$

(3) according to following regular initialization Q value:

Q _Init(s _i，a)＝r+γV _Init*(s _j)

(4) observe current state s _t

(5) continue in complex environment, to explore, at current state s _tUnder select one the action a _tAnd carry out, ambient condition is updated to new state s ' _t, and r is repaid in reception immediately _t

(6) observe new state s ' _t

(7) upgrade list item Q (s according to following formula _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\underset{a_t^{\&prime;}}{\mathrm{\&gamma;}\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

(8) judge that (the maximum study number of times of setting can guarantee that learning system restrains to the maximum study number of times whether robot arrived target or learning system and reached setting in maximum study number of times; Maximum study number of times is set to 300 in experimental situation of the present invention); If both satisfy one of which; Then study finishes, and continues study otherwise turn back to step (4).

For intensified learning Q value initialization of robot process is described, select robot impact point neighborhood to demonstrate.When neural network reaches equilibrium state; Neuron output value is shown in numerical value in the node among Fig. 3 in the neighborhood, and each node is corresponding to a discrete state, and the repayment of the cumulative maximum of each state equals the neuronic output valve of this state; Red node is represented dbjective state, and the grey node is represented barrier.Each arrow is represented an action; If the repayment immediately that the target goal state G of robot then obtains is 1; If with barrier or the other machines people repayment immediately that obtains that bumps then be-0.2, if robot is-0.1 in the repayment immediately that free space moves then acquisition.γ is a commutation factor, selects γ=0.95, can obtain the initial value of Q function according to Q value initialization formula, and the right initialization Q value of each state-action is represented shown in the numerical value like arrow among Fig. 4.After initialization is accomplished; Robot when robot faces than complex environment, just has certain purpose in the starting stage of study selecting appropriate action under the original state arbitrarily; Rather than the selection action of completely random ground, thereby accelerate algorithm the convergence speed.

Mobile robot's environmental modeling of setting up in the laboratory with explore on the software platform, carried out emulation experiment.The robot planning path that Fig. 5 obtains for the existing robots reinforcement Learning Method; Fig. 6 is an existing robots intensified learning algorithm convergence process.Learning algorithm begins later convergence through 145 study, and (like preceding 20 study) robot basically all can not arrive impact point in maximum iteration time in the starting stage of study.This is because the Q value is initialized to 0, makes robot have no priori, can only select action randomly, thereby it is lower to cause learning starting stage efficient, and algorithm the convergence speed is slower.

Fig. 7 is robot planning of the present invention path; Fig. 8 is a convergence process of the present invention.Learning algorithm begins convergence after through 76 study; And robot also can both arrive impact point basically in the starting stage of study within maximum iteration time; The present invention has effectively improved the learning efficiency in robot initial stage, has obviously accelerated the speed of convergence of learning process.

Claims (1)

1. robot intensified learning initial method based on neural network; Neural network has identical topological structure with the robot working space, and each neuron at first develops to neural network according to known component environment information corresponding to a discrete state in the state space; Up to reaching equilibrium state; At this moment each neuron output value is just represented the cumulative maximum repayment that its corresponding states can obtain, and then the repayment immediately of current state is added that follow-up state follows the maximum conversion accumulation repayment that optimal strategy obtains, Q (s that can be right to all state-actions; A) set rational initial value; Can priori be dissolved in the learning system through Q value initialization, the study in robot initial stage is optimized, thereby a learning foundation preferably is provided for robot; Specifically may further comprise the steps:

(1) sets up neural network model

Neutral net has identical topological structure with the structure space of robot work, each neuron only with its local neighborhood in neuron be connected, type of attachment is all identical; Connecting power all equates; The propagation of information is two-way between the neuron, and neutral net has the architecture of highly-parallel, and each neuron is corresponding to one of the robot working space discrete state; Whole neutral net is formed two dimensional topology by N * N neuron; The state in its neighborhood is upgraded in neutral net input based on each discrete state in evolutionary process, reaches poised state up to neutral net, when reaching poised state; Neuron output value just forms a single-peaked curved surface in the neutral net, and the value of every bit is just represented the cumulative maximum repayment that institute's corresponding states can obtain on the curved surface;

(2) design repayment function

In learning process; Robot can move on 4 directions; Select 4 actions up and down in free position, action is selected based on current state by robot, and the repayment immediately that obtains if this action makes robot arrive target is 1; If bumping, robot and barrier or other machines people obtain immediately that repayment is-0.2, if robot moves then the repayment immediately that obtains is-0.1 at free space;

(3) calculate cumulative maximum repayment initial value

When neural network reaches equilibrium state, define the cumulative maximum repayment V of the corresponding state of each neuron ^* _Init(s _i) equal this neuron output value x _i, its relation formula is following:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i,$

In the formula, x _iI neuronic output valve when reaching equilibrium state for neural network, V ^* _Init(s _i) be from state s _iSet out and follow the obtainable cumulative maximum repayment of optimal strategy institute;

(4) Q value initialization

Q (s _i, initial value a) is defined as at state s _iFollowing select action a obtained repay the maximum conversion accumulation repayment that r adds follow-up state immediately:

$Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\&gamma;}{V}_{\mathrm{Init}}^{*}\left(s_j\right)$

In the formula, s _jFor robot at state s _iThe following action follow-up state that a produced, the Q of selecting _Init(s _i, be that state-action is to (s a) _i, initial Q value a); γ is a commutation factor, selects γ=0.95;

(5) based on the robot intensified learning step of neural network

(a) neural network develops according to initialization context information, up to reaching equilibrium state;

(b) state s _iThe initial value of the cumulative maximum repayment that obtains is defined as neuron output value x _i, relation formula is following:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i$

(c) according to following regular initialization Q value: $Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\&gamma;}{V}_{\mathrm{Init}}^{*}\left(s_j\right),$

(d) observe current state s _t

(e) continue in complex environment, to explore, at current state s _tUnder select one the action a _tAnd carry out, ambient condition is updated to new state s ' _t, and r is repaid in reception immediately _t

(f) observe new state s ' _t

(g) upgrade list item Q (s according to following formula _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\underset{a_t^{\&prime;}}{\mathrm{\&gamma;}\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

In the formula, α _tBe learning rate, span is (0,1), and value is 0.5 usually, and decays with learning process; Q _T-1(s _t, a _t) and Q _T-1(s ' _t, a ' _t) state of being respectively-action is to (s _t, a _t) and (s ' _t, a ' _t) at t-1 value constantly, a ' _tFor at new state s ' _tThe action of following selection;

(h) judge whether robot has arrived the maximum study number of times that target or learning system have reached setting; The maximum study number of times of setting should guarantee that learning system restrains in maximum study number of times; If both satisfy one of which, then study finishes, and continues study otherwise turn back to step (d).

Images