CN102402712B - Robot reinforced learning initialization method based on neural network - Google Patents

Abstract

The invention provides a robot reinforced learning initialization method based on a neural network. The neural network has the same topological structure as a robot working space, and each neuron corresponds to a discrete state of a state space. The method comprises the following steps of: evolving the neural network according to the known partial environmental information till reaching a balance state, wherein at the moment, the output value of each neuron represents maximum cumulative return acquired when the corresponding state follows the optimal strategy; defining the initial value of a Q function as the sum of the immediate return of the current state and the maximum converted cumulative return acquired when the subsequent state follows the optimal strategy; and the mapping the known environmental information into the initial value of the Q function by the neural network. Therefore, the prior knowledge is fused into a robot learning system, and the learning capacity of the robot at the initial stage of reinforced learning is improved; and compared with the conventional Q learning algorithm, the method has the advantages of effectively improving the learning efficiency of the initial stage and increasing the algorithm convergence speed.

Description

Robot reinforced learning initialization method based on neural network

Technical field

The method that the present invention relates to priori to be dissolved into Q value initialization in mobile robot's learning system Zhong，Dui robot intensified learning process, belongs to machine learning techniques field.

Background technology

Continuous expansion along with robot application field, the task that robot faces also becomes increasingly complex, although pre-programmed is carried out in the repetition behavior that in a lot of situations, researchist can may carry out robot, but for realizing whole expected behavior, carry out Activity design and become more and more difficult, designer often can not make rational prediction to all behaviors of robot in advance.Therefore, autonomous robot that can perception environment must, by obtaining new behavior with the mutual on-line study of environment, make robot can reach the optimum action of target according to specific task choosing.

Intensified learning utilization is similar to the method for the trial and error (trial-and-error) in human thinking and finds optimum behavior strategy, aspect robot behavior study, is showing good learning performance at present.Q learning algorithm is a kind of intensified learning method that solves INFORMATION OF INCOMPLETE Markov decision problem, according to the return immediately of ambient condition and previous step study acquisition, the mapping policy of modification from state to action, so that the accumulation return value that behavior obtains from environment is maximum, thereby obtain optimum behavior strategy.Standard Q learning algorithm is generally Q or random number by Q value initialization, and robot is the priori to environment not, and the starting stage of study can only be selected action randomly, and therefore, in complex environment, algorithm the convergence speed is slower.In order to improve algorithm the convergence speed, researchist has proposed the method for many improvement Q study, improves Algorithm Learning efficiency, improves learning performance.

Generally, the method for acceleration Q study speed of convergence mainly comprises two aspects: a kind of method is the suitable return function of design, and another kind of method is reasonable initialization Q function.At present, researchist has proposed many improved Q learning algorithm ，Shi robot can obtain more effectively return in the process of intensified learning, mainly comprises: associated Q learning algorithm, inertia Q learning algorithm, Bayes Q learning algorithm etc.Its fundamental purpose will be dissolved in return function for the valuable implicit information of robot exactly, thus accelerating algorithm speed of convergence.Associated Q study compares current return and the return immediately in the moment in the past, selects the larger action of return value, can improve the learning ability of system by associated method for returning, reduces and obtains the needed iterative steps of optimal value.The target of inertia Q study is to provide a kind of method that predicted state is returned immediately, in learning process, utilize message delay principle, in the case of necessary new target is predicted, action comparer checks the expected returns of each situation, then selects the action executing of expected returns maximum.Bayes Q study utilizes probability distribution to describe the uncertainty estimation of robotary-action to Q value, in learning process, need to consider the distribution of previous moment Q value, and utilize robot learning to experience previous distribution is upgraded, utilize Bayes's variable to represent the cumulative maximum return of current state, bayes method, from having improved in essence the exploration strategy of Q study, has improved the performance of Q study.

Because enhanced signal in standard intensified learning is all the scalar value being calculated by state value function, people's knowledge form and behavior pattern cannot be dissolved in learning system.And in robot learning process, people often has the experience and knowledge of association area, therefore, in learning process, people's cognition and the intelligent form with enhanced signal are fed back to robot, can reduce state space dimension, accelerate algorithm the convergence speed.The problem existing in interactive process for standard intensified learning, provides external strengthening signal in real time by people in Thomaz Deng robot intensified learning process, and people is according to self experience adjustments training behavior, and guided robot carries out perspective exploration.Arsenio proposed a kind of to training data carry out online, the learning strategy of automatic marking, in interactive process, by triggering specific event, obtain training data, thereby the person of teaching be embedded into the backfeed loop of intensified learning.Mirza etc. have proposed the architecture based on interactive history, and robot can utilize and carry out social mutual historical experience with people and carry out intensified learning ，Shi robot and can in the easy game carrying out with people, obtain gradually suitable behavior.

Another kind improves the method for Q learning algorithm performance to be dissolved into priori in learning system exactly, and Q value is carried out to initialization.At present, Q value is carried out to initialized method and mainly comprise the method for approximate function, fuzzy rule method, potential function method etc.The method of approximate function utilizes the intelligent systems such as neural network to approach optimal value function, priori is become to return functional value ，Shi robot and in the subset of whole state space, learn, thereby can accelerate algorithm the convergence speed.Fuzzy rule method is set up fuzzy rule base according to initial environment information, then utilizes fuzzy logic to carry out initialization to Q value.The fuzzy rule that profit is set up in this way is all set according to environmental information is artificial, often can not reflect objectively the ambient condition of robot, causes algorithm unstable.Potential function method defines corresponding state potential function at whole state space, every bit potential energy value is corresponding to a certain discrete state value in state space, then utilize state potential function to carry out initialization to Q value, the Q value of learning system can be expressed as the change amount that initial value adds each iteration.In the middle of the various actions of robot, a series of code of conduct must be observed by robot, robot emerges corresponding behavior and intelligence by cognitive with reciprocation, and intensified learning Q value initialization of robot will become corresponding robot behavior by priori exactly.Therefore, how to obtain the regularization expression-form of priori, particularly realize domain expert's experience and the machine inference of general knowledge, it is robot behavior study urgent problem that people's cognition and intelligence are converted into the calculating of machine and human-machine intelligence's integration technology of reasoning.

Summary of the invention

For the present Research of existing robot intensified learning technology and the deficiency of existence, the present invention proposes a kind ofly can effectively improve the learning efficiency of starting stage, the robot reinforced learning initialization method based on neural network of convergence speedup speed, the method can be dissolved into priori in learning system by Q value initialization, study to the robot initial stage is optimized, thereby provides a good learning foundation for robot.

Robot reinforced learning initialization method based on neural network of the present invention, neural network has identical topological structure with robot working space, and each neuron is corresponding to a discrete state in state space.First according to known component environment information, neural network is developed, until reach equilibrium state, at this moment each neuron output value just represents the cumulative maximum return that its corresponding states can obtain, then the return immediately of current state is added to follow-up state follows the maximum conversion accumulation return (commutation factor is multiplied by cumulative maximum return) that optimal strategy obtains, can be to the right Q (s of all states-move, a) set rational initial value, by Q value initialization, priori can be dissolved in learning system, study to the robot initial stage is optimized, thereby for robot provides a good learning foundation, specifically comprise the following steps:

(1) set up neural network model

Neural network has identical topological structure with the structure space of robot work, each neuron is only connected with the neuron in its local neighborhood, type of attachment is all identical, connection weight all equates, between neuron, the propagation of information is two-way, neural network has the architecture of highly-parallel, each neuron is corresponding to one of robot working space discrete state, whole neural network forms two dimensional topology by N * N neuron, neural network is upgraded the state in its neighborhood according to the input of each discrete state in evolutionary process, until neural network reaches equilibrium state, while reaching equilibrium state, in neural network, neuron output value just forms a single-peaked curved surface, on curved surface, the value of every bit just represents the cumulative maximum return that institute's corresponding states can obtain,

(2) design return function

In learning process, robot can move up 4 sides, in condition selecting 4 actions up and down arbitrarily, action is selected according to current state by robot, if this action makes robot arrive target, the return immediately obtaining is 1, if robot and barrier or other robot bump, obtain return immediately for-0.2, if robot moves at free space, the return immediately obtaining is-0.1;

(3) calculate cumulative maximum return initial value

When neural network reaches equilibrium state, define the cumulative maximum return V of the corresponding state of each neuron ^* _init(s _i) equal this neuron output value x _i, its relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\←x_i,$

In formula, x _ii neuronic output valve while reaching equilibrium state for neural network, V ^* _init(s _i) be from state s _iset out and follow the obtainable cumulative maximum return of optimal strategy institute;

(4) Q value initialization

Q(s _i, initial value a) is defined as at state s _ilower selection action a obtains returns the maximum conversion accumulation return that r adds follow-up state immediately:

$Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\γ}{V}_{\mathrm{Init}}^{*}\left(s_j\right)$

In formula, s _jfor robot is at state s _ithe follow-up state that lower selection action a produces, Q _init(s _i, be a) that state-action is to (s _i, initial Q value a); γ is commutation factor, selects γ=0.95;

(5) the robot intensified learning step based on neural network

(a) neural network develops according to initialization context information, until reach equilibrium state;

(b) state s _ithe initial value of the cumulative maximum return obtaining is defined as neuron output value x _i, relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\←x_i$

(c) according to following regular initialization Q value: $Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\γ}{V}_{\mathrm{Init}}^{*}\left(s_j\right),$

(d) observe current state s _t;

(e) continue to explore in complex environment, at current state s _taction a of lower selection _tand carry out, ambient condition is updated to new state s ' _t, and r is returned in reception immediately _t;

(f) observe new state s ' _t;

(g) according to following formula, upgrade list item Q (s _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\α}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\α}}_t(r_t+\mathrm{\γ}\underset{a_t^{\′}}{\arg }\max Q_{t-1}(s_t^{\′},a_t^{\′}))$

In formula, α _tfor learning rate, span is (0,1), and value is 0.5 conventionally, and decays with learning process; Q _t-1(s _t, a _t) and Q _t-1(s ' _t, a ' _t) state of being respectively-action is to (s _t, a _t) and (s ' _t, a ' _t) at t-1 value constantly, a ' _tfor at new state s ' _tthe action of lower selection;

(h) judge whether robot has arrived the maximum study number of times that target or learning system have reached setting, the maximum study number of times of setting should guarantee that learning system restrains in maximum study number of times, if both meet one, study finishes, otherwise turns back to step (d) continue studying.

The present invention becomes Q function initial value by neural network by known environment information, thereby priori is dissolved in robotics learning system, improved the learning ability of robot in intensified learning starting stage, compare with traditional Q-learning algorithm, can effectively improve the learning efficiency of starting stage, accelerate algorithm the convergence speed.

Accompanying drawing explanation

Fig. 1 is i neuron neighbour structure schematic diagram.

Fig. 2 is neuron output value schematic diagram in robot target vertex neighborhood.

Fig. 3 is cumulative maximum return initial value V* _init(s, a) schematic diagram.

Fig. 4 is neural network neuron output value schematic diagram while reaching equilibrium state.

Fig. 5 is the robot planning path schematic diagram that existing Q study obtains.

Fig. 6 is existing Q learning algorithm convergence process schematic diagram.

Fig. 7 is robot planning of the present invention path schematic diagram.

Fig. 8 is study convergence process schematic diagram of the present invention.

Embodiment

The present invention is based on neural network robot intensified learning is carried out to initialization, neural network has identical topological structure with robot working space, when neural network reaches equilibrium state, neuron output value represents the cumulative maximum return of corresponding states, utilizes the return immediately of current state and the maximum conversion of follow-up state to accumulate the initial value that return obtains Q function.By Q value initialization, priori can be dissolved in learning system, the study in robot initial stage is optimized, thereby provide a good learning foundation for robot; Specifically comprise the following steps:

1 neural network model

Neural network has identical topological structure with robot working space, and each neuron is corresponding to robot working space's a discrete state.All neurons are all only connected with the neuron in its local neighborhood, and its type of attachment is all identical, and whole neural network forms two dimensional topology by N * N neuron.Neural network has the architecture of highly-parallel, and all connection weights all equate, between neuron, the propagation of information is two-way.Neural network is upgraded the state in its neighborhood according to the input of each discrete state in evolutionary process, and whole neural network can be regarded a discrete time dynamical system as.

Neural network is in evolutionary process, according to impact point and Obstacle Position information, the mapping in neural network topology structure produces the outside input of neural network, the neuron that barrier region is corresponding has negative outside input, and impact point neuron has positive outside input.Neural network develops according to outside input, and the positive neuron output value in impact point position can propagate into whole state space gradually damply by neuronic local connection, until reach equilibrium state.S type activation function has guaranteed that impact point position neuron has the maximum positive neuron output value of the overall situation, and the neuronic output valve of barrier region is suppressed to zero.After neural network reaches balance, all neuron output values have just formed a single-peaked curved surface, and on curved surface, the value of each point just represents the obtainable cumulative maximum return of its corresponding states.

Suppose that robot working space is comprised of 20 * 20 grids, neural network has identical topological structure with robot working space, also comprises 20 * 20 neurons, and each neuron is corresponding to a discrete state of work space.Each neuron is only connected with the neuron in its local neighborhood, and wherein the interior neuronic type of attachment of i neuron and its neighborhood as shown in Figure 1.Whole neural network forms two dimensional topology by 20 * 20 neurons.Neural network has the architecture of highly-parallel, and all connection weights all equate.In neural network evolutionary process, all neurons are input neuron and output neuron, and between neuron, the propagation of information is two-way, and whole neural network can be regarded a discrete time dynamical system as.

I neuron of neural network is corresponding to i discrete state of structure space, and i neuron discrete time kinetics equation is:

$x_i(t+1)=\left\{\begin{array}{cc}1& \mathrm{ifi}=i_{*}\\ f(\underset{j=1}{\overset{N}{\mathrm{\Σ}}}{w}_{\mathrm{ij}}{x}_j\left(t\right)+I_i\left(t\right))& \mathrm{otherwise}\end{array}\right.$

In formula, i _*for target nerve unit index value, x _i(t) be i neuron in t output valve constantly, N is the neuron number in i neuron neighborhood, I _i(t) be that i neuron inputted in t outside constantly, f is activation function, w _ijbe j neuron to i neuronic connection weight, computing formula is shown below:

$w_{\mathrm{ij}}=\left\{\begin{array}{cc}{e}^{-\mathrm{\&eta;}{|i-j|}^2}& \mathrm{if}|i-j|\&le;\mathrm{<figure-callout\; id="0"\; label="r"\; filenames="HDA0000088055920000012.png,HDA0000088055920000013.png"\; state="\{\{state\}\}">r</figure-callout>}\\ 0& \mathrm{if}|i-j|>r\end{array}\right.$

In formula, | i-j| is vector x in structure space _iand x _jbetween Euclidian distance, because each neuron is only connected with neuron in its local neighborhood, r value is 1, neuron output forms single-peaked curved surface when guaranteeing that neural network reaches equilibrium state, η span is (1,2), obvious w _ij=w _ji, i.e. w _ijfor symmetry; Neuron activation functions is selected S type function, is defined as follows:

$f\left(x\right)=\left\{\begin{array}{cc}0& \mathrm{ifx}\&le;0\\ \mathrm{kx}& \mathrm{if}0<x<1\\ 1& \mathrm{ifx}\&GreaterEqual;1\end{array}\right.$

In formula, k is the slope of linear model, span is (0,1), f (x) has guaranteed that the positive neuron output value in impact point position can propagate into whole state space gradually damply, and impact point has the maximum positive neuron output value of the overall situation, and the neuronic output valve of barrier region is suppressed to zero; I neuronic outside input mapping generation in neural network topology structure by impact point and Obstacle Position information, be defined as follows:

$I_i\left(t\right)=\left\{\begin{array}{cc}V& \mathrm{if}{x}_i\left(t\right)=t\arg \mathrm{et}\\ -V& \mathrm{if}{x}_i\left(t\right)=\mathrm{<figure-callout\; id="0"\; label="obstacle"\; filenames="HDA0000088055920000012.png,HDA0000088055920000013.png"\; state="\{\{state\}\}">obstacle</figure-callout>}\\ 0& \mathrm{otherwise}\end{array}\right.$

In formula, V is larger constant, and in order to guarantee that impact point neuron has the maximum neuron output value of the overall situation, and barrier region neuron has the minimum neuron output value of the overall situation, the value of V should be greater than neuronic input summation, and span is to be greater than 4 real number.

2 return function designs

In learning process, robot can move up 4 sides, in free position, can select 4 actions up and down, action is selected according to current state by robot, if this action makes robot arrive target, the return immediately obtaining is 1, if robot and barrier or other robot bump, obtain return immediately for-0.2, if robot moves at free space, the return immediately obtaining is-0.1.

3 calculate cumulative maximum return initial value

According to impact point and Obstacle Position information, the mapping in neural network topology structure produces the outside input of neural network, and the neuron that barrier region is corresponding has negative outside input, and impact point neuron has positive outside input.Neural network develops according to outside input, and the positive neuron output value in impact point position can propagate into whole state space gradually damply by neuronic local connection, until reach equilibrium state.S type activation function has guaranteed that impact point position neuron has the maximum positive neuron output value of the overall situation, and the neuronic output valve of barrier region is suppressed to zero.After neural network reaches equilibrium state, all neuron output values have just formed a single-peaked curved surface, and as shown in Figure 2, on curved surface, the value of each point just represents the obtainable cumulative maximum return of its corresponding states.

Robot is from arbitrary initial state s _tthe accumulation return obtaining of setting out is defined as follows:

$V^{\mathrm{\&pi;}}\left(s_t\right)=r_t+\mathrm{\&gamma;}{r}_{t+1}+{\mathrm{\&gamma;}}^2{r}_{t+2}+\mathrm{\&Lambda;}=\underset{i=0}{\overset{\&infin;}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}^i{r}_{t+i}$

In above formula, π is control strategy, the immediately return sequence of r for obtaining, and γ is commutation factor, span is (0,1), selects γ=0.95 here; Robot follows from state s the cumulative maximum return V that optimal strategy obtains ^*(s) be calculated as follows:

$V^{*}\left(s\right)=\underset{\mathrm{\&pi;}}{\arg \max }{V}^{\mathrm{\&pi;}}\left(s\right),(\&ForAll;s)$

When neural network reaches equilibrium state, define the cumulative maximum return V of the corresponding state of each neuron ^* _init(s _i) equal this neuron output value x _i, its relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i.$

In formula, x _ii neuronic output valve while reaching equilibrium state for neural network, V ^* _init(s _i) be from state s _iset out and follow the obtainable cumulative maximum return of optimal strategy institute.

4 robot intensified learnings based on neural network

4.1 traditional Q-learning algorithms

In Markovian decision process Zhong， robot, by sensor senses surrounding environment, know current state, and select the current action that will carry out, this action of environmental response also provides return immediately, and produce follow-up state.The task of robot intensified learning is exactly to obtain an optimal strategy to make robot obtain maximum conversion accumulation return from current state.The accumulation return that robot follows any tactful π acquisition from arbitrary initial state is defined as:

$V^{\mathrm{\&pi;}}\left(s_t\right)\&equiv;r_t+\mathrm{\&gamma;}{r}_{t+1}+{\mathrm{\&gamma;}}^2{r}_{t+2}+\mathrm{\&Lambda;}\&equiv;\underset{i=0}{\overset{\&infin;}{\mathrm{\&Sigma;}}}{\mathrm{\&gamma;}}^i{r}_{t+i}$

In formula, r _tfor t return immediately constantly, γ is commutation factor, and span is (0,1), selects γ=0.95 here.

The optimal strategy π * that robot can obtain cumulative maximum return from state s is defined as follows:

${\mathrm{\&pi;}}^{*}\&equiv;\underset{\mathrm{\&pi;}}{\arg \max }{V}^{\mathrm{\&pi;}}\left(s\right),(\&ForAll;s)$

Robot follows from state s the cumulative maximum return that optimal strategy π * can obtain and is defined as V* (s), and the return immediately that value of Q function is current state adds the maximum conversion accumulation return of follow-up state, and computing formula is as follows:

Q(s，a)≡(1-α _t)Q(s，a)+α _t(r(s，a)+γV*(s′))

In formula, α _tfor learning rate, span is (0,1), conventionally selects α _tinitial value is 0.5, and with the decay of study number of times; V* (s ') and Q (s ', a ') relational expression is as follows:

$V*\left(s^{\&prime;}\right)=\underset{a^{\&prime;}}{\max Q(s^{\&prime;},a^{\&prime;})}$

Q (s _t, a _t) according to following Policy Updates:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\underset{a_t^{\&prime;}}{\mathrm{\&gamma;}\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

In formula, Q _t-1(s _t, a _t) and Q _t-1(s ' _t, a ' _t) state of being respectively-action is to (s _t, a _t) and (s ' _t, a ' _t) at t-1 value constantly, a ' _tfor at new state s ' _tthe action of lower selection.

4.2Q value initialization

According to known environment information, neural network is developed, until reach equilibrium state, at this moment define the obtainable cumulative maximum return of each discrete state and equal its corresponding neuronic output valve.Then by carry out return immediately that selected action obtains from current state, add that follow-up state follows the maximum conversion accumulation return that optimal strategy obtains, can be to the right Q (s of all states-move _i, a) rational initial value is set.Q(s _i, calculation of initial value formula a) is as follows:

$Q_{\mathrm{Init}}(s_i,a)=r+\mathrm{\&gamma;}{V}_{\mathrm{Init}}^{*}\left(s_j\right)$

In formula, r is at state s _ithe return immediately that lower selection action a obtains, γ is commutation factor, span is (0,1), selects γ=0.95 here; s _jfor robot is at state s _ithe follow-up state that lower selection action a produces, Q _init(s _i, be a) that state-action is to (s _i, initial Q value a);

The 4.3 Q learning algorithms based on neural network of the present invention

(1) neural network develops according to initialization context information, until reach equilibrium state.

(2) utilize neuron output value x _ito state s _iinitialization is carried out in obtainable cumulative maximum return, and relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i$

(3) according to following regular initialization Q value:

Q _Init(s _i，a)＝r+γV _Init*(s _j)

(4) observe current state s _t.

(5) continue to explore in complex environment, at current state s _taction a of lower selection _tand carry out, ambient condition is updated to new state s ' _t, and r is returned in reception immediately _t.

(6) observe new state s ' _t.

(7) according to following formula, upgrade list item Q (s _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\underset{a_t^{\&prime;}}{\mathrm{\&gamma;}\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

(8) judge that (the maximum study number of times of setting can guarantee that learning system restrains in maximum study number of times to the maximum study number of times whether robot arrived target or learning system and reached setting, in experimental situation of the present invention, maximum study number of times is set to 300), if both meet one, study finishes, otherwise turns back to step (4) continue studying.

For intensified learning Q value initialization of robot process is described, select robot impact point neighborhood to demonstrate.When neural network reaches equilibrium state, in neighborhood, neuron output value is as shown in numerical value in node in Fig. 3, and each node is corresponding to a discrete state, and the return of the cumulative maximum of each state equals the neuronic output valve of this state, red node represents dbjective state, and grey node represents barrier.Each arrow represents an action, if the return immediately that the target goal state G of robot obtains is 1, if with bump the return immediately that obtains of barrier or other robot for-0.2，Ruo robot moves at free space the return immediately obtaining, be-0.1.γ is commutation factor, selects γ=0.95, can obtain the initial value of Q function according to Q value initialization formula, and the right initialization Q value of each state-move represents as shown in numerical value as arrow in Fig. 4.After initialization completes, when robot can both select appropriate action ，Dang robot to face compared with complex environment under any original state, in the starting stage of study, just there is certain purpose, rather than the selection action of completely random ground, thereby accelerate algorithm the convergence speed.

On mobile robot's environmental modeling of setting up in laboratory and exploration software platform, carried out emulation experiment.Fig. 5 is the robot planning path that existing robot intensified learning method obtains; Fig. 6 is existing robot intensified learning algorithm convergence process.Learning algorithm starts convergence after 145 study, and in the starting stage of study, (as front 20 study) robot substantially all can not arrive impact point in maximum iteration time.This is because Q value is initialized to 0, makes robot without any priori, can only select randomly action, thereby it is lower to cause learning starting stage efficiency, and algorithm the convergence speed is slower.

Fig. 7 is robot planning of the present invention path; Fig. 8 is convergence process of the present invention.Learning algorithm is starting convergence after 76 study, and robot also can arrive impact point substantially in the starting stage of study within maximum iteration time, the present invention has effectively improved the learning efficiency in robot initial stage, has obviously accelerated the speed of convergence of learning process.

Claims (1)

1. the robot reinforced learning initialization method based on neural network, neural network has identical topological structure with robot working space, each neuron is corresponding to a discrete state in state space, first according to known component environment information, neural network is developed, until reach equilibrium state, at this moment each neuron output value just represents the cumulative maximum return that its corresponding states can obtain, then the return immediately of current state is added to follow-up state follows the maximum conversion accumulation return that optimal strategy obtains, can be to the right Q (s of all states-move, a) set rational initial value, by Q value initialization, priori can be dissolved in learning system, study to the robot initial stage is optimized, thereby for robot provides a good learning foundation, specifically comprise the following steps:

(1) set up neural network model

Neural network has identical topological structure with the structure space of robot work, each neuron is only connected with the neuron in its local neighborhood, type of attachment is all identical, connection weight all equates, between neuron, the propagation of information is two-way, neural network has the architecture of highly-parallel, each neuron is corresponding to one of robot working space discrete state, whole neural network forms two dimensional topology by N * N neuron, neural network is upgraded the state in its neighborhood according to the input of each discrete state in evolutionary process, until neural network reaches equilibrium state, while reaching equilibrium state, in neural network, neuron output value just forms a single-peaked curved surface, on curved surface, the value of every bit just represents the cumulative maximum return that institute's corresponding states can obtain,

(2) design return function

In learning process, robot can move up 4 sides, can select action according to current state, in free position, can both in 4 actions of advancing, retreat, turn left, turn right, select a suitable action, if this action makes robot arrive target, the return immediately obtaining is 1, if robot and barrier or other robot bump, obtain return immediately for-0.2, if robot moves at free space, the return immediately obtaining is-0.1;

(3) calculate cumulative maximum return initial value

When neural network reaches equilibrium state, define the cumulative maximum return V of the corresponding state of each neuron ^* _init(s _i) equal this neuron output value x _i, its relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i,$

In formula, xi is neural network i neuronic output valve while reaching equilibrium state, V ^* _init(s _i) be from state s _iset out and follow the obtainable cumulative maximum return of optimal strategy institute;

(4) Q value initialization

Q(s _i, initial value a) is defined as at state s _ilower selection action a obtains returns the maximum conversion accumulation return that r adds follow-up state immediately:

$Q_{\mathrm{Init}}(s_i,a)=r+{\mathrm{\&gamma;V}}_{\mathrm{Init}}^{*}\left(s_j\right)$

In formula, s _jfor robot is at state s _ithe follow-up state that lower selection action a produces, Q _init(s _i, be a) that state-action is to (s _i, initial Q value a); γ is commutation factor, selects γ=0.95;

(5) the robot intensified learning step based on neural network

(a) neural network develops according to initialization context information, until reach equilibrium state;

(b) state s _ithe initial value of the cumulative maximum return obtaining is defined as neuron output value x _i, relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i$

(c) according to following regular initialization Q value: $Q_{\mathrm{Init}}(s_i,a)=r+{\mathrm{\&gamma;V}}_{\mathrm{Init}}^{*}\left(s_j\right),$

(d) observe current state s _t;

(e) continue to explore in complex environment, at current state s _taction a of lower selection _tand carry out, ambient condition is more

New is new state s' _t, and r is returned in reception immediately _t;

(f) observe new state s' _t;

(g) according to following formula, upgrade list item Q (s _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\mathrm{\&gamma;}\underset{a_t^{\&prime;}}{\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

In formula, α _tfor learning rate, value is 0.5, and decays with learning process; Q _t-1(s _t, a _t) and Q _t-1(s' _t, a' _t) state of being respectively-action is to (s _t, a _t) and (s' _t, a' _t) at t-1 value constantly, a' _tfor at new state s' _tthe action of lower selection;

(h) judge whether robot has arrived the maximum study number of times that target or learning system have reached setting, the maximum study number of times of setting should guarantee that learning system restrains in maximum study number of times, if both meet one, study finishes, otherwise turns back to step (d) continue studying.

Images