CN101599137A - Autonomous operant conditioning reflex automat and the application in realizing intelligent behavior - Google Patents

Abstract

Autonomous operant conditioning reflex automat and the application in realizing intelligent behavior belong to bionical field.A kind of autonomous operant conditioning reflex automat AOC relates to a kind of discrete calculation machine model of describing autonomous formula automaton, mainly comprise: operational set, state set, " condition-operation " regular collection, observable state transitions, and operant conditioning reflex law of learning, and, defined behavior entropy, stipulated the recurrence working procedure of AOC based on AOC state orientation value.The key character of AOC is to simulate biological operant conditioning reflex mechanism, thereby has a bionical self organizing function, comprise self study and adaptation function, can be used for describing, simulation designs various self-organizing systems, especially, should be applied to describe simulation, the various intelligent behaviors of planing machine robot system.

Description

Autonomous operant conditioning reflex automat and the application in realizing intelligent behavior

Technical field

The present invention relates to a kind of automat, a kind of biorobot based on the operant conditioning reflex principle.

Background technology

The automaton model that is used for learning system has just had at 1960s, and is called as learning automaton, in the past few years, mainly is that the structure that changes learning automaton satisfies different application requirements, generally is that existing input has output again.The present invention is a self-organizing system based on Si Jinna operant conditioning reflex theory, has self study and adaptation function.This Jenner just began the experimental study of animal learning and proposed two kinds of study forms from late 1920s: a kind of is through canonical form conditioned reflex study, in order to mould organic respondent behavior; Another kind is operant conditioned reflex study, in order to mould organic operation behavior.Western scholar thinks that these two kinds of reflections are two kinds of different connection processes: classical conditioning is the connection process of S--R; Operant conditioned reflex is the connection process of R--S.

The nearly more than ten years, the academic attention rate of autonomous system increases year by year, and the document total amount relevant with autonomous system increases year by year.The present invention is an autonomous formula automat, and with Non-Self-Governing formula automat difference, its output does not need the driving of external command, is that automat is made according to the needs of self.Relevant patent as: application number is 98115560.X, and name is called the screen generating method of customer operated type automatic computer and customer operated type automatic computer, application number and is 200710071071.0 and is called based on band memory and determines that the regular expression method for matching in speedup etc. of finte-state machine all is automat to be intersected with external environment realize certain function.At present, autonomous formula operant conditioning reflex automat did not also occur.

The present invention proposes an abstract self-organizing model based on Si Jinna operant conditioning reflex theory, be used for describing, simulation, design various self-organizing systems, make it show self study and adaptive characteristic, especially, should be applied to describe, simulation, the various intelligent behaviors of planing machine robot system.

Summary of the invention

The invention provides a kind of description that can be used for, simulation, the autonomous operant conditioning reflex automat that design has self-organization (comprising self study and self-adaptation) function.

Operant conditioning reflex automat of the present invention is one nine tuple, comprise: the incoming symbol set, internal state set, built-in function set, output symbol set, " condition-operation " regular collection, state transitions unit, observing unit, state orientation unit at random, and operant conditioning reflex unit, and, stipulated the recurrence working procedure of AOC.The key character of AOC is to simulate biological operant conditioning reflex mechanism, thereby has bionical self organizing function, comprises self study and adaptation function, can be used for describing, and simulation designs various self-organizing systems with interactive function.

A general finite-state automata is five-tuple a: FA={A, Z, S, f, g}.Wherein, A represents limited incoming symbol set, state symbol set (s (0) ∈ S is an original state) that S represents limited (inside), and Z represents limited output (receive status) assemble of symbol, and f:S * A → S represents state transition function, and g:S → Z represents output function.Finite-state automata FA is a non-autonomous system.

Among the AOC functional symbol be not equal to finite-state automata FA in incoming symbol, the functional symbol representative is the built-in function of AOC among the AOC, and among the FA incoming symbol representative be external command, on this meaning, AOC and finite-state automata FA be non-equivalence seemingly.Functional symbol set omega among the AOC is not the incoming symbol set among the FA, but the built-in function of AOC.It on the incoming symbol collective entity among the FA set of the instruction that may import of outside.Do not have the output symbol set among the AOC, do not have output function naturally yet.As autonomous formula system, AOC needs output symbol set and output function.Autonomous system also can or also need to act on environment or objective world.From the form of state space equation, output is the combination of state, or the combination of state and operation, and therefore, internal state set that we can say AOC itself is exactly a kind of set of output symbol, and the state of AOC can be observed; " state of AOC can be observed " is meant that AOC self has receptor in the definition, can detect the change of oneself state, and do not mean that the external world can observe this tittle; Autonomous formula automat also needs output, and this output does not need the driving of external command, is that automat is made according to the needs of self.

Autonomous formula automat is compared with Non-Self-Governing formula automat, it is advantageous that its output does not need the driving of external command, certain that to be automat make according to the needs of self acts on the action of environment, even that is to say external environment condition changes, autonomous formula automat still can be worked as usual, and Non-Self-Governing formula automat need change the variation that structural model or parameter adapt to external environment condition.Non-autonomous system always can be converted into autonomous system, so always can find an autonomous operant conditioning reflex AOC corresponding with corresponding Non-Self-Governing operant conditioning reflex automat.Autonomous operant conditioning reflex AOC uses more extensive.

In information theory, entropy can be used as measuring of certain incident uncertainty.Quantity of information is big more, and architecture is regular more, and function is perfect more, and entropy is just more little.Utilize the notion of entropy, theoretically the metering of research information, transmission, conversion, storage.The present invention introduces the notion of operation entropy, the convergence of proof AOC operation entropy ψ (t), because the process of system's self-organization is the process of absorption information, it is the process of drawing negentropy, it is the process that removes uncertainty, so also just illustrate the self-organization characteristic of AOC, AOC has self study and adaptation function really.

The present invention proposes a kind of autonomous operant conditioning reflex automat, and simulate the zoopery of Si Jinna with it, realized simulated operation conditioned reflex learning mechanism to prove this automat, also realize the Balance Control of double-wheel self-balancing robot simultaneously, illustrate that AOC can be used for the various intelligent behaviors of planing machine robot system with it.

Automat of the present invention is the autonomous operant conditioning reflex automat of one nine tuple:

AOC＝<t，Ω，S，Γ，δ，ε，η，ψ，s ₀>

Wherein

(1) discrete time of AOC: t ∈ 0,1,2 ..., n _t, t=0 is the initial moment of AOC;

(2) functional symbol of AOC set: Ω={ α _k| k=1,2 ..., n _Ω, α _kK functional symbol for AOC;

(3) state set of AOC: S={s _i| i=0,1,2 ..., n _S, s _iI state for AOC;

(4) working rule of AOC set: Γ={ r _Ik(p) | p ∈ P; I ∈ 0,1,2 ..., n _S; K ∈ 0,1,2 ..., n _Ω, " condition-operation " regular r at random _Ik(p): s _i→ α _k(p) mean that AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at state _iCondition under implementation and operation α _kProbable value, P represents p _IkSet;

(5) state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), AOC t+1 state s (t+1) ∈ S is constantly determined by t state s (t) ∈ S and t operation α (t) ∈ Ω constantly constantly, and is irrelevant with state and the operation of its t before the moment; The determined state transitions process of δ is known or unknown, but the result of its state transitions can observe;

(6) orientation function of AOC: ε: S → E={ ε _i| i=0,1,2 ..., n _S, ε _i=ε (s _i) ∈ E is state s _iThe orientation value of ∈ S;

(7) the operant conditioning reflex law of learning of AOC:

Regulate working rule r _Ik(p) the enforcement Probability p ∈ P of ∈ Γ: suppose that t state constantly is s (t), implementation and operation α (t) ∈ Ω, the state s (t+1) that t+1 observes constantly, operant conditioning reflex theory according to Si Jinna, if ε (s (t+1))-ε (s (t))＜0 then p (α (t) | s (t)) tend to reduce, otherwise, if ε (s (t+1))-ε (s (t))＞0 then p (α (t) | s (t)) tend to increase.Autonomous operant conditioning reflex automat AOC is in state s (t)=s in the t moment _iAnd current selection operation α (t)=α _k, simultaneously according to the state transitions transfer function, next state s (t+1)=s constantly _jThe operant conditioning reflex mechanism that simulation is biological, next is that the probability of t+1 current operation constantly will change constantly so, its value increases Δ on original basis, here Δ is relevant with orientation value ε, orientation value is big more to show that the result of operation is good more, Δ is big more simultaneously, the probability of t+1 all the other operations constantly all correspondingly deducts a value, and the value that deducts and just in time be Δ, the value that corresponding operating deducts just in time is that the probability of last one this operation constantly accounts for these ratios of operating (not comprising t selected operation of the moment) sum and multiply by Δ.Guaranteed that like this each probability sum of selecting each operation constantly all is 1.Be described as with formula more formally: as s (t)=s _i, α (t)=α _kAnd s (t+1)=s _jThe time, p so _Ik(t+1)=p _Ik(t)+and Δ, the probability tables of other operation is shown p _Iu(t+1)=p _Iu(t)-and Δ ξ, u represents 0 to n here _ΩBetween be not equal to any one numerical value of k.Wherein, p _Ik(t) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly; p _Ik(t+1) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t+1 value constantly;

And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value;

Be monotonic increasing function, satisfy And if only if x=0; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right),$ Here v represents 0 to n _ΩBetween be not equal to all numerical value of k,

Expression AOC state is in s _iImplementation and operation α under the condition of ∈ S _uThe probability sum of ∈ Ω is in t value constantly; p _Iu(t) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _uThe probability of ∈ Ω is at t value constantly, p _Iu(t+1) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _uThe probability of ∈ Ω is in t+1 value constantly.

(8) the operation entropy of AOC: ψ: P * E → R ⁺, R ⁺Be positive set of real numbers, AOC is in s at t operation entropy ψ (t) expression t moment state constantly _iOperation entropy sum under the condition:

$\mathrm{\&psi;}\left(t\right)=\mathrm{\&psi;}\left(\mathrm{\&Omega;}\left(t\right)\right|S)=\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i{\mathrm{\&psi;}}_i\left(t\right)=\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right){\mathrm{\&psi;}}_i\left(\mathrm{\&Omega;}\left(t\right)\right|s_i)$

It is in state s (t)=s constantly by t _iEvolutionary operator probability set under the condition and orientation function set decision.ψ _i(t) be that AOC is in state s _iOperation entropy under the condition:

${\mathrm{\&psi;}}_i\left(t\right)={\mathrm{\&psi;}}_i\left(\mathrm{\&Omega;}\left(t\right)\right|s_i)=-\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i)$

Know that operation entropy and weighted sum under each state just can draw AOC at t operation entropy constantly

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i).$ If the operation entropy ψ (t) of AOC is more and more littler and trend towards minimum when t → ∞, so just illustrate that AOC operation entropy ψ (t) is a convergent.AOC is a self-organizing system based on Si Jinna operant conditioning reflex theory, has self study and adaptation function.The process of system's self-organization is the process of absorption information, is the process of drawing negentropy, is the process that removes uncertainty.In order to illustrate the self-organization characteristic of AOC, we need prove the convergence of AOC operation entropy ψ (t).

(9) initial state of AOC: s ₀=s (0) ∈ S.

Key character of the present invention is to simulate biological operant conditioning reflex mechanism, thereby has bionical self organizing function, comprises self study and adaptation function, can be used for describing, and simulates, and designs the system of various self-organizations.

Autonomous operant conditioning reflex automat AOC among the present invention recursively moves according to following program step:

(1) initialization: t=0 is set, the original state s of given AOC (0) at random, given learning rate a, given initial operation Probability p _Ik(0)=1/n _Ω(i=0,1,2 ..., n _SK=1,2 ..., n _Ω); Given stop time T _f

(2) selection operation: according to the regular r among " condition-operation " regular collection Γ among the operational set Γ _Ik(p): s _i→ α _k(p), promptly AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at its state _iCondition under implementation and operation α _kProbable value, select the AOC state to be in operation α (t) the ∈ Ω of s (t) ∈ S randomly;

(3) in implementation and operation: the t moment, AOC is in state s (t) ∈ S and implements operation α (t) the ∈ Ω that previous step has been chosen, and δ (s (t), α (t))=δ (s takes place to shift current state _i, α _k);

(4) observer state: according to the state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), the result of state transitions can observe fully, promptly exist j ∈ 0,1,2 ..., n _SMake s (t+1)=s _j

(5) operant conditioning reflex: at t moment implementation and operation, not only the state of AOC shifts, and it each operates in next enforcement probability constantly and also change, then according to the operant conditioning reflex law of learning

Regulate working rule r _Ik(p) the enforcement Probability p ∈ P of ∈ Γ.T is s (t)=s constantly _iAnd α (t)=α _k, t+1 evolutionary operator probability is constantly complied with so

Upgrade.Wherein, And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right);$

(6) calculating operation entropy: according to the formula of the operation entropy that defines

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i).$ Calculate t operation entropy constantly, wherein, p (s _i) be AOC state s _iThe probability of occurrence of ∈ S is at t value constantly, p (α _k| s _i) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly.

(7) recurrence shifts: if t+1≤T _f, t=t+1 also repeats (2)-(7) so;

(8) as t+1＞T _fIn time, shut down.

The process flow diagram of the inventive method is seen Fig. 2.

Description of drawings

Accompanying drawing 1, the structural representation of autonomous operant conditioning reflex automat among the present invention;

T is discrete (1) constantly, and Ω is operation α _k(k=1,2 ..., n _Ω) set (2), S is state s _i(i=0,1,2 ..., n _S) set (3), δ is state transition function (4), Γ is " condition-operation " regular r _Ik(i ∈ 0,1,2 ..., n _S; K ∈ 1,2 ..., n _Ω) set (5), ε is orientation function (6), η is conditioned reflex law of learning (7), ψ is behavior entropy (8), s ₀Be original state (9).

Accompanying drawing 2, autonomous operant conditioning reflex automat AOC program flow diagram;

Accompanying drawing 3, the operation behavior probability curve of small white mouse;

Accompanying drawing 4, the operation entropy curve of small white mouse experiment;

Accompanying drawing 5, the operation behavior probability curve of machine dove;

Accompanying drawing 6, the operation entropy curve of machine dove experiment;

Accompanying drawing 7, double-wheel self-balancing robot is at erectility each operation behavior probability curve when being bias angle theta=0 °;

Accompanying drawing 8, double-wheel self-balancing robot be each the operation behavior probability curve during 0 °＜θ＜12 ° in the drift angle;

Accompanying drawing 9, double-wheel self-balancing robot each operation behavior probability curve during in bias angle theta=12 °;

Accompanying drawing 10, double-wheel self-balancing robot be each the operation behavior probability curve during-12 °＜θ＜0 ° in the drift angle;

Accompanying drawing 11, double-wheel self-balancing robot each operation behavior probability curve during in bias angle theta=-12 °;

Accompanying drawing 12, the operation entropy curve of double-wheel self-balancing robot experiment;

Embodiment

The small white mouse that one: one minimum system one of embodiment has learning ability, the small white mouse experiment of simulation Si Jinna.Describe the small white mouse experiment of Si Jinna briefly: put white mouse in the Skinner box into, and establish a lever, the structure of chest is got rid of all outside stimuluss as far as possible.Small white mouse can be free movable in case, when its presses lever, just has in the dish that a food falls into the chest below, and small white mouse just can be had food.Case has the action of a device recording animal outward.Small white mouse can learn ceaselessly to press lever, rewards by the action acquisition food of oneself.This experiment realizes the small white mouse experiment of Si Jinna by autonomous operant conditioning reflex automat.It is to press lever α that small white mouse has one of two operation behavior ₁, another is not press lever α ₂, i.e. operational set Ω={ α ₁, α ₂, probability is represented with p1, p2 respectively.Its state set S={s ₀, s ₁, s ₀The expression starvation, s ₁Represent non-starvation.Its working rule: Γ={ r _Ik(p) | p ∈ P; I ∈ 0,1}; K ∈ 0,1}}, " condition-operation " regular r at random _Ik(p): s _i→ α _k(p) mean that AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at state _iCondition under implementation and operation α _kProbable value.Its state transition function: δ: S (t) * Ω (t) → S (t+1), concrete condition is:

s ₀×p ₁→s ₁，s ₀×p ₂→s ₀，s ₁×p ₁→s ₁，s ₁×p ₂→s ₀。Its orientation function: ε: S → E={ ε _i| i=0,1}, ε _i=ε (s _i) ∈ E is state s _iThe orientation value of ∈ S, definition simultaneously $\mathrm{\&Delta;}=a\&times;{\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}\&times;(1-p_1).$ Wherein a is a learning rate,

It is the increment of orientation value.The probability of two behaviors of initial time all is 0.5, as long as small white mouse presses lever just can obtain award, press the probability of lever also to increase simultaneously to be next constantly small white mouse select the possibility increase of pressure lever, its probability is according to the operant conditioning reflex law of learning

Upgrade, through study once and again, small white mouse selects to press the Probability p of lever ₁Increasing.The learning rate a=0.01 of this experiment, after the study through 668 steps, small white mouse association presses lever to obtain food, can find out easily from accompanying drawing 3, and small white mouse is pressed the Probability p of lever ₁Finally trend towards 1.In the process of experiment, according to the formula of the operation entropy that defines $\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i)$ Calculated each operation entropy constantly, As time goes on the operation entropy ψ (t) of AOC is more and more littler and trend towards minimum when t → ∞, sees accompanying drawing 4, illustrates that so AOC operation entropy ψ (t) is a convergent.AOC is a self-organizing system based on Si Jinna operant conditioning reflex theory, has self study and adaptation function.The process of system's self-organization is the process of absorption information, is the process of drawing negentropy, is the process that removes uncertainty.Since the convergence of verified AOC operation entropy ψ (t) has so also just been illustrated the self-organization characteristic of AOC.

The concrete implementation step of this experiment is as follows:

(1) initialization: t=0 is set, the original state s of given AOC (0) at random, given learning rate a=0.01, given initial operation Probability p _Ik(0)=0.5 (i=0,1; K=1,2}; Given stop time T _f=1000;

(2) selection operation: according to the regular Γ={ r among " condition-operation " regular collection Γ among the operational set Γ _Ik(p) | p ∈ P; I ∈ 0,1}; K ∈ 1,2}}, " condition-operation " regular r at random _Ik(p): s _i→ α _k(p), promptly AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at 0 state _iCondition under implementation and operation α _kProbable value, select the AOC state to be in operation α (t) the ∈ Ω of s (t) ∈ S randomly;

(3) in implementation and operation: the t moment, AOC is in state s (t) ∈ S and implements operation α (t) the ∈ Ω that previous step has been chosen, and current state is according to δ: S (t) * Ω (t) → S (t+1), and concrete condition is:

s ₀* p ₁→ s ₁, s ₀* p ₂→ s ₀, s ₁* p ₁→ s ₁, s ₁* p ₂→ s ₀Shift;

(4) observer state: according to the state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), though the state transitions process is known or unknown, the result of its state transitions can observe fully, and { 0,1} makes s (t+1)=s promptly j ∈ _j

(5) operant conditioning reflex: at t moment implementation and operation, not only the state of AOC shifts, and it each operates in next enforcement probability constantly and also change, then according to the operant conditioning reflex law of learning

Regulate working rule r _Ik(p) the enforcement Probability p ∈ P of ∈ Γ.T is s (t)=s constantly _iAnd α (t)=α _k, t+1 evolutionary operator probability is constantly complied with so

Upgrade.Wherein,

And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right);$

(6) calculating operation entropy: according to the formula of the operation entropy that defines

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i).$ Calculate t operation entropy constantly, wherein, p (s _i) be AOC state s _iThe probability of occurrence of ∈ S is at t value constantly, p (α _k| s _i) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly.

(7) recurrence shifts: if t+1≤T _f, t=t+1 also repeats (2)-(7) so;

(8) as t+1＞T _fIn time, shut down.

Embodiment two: have the machine dove of learning ability, the pigeon experiment of simulation Si Jinna.Obtain food (positive reinforcement stimulation) when the machine dove pecks red button in this experiment, do not have any stimulation when pecking yellow button, shock by electricity when pecking blue buttons (negative reinforcement stimulation), pigeon pecks redly during beginning, and yellow and blue three buttons are at random.After a period of time, pigeon peck get red button number of times apparently higher than pecking the number of times of getting other two buttons.Be the autonomous operant conditioning reflex automat of one 3 operation of machine dove definition, 3 states, its operational set Ω={ α ₀, α ₁, α ₂, its element is respectively to peck red button α ₀, peck yellow button α ₁With peck blue buttons α ₂, probability is represented with p0, p1, p2 respectively.State set S={s ₀, s ₁, s ₂, i.e. Zero Hunger state (non-starvation) s ₀, semistarvation state s ₁, starvation s ₂, the state transitions rule is:

δ(s ₀×α ₀)＝s ₀ δ(s ₀×α ₁)＝s ₁ δ(s ₀×α ₂)＝s ₁

δ(s ₁×α ₀)＝s ₀ δ(s ₁×α ₁)＝s ₂ δ(s ₁×α ₂)＝s ₂

δ(s ₂×α ₀)＝s ₁ δ(s ₂×α ₁)＝s ₂ δ(s ₂×α ₂)＝s ₂

Show as following table 1 with form.Its orientation function: ε: S → E={ ε _i| i=0, ± 0.5, ± 1}, ε _i=ε (s _i) ∈ E is state s _iThe orientation value of ∈ S, definition simultaneously $\mathrm{\&Delta;}=a\&times;{\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}\&times;(1-p_0).$ Its orientation: s ₀→ s ₀: zero orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{00}=0);$ s ₀→ s ₁: zero orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{01}=0);$ s ₁→ s ₀: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{10}=0.5);$ s ₁→ s ₂: negative orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{12}=-0.5);$ s ₂→ s ₁: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{21}=1.0);$ s ₂→ s ₂: negative orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{22}=-1.0).$ According to the operant conditioning reflex law of learning

Current operation is awarded $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}>0)$ The time correspondence the enforcement probability tend to increase corresponding the reducing of enforcement probability of other operations; Current operation is awarded $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=0)$ The time all operations probability all constant; Current operation is awarded $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}<0)$ The time correspondence the enforcement probability tend to reduce the corresponding increase of enforcement probability of other operations.The initial probability of each operation all is 1/3, study through general 5000 steps, the machine dove substantially only pecks red button, do not peck red button and blue buttons, from accompanying drawing 5 as can be seen the machine dove Probability p 0 of pecking red button trend towards 1, peck the Probability p 1 of yellow button and the Probability p 2 of blue buttons and all trend towards 0.

The state transitions of table 1 machine dove

In the process of experiment, each is constantly all according to the formula of the operation entropy of definition

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i)$ Calculated the operation entropy, As time goes on the operation entropy ψ (t) of AOC is more and more littler and trend towards minimum when t → ∞, sees accompanying drawing 6, illustrates that so AOC operation entropy ψ (t) is a convergent.AOC is a self-organizing system based on Si Jinna operant conditioning reflex theory, has self study and adaptation function.The process of system's self-organization is the process of absorption information, is the process of drawing negentropy, is the process that removes uncertainty.Since the convergence of verified AOC operation entropy ψ (t) has so also just been illustrated the self-organization characteristic of AOC.

The concrete implementation step of this experiment is as follows:

(1) initialization: t=0 is set, the original state s of given AOC (0) at random, given learning rate a=0.01, given initial operation Probability p _Ik(0)=1/3 (i=0,1,2; K=0,1,2); Given stop time T _f=5000;

(2) selection operation: according to " condition-operation " regular collection among the operational set Γ

Γ={ r _Ik(p) | p ∈ P; I ∈ 0,1,2}; K ∈ 0,1,2}}, the regular r of " condition-operation " at random _Ik(p): s _i→ α _k(p), promptly AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at its state _iCondition under implementation and operation α _kProbable value, select the AOC state to be in operation α (t) the ∈ Ω of s (t) ∈ S randomly;

(3) in implementation and operation: the t moment, AOC is in state s (t) ∈ S and implements operation α (t) the ∈ Ω that previous step has been chosen, and current state δ: S (t) * Ω (t) → S (t+1) shifts, and concrete condition is:

δ(s ₀×α ₀)＝s ₀ δ(s ₀×α ₁)＝s ₁ δ(s ₀×α ₂)＝s ₁

δ(s ₁×α ₀)＝s ₀ δ(s ₁×α ₁)＝s ₂ δ(s ₁×α ₂)＝s ₂

δ(s ₂×α ₀)＝s ₁ δ(s ₂×α ₁)＝s ₂ δ(s ₂×α ₂)＝s ₂

(4) observer state: according to the state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), though the state transitions process is known or unknown, the result of its state transitions can observe fully, promptly exists

{ 0,1,2} makes s (t+1)=s to j ∈ _j

(5) operant conditioning reflex: at t moment implementation and operation, not only the state of AOC shifts, and it each operates in next enforcement probability constantly and also change, then according to the operant conditioning reflex law of learning Regulate working rule r _Ik(p) the enforcement Probability p ∈ P of ∈ Γ.T is s (t)=s constantly _iAnd α (t)=α _k, t+1 evolutionary operator probability is constantly complied with so

Upgrade.Wherein,

And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right);$

(6) calculating operation entropy: according to the formula of the operation entropy that defines

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i).$ Calculate t operation entropy constantly, wherein, p (s _i) be AOC state s _iThe probability of occurrence of ∈ S is at t value constantly, p (α _k| s _k) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly.

(7) recurrence shifts: if t+1≤T _f, t=t+1 also repeats (2)-(7) so;

(8) as t+1＞T _fIn time, shut down.

Embodiment three: the Balance Control that realizes double-wheel self-balancing robot by autonomous operant conditioning reflex automat.The two-wheeled vertical type robot is move left and right on the level land freely.When the drift angle exceeds ± 12 ° of robots meeting out of trim.The state set of She Ji AOC automat is exactly the robot drift angle for this reason, comprise 6 state: θ=0 °, 0 °＜θ＜12 °, θ=12 ° ,-12 °＜θ＜0 °, θ=-12 °, | θ |＞12 °, use s respectively ₀, s ₁, s ₂, s ₃, s ₄, s ₅, s ₆Represent, therefore, state set S={s ₀, s ₁, s ₂, s ₂, s ₃, s ₄, s ₅, s ₆, }.Its operational set Ω={ α ₀, α ₁, α ₂, comprise not mobile α ₀, α moves right ₁, be moved to the left α ₂Its state transitions rule is as follows:

δ(s ₀×α ₀)＝s ₀ δ(s ₀×α ₁)＝s ₃ δ(s ₀×α ₂)＝s ₁

δ(s ₁×α ₀)＝s ₂ δ(s ₁×α ₁)＝s ₀ δ(s ₁×α ₂)＝s ₂

δ(s ₂×α ₀)＝s ₅ δ(s ₂×α ₁)＝s ₁ δ(s ₂×α ₂)＝s ₅

δ(s ₃×α ₀)＝s ₄ δ(s ₃×α ₁)＝s ₄ δ(s ₃×α ₂)＝s ₀

δ(s ₄×α ₀)＝s ₅ δ(s ₄×α ₁)＝s ₅ δ(s ₄×α ₂)＝s ₃

See Table 2.Its orientation function: ε: S → E={ ε _i| i=0, ± 0.5, ± 1}, ε _i=ε (s _i) ∈ E is state s _iThe orientation value of ∈ S, simultaneously $\mathrm{\&Delta;}=a\&times;{\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}\&times;(1-p_{\mathrm{ik}}).$ Its orientation: s ₀→ s ₀: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{00}=0);$ s ₀→ s ₃: zero orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{03}=0);$ s ₀→ s ₁: zero orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{01}=0);$ s ₁→ s ₀: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{10}=1.0);$ s ₁→ s ₂: negative orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{12}=-0.5);$ s ₂→ s ₁: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{21}=1.0);$ s ₂→ s ₅: negative orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{25}=-1.0);$ s ₃→ s ₄: negative orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{34}=-0.5);$ s ₃→ s ₀: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{30}=1.0);$ s ₄→ s ₅: negative orientation $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{45}=-1.0);$ s ₄→ s ₃: just be orientated $({\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{43}=1.0).$ P wherein _IkThe expression robot is in state s _iFollowing implementation and operation a _kProbability.According to the operant conditioning reflex law of learning

Its probability is brought in constant renewal in.Initial probability all is 1/3, through the study in general 1500 steps, robot all can keep the balance of self with the operation that chooses near 1 probability under every kind of state, the operation that it generally can both choose under preceding 5 kinds of states allows θ trend towards 0 °, from accompanying drawing 7-11 as can be seen.In the process of experiment, each is constantly all according to the formula of the operation entropy of definition

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i)$ Calculated the operation entropy, As time goes on the operation entropy ψ (t) of AOC is more and more littler and trend towards minimum when t → ∞, sees accompanying drawing 12, illustrates that so AOC operation entropy ψ (t) is a convergent.AOC is a self-organizing system based on Si Jinna operant conditioning reflex theory, has self study and adaptation function.The process of system's self-organization is the process of absorption information, is the process of drawing negentropy, is the process that removes uncertainty.Since the convergence of verified AOC operation entropy ψ (t) has so also just been illustrated the self-organization characteristic of AOC.

The state transitions of table 2 double-wheel self-balancing robot and orientation mechanism

The concrete implementation step of this experiment is as follows:

(1) initialization: t=0 is set, the original state s of given AOC (0) at random, given learning rate a=0.01, given initial operation Probability p _Ik(0)=1/3 (i=0,1,2; K=0,1,2); Given stop time T _f=1500;

(2) selection operation: according to " condition-operation " rule among the operational set Γ

Γ={ r _Ik(p) | p ∈ P; I ∈ 0,1,2,3,4}; K ∈ 0,1,2}}, the regular r of " condition-operation " at random _Ik(p): s _i→ α _k(p), promptly AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at its state _iCondition under implementation and operation α _kProbable value, select the AOC state to be in operation α (t) the ∈ Ω of s (t) ∈ S randomly;

(3) in implementation and operation: the t moment, AOC is in state s (t) ∈ S and implements operation α (t) the ∈ Ω that previous step has been chosen, current state δ: S (t) * Ω (t) → S (t+1), and concrete condition is:

δ(s ₀×α ₀)＝s ₀ δ(s ₀×α ₁)＝s ₃ δ(s ₀×α ₂)＝s ₁

δ(s ₁×α ₀)＝s ₂ δ(s ₁×α ₁)＝s ₀ δ(s ₁×α ₂)＝s ₂

δ(s ₂×α ₀)＝s ₅ δ(s ₂×α ₁)＝s ₁ δ(s ₂×α ₂)＝s ₅

δ(s ₃×α ₀)＝s ₄ δ(s ₃×α ₁)＝s ₄ δ(s ₃×α ₂)＝s ₀

δ(s ₄×α ₀)＝s ₅ δ(s ₄×α ₁)＝s ₅ δ(s ₄×α ₂)＝s ₃

Shift;

(4) observer state: according to the state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), though the state transitions process is known or unknown, the result of its state transitions can observe fully, promptly exists

{ 0,1,2,3,4} makes s (t+1)=s to j ∈ _j

(5) operant conditioning reflex: at t moment implementation and operation, not only the state of AOC shifts, and it each operates in next enforcement probability constantly and also change, then according to the operant conditioning reflex law of learning Regulate working rule r _Ik(p) the enforcement Probability p ∈ P of ∈ Γ.T is s (t)=s constantly _iAnd α (t)=α _k, t+1 evolutionary operator probability is constantly complied with so

Upgrade.Wherein,

And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right);$ Here the operation of the optimum under every kind of state is different, so must calculate the probability of every kind of different operating correspondence under the state, one has 15 probability.

(6) calculating operation entropy: according to the formula of the operation entropy that defines

$\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i).$ Calculate t operation entropy constantly, wherein, p (s _i) be AOC state s _iThe probability of occurrence of ∈ S is at t value constantly, p (α _k| s _i) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly.

(7) recurrence shifts: if t+1≤T _f, t=t+1 also repeats (2)-(7) so;

(8) as t+1＞T _fIn time, shut down.

Claims (2)

1, a kind of autonomous operant conditioning reflex automat is one nine tuple hereinafter to be referred as AOC:

$\mathrm{AOC}=<t,\mathrm{\&Omega;},S,\mathrm{\&Gamma;},\mathrm{\&delta;},\mathrm{\&epsiv;},\mathrm{\&eta;},\mathrm{\&psi;},s_0>$

Wherein

(1) discrete time of AOC: t ∈ 0,1,2 ..., n _t, t=0 is the initial moment of AOC;

(2) functional symbol of AOC set: Ω={ α _k| k=1,2 ..., n _Ω, α _kK functional symbol for AOC;

(3) state set of AOC: S={s _i| i=0,1,2 ..., n _S, s _iI state for AOC;

(4) working rule of AOC set: Γ={ r _Ik(p) | p ∈ P; I ∈ 0,1,2 ..., n _S; K ∈ 0,1,2 ..., n _Ω, " condition-operation " regular r at random _Ik(p): s _i→ α _k(p) mean that AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at state _iCondition under implementation and operation α _kProbable value, P represents p _IkSet;

(5) state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), AOC t+1 state s (t+1) ∈ S is constantly determined by t state s (t) ∈ S and t operation α (t) ∈ Ω constantly constantly, and is irrelevant with state and the operation of its t before the moment; The determined state transitions process of δ is known or unknown, but the result of its state transitions can observe;

(6) orientation function of AOC: ε: S → E={ ε _i| i=0,1,2 ..., n _S, ε _i=ε (s _i) ∈ E is state s _iThe orientation value of ∈ S;

(7) the operant conditioning reflex law of learning of AOC:

The operant conditioning reflex mechanism that simulation is biological is regulated working rule r _Ik(p) the enforcement probability of ∈ Γ supposes that t state constantly is s (t)=s _i, implementation and operation α (t)=α _kState s (t+1)=s that ∈ Ω, t+1 observe constantly _j, t+1 evolutionary operator probability is constantly complied with so

Upgrade; Here wherein, p _Ik(t) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly; p _Ik(t+1) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t+1 value constantly; And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value;

Be monotonic increasing function, satisfy

And if only if x=0; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right),$ Here u represents 0 to n _ΩBetween be not equal to any one numerical value of k,

Expression AOC state is in s _iImplementation and operation α under the condition of ∈ S _uThe probability sum of ∈ Ω is in t value constantly, and v represents 0 to n _ΩBetween be not equal to all numerical value of k; p _Iu(t) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _uThe probability of ∈ Ω is at t value constantly, p _Iu(t+1) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _uThe probability of ∈ Ω is in t+1 value constantly;

(8) the operation entropy of AOC: ψ: P * E → R ⁺, R ⁺Be positive set of real numbers, AOC is in s at t operation entropy ψ (t) expression t moment state constantly _iOperation entropy sum under the condition:

$\mathrm{\&psi;}\left(t\right)=\mathrm{\&psi;}\left(\mathrm{\&Omega;}\left(t\right)\right|S)=\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i{\mathrm{\&psi;}}_i\left(t\right)=\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right){\mathrm{\&psi;}}_i\left(\mathrm{\&Omega;}\left(t\right)\right|s_i)$

It is in state s (t)=s constantly by t _iEvolutionary operator probability set under the condition and orientation function set decision; ψ _i(t) be that AOC is in state s _iOperation entropy under the condition:

${\mathrm{\&psi;}}_i\left(t\right)={\mathrm{\&psi;}}_i\left(\mathrm{\&Omega;}\left(t\right)\right|s_i)=-\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{{\log }_{2}p}_{\mathrm{ik}}=-\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({{\mathrm{\&alpha;}}_k|s}_i\right){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i)$

Know that operation entropy and weighted sum under each state just can draw AOC at t operation entropy constantly $\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i);$ Wherein, p (s _i) be AOC state s _iThe probability of occurrence of ∈ S is at t value constantly, p (α _k| s _i) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly;

(9) initial state of AOC: s ₀=s (0) ∈ S.

2, autonomous operant conditioning reflex automat AOC according to claim 1 is characterized in that it complies with following program step and recursively move:

(1) initialization: t=0 is set, the original state s of given AOC (0) at random, given learning rate a, given initial operation Probability p _Ik(0)=1/n _Ω(i=0,1,2 ..., n _SK=1,2 ..., n _Ω); Given stop time T _f

(2) selection operation: according to the regular r among " condition-operation " regular collection Γ among the operational set Γ _Ik(p): s _i→ α _k(p), promptly AOC is in s at its state _iUnder the condition of ∈ S according to Probability p ∈ P implementation and operation α _k∈ Ω, p=p _Ik=p (α _k| s _i) be that AOC is in s at its state _iCondition under implementation and operation α _kProbable value, select the AOC state to be in operation α (t) the ∈ Ω of s (t) ∈ S randomly;

(3) in implementation and operation: the t moment, AOC is in state s (t) ∈ S and implements operation α (t) the ∈ Ω that previous step has been chosen, and δ (s (t), α (t))=δ (s takes place to shift current state _i, α _k);

(4) observer state: according to the state transition function of AOC: δ: S (t) * Ω (t) → S (t+1), the result of state transitions can observe fully, promptly exist j ∈ 0,1,2 ..., n _SMake s (t+1)=s _j

(5) operant conditioning reflex: at t moment implementation and operation, not only the state of AOC shifts, and it each operates in next enforcement probability constantly and also change, then according to the operant conditioning reflex law of learning

Regulate working rule r _Ik(p) the enforcement Probability p ∈ P of ∈ Γ; T is s (t)=s constantly _iAnd α (t)=α _k, t+1 evolutionary operator probability is constantly complied with so

Upgrade; Wherein,

And 0≤p _Ik+ Δ≤1; ${\stackrel{\&RightArrow;}{\mathrm{\&epsiv;}}}_{\mathrm{ij}}=\mathrm{\&epsiv;}\left(s_j\right)-\mathrm{\&epsiv;}\left(s_i\right)$ It is the increment of orientation value; A is a learning rate; $\mathrm{\&xi;}=p_{\mathrm{iu}}\left(t\right)/\underset{v\&NotEqual;k}{\mathrm{\&Sigma;}}{p}_{\mathrm{iv}}\left(t\right);$

(6) calculating operation entropy: according to the formula of the operation entropy that defines $\mathrm{\&psi;}\left(t\right)=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}{p}_i\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}{p}_{\mathrm{ik}}{\log }_2{p}_{\mathrm{ik}}=-\underset{i=0}{\overset{n_S}{\mathrm{\&Sigma;}}}p\left(s_i\right)\underset{k=1}{\overset{n_{\mathrm{\&Omega;}}}{\mathrm{\&Sigma;}}}p\left({\mathrm{\&alpha;}}_k\right|s_i){\log }_{2}p\left({\mathrm{\&alpha;}}_k\right|s_i);$ Calculate t operation entropy constantly, wherein, p (s _i) be AOC state s _iThe probability of occurrence of ∈ S is at t value constantly, p (α _k| s _i) be that the AOC state is in s _iImplementation and operation α under the condition of ∈ S _kThe probability of ∈ Ω is in t value constantly;

(7) recurrence shifts: if t+1≤T _f, t=t+1 also repeats (2)-(7) so;

(8) as t+1＞T _fIn time, shut down.

Images