CA1287175C - Back propagation system - Google Patents
Back propagation systemInfo
- Publication number
- CA1287175C CA1287175C CA000561224A CA561224A CA1287175C CA 1287175 C CA1287175 C CA 1287175C CA 000561224 A CA000561224 A CA 000561224A CA 561224 A CA561224 A CA 561224A CA 1287175 C CA1287175 C CA 1287175C
- Authority
- CA
- Canada
- Prior art keywords
- input
- output
- synaptic
- error
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Abstract
INVENTION: Back Propagation System INVENTOR: Max Stanford Tomlinson Jr.
ABSTRACT OF OF THE DISCLOSURE
A continuous-time information processing system having an input layer, which receives an input stimulus vector, a hidden layer which processes the input stimuli vector and passes the information to an output layer which provides an output vector and receives an error stimulus vector. Each layer consists of one or more processing elements. Two processing elements may be coupled together by a synaptic element. Each synaptic element has a unique transfer function. Each synaptic element provides a forwards transfer of information and a backwards transfer of error between processing elements. The output value of each process-ing element is dependent upon a function of the information transferred to the processing element by the synaptic elements transferring information to it. The error value of each hidden processing element is similarly dependent upon a different function of the error transferred to the hidden processing element by the synaptic elements transferring error thereto. The system further includes a means for adjusting the transfer characteristics of each synaptic elements.
ABSTRACT OF OF THE DISCLOSURE
A continuous-time information processing system having an input layer, which receives an input stimulus vector, a hidden layer which processes the input stimuli vector and passes the information to an output layer which provides an output vector and receives an error stimulus vector. Each layer consists of one or more processing elements. Two processing elements may be coupled together by a synaptic element. Each synaptic element has a unique transfer function. Each synaptic element provides a forwards transfer of information and a backwards transfer of error between processing elements. The output value of each process-ing element is dependent upon a function of the information transferred to the processing element by the synaptic elements transferring information to it. The error value of each hidden processing element is similarly dependent upon a different function of the error transferred to the hidden processing element by the synaptic elements transferring error thereto. The system further includes a means for adjusting the transfer characteristics of each synaptic elements.
Description
lZ8717S
, . . . ., .. ~. .
INVENTION: Back Prop~gation System INVENTOR: Max Stanford Tomlinson Jr.
BACKGROl[JND OF TH:13 IN~ NTION
I. Field of Invention The present invention is a continuous-time adaptive neural system which learns to translate a set of input signals (input stimulus vectors) to a set of output signals (output vectors) based on a set of expected output signals (target vectors) through the back propagation of error signals. The error signals (error vector) are ~2~ 7S
a function of the actual and the e~cpected OlltpUt ~ignals.
II. Background.~rt .~s used herein. a neural s~stem produces an output v ector uhich is a func-tion of an input vector. The mapping function bet~een the input and output vectors is learned. Ideally. the output vector should match the target vector: the difference between the two vectors can be used in an adjustment mechanism.
Original theoretical approaches tot~ards neural networks are based upon the idea that when t~o neurons in the brain are active there is a correlation between them. One earl-; rule developed bv D. O. Hebb is described in his book "The Organization of Behaviour", ~ iley, 1949. The Hebbian rule states that hen two neurons are firing simultaneousl~ an association link between them is strengthened. Accordin~l-. the ne.Yt time either of the two neurons fires. the other one is more probable to fire also. Houever. the Hebbian rule is not a sufflcientmodel to explain the learning process. I:~nder the Hebbian rule. the connection strengths between neurons grow ~ithout bound. If maximums are placed on the connection strengths. these maxirIlums are always reached.
~ ubsequently. the Perceptron ~lodel was developed b~ Frank Rosenblatt.
and is discussed in his book "Principles of ~eurod~namics". SparTan. 196~. The Perceptron Model was originally believed po-~rerful enough to enable a machine to learn in a human-like manner.
The Perceptron ~odel includes input. hidden and output layers: each compriscd of one or more processing elements. In response to input stimuli. the input layer provides information to the hidden la~er. ~imilarl~. the hidden la~er pro- ides information to the output la.~er. Corlnections betv- een the input andhidden processing elements are fi~sed: connections be~-een the hidden and outputprocessing elements are adjustable.
In the Perceptron ~Iodel. if the inp-lts are boolean (i.e. either zero or one).
then the intended purpose of the hidden la~:er is to extract some kind of features from the input data. Ho~ever. if the inputs to the ~Iodel are continuous numbers(i.e. ha~ing more than t~o distinct ~alues. rather than just two boolean values).
then the hid(len layer is not used. Instead. Ihe outputs of the input layer are con-71~5 nected directl~ to the inputs of the output la~er.
In the Perceptron ~Iodel. all le~rning takes place in the output la~ er.I, nder the Perceptron ~Iodel many problems have been experimentaliy and mathematically shown to be representable bv connection strengths between la~ers.Rosenblatt s Perceptron Learning .~lgorithm enables a neural network to find a solution if there exists a representation for that problem by some set of connection strengths. Rosenblatt's Perceptron Convergence Proof is a well l;nown mathemat-ical proof that a Perceptron System will find a solution if it exists, In operation, the Perceptron Model modifies the strengths of the weighted connections between the processing elements, to learn an appropriate output response corresponding to a particular input stimulus ~ector. The n1odification of the connection weights occurs when an incorrect output response is given. This modification of the weights changes the transfer of information from the input to output processing elements so that e-entuall~ the appropria~e output response ill be pro-~ided. However. through experimentation, it was discovered that thePerceptron ~Iodel was unable to learn all possible functions. It uas hoped thal these unlearnable functions ere onl,v pathological cases. analogous to certainproblems that humans cannot solve. This is not the case. Perceptron ~ tems cannot represent and learn some very simple problems that humans are able to learn and represent An example of a problem that the Perceptron ~lodel is unable to represen~
(without 2~'~ hidden processing elements. uhere .~- is the number of input nodes), and therefore cannot learn. is the parit~ or "exclusive-or" boolean function. Toperform such a problem (u ith feuer than ~ ~ hidden processing elements) a system would require two layers of modifiable ueights. The Perceptron System cannot properly adjust more than one layer of modifiable weights. It - as speculated that no learning mechanism for a s-stem uith multiple layers of modifiable weights would ever be discovered because none existed !2~1insky & Papert. 1969. in "Per-ceptrons" ) .
(The problem with using ~ ~ hidden units is three-fold. First. since the hid-den units. in the Perceptron ~Iodel. do not adapt. ali the units must be present.
regardless of the function which needs to be learned. so that all functions can be learned. Second, the number of units required gro- s phenomenall.~: for e:;ample, ~Z~7175 2V4 is approximately 17 bi]lion, more neurons than in a human brain: this means that the largest parity problem the hurnan brain could solve, if wired in this manner, would have at most 32 inputs. Third. the s~-stem would not generalize:
given two input/output vector pairs near one another, one trained and the other not, the system should be able to interpolate the answer from the first: w,ith alarge number of hidden units. it has been e~;perimentally shown that this is notthe case.) Almost ~ll adaptive neural systems share several features in common. Typi-cally the processing elements of all systems have an output which is a function of the sum of the weighted inputs of the processing element. Almost all systems have a single layer of modifiable weights which affect the data transferred from the in-put to the output of the system.
The etolution of adapti~e neural systems took a dramatic step forward with the det elopment of an algorithm called "Back Propagation". This algorithm is fully described in the reference text "Parallel Distributed Processing. the ~Ii-crostructure of Cognition". Rumelhart. Hinton.& U illiams. ~lIT Press, 1986.
A back propagation system typically consists of three or more la,vers. each layer consisting of one or more processing elements. In one basic example, the system is comprised of an input layer, at least one hidden layer and an output layer. Each layer contains arbitrary. directed connections from the processing ele-ments in the inpùt layer to the hidden layer, and from the hidden layer to the output layer. There are no connections from processing elements to processing ele-ments in the same layer nor connections from the output to the hidden layer nor from the hidden to the input layer: i.e. there are no cycles (loops) in the connec-tion graph. (There are hypothesized mechanisms for networks t ith cycles in them~ but they are not being scrutinized herein.) In the Perceptron ~Iodel the idea of error ~as introduced. In a back propa-gation system, at each output processing element of the netuork. the error is quite easily realized. The error is typically the difference between an e~pected ~alue and the output ~alue. This error is used to modify the strength of the connection between a processing element and the output processing element. Ideally. this reduces the error between the e~cpected outpul and the ~-alue output by the pro-cessing element in response to the input. The Perceptron ~lodel lacks the ability 128~175 to allocate an error value to the hidden processing elements and therefore cannot adjust the weights of any connections not coupled to an output processing ele-ment. In a s~stem utilizing the Back Propagation algorithm. an error is assignedto the processing elements in hidden layers and th~ weights of the connections coupled to these hidden processing elements can be adjusted.
The derivation of the discrete time Back Propagation .~lgorithm is described in Chapter 8 of Parallel Distributed Processing, by Rumelhart et al, and is recounted here.
.~ weight associated with a connection is referred to as ~ji. The subscripts are used in the form ~to,from Hence, in the variable wji, i refers to the processing element from which data information is being received, and j refers to the process-ing element to which data information is sent. In the back propagation algo-rithm. a particular input stimulus vector is referred to collectivel~ b~ the v ariable p (for pattern). The elements of a particular output vector and particular target vector are referred to respectively as op; and Tpj, where j varies over the output processing elements. The Total Error of a s~stem is represented b~ the variable E The portion of the Error contributed bv a single input vector (one input pat-tern) is represented b~ the v ariable Fp The output of a processing element o;". in response to an inpur pattern p.
is is calculated by the following equation (uhich also defines the value netpj):
netpj = ~ u~j; op~ (1) (2) op~ = f(netpj) The techniques used b~ Back Propagation to minimize the Total Error is a varj-ant of Least ~Iean ~quared. The technique states that the total error is the square of the difference between the target vector and the output vector. Further-more, it is assumed that the total error for the s,vstem is the linear summation of the error for any individual pattern.
p (3) lZ~7175 Ep = 2 ~ (Tp,-Op,) ( ) In a Back Propagation network, error is minimized b~ adjusting the weights within the netuork. ~rhat is desired is to determine uhat amount to adjust a weight so that the error will be reduced. The following equation expresses that deslre:
a~ ~ aEp awji p awji The above expression can be expanded by the chain rule to get:
aE ~ aEp anetpl. (6) a~vji p k anetpk aWji anet k ~'e can expand the second component. a p bv noting Ihat netpk = ~ ulmom.
to get the following a(~ Wkmm) anetpk m ( ) a U~
It is eas~ to see that e~cept uhen m=i and ~=j, the abo-e is zero. Putting this back into the abo~e equation we get:
aE ~ aEp O (8) ~)'U ~i p anetpj ' a~p The first portion of the equation. a t- b~ e~cpansion using the chain rule.
gi~ es:
-~EC aEp ~pl (91 ~netpj j ~opl ~netp~
and a p can be simplified b~ recognizing Op/ = I(~-~etp~). B~ substiluting this in, the expression becomes:
' "~ " . .
~2~7175 I
aop, aJ(rle~pl) (10) anetp~ anetp~
t can nou be noted that af(netpl) is zero except when l=j; this gi--es us finallv ~netp j ~Opj af(netpj) = J'(netp;) (11) 3netpj anetpj and this can be substituted back in to get:
awj~ aoP J'(net aEp If we are examining an output node, the value of ao is readily apparent from the definition of Ep. as in:
Ep = 1 ~ (Tpj-Opj) (13) Partial differentiation of this expression with respect to Opj gives the following expression for output processing elements:
--= Tpj- Opj (l4!
Thus the error equation for an output value is:
a ~ = ( Tp j- Op~ ) J ' ( net p j) o~
The problem remains ~-~hal the error ~alue is for the hidden processing elements.
To determine this. Iet the definition of ~p, be:
aEp (16) Pj ~netpj From the expansion from above. ~-e see:
aEp = aEp J'(net ) anetp; aOpj ~xpanding by the chain rule on opj, ~, e get:
12~ f i~S
a~p aEp anetpk (18) aOp,, ~ an'etpk aOp, Expanding a ~ ~ b~ the definition netp~ u!k~ol. ue get:
a(~ U'klOpl) anetpk I (19) aOp,' aOp,, It is easv to see that the above is zero. e~cept u-hen l= j, so that we can state:
anetpk 3U~kjOp. (20) U!k ~
aOp~ aOpj ~ubs~ituting this back into the above equation. ue get:
U!k ~
By the definition of ~pJ~ we can then state:
a O ~ ( ~pk U'k j ) ( '' '~ ) Therefore. op~ for a hidden node can be expressed as:
~pj = I (netp.~)~ (pkU'kJ) ~'~3) k C o;nbining all the above elements together. we get:
( '~ ) , o p, and from this. the Total Error equation can 't>e forl1lulated:
~aE = ~ ~;" op, (-~
For an output processing element. ~P! i}: ('~G) ~p, = J'(netpJ) (Tpj-Op,) For a hidden processing element. ~p~ is:
~21~7175 = J (netpj) ~ (~SpkWkj) (21) k ~'o~ the change of the ~eight is set proportional to the above partial differentiation This is given by the follo- ing equation (28) ~,w~ opi The constant of proportionality (~) is the Learn Rate Experimentally, this constant has been found to be in the range of a to ver~ small, depending onthe number of u eights processing elements and patterns which are to be presented ~ ote that there is no guarantee that anS one particular veight change for a particular pattern uill decrease the total error it is actually quite probable that during one of the patterns the total error will increase, just that over all the pat-terns the total error should decrease In summary back propagation may be described as follo~-s On a forward pass of information through the netuor}; all of the processing element outputs are calculated b~ propagating the information input foruar(l through the net- ork i e from the input la, er to each hidden layer in turn and finall- to the output !a-er On a backuard pass i e frorn the output la~ er to the hidden la-ers each in reverse order from before and finally to the input layer all the errors are calcu-lated by propagating the associated errors back-vards through the net- ork FinallS~ all the ueights are changed according to the errors in the processing ele-ments above and the outputs of the processing elements belo-~
Prior art neural systems using the Back Propagation ~lgorithn-l ha-e been f'requently implemented on computer sS~stems These systems suffer se~ere limita-tions as the anlount of time required to perform the calculations is exorhitant ~s the size of the netuork (counting the processing elements) gro- s the com?uta-tional time gro-~ j exponentially (assurning a full- interconnected graph) It is therefore an object of the present invention to pro~ ide a ne- and improved adapti-e neural net- ork utilizing a modification of the Back Propaga-tion .~lgorithm to enable the system to adapt to a specified output response in response to a certain input stimUluc~
7:175 It is a further object of the in~ention to provide a continuous-time system, ie. the three phases described abo~ e are done simultaneousl~: all for~ ard propaga-tion. backward propagation and u eight updates proceed continuously in time.
It is a further object to pro~ide a neural system uherein the calculations are performed. not bv digital methods. but rather by analog methods.
It is a further object to provide a neural system for which the computa-tional time remains relati~elv constant as the network size increases.
III. Sumrnary of Invention An adapti-e. continuous'time information processing s-stem including an input laver having at least one input processing element. a hidden layer ha- ing at least one hidden processing element and an output la- r having at least one out-put processing element. Input synaptic elements are included ~vith each element coupling an input processing element tO a hidden node. The input synaptic ele-ments may couple an input processing element to more than one hidden process-ing elements. Output synaptic elements are also included with each element cou-pling a hidden processing element to an output node. The output synaptic ele-ments may couple a hidden processing element to more than one output process-ing element. The function of each input processing element is to receive an input stimulus. The input stimulus is transferred as information by the input synapticelement,s to each coupled hidden processing element. The information transferredby a coupling input synaptic element is transferred in accordance ~-ith a ueight-ing factor associated ~ith the coupling input synaptic element. Informatioll is also transferred from each hidden processing element to a coupled output processing element by a coupling output synaptic element. The information transferred b~ a coupling output synaptic element is transferred in accordance u ith another weighting factor associated with the coupling output synaptic element. The infor-mation output from the output processing element} is the output response of the system to the input stimulus. The output processing eiement also receives an error stimulus u hich is dependent upon the output response of the output processing element and an expected response. This error is propagated backwards through the output synaptic elements. hidden processing elements and input synaptic 121~7175 elements for modifying the respective weighting factors of the input and output synaptic elements. Back- ard propagation of the error through the system permitsthe system to adaptivel~ provide an e~;pected output response corresponding to an input stimulus.
IV. BRIE~ DE~CRIPTION OF DRAWINGS
The features. objects. and advantages of the present invention ~ ill become apparent from the following detailed description, taken in conjunction with the accompanying dra~ings, in which like reference numerals correspond throughout and wherein:
Figure 1 is a diagrammatic representation of a synaptic element of the present in~ention:
Figure 2 is a diagrammatic representation of a processing element of the present invention:
Figure 3 is a diagrammatic representation of a small. representative systeni of the present int ention having a single hidden processing element:
Figure 4 is a diagrammatic representation of a small. representative system of the present invention having a hidden s,vnaptic element:
Figure ~ is a diagrammatic representation of yet another embodiment of a s- stem of the present invention.
Figure 6 is a circuit block diagram of an implementation of the synaptic element of the diagrammatic representation of Figure 1.
Figure ~ is a circuit block diagram of an impiementation of the processing element of the diagrammatic representa~ion of Figure ''.
Figure ~ is a biock diagram of a circuit implementation of the syslem of Figure ~ using the circuits of Figures 6 and /:
Figures 9a and 9b are graphical representations of typical functions pro-vided by the function generators of the circuit of Figure 8:
12~7175 V DETAILED DESCRIPTION OF THE PREFERRED EMBODIME~TS
The Back Propagation .~lgorithm as originally developed and as described in the previous section is a discrete time algorithm. in that there is a forward pass.
a backwards pass and modification to the weights, and then a recycling. However.this is not an optimal implementation of the s~stem. There is an implicit assump-tion of linearit~ during these discrete time intervals. This is generally not a good assumption.
A better implementation of the system is for the network to run continu-ousl~, performing each of the operations simultaneously: this is the basis of what is called herein a continuous time s~stern. The follouing is a derivation of the con-tinuous time model of the Back Propagation .~lgorithm, as developed by this inventor One primary innovation is that time should be used to differentiate theError, to ascertain that the Error is monotonically decreasing. Hence, instead of taking the derivative of the Error with respect to any particular weight. the derivative is taken uith respect to time. The chain rule can then be applied tak-ing the partial differentiation with respect to 1Lji.
dE ~ aED du~j; (29) dt ~ dt Repeating equation (2~). as derived in the discrete time algorithm:
~E = ~;, aEp = ~, 6pjop, his can then be replaced into equation (~9) to give:
dE = ~ p jp,) dt To ensure that the derivative of the Error is nlonotonicall~ decreasing. the sign of ddt must be negati- e. The onl.v wa. to do this is to ensure that the sign of dti is the opposite sign of ~ ~pjop,.
By arbitrarily setting lZ8717S
p~pi (~2) this constraint is satisfied. b~ giving us:
dE = _ ~ pj pt ) Since the derivati-e of the Error is monotonicall,v decreasing, the s,vstem will con-verge at some final error alue. As derived. a s~stem is not guaranteed to converge to a zero Error. Experimental results show that a s,vstem generall,v will converge to zero error if the problem to be solved is representable b,v the net~ ork. If the s~rstem does not converge to a small error. or does not reliably converge, adding a small nurnber of additional processing elements and connections will lead to con-~ergence.
One further object of interest is that there exists a mechanism to maintain onl~ positive weight values. The Error remains monotonic decreasing u-hen an e xponential term, W~ is multiplied into the dtj function of equation 13" ) .
~:pecificall,~:
~ WR~ ~ ~p~op !3~!
and dt = ~ W~ p;P )' 13~) It uill become clear from the following detailed descriptions of the drau-ings how thesè eqllationC have been reduced to practice.
.~ neural s!stem utilizing backwards error propagation can be represented bv two kinds of elements: processing elements and s,vnaptic elements.
.~ s~naptic element connects two processing elements and ils primar~ func-tion is to store the connection strength.
A processing element receives a net data and a net error signal. and pro-duces a data and an error signal~ uhich are functions of the tuo received signals.
The functions can be mathematicall~ expressed as:
12~ S
(36) Output = J( .~etInput ) (3l ) E;rror~ = J ( .~etInp~ ) x .~\~et~rror~
.~ s~naptic element recei-es a data and an error sic~nal and produces a net dataand a net error signal ~hich are a function of the t~o recei~ed signals. The func-tions can be mathematicall~- expressed as:
.~etInputi = ~ U eightjj ~ Output~ (38) .~ietError~ = ~ Il eightji~ O~tput (39) dti = Learn_Rateji~ rror~ Outputi (~) The exact implementational details ~ ill become clear when the full s~-stem circui-tr~ is described.
The layout of the continuous time back propagation algorithrn is essen-tiall~, topologicall~ identical to that of the discrete time algorithm. Ho~-e~er. for bre-it~ of uords, there is a more elegant mechanism utilized to describe the inter-connectiolls and layout of the processing an~:l synaptic elements.
There are three layers of processing elements. onl~: the input. the hidden and the ourput layers. Each layer. as before. consists of one or more processingelements. There ma~ be connections from the input to the hidden larer (inpu~
synaptic elemeIIts). from the hidden to the output la~ er (output sy naptic ele-ments). from the input to the output la!er (direct s-r.aptic elements). and fromhid(1en processing elements to other hidden processing elements (hidden synapticelerrlents). There is a large constraint placed on hidden s~-naptic elements: if the hidden processing elements are numbered 1 to ~\. a s~-naptic element ma! onl~
connect from a io~er numbered hidden processing element to a higher numbered processing element: remernber that Ihe directionality of a synaptic elernent is important .
This three-la~er descriplion actuall- produces ail possible layered en~iron-ments: it describec an acyclic graph.
Figure l is a s~mbolic diagrammatic representation of a ~ynaptic Elemen~
denoted generall~ as 10. ~;~ naptic Element 10 has four !ines: ~ o input lines.
Dat.a 11a and Error 1~a: and t--o output lines. .~et Dat<I 13a and ~et E;rror l"a.
~Z87~75 Transfer function 1;~ is s~ mbolicl~ shoun to operate on and betv-een the data and error lines. The arrov- 16 signifies that the transfer function 1~ is ~ariable. Indi~
cator 1/ designates the ~\'et Data line.
Figure 2 is a s,tmbolic diagrammatic representation of a Processing Ele-ment denoted generall,~ as ~0. Processin(r Element 20 has four lines: two outputlines. Data 11b and Error 14b and tuo input lines, ~'et Data 13b and ~iet Error 12b. Internal drawings 28 are symbolic of the information flo~ within a Process-ing E;lement and will be explained in detail further in the description.
Figure 3 is a diagrammatic representation of a near minimal system of the present in~ention. Three processing elements. input processing element 20a, hid-den processing element 20b, and output processing element 20c are connected by three synaptic elements, input synaptic element 10a, direct synaptic element 10b~
and output s~ naptic element 10c. . The data. error. net data and net error lines of the synaptic elements are connected to the similarly named lines of the collnected processing elemenls. as diazrammed.
The input vector is transferred to the system via Input Stimulus line 31 v~hich is connect,ed tO the net data line of input processing element 20a. The input vector is comprised of onl- one element in this e:;ample. as there is onl~ one input processin~ element. The input ector is processed by the s~rsTem and ~ields an output ~ector (of onl,~ one element in this e~cample. as there is onl- one output processing element) on output response line 33 which is connected tO the data line of output processing element 20c. The error stimulus vector !of oniy one elementin this e.~;ample) is transferred to the system on error stimulus line 32 hhich is connect,ed t'o the net error line of output processing element 20c.
Figure 4 is a diagrammatic representation of an e~;pansion of the system represented in Figure 3. This s- stem can represent and learn a larger set of ma?-ping fimctions that the system of tigure 3. The processing elements. 20a. 20b and 20c correlate to the processing elements 20a. "Ob and 20c of Figure 3. The synaptic elements. 10a. 10b- and 10c correlate to the processing elements lOa.
10b and lOc of Figure 3. .~dditionall.~ there is another hidden processing element 20~'. another input synaptic element lOd. another output s~naptic element lOf and a hidden synaptic element 10e'.
lZ~717S
Figllre ;~ is a diagrammatic representation of an expansion of the s~stem represented in Figure 4, Figure ~ illustrates all possible connections bet~een t~o input processing elements~ t~o hidden processing elements and two output pro-cessing elements. The processing elements of similar numbers represent similar processing elements in Figure 3. ~dditionall~ there is another output processingelement 20~'. another input processing element '~Od''. and a plethora o~ additional synapt,ic elements. lOt" to 10z''.
The tuo elements of the input ector are transferred respectivel~ to the s!,stem via input stimulus 1 line ~1 and input stimulus line ~ hich are con-nected to the ~et Dala lines of input processing elements 20a'' and ~Od''. respec-tivel~. The two elements of the output ector are available on Output ~'alue 1 line ~ and Output ~ralue 2 line ~6, respectivelv. and are generated by the data lines of output processing elements 20c" and 20f'. respectively. The tv~o elements of the error vector are transferred respectively to the system via Error ~timulus 1 line 33 and Error ~timulus '' line ~4, which are connected to the ~'et Error lines of input processing elements 20a' and 20d-', respectivel,v, Figures 6. 7 aDd 8 illustrate a block diagram of a circuit implernentation of the neural syst.em of Figure ~, Figure & is a complete bloc}i diagram represen-tation of the s~stem depicted in Figure ~. Figures 6 and, are modular com-ponents of the sy,stem of Figure 8.
Figure 6 is a blocli diagram of a s-naptic circuit. generall~ denoted a~ 60, ~hich implernents the function of a s~naptic element 10 Svnaptic circuit 60. aspresented in Figure 6. is an adapti-e. continuous. bi-directional. m.atri:; multiplier element .
Synaptic C'ircuit 60 is a modular building blocli in a planar Ir~-o dimen-sionali matri:;. Data line 11a' and ~'et E;rror line 1''a' couple tO horizonta;l~ adja-cenr circuits. ~et Data line 13a' and Error line 1~a' similarl~; couple lo erticall-ad,jacent circuits.
Synaptic circuit 60 generates output signals ~'et Data, provided on line 13a'. and ~et Error. provided on line 1''a'. S~naptic circuit 60 receives as input signals E;rror on line 14a' and Data on line 11a'.
Synaptic circuit 60 includes an internal alue called "ueight", This alue is a- ailable on line 66. and is stored in inregrator 6~.
.
lZ~717S
--1, --The output of integrator 64. ~ia line 66. is coupled to the inputs of multi-plier 62. multiplier 63 and pouer function 6~. The Data signal provided to s~nap-tic circuit 60 on line 11a' is coupled to an input of multiplier 61 an'd an input of multiplier 62. .~n Error signal provided to s,vnaptic circuit 60 on line 14a' is cou-pled to an input of multiplier 61 and an input of multiplier 63. The output of the power function 6~ is coupled to an input of multiplier 61. Also coupled to an input of multiplier 61 is the excternally provided Learn Rate signal 6" defined in equation (28). The output of multiplier 61 is coupled to the input of integrator64.
dw j The output of multiplier 61. available on line ~9, is the equivalent of dt~
as deri~ed in equation (34). The product output of multiplier 61 is integrated over time by integrator 64. This integrated output signal from integrator 64 is a signal corresponding to the "weight" value.
The weight factor is coupled to the power function 6~ here it is raised to the po~er of a predetermined. fixed constant (this constant ma~ be different forever~ s,vnaptic element in a system: the optimal alue of this constant is empiri-call~ determilled). The output of the power function 6~ i5 coupled to the input of multiplier 61.
The eigllt value on line 66 is multiplied with the Data signal on line 11a`
in multiplier 6". The output of multiplier 62 is added to ~'et Data line 13a'. via summing element 69. and provides the ~'et Data output for synaptic element 60 on line 13a'.
The output of rnultiplier 63 is coupled to ~'et E;rror line 12a'. and provides the ~et Error output for s~naplic element 60.
The ~eighl ~ahle on line 66 is multiplied with the Error signal on Error line 14a' in multiplier 63. The output of multiplier 63 is added to ~\'et Error line 12a' v.a summing element 68. and provides the ~'et Error output for s~naptic ele-ment 60.
The signals in the s~naptic element ma~ be of an.v suitable character. as.
for example. voltages, currents. frequencies. Iight amplitudes. etc.
T~-picallv the output signals of this circuit. the signals on ~'et Error output line 12a' and ~'et Data line 13a'. if implemented electronicall~. are current signals.
~Z~7~75 These current signals are added to the total current flo~ from other circuits onthese respective lines. Hence. the total current on line 13a' is representative of the sum total output of all the s~naptic elements connected to line 13a': Summation units 68 and 69 can be implemented as wires in this t~pical case. Of course. other summing mechanisms can be used.
Figure 7 illustrates a processing element circuit generally denoted l 0.
Processing element circuit l0 recei-es a Net Data input signal on ~,'et Data line 13b' and a ~et Error input signal on ~iet ~rror line 12b'. Processing element cir-cuit l0 produces a Data signal on Data line 11b' and an Error signal on Error line 14b'.
The ~'et Data signal on line 13b~ is coupled to the input of Function Gen-erator l1 which provides for the processing element ,0 an output Data signal on Data line 11b'.
The ~i'et Input sigIlal from ~'et Data line 13b' is also coupled to the input of Function Generator 72. The output of Function Generator 1'~ and the ~\'et Error signal on ~'et Error line 1''b' are coupled to the two inputs of multiplier ,3.
~lultiplier ~3 outputs E;rror signal for the processing element l0 on Error line14b-.
For input and hidàen processing elements. the Data output on Data line 11b- is connected to the input Data line 11a of Synaptic Circuit ~0. For hidden and output processing elernents. the Error output on E;rror line 14b' is connected to the input ~rror lines 14a' of ..ynaptic Circuit 60. For outpui processing ele-ments. the Data output on Data line 11b' is provided as an output response. as part of an output vector. For inpul processing elements. the Error output on E;rror line 14b- is generall~ not used.
For hidden and OUIpUt processing elernents. the ~'et Data input signal on ~'ee Data line 13b' is connected to the ~'et Data output lines 13a' of S~naptic Cir-cuits 60. For input and hidden processing elements. the ?~et Error input signal on ~'et Error line 12b' is connected to the ~\'et Error output lines 12a' of S~naptic Circuits 60. For input processing elements. the ~'et Data input line 13b' receives an in.put stimulus t~hich is a single element of the input vector. For output pro-cessing elements. the ~et ~;rror input line 12b' receives an error stimulus ~-hich is a single element of the error v ector.
lZ~3~71~5 Figur~ 8 illustrates a schemati~:al ~locl; diagram illustral~ n of the la-out of the system diagrammaticall~ represenTed in Figure ~ us ng the modular circuitelements of Figures 6 and ,.
Processing elements 20a'' through 20f'' correlate with processing elements 20a' through 20f. respectivel~. of Figure ~. S~naptic elements 10a" through 10f''.
and 10t'' through 10z`' correlate with s~naptic elements 10a' through lOf and 10t' through 10z' respectivel~. of Figure ~. All signal lines, 11c through 1~c correlate to both the signal lines 11a' through 14a' of Figure 6 and signal lines 11b' through 14b' of Figure 7. The input. output and error lines 51' through ~6' correlate ~vith the input. output and error lines 51 though a~ of Figure 5. Synàptic elements 10a ' through 10f" and 10t'- through 10z'' mav be implemented as the circuit detailed in Figure 6. Processing elements 20a'' through 20f may be implemented as the circuit detailed in Figure ,.
An input stimulus vector, comprised of input stirnuli Input l on line ~1 and Input 2 on line ~2. are connected to processing elements 20a'' and ~Od''.
respectively. as is done in Figure ~. The output of processing element 20a'' is con-nected to synaptic elements 10a''. 10d", 10b''. and 10-v ' via Data line 11c'. Simi-larlv. the output of processing element 20d'' is connect to synaptic elements 10t''.
10u''. 10~' and 10x''. via Data line 11c''. S~naptic Elements 10a' and 10t'' sumtheir ~et Data outputs on ~'et Data line 13c. This summation on 13c' is pro-v ided as the ~'et Data inpuT to processing element 20b". Processing Element 20b'' pro- ides its Data OUtpUT signal on Data line 11c"'. to the Data input line of ~naptic l~lements 10e''. 10c'' and lOy''. S~naptic Elements 10d''. lOu'' and 10e'' sum their ~'et Data output signals on ~et Data line 13c''. ~ hich is provided asthe ~'et Dala input signal TO Processing Element 20e''. Processing Element 20e''provides its Data output signal on Data line 11c'"'. to the Data input line of .~naptic Elements lOf' and 10z''. Processing elements 10b". 10~'', lOc" and lOf'sum their :~'et Data output signals on ~et Data line 13c"'. u hich is provided as the ~\'et Data input signal to S-naptic E;lement ''Oc". Processing elements 10~
lOx''. 10~' and 10z sum their ~et Data output signals on ~'et Data line 13c'` '. hich is provided as the ~'et Data input jignal to ~naptic Element 20f'.
Processing elements ~Oc'' and 20f'' provide output signals Output 1 and Output 2. respectively. on lines 5~' and ~'. respectively. These outputs form the 128'7175 -2~-output ~ector.
An error stimulus vector. composed of error stimuli Error 1 on line ~3' and Error 2 on line ~4' are received b~ the ~et ~;rror lines of Processing Elements 20c and 20f, respectively, The ~rror output signal of Processing Elements ~Of'' is pro-~ided on E;rror line 14c'"' to ~ynaptic Elements 1O~A~' 10x'', 10y'' and 10z". The Error output signal of Processing Elements 20c'' is pro~ ided on Error line 1~c"' to Synaptic Elements 10f", 10c"~ 10~'' and 10b''. The ~\'et Error outputs of Synap-tic Elements lOf-' and 10z'' are summed on ~'et Error line 12c"'' and is pro~ided to the Net E;rror input line of Synaptic Element ''Oe''. The Error output signal of Processing Elements 20e'' is provided on Error line 1~c" to Synaptic Elements 10e''. 10u' and 10d'-. The ~-et Error outputs of ~ynaptic Elements lOe'', 10c".
and 10y" are summed on ~et Error line 12c'' and is provided to the ~'et Error input line of Synaptic Element 20b''. The Error output signal of Processing Ele-ments 20b'' is provided on Error line 1~c' to ~ynaptic Elements 10a' and 10t''.
The ~'et Error outputs of Synaptic Elements lOt'. 10u'. lOv and 10x'' are summed on ~'et Error line 12c" and is provided to the ~et Error input line of ~;~ naptic Element 20d". The ~et Error outputs of S- naptic Elements 10a''. 10d''.
I()b'' and 10~-'' are summed on ~'et Error line 1~c' and is pro~-ided tO the :~'et Error input line of .~! naptic Element 20a''.
In the example the Error output signals of Processing Elements . Oa'' and 20d'- are not used: often this will be the case. and as such a minimal system does not include the functional parts necessar- to pro~ide the Error output signal ofinput processing elernents. such as 20a' and "Od-'. nor the functional parts to pro-ide a .~'et Error output for the synaptic element.s connected to the input process-ing elenlents. The example is provided ~ith ~he Error output signals of Process-ing Elements 20a'' and "Od'' and the ~\'el Error outpur signals for ~naptic Ele-ments 10a". 10d". 10b'-. 10- ''. lOt''. 10u' . 10--'' and 10x for clarity and unifor-mit~ . A system Cell be built in this manner ~ ith no loss of generalit~ .
The input and output signals. along ith an- other signals of the circuitr .
may be either a voltage. current. frequenc~:. Iight amplitude. or other mechanism ~hich is capable of representing a ~ alue.
Figure 9a and 9b illustrate typical functions of the function generators 160 and 168 of figure 8. Function generator 16~ -hich provides the function F is 12871~S
t~picall~ characterized bv the equation:
F ( ~
1-e A t~p]cal function pro~ided b~- function generator lG8 is illustrated in figure 9b here~n F (~ e ~ 2~
The functions F and F are merel e~;emplar~ functions that can be approximatel~ implemented as a ladder net~A~ork of diodes and resistors. Howe~er.
other functions F and F- may be readil~ substituted. The function F is usuall~
the derivati~e of the function F.
It can be seen that by e~cpansion of the circ.uit of Figure 8 that much larger c ontinuous time neural s~stems can be constructed.
T~picall- the circuit eiements are formed in an ~ high (vertical) b~ ~I
wide (horizontal) matrix. .~s such~ there are ~ net error and ~ input lines. uith each being paired and running horizontall~ through ~;1 matrix elements. ;:imilarl~.
there are ~I delTa alld ~1 ner inpuf lines which are paired and each p~ir runninO
~erticall~ through ~ matrix elen-ienrs. The ~\~1 pairs of ~ertical delta and ner input lines are brought pair-- ise to the inpllt signal net input and the o uti-ut si mal delta of ~I processillg eiement circllit . The ~i pairs output and neT error iines from the processing elemenr circuirs can then be run through another ~I higsl b~L Y.-ide matrix b- connecting the siOnals respective!~ into t}le input signal input and output signal net error of the ~\I b~ L matrix.
The s~naptic circuiT and proessing element circuits are inserted in-o rr.atri:;-like con~gurations. The s~ naptic circuits are placed illtO a rectangular matri~; e~er~where a ~eight connection is desire(l het~ een tuo neurons. The pro-ccssing element circuits are placed e~er~here the rerminus of a neuron a:~on areplaed. Implemented in this fashion the s~stem ~ould then comprise a net~ or~;
hich i.c capable of performing the learning functior..
Howe~er. man~ net~orks ~ill not nee~l the le.lrning capabilit~. Once a s~c-tem has learned a particular function. the eighr ~alues ould be etched in~o a 128~7175 ., permanent memor~ for a lo~er cosr ~ersion than a ~er~ion capable of learning.
~uch a s!-stem ~ould need onl, the for~ ar~ flo~- of information; the back~ard Ho-~- and the ~veight update functions could be eliminated.
.~lthough in smaller s~stem the integrator ma- be implemented with capa-citors onlv~ this presents a dra~back in larger s.vstems due to the deca~ rate of the capacitors in the integrators. Howe-er. it is feasible that the weights rna~ be par-tiall~ stored digitallv and capacitors used onl~ for the low order bits of the data.
Learning in the Back Propagation s~stem is most often implemented b,v presenting an input and output pattern to the network and allouing it to adapt partially to the pattern before presenting the ne:;t pattern. It is noted that during the transition period between changing patterns the learnin~ rate tO should be set to zero to inhibit learning of improper information. An input pattern is presented to inputs of a total network ~hich then propaga~es through the nen ork to pro--ide an output pattern. .~n error is generated based from target or e~pected pat-tern. This error is then propagated back~- ard Ihrough the net~ orl; u ith a corresponding adjustment of the ~eights durinz the backvlvard propagations. Dur-ing this backward propagalion of error the weights are being updated bv e~;amin-inz rhe input and the error lines. The svnaptic and processing element circuits as described herein are quite adaptable to any learning system ~ hich implements the Back Propagation theory. There are probablv other learning s~stems to hich this svstem is also well fitted, In creating a pattern presentation svstem a ~imple e:;ternal circuit can be readily constructed. The svstem consists of rnemorv ~ ide enou"h to accommodate all the bil necessar.v for both the input and OUtpUT patterns. If an input or an OUT-put is tO be continuous analog (rather than having t~'O discrete analog values) then there must be a digital to analog converler for each input and output ~ hich is continuous. The memory for that channel ~ill need to be appropriatel~ - ide. In order to present the data. a simple sequencer and counter is all that is required.
The sequencer turns off the learning rate. steps the counter and turns the learning rate back on. The sequencer uould then ~ait for an appropriate time. and then repeat the steps. The counter ~ ould step through The memor~. c~cling one at ~
time for all of the patterns. .~lthough this s~stem could be stand-alone, it is more likel~ to be a co-processor or peripheral of a srandard computer. The standard ~L%~'7175 computer ~ould ~irst doun-load the information to the memory. The netttork would then undergo the learning process. .~t the end of the learning process thehost computer v.ould read bac~i the learned t~eights.
ln another e~cample. the circuit ~ ould also be able to perform in a continu-ous s,tstem. In specific exar.-.i,le. the netuork might be attached to the sensors and controllers of a robotic arm. The sensors ould pro--ide a continuous time representation of the external world. If the sensors are connected so as to provide both input and error signals the arm would learn to minimize the error signals.
In the feed forward function there is a requirement that there is a location to store the weight. This requirement is solved b~ the integrator which is assumed to hold a valid weight. The input stimulus to the neuron' comes in on the input line. This input line is shared b~ all the synaptic circuits which read this line.
I,'nless this input is a direct input from the external world this line would be con-nected to the output of a prior neuron. .~s such, this input line rnav sometimes be referred to as an output line. but rather than being the output of the current neu-ron. it is the output line of the prior neuron.
The s,vnaptic and processing element circuits as block diagramed in the figures ma,v be readilv implemented as electronic components. The multipliers are typically four quadrant multipliers. The integrator is simpl~ an operational amplifier with a feedback capacitor. The F and F' functions are readil~ made using simple ladder networks of resistors and diodes. The precision of the F and F' functions are not critical. however it is necessary that the F' function is the derivative of the function F. There ma!~ be other~ more readil! implemenTed function generators available.
The po--er function for the feedbacli raised weight is of the form ~I~R. ~here R is a positive constant. From the mathematics it is clear that R can be an- non-negative constant. Experimental results indicate 0.~ wor};s well (square root).
This is a readilv implemented function in a circuit element. .9.lso, a t alue of 1.0 for R has been used: this function is easil~ implemented as a wire. Finall,v. the standard bac}c propagation uses a talue of 0. uherein the uhole circuit can be remot-ed (as w = 1, assuminV 0 = 1).
Implementation of the s!~naptic and processing element circuits is readil~
adaptable to fabrication in inte~rated circuits. Based upon the horizontal and ~2~717S
~-ertical matri~; lavout the topovraph~v of the circuit ~ould be rather fundamental in design in building a net- or~i based on these builcling blocks. The synaptic and processing e]ement circuits are analog building blocXs of entire neural ne~worl;s useful in .~rtificial Intelligence systems.
The input processing elements. rather than being implemented as detailed.
may be implemented as feed-through units. not performing an~ calculations on the input stimulus. This would provide. as the input to the s~,naptic elements adirect fanout of the input stimulus to the s~naptic elements and the input pro-cessing elements could be implemented as a wire in an electronic implementation In manv applications. this is the method of choice. rather than as detailed l)v the drawings. utilizing a complete processing element.
.4.lthough particular embodiments of the invention ha~ e been illustrated and described. modifications and changes will become apparent to those skilled in the art. and it is intended to cover in the appended claims such rnodifications and changes as corne ~-ithin the true spirit and scope of the invelltion.
~hat I claim is:
, . . . ., .. ~. .
INVENTION: Back Prop~gation System INVENTOR: Max Stanford Tomlinson Jr.
BACKGROl[JND OF TH:13 IN~ NTION
I. Field of Invention The present invention is a continuous-time adaptive neural system which learns to translate a set of input signals (input stimulus vectors) to a set of output signals (output vectors) based on a set of expected output signals (target vectors) through the back propagation of error signals. The error signals (error vector) are ~2~ 7S
a function of the actual and the e~cpected OlltpUt ~ignals.
II. Background.~rt .~s used herein. a neural s~stem produces an output v ector uhich is a func-tion of an input vector. The mapping function bet~een the input and output vectors is learned. Ideally. the output vector should match the target vector: the difference between the two vectors can be used in an adjustment mechanism.
Original theoretical approaches tot~ards neural networks are based upon the idea that when t~o neurons in the brain are active there is a correlation between them. One earl-; rule developed bv D. O. Hebb is described in his book "The Organization of Behaviour", ~ iley, 1949. The Hebbian rule states that hen two neurons are firing simultaneousl~ an association link between them is strengthened. Accordin~l-. the ne.Yt time either of the two neurons fires. the other one is more probable to fire also. Houever. the Hebbian rule is not a sufflcientmodel to explain the learning process. I:~nder the Hebbian rule. the connection strengths between neurons grow ~ithout bound. If maximums are placed on the connection strengths. these maxirIlums are always reached.
~ ubsequently. the Perceptron ~lodel was developed b~ Frank Rosenblatt.
and is discussed in his book "Principles of ~eurod~namics". SparTan. 196~. The Perceptron Model was originally believed po-~rerful enough to enable a machine to learn in a human-like manner.
The Perceptron ~odel includes input. hidden and output layers: each compriscd of one or more processing elements. In response to input stimuli. the input layer provides information to the hidden la~er. ~imilarl~. the hidden la~er pro- ides information to the output la.~er. Corlnections betv- een the input andhidden processing elements are fi~sed: connections be~-een the hidden and outputprocessing elements are adjustable.
In the Perceptron ~Iodel. if the inp-lts are boolean (i.e. either zero or one).
then the intended purpose of the hidden la~:er is to extract some kind of features from the input data. Ho~ever. if the inputs to the ~Iodel are continuous numbers(i.e. ha~ing more than t~o distinct ~alues. rather than just two boolean values).
then the hid(len layer is not used. Instead. Ihe outputs of the input layer are con-71~5 nected directl~ to the inputs of the output la~er.
In the Perceptron ~Iodel. all le~rning takes place in the output la~ er.I, nder the Perceptron ~Iodel many problems have been experimentaliy and mathematically shown to be representable bv connection strengths between la~ers.Rosenblatt s Perceptron Learning .~lgorithm enables a neural network to find a solution if there exists a representation for that problem by some set of connection strengths. Rosenblatt's Perceptron Convergence Proof is a well l;nown mathemat-ical proof that a Perceptron System will find a solution if it exists, In operation, the Perceptron Model modifies the strengths of the weighted connections between the processing elements, to learn an appropriate output response corresponding to a particular input stimulus ~ector. The n1odification of the connection weights occurs when an incorrect output response is given. This modification of the weights changes the transfer of information from the input to output processing elements so that e-entuall~ the appropria~e output response ill be pro-~ided. However. through experimentation, it was discovered that thePerceptron ~Iodel was unable to learn all possible functions. It uas hoped thal these unlearnable functions ere onl,v pathological cases. analogous to certainproblems that humans cannot solve. This is not the case. Perceptron ~ tems cannot represent and learn some very simple problems that humans are able to learn and represent An example of a problem that the Perceptron ~lodel is unable to represen~
(without 2~'~ hidden processing elements. uhere .~- is the number of input nodes), and therefore cannot learn. is the parit~ or "exclusive-or" boolean function. Toperform such a problem (u ith feuer than ~ ~ hidden processing elements) a system would require two layers of modifiable ueights. The Perceptron System cannot properly adjust more than one layer of modifiable weights. It - as speculated that no learning mechanism for a s-stem uith multiple layers of modifiable weights would ever be discovered because none existed !2~1insky & Papert. 1969. in "Per-ceptrons" ) .
(The problem with using ~ ~ hidden units is three-fold. First. since the hid-den units. in the Perceptron ~Iodel. do not adapt. ali the units must be present.
regardless of the function which needs to be learned. so that all functions can be learned. Second, the number of units required gro- s phenomenall.~: for e:;ample, ~Z~7175 2V4 is approximately 17 bi]lion, more neurons than in a human brain: this means that the largest parity problem the hurnan brain could solve, if wired in this manner, would have at most 32 inputs. Third. the s~-stem would not generalize:
given two input/output vector pairs near one another, one trained and the other not, the system should be able to interpolate the answer from the first: w,ith alarge number of hidden units. it has been e~;perimentally shown that this is notthe case.) Almost ~ll adaptive neural systems share several features in common. Typi-cally the processing elements of all systems have an output which is a function of the sum of the weighted inputs of the processing element. Almost all systems have a single layer of modifiable weights which affect the data transferred from the in-put to the output of the system.
The etolution of adapti~e neural systems took a dramatic step forward with the det elopment of an algorithm called "Back Propagation". This algorithm is fully described in the reference text "Parallel Distributed Processing. the ~Ii-crostructure of Cognition". Rumelhart. Hinton.& U illiams. ~lIT Press, 1986.
A back propagation system typically consists of three or more la,vers. each layer consisting of one or more processing elements. In one basic example, the system is comprised of an input layer, at least one hidden layer and an output layer. Each layer contains arbitrary. directed connections from the processing ele-ments in the inpùt layer to the hidden layer, and from the hidden layer to the output layer. There are no connections from processing elements to processing ele-ments in the same layer nor connections from the output to the hidden layer nor from the hidden to the input layer: i.e. there are no cycles (loops) in the connec-tion graph. (There are hypothesized mechanisms for networks t ith cycles in them~ but they are not being scrutinized herein.) In the Perceptron ~Iodel the idea of error ~as introduced. In a back propa-gation system, at each output processing element of the netuork. the error is quite easily realized. The error is typically the difference between an e~pected ~alue and the output ~alue. This error is used to modify the strength of the connection between a processing element and the output processing element. Ideally. this reduces the error between the e~cpected outpul and the ~-alue output by the pro-cessing element in response to the input. The Perceptron ~lodel lacks the ability 128~175 to allocate an error value to the hidden processing elements and therefore cannot adjust the weights of any connections not coupled to an output processing ele-ment. In a s~stem utilizing the Back Propagation algorithm. an error is assignedto the processing elements in hidden layers and th~ weights of the connections coupled to these hidden processing elements can be adjusted.
The derivation of the discrete time Back Propagation .~lgorithm is described in Chapter 8 of Parallel Distributed Processing, by Rumelhart et al, and is recounted here.
.~ weight associated with a connection is referred to as ~ji. The subscripts are used in the form ~to,from Hence, in the variable wji, i refers to the processing element from which data information is being received, and j refers to the process-ing element to which data information is sent. In the back propagation algo-rithm. a particular input stimulus vector is referred to collectivel~ b~ the v ariable p (for pattern). The elements of a particular output vector and particular target vector are referred to respectively as op; and Tpj, where j varies over the output processing elements. The Total Error of a s~stem is represented b~ the variable E The portion of the Error contributed bv a single input vector (one input pat-tern) is represented b~ the v ariable Fp The output of a processing element o;". in response to an inpur pattern p.
is is calculated by the following equation (uhich also defines the value netpj):
netpj = ~ u~j; op~ (1) (2) op~ = f(netpj) The techniques used b~ Back Propagation to minimize the Total Error is a varj-ant of Least ~Iean ~quared. The technique states that the total error is the square of the difference between the target vector and the output vector. Further-more, it is assumed that the total error for the s,vstem is the linear summation of the error for any individual pattern.
p (3) lZ~7175 Ep = 2 ~ (Tp,-Op,) ( ) In a Back Propagation network, error is minimized b~ adjusting the weights within the netuork. ~rhat is desired is to determine uhat amount to adjust a weight so that the error will be reduced. The following equation expresses that deslre:
a~ ~ aEp awji p awji The above expression can be expanded by the chain rule to get:
aE ~ aEp anetpl. (6) a~vji p k anetpk aWji anet k ~'e can expand the second component. a p bv noting Ihat netpk = ~ ulmom.
to get the following a(~ Wkmm) anetpk m ( ) a U~
It is eas~ to see that e~cept uhen m=i and ~=j, the abo-e is zero. Putting this back into the abo~e equation we get:
aE ~ aEp O (8) ~)'U ~i p anetpj ' a~p The first portion of the equation. a t- b~ e~cpansion using the chain rule.
gi~ es:
-~EC aEp ~pl (91 ~netpj j ~opl ~netp~
and a p can be simplified b~ recognizing Op/ = I(~-~etp~). B~ substiluting this in, the expression becomes:
' "~ " . .
~2~7175 I
aop, aJ(rle~pl) (10) anetp~ anetp~
t can nou be noted that af(netpl) is zero except when l=j; this gi--es us finallv ~netp j ~Opj af(netpj) = J'(netp;) (11) 3netpj anetpj and this can be substituted back in to get:
awj~ aoP J'(net aEp If we are examining an output node, the value of ao is readily apparent from the definition of Ep. as in:
Ep = 1 ~ (Tpj-Opj) (13) Partial differentiation of this expression with respect to Opj gives the following expression for output processing elements:
--= Tpj- Opj (l4!
Thus the error equation for an output value is:
a ~ = ( Tp j- Op~ ) J ' ( net p j) o~
The problem remains ~-~hal the error ~alue is for the hidden processing elements.
To determine this. Iet the definition of ~p, be:
aEp (16) Pj ~netpj From the expansion from above. ~-e see:
aEp = aEp J'(net ) anetp; aOpj ~xpanding by the chain rule on opj, ~, e get:
12~ f i~S
a~p aEp anetpk (18) aOp,, ~ an'etpk aOp, Expanding a ~ ~ b~ the definition netp~ u!k~ol. ue get:
a(~ U'klOpl) anetpk I (19) aOp,' aOp,, It is easv to see that the above is zero. e~cept u-hen l= j, so that we can state:
anetpk 3U~kjOp. (20) U!k ~
aOp~ aOpj ~ubs~ituting this back into the above equation. ue get:
U!k ~
By the definition of ~pJ~ we can then state:
a O ~ ( ~pk U'k j ) ( '' '~ ) Therefore. op~ for a hidden node can be expressed as:
~pj = I (netp.~)~ (pkU'kJ) ~'~3) k C o;nbining all the above elements together. we get:
( '~ ) , o p, and from this. the Total Error equation can 't>e forl1lulated:
~aE = ~ ~;" op, (-~
For an output processing element. ~P! i}: ('~G) ~p, = J'(netpJ) (Tpj-Op,) For a hidden processing element. ~p~ is:
~21~7175 = J (netpj) ~ (~SpkWkj) (21) k ~'o~ the change of the ~eight is set proportional to the above partial differentiation This is given by the follo- ing equation (28) ~,w~ opi The constant of proportionality (~) is the Learn Rate Experimentally, this constant has been found to be in the range of a to ver~ small, depending onthe number of u eights processing elements and patterns which are to be presented ~ ote that there is no guarantee that anS one particular veight change for a particular pattern uill decrease the total error it is actually quite probable that during one of the patterns the total error will increase, just that over all the pat-terns the total error should decrease In summary back propagation may be described as follo~-s On a forward pass of information through the netuor}; all of the processing element outputs are calculated b~ propagating the information input foruar(l through the net- ork i e from the input la, er to each hidden layer in turn and finall- to the output !a-er On a backuard pass i e frorn the output la~ er to the hidden la-ers each in reverse order from before and finally to the input layer all the errors are calcu-lated by propagating the associated errors back-vards through the net- ork FinallS~ all the ueights are changed according to the errors in the processing ele-ments above and the outputs of the processing elements belo-~
Prior art neural systems using the Back Propagation ~lgorithn-l ha-e been f'requently implemented on computer sS~stems These systems suffer se~ere limita-tions as the anlount of time required to perform the calculations is exorhitant ~s the size of the netuork (counting the processing elements) gro- s the com?uta-tional time gro-~ j exponentially (assurning a full- interconnected graph) It is therefore an object of the present invention to pro~ ide a ne- and improved adapti-e neural net- ork utilizing a modification of the Back Propaga-tion .~lgorithm to enable the system to adapt to a specified output response in response to a certain input stimUluc~
7:175 It is a further object of the in~ention to provide a continuous-time system, ie. the three phases described abo~ e are done simultaneousl~: all for~ ard propaga-tion. backward propagation and u eight updates proceed continuously in time.
It is a further object to pro~ide a neural system uherein the calculations are performed. not bv digital methods. but rather by analog methods.
It is a further object to provide a neural system for which the computa-tional time remains relati~elv constant as the network size increases.
III. Sumrnary of Invention An adapti-e. continuous'time information processing s-stem including an input laver having at least one input processing element. a hidden layer ha- ing at least one hidden processing element and an output la- r having at least one out-put processing element. Input synaptic elements are included ~vith each element coupling an input processing element tO a hidden node. The input synaptic ele-ments may couple an input processing element to more than one hidden process-ing elements. Output synaptic elements are also included with each element cou-pling a hidden processing element to an output node. The output synaptic ele-ments may couple a hidden processing element to more than one output process-ing element. The function of each input processing element is to receive an input stimulus. The input stimulus is transferred as information by the input synapticelement,s to each coupled hidden processing element. The information transferredby a coupling input synaptic element is transferred in accordance ~-ith a ueight-ing factor associated ~ith the coupling input synaptic element. Informatioll is also transferred from each hidden processing element to a coupled output processing element by a coupling output synaptic element. The information transferred b~ a coupling output synaptic element is transferred in accordance u ith another weighting factor associated with the coupling output synaptic element. The infor-mation output from the output processing element} is the output response of the system to the input stimulus. The output processing eiement also receives an error stimulus u hich is dependent upon the output response of the output processing element and an expected response. This error is propagated backwards through the output synaptic elements. hidden processing elements and input synaptic 121~7175 elements for modifying the respective weighting factors of the input and output synaptic elements. Back- ard propagation of the error through the system permitsthe system to adaptivel~ provide an e~;pected output response corresponding to an input stimulus.
IV. BRIE~ DE~CRIPTION OF DRAWINGS
The features. objects. and advantages of the present invention ~ ill become apparent from the following detailed description, taken in conjunction with the accompanying dra~ings, in which like reference numerals correspond throughout and wherein:
Figure 1 is a diagrammatic representation of a synaptic element of the present in~ention:
Figure 2 is a diagrammatic representation of a processing element of the present invention:
Figure 3 is a diagrammatic representation of a small. representative systeni of the present int ention having a single hidden processing element:
Figure 4 is a diagrammatic representation of a small. representative system of the present invention having a hidden s,vnaptic element:
Figure ~ is a diagrammatic representation of yet another embodiment of a s- stem of the present invention.
Figure 6 is a circuit block diagram of an implementation of the synaptic element of the diagrammatic representation of Figure 1.
Figure ~ is a circuit block diagram of an impiementation of the processing element of the diagrammatic representa~ion of Figure ''.
Figure ~ is a biock diagram of a circuit implementation of the syslem of Figure ~ using the circuits of Figures 6 and /:
Figures 9a and 9b are graphical representations of typical functions pro-vided by the function generators of the circuit of Figure 8:
12~7175 V DETAILED DESCRIPTION OF THE PREFERRED EMBODIME~TS
The Back Propagation .~lgorithm as originally developed and as described in the previous section is a discrete time algorithm. in that there is a forward pass.
a backwards pass and modification to the weights, and then a recycling. However.this is not an optimal implementation of the s~stem. There is an implicit assump-tion of linearit~ during these discrete time intervals. This is generally not a good assumption.
A better implementation of the system is for the network to run continu-ousl~, performing each of the operations simultaneously: this is the basis of what is called herein a continuous time s~stern. The follouing is a derivation of the con-tinuous time model of the Back Propagation .~lgorithm, as developed by this inventor One primary innovation is that time should be used to differentiate theError, to ascertain that the Error is monotonically decreasing. Hence, instead of taking the derivative of the Error with respect to any particular weight. the derivative is taken uith respect to time. The chain rule can then be applied tak-ing the partial differentiation with respect to 1Lji.
dE ~ aED du~j; (29) dt ~ dt Repeating equation (2~). as derived in the discrete time algorithm:
~E = ~;, aEp = ~, 6pjop, his can then be replaced into equation (~9) to give:
dE = ~ p jp,) dt To ensure that the derivative of the Error is nlonotonicall~ decreasing. the sign of ddt must be negati- e. The onl.v wa. to do this is to ensure that the sign of dti is the opposite sign of ~ ~pjop,.
By arbitrarily setting lZ8717S
p~pi (~2) this constraint is satisfied. b~ giving us:
dE = _ ~ pj pt ) Since the derivati-e of the Error is monotonicall,v decreasing, the s,vstem will con-verge at some final error alue. As derived. a s~stem is not guaranteed to converge to a zero Error. Experimental results show that a s,vstem generall,v will converge to zero error if the problem to be solved is representable b,v the net~ ork. If the s~rstem does not converge to a small error. or does not reliably converge, adding a small nurnber of additional processing elements and connections will lead to con-~ergence.
One further object of interest is that there exists a mechanism to maintain onl~ positive weight values. The Error remains monotonic decreasing u-hen an e xponential term, W~ is multiplied into the dtj function of equation 13" ) .
~:pecificall,~:
~ WR~ ~ ~p~op !3~!
and dt = ~ W~ p;P )' 13~) It uill become clear from the following detailed descriptions of the drau-ings how thesè eqllationC have been reduced to practice.
.~ neural s!stem utilizing backwards error propagation can be represented bv two kinds of elements: processing elements and s,vnaptic elements.
.~ s~naptic element connects two processing elements and ils primar~ func-tion is to store the connection strength.
A processing element receives a net data and a net error signal. and pro-duces a data and an error signal~ uhich are functions of the tuo received signals.
The functions can be mathematicall~ expressed as:
12~ S
(36) Output = J( .~etInput ) (3l ) E;rror~ = J ( .~etInp~ ) x .~\~et~rror~
.~ s~naptic element recei-es a data and an error sic~nal and produces a net dataand a net error signal ~hich are a function of the t~o recei~ed signals. The func-tions can be mathematicall~- expressed as:
.~etInputi = ~ U eightjj ~ Output~ (38) .~ietError~ = ~ Il eightji~ O~tput (39) dti = Learn_Rateji~ rror~ Outputi (~) The exact implementational details ~ ill become clear when the full s~-stem circui-tr~ is described.
The layout of the continuous time back propagation algorithrn is essen-tiall~, topologicall~ identical to that of the discrete time algorithm. Ho~-e~er. for bre-it~ of uords, there is a more elegant mechanism utilized to describe the inter-connectiolls and layout of the processing an~:l synaptic elements.
There are three layers of processing elements. onl~: the input. the hidden and the ourput layers. Each layer. as before. consists of one or more processingelements. There ma~ be connections from the input to the hidden larer (inpu~
synaptic elemeIIts). from the hidden to the output la~ er (output sy naptic ele-ments). from the input to the output la!er (direct s-r.aptic elements). and fromhid(1en processing elements to other hidden processing elements (hidden synapticelerrlents). There is a large constraint placed on hidden s~-naptic elements: if the hidden processing elements are numbered 1 to ~\. a s~-naptic element ma! onl~
connect from a io~er numbered hidden processing element to a higher numbered processing element: remernber that Ihe directionality of a synaptic elernent is important .
This three-la~er descriplion actuall- produces ail possible layered en~iron-ments: it describec an acyclic graph.
Figure l is a s~mbolic diagrammatic representation of a ~ynaptic Elemen~
denoted generall~ as 10. ~;~ naptic Element 10 has four !ines: ~ o input lines.
Dat.a 11a and Error 1~a: and t--o output lines. .~et Dat<I 13a and ~et E;rror l"a.
~Z87~75 Transfer function 1;~ is s~ mbolicl~ shoun to operate on and betv-een the data and error lines. The arrov- 16 signifies that the transfer function 1~ is ~ariable. Indi~
cator 1/ designates the ~\'et Data line.
Figure 2 is a s,tmbolic diagrammatic representation of a Processing Ele-ment denoted generall,~ as ~0. Processin(r Element 20 has four lines: two outputlines. Data 11b and Error 14b and tuo input lines, ~'et Data 13b and ~iet Error 12b. Internal drawings 28 are symbolic of the information flo~ within a Process-ing E;lement and will be explained in detail further in the description.
Figure 3 is a diagrammatic representation of a near minimal system of the present in~ention. Three processing elements. input processing element 20a, hid-den processing element 20b, and output processing element 20c are connected by three synaptic elements, input synaptic element 10a, direct synaptic element 10b~
and output s~ naptic element 10c. . The data. error. net data and net error lines of the synaptic elements are connected to the similarly named lines of the collnected processing elemenls. as diazrammed.
The input vector is transferred to the system via Input Stimulus line 31 v~hich is connect,ed tO the net data line of input processing element 20a. The input vector is comprised of onl- one element in this e:;ample. as there is onl~ one input processin~ element. The input ector is processed by the s~rsTem and ~ields an output ~ector (of onl,~ one element in this e~cample. as there is onl- one output processing element) on output response line 33 which is connected tO the data line of output processing element 20c. The error stimulus vector !of oniy one elementin this e.~;ample) is transferred to the system on error stimulus line 32 hhich is connect,ed t'o the net error line of output processing element 20c.
Figure 4 is a diagrammatic representation of an e~;pansion of the system represented in Figure 3. This s- stem can represent and learn a larger set of ma?-ping fimctions that the system of tigure 3. The processing elements. 20a. 20b and 20c correlate to the processing elements 20a. "Ob and 20c of Figure 3. The synaptic elements. 10a. 10b- and 10c correlate to the processing elements lOa.
10b and lOc of Figure 3. .~dditionall.~ there is another hidden processing element 20~'. another input synaptic element lOd. another output s~naptic element lOf and a hidden synaptic element 10e'.
lZ~717S
Figllre ;~ is a diagrammatic representation of an expansion of the s~stem represented in Figure 4, Figure ~ illustrates all possible connections bet~een t~o input processing elements~ t~o hidden processing elements and two output pro-cessing elements. The processing elements of similar numbers represent similar processing elements in Figure 3. ~dditionall~ there is another output processingelement 20~'. another input processing element '~Od''. and a plethora o~ additional synapt,ic elements. lOt" to 10z''.
The tuo elements of the input ector are transferred respectivel~ to the s!,stem via input stimulus 1 line ~1 and input stimulus line ~ hich are con-nected to the ~et Dala lines of input processing elements 20a'' and ~Od''. respec-tivel~. The two elements of the output ector are available on Output ~'alue 1 line ~ and Output ~ralue 2 line ~6, respectivelv. and are generated by the data lines of output processing elements 20c" and 20f'. respectively. The tv~o elements of the error vector are transferred respectively to the system via Error ~timulus 1 line 33 and Error ~timulus '' line ~4, which are connected to the ~'et Error lines of input processing elements 20a' and 20d-', respectivel,v, Figures 6. 7 aDd 8 illustrate a block diagram of a circuit implernentation of the neural syst.em of Figure ~, Figure & is a complete bloc}i diagram represen-tation of the s~stem depicted in Figure ~. Figures 6 and, are modular com-ponents of the sy,stem of Figure 8.
Figure 6 is a blocli diagram of a s-naptic circuit. generall~ denoted a~ 60, ~hich implernents the function of a s~naptic element 10 Svnaptic circuit 60. aspresented in Figure 6. is an adapti-e. continuous. bi-directional. m.atri:; multiplier element .
Synaptic C'ircuit 60 is a modular building blocli in a planar Ir~-o dimen-sionali matri:;. Data line 11a' and ~'et E;rror line 1''a' couple tO horizonta;l~ adja-cenr circuits. ~et Data line 13a' and Error line 1~a' similarl~; couple lo erticall-ad,jacent circuits.
Synaptic circuit 60 generates output signals ~'et Data, provided on line 13a'. and ~et Error. provided on line 1''a'. S~naptic circuit 60 receives as input signals E;rror on line 14a' and Data on line 11a'.
Synaptic circuit 60 includes an internal alue called "ueight", This alue is a- ailable on line 66. and is stored in inregrator 6~.
.
lZ~717S
--1, --The output of integrator 64. ~ia line 66. is coupled to the inputs of multi-plier 62. multiplier 63 and pouer function 6~. The Data signal provided to s~nap-tic circuit 60 on line 11a' is coupled to an input of multiplier 61 an'd an input of multiplier 62. .~n Error signal provided to s,vnaptic circuit 60 on line 14a' is cou-pled to an input of multiplier 61 and an input of multiplier 63. The output of the power function 6~ is coupled to an input of multiplier 61. Also coupled to an input of multiplier 61 is the excternally provided Learn Rate signal 6" defined in equation (28). The output of multiplier 61 is coupled to the input of integrator64.
dw j The output of multiplier 61. available on line ~9, is the equivalent of dt~
as deri~ed in equation (34). The product output of multiplier 61 is integrated over time by integrator 64. This integrated output signal from integrator 64 is a signal corresponding to the "weight" value.
The weight factor is coupled to the power function 6~ here it is raised to the po~er of a predetermined. fixed constant (this constant ma~ be different forever~ s,vnaptic element in a system: the optimal alue of this constant is empiri-call~ determilled). The output of the power function 6~ i5 coupled to the input of multiplier 61.
The eigllt value on line 66 is multiplied with the Data signal on line 11a`
in multiplier 6". The output of multiplier 62 is added to ~'et Data line 13a'. via summing element 69. and provides the ~'et Data output for synaptic element 60 on line 13a'.
The output of rnultiplier 63 is coupled to ~'et E;rror line 12a'. and provides the ~et Error output for s~naplic element 60.
The ~eighl ~ahle on line 66 is multiplied with the Error signal on Error line 14a' in multiplier 63. The output of multiplier 63 is added to ~\'et Error line 12a' v.a summing element 68. and provides the ~'et Error output for s~naptic ele-ment 60.
The signals in the s~naptic element ma~ be of an.v suitable character. as.
for example. voltages, currents. frequencies. Iight amplitudes. etc.
T~-picallv the output signals of this circuit. the signals on ~'et Error output line 12a' and ~'et Data line 13a'. if implemented electronicall~. are current signals.
~Z~7~75 These current signals are added to the total current flo~ from other circuits onthese respective lines. Hence. the total current on line 13a' is representative of the sum total output of all the s~naptic elements connected to line 13a': Summation units 68 and 69 can be implemented as wires in this t~pical case. Of course. other summing mechanisms can be used.
Figure 7 illustrates a processing element circuit generally denoted l 0.
Processing element circuit l0 recei-es a Net Data input signal on ~,'et Data line 13b' and a ~et Error input signal on ~iet ~rror line 12b'. Processing element cir-cuit l0 produces a Data signal on Data line 11b' and an Error signal on Error line 14b'.
The ~'et Data signal on line 13b~ is coupled to the input of Function Gen-erator l1 which provides for the processing element ,0 an output Data signal on Data line 11b'.
The ~i'et Input sigIlal from ~'et Data line 13b' is also coupled to the input of Function Generator 72. The output of Function Generator 1'~ and the ~\'et Error signal on ~'et Error line 1''b' are coupled to the two inputs of multiplier ,3.
~lultiplier ~3 outputs E;rror signal for the processing element l0 on Error line14b-.
For input and hidàen processing elements. the Data output on Data line 11b- is connected to the input Data line 11a of Synaptic Circuit ~0. For hidden and output processing elernents. the Error output on E;rror line 14b' is connected to the input ~rror lines 14a' of ..ynaptic Circuit 60. For outpui processing ele-ments. the Data output on Data line 11b' is provided as an output response. as part of an output vector. For inpul processing elements. the Error output on E;rror line 14b- is generall~ not used.
For hidden and OUIpUt processing elernents. the ~'et Data input signal on ~'ee Data line 13b' is connected to the ~'et Data output lines 13a' of S~naptic Cir-cuits 60. For input and hidden processing elements. the ?~et Error input signal on ~'et Error line 12b' is connected to the ~\'et Error output lines 12a' of S~naptic Circuits 60. For input processing elements. the ~'et Data input line 13b' receives an in.put stimulus t~hich is a single element of the input vector. For output pro-cessing elements. the ~et ~;rror input line 12b' receives an error stimulus ~-hich is a single element of the error v ector.
lZ~3~71~5 Figur~ 8 illustrates a schemati~:al ~locl; diagram illustral~ n of the la-out of the system diagrammaticall~ represenTed in Figure ~ us ng the modular circuitelements of Figures 6 and ,.
Processing elements 20a'' through 20f'' correlate with processing elements 20a' through 20f. respectivel~. of Figure ~. S~naptic elements 10a" through 10f''.
and 10t'' through 10z`' correlate with s~naptic elements 10a' through lOf and 10t' through 10z' respectivel~. of Figure ~. All signal lines, 11c through 1~c correlate to both the signal lines 11a' through 14a' of Figure 6 and signal lines 11b' through 14b' of Figure 7. The input. output and error lines 51' through ~6' correlate ~vith the input. output and error lines 51 though a~ of Figure 5. Synàptic elements 10a ' through 10f" and 10t'- through 10z'' mav be implemented as the circuit detailed in Figure 6. Processing elements 20a'' through 20f may be implemented as the circuit detailed in Figure ,.
An input stimulus vector, comprised of input stirnuli Input l on line ~1 and Input 2 on line ~2. are connected to processing elements 20a'' and ~Od''.
respectively. as is done in Figure ~. The output of processing element 20a'' is con-nected to synaptic elements 10a''. 10d", 10b''. and 10-v ' via Data line 11c'. Simi-larlv. the output of processing element 20d'' is connect to synaptic elements 10t''.
10u''. 10~' and 10x''. via Data line 11c''. S~naptic Elements 10a' and 10t'' sumtheir ~et Data outputs on ~'et Data line 13c. This summation on 13c' is pro-v ided as the ~'et Data inpuT to processing element 20b". Processing Element 20b'' pro- ides its Data OUtpUT signal on Data line 11c"'. to the Data input line of ~naptic l~lements 10e''. 10c'' and lOy''. S~naptic Elements 10d''. lOu'' and 10e'' sum their ~'et Data output signals on ~et Data line 13c''. ~ hich is provided asthe ~'et Dala input signal TO Processing Element 20e''. Processing Element 20e''provides its Data output signal on Data line 11c'"'. to the Data input line of .~naptic Elements lOf' and 10z''. Processing elements 10b". 10~'', lOc" and lOf'sum their :~'et Data output signals on ~et Data line 13c"'. u hich is provided as the ~\'et Data input signal to S-naptic E;lement ''Oc". Processing elements 10~
lOx''. 10~' and 10z sum their ~et Data output signals on ~'et Data line 13c'` '. hich is provided as the ~'et Data input jignal to ~naptic Element 20f'.
Processing elements ~Oc'' and 20f'' provide output signals Output 1 and Output 2. respectively. on lines 5~' and ~'. respectively. These outputs form the 128'7175 -2~-output ~ector.
An error stimulus vector. composed of error stimuli Error 1 on line ~3' and Error 2 on line ~4' are received b~ the ~et ~;rror lines of Processing Elements 20c and 20f, respectively, The ~rror output signal of Processing Elements ~Of'' is pro-~ided on E;rror line 14c'"' to ~ynaptic Elements 1O~A~' 10x'', 10y'' and 10z". The Error output signal of Processing Elements 20c'' is pro~ ided on Error line 1~c"' to Synaptic Elements 10f", 10c"~ 10~'' and 10b''. The ~\'et Error outputs of Synap-tic Elements lOf-' and 10z'' are summed on ~'et Error line 12c"'' and is pro~ided to the Net E;rror input line of Synaptic Element ''Oe''. The Error output signal of Processing Elements 20e'' is provided on Error line 1~c" to Synaptic Elements 10e''. 10u' and 10d'-. The ~-et Error outputs of ~ynaptic Elements lOe'', 10c".
and 10y" are summed on ~et Error line 12c'' and is provided to the ~'et Error input line of Synaptic Element 20b''. The Error output signal of Processing Ele-ments 20b'' is provided on Error line 1~c' to ~ynaptic Elements 10a' and 10t''.
The ~'et Error outputs of Synaptic Elements lOt'. 10u'. lOv and 10x'' are summed on ~'et Error line 12c" and is provided to the ~et Error input line of ~;~ naptic Element 20d". The ~et Error outputs of S- naptic Elements 10a''. 10d''.
I()b'' and 10~-'' are summed on ~'et Error line 1~c' and is pro~-ided tO the :~'et Error input line of .~! naptic Element 20a''.
In the example the Error output signals of Processing Elements . Oa'' and 20d'- are not used: often this will be the case. and as such a minimal system does not include the functional parts necessar- to pro~ide the Error output signal ofinput processing elernents. such as 20a' and "Od-'. nor the functional parts to pro-ide a .~'et Error output for the synaptic element.s connected to the input process-ing elenlents. The example is provided ~ith ~he Error output signals of Process-ing Elements 20a'' and "Od'' and the ~\'el Error outpur signals for ~naptic Ele-ments 10a". 10d". 10b'-. 10- ''. lOt''. 10u' . 10--'' and 10x for clarity and unifor-mit~ . A system Cell be built in this manner ~ ith no loss of generalit~ .
The input and output signals. along ith an- other signals of the circuitr .
may be either a voltage. current. frequenc~:. Iight amplitude. or other mechanism ~hich is capable of representing a ~ alue.
Figure 9a and 9b illustrate typical functions of the function generators 160 and 168 of figure 8. Function generator 16~ -hich provides the function F is 12871~S
t~picall~ characterized bv the equation:
F ( ~
1-e A t~p]cal function pro~ided b~- function generator lG8 is illustrated in figure 9b here~n F (~ e ~ 2~
The functions F and F are merel e~;emplar~ functions that can be approximatel~ implemented as a ladder net~A~ork of diodes and resistors. Howe~er.
other functions F and F- may be readil~ substituted. The function F is usuall~
the derivati~e of the function F.
It can be seen that by e~cpansion of the circ.uit of Figure 8 that much larger c ontinuous time neural s~stems can be constructed.
T~picall- the circuit eiements are formed in an ~ high (vertical) b~ ~I
wide (horizontal) matrix. .~s such~ there are ~ net error and ~ input lines. uith each being paired and running horizontall~ through ~;1 matrix elements. ;:imilarl~.
there are ~I delTa alld ~1 ner inpuf lines which are paired and each p~ir runninO
~erticall~ through ~ matrix elen-ienrs. The ~\~1 pairs of ~ertical delta and ner input lines are brought pair-- ise to the inpllt signal net input and the o uti-ut si mal delta of ~I processillg eiement circllit . The ~i pairs output and neT error iines from the processing elemenr circuirs can then be run through another ~I higsl b~L Y.-ide matrix b- connecting the siOnals respective!~ into t}le input signal input and output signal net error of the ~\I b~ L matrix.
The s~naptic circuiT and proessing element circuits are inserted in-o rr.atri:;-like con~gurations. The s~ naptic circuits are placed illtO a rectangular matri~; e~er~where a ~eight connection is desire(l het~ een tuo neurons. The pro-ccssing element circuits are placed e~er~here the rerminus of a neuron a:~on areplaed. Implemented in this fashion the s~stem ~ould then comprise a net~ or~;
hich i.c capable of performing the learning functior..
Howe~er. man~ net~orks ~ill not nee~l the le.lrning capabilit~. Once a s~c-tem has learned a particular function. the eighr ~alues ould be etched in~o a 128~7175 ., permanent memor~ for a lo~er cosr ~ersion than a ~er~ion capable of learning.
~uch a s!-stem ~ould need onl, the for~ ar~ flo~- of information; the back~ard Ho-~- and the ~veight update functions could be eliminated.
.~lthough in smaller s~stem the integrator ma- be implemented with capa-citors onlv~ this presents a dra~back in larger s.vstems due to the deca~ rate of the capacitors in the integrators. Howe-er. it is feasible that the weights rna~ be par-tiall~ stored digitallv and capacitors used onl~ for the low order bits of the data.
Learning in the Back Propagation s~stem is most often implemented b,v presenting an input and output pattern to the network and allouing it to adapt partially to the pattern before presenting the ne:;t pattern. It is noted that during the transition period between changing patterns the learnin~ rate tO should be set to zero to inhibit learning of improper information. An input pattern is presented to inputs of a total network ~hich then propaga~es through the nen ork to pro--ide an output pattern. .~n error is generated based from target or e~pected pat-tern. This error is then propagated back~- ard Ihrough the net~ orl; u ith a corresponding adjustment of the ~eights durinz the backvlvard propagations. Dur-ing this backward propagalion of error the weights are being updated bv e~;amin-inz rhe input and the error lines. The svnaptic and processing element circuits as described herein are quite adaptable to any learning system ~ hich implements the Back Propagation theory. There are probablv other learning s~stems to hich this svstem is also well fitted, In creating a pattern presentation svstem a ~imple e:;ternal circuit can be readily constructed. The svstem consists of rnemorv ~ ide enou"h to accommodate all the bil necessar.v for both the input and OUtpUT patterns. If an input or an OUT-put is tO be continuous analog (rather than having t~'O discrete analog values) then there must be a digital to analog converler for each input and output ~ hich is continuous. The memory for that channel ~ill need to be appropriatel~ - ide. In order to present the data. a simple sequencer and counter is all that is required.
The sequencer turns off the learning rate. steps the counter and turns the learning rate back on. The sequencer uould then ~ait for an appropriate time. and then repeat the steps. The counter ~ ould step through The memor~. c~cling one at ~
time for all of the patterns. .~lthough this s~stem could be stand-alone, it is more likel~ to be a co-processor or peripheral of a srandard computer. The standard ~L%~'7175 computer ~ould ~irst doun-load the information to the memory. The netttork would then undergo the learning process. .~t the end of the learning process thehost computer v.ould read bac~i the learned t~eights.
ln another e~cample. the circuit ~ ould also be able to perform in a continu-ous s,tstem. In specific exar.-.i,le. the netuork might be attached to the sensors and controllers of a robotic arm. The sensors ould pro--ide a continuous time representation of the external world. If the sensors are connected so as to provide both input and error signals the arm would learn to minimize the error signals.
In the feed forward function there is a requirement that there is a location to store the weight. This requirement is solved b~ the integrator which is assumed to hold a valid weight. The input stimulus to the neuron' comes in on the input line. This input line is shared b~ all the synaptic circuits which read this line.
I,'nless this input is a direct input from the external world this line would be con-nected to the output of a prior neuron. .~s such, this input line rnav sometimes be referred to as an output line. but rather than being the output of the current neu-ron. it is the output line of the prior neuron.
The s,vnaptic and processing element circuits as block diagramed in the figures ma,v be readilv implemented as electronic components. The multipliers are typically four quadrant multipliers. The integrator is simpl~ an operational amplifier with a feedback capacitor. The F and F' functions are readil~ made using simple ladder networks of resistors and diodes. The precision of the F and F' functions are not critical. however it is necessary that the F' function is the derivative of the function F. There ma!~ be other~ more readil! implemenTed function generators available.
The po--er function for the feedbacli raised weight is of the form ~I~R. ~here R is a positive constant. From the mathematics it is clear that R can be an- non-negative constant. Experimental results indicate 0.~ wor};s well (square root).
This is a readilv implemented function in a circuit element. .9.lso, a t alue of 1.0 for R has been used: this function is easil~ implemented as a wire. Finall,v. the standard bac}c propagation uses a talue of 0. uherein the uhole circuit can be remot-ed (as w = 1, assuminV 0 = 1).
Implementation of the s!~naptic and processing element circuits is readil~
adaptable to fabrication in inte~rated circuits. Based upon the horizontal and ~2~717S
~-ertical matri~; lavout the topovraph~v of the circuit ~ould be rather fundamental in design in building a net- or~i based on these builcling blocks. The synaptic and processing e]ement circuits are analog building blocXs of entire neural ne~worl;s useful in .~rtificial Intelligence systems.
The input processing elements. rather than being implemented as detailed.
may be implemented as feed-through units. not performing an~ calculations on the input stimulus. This would provide. as the input to the s~,naptic elements adirect fanout of the input stimulus to the s~naptic elements and the input pro-cessing elements could be implemented as a wire in an electronic implementation In manv applications. this is the method of choice. rather than as detailed l)v the drawings. utilizing a complete processing element.
.4.lthough particular embodiments of the invention ha~ e been illustrated and described. modifications and changes will become apparent to those skilled in the art. and it is intended to cover in the appended claims such rnodifications and changes as corne ~-ithin the true spirit and scope of the invelltion.
~hat I claim is:
Claims (7)
1. A continuous time information processing system which adapts, through the backward propagation of error stimuli, to translate input stimuli to expected output responses, comprising:
an input layer comprising at least one input processing element: a hid-den layer comprising at least one hidden processing element: an output layer comprising at least one output processing element:
wherein:
each input processing element comprises an input Net Data, an output Data, and a Data transfer function:
each hidden processing element comprises an input Net Data, an out-put Data an input Net Error, an output Error a Data transfer function, an Error transfer function a Net Data Summation function and a Net Error Summation function:
each output processing element comprises an input Net Data, an out-put Data an input Net Error, an output Error a Data transfer function, an Error transfer function and a Net Data Summation function:
an output Data is a value: an output Error is a value: an input Net Data is a value: an input Net Error is a value:
said system further comprising:
an input vector comprised of one or more input stimuli: each input stimulus consisting of a value: each said input stimulus being associated with one said input processing element: each said input stimulus connected to said input Net Data of its said associated input processing element:
an output vector comprised of one or more output responses: each out-put response consisting of a value: each said output response being associated with one said output processing element; each said output response con-nected to said output Data of its said associated output processing element:
an error vector comprised of one or more error stimuli: each error stimulus consisting of a value: each said error stimulus being associated with one said output processing element: each said error stimulus connected to said input Net Error of its said associated output processing element: and at least two synaptic elements. at least one being an input synaptic element, at least one being an output synaptic element and zero or more being a direct synaptic element:
wherein:
each synaptic element couples two processing elements, one defined as the processing element below it and the other defined as the processing ele-ment above it:
each input synaptic element couples an input processing element below it with a hidden processing element above it: each output synaptic element couples a hidden processing element below it with an output processing ele-ment above it:
each input synaptic element comprises an input Data, an output Net Data, an input Error and a modifiable synaptic transfer function:
each output synaptic element comprises an input Data, an output Net Data an input Error, an output Net Error and a modifiable synaptic transfer function:
an input Data is a value: an input Error is a value: an output Net Data is a value: an output Net Error is a value:
each input Data of a synaptic element is connected to the output Data of the processing element below it, each input Error of a synaptic element is connected to the output Error of the processing element above it, each output Net Data of a synaptic element is connected to an input of the Net Data Summation function of the processing element above it:
each output Net Error of an output synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
the input Net Data of an input processing element receives the value of its associated input stimulus which then becomes the value of input Net Data of that input processing element:
the Data transfer function of an input processing element operates on the input Net Data value of that input processing element to produce the output Data value of that input processing element:
the input Data of an input synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that input synaptic element:
the Synaptic Transfer function of the input synaptic element operates on the input Data value of that input synaptic element to produce the output Net Data value of that input synaptic element:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that hidden processing element:
the Data Transfer function of a hidden processing element operates on the input Net Data value of that hidden processing element to produce the output Data value of that hidden processing element:
the input Data of an output synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Data value of that output synaptic element to produce the output Net Data value of that output synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the output synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that output processing element:
the Data Transfer function of an output processing element operates on the input Net Data value of that output processing element to produce the output Data value of that output processing element:
an output response of the output vector receives the output Data value of its associated output processing element which then becomes the output response value:
the input Net Error of an output processing element receives the value of its associated error stimulus which then becomes the value of input Net Error of that output processing element:
the Error transfer function of an output processing element operates on the input Net Data value and the input Net Error value of that output processing element to produce the output Error value of that output process-ing element:
the input Error of an output synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Error value of that output synaptic element to produce the output Net Error value of that output synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the output synaptic elements coupled thereto, to produce a value which becomes the input Net Error value of that hidden processing element:
the Error Transfer function of a hidden processing element operates on the input Net Data value and the input Net Error value of that hidden pro-cessing element to produce the output Error value of that hidden processing element: and the input Error of an input synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that input synaptic element:
said system further comprising:
means for modifying the synaptic transfer function of the synaptic ele-ments: wherein: each synaptic transfer function being modified as a function of the input Error value and the input Data value of that synaptic element:
an input layer comprising at least one input processing element: a hid-den layer comprising at least one hidden processing element: an output layer comprising at least one output processing element:
wherein:
each input processing element comprises an input Net Data, an output Data, and a Data transfer function:
each hidden processing element comprises an input Net Data, an out-put Data an input Net Error, an output Error a Data transfer function, an Error transfer function a Net Data Summation function and a Net Error Summation function:
each output processing element comprises an input Net Data, an out-put Data an input Net Error, an output Error a Data transfer function, an Error transfer function and a Net Data Summation function:
an output Data is a value: an output Error is a value: an input Net Data is a value: an input Net Error is a value:
said system further comprising:
an input vector comprised of one or more input stimuli: each input stimulus consisting of a value: each said input stimulus being associated with one said input processing element: each said input stimulus connected to said input Net Data of its said associated input processing element:
an output vector comprised of one or more output responses: each out-put response consisting of a value: each said output response being associated with one said output processing element; each said output response con-nected to said output Data of its said associated output processing element:
an error vector comprised of one or more error stimuli: each error stimulus consisting of a value: each said error stimulus being associated with one said output processing element: each said error stimulus connected to said input Net Error of its said associated output processing element: and at least two synaptic elements. at least one being an input synaptic element, at least one being an output synaptic element and zero or more being a direct synaptic element:
wherein:
each synaptic element couples two processing elements, one defined as the processing element below it and the other defined as the processing ele-ment above it:
each input synaptic element couples an input processing element below it with a hidden processing element above it: each output synaptic element couples a hidden processing element below it with an output processing ele-ment above it:
each input synaptic element comprises an input Data, an output Net Data, an input Error and a modifiable synaptic transfer function:
each output synaptic element comprises an input Data, an output Net Data an input Error, an output Net Error and a modifiable synaptic transfer function:
an input Data is a value: an input Error is a value: an output Net Data is a value: an output Net Error is a value:
each input Data of a synaptic element is connected to the output Data of the processing element below it, each input Error of a synaptic element is connected to the output Error of the processing element above it, each output Net Data of a synaptic element is connected to an input of the Net Data Summation function of the processing element above it:
each output Net Error of an output synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
the input Net Data of an input processing element receives the value of its associated input stimulus which then becomes the value of input Net Data of that input processing element:
the Data transfer function of an input processing element operates on the input Net Data value of that input processing element to produce the output Data value of that input processing element:
the input Data of an input synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that input synaptic element:
the Synaptic Transfer function of the input synaptic element operates on the input Data value of that input synaptic element to produce the output Net Data value of that input synaptic element:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that hidden processing element:
the Data Transfer function of a hidden processing element operates on the input Net Data value of that hidden processing element to produce the output Data value of that hidden processing element:
the input Data of an output synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Data value of that output synaptic element to produce the output Net Data value of that output synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the output synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that output processing element:
the Data Transfer function of an output processing element operates on the input Net Data value of that output processing element to produce the output Data value of that output processing element:
an output response of the output vector receives the output Data value of its associated output processing element which then becomes the output response value:
the input Net Error of an output processing element receives the value of its associated error stimulus which then becomes the value of input Net Error of that output processing element:
the Error transfer function of an output processing element operates on the input Net Data value and the input Net Error value of that output processing element to produce the output Error value of that output process-ing element:
the input Error of an output synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Error value of that output synaptic element to produce the output Net Error value of that output synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the output synaptic elements coupled thereto, to produce a value which becomes the input Net Error value of that hidden processing element:
the Error Transfer function of a hidden processing element operates on the input Net Data value and the input Net Error value of that hidden pro-cessing element to produce the output Error value of that hidden processing element: and the input Error of an input synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that input synaptic element:
said system further comprising:
means for modifying the synaptic transfer function of the synaptic ele-ments: wherein: each synaptic transfer function being modified as a function of the input Error value and the input Data value of that synaptic element:
2. The information processing system of Claim 1, additionally comprising:
a plurality of hidden processing elements in the hidden layer, identified with identifying numbers 1 to N: and at least one hidden synaptic element; wherein each hidden synaptic element couples one hidden processing element below it with another above it: and each hidden processing element below a hidden synaptic element hav-ing an identifying number less than the identifying number of the hidden processing element above that hidden synaptic element, each hidden synaptic element comprising an input Data, all output Net Data, an input Error, an output Net Error and a modifiable synaptic transfer function:
each output Net Error of a hidden synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
wherein:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements and hidden synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that hidden processing element:
the input Data of an hidden synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that hidden synaptic element:
the Synaptic Transfer function of the hidden synaptic element operates on the input Data value of that hidden synaptic element to produce the output Net Data value of that hidden synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the hidden synaptic elements and output synaptic elements coupled thereto, to produce a value which becomes the input Net Error value of that hidden processing element:
the input Error of a hidden synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that hidden synaptic element:
the Synaptic Transfer function of the hidden synaptic element operates on the input Error value of that hidden synaptic element to produce the output Net Error value of that hidden synaptic element:
a plurality of hidden processing elements in the hidden layer, identified with identifying numbers 1 to N: and at least one hidden synaptic element; wherein each hidden synaptic element couples one hidden processing element below it with another above it: and each hidden processing element below a hidden synaptic element hav-ing an identifying number less than the identifying number of the hidden processing element above that hidden synaptic element, each hidden synaptic element comprising an input Data, all output Net Data, an input Error, an output Net Error and a modifiable synaptic transfer function:
each output Net Error of a hidden synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
wherein:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements and hidden synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that hidden processing element:
the input Data of an hidden synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that hidden synaptic element:
the Synaptic Transfer function of the hidden synaptic element operates on the input Data value of that hidden synaptic element to produce the output Net Data value of that hidden synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the hidden synaptic elements and output synaptic elements coupled thereto, to produce a value which becomes the input Net Error value of that hidden processing element:
the input Error of a hidden synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that hidden synaptic element:
the Synaptic Transfer function of the hidden synaptic element operates on the input Error value of that hidden synaptic element to produce the output Net Error value of that hidden synaptic element:
3. The information processing system of Claim 1, additionally comprising:
at least one direct synaptic element: wherein each direct synaptic element couples an input processing element below it with an output processing elements above it: and each direct synaptic element comprising an input Data, an output Net Data, an input Error, and a modifiable synaptic transfer function:
wherein:
the input Data of a direct synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that direct synaptic element:
the Synaptic Transfer function of the direct synaptic element operates on the input Data value of that direct synaptic element to produce the out-put Net Data value of that direct synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the direct synaptic elements in addition to that of the output synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that output processing element:
the input Error of a direct synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that direct synaptic element;
at least one direct synaptic element: wherein each direct synaptic element couples an input processing element below it with an output processing elements above it: and each direct synaptic element comprising an input Data, an output Net Data, an input Error, and a modifiable synaptic transfer function:
wherein:
the input Data of a direct synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that direct synaptic element:
the Synaptic Transfer function of the direct synaptic element operates on the input Data value of that direct synaptic element to produce the out-put Net Data value of that direct synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the direct synaptic elements in addition to that of the output synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that output processing element:
the input Error of a direct synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that direct synaptic element;
4. A continuous time information processing system which adapts, through the backward propagation of error stimuli, to translate input stimuli to expected output responses, comprising:
an input layer comprising at least one input processing element: a hid-den layer comprising at least one hidden processing elements, identified with identifying numbers 1 to N; an output layer comprising at least one output processing element:
wherein:
each input processing element comprises an input Net Data, an output Data, and a Data transfer function:
each hidden processing element comprises an input Net Data, an out-put Data, an input Net Error, an output Error, a Data transfer function, an Error transfer function, a Net Data Summation function and a Net Error Summation function:
each output processing element comprises an input Net Data, an out-put Data, an input Net Error, an output Error, a Data transfer function, an Error transfer function, and a Net Data Summation function:
an output Data is a value: an output Error is a value: an input Net Data is a value: and an input Net Error is a value:
said system further comprising:
an input vector comprised of one or more input stimuli: each input stimulus consisting of a value: each said input stimulus being associated with one said input processing element: each said input stimulus connected to said input Net Data of its said associated input processing element:
an output vector comprised of one or more output responses: each out-put response consisting of a value: each said output response being associated with one said output processing element: each said output response connected to said output Data of its said associated output processing element:
an error vector comprised of one or more error stimuli: each error stimulus consisting of a value: each said error stimulus being associated with one said output processing element: each said error stimulus connected to said input Net Error of its said associated output processing element:
at least two synaptic elements. at least one being an input synaptic element. at least one being an output synaptic element. zero or more being a direct synaptic element and zero or more being a hidden synaptic element:
each synaptic element couples two processing elements, one defined as the processing element below it and the other defined as the processing ele-ment above it:
each input synaptic element couples an input processing element below it with a hidden processing element above it: each direct synaptic element couples an input processing element below it with an output processing ele-ment above it: each hidden synaptic element couples one hidden processing element below it with another hidden processing element above it: each hid-den processing element below a hidden synaptic element having an identify-ing number less than the identifying number of the hidden processing ele-ment above that hidden synaptic clement each output synaptic element couples a hidden processing element below it with an output processing ele-ment above it:
each input synaptic element comprises an input Data. an output Net Data. an input Error. and a modifiable synaptic transfer function:
each direct synaptic element comprises an input Data. an output Net Data. an input Error. and a modifiable synaptic transfer function:
each hidden synaptic element comprises an input Data. an output Net Data. an input Error. an output Net Error. and a modifiable synaptic transfer function:
each output synaptic element comprises an input Data. an output Net Data. an input Error an output Net Error. and a modifiable synaptic transfer function;
an input Data is a value: an input Error is a value: an output Net Data is a value: and an output Net Error is a value:
each input Data of a synaptic element is connected to the output Data of the processing element below it.
each input Error of a synaptic element is connected to the output Error of the processing element above it.
each output Net Data of a synaptic element is connected to an input of the Net Data Summation function of the processing element above it:
each output Net Error of a hidden synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
each output Net Error of an output synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
wherein:
the input Net Data of an input processing element receives the value of its associated input stimulus which then becomes the value of input Net Data of that input processing element:
the Data transfer function of an input processing element operates on the input Net Data value of that input processing element to produce the output Data value of that input processing element:
the input Data of an input synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that input synaptic element:
the synaptic Transfer function of the input synaptic element operates on the input Data value of that input synaptic element to produce the output Net Data value of that input synaptic element:
the input Data of a direct synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that direct synaptic element:
the Synaptic Transfer function of the direct synaptic element operates on the input Data value of that direct synaptic element to produce the output Net Data value of that direct synaptic element:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements and hidden synaptic elements coupled thereto to produce a value which becomes the input Net Data value of that hidden processing element the Data Transfer function of a hidden processing element operates on the input Net Data value of that hidden processing element to produce the output Data value of that hidden processing element the input Data of an output synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that output synaptic element:
the input Data of an hidden synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that hidden synaptic element;
the Synaptic Transfer function of the hidden synaptic element operates on the input Data value of that hidden synaptic element to produce the output Net Data value of that hidden synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Data value of that output synaptic element to produce the output Net Data value of that output synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the direct synaptic elements in addition to that of the output synaptic elements coupled thereto to produce a value which becomes the input Net Data value of that output processing element:
the Data Transfer function of an output processing element operates on the input Net Data value of that output processing element to produce the output Data value of that output processing element:
an output response of the output vector receives the output Data value of its associated output processing element which then becomes the output response value:
the input Net Error of an output processing element receives the value of its associated error stimulus which then becomes the value of input Net Error of that output processing element:
the Error transfer function of all output processing element operates on the input Net Data value and the input Net Error value of that output processing element to produce the output Error value of that output process-ing element:
the input Error of a direct synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that direct synaptic element:
the input Error of an output synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Error value of that output synaptic element to produce the output Net Error value of that output synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the hidden synaptic elements and output synaptic elements coupled thereto. to produce a value which becomes the input Net Error value of that hidden processing element:
the Error Transfer function of a hidden processing element operates on the input Net Data value and the input Net Error value of that hidden pro-cessing element to produce the output Error value of that hidden processing element:
the input Error of a hidden synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that hidden synaptic element:
the Synaptic Transfer function of the hidden synaptic element operates on the input Error value of that hidden synaptic element to produce the output Net Error value of that hidden synaptic element:
the input Error of an input synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that input synaptic element:
means for modifying the synaptic transfer function of the synaptic ele-ments. each synaptic transfer function being modified as a function of the input Error value and the input Data value of that synaptic element.
an input layer comprising at least one input processing element: a hid-den layer comprising at least one hidden processing elements, identified with identifying numbers 1 to N; an output layer comprising at least one output processing element:
wherein:
each input processing element comprises an input Net Data, an output Data, and a Data transfer function:
each hidden processing element comprises an input Net Data, an out-put Data, an input Net Error, an output Error, a Data transfer function, an Error transfer function, a Net Data Summation function and a Net Error Summation function:
each output processing element comprises an input Net Data, an out-put Data, an input Net Error, an output Error, a Data transfer function, an Error transfer function, and a Net Data Summation function:
an output Data is a value: an output Error is a value: an input Net Data is a value: and an input Net Error is a value:
said system further comprising:
an input vector comprised of one or more input stimuli: each input stimulus consisting of a value: each said input stimulus being associated with one said input processing element: each said input stimulus connected to said input Net Data of its said associated input processing element:
an output vector comprised of one or more output responses: each out-put response consisting of a value: each said output response being associated with one said output processing element: each said output response connected to said output Data of its said associated output processing element:
an error vector comprised of one or more error stimuli: each error stimulus consisting of a value: each said error stimulus being associated with one said output processing element: each said error stimulus connected to said input Net Error of its said associated output processing element:
at least two synaptic elements. at least one being an input synaptic element. at least one being an output synaptic element. zero or more being a direct synaptic element and zero or more being a hidden synaptic element:
each synaptic element couples two processing elements, one defined as the processing element below it and the other defined as the processing ele-ment above it:
each input synaptic element couples an input processing element below it with a hidden processing element above it: each direct synaptic element couples an input processing element below it with an output processing ele-ment above it: each hidden synaptic element couples one hidden processing element below it with another hidden processing element above it: each hid-den processing element below a hidden synaptic element having an identify-ing number less than the identifying number of the hidden processing ele-ment above that hidden synaptic clement each output synaptic element couples a hidden processing element below it with an output processing ele-ment above it:
each input synaptic element comprises an input Data. an output Net Data. an input Error. and a modifiable synaptic transfer function:
each direct synaptic element comprises an input Data. an output Net Data. an input Error. and a modifiable synaptic transfer function:
each hidden synaptic element comprises an input Data. an output Net Data. an input Error. an output Net Error. and a modifiable synaptic transfer function:
each output synaptic element comprises an input Data. an output Net Data. an input Error an output Net Error. and a modifiable synaptic transfer function;
an input Data is a value: an input Error is a value: an output Net Data is a value: and an output Net Error is a value:
each input Data of a synaptic element is connected to the output Data of the processing element below it.
each input Error of a synaptic element is connected to the output Error of the processing element above it.
each output Net Data of a synaptic element is connected to an input of the Net Data Summation function of the processing element above it:
each output Net Error of a hidden synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
each output Net Error of an output synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
wherein:
the input Net Data of an input processing element receives the value of its associated input stimulus which then becomes the value of input Net Data of that input processing element:
the Data transfer function of an input processing element operates on the input Net Data value of that input processing element to produce the output Data value of that input processing element:
the input Data of an input synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that input synaptic element:
the synaptic Transfer function of the input synaptic element operates on the input Data value of that input synaptic element to produce the output Net Data value of that input synaptic element:
the input Data of a direct synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that direct synaptic element:
the Synaptic Transfer function of the direct synaptic element operates on the input Data value of that direct synaptic element to produce the output Net Data value of that direct synaptic element:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements and hidden synaptic elements coupled thereto to produce a value which becomes the input Net Data value of that hidden processing element the Data Transfer function of a hidden processing element operates on the input Net Data value of that hidden processing element to produce the output Data value of that hidden processing element the input Data of an output synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that output synaptic element:
the input Data of an hidden synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that hidden synaptic element;
the Synaptic Transfer function of the hidden synaptic element operates on the input Data value of that hidden synaptic element to produce the output Net Data value of that hidden synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Data value of that output synaptic element to produce the output Net Data value of that output synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the direct synaptic elements in addition to that of the output synaptic elements coupled thereto to produce a value which becomes the input Net Data value of that output processing element:
the Data Transfer function of an output processing element operates on the input Net Data value of that output processing element to produce the output Data value of that output processing element:
an output response of the output vector receives the output Data value of its associated output processing element which then becomes the output response value:
the input Net Error of an output processing element receives the value of its associated error stimulus which then becomes the value of input Net Error of that output processing element:
the Error transfer function of all output processing element operates on the input Net Data value and the input Net Error value of that output processing element to produce the output Error value of that output process-ing element:
the input Error of a direct synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that direct synaptic element:
the input Error of an output synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Error value of that output synaptic element to produce the output Net Error value of that output synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the hidden synaptic elements and output synaptic elements coupled thereto. to produce a value which becomes the input Net Error value of that hidden processing element:
the Error Transfer function of a hidden processing element operates on the input Net Data value and the input Net Error value of that hidden pro-cessing element to produce the output Error value of that hidden processing element:
the input Error of a hidden synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that hidden synaptic element:
the Synaptic Transfer function of the hidden synaptic element operates on the input Error value of that hidden synaptic element to produce the output Net Error value of that hidden synaptic element:
the input Error of an input synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that input synaptic element:
means for modifying the synaptic transfer function of the synaptic ele-ments. each synaptic transfer function being modified as a function of the input Error value and the input Data value of that synaptic element.
5. A continuous time information processing system which adapts. through the backward propagation of error stimuli to translate input stimuli to expected output responses.
comprising:
an input layer comprising at least one input processing element: a hid-den layer comprising at least one hidden processing element: an output layer comprising at least one output processing element:
wherein:
each input processing element comprises an input Net Data. an output Data. and a Data transfer function:
each hidden processing element comprises an input Net Data an out-put Data an input Net Error, an output Error a Data transfer function an Error transfer function a Net Data Summation function and a Net Error Summation function:
each output processing element comprises an input Net Data an out-put Data an input Net Error an output Error a Data transfer function an Error transfer function and a Net Data Summation function:
an output Data is a value: an output Error is a value: an input Net Data is a value: all input Net Error is a value:
said system further comprising:
an input vector comprised of one or more input stimuli: each input stimulus consisting of a value: each said input stimulus being associated with one said input processing element: each said input stimulus connected to said input Net Data of its said associated input processing element:
an output vector comprised of one or more output responses: each out-put response consisting of a value: each said output response being associated with one said output processing element: each said output response con-nected to said output Data of its said associated output processing element:
an error vector comprised of one or more error stimuli: each error stimulus consisting of a value each said error stimulus being associated with one said output processing element each said error stimulus connected to said input Net Error of its said associated output processing element: and at least two synaptic elements at least one being an input synaptic element at least one being an output synaptic element and zero or more being a direct synaptic element:
wherein:
each synaptic element couples two processing elements one defined as the processing element below it and the other defined as the processing ele-ment above it:
each input synaptic element couples an input processing element below it with a hidden processing element above it each output synaptic element couples a hidden processing element below it with an output processing ele-ment above it:
each input synaptic element comprises an input Data an output Net Data an input Error and a modifiable synaptic transfer function:
each output synaptic element comprises an input Data an output Net Data an input Error an output Net Error and a modifiable synaptic transfer function:
an input Data is a value an input Error is a value an output Net Data is a value: an output Net Error is a value:
each input Data of a synaptic element is connected to the output Data of the processing element below it.
each input Error of a synaptic element is connected to the output Error of the processing element above it.
each output Net Data of a synaptic element is connected to an input of the Net Data Summation function of the processing element above it each output Net Error of an output synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
the input Net Data of an input processing element receives the value of its associated input stimulus which then becomes the value of input Net Data of that input processing element:
the Data transfer function of an input processing element operates on the input Net Data value of that input processing element to produce the output Data value of that input processing element:
the input Data of an input synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that input synaptic element:
the Synaptic Transfer function of the input synaptic element operates on the input Data value of that input synaptic element to produce the output Net Data value of that input synaptic element:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that hidden processing element:
the Data Transfer function of a hidden processing element operates on the input Net Data value of that hidden processing element to produce the output Data value of that hidden processing element:
the input Data of an output synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Data value of that output synaptic element to produce the output Net Data value of that output synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the output synaptic elements coupled thereto to produce a value which becomes the input Net Data value of that output processing element:
the Data Transfer function of an output processing element operates on the input Net Data value of that output processing element to produce the output Data value of that output processing element:
an output response of the output vector receives the output Data value of its associated output processing element which then becomes the output response value:
the input Net Error of an output processing element receives the value of its associated error stimulus which then becomes the value of input Net Error of that output processing element:
the Error transfer function of an output processing element operates on the input Net Data value and the input Net Error value of that output processing element to produce the output Error value of that output process-ing element:
the input Error of an output synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Error value of that output synaptic element to produce the output Net Error value of that output synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the output synaptic elements coupled thereto. to produce a value which becomes the input Net Error value of that hidden processing element:
the Error Transfer function of a hidden processing element operates on the input Net Data value and the input Net Error value of that hidden pro-cessing element to produce the output Error value of that hidden processing element: and the input Error of an input synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that input synaptic element:
said s stem further comprising:
means for modifying the synaptic transfer function of the synaptic ele-ments: wherein: each synaptic transfer function being modified as a function of the input Error value and the input Data value of that synaptic element:
wherein:
said output synaptic element is a circuit comprising:
first multiplication means for receiving an input Data value a power value and an input Error value and providing a product value:
integration means for receiving said product value and providing a weight value corresponding to this integral of said product value over time second multiplication means for receiving and multiplying said input Data value and said weight value so as to provide an output Net Data value means for raising by a predetermined constant said weight value as provided from said integration means and providing a power value to said first multiplication means third multiplication means for receiving and multiplying said input Error value and said weight value so as to provide an output Net Error out-put value
comprising:
an input layer comprising at least one input processing element: a hid-den layer comprising at least one hidden processing element: an output layer comprising at least one output processing element:
wherein:
each input processing element comprises an input Net Data. an output Data. and a Data transfer function:
each hidden processing element comprises an input Net Data an out-put Data an input Net Error, an output Error a Data transfer function an Error transfer function a Net Data Summation function and a Net Error Summation function:
each output processing element comprises an input Net Data an out-put Data an input Net Error an output Error a Data transfer function an Error transfer function and a Net Data Summation function:
an output Data is a value: an output Error is a value: an input Net Data is a value: all input Net Error is a value:
said system further comprising:
an input vector comprised of one or more input stimuli: each input stimulus consisting of a value: each said input stimulus being associated with one said input processing element: each said input stimulus connected to said input Net Data of its said associated input processing element:
an output vector comprised of one or more output responses: each out-put response consisting of a value: each said output response being associated with one said output processing element: each said output response con-nected to said output Data of its said associated output processing element:
an error vector comprised of one or more error stimuli: each error stimulus consisting of a value each said error stimulus being associated with one said output processing element each said error stimulus connected to said input Net Error of its said associated output processing element: and at least two synaptic elements at least one being an input synaptic element at least one being an output synaptic element and zero or more being a direct synaptic element:
wherein:
each synaptic element couples two processing elements one defined as the processing element below it and the other defined as the processing ele-ment above it:
each input synaptic element couples an input processing element below it with a hidden processing element above it each output synaptic element couples a hidden processing element below it with an output processing ele-ment above it:
each input synaptic element comprises an input Data an output Net Data an input Error and a modifiable synaptic transfer function:
each output synaptic element comprises an input Data an output Net Data an input Error an output Net Error and a modifiable synaptic transfer function:
an input Data is a value an input Error is a value an output Net Data is a value: an output Net Error is a value:
each input Data of a synaptic element is connected to the output Data of the processing element below it.
each input Error of a synaptic element is connected to the output Error of the processing element above it.
each output Net Data of a synaptic element is connected to an input of the Net Data Summation function of the processing element above it each output Net Error of an output synaptic element is connected to an input of the Net Error Summation function of the processing element below it:
the input Net Data of an input processing element receives the value of its associated input stimulus which then becomes the value of input Net Data of that input processing element:
the Data transfer function of an input processing element operates on the input Net Data value of that input processing element to produce the output Data value of that input processing element:
the input Data of an input synaptic element receives the output Data value of the processing element below it which then becomes the value of the input Data of that input synaptic element:
the Synaptic Transfer function of the input synaptic element operates on the input Data value of that input synaptic element to produce the output Net Data value of that input synaptic element:
the Net Data Summation function of a hidden processing element receives and operates on the output Net Data values of the input synaptic elements coupled thereto, to produce a value which becomes the input Net Data value of that hidden processing element:
the Data Transfer function of a hidden processing element operates on the input Net Data value of that hidden processing element to produce the output Data value of that hidden processing element:
the input Data of an output synaptic element receives the output Data value of the processing element below it which then becomes the input Data value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Data value of that output synaptic element to produce the output Net Data value of that output synaptic element:
the Net Data Summation function of an output processing element receives and operates on the output Net Data values of the output synaptic elements coupled thereto to produce a value which becomes the input Net Data value of that output processing element:
the Data Transfer function of an output processing element operates on the input Net Data value of that output processing element to produce the output Data value of that output processing element:
an output response of the output vector receives the output Data value of its associated output processing element which then becomes the output response value:
the input Net Error of an output processing element receives the value of its associated error stimulus which then becomes the value of input Net Error of that output processing element:
the Error transfer function of an output processing element operates on the input Net Data value and the input Net Error value of that output processing element to produce the output Error value of that output process-ing element:
the input Error of an output synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that output synaptic element:
the Synaptic Transfer function of the output synaptic element operates on the input Error value of that output synaptic element to produce the output Net Error value of that output synaptic element:
the Net Error Summation function of a hidden processing element receives and operates on the output Net Error values of the output synaptic elements coupled thereto. to produce a value which becomes the input Net Error value of that hidden processing element:
the Error Transfer function of a hidden processing element operates on the input Net Data value and the input Net Error value of that hidden pro-cessing element to produce the output Error value of that hidden processing element: and the input Error of an input synaptic element receives the output Error value of the processing element above it which then becomes the input Error value of that input synaptic element:
said s stem further comprising:
means for modifying the synaptic transfer function of the synaptic ele-ments: wherein: each synaptic transfer function being modified as a function of the input Error value and the input Data value of that synaptic element:
wherein:
said output synaptic element is a circuit comprising:
first multiplication means for receiving an input Data value a power value and an input Error value and providing a product value:
integration means for receiving said product value and providing a weight value corresponding to this integral of said product value over time second multiplication means for receiving and multiplying said input Data value and said weight value so as to provide an output Net Data value means for raising by a predetermined constant said weight value as provided from said integration means and providing a power value to said first multiplication means third multiplication means for receiving and multiplying said input Error value and said weight value so as to provide an output Net Error out-put value
6. An adaptive continuous-time, bidirectional matrix multiplier element comprising:
first multiplication means for receiving an input Data signal, a power signal and an input Error signal and providing a product signal:
integration means for receiving said product signal and providing a weight signal corresponding to the integral of said product signal over time:
second multiplication means for receiving and multiplying said input Data signal and said weight signal so as to provide an output Net Data sig-nal:
means for raising by a predetermined constant said weight signal as provided from said integration means and providing a power signal to said first multiplication means:
third multiplication means for receiving and multiplying said input Error signal and said weight signal so as lo provide an output Net Error out-put signal:
means for summing the outputs of all matrix elements to compute the vector result of the matrix multiplication.
first multiplication means for receiving an input Data signal, a power signal and an input Error signal and providing a product signal:
integration means for receiving said product signal and providing a weight signal corresponding to the integral of said product signal over time:
second multiplication means for receiving and multiplying said input Data signal and said weight signal so as to provide an output Net Data sig-nal:
means for raising by a predetermined constant said weight signal as provided from said integration means and providing a power signal to said first multiplication means:
third multiplication means for receiving and multiplying said input Error signal and said weight signal so as lo provide an output Net Error out-put signal:
means for summing the outputs of all matrix elements to compute the vector result of the matrix multiplication.
7. A continuous time information processing system which adapts utilizing the backward propagation of error signals to translate input signals to expected output response signals.
comprising:
at least one input fan-out circuit: at least one input synaptic circuit: at least one hidden processing circuit: at least one output synaptic circuit: and at least one output processing circuit:
wherein:
each input fan-out circuit comprises an input Net Data line and an output Data line:
each hidden processing circuit comprises one or more input Net Data lines one or more input Net Error lines an output Data line an output Error lines a first transfer function a second transfer function a first multiplication means a first summation means and a second summation means:
each output processing circuit comprises one or more input Net Data lines. an input Net Error line.
an output Data line an output Error line a first transfer function a second transfer function a first multiplication means and a second summa-tion means:
each input synaptic circuit comprises an input Data line an output Net Data line. an input Error line a distribution weight line. a distribution weight prime line. a second multiplication means a first integration means and a third multiplication means:
each output synaptic circuit comprises an input Data line an output Net Data line. an input Error line. an output Net Error line a distribution weight line a distribution weight prime line. a fourth multiplication means a second integration means a fifth multiplication means and a sixth multiplica-tion means: and each input output and distribution line transmits a signal from a sin-gle input to one or more outputs:
said system further comprising:
an input vector comprised of one or more input data stimulus lines:
wherein: each said input data stimulus line is associated with one said input fan-out circuit: each said input data stimulus line connects said input Net Data line of its said associated input fan-out circuit:
an output vector comprised of one or more output response lines:
wherein: each said output response line is associated with one said output processing circuit: each said output response line connects said output Data line of its said associated output processing circuit:
an error vector comprised of one or more input error stimulus lines:
wherein: each said input error stimulus line is associated with one said output processing circuit: each said input error stimulus line connects said input Net Error line of its said associated output processing circuit:
wherein:
an input synaptic circuit couples between an input fan-out circuit and a hidden processing circuit:
an output synaptic circuit couples between a hidden processing circuit and an output processing circuit:
each output Data line of an input fan-out circuit connects to an input Data line of an input synaptic circuit:
each input Data line of an input synaptic circuit connects to the out-put Data line of an input fan-out circuit: each input Error line of an input synaptic circuit connects to the output Error line of a hidden processing cir-cuit:
each output Net Data line of an input synaptic circuit connects to one of the input Net Data lines of a hidden processing circuit:
each output Net Data line of an output synaptic circuit connects to one of the input Net Data lines of an output processing circuit: each output Net Error line of an output synaptic circuit connects to one of the input Net Error lines of a hidden processing circuit:
each input Data line of a hidden synaptic circuit connects to the out-put Data line of a hidden processing circuit: each input Error line of a hidden synaptic circuit connects to the output Error line of a hidden processing cir-cuit:
each summing means receives one or more signals from separate lines and produces a signal which is the mathematical summation of the received signals:
each multiplication means receives signals from a plurality of lines and produces a signal which is the mathematical multiplication of the received signals:
each integration means receives a signal from a line and produces a signal which is the mathematical integration over time of the received signal:
each transfer function means receives a signal from a line and produces some predetermined function of that received signal:
each input Net Data line of an input fan-out circuit is connected directly to the output Data line of that input fan-out circuit and passes the signal thereon directly without change:
each first summation means receives one or more signals from the input Net Data lines of the processing circuit to produce a signal on the dis-tribution Net Data Summation line:
each first transfer function means receives a signal from the first sum-ming means on the distribution Net Data Summation line of the processing circuit to produce a signal on the output Data line of The processing circuit:
each second transfer function means receives a signal from the first summing means on the distribution Net Data Summation line of the process-ing circuit to produce a signal on a distribution intermediate line which is con-nected to an input of the first multiplication means:
each second summation means receives one or more signals from the input Net Error lines of the processing circuit to produce a signal on tile dis-tribution Net Error Summation line:
each first multiplication means receives signals from said distribution intermediate line of the processing circuit and from the second summing means on the distribution Net Error Summation line of the processing circuit to produce a signal on the output Error line of the processing circuit:
each second multiplication means receives signals from the input Data line and input Error line of the input synaptic circuit to produce a signal on the distribution weight prime line of the input synaptic circuit:
each first integration means receives a signal on the weight prime line of the input synaptic circuit and produces a signal on the distribution weight line of the input synaptic circuit:
each third multiplication means receives signals from the input Data line and distribution weight line of the input synaptic circuit to produce a signal on the output Net Data line of the input synaptic circuit:
each fourth multiplication means receives signals from the input Data line and input Error line of the output synaptic circuit to produce a signal on the distribution weight prime line of the output synaptic circuit;
each second integration means receives a signal on the weight prime line of the output synaptic circuit and produces a signal on the distribution weight line of the output synaptic circuit:
each fifth multiplication means receives signals from the input Data line and distribution weight line of the output synaptic circuit to produce a signal on the output Net Data line of the output synaptic circuit:
each sixth multiplication means receives signals from the input Error line and distribution weight line of the output synaptic circuit to produce a signal on the output Net Error line of the output synaptic circuit.
comprising:
at least one input fan-out circuit: at least one input synaptic circuit: at least one hidden processing circuit: at least one output synaptic circuit: and at least one output processing circuit:
wherein:
each input fan-out circuit comprises an input Net Data line and an output Data line:
each hidden processing circuit comprises one or more input Net Data lines one or more input Net Error lines an output Data line an output Error lines a first transfer function a second transfer function a first multiplication means a first summation means and a second summation means:
each output processing circuit comprises one or more input Net Data lines. an input Net Error line.
an output Data line an output Error line a first transfer function a second transfer function a first multiplication means and a second summa-tion means:
each input synaptic circuit comprises an input Data line an output Net Data line. an input Error line a distribution weight line. a distribution weight prime line. a second multiplication means a first integration means and a third multiplication means:
each output synaptic circuit comprises an input Data line an output Net Data line. an input Error line. an output Net Error line a distribution weight line a distribution weight prime line. a fourth multiplication means a second integration means a fifth multiplication means and a sixth multiplica-tion means: and each input output and distribution line transmits a signal from a sin-gle input to one or more outputs:
said system further comprising:
an input vector comprised of one or more input data stimulus lines:
wherein: each said input data stimulus line is associated with one said input fan-out circuit: each said input data stimulus line connects said input Net Data line of its said associated input fan-out circuit:
an output vector comprised of one or more output response lines:
wherein: each said output response line is associated with one said output processing circuit: each said output response line connects said output Data line of its said associated output processing circuit:
an error vector comprised of one or more input error stimulus lines:
wherein: each said input error stimulus line is associated with one said output processing circuit: each said input error stimulus line connects said input Net Error line of its said associated output processing circuit:
wherein:
an input synaptic circuit couples between an input fan-out circuit and a hidden processing circuit:
an output synaptic circuit couples between a hidden processing circuit and an output processing circuit:
each output Data line of an input fan-out circuit connects to an input Data line of an input synaptic circuit:
each input Data line of an input synaptic circuit connects to the out-put Data line of an input fan-out circuit: each input Error line of an input synaptic circuit connects to the output Error line of a hidden processing cir-cuit:
each output Net Data line of an input synaptic circuit connects to one of the input Net Data lines of a hidden processing circuit:
each output Net Data line of an output synaptic circuit connects to one of the input Net Data lines of an output processing circuit: each output Net Error line of an output synaptic circuit connects to one of the input Net Error lines of a hidden processing circuit:
each input Data line of a hidden synaptic circuit connects to the out-put Data line of a hidden processing circuit: each input Error line of a hidden synaptic circuit connects to the output Error line of a hidden processing cir-cuit:
each summing means receives one or more signals from separate lines and produces a signal which is the mathematical summation of the received signals:
each multiplication means receives signals from a plurality of lines and produces a signal which is the mathematical multiplication of the received signals:
each integration means receives a signal from a line and produces a signal which is the mathematical integration over time of the received signal:
each transfer function means receives a signal from a line and produces some predetermined function of that received signal:
each input Net Data line of an input fan-out circuit is connected directly to the output Data line of that input fan-out circuit and passes the signal thereon directly without change:
each first summation means receives one or more signals from the input Net Data lines of the processing circuit to produce a signal on the dis-tribution Net Data Summation line:
each first transfer function means receives a signal from the first sum-ming means on the distribution Net Data Summation line of the processing circuit to produce a signal on the output Data line of The processing circuit:
each second transfer function means receives a signal from the first summing means on the distribution Net Data Summation line of the process-ing circuit to produce a signal on a distribution intermediate line which is con-nected to an input of the first multiplication means:
each second summation means receives one or more signals from the input Net Error lines of the processing circuit to produce a signal on tile dis-tribution Net Error Summation line:
each first multiplication means receives signals from said distribution intermediate line of the processing circuit and from the second summing means on the distribution Net Error Summation line of the processing circuit to produce a signal on the output Error line of the processing circuit:
each second multiplication means receives signals from the input Data line and input Error line of the input synaptic circuit to produce a signal on the distribution weight prime line of the input synaptic circuit:
each first integration means receives a signal on the weight prime line of the input synaptic circuit and produces a signal on the distribution weight line of the input synaptic circuit:
each third multiplication means receives signals from the input Data line and distribution weight line of the input synaptic circuit to produce a signal on the output Net Data line of the input synaptic circuit:
each fourth multiplication means receives signals from the input Data line and input Error line of the output synaptic circuit to produce a signal on the distribution weight prime line of the output synaptic circuit;
each second integration means receives a signal on the weight prime line of the output synaptic circuit and produces a signal on the distribution weight line of the output synaptic circuit:
each fifth multiplication means receives signals from the input Data line and distribution weight line of the output synaptic circuit to produce a signal on the output Net Data line of the output synaptic circuit:
each sixth multiplication means receives signals from the input Error line and distribution weight line of the output synaptic circuit to produce a signal on the output Net Error line of the output synaptic circuit.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US2499887A | 1987-03-12 | 1987-03-12 | |
| US24,998 | 1987-03-12 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1287175C true CA1287175C (en) | 1991-07-30 |
Family
ID=21823481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000561224A Expired - Lifetime CA1287175C (en) | 1987-03-12 | 1988-03-11 | Back propagation system |
Country Status (3)
| Country | Link |
|---|---|
| AU (1) | AU1496588A (en) |
| CA (1) | CA1287175C (en) |
| WO (1) | WO1988007234A1 (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2639736B1 (en) * | 1988-11-25 | 1991-03-15 | Labo Electronique Physique | GRADIENT BACKPROPAGATION METHOD AND NEURON NETWORK STRUCTURE |
| US5046020A (en) * | 1988-12-16 | 1991-09-03 | E. I. Du Pont De Nemours And Company | Distributed parallel processing network wherein the connection weights are generated using stiff differential equations |
| DE69013716T2 (en) * | 1989-02-23 | 1995-06-01 | Matsushita Electric Ind Co Ltd | Learning machine. |
| FR2648251B1 (en) * | 1989-06-09 | 1991-09-13 | Labo Electronique Physique | LEARNING METHOD AND NEURON NETWORK STRUCTURE |
| FR2648253B1 (en) * | 1989-06-09 | 1991-09-13 | Labo Electronique Physique | TREATMENT METHOD AND NEURON NETWORK STRUCTURE IMPLEMENTING THE METHOD |
| EP0416173B1 (en) * | 1989-08-23 | 1997-01-15 | Hitachi, Ltd. | A method for adjusting network parameters in a multi-layer perceptron device, and perceptron device provided with means for executing the method |
| JP3168204B2 (en) * | 1989-12-20 | 2001-05-21 | 富士通株式会社 | Network configuration data processing device |
| GB2258554A (en) * | 1991-08-07 | 1993-02-10 | Haneef Akhter Fatmi | Function-neuron net analyzer |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4518866A (en) * | 1982-09-28 | 1985-05-21 | Psychologics, Inc. | Method of and circuit for simulating neurons |
| US4593367A (en) * | 1984-01-16 | 1986-06-03 | Itt Corporation | Probabilistic learning element |
| US4599692A (en) * | 1984-01-16 | 1986-07-08 | Itt Corporation | Probabilistic learning element employing context drive searching |
-
1988
- 1988-03-11 WO PCT/US1988/000755 patent/WO1988007234A1/en unknown
- 1988-03-11 CA CA000561224A patent/CA1287175C/en not_active Expired - Lifetime
- 1988-03-11 AU AU14965/88A patent/AU1496588A/en not_active Abandoned
Also Published As
| Publication number | Publication date |
|---|---|
| WO1988007234A1 (en) | 1988-09-22 |
| AU1496588A (en) | 1988-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1312383C (en) | Spike transmission for neural networks | |
| Rumelhart et al. | A general framework for parallel distributed processing | |
| Charalambous | Conjugate gradient algorithm for efficient training of artificial neural networks | |
| US5056037A (en) | Analog hardware for learning neural networks | |
| CA1287175C (en) | Back propagation system | |
| WO1993000655A1 (en) | Sparse comparison neural network | |
| Farhat | Optoelectronic neural networks and learning machines | |
| Ellison | On the convergence of the multidimensional Albus perceptron | |
| JP3345478B2 (en) | Neural network device and learning method thereof | |
| US5170071A (en) | Stochastic artifical neuron with multilayer training capability | |
| Chen et al. | Optimization schemes for neural network training | |
| Christodoulou et al. | Design of gratings and frequency selective surfaces using fuzzy ARTMAP neural networks | |
| Sato et al. | Learning Nonlinear Dynamics by Recurrent Neural Networks (Some Problems on the Theory of Dynamical Systems in Applied Sciences) | |
| Ho et al. | Analogue circuit design and implementation of an adaptive resonance theory (ART) neural network architecture | |
| Duong | Cascade error projection: an efficient hardware learning algorithm | |
| Barua | Neural networks and their applications to computer data security | |
| Zhang et al. | Neuron-adaptive higher order neural network group models | |
| Wilke | Simulation of neural networks and genetic algorithms in a distributed computing environment using NeuroGraph | |
| Ferrari et al. | Algebraic training of a neural network | |
| Korogod | Synthesis of mathematical models of branching axons and dendrites | |
| JP3288066B2 (en) | Learning processing method and system for network configuration data processing device | |
| JP2517662B2 (en) | Chain controller | |
| Constant et al. | Modeling of electromechanical systems using neural networks | |
| Dayhoff | Pattern recognition with a pulsed neural network | |
| Tanomaru | Comparative study of two neural network approaches for nonlinear identification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKLA | Lapsed |