JPH0363870A - Learning method and neural network structure - Google Patents

Abstract

PURPOSE: To accelerate the learning of an input layer, and to decelerate the learning of an output layer by correction added to a neuron state by proportionating change to a component and a constant by steps for multiplying the component by a constant decided according to each layer, and decreasing the constant according to the number of the layers from an input layer to an output layer. CONSTITUTION: In a learning method including each layer of the update of a synapse coefficient based on change ▵Xj ,L, a gradient component gj ,L is multiplied by a parameter θj ,L for deciding the next change of a neuron state. Thus, change ▵Xj ,L proportional to -θj ,L gj ,L' is calculated. Here, θj ,L is decided according to the state of a neuron (j) of a layer 1, and when 0<=θ1 <+> <=1, and -gj ,L has different codes, θj ,L=1, and when -gj ,L and Xj ,L have the same codes, θj ,L=θL<+> . Thus, the learning of the input layer can be accelerated, and the learning of the output layer can be decelerated by the correction added to the neuron state.

Description

DETAILED DESCRIPTION OF THE INVENTION The present invention relates to a learning method in a neural network that performs a learning phase in which synaptic coefficients are determined based on actual examples using an error gradient back propagation algorithm. The invention further relates to a neural network structure and a computer programmed to carry out the method described above.

Yijing narrow distortion neural network is image processing,
Used for speech processing, etc.

Neural networks are automatic devices interconnected by synapses with corresponding synaptic coefficients. These can lead to solutions to problems that are difficult to solve with traditional sequential computers.

In order to perform a given process operation, a neural network must previously learn how to perform such an operation. This so-called learning phase uses actual examples (examples), but in learning, the desired output result for certain input data is known in advance.

In the first cycle, the neural network is not yet suitable for performing the desired task and produces inaccurate results.

Error E between the result obtained and the result that should be obtained
Determine p and vary the synaptic coefficients based on the principle of adaptation, allowing the neural network to learn certain selected examples. Repeat this step for as many examples as you think are necessary for the neural network to learn satisfactorily.

A widely used method for performing such adaptation is gradient back propagation. The gradient gj, L component of the previous error E9 (calculated in the final phase) is expressed as follows for each neuron state x
,,determine for L.

These components are then back-propagated into the neural network. This starts from the output to determine all internal components g=, t (f≠L), and then performs the correction to be made to the synaptic coefficient 'ij+L of the relevant neuron. This method is described, for example, in the following document:

D, Ej Rumelh
art) Day Ehinton (D, EjHinton)
) and R.J. Williams (R.J., Wil
liams) “Learning Internal”
Representation by ErrorP
ropagation D, Ej Rumelh
art) J, L, Mc
Clelland) “Parallel Distr.
ibutedProcess: Explo
ration in the Microstructure
re of Cognition” VoI!,,I
Foundations, MIT Press (1
986) D.J. Burr
)”Experiments on neural n.
recognition of spoken a
nd written text”, TEEE
Trans, onAcoustic, 5peech
and signal processing + V
ol.

36, No. 7. July 1988, page 1162.

However, when these methods are performed within a neural network, the learning period becomes extremely long for certain applications. Such difficulties arise, for example, with parity problems.

For example, the parity problem is such that the input is linked to the binary signal I1 term, and the output outputs state l when the number of inputs "'1" is odd, and outputs state O when it is even. This occurs in a neural network that is connected to a computer. The learning problem in this case is that the output must change state if only one of the inputs changes state, and the output must not change if an even number of input states change. to cause.

Furthermore, when using a neural network for a classification problem, for example, there is a problem in that it is difficult to separate classes whose minimum distance is small. This is because neural networks require a long training time to discriminate between these classes. This defect makes it difficult to separate input data that are encoded sequentially, especially when some of the examples belong to different classes and the inputs differ only slightly from each other.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to reduce the training time of neural networks while minimizing the additional hardware required.

The present invention is a learning method performed by a neural network consisting of L layers, in which each of the following steps is performed: ・Nl neurons are supplied with synaptic coefficients W(j+L′ from neurons in the previous layer) A step of determining the state Xj, L of the neuron in layer l as follows: Xj+L"'ΣiJ+f' Yt++-+・In the step of determining the potential Yj1 of the output neuron using the nonlinear function F as follows, Yi,, i = F
(X, t) Here, l: layer index j when 1≦i≦L: output layer l
, the index of the neuron in the input layer is the index of the neuron in the input layer -1. phase, and these learning phases.

6. Setting of the matrix Wij+1 of synaptic coefficients of the neural network. - Input data yj of each example p to be learned. 10. Comparison of the results Yj, L obtained in each layer by the output Yj provided to the input for this example p provided in order to define the partial error Ej, 10 for each output neuron and each Conditions for the output layer observed for example p: 1. Xi
, t, each gradient fraction of error E with respect to g44. −θE /'θXj+( Determination of ・The back propagation method of the gradient component glt is performed, thereby determining the gradient component g1L for other layers based on the replaced synaptic coefficient 71 lix by neural net work, For application to neural networks, the next change ΔXj, which has a sign opposite to that of the corresponding coefficient g11.
Determination of L, ・Based on the change ΔXj+L, the synaptic coefficient A・7
In the learning method that includes each phase of the update, in order to determine the next change in the neuron state, the gradient component gj,L is multiplied by the parameter θ1L, thereby making it proportional to −θ1°gl,′ Change ΔXj
+L, where θ,, is determined depending on the state of neuron j in layer l, and 0≦θ1゛≦1, g19. When and Xj+L have different signs, θj
+ 1 = 1, and when gj+ t and XjrL have the same sign, θ
It is characterized in that j+L=θL+.

During the learning process, a given example p is presented.

The data attached to example p and introduced through the input of the neural network produce a result Yj+L for a given output neuron j in the final layer of the network. At this time, the result YJ to be achieved at the starting point is known. Therefore, the error can be calculated as follows, for example.

This formula is for the root mean square error. Other comparison criteria can also be used.

According to the known gradient pack propagation method, the components g, t of the error gradient are:
Determined for each contribution of the neuron's state XjrL.

Therefore, gj, t = a E'7θx,, L where Xj,
L represents the state of the neuron before applying the nonlinear function.

Therefore, the component gj,L=θE'/θXj,L is related to the output layer, and therefore gj,t,
= (yj, t, -Yj) ・P'j+L.

Here F'j+L is the derivative of the nonlinear output function.

At this time, the neural network is loaded with the replaced synaptic coefficient matrix 'ji+L, and the components g,
L is pack-propagated from the output of the network. The network thereby determines the other components of the gradient for l≠L. These components g=+
t is used for determining the change ΔXj+L, which is used to modify the synaptic coefficient Wij+1 and adapt the network to the relevant example.

Generally speaking, this known modification method is performed as follows.

W = j, t (new) = Wi 1t (old)
+ k .DELTA. This neuron state Xj+ is multiplied in advance by an associated parameter θj+L determined based on the signs of the components of the gradient gJ,L.

These parameters are θ1° when glt and XjrL have different signs, and θj+L””θ1 when −g, , and XjrL have the same sign and 0≦θ1゛≦].

However, to facilitate the learning process, during the first learning iteration, θL+ can be chosen to be close to or equal to zero for each given example.

Furthermore, in a later learning iterative course, θ.

increases towards 1 for each given example.

The application of codes in the present invention is such that at the beginning of learning, corrections are made taking into account the signs of observed errors, and as learning progresses, corrections are gradually made with higher accuracy and with a smaller degree of coarseness. It is possible to make it happen at any time.

The non-linear function determining the output potential can be selected to be slightly non-linear or strongly non-linear. In order to increase the efficiency of the adoption of codes according to the invention, the selection of non-linear functions can be changed during the learning process. death.

However, the change Δx47 obtained by the gradient pack propagation method. does not cause excessive changes in synaptic coefficients. Therefore, in the interpolated version according to the invention, a standardization is performed, so that the sum of the squares of the synaptic coefficients remains quasi-definite.

For this reason, the nonlinear function at the start of learning is selected to be slightly nonlinear, and at the end of learning, it approaches the coded form. To enable this selection, the synaptic coefficients are made to maintain a standard, metastable value of Σ (Yi7.t)2 that converges towards a given neuron j.

The nonlinear function F is expressed as Yjt = tanh (XJ, t/
Tt), where T+ is a layer-related parameter called the temperature of layer l.

Regarding the degree of nonlinearity of the nonlinear function, changes applied during learning were obtained from changes in the parameter TL for each layer.

The principle of application of the code according to the present invention is to first perform coarse correction (θ゛ is small and positive) using the error code, and then perform fine correction using a parameter β゛ close to l in order to perform high-precision correction. , produce similar effects at the level of the overall structure. This is multiplied by meter ηj+L. As a result, the modification (parameter ηj+L) applied simultaneously to all neuron states can be applied to each state (parameter θ
+) is superimposed on each individual modification made to

Due to the effect of the previously submitted sign, it is possible to introduce a modification factor ηj+L depending on each output neuron j of the final layer. The error Hp in this case is determined by the following equation.

! g = 'A (YJY4.L)2.

This error is a square function.

Generally speaking, for each output neuron j of layer L (for a given example p), this error EJ is: EJ=H(
Y7 Ylt ), where H is a function of the difference between the obtained result Yj+L and the desired result YJ.

The error Ep thus obtained is used to determine the components gj, L and gj+L (where l≠L) of the gradient formed as described above.

In order to determine the components of the gradient gj+L, the method of the invention comprises the step of determining the error E2 beforehand by adding a correction factor ηj+L determined by the neuron j of the final layer, so that here j=1 , steps are provided to facilitate the start of learning. However, when E:, yj, and L have different signs, η3. t=1
, Yj and Yj, when L have the same sign, ηj+L"η"
, where 0≦η3≦l.

In the current case, η,1=θj+L.

is allowed.

According to a supplementary bar of the present invention, the 9 strategies can be developed at the level of each layer of the neural network. Learning is accelerated for the input layer and decelerated for the output layer, taking into account the dominant role entrusted to the input layer.

In the general application of the gradient bank propagation method, the state of the neuron Xj,L is determined by the gradient -ΔXj,
Modified by L. It has the same proportionality coefficient (propo) for all layers of the neuron network.
gradient g41. This occurs by multiplying each component of .

According to a complementary version, the invention also performs the correction by assigning a proportionality factor βj+L to each neuron in each layer, and each correction −ΔXj,L is reduced to β5. .

It is proposed to make it proportional to gll.

The parameter βj+L is a correction value Δχ41. related to the coding strategy described above by setting it to be proportional to the parameter θj+L that helps determine .

In this way, β41. is proportional to β,·θj+L. Here, β is the same for any given layer l. According to this supplementary version, a parameter β, which enables control over the learning speed in the input layer and the learning speed in the output layer, is assigned to each layer l. Therefore, the parameter β, decreases as l increases from the input layer to the output layer.

In this way, the method of the present invention uses the constant β determined depending on each layer.
, the component θ12. - A step of multiplying gj+t, and by this step -ΔX59. β1
・θj+L' gj, proportional to t, where β
, is 10 for the input layer, and decreases strictly with the number of layers toward each layer, so that modifications made to the neuron state accelerate the learning of the input layer and slow down the learning of the output layer. ensure that

〔Example〕

The present invention will be explained below with reference to the drawings.

Figure 1 shows input signals Yl+ L-1+ YZ+, respectively.
L-1+y, (L-11, L-1 if its state is Xj+
1 shows the general operation diagram performed by a single neuron network formed by an input layer of multiple neurons 10+101 (1-11) feeding a single output neuron of 1. In this case, the above situation is It is determined as follows.

Xj・1=ΣWi+j・,′Y・・L−1 This state χ4
1. is influenced by a nonlinear function (block 12), and after applying this function F, the output potential Y4. ．． give.

Yj, L= F (X1t) Therefore, this output potential Y41. can serve as input state for subsequent layers.

Thus, input layer i=1, hidden layer (hidden 1a
A plurality of layers as shown in FIG. 2 are obtained, including yer) 1=2.3 and output layer 1=L. Neurons in a layer are exclusively linked to neurons in subsequent layers via synaptic coefficients 'ij+L.

The state of each neuron is determined by the above equation starting from layer 2=1.

In order to carry out the learning process, i.e. to adapt the synaptic coefficients for a given task, the result yj on the output layer is known in advance (Example).
is provided as input, and for each instance, after calculating the error Ep for all output states, the infinitesimal variable θ of each intermediate state
Determine the variables for Xj,L. In this case, the gradient components gj, L are given by the following equation.

gjl, = θEp/θxj1 Thus, the component g4.g in the output layer. After L is calculated, it is inverted Tti (pack propagated) to the neuron network, where the other components gj+t of the error gradient are recovered. These components adapt the neural network to the task it faces, so the state χ
11. This makes it possible to determine the variables Δχj+L that can be deduced from it for use. This operation, as mentioned above,
This is performed prior to updating the synaptic coefficient W==, t.

These steps of the method of the invention may be carried out in a dedicated neuron network structure, as shown in FIG. 3, or in a computer programmed to carry out the method.

The memory 30 stores, for example, the synaptic coefficient matrix -15 initially supplied by the input means 29 and the matrix -5i1 . remember.

The synaptic coefficient is the input potential Yi+L from the previous layer.
-1, and these means 31 determine Xj+L=Σ wtj, t'Yi+L-1.

The input of the network is the input neuron state Yi+L
-1 as a base, example Y.

supply. These examples are provided by example memory 32.

The selector 33 has the skill to make this selection possible.

The example memory 32 also stores the results Yj to be obtained for each example p and each output potential j.

The state of the output neuron, X,, t, is made to follow a non-linear function in member 34. Said member 34 provides the last layer output potential Yj,L as provided by the system for each example. Layer l
The output potential Yj+L of is temporarily stored in a state memory 37 for performing intermediate calculation steps from one layer to another, so that it can be used as an input state for the next layer. Each potential Yj+L is compared with the intended state Yj in a comparator 35. Said comparator 35 furthermore detects all detected errors EJ
are stored and these errors are added to give the error Ep for each exembre.

The components of the gradient gj+L are determined by the host computer 36. The computer therefore receives the error E11 output potential Yj+L and the intended state y. host computer 3
6, the component gj+t is determined so that the following equation holds true.

g;+t”θj+L ・(Yj, t −Yj) ・
F'j+where 1≦j≦I(L), and F'j+L is a derivative of each nonlinear function of the output layer.

These components gj, L are fed to calculation means 31 which make it possible to carry out a gradient backpropagation method (Cladiend pack propagation method). That is, the components gj+L are fed to the output layer so that their effects are back-propagated to the input layer.

Therefore, the gradient gj, , = θE9/θ.

(However, l-#L) uses the calculation means 31, and the error E
' is determined by backpropagation of the gradient.

This component gj,t is used by the host computer 36 to determine the next variable ΔXj,L for each neuron state.
supply. For this purpose, in the case of the present invention:
The computer 36 converts each component gj+t into its parameter θ
4. ．． so that it is multiplied by

All variables ΔXj+L update these member 38
supply to. Said update member 38 has the function of determining new synaptic coefficients Wij+1 and supplying these coefficients to the memory 3 term.

This process is made iterative to perform all learning phases. In the process, the host computer 3
6 may supply for the first iteration a correction parameter θ equal to or approximately equal to the term, after which the computer 36 causes this parameter to approach the value 1 in the course of subsequent iterations. can be increased. Furthermore, in order to perform gradient backpropagation in the calculation means 31, the host computer 36 calculates Ej before calculating the components gj, t.
Multiplication is performed using the parameter η1L.

β,・θj+L' Variable -Δ proportional to gj+L
×1. To determine L, the constant β for each layer is modified by a modified value θ18. gj+1, the host computer updates the synaptic coefficient '1i by update member 38.
Proceed before updating rt.

Thus, the hierarchical neural network structure according to the invention comprises means for carrying out the learning method described above;
For that purpose the structure is one synaptic coefficient (continuous coefficient)
- means for storing the examples to be learned and introduced into the neural network; and - for each example, comparing the resulting neuron potential at the output with the result faced for each example. and means for providing an error corresponding to the observed difference; computing an output neuron state based on the one-person neuron potential, and performing gradient backpropagation of the error to obtain a component g1 of the gradient. ．． - means for providing a non-linear function at the output; - calculating new synaptic coefficients taking into account the components of the gradient gj,L and the multiplier parameters for the method, and calculating the new synaptic coefficients of the iterative cycle; means for enabling new synaptic coefficients by assigning a significance to a given iteration or to a given neuron or a given layer of the neural network.

The system shown in the third diagram is provided in the form of a neural network structure consisting of functional blocks controlled by a host computer.
Preferably, the functionality to be implemented is integrated within the computer itself. In that case, the invention also relates to a programmed computer implementing the steps of the method described above.

Table 1 shows a flowchart containing the main steps of an example program according to the invention.

-2j2ot1''- initializes η゛ and θ° to small positive values and fixes the temperature TL. For layer (rare) j2=l, the value TL is equal to the average of the absolute values of the inputs with respect to example p; for layer l≠1, the value TL is of the order of 1 (loop to 1 ).

The synaptic coefficient -ij+L is initialized by random selection or set to a known value (loop to i an
d j ).

- Step 2 is the input value Yi for example p.

Insert into the neural network.

- Topsoil 1 work is in condition X41. and output potential Y
Calculate j+L. The calculation of state Xj+1 is done by Threshorubi S11. including. The threshold is a nonlinear function F
It can also be guided inward.

-2 Otsu j 2 and 1” U uses a sign strategy (sign
strategy). For this purpose the product (
product) Yj-Yj, L and consider its sign. If the product is negative or O, ηj+L takes the value 1; in the opposite case, ηj+L takes the value η゛.

Determine the error E' in the output layer and calculate the gradient g
Calculate the components of j and L.

- Step 5 Calculate the derivative F'j+L of the nonlinear function. Next, the components of gradient gj, L-1 are calculated by backpropagation of the gradient. Product
glt・X49. Check. If this product is negative or 0, θ,L equals 1; if this product is positive, θ51. is equal to θ° (0≦θ1≦1). Next, β1. Calculate.

- Step 6 Use the components of the gradient gj,L to determine the next variable ΔXj+L. This step creates an example auto-ad function that allows the component gj,L to have an influence on the variable ΔX, .
aptiue function example)
give you a choice.

This function is the gradient gj, the modulus G2 of L,
Factors γ, ξ that control the amplitude of the modification and the rings β5 associated with the various neurons. ．． includes the average value T.

−Step 7 This step calculates the synapse coefficient of the variable ΔXj+L calculated for example P −□j+L
and thresholds Sj+L. The distribution coefficient is the norm ΣY□.

is controlled by the parameter σ, which applies This step 7 represents an example of a distribution that allows the criterion of synaptic coefficients to be held at a quasi-constant value for a given output neuron. Changes should be realized with as small a variation of thresholds and weights as possible.

- Step 8 If the shadow table 1 for all examples is small, end the learning. If this error is greater than ε, the procedure continues with the next step.

Step 9: Slightly lower the temperature TL. Therefore, the initial value is the parameter e between O and 1.

is multiplied by

Step 10: Readjust the values of η and θ.

-Yu±F”ll Select another example p′,
The operation is restarted in step 2.

[Brief explanation of drawings]

FIG. 1 shows the mechanism of processing carried out by a structure containing a layer of input neurons and a single output neuron; FIG. 2 shows a structure containing several layers: an input layer, a hidden layer and an output layer. , FIG. 3 is a diagram showing a neural network structure for implementing the method of the invention. 10+, 10g---101(L-1)"'knee L-
Ron (nerve cells) 11, 31...Calculation means 12...Non-linear function 29...Input means 30...Memory 32...Example memory (example memory) 33.
... Selector 34 ... Nonlinear member 35 ... Comparator 36 ... Host computer 37 ... State memory 38 ... Update member

Claims (1)

[Claims] 1. A learning method performed by a neural network consisting of L layers, which includes the following steps: - Setting synaptic coefficients W_i_j_,_L to neurons in layer l;
The state of the neuron in layer l is determined based on the output potential Y_i_,_L_-_1 supplied from the neuron in the previous layer connected by ', or based on the input data Y_i_,_0 for layer l=1. X_j_,_L, X_j_,_L=Σij,l・Y_i_,_L_-_1
A step of determining the potentials Y_i_, _L of the output neuron using the nonlinear function F as follows, Y_i_, _
L=F(X_j_,_L) where l: Index of the layer when 1≦l≦L j: Index of the neuron in the output layer l i: Index of the neuron in the input layer l-1. A method, the method comprising a learning phase by repetition of examples p that are successively supplied to the input of the neural network, and these learning phases include: - setting of the matrix of synaptic coefficients W_i_j_,_L of the neural network; , the introduction of input data Y_j_,_0 for each example p to be learned; the result obtained in the output layer L by the direct output Y_j for this example p provided at the input to define the partial error E_j; Comparison of Y_j_,_L, - all partial errors E_j observed for each output neuron and each example p
Determination of the sum E of,・state X_j_,_L for output layer L
Determination of each gradient component g_j_,_L=∂E/∂X_j_,_L of the error E for Determination of the gradient components g_j_,_L for other layers, and the corresponding coefficients g_j_,_L for application to the neural network.
Determination of the next change ΔX_j_,_L with a sign opposite to that of , Determine the next change of the neuron state based on the change ΔX_j_,_L in a learning method that includes each phase of updating of the synaptic coefficients. Therefore, there is a step of multiplying the gradient component g_j_,_L by the parameter θ_j_,_L, thereby -θ_j_,_L
・Calculating the change ΔX_j_,_L proportional to g_j_,_L′, where θ_j_,_L is determined according to the state of neuron j in layer l, and 0≦θ_1^+≦1, −g_j_,_L When and How to learn. 2. Make the first learning iteration θ_L^+ approximately equal to zero, or
Claim 1: Each predetermined value is approximately zero.
Method described. 3. The method of claim 2, wherein in subsequent learning iterations θ_L^+ increases toward 1 for each given example. 4. Select the non-linear function to be slightly non-linear at the beginning of the training, and then make it approach the signed function at the end of the training, and to allow such selection, converge towards a given neuron j. The synaptic coefficient is the standard Σ(W_
4. The method according to claim 1, wherein the i_j_,_L)^2 quasi-constant is maintained. 5. The nonlinear function F is Y_j_,_L=tanh(X_
5. The method according to claim 4, wherein T_L is a layer-related parameter referred to as the temperature of layer l. 6. The method according to claim 5, wherein changes in the degree of nonlinearity of the nonlinear function applied during learning are obtained from changes in the parameter T_L for each layer. 7. To determine the components of the gradient g_j_,_L, the method adds in advance a correction factor η_j_,_L determined by the neuron j of the final layer,
determining the error E^p, thereby E^p≒Σ^I^(^L^)_j_=_1η_j_,_
A step that favors the start of learning as L・E^p_j, provided that η_j when E_j and Y_j_,_L have different signs.
_,_L=1, and when Y_j and Y_j_,_L have the same sign, η_j_,_L=η^+, where 0≦η^+≦1, according to any one of claims 1 to 6. the method of. 8. The method according to claim 7, wherein η_j_,_L=θ_j_,_L. 9. The method according to any one of claims 1 to 8, wherein the partial error E_j is the squared error 1/2 (Y_j - Y_j_,_L)^2. 10. The component θ is determined by the constant β_L determined according to each layer.
It has a step of multiplying _j_,_L・g_j_,_L, and by this step -ΔX_j_,_L becomes β_L
・Proportional to θ_j_, _L ・g_j_, _L, where β_L decreases strictly according to the number of layers from the input layer to the output layer, so that the modification applied to the neuron state is proportional to that of the input layer. Any one of claims 1 to 9, ensuring acceleration of learning and deceleration of learning of the output layer.
The method described in section. 11. A neural network comprising means for performing the learning method according to claims 1 to 10, comprising: - means for accumulating synaptic coefficients; - means for accumulating examples to be learned and introducing these examples into the neural network; means for comparing, for each example, the neuron potential obtained at the output of the neuron with the direct result for each example and providing an error corresponding to the observed difference; - output based on the input neuron potential; Compute the neuron state, perform gradient back propagation of the error, and calculate the component g_j of the gradient.
_, _L and the multiplication parameters assigned to the method to calculate new synaptic coefficients, thereby controlling the significance contributing to a given iteration of the iterative cycle, or to a given layer or neural network. A neural network structure characterized in that it has means for controlling the significance assigned to a predetermined neuron. 12. A computer programmed to carry out the learning method according to any one of claims 1 to 10.