JPH0696046A - Learning processor of neural network - Google Patents

Abstract

PURPOSE:To provide the learning processor of the neural network which increases the efficiency of the whole learning and makes the learning fast and precise. CONSTITUTION:The structure of the neural network is changed between pattern conversion and the learning. At the time of the pattern conversion, a sigmoid function is applied to for units of an output layer as before. At the time of the learning, on the other hand, nonlinear conversion is not applied to the units of the output layer 6, an inverse sigmoid function 8 is applied to a tutor signal 9, and the difference is an error signal 11. A load calculating circuit 13 learns a synapsis load 5 by least square algorithm and a load calculating circuit 14 learns a synapsis load 3 by an error inverse propagating method.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a high-speed learning processing device for a hierarchical neural network used for memory, inference, judgment, prediction, pattern recognition, control, optimization and the like.

[0002]

2. Description of the Related Art A neural network is a signal processing system in which a plurality of input-output artificial neural elements (units) simulating the functions of biological neural elements are connected in layers.
It is a general term for networks that realize information processing.

FIG. 2 shows an example of the structure of a three-layer hierarchical neural network, which comprises an input layer 2 having i units, an intermediate layer 4 having j units, and an output layer 7 having k units. . Here, the intermediate layer 4 is a single layer, but there may be a plurality of layers. In FIG. 2, signal transmission is as follows.

When x _i is the input signal of the network 1, w _ji is the synapse weight between the input layer and the intermediate layer, and θ _j is the offset amount, the internal state signal u _j of each unit of the intermediate layer 4 is represented.
Is expressed by the following equation.

[0005]

[Equation 1]

In order to simplify the description, the above equation is newly replaced with the following equation.

[0007]

[Equation 2]

From this, the output h of the j unit of the intermediate layer 4
_j is expressed by the following equation.

[0009]

## EQU00003 ## h _j = f (u _j ) (Equation 3) Here, for f (·), for example, a sigmoid function of the following equation is generally used.

[0010]

[Equation 4]

Similarly, when v _kj is the synapse load 5 between the intermediate layer and the output layer and φ _k is the offset amount, the internal state signal s _k of each unit of the output layer 7 is expressed by the following equation.

[0012]

[Equation 5]

In order to simplify the description, the above equation is newly replaced with the following equation.

[0014]

[Equation 6]

From this, the output y of the k unit of the output layer 7
_k is expressed by the following equation.

[0016]

Y _k = f (s _k ) (Equation 7) As described above, in the hierarchical neural network, each unit processes the input data 1 given to the input layer 2 and transfers it to the next layer. The output data 7 corresponding to the input data is obtained from the output layer 7.

The error back-propagation method has been widely used as a learning method for synapse weights of a hierarchical neural network, as previously described by Ramelhardt, Hilton and Williams; “Learning Internal Representation By Error Bass Propagation” (R
umelhart, Hinton, and Williams: “Learning InternalRe
presenations by Error Back Propagation ”, In Paral
lel Distributed Processing, Vol.1, pp318-36
2, MIT Press (1986)).

FIG. 3 is an example of a configuration in which the error back propagation method is applied to the hierarchical neural network of FIG. The error back propagation method will be described below with reference to FIG.

When the input data 1 of the pattern P is input to the input layer 2, the output data which the unit k of the output layer 7 wants to output is the teacher signal y _mk 9. At this time, the error 12 between the teacher signal 9 and the actual output data 10

[0020]

## EQU00008 _## Defining e _k = y _mk -y _k (Equation 8), the square error evaluation function E _P for a certain pattern _P is expressed by the following equation.

[0021]

[Equation 9]

First, the load calculation circuit 15 is designed. This is as follows based on the steepest descent method for the amount of change in the synaptic load v _kj .

[0023]

[Equation 10]

Next, the load calculation circuit 16 is designed. This is as follows from the steepest descent method for the amount of change in the synaptic weight w _kj .

[0025]

[Equation 11]

Similarly, when the number of layers is four or more, the synapse load between all the layers can be determined in the same manner by repeatedly converting the error into the error in the previous layer.

In addition, a learning method is known in which the previous correction amount is taken into consideration in order to achieve the speedup of the error backpropagation method of Expressions 9 and 10. The correction amount of the previous (n-1) step is Δv
(N−1), Δw (n−1), and the correction amounts of the (n) th step this time are Δv (n) and Δw (n), the following equation is obtained.

[0028]

[Equation 12]

[0029]

[Equation 13]

This is because by adding the correction amount of the previous time, a kind of inertia is generated in the change of the synapse load, and the effect of ignoring the fine irregularities of the error curved surface can be obtained.

By the way, the above learning is to minimize the error E _P with respect to the input / output group of a certain pattern P, and is sequentially called correction learning. On the other hand, in order to minimize the following error amount E _T with respect to the input / output pairs of all patterns, it is necessary to sequentially add the synaptic loads obtained by the correction learning and perform correction with the added synaptic loads for all patterns. is there. This is called batch modification learning.

[0032]

[Equation 14]

Further, conventionally, a method for increasing the learning speed of the above-mentioned error back-propagation method is described in Japanese Patent Laid-Open No. 3-252887. There, the internal signal of the output (s _k : Equation 6)
And a teacher signal (y _mk ) and a teacher internal signal subjected to inverse sigmoid transformation are used to perform learning by the above error propagation method.

[0034]

However, the above-mentioned conventional back-propagation method and the method disclosed in Japanese Patent Application Laid-Open No. 3-252887 disclose the synapse load w _ji between the input layer and the intermediate layer,
Since the _weights of both the synapse weights v _kj between the intermediate layer and the output layer are determined based on the steepest descent method, the sum of the squared errors E _P described above is made sufficiently small and the learning required until the learning is finished. There is a problem that the number of iterations becomes a huge value and efficient learning processing cannot be performed.

More specifically, the learning procedure of the conventional error back-propagation method is as follows, when updating the synapse weight w _ji between the input layer and the intermediate layer, as shown in _Equation 11. perform learning as synaptic weights v _kj of indicates the correct value, likewise, synapse load v when updating _kj, output h of the intermediate layer as shown by the number 10
The information of _j is needed, and the learning is performed assuming that the synaptic weight w _ji is a correct value. That is, although the conventional error back-propagation method learns the update of the synapse weights w _ji and v _kj independently of each other, both synapse weights of the error curved surface, which are generally said to have slow convergence, are calculated. Since the learning is performed by the steepest descent method that is determined based on the gradient of, there is a problem that the learning time becomes a huge value.

In view of the conventional problems, an object of the present invention is to improve the learning efficiency of the whole learning by improving the learning speed of the synapse weight between the intermediate layer and the output layer of the hierarchical neural network with high accuracy. It is to provide a learning processing device for a neural network that achieves high speed and high accuracy.

[0037]

In order to achieve the above-mentioned object, the present invention comprises a plurality of units which have a sigmoid-like nonlinear function inside and which perform signal processing corresponding to an artificial neural element, A signal processing unit including an input layer, an intermediate layer, and an output layer, and a coupling between the units based on an error signal between an output value of the output layer and a teacher signal with respect to an input signal pattern input to the input layer. In a learning processing device of a neural network, comprising: a learning processing unit that sequentially and repeatedly calculates a coefficient of strength from the output layer side toward the input layer side, the strength of the coupling between the intermediate layer and the output layer. The second learning processing unit different from the first learning processing unit that learns the coefficient and the other first learning processing unit that learns the coefficient of the coupling strength is provided.

Further, in the present invention, the first learning processing unit and the second learning processing unit use the value obtained by passing the teacher signal through an inverse function of the sigmoid-like nonlinear function to obtain the error signal. A learning processing unit for determining and calculating the coefficient of the coupling strength is provided.

Further, in the present invention, the first learning processing section uses the least squares method for calculating the coefficient of the coupling strength toward the minimum value of the error curved surface obtained from the error signal, The second learning processing unit is provided with a learning processing unit that calculates the coefficient of the coupling strength in the steepest descent direction of the error curved surface.

[0040]

According to the learning method of the present invention, learning of the synapse weight between the intermediate layer and the output layer is performed at high speed and with high accuracy, thereby achieving high speed and high accuracy of learning of the hierarchical neural network. That is, the synapse weight v _kj is learned at high speed and high accuracy based on the least squares method algorithm, and other synapse weights are learned by the conventional error backpropagation method, so that the learning of the synapse weight between the intermediate layer and the output layer is minimized. It can be converged to the minimum value without falling.

[0041]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the drawings.

FIG. 1 shows a three-layer neural network structure when learning is performed by the learning method of the present invention. The conventional error backpropagation method uses pattern conversion (Fig. 2) and learning (Fig. 3).
There is no change in the structure of the neural network. Here, the time of pattern conversion refers to a period during which learning is completed, a synapse weight is fixed, a pattern entering the input layer is converted, and a neural network solution is output from the output layer. “Learning time” refers to a time when learning the synapse weight according to a certain evaluation function.

On the other hand, the learning method of the present invention changes the structure of the neural network at the time of pattern conversion (FIG. 2) and at the time of learning (FIG. 1). At the time of pattern conversion, the sigmoid function of the equation 4 is applied to the unit of the output layer 7 similarly to the error back propagation method, but at the time of learning, the teacher signal 9 is given the inverse sigmoid of the following equation without performing the nonlinear conversion on the unit of the output layer 6. Function 8 is applied and converted.

[0044]

[Equation 15]

From this, the output s _kP of the k unit of the output layer 6 at the time of learning is given by the following equation.

[0046]

[Equation 16]

However,

[0048]

[Equation 17]

[0049]

[Equation 18]

It is Here, the subscript P is the pattern P
Is a signal to. Further, the inverse sigmoid transformation of the teacher signal y _mkP 9 for the pattern P is _defined as f ~ ¹ (y _mkP ) 8.

Here, the error signal e _kP for the pattern P
11

[0052]

Equation 19] e _kP = f ~ ¹ and _(y mkP) -s _kP ... (Equation 19), the evaluation function of the square error of each output layer to the total pattern is defined as the following equation.

[0053]

[Equation 20]

First, the load calculation circuit 13 is designed.
Now consider minimizing the error E _kA with respect to the synaptic weight V _k (equation 17). Then, the influence on the error E _kA with respect to the minute change of the synapse load V _k can be decomposed as follows, and since the minimum point exists, the following equation is set to zero.

[0055]

[Equation 21]

From this, if the first term on the right-hand side of the above equation is regular, Vk is set as _VkP.

[0057]

[Equation 22]

V _kP may be set as Rewriting this into a sequential formula gives the following formula.

[0059]

[Equation 23]

[0060]

[Equation 24]

However,

[0062]

[Equation 25]

[0063]

[Equation 26]

For example, if λ _1P = 1 and λ _2P = 1 then the above equation is a recursive least squares algorithm.

Also,

[0066]

[Equation 27]

[0067]

[Equation 28]

By setting, the trace of Γ _P can be made constant.

Next, the load calculation circuit 14 is designed. As a learning method of the synapse weight w _ji 3 between the input layer 2 and the intermediate layer 4, the error signal 11 (Equation 19) is learned by the conventional error back propagation method. First, the squared error for the pattern P is defined.

[0070]

[Equation 29]

The amount of change in the synaptic weight w _kj is determined by the steepest descent method as follows.

[0072]

[Equation 30]

It is also possible to increase the speed by considering the previous correction amount as shown in the above equation.

FIG. 4 is a diagram showing an execution procedure of an embodiment of the present invention. First, the learning of the synapse load (step 401) is repeated with the configuration of FIG. 1 until the absolute value error of the following equation becomes equal to or less than a set value E _R (step 402).

[0075]

[Equation 31]

Next, the synapse load is fixed and pattern conversion is carried out with the configuration of FIG. 2 (step 403). In this case, the learning shown in FIG. 1 can be executed by the read only memory (ROM) and the random access memory (RAM), and the pattern conversion shown in FIG. 2 can be executed by the ROM.

FIG. 5 is a diagram showing an execution procedure of an embodiment of the present invention. First, as described with reference to FIG. 5, learning of synapse weights (step 501) in the configuration of FIG. 1 is repeated until the absolute value error of (Equation 31) becomes equal to or less than a set value E _R (step 502). .

Next, pattern conversion is performed with the configuration of FIG. 2 (step 503). At this time, the absolute value of the equation (Equation 31) is monitored for each pattern conversion (step 504).
If the value is less than or equal to a set value E _{S, the} synaptic weight is fixed as it is and the pattern conversion is repeated (step 50).
6) If the value is above a certain set value, the structure of the network is changed to the configuration of FIG. 1 to learn the synaptic weight.
(Step 5050). In this case, the read only memory (ROM) and the random access memory (RAM) are used to implement the network.

In order to confirm the effectiveness of the learning processing device of the present invention, the learning result of exclusive OR (XOR) will be shown below.
Various pattern recognitions can be considered as applications of this problem.

Table 1 below shows the input / output relationship of the exclusive OR (XOR).

[0081]

[Table 1]

In order for the neural network to acquire this relationship, learning of the intermediate layer is necessary. Learning was performed with a three-layer neural network configuration. The number of input layer units is 2, the number of intermediate layer units is 2, and the number of output layer units is 1. All synapse weights were initialized with random numbers in the range of ± 1, and all offset amounts were initialized with random numbers in the range of 0 and +1.

FIG. 6 shows the learning results for the learning parameters γ ₀ and η according to the present invention, and FIG. 7 shows the learning results for the learning parameters η and α by the conventional error back propagation method. The vertical axis represents the number of learning steps required for the absolute value error defined by (Equation 31) to become 0.1 or less.

In the learning method of the present invention, σ in the equation 27 is set to 1, and learning of w _ji is performed by the _equation (30).

On the other hand, in the conventional error back propagation method,
Learning is performed using Equation 13.

In the learning method of the present invention, the learning is completed in the shortest 27 steps (when γ ₀ = 10.0 and η = 0.01), whereas the error back propagation method has the shortest 153 steps. The learning is completed at (η = 1.0, α = 0.9). From the figure, the error backpropagation method linearly reduces the number of learning steps with respect to α, whereas the learning method of the present invention uses γ
_It can be seen that the number of learning steps decreases exponentially with respect to ₀ . The learning method of the present invention as a whole can realize 5 to 10 times higher speed than the back propagation method.

In the above-described embodiment, the method of learning the synapse weight between the intermediate layer and the output layer by using the least square method algorithm is shown. However, the present invention is not limited to the least square method algorithm. Instead, for example, a conjugate gradient method that enables high-speed learning or various optimization algorithms may be used.

In the above-mentioned embodiment, the description has been given for the three-layer neural network, but the present invention does not limit the number of layers.

[0089]

According to the learning processing apparatus and the learning method of the present invention, the synapse weight between the intermediate layer and the output layer can be learned at high speed and with high accuracy, so that the learning speed and learning accuracy as a whole are improved. You can

[Brief description of drawings]

FIG. 1 is an explanatory diagram of a neural network structure during learning showing an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a neural network structure during pattern conversion.

FIG. 3 is an explanatory diagram of a neural network structure at the time of learning by the conventional back propagation method.

FIG. 4 is a flowchart of an execution procedure showing an embodiment of the present invention.

FIG. 5 is a flowchart of an execution procedure showing an embodiment of the present invention.

FIG. 6 is an explanatory diagram of an execution result according to an embodiment of the present invention.

FIG. 7 is an explanatory diagram of a learning result by the conventional error back propagation method.

[Explanation of symbols]

1 ... input data, 2 ... input layer, 3 ... synaptic load, 4 ...
Intermediate layer, 5 ... Synapse load, 6 ... Output layer, 8 ... Inverse sigmoid function, 9 ... Teacher signal, 11 ... Error signal, 13 ... Weight calculation circuit, 14 ... Weight calculation circuit.

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] March 22, 1993

[Procedure Amendment 1]

[Document name to be amended] Statement

[Name of item to be corrected] Brief description of the drawing

[Correction method] Change

[Correction content]

[Brief description of drawings]

FIG. 1 is a diagram illustrating a new example during learning showing an embodiment of the present invention.
Explanatory drawing of a Lar network structure.

[Fig. 2] Neural network at the time of pattern conversion
FIG.

FIG. 3 is an explanatory diagram of a neural network structure at the time of learning by the conventional back propagation method.

FIG. 4 is a flowchart of an execution procedure showing an embodiment of the present invention.

FIG. 5 is a flowchart of an execution procedure showing an embodiment of the present invention.

FIG. 6 is an explanatory diagram of an execution result according to an embodiment of the present invention.

FIG. 7 is an explanatory diagram of a learning result by the conventional error back propagation method.

[Explanation of Codes] 1 ... Input data, 2 ... Input layer, 3 ... Synapse load, 4 ...
Intermediate layer, 5 ... Synapse load, 6 ... Output layer, 8 ... Inverse sigmoid function, 9 ... Teacher signal, 11 ... Error signal, 13 ... Weight calculation circuit, 14 ... Weight calculation circuit.

[Procedure Amendment 2]

[Document name to be corrected] Drawing

[Correction target item name] All drawings

[Correction method] Change

[Correction content]

[Figure 1]

[Fig. 2]

[Figure 4]

[Figure 3]

[Figure 5]

[Figure 6]

[Figure 7]

Claims (1)

[Claims] 1. A sigmoid-like nonlinear function is provided inside,
A signal processing unit including an input layer, an intermediate layer, and an output layer configured by a plurality of units that perform signal processing corresponding to an artificial neural element, and an output of the output layer with respect to an input signal pattern input to the input layer A neural network including a learning processing unit that sequentially and repeatedly calculates a coefficient of coupling strength between the units from the output layer side to the input layer side based on an error signal between a value and a teacher signal. In the learning processing device, the first learning processing unit that learns the coefficient of the bond strength between the intermediate layer and the output layer, and the second learning process that learns the other coefficient of the bond strength between them. A learning processing device for a neural network, which is provided with a section.