JP3114276B2 - Learning method of hierarchical neural network - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an MRII (MAD) for a hierarchical neural network constituted by units using a signum function (sign function) as an output function.
ALINERule II).

[0002]

2. Description of the Related Art A neural network is used, for example, for classifying (recognizing) input signals. For example, as shown in FIG. 3, when a pattern in an input image is classified using a hierarchical neural network including an input layer 11, an intermediate layer 12, and an output layer 13, the number of units 14 of the input layer 11 is determined by the number of hierarchical neural networks. Is determined by the number of pixels of the image 15 to be input to the. Similarly, the number of units 16 in the output layer 13 is determined by the number of pixels of the output image, the number of categories of classification, and the like. The number of units 17 in the intermediate layer 12 needs to be appropriately selected depending on the number and complexity of patterns to be recognized, but a method for determining an appropriate number of units has not been established.

FIG. 4 shows a neuron model used for the units 17 and 16 of the intermediate layer 12 and output layer 13 of the hierarchical neural network. This neuron model is
When a binary signal X (x ₁ , x ₂ ..., X _n ) of ± 1 is input, a sum y is obtained by multiplying the input signal by a connection weight, and a binary signal q = SGN (y) is output. I do. The signum function SGN (y) used for the output function is a function that outputs +1 or −1 by looking at the sign of y having a real value. x ₀ = 1 is the threshold input.

As a learning method of a hierarchical neural network using a signum function as an output function of each unit in FIG. 3, that is, for example, when an input image is input, an output is obtained at an output terminal corresponding to the pattern, and the image is classified. As a method of determining each coupling load to enable
The MRII method will be described with reference to FIG. An appropriate number of units 17 of the intermediate layer 12 are prepared, for example, the number of signals prepared for learning, and a decimal number is randomly given to the connection weights of all the units 17 and 16 to initialize them (S ₁ ). Next, the total error is initialized to zero, and the number of presentations of the learning set is initialized to zero (S ₂ ), and one of the prepared learning sets (a set of the input signal X and the teacher signal D used for learning) is set.
The set is presented to the neural network, that is, the input signal X is input to the neural network (S ₃ ). The output of the intermediate layer 12 is calculated for the input signal, and the output of the output layer 13 is calculated to obtain an output signal Q (S ₄ ).

[0005] The error E between the output signal Q and the teacher signal D calculated (S _5), which it adds the error E to the total error as a new total error (S _6). Next, the number of trials is initialized to 0 (S ₇ ), and the intermediate layer unit in which the internal state value y of the intermediate layer unit 17 is the number of trials + 1 plus the first zero is selected, that is, the absolute value of the internal state value y is the number of trials. the intermediate layer unit selects small +1 th (S _8).

The binary output q of the selected intermediate layer unit
To create a new intermediate layer output signal (hereinafter referred to as a trial pattern) (S ₉ ). Enter the attempt pattern at the output layer 13, calculates and 'seeking (S _10), the output signal Q' output signal Q error E between the teacher signal D and '
(S ₁₁ ). 'Compares and the error signal E obtained in step _{S 5 (S 12), E} >E' the error signal E of the coupling weight of the intermediate layer units selected in the case of, actually the sign of the output of the unit updated by the LMS algorithm to invert (S _13). In other words, the current connection weight is W
k, W _{k + 1} after the update, α as the learning coefficient, and d (the binary output after sign inversion) as the teacher signal, W _{k + 1} = W _k + α
εX / | X | ^2, calculates the _{^{ε = d-X T W k}} . E ≦
For E 'undoing the inverted symbols in the trial pattern, not updated connection weights (S _14).

Next, the number of trials is incremented by one, and a new trial number is obtained.
(S₁₅), The number of trials is the same as the number of units in the middle layer 12
Check if they match (S₁₆), If they do not match,
S ₈Return to In this way all of the middle tier units
In order of those whose internal state value y is close to zero.
The total load is updated or not. Then enter
Input the force signal X again and find the output signal Q again
(S₁₇), And compares the output signal Q with the teacher signal D (S
₁₈), In the case of a mismatch, the combination of the units 16 of the output layer 13
The load is updated by the LMS algorithm (S₁₉), Match
If it is, leave the coupling load of the output layer unit as it is.
You.

Next, the number of presentations of the learning set is incremented by one, and
A new number of learning set presentations (S ₂₀), The learning set
The number of presentation matches the number of learning sets given in advance
Check whether (S_{twenty one}), If not, step S_Three
And do the same for another training set.
Steps are repeated for all training sets in the same way.
Top S_Three~ S_{twenty one}After the execution (learning) is completed (one cycle
Check if total error is zero)
Click (S_{twenty two}), If not zero, step S_TwoBack to
Again zero total error for all training sets
Step S until_Two~ S_{twenty two}Repeatedly (learn)
You. When the total error becomes zero, the learning ends.

In the learning of FIG. 5, the output signal of the intermediate layer 12 is calculated as shown in FIG.
And the sum of the products of the respective intermediate layer units multiplied by the coupling load is obtained to obtain the internal state value, and the internal state value is substituted for the signum function to obtain a binarized intermediate layer output. As shown in FIG. 7, the output signal of the output layer 13 is calculated by summing the output of each intermediate layer and the coupling weight of each output layer unit, obtaining the internal state value, and obtaining the internal state value. By substituting the value into the signum function, an output signal of the binarized output layer is obtained. The error between the output signal Q of the output layer and the teacher signal D is calculated as shown in FIG.

The generation of the trial patterns in steps S ₈ and S ₉ is performed as shown in FIG. Sorted in ascending order of the absolute value of the internal state value of each unit of the intermediate layer, (number of trials + 1)
Find the middle layer unit with the smallest internal state value,
The output sign of the unit is inverted, and this and the output of the other hidden units are used as the trial pattern. Step S
The update of the coupling load of the intermediate layer unit in ₁₃ is performed as shown in FIG. That is, the difference between the output of the selected hidden unit and the internal state value of the unit is obtained, and the product of the difference ε, the learning coefficient α and each input pixel signal is divided by the number of input pixels to obtain the value of the pixel signal. Is added to the current connection load to obtain the updated connection load. Update connection weights in the output layer unit in step S ₁₉ is performed as shown in FIG. 11. First, the output of the hidden layer is calculated, and then the output Q (n) of each output layer unit is calculated.
(N), and when they do not match, each output layer unit n
, The difference ε between its internal state value and the teacher signal D (n) is calculated, and the product of the ε and α and the internal state value of each intermediate layer unit divided by the number M of intermediate layer units is , Is added to the coupling load with the intermediate layer unit to obtain a new coupling load with the intermediate layer unit. This is performed for each output layer unit.

[0011]

The performance of a hierarchical neural network strongly depends on the network structure such as the number of intermediate layers and the number of units. For example, if the number of units in the intermediate layer is too large, the vector space of the input signal is unnecessarily divided, and the generalization ability of the hierarchical neural network is reduced. However, since a method for obtaining an appropriate number of intermediate layer units has not been established, the structure of a hierarchical neural network must be determined by trial and error.

When determining the number of units of the intermediate layer by trial and error, it is common to use a redundant number of intermediate layer units. For this reason, there is a problem that the structure of the hierarchical neural network becomes large, and the learning time and the amount of calculation increase.

[0013]

According to the present invention, in an MRII learning method for a hierarchical neural network, the learning (learning) for each learning set is completed once (one cycle end). , The intermediate layer unit that always has E = E ′ is deleted as a non-contributing intermediate layer unit that does not contribute to the operation of the network.

[0014]

When a hierarchical neural network composed of neuron models used in the method of the present invention is used for pattern recognition, an intermediate layer unit has a function of dividing a multidimensional vector space created by an input pattern. Since one intermediate layer unit divides a multidimensional vector space into two, if there are a plurality of intermediate layer units, the multidimensional vector space is finely divided. The output layer unit outputs recognition results by checking where the input pattern exists in the divided multidimensional vector space. At this time, if the multidimensional vector space is appropriately divided, the generalization ability of the hierarchical neural network increases, and it is possible to have excellent recognition ability. Conversely, if there are more intermediate layer units than necessary and the multidimensional vector space is finely divided for that reason, if the number of units is appropriate but the division is not appropriate, the generalization ability will be low. In the above learning procedure, when the sign of the output signal of the hidden layer unit is inverted, the degree of contribution of the hidden layer unit is determined depending on whether or not the influence appears in an error of the output of the hidden layer. When the error decreases, updating the connection weight so that the sign of the selected hidden unit is reversed is to correct the division of the multidimensional vector space so as to be appropriate. The layer units can be considered to contribute to the pattern recognition performed by the network. If the error increases, it can be considered that the selected middle layer unit may have contributed to the pattern recognition performed by the network. On the contrary, it is considered that the intermediate layer unit in which the error does not increase or decrease at least once in one cycle of learning does not contribute to the pattern recognition performed by the network or the degree of the contribution is very low. In the learning of the present invention, if the error does not decrease, the policy is not to update the connection weight. Therefore, the intermediate layer unit in which the error does not increase or decrease does not receive the optimization of the division of the multidimensional vector space. , It is reasonable to consider it as not contributing and delete it.

[0015]

FIG. 1 shows an embodiment of the present invention. Steps corresponding to those in FIG. 5 are denoted by the same reference numerals. In the present invention, an intermediate layer unit table is prepared, one bit is allocated to each unit, and this bit is set to "1" so that a flag can be set. Only parts different from FIG. 5 will be described. In step S ₂ is initialized to erase the flag each bit zero of the intermediate layer unit table. 'If it is determined, a flag in the intermediate layer unit table for the intermediate layer Sonitto selected (S _23), the procedure proceeds to step _{S 13, E <E' E} > E at step S ₁₂ selected Similarly for the intermediate layer unit flags the intermediate layer unit table (S _24), step S ₁₄
Move on to If E = E ', no flag is set.

The training set presentation number at step S ₂₁ is consistent with the number of training set and terminates the learning for all the training set, i.e. when one cycle of the learning is completed, whether all the bits of the intermediate layer unit table 1 Check (S ₂₅ ), if all are not 1, 0 bit,
That locate the intermediate layer unit flag is not set from the intermediate layer unit table, delete the hidden units as non-contributing intermediate layer unit (S _20), step S ₂₂
And all the bits of the intermediate layer unit table are 1
For immediately proceeds to step S _22.

The elimination of the non-contributing intermediate layer unit is described in FIG.
This is performed as shown in FIG. The one intermediate layer unit m is read from the intermediate layer unit table, and it is checked whether or not this is 1 (S ₃₁ ). If this is not 1, that is, if the flag is not set, the intermediate layer unit m and each input are read. Coupling load Wm (ij) of each intermediate unit with signal
(Suffix mid for W is omitted) is set to zero (S ₃₂ ), and then each coupling load W _n (m) between the intermediate layer unit m and each output layer unit (the superscript ou for W)
t is omitted) is set to zero ( _S33 ). Thus not one, i.e. flag deletes the non-contribution intermediate layer unit the steps S ₃₁ to S ₃₃ for each intermediate layer unit is not running standing.

In the above description, the corresponding connection load is set to zero in order to eliminate the non-contributing intermediate layer unit. However, a portion where the connection load is set to zero in the connection load memory may be reduced and removed. In this case, if the stuffing process is performed during the learning for each cycle during the learning, the relationship between the intermediate layer and the output layer will be different. Therefore, it is necessary to correct this relationship by a learning algorithm. However, when the connection weight is deleted as zero as described above, the processing is simplified in that the learning algorithm does not need to be modified in the middle.

Hierarchical neural networks are not limited to application to pattern recognition devices. Learning can be performed on an electronic computer, and the connection weights after learning can be copied to a ROM, and the ROM can be used for other devices. it can.

[0020]

As described above, according to the present invention, since the non-contributing intermediate unit is deleted during the learning, the amount of calculation in the learning is reduced, and the learning time is shortened.
Also, a hierarchical neural network having an appropriate number of intermediate layer units is configured, and a hierarchical neural network with high generalization ability can be obtained.

[Brief description of the drawings]

FIG. 1 is a flowchart showing an embodiment of the present invention.

Figure 2 is a flow diagram showing a specific example of a deletion step S ₂₆ of the non-contribution hidden unit in FIG.

FIG. 3 is a block diagram showing a hierarchical neural network.

FIG. 4 is a block diagram showing an example of a neuron model (unit).

FIG. 5 is a flowchart showing a future learning method.

FIG. 6 is a flowchart showing the calculation of the output of the hidden layer.

FIG. 7 is a flowchart showing calculation of an output layer output.

FIG. 8 is a flowchart showing calculation of an error.

FIG. 9 is a flowchart showing generation of a trial pattern.

FIG. 10 is a flowchart showing a process of updating the coupling load of the intermediate layer unit.

FIG. 11 is a flowchart showing a process of updating the connection load of the output layer unit.

Continuation of the front page (56) References "Error Back Propagation Learning Algorithm for Automatically Reducing the Number of Middle Layer Elements in a Hierarchical Neural Network" Yutaka Matsunaga, Yoshiaki Nakade, Kazuyuki Murase, IEICE Technology Research Report, Vol. 91, No. 25, pp. 9-14, 1991, "Back Provision with Selection Function (Reduction of Number of Intermediate Units and Faster Convergence)", Masafumi Hagiwara, IEICE Technical Report Research Report, Vol. 89, No. 104, pp. 85-90, 1990, "30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation," Widow B. and Lehr M.S. A. , Proceedings of the IEEE, vol. 78, no. 9, pp. 1415-1442, 1990 (58) Field surveyed (Int.Cl. ⁷ , DB name) G06G 7/60 G06F 15/18

Claims (1)

(57) [Claims] 1. A method for learning a hierarchical neural network constituted by units using a signum function as an output function, comprising: a. Giving an appropriate decimal to the combined load of all units in the middle and output layers; b. Training set prepared (pair of input signal and teacher signal)
Is input to the neural network, c. Calculating an error E between the output signal at that time and the teacher signal; d. Selecting an intermediate layer unit in the order in which its internal state value is close to zero and inverting the sign of the binary output of the selected unit to produce a new intermediate layer output signal (referred to as a trial pattern); e. The trial pattern is input to the output layer to determine an output signal, and an error E 'between the output signal and the teacher signal is determined. F. The error E 'is compared with the error E, and when E>E', the coupling load of the selected intermediate layer unit is updated so that the sign of the binary output is actually inverted; g. When E ≦ E ′, the sign obtained by inverting the trial pattern is restored, h. Repeat dg above for all intermediate layer units, i. Thereafter, the input signal is input again to obtain an output signal, and an error between the output signal and the teacher is determined; j. If the error is not zero, update the coupling weight of the output layer unit; k. Perform steps b through j for each of the other learning sets, l. Thereafter, it is determined whether or not the total error (total error) of i obtained for each learning set is zero, and m. If the value is not zero, the above b to l are repeated, and the process ends when the value is zero. In the learning method of the hierarchical neural network, when the execution of all the learning sets in k is completed, the middle layer unit where E = E 'is always deleted in the execution. How to learn the network.