JPH0415860A - Method for making learning of neural network efficient - Google Patents

Abstract

PURPOSE:To improve learning efficiency by enlarging an update step width as against a pair of an input pattern and a teach signal by prescribed multiple when learning effect as against a pair of the specified input pattern and the teach signals is not improved. CONSTITUTION:The sum of differences between the desired output values and real output values of all output units as against the prescribed input pattern is obtained at every repetitive calculation of learning using a back propagation method. When the sum is larger than a previously set threshold, the update step width as against a pair of the input pattern and the teach signal is enlarged by prescribed multiple. 1.2-1.6 multiple is suitable for the update step width. Thus, learning efficiency improves.

Description

[Detailed Description of the Invention] [Industrial Application Fields] The present invention relates to a method for improving the learning efficiency of neural networks, and more specifically, to neural networks used for pattern recognition such as character recognition and speech recognition.
This paper relates to a method for improving learning efficiency when learning using the backpropagation method.

[Prior art 1] The learning algorithm for backblowing that the present invention seeks to improve is one that trains a multilayer neural network formed by an input layer, an output layer, and an intermediate layer as shown in FIG. Yes, and will be explained below.

The units of the network are connected from the human power layer to the output layer, and as shown in FIG. The connection weight coefficient h W
Take the sum of the products neti = ΣWji - Oi, and further pass it through the input/output function f(x') to obtain the output signal Oj
Output = f (net i).

In other words, when a certain input signal pattern is input to the input layer of the network, the calculations described above are performed in all units (however, in the human layer, it is often not passed through the input/output function), and the final output is The coupling weight coefficients between units are determined so that the signal pattern output from the layer becomes a desired pattern (hereinafter referred to as a teacher signal). Here, the sum of squares Ep of errors between the teacher signal and the output signal of the neural network is used as the evaluation function.

Ep=(1/2) ・Σ(tpj Opi)2−・
(1) Here, tpj is the teacher signal of the output unit j for the input pattern P, and Opj is the output signal of the output unit j. This error function Ep needs to be minimized for all input patterns. Therefore, the problem becomes a minimization problem of determining the coupling weighting coefficients that minimize E=ΣEp.

To solve this problem, the backpropagation method uses the steepest descent method. For more information on the backpropagation method, please refer to Qi Yi'ylhart et al.'s ``Parallel Distributed Processing'' engineer. Ite Publishing (D, E, Rumelh
Artet al: Parallel Distri
Butted Processing, MITPres
s f198611.

In the backpropagation method, the connection weighting coefficients are determined as follows.

Regarding the connection weighting coefficient WH from unit 1 to unit j, the amount of update (amount of change) Δ for the second input/output data
ρWjj is ΔρWji:r)・δpj・Opi・−(2
) is given by However, η = update step width (constant) δpj”Opj・(10pj)・(tpj−O
pj) (When unit J is a crab unit) δpj=opj ・(l 0pj) ・Σδpic
Wkj (when unit j is an intermediate unit) tpj: Teacher signal of unit J for the second input pattern Opj: Actual output value of unit J for the second input pattern.

In this way, the basic principle of the backpropagation method is the steepest descent method, so in order to reach the minimum value in the shortest distance, the update width ΔWji must be made infinitely small. As the number of times increases, the convergence speed becomes slower. Therefore, the update width ΔWji is as large as possible.
To obtain , the value of η in equation (2) above is increased (
I want to do this, but the update direction tends to oscillate. So Ru
In Melhart's literature, the previous update amount (change amount) is added as an inertia term.

ΔρWj ((n+1.) = η・δpj−Ocsi+ a・△ρWj<(n)・・
・(3) It is proposed that However, a: is a constant.

-M In equation (3), the values of the constants η and a are determined empirically. In Japanese Unexamined Patent Publication No. 1-320565, this constant η is set so that the output error of the neural network takes a minimum value or a value close to the minimum value for each repeated learning calculation or once for several repeated learning calculations. It is proposed that the value of α be changed.

[Problems to be Solved by the Invention] In the above equation (3), if the values of the constants η and a were determined empirically, learning would take a long time and an inappropriate local minimum value (local minimum value) would be generated. ) is easy to fall into.

JP-A-1-320565 discloses that in order to determine the constants η and α so that the output error Ep becomes small for each repeated learning calculation, a finite number of values of the constants η and α are prepared and Ep is the smallest among them. The constants η and a are rejected. This method has the problem that each time the values of the constants η and a are changed, Ep must be calculated for the number of combinations of the constants η and a prepared, and the minimum Ep must be calculated from among them. be.

In addition, during learning, calculations are repeated for a set of pairs of input patterns and teacher signals, but the learning effect for any particular input pattern is not as good as for others, and learning often takes a long time. . This is thought to be due to the fact that the same conditions are used for all input pattern and teacher signal pairs in one learning iteration calculation, that is, the values of the constants η and α are the same in equation (3). . An object of the present invention is to improve the learning efficiency of a neural network by changing the value of the constant η and changing the learning conditions for that pair when the learning effect for a particular input pattern and teacher signal pair is not improved. The purpose of this invention is to provide a method for improving network learning efficiency.

[Means for Solving the Problems 1-B] The teacher signal in the pack propagation method has the same number of data as the number of output units, each representing a desired output value of the unit. When observing the value of output kaninits at that point for each repeated learning calculation,
In many cases, the value of a particular unit does not change and remains at a value that is significantly different from the desired output value. In other words, the effect of learning based on the input pattern for setting the output value to the desired output value is not improved.

The present invention is applied to learning a hierarchical neural network using the backpropagation method, and employs the following technical means. In other words, for each repeated calculation of learning, the sum of the errors between the desired output value and the actual output value of all output units for a certain input pattern is calculated, and when the sum is greater than a preset threshold, the difference between the input pattern and the teacher signal is calculated. This is a method for improving the learning efficiency of a neural network, which is characterized by increasing the update step width for a pair by a predetermined multiple.

The update step width is preferably 1.2 to 16 times.

That is, if the update step width is within the range of 1.2 to 1.6 times, the number of times of learning is the smallest and the learning efficiency is improved. The reason for this is that when the update step width is less than 1.2 times, the update effect is too small and the number of learning increases. On the other hand, if the update step width exceeds 1.6 times, the update direction tends to oscillate, and the update amounts Δ and Wji become difficult to converge to a constant value, so that the learning efficiency decreases on the contrary.

[Operation] In the present invention, p
The update amount ΔρWji for the th input/output data is determined.

△pWji(n+1) =η・η′ ・δρj・Opi ・・−(4) or ΔpWji(n+1) =η ゛ η′ ° δpj °Oρi tena・ΔpWj
i(n) ... 10pj each in (5) When tpjl>s η'=t 10p each
j When tpjl≦S η'=1 Here, S is the threshold and t is the update step width. Strictly speaking, appropriate values for the threshold S and update step width t depend on the task, but in the experimental example In this case, learning efficiency could be improved by setting the threshold value s = 0.7 and the update step width 1.2≦t≦1.6.

[Embodiment 1] FIG. 1 is an explanatory diagram showing the structure of a hierarchical neural network for character recognition, which is an embodiment of the present invention. The experimental conditions are as follows.

(1) The structure of the neural network is that there are 160 units in the input layer, 20 units in the middle layer, and 10 units in the output layer, and the input/output function of the unit is f (x) = 1/ ( 1+exp (-x)) is used.

(2) The objects to be recognized are 10 characters from numbers 0 to 9, and the output characters correspond to each character.

(3) Characters are divided into 160 segments, and each segment is made to correspond to an inkaninit, and the character is characterized by l for the character part and 0 for the background part. (4) Learning is performed by pack propagation method. The previous connection weighting coefficient is given by a random number.

Equations (3) and (5) were used to update the connection weighting coefficients.

In each equation, n=0.75 and a=0.8. The convergence condition that determines the end of learning is E, <0
．． 08.

(5) In the experiment, Nol: In equation (5), s = 0.7, t = 1.2 and Shimono No2:
In formula (5), s = 0.7, t = 1.4, and Shimono No. 3:
In formula (5), s = 0.7 and t = 1.6 No.4:
Three experiments were conducted using the formula (3), each with a different initial state.

Table 1 shows the above experimental results. here.

The number of learning times is the number of times for a pair of an input pattern and a teacher signal, and in one learning repetition, 10 characters are learned 10 times.

Table l As shown above, according to the present invention, the values of S and t in equation (4) or (5) are appropriately determined as described above, and appropriate values are determined for each learning using a pair of input pattern and teacher signal. By setting the update step width, we were able to improve learning efficiency.Also, in order to determine this update step width, we calculated the sum of all output kaninits for each presentation of the input pattern. The value S is set in advance
Although it is necessary to compare the sizes of

[Effects of the Invention] The present invention has excellent effects on improving the learning efficiency of neural networks.

[Brief explanation of drawings]

FIG. 1 is an explanatory diagram showing the structure of a hierarchical neural network according to an embodiment of the present invention, FIG. 2 is an explanatory diagram showing the general structure of a hierarchical neural network, and FIG. FIG. 2 is an explanatory diagram showing the input/output relationship. Wji... Connection weight coefficient

Claims (1)

[Claims] 1. In learning a hierarchical neural network using the backpropagation method, the error between the desired output value and the actual output value of all output units for a certain input pattern is determined for each repeated learning calculation. 1. A method for improving the learning efficiency of a neural network, characterized in that the total sum is calculated, and when the sum is larger than a preset threshold, the update step width for the input pattern and teacher signal pair is increased by a predetermined multiple. 2. The method for improving learning efficiency of a neural network according to claim 1, wherein the update step width is 1.2 to 1.6 times.