JP3220021B2 - Neural network learning acceleration system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a neural network, and more particularly to a method for reducing the number of repetitions during learning.

[0002]

2. Description of the Related Art Conventionally, in neural network learning, BP (Back Propagation Alg) has been used.
orthm, an error back propagation algorithm) is often used. The neural network has a hierarchical structure composed of an input layer, an output layer, and at least one intermediate layer composed of one or more neurons.

FIG. 4 is a flowchart showing an example of the operation of a conventional neural network. In this figure,
The number of neurons is 3, and it is composed of one intermediate layer.

Generally, the number of layers is K, the number of neurons in the k-th layer is N _k , and the output of the k-th layer is X ^(k) = X ^(k) i: i = 1 to
N _k , the coupling coefficient between the (k−1) th layer and the kth layer is W ^(k) = W
^(k) ij: If i = 1 to N _k−1 and j = 1 to N _k , the operation of the neural network is (k−1)
The output X of the layers of the ^(k-1) from the k-th layer output X ^(k),

[0005]

(Equation 1)

And converting the data finally input from the input layer into data output from the output layer. Here, the layer of k = 1 is the input layer, and the layer of k = K is the output layer. The function f is a monotonically increasing function having a saturation characteristic.

[0007] Learning of the neural network is an operation of correcting the coupling coefficients W ^(k) ij of all the layers so that the output value of the output layer with respect to the input value entering the input layer becomes an expected value. That is.

The output X ^(K) of the output layer and the expected value T =
The difference from (Ti: i = 1 to N _k ) gives an evaluation of the learning result. In addition, the square error ^{E = 1/2 | X (} k) -T | 2 of W
^(k) ij Partial differential value δE / δW ^(k) ij approaches 0, the effect of iterative learning is saturated (converged)
Is shown.

[0009]

In the prior art described above, a great number of repetitive learnings must be performed for the convergence of the learning. Therefore, there is a problem that it takes an enormous amount of learning time when learning the neural network.

Accordingly, an object of the present invention is to reduce the number of repetitions at the time of learning of a neural network, thereby shortening the learning time.

[0011]

SUMMARY OF THE INVENTION The present invention providesOne or more d
And at least one of the input and output layers
Composed of an intermediate layerNeural networks,Previous
Coupling coefficients calculated by the neural network, and
Error, the difference between the output of the output layer and the expected value
And a data storage device that saves each time learning is performed.
The number of times of learning is counted, and the number of times of learning is constant
If not reached,The neural network
A learning counter that returns the process to the learning counter.
If a certain number of learnings have been completed by the
Use the coupling coefficient and evaluation error stored by the storage device
Calculate the optimal linear sum of the coupling coefficients,
As a coefficientThe neural networkLearning to pass to
Device and the optimal linear sum calculated by the learning accelerator
Neural network implemented as new coupling coefficient
Sometimes learning to determine whether the effect of learning acceleration was effective
Equipped with a learning acceleration judgment device to reduce the number of repetitions during learning
Neural network learning
Speed system.

It is preferable that the data storage device has a data output unit for writing the coupling coefficient and the evaluation error to a file, and a disk device for storing the file.

[0013] Further, the learning number counter includes an initialization unit for substituting 0 for the counter variable, a count-up unit for increasing the value of the counter variable by 1, and a judgment for comparing the counter variable with a predetermined constant and magnitude. It is preferable to have a part.

Further, it is preferable that the predetermined constant is 8 in the determination section. Further, it is preferable that a predetermined constant can be freely set by the user.

[0015] Further, the learning acceleration device calculates the optimum linear sum so that the linear prediction error of the linear sum is optimal in the minimum sense, and a data input unit for reading the coupling coefficient and the evaluation error from the file on the disk device. It is preferable to have an optimal linear sum calculation unit that performs the calculation.

The neural network value learning acceleration system of the present invention improves the coupling coefficient W during learning of the neural network.

More specifically, means for storing a coupling coefficient for each learning iteration and a difference (evaluation error) | X−T | between the output value X of the output layer and the expected value T; It has means for counting the number of times, means for improving the coupling coefficient W, and means for determining the effect of learning acceleration.

A means for improving the coupling coefficient W by using the coupling coefficient and the evaluation error for each learning iteration stored by the means for storing the coupling coefficient and the evaluation error is used to reduce the coupling coefficient and the evaluation error. Improve. The improvement of the coupling coefficient W is performed by using the stored optimal linear sum of the coupling coefficients so far as a new coupling coefficient (the final linear sum will be described later). At the start of learning, since there is no past coupling coefficient for obtaining an optimal linear sum, learning is repeated a predetermined number of times using a conventional BP. The number of repetitions is counted by means for counting, and the time when the improvement of the coupling coefficient W by the optimal linear sum is started is determined. In some cases, the effect of improving the coupling coefficient W by the optimal linear sum does not appear, and the result is determined by means for determining the effect of learning acceleration.

Although the definition and determination method of the optimal linear sum are not unique, here, a method for determining the optimal linear sum such that the linear prediction error of the linear sum is optimal in the sense of least square will be described. In addition to this, for example, selection (Chebyshev acceleration) for making the linear prediction error optimal in the sense of minimax is also effective.

First, the linear sum of the accumulated coupling coefficients of the layer k is

[0021]

(Equation 2)

And the linear estimation error D ′ for the linear sum W ^(k) (k = 2 to K) is

[0023]

(Equation 3)

It is assumed that Here, (A, B) represents the range of the accumulated data used for the linear sum, and the linear sum coefficient C (n)
Is

[0025]

(Equation 4)

An unknown that satisfies.

Under the condition of (Equation 3), a C that minimizes the square norm | D '| ² of (Equation 2) of the error vector
(N): n = AB is easily obtained by, for example, the undetermined constant method.

Whether or not the coupling coefficient obtained from the optimal linear sum approaches a true coupling coefficient that reduces the evaluation error can be explained as follows. (1) If "the difference between the true coupling coefficient W and the coupling coefficient Wn of each time" and the "evaluation error" have a linear relationship, | Wn-W
If the space spanned by | is perfect, the optimal linear sum matches the true coupling coefficient W. (2) If the “difference between the true coupling coefficient W and the coupling coefficient Wn of each time” and the “evaluation error” are in a linear relationship, the optimal space in the finite N | Wn−W | The linear sum is
"Optimal coupling coefficient".

Originally, the optimum linear sum is obtained so as to minimize the evaluation error, so that the optimum coupling coefficient is obtained in a base space spanned by | Wn-W |. That is, | Wn-W
Since | and the evaluation error are assumed to be in a linear relationship, minimizing the evaluation error means that | Wn-W | is minimum.

Further, the n-th evaluation error is given by | Wn-W |
Can be tailored with each component. At this time,
If Wn is sufficiently close to W, higher-order expressions of second and higher orders can be ignored in most cases, and a linear relationship holds.

When calculating the linear sum of Wn, higher-order terms may cancel each other. Therefore, the optimal linear sum is more likely to be closer to the true coupling coefficient W than any of the Wn.

[0032]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of the neural network learning acceleration system of the present invention. This system uses a conventional neural network 1
, A data storage device 2, a learning counter 3, a learning acceleration device 4, and a learning acceleration determination device 5. For each function, the conventional neural network 1 first performs the conventional BP learning. Next, in the data storage device 2, the coupling coefficient calculated by the conventional neural network 1 and the evaluation error at that time (the difference between the output of the output layer and the expected value) are stored every time learning is performed. . The learning number counter 3 counts the number of times of learning, and passes the processing to the learning acceleration device 4 when the learning of a certain number of times has been completed. If the number of times of learning has not reached the certain number, the process returns to the conventional neural network 1. The learning acceleration device 4 calculates the optimal linear sum of the coupling coefficients by using the coupling coefficients and the evaluation error stored by the data storage device 2 and passes the calculated linear sum to the conventional neural network 1 as a new coupling coefficient. The learning acceleration determination device 5 determines whether or not the effect of learning acceleration has been obtained when the neural network is executed using the optimal linear sum calculated by the learning acceleration device 4 as a new coupling coefficient.

FIG. 2 is a flowchart showing the operation of the embodiment of the neural network learning acceleration system according to the present invention. The operation will be described with reference to FIG. 1. First, an initial value is set for the coupling coefficient W between each layer of the neural network (step A1). 2. The input data for learning is read (step A2). The learning input data is data for which a correct output value is known in advance. 3. The learning counter is initialized (step A3). 4. A neural network is executed with the learning input data, and an output value X is obtained from an output layer (step A4). 5. An evaluation error | X−T | is calculated (step A5). 6. It is checked whether the evaluation error is smaller than the convergence judgment value (step A6).

If the evaluation error is smaller than the convergence judgment value, the learning is terminated. If the evaluation error is larger than the convergence error, step A7 is performed. 7. Before executing the neural network, it is checked whether learning acceleration is performed (step A7).

If learning acceleration has been performed, step A14
Is performed, and if learning acceleration is not performed, step A8 is performed. 8. The coupling coefficient W is compared with the evaluation error (step A
8). 9. The learning counter is counted up (step A9). 10. It is checked whether the number of times of learning is greater than a certain number (step A10).

If it is smaller, learning by the conventional BP (step A11) is performed, the coupling coefficient is updated, and the process returns to step A4 to execute the neural network with the updated coupling coefficient.

If it is larger, the past coupling coefficient and the evaluation error stored in step A8 are read (step A1).
2) Calculate the optimal linear sum of the coupling coefficients (step A1)
3). The calculated optimal linear sum is used as a new coupling coefficient, and the process returns to step A3. 11. After the first neural network is executed after the completion of the learning acceleration, the learning acceleration is evaluated (step A1).
4). 12. It is determined whether or not the learning acceleration is effective (step A15). If there is no effect, the coupling coefficient and the evaluation error immediately before the learning acceleration are reset as the current coupling coefficient and the evaluation error, and step A8 is performed. If there is an effect, step A8 is performed as it is.

The operation of the embodiment of the present invention has been described above. Next, the apparatus in which the operation in each step is performed will be described.

Step A1, Step A2, Step A
4 to A6 and step A11 are performed by the conventional neural network 1 in FIG. Steps A3, A9 and A10 correspond to learning counter 3 in FIG.
Do with. Step A8 is performed by the storage device 2 in FIG. Steps A12 and A13 are performed by the learning acceleration device 4 in FIG. Step A7, Step A14
Steps A16 to A16 are performed by the learning acceleration determination device 5 in FIG.

FIG. 3 is a block diagram showing in detail the configuration of an embodiment of the neural network learning acceleration system according to the present invention. According to this, the data storage device 2
Is composed of a data output unit 21 for writing a coupling coefficient and an evaluation error to a file, and a disk device 22 for storing the file. The learning number counter 3 includes an initialization unit 31 that substitutes 0 for a counter variable, a count-up unit 32 that increases the value of a counter variable by 1, and a determination unit 33 that compares a counter variable with a predetermined constant and size. Be composed. Here, it is assumed that the predetermined constant of the determination unit 33 is set to 8 here. The learning acceleration device 4 includes a data input unit 41 that reads a coupling coefficient and an evaluation error from a file on the disk device 22 and an optimal device that calculates an optimal linear sum so that a linear prediction error of the linear sum is optimal in a minimum sense. It comprises a linear sum calculation unit 42. The learning acceleration determination device 5 compares the evaluation errors immediately before and immediately after the learning acceleration, and determines whether the learning acceleration is effective.

Next, the operation of the embodiment of the present invention will be described in more detail with reference to FIG.

First, an initial value of the coupling coefficient W is set in the conventional neural network 1 (step A).
1).

Learning input data such that the correct output value is (1, 2, 3) is input to this neural network (step A2).

The initialization section 31 of the learning counter 3 substitutes 0 for the counter variable (step A3), and executes the neural network with the coupling coefficient of the initial value (step A).
4). The output X1 (X11, X
12, X13) and the correct output value (1, 2, 3) (X
11-1, X12-2, X13) are calculated as evaluation errors (step A5). It is determined whether the RMS (root mean square, root mean square) of the evaluation error is smaller than the convergence determination value (step A6). If it is smaller, the learning ends. If it is larger, check whether learning acceleration is performed before executing the neural network. In this case, since the processing has not been performed, the data output unit 21 of the data storage device 2 creates a file on the disk device 22 and writes the coupling coefficient and the evaluation error in the file (step A8). If the learning acceleration has been performed, the learning acceleration is evaluated (step A14, described later). The count-up unit 32 of the learning counter 3 increments the value of the counter variable by 1 (step A9), and checks whether the value of the counter variable of the determination unit 33 is larger than a certain value (step A10). It is assumed that a certain constant value is set to 8 here. If the value of the counter variable is smaller than 8, the coupling coefficient is updated by the conventional learning method using BP (step A11), and the process returns to the execution of the neural network using the coupling coefficient again to repeat the subsequent processing. At the time of repetition, when the coupling coefficient and the evaluation error are stored, a file is created on the disk, and the file is appended (added) to the file. When the counter variable is larger than 8, the learning acceleration device 4 accelerates the learning. If the learning is repeated less than eight times and the learning is completed, the learning acceleration of the present invention is not performed.

In the data input section 41 of the learning acceleration device 4,
The coupling coefficient and the evaluation error are read from a file on the disk (step A12). Since eight sets of coupling coefficients and evaluation errors are stored in the file, they are all read.
The optimum linear calculator 42 calculates the optimum linear sum of the eight coupling coefficients (step A13). The calculation of the optimal linear sum is
A linear sum coefficient is calculated such that the square norm | D ′ | ² of the linear sum D ′ of the evaluation errors calculated by (Equation 2) is minimized,
As shown in (Equation 1), the optimal linear sum of the coupling coefficients is calculated using the linear sum coefficient. The optimal linear sum of the coupling coefficients calculated in this manner is a coupling coefficient whose evaluation error is smaller than that of the linearly summed coupling coefficient. The process returns to step A3 with this optimum linear sum as a coupling coefficient, and the subsequent processing is repeated. At this time, in step A7, since the learning acceleration has just been performed, the flow branches to evaluation of learning acceleration (step A14). The learning acceleration determination device 5 compares the current evaluation error with the evaluation error immediately before the learning acceleration. If the current evaluation error is smaller, the current coupling coefficient and the evaluation error are stored as they are (step A8). If the current evaluation error is larger, it is determined that there is no effect of the learning acceleration, and the coupling coefficient and the evaluation error immediately before the learning acceleration are reset and stored.

Further, in step A9 of FIG. 2, it is determined whether the number of times of learning is equal to or less than a certain number (8 in the embodiment). It is preferable that the user can use the certain number of times.

[0048]

According to the present invention, when learning is repeated,
An optimal coupling coefficient can be calculated by the learning accelerator. Therefore, it is possible to reduce the number of times of learning during the neural network learning, and it is possible to end the learning in a short time.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a learning acceleration device for a neural network according to the present invention.

FIG. 2 is a flowchart showing a processing operation of a learning acceleration device for a neural network according to the present invention.

FIG. 3 is a block diagram showing a configuration of an embodiment of a learning acceleration system for a neural network according to the present invention.

FIG. 4 is a diagram showing the operation of a conventional neural network.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Conventional neural network 2 Data storage device 3 Learning number counter 4 Learning acceleration device 5 Learning acceleration judgment device 21 Data output unit 22 Disk device 31 Initialization unit 32 Count up unit 33 Judgment unit 41 Data input unit 42 Optimal linear sum calculation unit

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-5-6350 (JP, A) Masatake Mori, Masaaki Sugihara, Kazuo Murota, "Iwanami Course Applied Mathematics Linear Calculation", Iwanami Shoten, p. 34-p. 38 (1994) (58) Field surveyed (Int. Cl. ⁷ , DB name) G06N 3/00-3/08 JICST file (JOIS) INSPEC (DIALOG)

Claims (6)

(57) [Claims] 1. An input layer comprising one or more neurons.
A neural network comprising an output layer and at least one hidden layer; a coupling coefficient calculated by the neural network;
And a data storage device for storing an evaluation error , which is a difference between the output of the output layer and an expected value, every time learning is performed, and counting the number of times of learning, and the number of times of learning reaches a certain number of times. If not, use the neural network
A learning number counter that returns the process to the network, and if a certain number of learnings have been completed by the learning number counter, the optimal linear sum of the coupling coefficient is calculated by using the coupling coefficient and the evaluation error stored by the data storage device. calculated, the neural net it as a new coupling coefficient
A learning accelerator to be passed to the workpiece , and when the neural network is executed with the optimal linear sum calculated by the learning accelerator as a new coupling coefficient,
A neural network learning acceleration system comprising: a learning acceleration determining device for determining whether or not the learning acceleration is effective; and reducing the number of repetitions during learning. 2. The neural network learning according to claim 1, wherein the data storage device has a data output unit for writing a coupling coefficient and an evaluation error to a file, and a disk device for storing the file. Acceleration system. 3. The learning counter according to claim 1, wherein the initialization unit substitutes 0 for the counter variable, a count-up unit increases the value of the counter variable by 1, and compares the counter variable with a predetermined constant and magnitude. The neural network learning acceleration system according to claim 1, further comprising a determination unit. 4. The neural network learning acceleration system according to claim 3, wherein said predetermined constant is 8 in said determination unit. 5. The neural network learning acceleration system according to claim 3, wherein said predetermined constant can be freely set by a user in said determination unit. 6. A data input unit for reading a coupling coefficient and an evaluation error from a file on the disk device, wherein the learning acceleration device has an optimal linear sum so as to minimize a linear prediction error of the linear sum. The neural network learning acceleration system according to any one of claims 1 to 5, further comprising: