JPH064506A - Neural network learning method - Google Patents

Abstract

PURPOSE:To reduce the error with movement of a separate plane and to learn a neural network at a high speed by using only a pattern of a large error at the first stage of learning. CONSTITUTION:The input signal pattern of the teacher signal data 10 is inputted to the input layer neuron (S1). Then, the input signals are successively transmitted to the neurons of an output layer and the output of the output layer neuron is finally obtained (S2). An output error is calculated from the output signal of the data 10 and the calculated output of the output layer neuron (S3). Then, it is decided whether the learning pattern under display satisfies the weight correction conditions or not (S4). Then, the sharpest falling slope of an input pattern is calculated to the weight of each neuron (S5), and the correction value of this time is calculated from the error slope and the precedent correction value (S6). This operation is repeated until the output error is set less than the upper limit level of a fixed error that serves as a standard for the end of learning.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of learning network weights using a teacher signal in the processing field of pattern recognition and data compression using a neural network.

[0002]

2. Description of the Related Art A neural network is a simplified model of a neural network in the human brain, which is a network in which neuron neurons are connected via synapses through which signals pass in only one direction. . Signal transmission between neurons is performed through this synapse, and various information processing can be performed by appropriately adjusting the resistance of the synapse, that is, the weight. In each neuron, outputs from other neurons are weighted with synapses and input, and the sum of them is output to other neurons again by modifying the nonlinear response function.

The structure of the neural network is roughly divided into two types, that is, an interconnection type and a multilayer type. The former is suitable for the optimization problem and the latter is suitable for the recognition problem such as pattern recognition. The present invention relates to multilayer neural networks.

When a multi-layered neural network is used for pattern recognition and data compression, a teacher signal in which an input pattern and an output pattern are paired is prepared so that the network outputs a correct output, and the weight is applied to the network. It is necessary to learn in advance. This is usually done using an optimization technique called backpropagation. This method is discussed in, for example, the PDP model, Industrial Books (1989), pages 325 to 331.

The characteristic of this method is to reduce the output error of the network by using a non-decreasing continuous function such as a sigmoid function for the non-linear response function of the neuron and updating the weights as shown in the following equation. Especially.

[0006]

Dw / dt = −∂E / ∂w (Equation 2) w: weight E: output error of network t: time A processing procedure of the method is specifically shown using FIG.

First, the data 10 in FIG. 2 will be described. The learning data 10 is composed of input pattern data and teacher pattern data.
X learning pattern number P), (input layer neuron number No x learning pattern number P). The input signal pattern and the teacher signal pattern correspond to each other. For example, the input signal pattern 1 is input to the input layer, the error between the output from the output layer and the teacher signal pattern 1 is calculated, and the error is used for learning. .

In step 20, the input signal pattern of the teacher signal data 10 is input to the input layer neuron.

In step 21, the input signal is successively propagated to the neurons on the output layer side in accordance with equation 3, and finally the output of the neuron on the output layer is obtained.

[0010]

[Equation 3]

In step 22, the output error is calculated using the equation 4 by the output signal of the teacher signal data 10 and the output of the output layer neuron calculated in step 23.

[0012]

[Equation 4]

In step 23, the steepest descent gradient of the input pattern q is obtained for the weight of each neuron according to the equation (5).

[0014]

[Equation 5]

In step 24, the present correction amount is obtained from the error gradient and the previous correction amount using the following equation.

[0016]

[Equation 6]

Η: Learning coefficient α: Moment coefficient However, the second term on the right side is called a moment term and is an term added empirically to accelerate learning.

In step 25, the weight of each neuron is modified according to equation (7).

[0019]

[Equation 7]

At step 26, the processing from step 20 to step 25 is performed for all learning patterns.

In step 27, step 26 is repeated until the output error falls within the upper limit of a constant error that serves as a reference for the end of learning. The learning end condition is, for example, when the output errors of the output layer neurons are within a certain upper limit value such as 0.1, and this is true for all learning patterns.

On the other hand, it has been pointed out that learning takes a long time as a problem of this back propagation.
Therefore, in order to speed up the learning, there is a method of omitting the calculation without correcting the weight for the pattern satisfying the convergence condition. First, for each learning pattern, "output error is 0.
Is it within 1? " For the satisfying pattern, the calculation for correcting the weight in step 23 to step 25 described above is not performed. This processing procedure will be specifically described with reference to FIG.

In steps 30 to 32, the error of the output layer neuron is calculated according to the procedure similar to steps 23 to 25 described in the explanation of the back propagation.

In step 33, it is judged whether or not the learning pattern currently presented satisfies the ending condition "whether the output error is within 0.1". If not satisfied, the process proceeds to step 34.

In steps 34 to 36, calculation and weight correction associated with the weight correction are performed in accordance with the same procedure as in step 23 to step 25 described above.

In step 37, the processes of steps 30 to 33 are performed for all learning patterns.

In step 38, step 37 is repeated until the convergence condition is satisfied for all learning patterns.

As a result of the above, backpropagation learning without calculation can be performed.

[0029]

A problem of the backpropagation learning is that if the learning is performed using all the patterns from the beginning, the learning becomes slow. This is because, if learning is performed using all patterns from the beginning, the learning effects of the individual patterns cancel each other out.

This problem will be described in detail with reference to FIGS. FIGS. 5 and 6 show a case where a neuron having a two-dimensional input is trained to perform linear separation of x and ◯, FIG. 5 shows a state in which the learning is finished, and FIG. 6 shows a state before the learning. It is a thing. When the input pattern represented by × is input, the neuron outputs 0.9,
When the input pattern represented by ◯ is input, it is learned to output 0.1. The equation for the output O of the neuron is given by

[0031]

[Equation 8]

F represents a sigmoid function, and net represents a total input to the neuron. The straight line G in FIGS. 5 and 6 is net = 0.
Is a straight line that represents, and is hereinafter referred to as a separation plane. 7 and 8 are output characteristic diagrams of the neuron taken along the broken lines in FIGS. 5 and 6, respectively. Sigmoid function is net
When the absolute value of increases, it approaches 0 or 1. In other words, as the distance from the separation plane increases, or the magnitude of the weight vector | W | increases, the absolute value of net increases and the output of the neuron approaches 1 or 0. Therefore, in order to change the state of FIG. 6 to the state of FIG. 5 by learning, it is necessary to bring the separation plane G between the patterns of × and ◯ and increase | W | at the same time.

Next, learning when one input pattern is presented will be described. In FIG. 6, in order to present the input pattern ◯ of a and learn that it outputs 0.1, the separation plane G is pushed up to separate the input pattern a from G, and at the same time | W | Needs to be increased to reduce net. On the other hand, in order to present the input pattern x of b and learn it to output 0.9, on the contrary, the separation plane G is pushed down to separate the input pattern b from G and at the same time increase | W | Then you have to make the net bigger. When the patterns a and b are presented at the same time during learning, as a result, the separation plane G does not move much and only | W | becomes large. The same thing occurs when learning is performed using all the learning patterns in the figure.

When | W | is increased and net is increased, the differentiation of the sigmoid function that enters the steepest descent in the form of a product becomes small for all patterns, as shown in Equation 5. That is, since | W | is large, net is large for any pattern, and df / dnet is small. Therefore, the correction amount ΔW of the weight calculated from the steepest descent gradient using Equation 6 becomes small, and the learning speed cannot be increased.

The above can also occur in backpropagation learning that omits calculations. This is because, when the convergence condition becomes strict, that is, when the upper limit value of the output error becomes small, all the learning patterns do not satisfy the convergence condition, and the learning is performed using all the patterns from the beginning.

An object of the present invention is to provide a method for realizing high-speed learning by preventing learning from being delayed by learning all patterns from the beginning in learning using backpropagation. It is in.

[0037]

The above problem is that the output error of the neural network is calculated for each learning pattern, and the absolute value of each output error is larger than the weight correction determination error defined by the equation (1). When learning is performed for a pattern including even one output neuron and learning is performed, a weight correction amount is calculated using the back propagation method, and the weight is corrected by a learning method.

[0038]

When the learning is performed using all the learning patterns from the beginning, only the magnitude of the weight | W | becomes large, and the separation plane does not move much. As a result, the differentiation of the sigmoid function that enters the steepest descent in the form of a product becomes small for all learning patterns. Then, the correction amount ΔW of the weight calculated from the steepest descent gradient also becomes small, and the learning cannot be speeded up.

By the above means, in the beginning of learning,
Learning is performed using only patterns with large errors. Therefore, the increase of | W | is relatively suppressed, and the separation plane moves to reduce the error, so that the learning is performed at high speed.
This will be described in detail with reference to FIG.

FIG. 9 is an output characteristic diagram showing a state before starting learning as in FIG. In the figure, ● represents a learning pattern with a large error. Due to the output characteristic of the neuron cut along the broken line, the output is larger than 0.5 above the separation plane G and is 1 apart from G.
Approach. Conversely, below G, it has the opposite characteristic. On the other hand, since the pattern of ◯ is learned so as to output 0.1, the learning pattern of ● above the G and separated from G has a large error.

At the beginning, consider learning using only the learning pattern of ●. By learning, it tries to bring the output for ● close to 0.1. Therefore, in learning, at the same time as trying to raise the separation plane G upward,
Attempts to reduce the output of the neuron by reducing | W |. Since | W | becomes small, the differential value df of the sigmoid function
/ Dnet becomes large, and therefore, according to Equations 3 and 4, Δ
W keeps a relatively large value. As a result, it becomes possible to perform learning at high speed.

[0042]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will be described below with reference to FIGS.
Will be explained. FIG. 1 is a flowchart of the entire processing of the embodiment, and FIG. 4 is learning data 1 for performing weight learning.
It is explanatory drawing which shows the detail of 0. Reference numeral 50 is input data input to the input layer neuron,
There are i × the number of learning patterns P). Reference numeral 53 is teacher data for obtaining an error by comparing with the output of the output layer neuron, and there is a data number of (output layer neuron No.times.learning pattern number P). The input data 50 and the teacher data 53 are paired, and for example, the input signal pattern 51
Corresponds to the teacher signal pattern 54.

Next, the operation of this embodiment will be described with reference to FIG. In the figure, step 4 is a feature of the present invention.

In step 1, the input signal pattern of the teacher signal data 10 is input to the input layer neuron.

In step 2, the input signals are successively propagated to the neurons on the output layer side according to the equation 3, and the output of the output layer neurons is finally obtained.

In step 3, the output error is calculated by using the following equation based on the output signal of the teacher signal data 10 and the output of the output layer neuron calculated in step 2.

[0047]

[Equation 9]

In step 4, it is determined whether or not the currently presented learning pattern satisfies the weight correction condition. That is, a comparison is made with the judgment error defined by Equation 1, and it is judged whether or not there is an output layer neuron having an output error larger than the judgment error. If it exists, it is determined that the weight correction condition is satisfied, and the process proceeds to step 5.

In step 5, the steepest descent gradient of the input pattern q is obtained for the weight of each neuron according to equation (5).

In step 6, the present correction amount is obtained from the error gradient and the previous correction amount using the equation (6).

In step 7, the weight of each neuron is modified according to equation (7).

In step 8, the processes of steps 1 to 7 are performed for all learning patterns.

In step 9, step 1 is repeated until the output error falls within the upper limit of a constant error which is a reference for learning completion. This convergence condition is, for example, when the output error of each output layer neuron is within a certain upper limit value, such as 0.1, and this holds for all learning patterns.

By such a method, since learning is performed only in the first part of learning using only the pattern having a large error, the learning can be performed at high speed.

The weight correction condition described in step 4 is not limited to that of this embodiment, and a learning error exceeding it may be corrected by using the judgment error shown in Expression 10.

[0056]

[Equation 10]

Next, a second embodiment of the present invention will be described with reference to FIGS.
Also, description will be made with reference to FIG. FIG. 10 is a block diagram of the apparatus of this embodiment.

The operation of this apparatus will be described below in association with the processing flow of FIG. Note that steps 1, 2, 3,
... indicate the steps in Fig. 1, respectively.

In step 1, the neuron output calculation unit 63 reads the input signal pattern of the learning signal data 10 stored in the learning data storage unit 62 and inputs it to the input layer neuron.

In step 2, the neuron output calculator 6
3 obtains the output of the neuron in each layer according to equation 3.

In step 3, the error calculation part 66 is generated by the teacher signal of the learning data 10 stored in the learning data storage part 62 and the output of the output layer neuron calculated in step 2.
Calculates the output error using Equation 9.

At step 4, the weight correction controller 68
It is determined whether the currently presented learning pattern satisfies the weight correction condition. That is, the weight correction control unit 68 compares the weight correction determination error defined by Equation 1 with the output error calculated in step 3, and determines whether there is an output layer neuron having an error larger than the determination error. To judge. If it exists, it is determined that the weight correction condition is satisfied, and the weight correction control unit 6 is added to the steepest descent gradient calculation unit 67, the weight correction amount calculation unit 65, and the weight correction unit 61.
8 sends an indication signal to perform the calculations to correct the weights. Then, the process proceeds to step 5.

In step 5, the steepest descent gradient calculator 67
Calculates the steepest descent gradient of the learning pattern q for each neuron weight according to Equation 5.

In step 6, the weight correction amount calculation unit 65
However, the previous weight correction amount is read from the weight correction amount storage unit 64, and the current correction amount is calculated using the equation 6.

In step 7, the weight correction unit 61 corrects the weight of each neuron stored in the network weight storage unit 60 according to the equation (7).

In step 8, steps 1 to 7 are performed for all learning patterns.

In step 9, step 1 is repeated until the output error falls within the upper limit of a constant error that serves as a reference for the end of learning. This convergence condition is, for example, when the output error of each output layer neuron is within a certain upper limit value, such as 0.1, and this holds for all learning patterns.

Next, a third embodiment of the present invention will be described with reference to FIGS. 11 and 4. FIG. 11 shows a flowchart of the entire processing of the embodiment. FIG. 4 is an explanatory diagram showing details of the learning data 10 for performing weight learning.

The operation of this embodiment will be described with reference to FIG. In the figure, step 75 and step 74 are features of the present invention.

In step 70, the input signal pattern of the teacher signal data 10 is input to the input layer neuron.

In step 71, the input signals are successively propagated to the neurons on the output layer side according to the equation 3, and finally the output of the output layer neurons is obtained.

In step 72, the output error is calculated by using the equation 9 by the output signal of the teacher signal data 10 and the output of the output layer neuron calculated in step 71.

At step 73, the processing from step 70 to step 72 is performed for all learning patterns.

As a result of the processing of step 73 and steps 70 to 72, the error of the output layer neuron is obtained for all learning patterns.

In step 72, the errors obtained in step 73 and steps 70 to 72 are sorted in descending order. Then, a learning pattern is selected from the top in the order of increasing error by a predetermined number of patterns, and this is used as a learning pattern for correcting the weight.

In step 74, the weights are corrected by performing the processes of steps 76 to 81 for the learning pattern for correcting the weights obtained in step 75.

In step 76, the input signal pattern of the teacher signal data 10 is input to the input layer neuron.

In step 77, the input signals are successively propagated to the neurons on the output layer side in accordance with equation 3, and finally the output of the output layer neurons is obtained.

In step 78, the output error is calculated using the equation 9 by the output signal of the teacher signal data 10 and the output of the output layer neuron calculated in step 77.

In step 89, the steepest descent gradient of the input pattern q is obtained for the weight of each neuron according to equation 5.

In step 80, the correction amount for this time is calculated from the gradient of the error and the correction amount for the previous time using the equation (6).

In step 81, the weight of each neuron is modified according to the equation (7).

In step 74, the processing from step 76 to step 81 is performed for the learning pattern for correcting all weights.

In step 82, steps 73 to 75 are repeated until the output error falls within the upper limit of a constant error that serves as a reference for learning completion. This convergence condition is set such that the output error of each output layer neuron is within a certain upper limit, such as 0.1, and this holds for all learning patterns.

According to such a method, since learning is performed using only a pattern having a large error, it is possible to perform learning at high speed.

The learning pattern selection conditions for correcting the weights described in step 75 are not limited to those of the present embodiment, and the learning pattern selection condition is
The learning patterns having large errors may be selected in order from the top by a predetermined fixed number.

[0087]

According to the present invention, the learning is performed by using only the pattern having a large error at the beginning of the learning depending on the correction condition of the weight. Therefore, the effect of learning is less likely to be canceled out as compared with the case of learning by using all patterns from the beginning. That is, mainly, only the magnitude of the weight | W | becomes large, and the separation plane does not move much. As a result, since the increase of | W | is suppressed relatively, the differential of the sigmoid function that enters the steepest descent gradient in the form of a product is not so small, and the steepest descent gradient keeps a certain degree. Therefore, the weight correction amount ΔW does not suddenly decrease, so that the learning is performed at high speed.

[Brief description of drawings]

FIG. 1 is a PAD showing the overall processing of the first embodiment of the present invention.
Flow chart.

FIG. 2 is a flowchart showing a processing procedure of a back propagation method.

FIG. 3 is a flowchart of a processing procedure of a backpropagation method in which a calculation for weight correction is omitted for a learning pattern that satisfies a learning convergence condition.

FIG. 4 is an explanatory diagram showing details of learning data.

FIG. 5 is an explanatory diagram showing a state after the neuron has finished learning to separate signals ◯ and ×.

FIG. 6 is an explanatory diagram showing a state before the neuron starts learning the separation of signals ◯ and ×.

7 is an output characteristic diagram of a neuron taken along the broken line in FIG.

8 is an output characteristic diagram of a neuron taken along the broken line in FIG.

FIG. 9 is an explanatory diagram showing a state before the neuron starts learning the separation of signals ◯ and ×.

FIG. 10 is a block diagram showing the overall functional configuration of a second embodiment of the present invention.

FIG. 11 is a PA showing the overall processing of the third embodiment of the present invention.
The flowchart of D.

[Explanation of symbols]

3 ... Input of input pattern to neural network, 4 ... Calculation of output of each neuron, 5 ... Calculation of error of output layer neuron, 6 ... Judgment of weight correction condition, 7 ... Calculation of steepest descent gradient, 8 ... Weight correction Calculation of quantity, 9 ... Correction of weight, 10 ...
Learning data.

Claims (5)

[Claims] 1. In learning of a neural network,
A neural network learning method characterized in that a learning pattern to be learned is controlled by using a judgment condition different from a learning end judgment condition for judging the end of learning. 2. In learning of a neural network using a back propagation method, a first step of calculating an output error of the neural network for each learning pattern, and learning of the pattern based on the output error. A neural network characterized by including a second step of determining whether or not, and a third step of calculating a correction amount of the weight by using the backpropagation method when learning and correcting the weight. Online learning method. 3. In the learning of a neural network using the backpropagation method, the first step of calculating the output error of the neural network for each learning pattern and the absolute value of the output error are given by A second step of determining that learning is to be performed for a learning pattern including an output neuron larger than the defined weight correction determination error; When learning, a third step of calculating a weight correction amount using the backpropagation method and correcting the weight is included. 4. In the learning of a multi-layered neural network using the back propagation method, the first step of calculating the output error of the neural network for each learning pattern and the absolute value of each output error are: The second step of determining that learning is to be performed for a learning pattern including an output neuron that is larger than the weight correction determination error defined by Equation 1, and when learning is performed,
And a third step of correcting the weight by calculating the correction amount of the weight by using the backpropagation method. 5. In the learning of a multilayer neural network using the back propagation method, a first step of calculating an output error of the neural network for each learning pattern, and an output error obtained in the first step. A second step of selecting a learning pattern from the top in the order of increasing error by a predetermined number of patterns and using it as a learning pattern for correcting the weight, and a learning pattern for correcting the weight,
A third step of calculating a weight correction amount by using the back propagation method and correcting the weight, the neural network learning method.