JPH04230570A - System for learning neural network - Google Patents

Abstract

PURPOSE:To improve sort accuracy without over learning requiring enormous learning time and means to save the learning by performing learning so as to reduce errors even to the learning sample while maintaining sort functions supplied initially when the neural networks are adjusted by using learning sample data. CONSTITUTION:This system is consisted of an error calculation means 11 calculating output errors against learning sample data 14 of neural network, a load difference calculation means 12 calculating the load difference which is the one between the coupled load of the neural network and the initial set value of the bias value, and a coupled load adjustment means 13 adjusting the coupled load and the bias value so as to reduce the sum of the output error and the load difference. The system ends the learning when learning sample data 14 are repeatedly supplied to the neural network to turn the error less than the constant value.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a learning method for a neural network, and in particular to a method for classifying data to be classified and a general classification rule for classifying data to be classified. This invention relates to a neural network learning method that uses data to construct a neural network with high classification accuracy. [0002] Conventionally, multilayer neural networks have been used for the purpose of data classification processing. Figure 3
shows a configuration diagram of a multilayer neural network. In the figure, an input layer 301 inputs feature amounts of data to be classified. The output layer 302 outputs the classification results. The intermediate layer 303 is located between the input layer 301 and the output layer 302,
It is composed of one layer or multiple layers. The input of the unit in each layer and the output of each unit in the layers before and after it are combined. Regarding the units at both ends of each bond, the unit in the previous layer is called the starting point unit, and the unit in the subsequent layer is called the ending point unit. Taking the layer in FIG. 3 as an example, the starting point units are units 30a, 30b, 30c, and 30d of the input layer 301.
, and the terminal units that are subsequent stages for units 30a, 30b, 30c, and 30d are units 31a, 31
b, 31c. The output of each unit is determined according to the following formula. oj (k) = tanh( a
j(k))
(1) [0004] [Formula 1] [0005] Here, oj (k) is k layer (k≧1, k
=1 is the output value of the j-th unit of the input layer). lol
ij(k) is from the i-th unit of the k-1 layer to the k layer j
The joint load to the th unit, N(k-1), is k-1
This is the total number of units in the layer. However, w0j(k) is the connection load for giving a bias to the j-th unit of the k layer, and the output value o0 of the j-th unit of the (k-1) layer
j(k-1) is always set to 1. In addition, the input layer 301 (
Each unit (k=1) outputs the feature amount of the input data to be classified as it is to the next stage. [0006] Data is added to such a neural network.
In order to classify data, when the feature values of the data to be classified are given to the input layer units, only the output layer units corresponding to the classification category to which the data belongs output a high value. It is necessary to set the coupling weight and bias value between each unit so that other output layer units output a low value. [0007] For this purpose, as a first method in the conventional technology, the connection weights and bias values are initially set to random values, and when the feature values of learning sample data whose classification results are known are input. A back error propagation learning method is known in which connection weights and bias values are repeatedly adjusted minute by minute so that the error between the actual output value and the desired output value is reduced. FIG. 4 shows a diagram for explaining the first conventional technique. The learning means 402 includes an error calculating means 410 that calculates the error between the actual output value and the desired output value when the feature quantity 403 of the learning sample data is input to the neural network 401 before learning, and the error calculating means The connection weight adjustment means 412 repeatedly adjusts connection weights and bias values minute by minute using a back error propagation learning method so that the error calculated in step 410 is reduced. [0009] A pre-learning neural network 401 is set in the learning means 402, and the feature values and classification results 403 of the learning sample data are stored in the neural network.
is input. As a result, the learning means 402 performs learning using the error calculation means 410 and the connection weight adjustment means 412,
The neural network 404 after learning is output. Next, the operation of the learning means 402 will be explained in detail. The error calculation means 410 of the learning means 402 calculates the output value of the neural network when the feature amount 403 of the learning sample data is input to the input layer of the neural network. and the squared error of the desired output value ej = oj
(K) − yj (4) is calculated. Here, K is the layer number of the output layer, ie, the total number of layers of the neural network. Further, yj is a correct classification result that is a desired output value for the learning sample data 403 of the j-th unit of the output layer. Next, the load change adjustment means 411 slightly adjusts the connection load and bias value of the neural network according to the following formula so that the calculated error is reduced. Coupling load or bias value adjustment amount Δwij(k)
is calculated according to the following formula, which is the product of the error signal dj (k) of the end point unit and the output value oi (k-1) of the start point unit. Δwij(k) = −ε1 dj
(k) oi (k-1)
(5) Here, ε1 is a parameter that determines the amount of adjustment in one repetition. The k layer is the output layer (k=
K), the error signal dj (k) of the end point unit is calculated by the following formula. dj (K) = ej (1-oj
(K) )(1+oj (K) ) (6) Here, ej is the output error calculated by the error calculation means 410 according to equation (4). When the k layer is an intermediate layer (1<k<K), the error signal dj (k) is obtained by the following equation. ##EQU3## The error calculation by the error calculation means 410 and the adjustment of the connection weights and bias values by the connection weight adjustment means 412 are executed by repeatedly applying the learning sample data 403, and the error is calculated by repeatedly applying the learning sample data 403. Learning ends when becomes below a certain value. Next, a second conventional method will be explained. In this method, when a data classification rule that is considered to be approximately correct is known, the connection weights and bias values of the neural network are initially set so that the neural network has a classification function equivalent to this data classification rule. Then, the training sample data
This method uses a back error propagation learning method to adjust connection weights and bias values using data. This method was proposed by the present inventor in Japanese Patent Application No. 2-15
It is described in Neural Net Learning Method, No. 570. This method converts known data classification rules, which classify data based on the presence or absence of features in training data, into a logical operation formula, and then applies the multiplicative and additive terms of the logical operation formula to each unit of the neural network. The structure and connection weights of the neural net are set so that it has a classification function equivalent to a logical operation formula, and the neural network is trained using example data. In addition, the known data classification rules, which classify the learning data based on the magnitude relationship by comparing the features of the input learning data with constant values, are converted into logical operation expressions, and the logical operation expressions are size comparison term,
Multiplicative terms and additive terms are assigned to each unit of the neural net, and the neural net has a classification function equivalent to a logical operation expression.
Set the structure and connection loads of the net, and set the learning data.
It uses data to teach students. FIG. 5 shows a diagram for explaining the second conventional method. In this figure, the same parts as those in FIG. 4 are given the same reference numerals, and the explanation thereof will be omitted. The logical operation conversion means 502 converts the data classification rule 501 into a logical operation. Connection load setting means 503
is a new model that performs an operation equivalent to the obtained logical operation expression.
Set the structure and connection weight of the network. The learning means 402 applies the feature quantity 403 of the learning sample data to the neural network obtained from the connection weight setting means 503 using a back error propagation learning method similar to the conventional first method. input, and adjust the coupling weight and bias value in minute increments so as to reduce the error between the actual output value and the desired output value. Next, the operation of each means will be explained. Here, it is assumed that the following rule is given as the known data classification rule 501. IF (x1 > a1 ) and (x2 > a2 )
THEN y (8)
IF (x3 > a3 ) and (x4 < a4 ) TH
EN y (9) (8)
The classification rule is ``the feature quantity x1 of the data to be classified is a constant a
1 and the feature quantity x2 is larger than the constant a2, the data to be classified belongs to category y.''The classification rule in (9) is ``If the feature quantity x3 of the data to be classified is is larger than the constant a3, and the feature x
4 is smaller than the constant a4, it means that the data to be classified belongs to category y. [0025] The logical operation conversion means 502 has the formula (8), (
9) From “(x1 is larger than a1, and x2
is greater than a2) or (x3 is greater than a3 and x4 is less than a4), then y is true.''however,"
・” represents logical product, “+” represents logical sum, and “¬” represents logical negation. y=(x1 > a1 )・(x2 > a2 )+(
x3 > a3 )・¬(x4 > a4 )


(10) Next, the joint load setting means 503
The operation will be explained. FIG. 6 shows a connection load setting means 50 in a conventional second method.
A diagram for explaining 3 is shown. Initial connection load setting means 5
03 is the neural network 6 according to the logical operation formula 601 (formula 10) given from the logical operation conversion means 502.
Determine the connection configuration and connection load of 02. As shown in FIG. 602, the connection configuration of the neural network 602 is such that one input layer unit is provided for each variable appearing on the right side of the logical operation formula 601 (Equation 10), and one input layer unit is provided for each comparison term between a variable and a constant. One second hidden layer unit is assigned to each multiplicative term in the first hidden layer unit so as to realize addition of all multiplicative terms in the output layer. The method for determining the bonding load w is described in the above-mentioned patent application No. 2
It is described in Neural Network Learning Method, No. 185570. This method sets the structure and connection weights of a neural net so that it has a classification function equivalent to a logical operation formula. [0028] Here, a formula for determining the joint load w will be shown. First, a method for determining the joint weight wi and bias wo of the addition unit (output layer) will be explained. Considering an n-input addition unit, if the input signal Ii, (1≦i≦n) is -1≦Ii≦-d, it is “false”, and if d≦Ii≦1, it is “true”. When the output signal o is −1≦o≦−d′, it is “false”, and when d′≦o≦1, it is “true”. Under the above conditions, in order to realize the additive function, the joint load wi (
1≦i≦n) and bias w0 are set as follows. wi = w≧{2tanh-1 (d')}/{d
(n+1)-(n-1)} (11) w
0 = {w(n-1)(d+1)}/2
(12)
However, when the input is a negative term (b of y=a+¬b), wi=-w. Also, d is d>(n-1)/
It is necessary to satisfy the condition (n+1). Next, a method for determining the joint load and bias of the multiplicative unit (second intermediate layer) will be explained. Define n, d, and d' in the same way as for addition. To realize the multiplicative function, the connection weight wi (1≦i≦n) and bias w0 are set as follows. wi = w≧{2tanh-1 (d')}/{d
(n+1)-(n-1)} (13) w
0 = {w(1 −n)(d+1)}/2
(14
) However, if the input is a negative term (b of y=a・¬b
) is assumed to be wi = -w. Also, d is d>(n-1
)/(n+1). Furthermore, a method for determining the coupling load and bias in the comparison unit (first intermediate layer) will be explained. Input I
The relationship between the connection weight w1 and the bias value w0 of the unit that outputs "true" when is larger than the constant A is set as follows. w0 = -w1 A

(15) In addition, there may be extra units and connections between units that do not correspond to the logical formula, and these units and connections between units are randomly selected with a sufficiently smaller absolute value than the connection weight determined above. set to the value. As described above, the neural network obtained by the connection weight setting means 503 is
02 adjusts the connection weight and bias value using the same back error propagation learning method as the first conventional method. [Problems to be Solved by the Invention] However, the conventional first
The method uses the back error propagation learning method to acquire training sample data.
Learning is performed so that the error between the actual output value and the desired output value of the neural network for the controller is reduced. Therefore, by repeating learning on training sample data, it is possible to obtain a neural network that gives correct classification results, but when inputting unknown data other than training sample data, it is possible to obtain a neural network that gives correct classification results. There is no guarantee that classification results will be obtained. In particular, when the training sample data is biased or the number of training sample data is insufficient, the problem arises that the classification function of the neural network is specified to classify only the training sample data. There is. Furthermore, in the second conventional method, the connection weights of the neural network are initially set so as to have a classification function equivalent to a known classification rule that is considered to be approximately correct; Furthermore, learning is performed on the learning sample data using the back error propagation learning method, which is the first conventional method. Therefore, it is expected that the classification accuracy for unknown data will be improved compared to the conventional first method. However, like the conventional first method, the training sample data
If the data is biased or contains noise,
There is a problem in that the initially set connection weights change greatly due to the back error propagation learning method, resulting in excessive learning such as learning not converging or requiring a huge amount of learning time, leading to a decrease in classification accuracy. [0037] The present invention has been made in view of the above points.
When adjusting a neural network using training sample data, learning is performed to reduce the error on the training samples while maintaining the initial classification function as much as possible, preventing overfitting. The purpose of the present invention is to provide a learning method for neural networks that can improve classification accuracy without having to do so. [Means for Solving the Problems] FIG. 1 shows a diagram of the basic configuration of the present invention. In a neural network learning method having a learning means 10 for training a neural network using training sample data 14 for which classification results are known, the learning means 10 performs the training of the neural network. An error calculation means 11 for calculating an output error with respect to sample data 14, a load difference calculation means 12 for calculating a load difference which is a deviation from the initial setting values of the connection load and bias value of the neural network, and an output It has a joint load adjusting means 13 that adjusts the joint load and bias value so that the sum of the error and the load difference is reduced. [Operation] The present invention learns a neural network that is initially set to have a classification function equivalent to a classification rule that is considered to be approximately accurate given in advance. When adjusting the connection weight so that it decreases, the connection weight is adjusted so that the sum of the output error for the learning sample data and the load difference, which is the deviation from the initial setting value of the connection weight, decreases. That is, learning is performed so that the error with respect to the learning sample is reduced while maintaining the initially given classification function as much as possible. Embodiment FIG. 2 shows a configuration diagram of an embodiment of the present invention. In the figure, the same parts as in FIGS. 1 and 5 are designated by the same reference numerals, and the explanation thereof will be omitted. In the figure, a load difference calculating means 12 calculates the difference between the joint load and the initial setting value. The other configurations are similar to those of the second conventional method shown in FIG. The operation of the present invention will be explained below. The known data classification rule 501 is converted into a logical operation formula (formula 10) by the logical operation conversion means 502, and the connected weight setting means 5
Sent on 03. The connection weight setting means 503 sets the structure and connection weight of the neural network according to the logical operation formula (Equation 10). The operation up to this point is similar to the second conventional method. Next, the operating principle of the learning means 10 will be explained. The learning means 10 changes the value of the connection weight for the neural network initialized by the connection weight setting means 503 so as to decrease the evaluation function E' expressed by the following equation. [Equation 4] Here, E on the right side is the square of the output value of the neural network and the desired output value when the features of the learning sample data shown in equation (3) are input. This is an error. Furthermore, the second term on the right side is a term representing the deviation of the joint load from the initial setting value. That is, the learning means 10 adjusts the connection weights so that the sum of the squared error of the output value of the neural network and the deviation of the connection weights from the initial setting value is reduced. Wij(k) in equation (16) is the joint load wij
(k) is the initial setting value. f(x, y) is a function in which the difference between x and y increases, for example, |x-y|, (x-y)
2 etc. ηij(k) is a parameter that determines the contribution of the difference between each connection load to the evaluation function E', and is usually a constant value η for all connection loads wij(k), or a constant value for each layer. Let the value η(k) be. In order to reduce the above evaluation function E',
The amount of change Δw'ij(k) in the joint load wij(k) is given by the following formula. ##EQU5## Here, Δwij(k) is the adjustment amount of the connection weight in the normal back error propagation learning method calculated by equation (5). According to equations (17) and (18), for example, f
When (x, y) = (x-y)2, Δw'ij(k) = Δwij(k) −2εηi
j(k) wij(k) ( wij(k) −Wij
(k) )

(19). The learning means 10 performs the above operations. Below f
When (x, y)=(x-y)2, that is, equation (19
), we will explain the case where the joint load is adjusted according to [0050] The error calculation means 11 is a neural network.
The error ej between the output value E of the neural network and the desired output value when the feature quantity 14 of the learning sample data is input to the input layer of the neural network is calculated using the formula (3 ), calculated according to formula (4). The load difference calculation means 12 stores the initial setting value Wij(k) of the joint load, and uses the current joint load value wij(k) to calculate the second value on the right side of equation (19). −2εηij(k) wij(k) (w
ij(k) −Wij(k) ) is determined, and the deviation of the connection load is determined. Similar to the conventional reverse error propagation learning method, the connection weight adjustment means 13 calculates the connection weight adjustment amount Δwij(k), which is the first term on the right side of equation (19), and calculates the connection weight adjustment amount Δwij(k), The value of the deviation of the second term on the right side calculated by -
2εηij(k) wij(k) (wij(k) −
Wij(k) ) and the connection load adjustment amount Δw'ij
(k) is calculated, and the equation (3) of the error calculation means 11 and the equation (
Adjust the joint load using the output error ej calculated in 4). Error calculation by the above error calculation means 11,
The calculation of the deviation of the connection loads from the initial setting value by the load difference calculation means 12 and the adjustment of the connection loads by the connection load adjustment means 13 are executed by repeatedly applying the learning sample data 14 to the neural network, Learning ends when the error becomes less than a certain value. [0054] According to the present invention, a neural network initially set to have a classification function equivalent to a pre-given classification rule that is considered to be approximately accurate is used as a learning sample. When adjusting using data, the connection weights are adjusted so that the sum of the output error for the learning sample data and the deviation from the initial setting value of the connection weights is reduced. Learning can be performed on learning samples to reduce errors while maintaining the classification function as much as possible, avoiding over-learning in the past and improving classification accuracy.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the principle configuration of the present invention.

FIG. 2 is a configuration diagram of an embodiment of the present invention.

FIG. 3 is a configuration diagram of a multilayer neural network.

FIG. 4 is a diagram for explaining a first conventional method.

FIG. 5 is a diagram for explaining a second conventional method.

FIG. 6 is a diagram for explaining connection load setting means in a second conventional method.

[Explanation of symbols]

10 Learning methods 11 Error calculation means 12 Load difference calculation means 13　Connection load adjustment means 14 Learning sample data 501 Known data classification rules 502 Logical operation conversion means 503 Combined load setting means

Claims (1)

[Claims] 1. A neural network learning method comprising a learning means for training a neural network using training sample data for which classification results are known, wherein the learning means is configured to train the neural network using training sample data for which classification results are known. an error calculation means for calculating an output error of the neural network with respect to the learning sample data; and a load difference calculation means for calculating a load difference that is a deviation of the connection weight and bias value of the neural network from the initial setting values. , a connection weight adjustment means for adjusting connection weights and bias values so that the sum of the output error and the load difference is reduced.