JPH09311846A - Neural network - Google Patents

Abstract

PROBLEM TO BE SOLVED: To attain higher learning and generalization capability by grouping the units of an output layer for learning, recognition and estimation, and then using an optimum unit in the group. SOLUTION: When various set values of a learning pattern, a teacher pattern, etc., are inputted to an input device 5, these input value store every pattern in a working memory 2 via an input control part 3. Then each input value is inputted to a neural network 1, and the learning is performed based on the set value of learning and teacher patterns, etc., of the memory 2 and the coupling coefficient is updated. The coupling coefficient is stored in the network 1, and the learning result is outputted to an output/display device and displayed there via an output control part 4. In such a constitution, the units of an output layer are divided into groups and the optimum unit of each group represents the group. Thus, the square error of the output value is reduced for the teacher value.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network type pattern discriminating apparatus and a value estimating apparatus for information processing and learning means for the same.

[0002]

2. Description of the Related Art A typical error back-propagation method (back propagation method) is used as a structure and learning means of a neural network (multi-layered hierarchical neural network).
(For example, "Neural network model and connectionism", Shunichi Amari, published in The University of Tokyo Press.) This outline is explained by the unit internal state explanatory diagram of FIG. The relationship is (1),
In the expressions (2) and (3), the input for the unit i is represented by Oj (j = 1 to N), and the coupling coefficient (weight) for each Oj is represented by Wij. At this time, the coupling coefficient Wij is initialized by a random number or the like.

Input sum of products Xi = Σ WijOj (1) The equation (1) is applied to the function f (Xi) for conversion. As a function, a sigmoid function like the following formula (2) that is generally different is generally used. f (Xi) = 1 / {1 + exp (−Xi)} ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (2) Output Yi = f (Xi) ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (3) where Yi The value of is a number between 0 and 1. On the other hand, the input layer unit uses the input value as it is as the output value.

The error backpropagation method is a learning method for minimizing the square error of a unit between an output pattern Yi obtained from a learning pattern Xk and a desired output value Ii (hereinafter referred to as a teacher pattern) for the learning pattern. . Here, an example of a three-layer structure as shown in FIG. 2 will be described, but the same applies to the case of multiple layers. The coupling coefficient is learned as follows. First, assuming that the loss function is E (square error), learning of the output layer is represented by equation (4), where i is the unit number of the output layer,
j is the unit number of the intermediate layer, k is the unit number of the input layer, Oj is the output of the intermediate layer, Wij is the coupling coefficient between the intermediate layer and the output layer, Yi is the output of the output layer, and Ii is the teacher pattern. E (W) = 1 / 2.SIGMA. (Yi-Ii). (Yi-Ii) .... (4) W represents all Wij.

When the steepest descent method (stochastic descent method) is applied, the coupling coefficient may be changed as shown in the equation (6) using the equation (5) as a learning signal. δE / δWij = (Yi−Ii) f ′ (ΣWijOj) Oj (5) New Wij = old Wij−c δE / ΣWij (6) Here, c is a learning constant, Oj is the output of the middle layer.

The sigmoid function of the expression (2) can be generally expressed by the following expression (7), and when differentiated, the original function is used and expressed as the expression (8). f (x) = 1 / {1 + exp (-x)} ... (7) f '(x) = f (x) (1-f (x)) ... (8) Actually, f (ΣWijOj) = Yi, so (8)
The formula is represented by f ′ (ΣWijOj) = Yi (1-Yi), and the formula (5) becomes the formula (9). [delta] E / [delta] Wij = (Yi-Ii) Yi (1-Yi) Oj (9) Therefore, if c = 1, then equation (6) gives new Wij = old Wij- (Yi-Ii) Yi ( 1-Yi) Oj ... (10) Here, D1i = (Ii−Yi) Yi
Assuming (1-Yi), equation (10) is represented by equation (11). New Wij = old Wij + D1iOj ... (11) In the learning of the intermediate layer, if the loss function is E (square error),
It is expressed by equation (12). E (V) = 1/2 (Yi-Ii) (Yi-Ii) (12) V is all Vjk. Here, Vjk represents a coupling coefficient between the intermediate layer and the input layer. When the steepest descent method (stochastic descent method) is applied, equation (13) is used as the learning signal (1
Change the coupling coefficient as shown in equation (4).

ΔE / δVjk = δE / δOj · δOj / δVjk = Σ (Yi-Ii) δYi / δOj · δOj / δVjk = {Σ (Yi-Ii) f ′ (WijOj) Wij} × f ′ (ΣVjkXk) Xk・ ・ ・ (13) New Vi = Old Vi-cδE / δVjk ・ ・ ・ ・ ・ ・ (14) Equation (14) is converted using Equations (15) to (18), and (19) Get the expression. Dj = − {Σ (Yi−Ii) f ′ (WijOj) Wij} = ΣD1iWij ... (15) f ′ (ΣVjkXk) Xk = Oj (1-Oj) Xk .. (16) .delta.E / .delta.Vjk = -DjOj (1-Oj) Xk (17) where Xk is the output of the input layer. Therefore, if c = 1, new Vjk = old Vjk + DjOj (1-Oj) Xk (18) where D2j = DjOj (1-Oj). Therefore, new Vjk = old Vjk + D2jXk (19) In this way, the learning equation for the coupling coefficient between the input layers and the intermediate layer can also be expressed. The calculation of the coupling coefficient starts from the unit of the output layer, moves to the unit of the intermediate layer, and successively moves to the intermediate layer of the previous stage. Therefore, learning proceeds as follows. First, a learning pattern is input and the result is calculated. Change all coupling coefficients to reduce the error with the resulting teacher pattern. The learning pattern is input again. This is repeated until it converges.

[0008]

However, the conventional techniques have the following drawbacks. (1) Since the comparison between the output value of each unit of the output layer and the corresponding teacher value at the time of learning of the conventional multi-layer hierarchical structure type neural network is performed on a one-to-one basis, learning of a specific unit of the output layer is performed. Since there is no progress, there is a sufficient risk that learning will not progress as a whole network. (2) Since the comparison between the output value of each unit of the output layer and the corresponding teacher value at the time of recognizing unknown data of the conventional multi-layer hierarchical structure type neural network is performed on a one-to-one basis, a specific unit of the output layer Due to the different values of, there is a sufficient risk that the recognition result of the entire network will be erroneous. (3) As a result of learning a fixed number of times, or learning until it falls to a local minimum, there is variation in the difference between the output of each learning pattern and the teacher pattern, or there is a learning pattern that does not match the teacher pattern. There are many things. In order to solve this, it takes execution time just to increase the number of times of learning, and a specific learning pattern may not converge even if the number of times of learning is increased.

[0009]

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and has as its object to provide the following improvement means for a neural network. The multilayer hierarchical structure type neural network according to claim 1, wherein the units of the output layer are grouped, and the optimum unit of each group represents the group,
The learning means is configured so that the squared error of the output value with respect to the teacher value becomes small. According to the present invention, the optimum unit of each group of the units of the output layer is made to represent the group at the time of recognition judgment of unknown data, and the means for recognition recognition is provided. In the multi-layer hierarchical structure type neural network according to claim 3, the units of the output layer are grouped, and the optimum unit of each group represents the group,
Learning means for reducing the squared error of the output value with respect to the teacher value, and means for performing the recognition determination such that the optimum unit of each group of the units in the output layer at the time of recognition determination of unknown data represents that group. It is configured to have.

In claim 4, the units of the output layer are grouped, and the optimum unit of each group, that is, the unit having the value closest to the corresponding teacher data is made to represent that group, and the value of each representative unit is set. 2. The neural network according to claim 1, further comprising a learning unit that makes one of learning end conditions the sum of squared errors of the teacher pattern. In claim 5, when determining the recognition of unknown data, each teacher pattern is compared with a unit in each group, the unit having the closest value is set as the representative unit of the group, and the teacher pattern closest to each representative unit value is recognized. The neural network according to claim 2, wherein the neural network is configured to be used for a determination result. 7. The unit of the output layer according to claim 6, wherein the optimum unit of each group, that is, the unit having the value closest to the corresponding teacher data represents that group, and the value of each representative unit and the teacher pattern Learning means that uses the total value of the squared error as one of the learning end conditions, and at the time of recognition determination of unknown data, each teacher pattern and the unit in each group are compared and the unit with the closest value is set as the representative unit of that group, The neural network according to claim 3, wherein the teacher pattern closest to each representative unit value is used for the recognition determination result.

The operation will be described in detail in the embodiments to be described later. In the neural network, the generalization ability is increased when learning character data and various discrimination data, and the recognition accuracy is increased at the time of recognition determination. Thus, each unit of the output layer of the neural network is grouped. Hereinafter, embodiments of the neural network of the present invention will be described in detail with reference to the drawings.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is an overall configuration diagram of an embodiment of a learning and recognizing device including a neural network of the present invention. 10 is a neural network learning and recognizing device, 1 is a neural network, 2 is a working memory ( A place for temporarily storing what is input or calculated) 3 is an input control unit, 4 is an output control unit, 5 is an input device, and 6 is a device for performing output / display.

In the neural network learning and recognition device 10, when various set values such as a learning pattern and a teacher pattern are input to the input device 5, the input values are stored in the working memory 2 via the input control unit 3. Remembered. Each input value is input to the neural network 1, where learning is performed based on the learning pattern on the working memory, the teacher pattern, and various set values to update the coupling coefficient. The coupling coefficient is stored in the neural network 1, and the learning result is output / displayed via the output control unit 4 to the device 6 for outputting / displaying. In the neural network learning and recognition device 10, if an unknown pattern is input,
As described above, the input value is stored in the working memory 2 via the input control unit 3. The neural network 1 performs recognition or value estimation based on the coupling coefficient and outputs / displays the result via the output control unit 4 to the device 6 for outputting / displaying.

Next, one embodiment of the present invention will be described with reference to FIGS. FIG. 2 is a schematic diagram showing a coupling state by a unit of a neural network including input, intermediate and output layers, and FIG. 3 is an explanatory diagram of an internal state of one unit. FIG. 4 shows an example of a neural network for recognizing numbers, in which a feature quantity of number data is given as an input pattern,
It is a system that discriminates which number based on the output pattern.

In this neural network, the unit of the input layer is 64 units representing the characteristic amount of characters, the unit of the intermediate layer is 8 units connected to the units of each input layer by a coupling coefficient, and the output layer is 1 to 9 and 0. It is made up of 20 units, each of which is designated by 2 units in turn, and consists of 10 groups. In the case of this example, as the teacher pattern, for example, in the case of the number 1, a pattern of 1000000000 is given. This means that the first 1 indicates that group 1 is 1, the next 0 indicates that group 2 is 0, and so on.
Up to 0. Other numbers similarly create teacher patterns. As the value of the teacher pattern of each group, the same value is associated with each unit in the group, and this value is used at the time of learning and recognition determination. When selecting the optimum unit for the total error value and the judgment, for example, the calculated values of the units in the output layer are 0.9, 0.5, 0.3, 0.5, ...
・ If 0.1, 0.2, select a unit with a value of 0.9 for group 1, select a unit of 0.3 for group 2, and select a unit of 0.1 for the final group 0. In this way, it is sufficient if any unit in the group is close to the teacher value, so that the number of learnings can be reduced and the recognition accuracy can be improved.

FIGS. 5 and 6 are flowcharts for explaining the learning means of the neural network, where the numbers of units forming the input, intermediate and output layers are IN, HN and ON. FIG. 7 is a flow chart for explaining the recognition means of the neural network. Hereinafter, the procedure for performing learning will be described using the flowcharts in FIGS. 5 and 6. 101 --- Start processing. 102 ・ ・ ・ Neural network model name setting. The learning pattern number is initialized to 0. Repeat count 0
Initialize to Set the repeat count. Input layer,
The numbers of units in the intermediate layer and the output layer are set to IN, HN, and ON. The value of various parameters such as the target value of the learning end condition is set. 103 ・ ・ ・ Sets all learning patterns and teacher values in working memory 2. Set the number of learning patterns. 104 ... Initializes each coupling coefficient between the input layer and the intermediate layer by a random number or the like. The coupling coefficient between the intermediate layer and the output layer is initialized by random numbers or the like.

105 ... Set a learning pattern in each unit of the input layer. 106 ・ ・ ・ The output value of the intermediate layer unit is calculated by the coupling coefficient between each unit of the input layer and the input layer and the intermediate layer. The output value of the output layer unit is calculated according to the coupling coefficient between each unit of the intermediate layer and the output layer. 107 ・ ・ ・ Calculates the error value between the teacher value and the output value of the output layer unit. Further, the error value of the output layer unit is propagated to the intermediate layer side to calculate the error value of the intermediate layer. 108 ... Updates the coupling coefficient between the intermediate layer and the output layer according to the error value. Similarly, the coupling coefficient between the input layer and the intermediate layer is also updated. 109 ・ ・ ・ Count up the learning pattern number. 110 ・ ・ ・ Determines whether the learning pattern is finished. If yes, go to next, if no go to 105. 111 ・ ・ ・ Determine if the sum of error values is less than the specified value. YE
If S, go to 115, and if NO, go to the next.

112: It is determined whether all the output patterns and the teacher pattern match. Judgment is made by the representative unit of each group. If yes, go to 115; if no, go to next. 113 ・ ・ ・ Count up the repeat count. The learning pattern number is reinitialized to 0. 114 ・ ・ ・ Determines whether the repetition is completed. If yes, go to next, if no go to 105. 115 ... Save the neural network model name. Stores the number of units in the input layer, hidden layer, and output layer.
Save various parameter values. Save the coupling coefficient. 116 ・ ・ ・ End.

Next, the procedure for recognition will be described with reference to the flowchart of FIG. 201 ・ ・ ・ Start processing. 202 ・ ・ ・ Read and set the neural network model name. The number of each unit of the input layer, the intermediate layer, and the output layer is read, and IN, HN, and ON are set. Read and set the values of various parameters. Read and set the learned coupling coefficients. 203 ・ ・ ・ Enter unknown data. 204 ・ ・ ・ Set unknown data in the input layer and calculate the output value of the intermediate layer and the output layer unit by each coupling coefficient value. 205 ・ ・ ・ Calculates the judgment result from the output value of the output layer unit and the judgment condition. Judgment is made by the representative unit of each group. 206 ・ ・ ・ Output and display the judgment result. 207 ... End? If yes, go next, if no, 20
To 3 208 ・ ・ ・ End.

[0020]

As described above, according to the neural network of the present invention, each unit of the output layer is grouped for learning, recognition and estimation of character data, various discrimination data, etc., and the optimum unit is used within the group. Since this is done, higher learning ability and generalization ability than before can be obtained.

[Brief description of drawings]

FIG. 1 is a system configuration diagram of an embodiment adopting a neural network according to the present invention.

FIG. 2 is a schematic diagram of unit connection of a three-layered neural network.

FIG. 3 is an explanatory diagram of the internal state of the unit.

FIG. 4 is an explanatory diagram of an embodiment of a neural network for performing numeral recognition.

FIG. 5 is a flowchart illustrating a learning means of a neural network configured by grouping units in an output layer.

FIG. 6 is a flowchart for explaining learning means of a neural network configured by grouping units in an output layer.

FIG. 7 is a flowchart illustrating a recognition and estimation unit of a neural network configured by grouping output layer units.

[Explanation of symbols]

 1 Neural network 2 Working memory 3 Input control unit 4 Output control unit 5 Input device 6 Output / display device 10 Neural network learning recognition device

Claims (6)

[Claims] 1. In a multi-layered hierarchical neural network, units in an output layer are grouped so that an optimum unit of each group represents the group so that a squared error of an output value with respect to a teacher value becomes small. Neural network that constitutes various learning means. 2. A neural network configured so that an optimum unit of each group of the units in the output layer represents the group at the time of recognition judgment of unknown data, and has a means for recognition judgment. 3. In a multi-layered hierarchical neural network, units in the output layer are grouped so that the optimum unit of each group represents that group so that the square error of the output value with respect to the teacher value becomes small. A neural network configured so that the learning unit and the optimum unit of each group of the units in the output layer at the time of recognition determination of unknown data represent that group, and the recognition determination unit is provided. 4. The units of the output layer are grouped, and the optimum unit of each group, that is, the unit having the value closest to the corresponding teacher data is represented as the group, and the value of each representative unit and the teacher pattern. The neural network according to claim 1, further comprising a learning unit that makes one of the learning end conditions the sum of squared errors of 5. When recognizing unknown data, each teacher pattern is compared with a unit in each group, the unit having the closest value is set as a representative unit of the group, and the teacher pattern closest to each representative unit value is recognized. The neural network according to claim 2, which is configured to be used for a determination result. 6. The units in the output layer are grouped, and the optimum unit of each group, that is, the unit having the value closest to the corresponding teacher data represents that group, and the value of each representative unit and the teacher pattern Learning means that uses the total value of the squared error as one of the learning end conditions, and at the time of recognition determination of unknown data, each teacher pattern and the unit in each group are compared and the unit with the closest value is set as the representative unit of that group, The neural network according to claim 3, wherein the teacher pattern closest to each representative unit value is used as a recognition determination result.