JPH05197821A - Method for optimizing hierarchical neural network and device therefor - Google Patents

Abstract

PURPOSE:To provide a hierarchical neural network optimizing method which can approximate the number and the weight of neurons arranged in an intermedi ate class to each optimum value. CONSTITUTION:It is judged whether the optimum solution is converged or not (learning state), whether a local solution is caused or not, and whether the number of neurons is proper or not in regard of a neural network which is learnt by the prescribed frequency with the detection of the changed variable of the error square sum and the correct answer rate and its changed variable. The neurons are added to an intermediate class as necessary, the defective neurons are deleted away, or the weight of the defective neurons is initia1ized. Then the neural. network is learnt again. Thus the optimum number of neurons is set to the intermediate class during the learning and at the same time an optimum solution is obtained. The defective neurons have the low affecting force to the analysis of the input data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for optimizing a hierarchical neural network, and more particularly to optimizing the structure of an intermediate layer in a hierarchical neural network.

[0002]

2. Description of the Related Art A hierarchical neural network is composed of an input layer, an intermediate layer (consisting of a plurality of layers as necessary) and an output layer. Each layer is provided with a predetermined number of neurons, Neurons are connected by synapses, and a predetermined coupling coefficient (hereinafter, simply referred to as “weight” given to neurons) is given to each synapse. Then, in order to set the weight to an appropriate value, the neural network itself performs self-learning.

An example of this self-learning is called back propagation. This method first initializes the weights given to each synapse with random numbers,
In that state, input data in the input layer is input, and from the input layer to the output layer are sequentially calculated based on the input data to obtain output data. Then, compare the output data with the expected output data that should be obtained by the input data,
The difference between the two is obtained, and the weight is changed so that this difference becomes smaller. Moreover, during this learning process, the number of neurons arranged in each layer is fixed.

[0004]

By the way, if the number of neurons installed in the intermediate layer is small, the feature quantity space of the input data cannot be properly divided, and even if the above learning processing is performed, it is sufficient. I / O relationship cannot be obtained. On the other hand, when the number of neurons increases, the correct input / output relationship can be obtained, but when it is actually built in the system, not only the memory capacity becomes huge, but also the processing time becomes long and it cannot be put to practical use. .. Moreover, the optimal number of neurons cannot be known in advance. Therefore, even if the learning process is performed, the structure of the neural network in the intermediate layer cannot be optimized because the number of neurons is not optimal. Further, as shown in FIG. 19, in the hierarchical neural network, the input / output relationship obtained by learning becomes a local solution, and there is a possibility that an optimal solution cannot be obtained. Here, the optimum solution is a value of a weight that minimizes the error sum of squares on the error curved surface, and the local solution is that the sum of error squares (amount of change) locally decreases on the error curved surface (the optimum solution). Not) indicates the weight value.

The present invention has been made in view of the above background, and an object of the present invention is to make an optimum neural network by making both the number of neurons arranged in the intermediate layer and the weights close to the optimum values. It is an object of the present invention to provide a method and an apparatus for optimizing a hierarchical neural network capable of obtaining the structure of.

[0006]

In order to achieve the above-mentioned object, in the optimization method of a hierarchical neural network according to the present invention, an intermediate process is performed while a predetermined learning is being performed on the hierarchical neural network. It is determined whether the weight given between the neurons in the layer falls into the local solution, and a new neuron is added to a predetermined position in the intermediate layer when the weight falls into the local solution.

Further, during the predetermined learning for the hierarchical neural network, a defective neuron evaluation value representing the degree of contribution to the recognition of the input data is obtained for each neuron forming the intermediate layer, The defective neuron may be determined based on the evaluation value, the defective neuron may be deleted at a predetermined timing, or the weight assigned to the defective neuron may be initialized, and then the learning may be continued.

Further, the above methods may be combined to appropriately add or delete neurons to or from the intermediate layer, and further, after the once deleted defective neuron is restored, learning may be continued. good.

Further, in the optimizing apparatus according to the present invention suitable for carrying out the above method, a neural network learning apparatus for learning a hierarchical neural network and learning situation data output from the neural network learning apparatus. A device for determining whether or not the weight given to the neurons in the intermediate layer falls into a local solution based on the above, and a new neuron is added to the intermediate layer based on a control signal sent from the determining device. And a device for adding an intermediate layer neuron.

A neural network learning device for learning a hierarchical neural network and recognition of input data for each neuron forming an intermediate layer based on learning status data output from the neural network learning device. A defective neuron evaluation value calculation device that calculates a defective neuron evaluation value that indicates the degree of contribution, and whether learning is close to an optimal solution is determined based on the learning situation data output from the neural network learning device. A device, a control signal sent from the device for judging the device, and an intermediate layer neuron reduction device for deleting the defective neuron in the intermediate layer based on the evaluation value data of the defective neuron sent from the defective neuron evaluation value calculation device. May be provided.

Further, a neural network learning device for learning a hierarchical neural network and recognition of input data for each neuron forming an intermediate layer based on the learning situation data output from the neural network learning device. A defective neuron evaluation value calculation device that calculates a defective neuron evaluation value that indicates the degree of contribution, and the weights assigned between neurons in the intermediate layer based on the learning situation data output from the neural network learning device A device that determines whether or not a solution is present, a control signal that is sent from the device that determines whether the solution is in solution, and a defective neuron evaluation value data that is sent from the defective neuron evaluation value calculation device. An intermediate layer neuron weight initialization device that initializes the weight of the neuron may be provided. And, preferably, the above-mentioned respective devices are appropriately combined and configured.

Further, in addition to the hierarchical neural network optimizing device having the above-mentioned various configurations, a storage means for storing output error weight data set based on a given required specification is provided, and the various determining devices are provided. However, the various determinations may be made in consideration of the output error weight data stored in the storage means.

When storage means for storing the output error weight data is provided, output error weight adjusting means for further correcting the output error weight data is further provided, and the judging means is provided with the output error weight data. It is preferable to add a function of determining whether or not to correct. When the means for storing the output error weight data and the adjusting means for correcting the data are provided as described above, the optimization can be performed without providing the neuron reduction / addition / weight initialization means in the intermediate layer. Can be planned.

[0014]

In the present invention, the learning situation (effect) is monitored during the learning of the hierarchical neural network, and if it is determined that the local solution is present, a neuron is added.
Then, the feature space can be divided into more curved surfaces, the recognition rate is improved, and the optimal solution is obtained by escaping from the local solution.

On the other hand, the defective neuron is the next stage (output side).
Influence on the input data (contribution to recognition of input data) is low. Therefore, when it is determined that the learning situation falls into a local solution, there is a possibility that the defective neuron may escape from the local solution due to its greater influence on the next stage.
Therefore, the weight of the defective neuron having low influence is reinitialized and learning is continued. On the other hand, when the learning situation is judged and it is judged that the optimum solution is reached, it can be considered that the correct input / output relationship can be maintained even without such a defective neuron. Therefore, the defective neuron is deleted and learning is continued. Then, if the state of the optimum solution can be maintained even after the deletion, the number of neurons forming the intermediate layer is reduced and high-speed processing can be performed. If the optimal solution cannot be obtained as a result of deleting the defective neuron, the optimal solution is obtained by restoring the deleted defective neuron or adding a new neuron.

Further, by performing various evaluations on the learned neural network in consideration of the output error weight data, it is possible to reach an optimum solution that satisfies the required specifications at a higher speed. When the output error weight adjusting means is provided, the initially set output error weight data is corrected according to the learning situation, and the corrected output error weight data is also subjected to various status evaluations. Therefore, learning is completed at a higher speed.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A preferred embodiment of a hierarchical neural network optimization method and apparatus according to the present invention will be described in detail with reference to the accompanying drawings. First, an outline of one embodiment of the present invention will be described. For example, during learning using back propagation or the like, defective neurons that do not contribute to detection are deleted from neurons forming the intermediate layer. Or add new neurons to set the optimal number of neurons, and if necessary, re-initialize bad neuron weights to prevent each weight from falling into a local solution. I am trying to do it. And
In this example, as a detection factor for performing such control,
The sum of error squares, the amount of change in the sum of error squares, the percentage of correct answers, and the amount of change in the percentage of correct answers are used.

The specific circuit configuration for executing them is as shown in FIG. That is,
This drawing shows the first embodiment of the present invention. As shown in the drawing, the outputs of the learning database 12, the weight database 14, and the intermediate layer optimization parameter database 16 are connected to the neural network learning device 10, respectively.

The learning database 12 records and holds, as learning data for the hierarchical neural network, input data to be provided to the input layer of the hierarchical neural network and expected output data to be provided to the output layer. The specific data structures are as shown in FIGS. 2A and 2B, respectively. Further, the weight database 14 stores and holds weight data representing a connection state from one neuron of the hierarchical neural network to another neuron, and neural net structure data (the number of neurons in each layer), and concrete data The structure is shown in FIG.
It is as shown in (B). Further, the intermediate layer optimization parameter database 16 stores various parameters necessary for optimizing the intermediate layer of the hierarchical neural network, and the specific data structure is as shown in FIG. Each threshold value and the like are registered in advance as fixed data. Then, the neural network learning device 10 receives each data from each of the databases 12, 14 and 16 to form a hierarchical neural network, and uses the back propagation algorithm to perform a predetermined number of times indicated by the optimization learning interval. Only to learn.

Here, in the present invention, the neural network learning device 10 and the databases 12, 14, 16 are described.
A hierarchical neural network optimization device 20 is connected to the. This hierarchical neural network optimization device 20
Error square sum and change amount calculation device 21, correct answer rate and change amount calculation device 22, defective neuron evaluation value calculation device 2
3, an intermediate layer neuron reduction device 24, an intermediate layer neuron addition device 25, an intermediate layer neuron weight initialization device 26, and an intermediate layer neuron optimization determination device 27. Then, the respective devices 21 to 27, the neural network learning device 10 and the respective databases 12, 1
The specific connection configuration with 4, 16 is as follows.

That is, the neural network learning device 10
Output of error sum of squares and change amount calculation device 21, correct answer rate and change amount calculation device 22, bad neuron evaluation value calculation device 23, hidden layer neuron reduction device 24, hidden layer neuron addition device 25, and hidden layer neurons It is input to the weight initialization device 26.

The learning database 12 outputs the sum of squared error and its variation calculating device 21, the correct answer rate and its variation calculating device 22, and the defective neuron evaluation value calculating device 2.
It is entered in 3.

The outputs of the error sum of squares and its variation calculating device 21 and the correct answer rate and its variation calculating device 22 are
It is input to the intermediate layer neuron optimization determination device 27. Further, the outputs of the bad neuron evaluation calculation device 23, the hidden layer neuron optimization determination device 27, and the hidden layer optimization parameter database 16 are the output of the hidden layer neuron reduction device 24, the hidden layer neuron addition device 25, and the hidden layer neuron weight initialization device. 26 respectively. The outputs of the respective devices 24, 25 and 26 are the neural network learning device 10 and the weight database 14.
Has been entered in.

Further, the output of the intermediate layer optimization parameter database 16 is the intermediate layer neuron optimization determining device 2
7, and the output of the intermediate layer neuron optimization determining device 27 is input to the neural network learning device 10.

Next, the function of each of the devices 21 to 27 constituting the hierarchical neural network optimizing device 20 will be described. * Sum of squared error and its variation calculation device First, this device 21 causes the neural network learning device 10 to perform a predetermined number of times of learning, and then the weighting data and neural network structure data at that time are obtained from the neural network learning device 10. At the same time, the learning data (input data, expected output data) is received from the learning database 12. Then, a hierarchical neural net is formed from the weight data and the neural net structure data, and the recall process is performed. Here, the recollection process is a calculation process of recollection from the input layer to the output layer of the hierarchical neural network by using the weights after learning by giving the input data to the neurons of the input layer of the hierarchical neural network. And obtain output data from the neurons in the output layer of the hierarchical neural network.

Then, using the obtained output data and the expected output data, the error square sum (E1) is calculated and held, and the calculated error square sum (E0) is obtained. The percentage of the absolute value of the difference is the error squared variation (d
Calculate as E). Then, it serves to supply the obtained error square sum and change amount to the intermediate layer neuron optimization determination device 27.

Here, the calculation formulas for obtaining the above respective values are shown below.

[0028]

[Formula 1] <Error sum of squares> <Amount of change in error sum of squares> dE = {| E0-E1 | * 100} / E0 where ptn: total number of patterns, pe3: number of output layer neurons, L:
Total number of layers, di [j]: Expected output for j-th pattern output layer i-th neuron yi [L]: Output value of neuron on output layer i-th surface.

Correct Answer Rate and Change Amount Calculation Device This device 22 receives predetermined data from the neural network learning device 10 and the learning database 12 like the above error square sum and change amount calculation device 21. At the same time, a correct answer rate (C0) is calculated and held by forming a hierarchical neural network and comparing the output data obtained by performing the recall process with the expected output data, and further calculating the value thereof, and calculating the value last time. Obtained correct answer rate (C
The percentage of the absolute value of the difference from 0) is calculated as the correct answer rate change amount (dC). Then, the obtained correct answer rate and the obtained variation amount are supplied to the intermediate layer neuron optimization determination device 27.

Here, the calculation formulas for obtaining the above respective values are shown below. That is, first, for all patterns ptn, * y [L] (output data) and * d
[P] (expected output data) is compared, and if both are the same, the pattern p is taught as a correct answer pattern.

[0031]

## EQU00002 ## where * y [L] = (y1 [L], y2
[L], ..., y (peL) [L]) * d [p] = (d1 [p], d2 [p], ..., d (peL
) [P]).

Then, the number of correct answer patterns at that time is ptn1,
If the total number of patterns is ptn, the correct answer rate C0 is obtained by the following equation.

C0 = ptn1 / ptn The correct answer rate change amount dC is obtained from dC = C0-C1.

* Defective Neuron Evaluation Value Calculating Device This device 23 calculates, for each neuron constituting the intermediate layer,
The defective neuron evaluation value is obtained, and the defective neuron evaluation value in the present invention means the degree of contribution to pattern recognition. Then, it can be said that the neuron having such a low degree of contribution has a small influence on the output layer. Therefore, (1) Even if the number of neurons having a low evaluation value is reduced, it is unlikely that the ability of the neural network is impaired, and it can be considered that correct output data can be obtained. (2) Further, when the neural network falls into a local solution during learning, it is possible to escape from the local solution and find the optimum solution by continuing the learning after initializing the weights of the bad neurons. That is, the present device 23 obtains a defective neuron evaluation value as a pre-process for obtaining the effects (1) and (2), and the specific means is as follows.

Similar to the above-mentioned devices 21 and 22, after the hierarchical neural network is learned a predetermined number of times, predetermined data is received from the neural network learning device 10 and the learning database 12, and a hierarchical neural network is formed. To do. Then, in this apparatus 23, the recalling process is performed on the input layer and the intermediate layer, and the defective neuron evaluation value evalm [i] of the m-th layer i-th neuron is obtained by the following formula.

[0036]

[Equation 3] Here, ptn: total number of patterns pe (m + 1): number of neurons in m + 1 layer wij [m]: neuron in m-th layer i-th layer and m + 1-th layer j
Weight with respect to the n-th neuron yi [m]: The output value of the neuron on the m-th layer and the i-th surface.

Then, the defective neuron evaluation value of each neuron forming the intermediate layer obtained by the above equation is supplied to the intermediate layer neuron reduction device 24 and the intermediate layer neuron weight initialization device 26.

The evaluation value eval obtained by the above equation
The smaller m [i], the smaller the absolute value of the weight of the upper layer of the i-th neuron in the intermediate layer, or the intermediate layer i
This means that the output of the numbered neuron is small. Therefore, by performing a predetermined process on the neuron having a low evaluation value, the above effect can be expected and the neural network can be optimized. The defective neuron evaluation value obtained by the above equation is stored and held as a data structure as shown in FIG.

* Intermediate Layer Neuron Reduction Device This device 24 reduces predetermined neurons based on the signal sent from the intermediate layer neuron optimization decision device 27. The specific processing is as follows. ing.

First, the defective neuron evaluation value calculation device 23
The poorer neuron evaluation value is received, and the optimization parameter is received from the hidden layer optimization parameter database 16. And among the optimization parameters,
The number of defective neurons in each hidden layer is reduced by "the number of defective neurons to be reduced". The numerical value to be reduced is stored (del pe num [m]) in the intermediate layer optimization parameter D / B 16 as described above. Further, the selection index of the neurons to be reduced is reduced for each layer in ascending order of the bad neuron evaluation value.

Here, the reduction of neurons means that the weights connected to the neurons to be deleted are deleted from the weight data stored in the weight database 14 and each layer stored in the neural network structure data is deleted. Perform processing to reduce the number of neurons in. Then, as a specific example, for example, when deleting the m-th layer j-th neuron, the following data is deleted from the weight data.

[0042]

[Mathematical formula-see original document] w1j [m-1], w2j [m-1], ..., w (pe (m-
1)) j [m-1] wj1 [m], wj2 [m], ..., wj (pe (m + 1)) [m] Also, if only del pe num [m] are reduced, the m-th layer Let pem = pem-del pe num [m] be the number of neurons pem in.

After performing the above-mentioned reduction processing, all the weight data and the neural network structure data are set to the neural network learning device 10 and the weight database 14.
Send to. As a method of the above deletion, the corresponding defective neuron may be physically deleted, or the weight of the defective neuron may be set to "zero" to eliminate the influence on the output side. ..

* Intermediate layer neuron adding device This device 25 adds a predetermined neuron based on the signal sent from the intermediate layer neuron optimization judging device 27. The specific processing is as follows. ing.

Intermediate layer optimization parameter database 16
Receives optimization parameters from. Then, among the optimization parameters, neurons are added to each intermediate layer by the number of neurons to be added. Here, the addition of a neuron is performed by newly installing a neuron, adding a weight that is coupled to the added neuron to the weight data, and increasing the number of neurons in each layer of the neural network structure data by the additional amount. And the number to add is
Store in the middle layer optimization parameter D / B16 (add pen n
um [m]) has been. Moreover, the value of the added weight is determined by a random number. As a specific example, for example, when the j-th neuron is added to the m-th layer, the following data is added to the weight data.

[0046]

[Mathematical formula-see original document] w1j [m-1], w2j [m-1], ..., w (pe (m-
1)) j [m-1] wj1 [m], wj2 [m], ..., wj (pe (m + 1)) [m] When only add pe num [m] are added, the m-th layer Let pem = pem + add pe num [m] be the number of neurons pem in.

After performing the above-mentioned additional processing, all the weight data and the neural network structure data are set to the neural network learning device 10 and the weight database 14.
Send to. In this way, by adding neurons to the intermediate layer, it becomes possible to divide the feature quantity space by more curved surfaces, and the recognition rate is improved.

* Intermediate layer neuron weight initialization device This device 26 initializes the weight of a predetermined defective neuron on the basis of the signal sent from the intermediate layer neuron optimization determination device 27. Is as follows.

That is, the defective neuron evaluation value calculation device 23 receives the defective neuron evaluation value and the optimization parameter from the hidden layer optimization parameter database 16. Then, among the optimization parameters, the neuron weights are initialized in each intermediate layer by the “number of neurons to be initialized” (new initial values are set by random numbers).
Then, as the selection index of the neuron to be initialized at this time, the weight of the neuron is initialized in ascending order of the bad neuron evaluation value. As a specific example, for example, when the j-th neuron is initialized in the m-th layer, the following weight values are re-assigned with random numbers.

[0050]

[Expression 6] w1j [m-1], w2j [m-1], ..., w (pe (m-
1)) j [m-1] wj1 [m], wj2 [m], ..., wj (pe (m + 1)) [m] Then, after performing the above initialization processing, all weight data and neural The net structure data is sent to the neural network learning device 10 and the weight database 14.

Another function of the intermediate layer neuron weight initialization device 26 is to initialize the weight of the defective neuron reduced by the intermediate layer neuron reduction device 24 to "0". A function of restoring a defective neuron by re-establishing it may be added.

As an initialization method, in addition to the setting by the random number as described above, for example, a specific value (can be large or small) can be used as the initial value weight data and set to the value. However, various means can be taken.
In other words, the point is that the initialization method can reduce the possibility that the weight after initialization and the weight when it is determined as a bad neuron before initialization are less likely to match (probability Is desirable, and it is desirable that the weight after initialization be extremely different from the weight before initialization).

* Intermediate Layer Neuron Optimization Judgment Device In the present device 27, the error square sum and its change amount calculation device 21
Also, various data are received from the correct answer rate and its change amount calculation device 22. Then, the given sum of squared errors, error 2
The state of the hierarchical neural network, that is, the learning proficiency (approaching the optimal solution) based on the amount of change in the sum of sums, the correct answer rate, the amount of change in the correct answer rate, and the number of times the defective neurons are initialized. Level) and whether or not a local solution has fallen into the learning state, the number of neurons in the intermediate layer is optimized, and each weight is optimized in order to optimize each weight. A predetermined command signal is sent to the intermediate layer neuron adding device 25 and the intermediate layer neuron weight initializing device 26. Also,
It also has a function of judging whether or not the learning is sufficient and sending a learning continuation signal to the neural network learning layer 10.

In this embodiment, the hidden layer neuron optimization judging device 27, the above-mentioned error sum of squares and its change amount calculating device 21, and the correctness rate and its change amount calculating device 22.
A device for determining whether or not the weight given to the neurons in the intermediate layer falls into a local solution based on the learning situation data output from the neural network learning device according to the present invention, and learning is an optimal solution. And a device for determining the learning state.

Next, the neural network optimizing method according to the present invention using the above apparatus will be described with reference to FIGS. 6 to 7. First, the weights of all neurons are initialized, the number of resets (the number of initializations) is set to 0, and the correct answer rate C1 and the error sum of squares E0 are calculated as initial data (S101 to 104). If the neural network learning device 10 has learned a predetermined number of times (S105, 106), the error sum of squares E1 is obtained (S107), and the error sum of squares E and the initial data is calculated.
The change amount dE of the sum of squared errors is calculated from 0 (S10
8) It is determined whether or not the value dE is smaller than a predetermined value (threshold value (given as an optimization parameter)) (S109). In this example, the determination formula is “dE <threshold”, but it is of course possible to set “dE ≦ threshold”. Similarly, “=” is included when comparing various data with each threshold. You may do it.

When the amount of change dE is large, the weight of the neural network is dynamically changing, which means that the weight is not close to the optimal solution (not converged), and learning is insufficient. Can be judged. Therefore, the process returns to step 104 to continue learning again. Note that the value of the error sum of squares E1 obtained in step 107 is the value in step 10
At 4, the error sum of squares E0 to be compared in the next determination is obtained.

On the contrary, when the change amount dE is small, it can be judged that the weight of the neural network is approaching the convergence, and therefore the defective neuron evaluation value is calculated (S1).
Along with 10), the correct answer rate C0 is calculated (S111), and it is determined whether or not the value C0 is smaller than a predetermined value (threshold value (given as an optimization parameter)) (S11).
2).

When the correct answer rate C0 is large, it can be determined that the solution is close to the optimum solution, and therefore it can be determined that the current number of neurons in the intermediate layer is sufficient (no need to add neurons). Therefore, in order to test whether the optimal number of neurons is obtained at least, the learning is continued after deleting some defective neurons having a low evaluation value at that time. Specifically, the correct answer rate C0 at that time is replaced with C1, and the weight data of all neurons is stored in the weight database 14 (S1).
13, 114). Then, after deleting the predetermined defective neuron and setting the reset count R to 0 (S11
5, 116) and returns to step 104 to repeat the above operation.

On the other hand, when the correct answer rate C0 is small, there is a possibility that learning is insufficient or there is a local solution. Therefore, the change rate of the correct answer rate is calculated, and this is calculated as a predetermined value (threshold (threshold ( (Given as optimization parameter))) is smaller than (S117). When the amount of change is large, it can be determined that the learning is insufficient, and the correct answer rate may be increased by further learning. Therefore, the correct answer rate C0 at that time is replaced with C1 (S118), and then step 10 is performed.
Return to 4 to continue learning. If the amount of change is small, it can be determined that a local solution has fallen. Therefore, in order to break out from that state and obtain an optimal solution, the weight of the defective neuron with a low evaluation value is initialized or a new one is added. Add to the middle of the neuron. Then, which means to take is determined depending on the number of resets.

That is, after the correct answer rate C0 at that time is replaced with C1 (S119), it is determined whether or not the reset number is smaller than a predetermined value (threshold value (given as an optimization parameter)) (S120). ), If it is small (the number of resets is small), the weight data of all the neurons at that time may be stored in the weight database 14 because there is a possibility that the defective neuron may be initialized to escape from the local solution. (S121), the weight of a predetermined defective neuron is initialized, and the reset count R is set to 1
After adding (S122, 123), the process returns to step 104 and the above operation is repeated.

On the other hand, if the number of resets is large, it is unlikely that the defective solution can be escaped from the local solution even if the defective neurons are initialized again. Add neurons. That is, the weight data of all the neurons at that time are stored in the weight database 14 (S124), a predetermined number of neurons are added to the intermediate layer, the weights of the added neurons are initialized (S125, 126), and then the number of resets is performed. After setting R to 0 (S127), step 10
Returning to step 4, the above operation is repeated. In this example, as a mode of this addition, not only a new neuron is added but also the weight of the defective neuron once deleted in step 115 is initialized to restore the neuron. Including those that cause

Then, the above-mentioned respective processes are repeated,
If it is determined in step 106 that the number of neurons and the weight data are sufficient, the neural network learning device 10
Finish the study in. Note that S109 and S in FIG.
112 to S119 and the flow portion shown in FIG. 7 show the functions of the intermediate layer neuron optimization determination device 27.

Further, in the above-mentioned embodiment, the function of the hidden layer neuron optimization judging device 27 is as shown in a part of FIG. 6 and the flow shown in FIG. 7, but instead of this, for example, as shown in FIG. It may be a device having a function.
That is, in the first embodiment described above, the change amount dE of the error sum of squares, the correct answer rate C0, and the change amount dC of the correct answer rate thereof are used as the determination criteria in the determining device 27 as to whether or not the created intermediate layer neuron is optimal. However, in this example, in addition to that,
First, the error sum of squares E1 was also added to the criterion.

Further, the thresholds to be compared with these four judgment criteria are also the same in the first embodiment, regardless of the types of subsequent processing (addition / deletion of neurons, weight initialization). In this example, reference values for deletion and other reference values are provided. That is, in the first embodiment, the threshold value of the error change amount is one “th d sq err”, but the two threshold values “dEth +” and “dEth−” and the threshold value of the correct answer rate are “th right”. "One was" Cth
"Th d right" was one of the thresholds for the change rate of the correct answer rate to "+" and "Cth-", but it was set to two of "dCth +" and "dCth-". New error 2
Two thresholds for sum of multiplication, "Eth +" and "Eth-", have been added. The respective reference values are stored in the intermediate layer optimization parameter D / B16 as shown in FIG. With this configuration, in this example, as compared with the first example,
It has become possible to obtain the optimal solution faster and more reliably. The processing steps shown in parentheses in FIG. 9 are the intermediate layer neuron reduction device 24, the intermediate layer neuron addition device 25, and the intermediate layer neuron weight that actually perform the processing determined by the intermediate layer neuron optimization determination device 27. This is performed by each device of the initialization device 26.

FIG. 11 shows a third embodiment of the present invention. This embodiment is further improved on the basis of the above-mentioned second embodiment (one in which the determination device 27 has four determination parameters and two threshold values for each parameter are set according to the processing performed after determination). It is a thing.

First, the concept of this embodiment will be explained. As the separating curved surface of the input / output pattern becomes more complicated, the number of neurons in the intermediate layer constituting the hierarchical NN becomes larger, and the learning speed and the recall speed are inevitably slower. turn into. By the way, for example, in a 2-input 1-output system, if the input / output pattern is as shown in FIG. 12A, the separation curved surface is K1. At this time, the output specifications (“○” or “×” in this example) have a priority order as required specifications, that is, for example, an area (necessity area) that should not be erroneously discriminated and a possibility that it exists (even if erroneous discrimination is made). If there is a region (possibility region) with "good", each of the separating curved surfaces K2 and K3 will be as shown in FIG. Therefore, the learning / remembering speed is increased by performing the learning with the possibility / necessity area being input.

In practice, the concept of output error weighting is provided as a coefficient for multiplying the output error, and the output error weighting is increased when the attribute of the output having a high priority is set, and the output error weighting of the attribute of the output having a low priority is set. On the contrary, by decreasing the output error weight, the effect of the attribute of the output with high priority on the recognition of the neural network is increased, and by learning in that state, the output with high priority can be surely recognized. become.

As a concrete example, if the neural network is used in an inspection device for discriminating (recognizing) a non-defective product or a defective product, for example, it is not recognized that the defective product is erroneously discriminated. However, since it is permissible to misidentify a non-defective product as a defect to some extent, the recognition rate of a defective product is 100.
%, But a required specification such as a recognition rate of a non-defective product is 99%, for example. In such a case, the output error weight is set small (for example, 0.8) for the learning data of the good product category, and the output error weight is set large (for example, 1.0) for the learning data of the defective product category. You should take. In this example, the maximum value of the output error weight is 1.0
Then, such a value is used as an initial value.

Here, in order to execute the concept of the output error weight based on the above priority, in this example, an output error weight database 18 storing the predetermined output error weight data OW is provided as shown in FIG. The output error weight data OW is taken into consideration for various evaluations, and in this example, an output error weight adjusting device 28 for correcting the weight data OW stored in the database 18 is provided. The output error weight data has a data structure as shown in FIG. Since the output error weight data OW is provided in this way, the optimization processing parameter P is also set to the number of resets of the output error weight as compared with that of the second embodiment as shown in FIG. 13B. The threshold value Nowth and the output error weight correction amount dOW are added. Further, dEth + and dCth + also serve as respective threshold values when adjusting the output error weight.

In this example, the output error weight correction process is added in addition to various corrections (deletion of defective neuron, addition of new neuron, etc.) associated with the learning result in each of the above-described examples. The function of the intermediate layer neuron optimization determination device 27 in the example needs to have a determination function of whether or not to correct the output error weight. Therefore, as shown in FIG. 11, an optimization processing selection device 29 is provided in place of the intermediate layer neuron optimization determination device 27 in the first and second embodiments, and the output of the optimization processing selection device 29 is set to the above output error. It is input to the weight adjusting device 28 and the defective neuron evaluation value calculating device 23 ', and the output of the defective neuron evaluation value calculating device 23' is deleted, added, and weight initialization device 24 ', 25', 26 'for each intermediate layer neuron. Is connected to. The connection method is shown in Fig. 1.
Similar to the one shown in FIG. 5, the output of the optimization processing selection device 29 is connected to the above devices 24 ', 25', 26 ', and the defective neuron evaluation value calculation device 23' is directly specified by the learning device 10 'or the like. Even if data is given, the same effect can be obtained.

Here, the optimization processing selection device 29 will be described. The hierarchical neural network weight data W output from the neural network learning device 10 ', and the error square obtained by the various calculation devices 21' and 22 '. The sum E, the correct answer rate C, and the variations dE and dC thereof (specifically, the form of the data structure shown in FIG. 14) are read, and the optimization parameter P is read from the optimization processing parameter database 16 '. Then, the processing method for optimizing is determined from the given data, and specifically, the determination processing is performed according to the flowchart shown in FIG.
The next process (device to be executed) is determined. The data regarding the correct answer rate, such as the calculated correct answer rate change amount dC, has the same data structure (matrix) as in FIG.
In this FIG. 15 as well, as in the case of FIG. 9 described above, the processing indicated by the curly braces is actually performed by the various devices (24'-26 ', 28) in the next stage.

If it is determined that the output error weight is to be corrected, the output error weight adjusting device 28 is selected. If it is determined that another correction is to be performed, the defective neuron evaluation value calculating device 23 'is selected. At the same time, predetermined data is sent. Then, the defective neuron evaluation value calculation device 23 'performs the same processing as in the first embodiment, obtains the evaluation value of each neuron, and according to the determined contents, a predetermined device (24'-26) Each of the devices 2'is configured to send predetermined data together with the evaluation value to any one of ‘
The specific processing contents in 4'to 26 'are the same as those in the above-mentioned embodiment, and therefore their explanations are omitted.

On the other hand, the output error weight adjusting device 28, when the optimization process selecting device 29 determines to correct the output error weight, as described above, the device 29 adjusts the optimization parameter P and the neural network weight data. Read W,
The output error weight data OW is read from the output error weight database 18 ', the output error weight data OW is corrected based on the data, and the corrected contents are stored / stored in the database 28. It looks like.

That is, basically, the output error weight for the output satisfying the required specifications is corrected so as to be reduced. That is, according to the following formula, the output error weight corresponding to the left side is subtracted by the correction amount dOW of the output error weight stored in the optimization parameter P.

If c lth + (i) ≤ C l (i) & dOW ≤ ow l (i) then ow l (i) = ow l (i) − dOW if c hth + (i) ≤ C h (i) & dOW ≤ ow h (i) then ow h (i) = ow h (i) -dOW where "cl (i)" is the i-th correct answer rate when di [j] = 0.0, and "ch (i) ”is the i-th correct answer rate when di [j] = 1.0. Similarly, "c lth + (i)" is di [j] = 0.
It is the i-th correct answer threshold value (required specification) when 0,
“C hth + (i)” is the threshold value (required specification) of the i-th correct answer rate when di [j] = 1.0 (both at the time of neuron weight reset / addition / output error weight correction). Also, "ow h
(i) ”is the i-th output error weight when di [j] = 1.0. "Ow l (i)" is the i-th output error weight when di [j] = 0.0, and "ow h (i)" is the i-th output error weight when di [j] = 1.0. It is a weight. Note that i = 1,2, ..., pe3
Here, the order of output layer neurons is shown (peL is the number of neurons in the output layer (Lth layer)), and di [j] is expected output data for the j-th pattern output layer and the i-th neuron.

With this configuration, the output error weight is reduced for outputs that satisfy the required specifications, that is, for outputs that can be recognized correctly by the neural nets learned up to that point. , In the subsequent learning, the learning result is faster, and the optimal solution is obtained by focusing the learning on the pair of the input data for the teacher and the expected output data that do not satisfy the required specifications be able to.

The rest of the configuration and operation are the same as those of the above-mentioned embodiments, so the description thereof will be omitted. Then, FIG. 16 shows a schematic processing flow of the operation of the entire apparatus in the present embodiment.

FIG. 17 shows a fourth embodiment of the present invention. This example is one of the simplest configurations of the present invention, and is obtained by multiplying the output error by the output error weight data using the concept of the possibility / necessity region as in the case of the third embodiment. The output error weight data adjusting device 28 'corrects the output error weight data according to the learning result, based on the evaluation of the learning status based on the calculation result.
The learning performed next is evaluated based on the corrected output error weight data. And, unlike the above-mentioned respective embodiments, various devices for performing the addition / reduction processing of the intermediate layer neurons and the weight reset processing (the above 24 to 24).
26 and 24 'to 26') are not provided.

Since no processing device is installed for the middle-layer neurons, the output error weight data is only corrected as an optimization process that is performed as necessary after the predetermined learning and before the next learning. The function of the intermediate layer neuron optimization determining device 27 'is also simplified as shown in FIG. With such a configuration, although its function is lower than that of the above-described third embodiment such as addition / deletion of hidden layer neurons, since it has a function of correcting the output error weight data, the output error during learning is The weights can be automatically determined, and the learning speed is sufficiently high compared to the one in which the possibility / necessity area is simply set.

The above-described embodiments are merely examples of the present invention, and the present invention is not limited to this. For example, it is necessary to judge whether or not the number of neurons should be optimized while performing learning processing. It is also possible to simply delete the bad neuron according to. On the contrary, if it is determined that the number of neurons in the intermediate layer is insufficient during learning, a new neuron may be added to the intermediate layer. Moreover,
Instead of the above-mentioned additional work, or the defective neuron once deleted may be restored together with it, and in the present invention, an arbitrary step can be appropriately extracted and combined to optimize the hierarchical neural network.

When the desired processing is performed as described above, the components of the optimizing device used accordingly (the defective neuron evaluation value calculating device 23, the intermediate layer neuron reducing device 24, the intermediate layer neuron adding device). 25 and the intermediate layer neuron weight initialization device 26, etc.), the configuration of the optimization device may be configured by appropriately combining other components except the configuration unnecessary for processing.

Further, in the configuration shown in the third embodiment (FIG. 11), for example, the output error weight adjusting device is omitted and various evaluations are performed using the output error weight data OW stored in the output error weight database 18. The predetermined processing (addition, reduction, weight initialization) for the intermediate layer neuron to be performed next may be determined. Further, in this case, as in the above-described modified examples, it is not necessary to further provide a processing device for performing one or two of the three processes performed on the intermediate layer neurons.

Furthermore, in the third embodiment, an example in which the output error weight data adjusting device is installed on the device having the structure of the second embodiment as a reference has been described. May be installed together with the output error weight database, and various changes can be made.

Further, in the above-mentioned embodiment, an example in which back propagation is used as the learning method of the neural network has been described, but any learning method can be used. Further, as the method of extracting defective neurons, in the above-described embodiment, a predetermined number of defective neuron evaluation values are extracted in order, but the present invention is not limited to this, and, for example, Various methods can be adopted, such as determining whether or not the value exceeds a predetermined value.

[0085]

As described above, in the method and apparatus for optimizing a hierarchical neural network according to the present invention, a neuron is added to the intermediate layer during learning and learning is continued again. Even if the obtained weight falls into a local solution, it is possible to escape from it and obtain an optimal solution.

Further, a defective neuron evaluation value representing the degree of contribution to the recognition of the input data is obtained for each neuron constituting the intermediate layer, the defective neuron is determined based on the evaluation value, and the defective neuron is predetermined. By deleting at the timing of, the capacity of the memory used can be reduced as the number of neurons is reduced, and high-speed processing becomes possible.

Furthermore, by initializing the weight of the defective neuron as necessary and continuing the learning, it is possible to escape from the local solution and obtain the optimum solution. Then, by increasing or decreasing the number of neurons according to the learning situation, it is possible to increase the recognition rate, increase the processing speed, and reduce the cost. Further, by allowing the once deleted neuron to recover, even if it deletes by mistake and a problem such as a decrease in recognition rate occurs, it can be recovered to the original state. Furthermore, by providing an output error weight adjusting device or the like, it is possible to perform learning at a higher speed and obtain an optimum solution.

Furthermore, when various evaluations are performed on the neural network after learning in consideration of the output error weight data, the concept of the possibility / necessity region is introduced,
The optimal solution that meets the required specifications can be reached at high speed. When the output error weight adjusting means is provided,
The output error weight data that has been initialized can be corrected and automatically determined according to the learning situation, and various evaluations will be performed using the corrected output error weight data, and learning will end faster. Can be made

[Brief description of drawings]

FIG. 1 is a block diagram showing a preferred embodiment of a hierarchical neural network optimization device according to the present invention.

FIG. 2 is a diagram showing an example of a data structure stored in a learning database.

FIG. 3 is a diagram showing an example of a data structure stored in a weight database.

FIG. 4 is a diagram showing an example of a data structure stored in a middle layer optimization parameter database.

FIG. 5 is a diagram showing an example of a data structure of a defective neuron evaluation value.

FIG. 6 is a part of a flow chart showing a preferred embodiment of the optimization method of the hierarchical neural network according to the present invention.

FIG. 7 is a partial diagram showing a continuation of the flowchart shown in FIG. 6;

8 is a process diagram showing a main part of the work procedure shown in FIGS. 6 and 7. FIG.

FIG. 9 is a flow chart diagram for explaining the function of an intermediate layer neuron optimization determination device, which is a main part of a second embodiment of a hierarchical neuron optimization device according to the present invention.

FIG. 10 is a diagram showing an example of a data structure of optimization parameters used in this embodiment.

FIG. 11 is a block diagram showing a third embodiment of the optimization apparatus for hierarchical neurons according to the present invention.

FIG. 12 is a diagram illustrating a function of output error weight data.

FIG. 13 is a diagram showing an example of a data structure of output error weight data and optimization parameters in the present embodiment.

FIG. 14 is a diagram showing an example of a data structure of a correct answer rate in the present embodiment.

FIG. 15 is a flowchart illustrating the functions of the optimization processing selection device of this embodiment.

FIG. 16 is a flow chart for explaining the overall operation of the apparatus of this embodiment.

FIG. 17 is a block diagram showing a fourth embodiment of the optimization apparatus for hierarchical neurons according to the present invention.

FIG. 18 is a flow chart diagram illustrating the function of the hidden layer neuron optimization determination apparatus of the present embodiment.

FIG. 19 is a graph showing an example of the relationship between the local solution and the optimum solution.

[Explanation of symbols]

10 Neural network learning device 18 'Output error weight data D / B 20, 20', 20 "Hierarchical neural network optimizing device 21, 21 ', 21" Error sum of squares and its variation calculation device 22, 22', 22 "Correct answer rate and its change amount calculation device 23, 23 ', 23" Bad neuron evaluation value calculation device 24, 24', 24 "Intermediate layer neuron reduction device 25, 25 ', 25" Intermediate layer neuron addition device 26, 26 ′, 26 ″ Intermediate layer neuron weight initialization device 27, 27 ′, 27 ″ Intermediate layer neuron optimization determination device 28, 28 ″ Output error weight adjustment device 29 Optimization processing selection device

Claims (12)

[Claims] 1. While performing a predetermined learning on a hierarchical neural network, it is determined whether or not the weight given to the neurons in the intermediate layer falls into a local solution,
A method for optimizing a hierarchical neural network, characterized in that a new neuron is added to a predetermined position of the intermediate layer when a local solution is encountered. 2. A neural network learning device that performs learning on a hierarchical neural network, and weights given to neurons in an intermediate layer based on learning status data output from the neural network learning device are local solutions. And a device for determining whether or not the device has fallen into the intermediate layer, and an intermediate layer neuron adding device for adding a new neuron to the intermediate layer based on a control signal sent from the determining device. Neural network optimization device. 3. A defective neuron evaluation value, which indicates the degree of contribution to the recognition of input data, is obtained for each neuron that constitutes an intermediate layer during the predetermined learning for a hierarchical neural network, A method for optimizing a hierarchical neural network, characterized in that a defective neuron is determined based on the evaluation value, the defective neuron is deleted at a predetermined timing, and then learning is continued. 4. A neural network learning device for learning a hierarchical neural network, and recognition of input data for each neuron constituting an intermediate layer based on learning situation data output from the neural network learning device. A defective neuron evaluation value calculation device that calculates a defective neuron evaluation value indicating the degree of contribution, and whether learning is close to an optimal solution is determined based on the learning situation data output from the neural network learning device. A device, a control signal sent from the device for judging the device, and an intermediate layer neuron reduction device for deleting the defective neuron in the intermediate layer based on the evaluation value data of the defective neuron sent from the defective neuron evaluation value calculation device. An apparatus for optimizing a hierarchical neural network, characterized by comprising: 5. A defective neuron evaluation value, which represents the degree of contribution to the recognition of input data, is obtained for each neuron that constitutes an intermediate layer during the predetermined learning performed on the hierarchical neural network, Optimal hierarchical neural network characterized by determining a defective neuron based on the evaluation value, initializing the weight given to the defective neuron at a predetermined timing, and continuing learning Method. 6. A neural network learning device for learning a hierarchical neural network, and recognition of input data for each neuron constituting an intermediate layer based on learning situation data output from the neural net learning device. A defective neuron evaluation value calculation device that calculates a defective neuron evaluation value that expresses the degree of contribution, and the weights assigned between neurons in the intermediate layer based on the learning situation data output from the neural network learning device. A device that determines whether or not a solution is present, a control signal that is sent from the device that determines whether the solution is in solution, and defective neuron evaluation value data that is sent from the defective neuron evaluation value calculation device. And a middle-layer neuron weight initialization device for initializing neuron weights. Optimization device for hierarchical neural networks. 7. A method of adding and deleting neurons in an intermediate layer is appropriately performed by using the method described in claim 1 or 3 while performing a predetermined learning on a hierarchical neural network. A method for optimizing a hierarchical neural network, which is characterized in that it is executed while being used properly. 8. A method for optimizing a hierarchical neural network, which comprises performing the process according to claim 3 or claim 7 to restore a once deleted defective neuron and then continuing learning. 9. A neural network learning device for learning a hierarchical neural network, and recognition of input data for each neuron constituting an intermediate layer based on learning situation data output from the neural network learning device. Defective neuron evaluation value calculation device for calculating a defective neuron evaluation value representing the degree of contribution, a device for judging a learning state based on learning situation data output from the neural network learning device, and a device for judging the learning state A device for adding a new neuron to the intermediate layer based on a control signal sent from the device, a control signal sent from the device for judging, and an evaluation of a defective neuron sent from the defective neuron evaluation value calculation device. An intermediate layer for eliminating bad neurons in the intermediate layer based on the value data and Yuron reduction apparatus and optimization system of the hierarchical neural network which is characterized in that an intermediate layer neuron weights initialization device for initializing the weights of the neurons of the intermediate layer. 10. The output error weight data set based on a required specification is stored in the hierarchical neural network optimizing device according to any one of claims 2, 4, 6, and 9. The optimization of the hierarchical neural network is characterized in that the device for making various judgments is adapted to make the various judgments in consideration of the output error weight data stored in the memory means. apparatus. 11. The optimization apparatus for a hierarchical neural network according to claim 10, further comprising output error weight adjustment means for correcting the output error weight data stored in the storage means, and the determination means, An optimization apparatus for a hierarchical neural network, further comprising a function of determining whether to correct the output error weight data. 12. A neural network learning device for performing learning on a hierarchical neural network, storage means for storing predetermined output error weight data, learning status data output from the neural network learning device, and the storage. Based on the output error weight data provided from the means, a means for determining the learning status to determine whether or not the output error weight data needs to be corrected, and the output error weight data based on the determination result of the determining means. An apparatus for optimizing a hierarchical neural network, which comprises an output error weight adjusting means for correcting and storing the corrected output error weight data in the storage means.