JP2001344590A - Neural network and method for learning the same and method for analyzing the same and method for judging abnormality - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a network having interchangeability with a conventional network, and whose inside analysis is simple, and to provide a method for learning the network, a method for analyzing the network, and a method for judging abnormality. SOLUTION: In this neural network in a hierarchical structure having plural input layer elements and plural intermediate layer elements and a method for learning the network, a method for analyzing the network, and a method for judging abnormality, non-dense connecting parts 12 constituted by connecting intermediate layer elements to a part of those plural input layer elements are generated by eliminating the connection of the input layer elements to the intermediate layer elements (turning the weight into 0).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to the field of information processing related to various controls, predictions, and diagnoses using a neural network.
And a learning method thereof, an analysis method of the neural network itself, and an abnormality determination method based on the analysis result.

[0002]

2. Description of the Related Art As is well known, a neural network has a learning ability, is excellent in non-linearity and pattern matching performance, and is used in many fields such as control, prediction, and diagnosis. Although many structures have been proposed as the neural network, most of the practically used neural networks are mostly of a hierarchical type, especially a three-layer type. The hierarchical neural network learns by an algorithm usually called a back propagation method (error back propagation method), and the internal connection state is adjusted. The neural network trained in this way, when given the same input data as the learning data, outputs almost the same output as the learning data. Further, there is a feature that when an input close to the learning data is given, an output close to the learning data is output. Compared to the regression formula constructed by the least squares method, the neural network is superior in nonlinearity, but it has a complicated internal structure and is difficult to analyze, so it is not known how to output unknown data There are drawbacks.

As a conventional method for analyzing a neural network, there are the following three typical examples as examples of directly analyzing the internal structure. "Discovery and Generalization of Regularity by Neural Network Structure Learning": Journal of the Japanese Neural Network Society, Vol.1, No.2 (199
There is 4). This method is a method of performing an internal analysis by a method in which unnecessary connections of a neural network are deleted by the concept of forgetting, and only necessary connections are left. Its effectiveness has been shown in the field of pattern recognition. "Construction and Learning Methods of Fuzzy Neural Networks": Journal of Fuzzy Society of Japan, Vol.4, No5. (1992)
A fuzzy neural network with a new structure that fuses fuzzy and neuron enables internal analysis. "Fuzzy IF-THEN using neural networks
Automatic extraction of rules ”: IEEJ Transactions on Electronics, C, Vol. 110-C, N
o.3, (1990) analyzes by extracting fuzzy rules from a neural network having a special structure called a distributed neural network.

Although no direct internal analysis is performed, Japanese Patent Application Laid-Open No. H10-74188, "Data Learning Apparatus and Plant Control Apparatus", and Japanese Patent Application No. H11-322130, describe a method for evaluating the reliability of output values. Evaluation device, evaluation method and storage medium for output value of neural network "
There is. Each method is a method of searching and displaying a value close to input data at the time of prediction / control from learning data.

[0005] A back propagation method is generally used as a learning method of a neural network, but a problem has been pointed out in terms of performance. That is, in the back propagation method, it is necessary to determine the number of layers and the number of elements of the neural network in advance, but information on these is not obtained in advance, and to optimize the neural network, the number of layers and the number of elements are It is necessary to search for the number by trial and error. In particular, when the internal structure of the neural network is complicated, the search takes time and effort. Also,
The neural network after learning obtained by the back propagation method becomes a black box, and it is often difficult to define the meaning of the intermediate layer element. In view of these problems, Japanese Patent Application No. 11-066165 “Neural Network Optimization Learning Method” and Japanese Patent Application No. 2000-710.
11 "Neural network optimization learning method", etc.
In recent years, learning methods for reducing unnecessary intermediate layer elements and coupling have been proposed. With these learning algorithms,
It is possible to construct a very compact and high-performance neural network.

[0006]

Neural networks are used in many fields because of their excellent abilities such as learning ability, non-linear ability, and pattern matching ability. It was difficult to easily understand and explain why the output value was output. The above prior art is a method of searching for and displaying learning data close to the input data in the target state, but cannot explain unknown input data that is not in the learning data because internal analysis is not performed. .

In the above-mentioned prior arts, since the internal analysis is performed, it is possible to know what output can be obtained even for an unknown input state. However, although the prior art is effective for discrete problems such as pattern recognition, it has a drawback that it cannot handle problems targeting continuous values.
The prior art uses a neural network having a special structure completely different from a normal neural network structure, and thus lacks versatility. In particular, in the prior art, due to its structure, the learning time increases exponentially as the number of input factors increases, and its ability is inferior to ordinary neural networks. Further, the prior art has problems that not only is the neural network structure complicated, but also its analysis method is difficult, and the extracted fuzzy rules are not simple. The above-mentioned prior art has an advantage that a compact structure can be obtained. However, since the structure itself is the same as that of a conventional neural network, it is impossible to perform an internal analysis.

[0008] Therefore, the object of the present invention is to provide a neural network having a structure which is completely compatible with a normal hierarchical neural network and whose internal analysis is easy,
It is intended to provide a learning method, an analysis method, and an abnormality determination method. For _{example, y = ax 1 + bx 2} + cx 1 x 2 + d (x 1, x 2 are input variables, y the output variables, a, b, c, d
In the case of the regression equation of the prior art as expressed as a coefficient, the output factor is clear (it is clear that it depends on the factors x ₁ and x ₂ ), and the function of each factor is clear. (Consisting of independent components of x ₁ and x ₂ ( _first and _second terms on the right side of the regression equation) and components of their interaction (third term)), the degree of influence of each factor on input and output is Is clear (x
₁ , x ₂ , and x ₁ × _{2 have} a degree of influence of a, b, and c, and there is d as a fixed component). These characteristics facilitate internal analysis.

Accordingly, the present invention aims to realize the easiness of analysis of such a regression equation by using an existing neural network, and furthermore, by using a structure that is compatible with a conventional neural network, an existing system can be implemented. An object of the present invention is to provide a neural network which is easy to apply and has the same performance as the current state, and a learning method and an analysis method thereof.

[0010]

In order to solve the above-mentioned problem, the present invention is directed to a neural network having a hierarchical structure having a plurality of input layer elements and a plurality of intermediate layer elements. Is characterized by having a loosely-coupled portion formed by coupling an intermediate layer element to a part thereof.

According to a second aspect of the present invention, in the neural network learning method for learning a neural network according to the first aspect, all weights between an input layer element and an intermediate layer element are initialized. A second step of removing a coupling between an arbitrary input layer element and an intermediate layer element, and an input layer such that the evaluation function is reduced using an evaluation function for evaluating a learning error. A third step of calculating a weight correction amount between the element and the intermediate layer element, a fourth step of setting the weight correction amount between an arbitrary input layer element and the intermediate layer element to 0, and a third step And a fifth step of correcting the weight between the input layer element and the intermediate layer element by using the final correction amount obtained through the fourth step. Processing of 3 steps or less It is intended to repeatedly run.

According to a third aspect of the present invention, in the third step of the second aspect, the evaluation function is a term for evaluating a learning error to reduce the learning error and for simplifying the structure of the neural network. And a term for reducing unnecessary intermediate layer elements.

According to a fourth aspect of the present invention, in the second or third aspect, as a sixth step, the variance obtained by using the output value sequence of one element in the intermediate layer element is less than a specified value. In some cases, this element is fused to a bias element, and when the correlation coefficient obtained using the output value series of two elements in the intermediate layer element is equal to or greater than another specified value, these two intermediate layers are used. The method has a step of reducing the number of intermediate layer elements by performing compact structuring by fusing elements as having the same function in information transmission.

According to a fifth aspect of the present invention, an intermediate layer element constituting a loosely coupled portion is obtained by rearranging the intermediate layer elements between the second step and the third step. A step of performing a process of putting together the intermediate layer elements constituting the entire coupling portion in which the intermediate layer elements are coupled to all of the layer elements into separate groups is inserted, and in the third step in claim 2, The weight correction amount is calculated using an evaluation function that causes the coupling of the intermediate layer elements forming the coupling portion to grow faster than the coupling of the intermediate layer elements constituting the entire coupling portion.

According to a sixth aspect of the present invention, between the second step and the third step in the second aspect, the partial neuron having the intermediate layer element constituting the loosely coupled portion by rearranging the intermediate layer element is fully coupled to the partial neuro. A step of alternately arranging partial neurons having an intermediate layer element constituting a part is inserted, and in the third step in claim 2, the coupling of any intermediate layer element is performed faster than the coupling of other intermediate layer elements. The amount of weight correction is calculated using an evaluation function that causes growth.

According to a seventh aspect of the present invention, in the third step of the second aspect, a term for growing an arbitrary intermediate layer element faster than the other intermediate layer elements and simplifying the structure of the neural network. The weight correction amount is calculated by using an evaluation function having a term for reducing unnecessary intermediate layer elements.

According to an eighth aspect of the present invention, the learning method according to the fifth aspect and the learning method according to the sixth aspect are alternately performed.

According to a ninth aspect of the present invention, there is provided a neural network constructed by the learning method according to any one of the second to eighth aspects, wherein an input layer element, an intermediate layer element, and an output layer element are interconnected. Is to analyze the degree of influence of input factors such as linearity and non-linearity on the output from the connection state of.

According to a tenth aspect of the present invention, the structure of the neural network is analyzed using the evaluation index indicating the effectiveness of the input layer element and the intermediate layer element for the neural network described in the entirety of the first aspect. Things.

According to an eleventh aspect of the present invention, the amount of information transmitted from the intermediate layer element to the output layer element when arbitrary input data is input is targeted for the neural network described in the entirety of the first aspect. This is to analyze the influence of the input factor on the output based on the size.

According to a twelfth aspect of the present invention, the amount of information transmitted from an intermediate layer element to an output layer element when arbitrary input data is input to the neural network described in the entirety of the first aspect is described. This is to analyze the influence of the input factor on the output based on the correlation with the input data.

According to a thirteenth aspect of the present invention, the amount of information transmitted from an intermediate layer element to an output layer element when arbitrary input data is input to the neural network described in the entirety of the first aspect is described. The upper and lower limits of the output value of the neural network are analyzed based on the correlation with the input data.

The invention according to claim 14 is the invention according to claims 9 to 13
The neural network is analyzed by the analysis method described in any one of the above, the quality of the learning state of the neural network is determined by comparing it with the abnormality determination database, and the result is output.

According to a fifteenth aspect of the present invention, in the abnormality determination method according to the fourteenth aspect, when the phenomenon to be learned is qualitatively known in advance, the neural network is based on a connection state between elements of the neural network. This is to determine the quality of the learning state of the network.

According to a sixteenth aspect of the present invention, in the abnormality determination method according to the fourteenth aspect, when the phenomenon to be learned is qualitatively determined, the input layer element is used as a numerical index indicating the internal state of the neural network. And determining whether the learning state of the neural network is good or not based on the evaluation index indicating the effectiveness of the intermediate layer element.

According to a seventeenth aspect of the present invention, in the abnormality determining method according to the fourteenth aspect, when a mathematical expression representing a phenomenon to be learned exists, the mathematical expression is compared with a correlation coefficient of an internal state of the neural network. To determine the learning failure.

The invention according to claim 18 is the abnormality determination method according to claim 14, wherein the upper limit value of the output of the neural network is smaller than the upper limit value of the learning data, or the lower limit value of the output of the neural network is smaller. The learning failure is determined when the learning data is larger than the lower limit value.

The invention according to claim 19 is the invention according to claims 14 to 1.
8, when learning failure of the neural network is found by the abnormality determination method described in any one of 8 above, re-learning is performed as necessary.

According to a twentieth aspect of the present invention, in the invention of the seventeenth aspect, a range in which learning is poor is automatically specified, and learning data is increased in the range and re-learned.

According to a twenty-first aspect of the present invention, in the neural network learning method for learning a neural network according to the first aspect, all weights between an input layer element and an intermediate layer element are initialized. A first step of removing a coupling between an arbitrary input layer element and an intermediate layer element and generating only an intermediate layer part of a loosely coupled part without generating a fully coupled part; and a learning error. A third step of calculating a correction amount of the weight between the input layer element and the intermediate layer element so as to reduce the evaluation function using an evaluation function for evaluating A fourth step of setting the correction amount of the weight between 0 and 0 to zero,
A fifth step of correcting the weight between the input layer element and the intermediate layer element by using the final correction amount obtained through the third step and the fourth step, and when the learning error becomes equal to or less than a specified value. And a seventh step of adding an intermediate layer element in order to execute the processing of the third step or less again when the learning error is equal to or more than the specified value.

The invention according to claim 22 is the invention according to claims 1 to 8,
21. In the second step in any one of 21., a plurality of input factors are classified into a plurality of groups based on an input / output relationship of learning data, and an intermediate layer portion of a loosely coupled portion is generated for each group. Things.

According to the twenty-third aspect of the present invention,
21. In the second step in any one of 21., a plurality of input factors are set to the maximum, minimum, average,
It is classified into a plurality of groups based on statistical indices such as standard deviations, and an intermediate layer portion of a loosely coupled portion is generated for each group.

[0033] The invention according to claim 24 is the invention according to claims 1 to 8,
21. In the second step in any one of 21., a plurality of input factors are classified into a plurality of groups based on a correlation coefficient between the input factors of the learning data, and a middle layer of a loosely coupled portion is grouped for each group. To generate the part.

According to the twenty-fifth aspect of the present invention,
21. In the second step in any one of 21., a plurality of input factors are classified into a plurality of groups based on a correlation coefficient between an input and an output of learning data, and a loosely-coupled part is Of the intermediate layer.

[0035]

Embodiments of the present invention will be described below. (1) Embodiment of the First Embodiment of the Invention First, a neural network structure according to the first embodiment of the invention will be described. In a normal hierarchical neural network, the input layer elements and the intermediate layer elements are all connected as shown in FIG. 2 (referred to as a fully connected portion), but the hierarchical neural network of the present invention is, as shown in FIG. Only any input layer element and any intermediate layer element are coupled. That is, in this embodiment, the entire coupling portion 11 composed of the intermediate layer element coupled to all the input layer elements and the loosely coupled portion 12 composed of the intermediate layer element coupled to some of the input layer elements are used. Has become. Here, the entire connection portion 11
Is not necessarily required, and in the present invention, it is sufficient that a part of the plurality of input layer elements has a loosely-coupled portion in which the intermediate layer element is coupled. As described above, by providing the loosely coupled portion 12 in which the value of the weight (weight coefficient or coupling coefficient) between some of the input layer elements is set to 0, complete compatibility with the conventional hierarchical neural network is obtained. be able to.

Next, compatibility with a conventional hierarchical neural network will be described. Specifically, it will be described that the neural network of the present invention is compatible with a conventional neural network and can be easily replaced with an existing system. Each weight of the neural network is represented by a numerical array as shown in FIG. The internal structure and properties of the neural network are determined by the number of elements in each layer and the value of the weight between elements. Table 1 shows an example of the database structure of a conventional neural network.
Shown in This database includes the number of elements in the input layer, the intermediate layer, and the output layer, and the magnitude of the weight between elements in each layer (w ₁₁ , w ₁₂ ,..., V ₁ , v _2, etc.).

[0037]

[Table 1]

The neural network of this embodiment is
This database can be expressed simply by setting an arbitrary weight w _ij between the input layer element and the intermediate layer element to 0, and the same database structure as that of the conventional neural network can be adopted. .

Next, FIG. 4 shows a conventional neural network apparatus (parts for prediction / diagnosis etc. excluding the learning part).
For simplicity, FIG. 4 shows only a minimum configuration. This device is a device or a computer that performs prediction / diagnosis, etc., and stores a learned neural network database in a storage device (HDD, ROM, RAM, external storage device, etc.) 13 and displays the result of the neural network operation. Output to a transmission device (display, printer, speaker, LAN, telephone line, etc.) 17. In addition, 1
4 is an input data reading unit, 15 is a weight reading unit, 1
6 is a neuro (neural network) calculator, 18
Is a storage unit for storing results of prediction / diagnosis. Normal,
Neural network databases are often re-learned by other computers and updated regularly. Since the neural network of the present invention is compatible with the current neural network at the database level, there is no need to modify the device.

The conventional hierarchical neural network is
Since the intermediate layer element is fully coupled to the input layer element, all the data from the input layer is mixed in the intermediate layer part, and it has been difficult to analyze the relationship between the input data and the output value. However, the neural network structure of the present invention has a loosely coupled portion, and the data from the input layer is separated for each arbitrary input factor in the intermediate layer portion, so that the analysis is easy. Elements that cannot be separated, such as the interaction of input factors, are processed at all connected parts, if any, so that it is possible to guarantee the same accuracy as before.
A specific analysis method of the neural network in the present embodiment will be described later.

(2) Embodiment of the Invention of Claim 2 The invention of claim 2 relates to the learning method of the neural network according to the invention of claim 1. An embodiment of the present invention will be described with reference to the flowchart of FIG. First
Step A1 is a normal neural network weight initialization process. Specifically, an initial value is given by a small number of random numbers for all weights between elements of each layer of the ordinary neural network of FIG. Here, a neural network structure without any connection between the input layer and the hidden layer may be defined programmatically. In this case, the processing of the following second and fourth steps is unnecessary.

The second step A2 is a process for changing the neural network with weights initialized to the neural network structure according to the first aspect of the present invention. That is, the coupling between an arbitrary input layer element and an intermediate layer element is deleted. Here, the simplest method for removing the connection is a method of replacing an arbitrary weight value with zero.

The third step A3 is the calculation of the weight correction amount of the ordinary neural network. The correction amount of the weight between the input layer element and the intermediate layer element is calculated so that the evaluation function for evaluating the learning error becomes small. An example of the evaluation function here is shown in Expression 1 below. Of course, other evaluation functions may be used as in the embodiments of the invention described in the claims.

[0044]

## EQU1 ## J = １ ／ · (ot) ²

In the equation 1, J: evaluation function,
o: neuro output, t: teacher signal (learning target value).

The fourth step A4 is the re-correction of the weight correction amount for the neural network structure according to the first aspect of the present invention. By the calculation in the third step A3, an arbitrary weight without coupling may be reconstructed. In order to prevent this, the correction amount of the weight of the arbitrary connection is forcibly set to zero.

The fifth step A5 is a weight correction process. The weight between the input layer element and the intermediate layer element is corrected according to the final correction amount calculated through the third and fourth steps. Assuming that the weight correction amount is △ w _ij , the weight is w _ij , and the learning coefficient is α, the weight can be corrected by Expression 2.

[0048]

## EQU2 ## w _ij = w _ij + α △ w _ij

The processing after the third step A3 is repeated until the learning error becomes equal to or less than the specified value and the completion of the learning is confirmed (step A6). Here, the determination of the end of the learning can be made based on whether an error with respect to the evaluation function or all the learning data has become a specified value or less, or whether the number of times of learning has reached a predetermined number.

(3) Embodiment of the Invention According to the Third Embodiment According to the evaluation function of Expression 1 described in the embodiment of the second embodiment, a learning error can be reduced, but an unnecessary connection is often included. . Unwanted connections complicate the structure of the neural network and are a major source of difficulty in its analysis. Therefore, the invention of claim 3 is an improvement of the third step A3 of the invention of claim 2. This makes it possible to automatically delete unnecessary connections.
The evaluation function used in this embodiment is represented by Expression 3.

[0051]

## EQU3 ## J _f = (term for evaluating output error) + ε ′ (term for evaluating complexity of neural network)

Specifically, due to the difference in terms for evaluating the complexity of the neural network, the following equations 4 to 7 are used.
There is.

[0053]

## EQU4 ## J _f = 1/2 · (ot) ² + ε′Σ | w _ij |

[0054]

## EQU5 ## J _f = 1/2 · (ot) ² + ε′Σw _ij ²

[0055]

J _f = 1/2 · (ot) ² + ε ′ ({| w _ij | + β} (weight of all connected parts) | w _ij |)

[0056]

(7) J _f = 1/2 · (ot) ² + ε ′ ({w _ij ² + β}
(Weight of all connected parts) w _ij ² )

Where J _{f is} an evaluation function of learning with forgetting,
ε ′: forgetting factor, w _ij : weight, β: coefficient. The learning algorithm of the neural network with forgetting is described in, for example, "Structure learning with forgetting of neural network" (Journal of the Japanese Fuzzy Society Vol.9, No.1, pp2-9 (1997)). I have. This learning algorithm with forgetting is to construct a neural network in which output errors are small and generation of unnecessary connections between layers is suppressed. In the evaluation function of Equation 3, the closer the term for evaluating the output error is to 0, the closer the output is to the learning target value, and the smaller the term for evaluating the complexity of the neural network is, the smaller the structure of the network is. Is simple. The smaller this evaluation function value is, the better.

In the evaluation functions of Equations (6) and (7), the weights of all connected parts are added to a large value, so that the growth of all connected parts can be suppressed. The fully connected portion is a portion that is difficult to analyze because it has the same structure as a conventional neural network. The growth of loosely coupled parts is indispensable for analysis. For this reason, the evaluation functions of Expressions 6 and 7 are characterized in that the growth of all the connection portions is suppressed, so that the analysis functions are easily structured. The terms for evaluating the output error in Equations 4 and 5 are all squares of the output error obtained by subtracting the learning target value from the neuro output value. However, the term for evaluating the complexity of the neural network is This is the sum of the absolute values of the weights, and in Equation 5, it is the sum of the squares of the weights. The reason why the weight is used to evaluate the complexity of the network is that, for example, if the weight is 0, there is no connection, and the less the connection, the simpler the network structure becomes. For example, when the evaluation function of Expression 4 is used, the following Expression 8 is used to correct the actual weight. In Equation 8, η is a learning coefficient.

[0059]

(Equation 8)

(4) Embodiment of the Invention of Claim 4 According to the invention of claim 4, in the learning method of claims 2 and 3, as a sixth step, unnecessary intermediate layer elements which are not effectively acting are removed. Fused with a bias element (meaning an intermediate layer element that behaves like a bias element that outputs a constant value without changing the output value even if the input value of the input layer changes). An object of the present invention is to provide a learning method for reducing the number of intermediate layer elements by fusing together intermediate layer elements having the same function with respect to change. The neural network structure according to the first aspect of the present invention is provided with a loosely coupled portion for the purpose of internal analysis. However, when there is an unnecessary coupling or an intermediate layer element, there is a problem that the internal analysis cannot be properly performed. The invention of claim 3 is characterized in that unnecessary portions are deleted in units of coupling between each layer element, but the invention of claim 4 is characterized in that it is deleted in units of intermediate layer elements. Further, since the number of intermediate layer elements is reduced during learning, the amount of calculation is also reduced, which is effective for speeding up learning.

As a method for reducing the number of intermediate layer elements,
The conventional compact structuring method shown in FIG. 6 (for a compact structuring method, see, for example, Tatsuya Masuda et al., "Compact structuring of hierarchical neural network by combining hidden units" (Transactions of the Society of Instrument and Control Engineers Vol.28, N
o.4, pp.519-527 (1992)), a relatively good result can be obtained as it is, but a better result can be obtained by performing the processing shown in FIG. For simplicity, FIG. 6 will be described first. The initialization step B1 in FIG. 6 corresponds to the first step A1 and the second step A2 in FIG. 5, and the learning processing step B2 in FIG. 6 corresponds to the third step A3, the fourth step A4, and the fourth step A4 in FIG. 5
Step B9 in FIG. 6 corresponding to step A5 and determining the end of learning corresponds to step A6 in FIG. FIG.
Steps B4 to B8 of the compact structured portion of (1) correspond to a sixth step unique to the present embodiment, which is not shown in FIG. Conditions for shifting to compact structuring in step B3 in FIG. 6 include, for example, that the number of times of learning has reached a predetermined value and that the learning error has not decreased.

In the compact structured portion, first, the variance of the output of each intermediate layer element is calculated (step B).
4). The variance is a statistical index used also as an evaluation index indicating the importance and effectiveness of the intermediate layer element, and is represented by, for example, Expression 9 using an output value sequence of the intermediate layer element.

[0063]

(Equation 9)

If at least one of the plurality of variances calculated in step B4 has a value equal to or less than the specified value, it is regarded as an unnecessary intermediate layer element and integrated with the bias element (step B4).
5, B6). When all variances are not less than the specified value,
Each correlation coefficient of the intermediate layer element is calculated (step B7).
For example, when there are three intermediate layer elements, three kinds of correlation coefficients of element 1-element 2, element 2-element 3, and element 3-element 1 are calculated, and among them, the correlation coefficient is a specified value or more. Are fused assuming that the elements of the intermediate layer are elements having the same function of transmitting information (steps B8 and B6). The specified value in step B8 is an index of −1 to +1 indicating the similarity. It is assumed that the closer to ± 1, the higher the correlation, and the closer to 0, the lower the correlation. Note that the correlation coefficient is represented by, for example, Expression 10 using output value sequences of two intermediate layer elements.

[0065]

(Equation 10)

When the variance of a plurality of intermediate layer elements is calculated and a large number of variances are equal to or smaller than a specified value, the intermediate layer element having the smallest variance and the bias element are fused. It is conceivable to fuse all the elements that can be fused at once, but in many cases, a better learning result can be obtained by limiting the number of fusions to one for one compact structure. When there are many combinations of the intermediate layer elements having the correlation coefficient value equal to or larger than the specified value, the combinations of the intermediate layer elements having the largest correlation coefficient value are merged. In this case, too, a good learning result is often obtained when the number of times of fusion is limited to one. In step B9, when the number of times of learning reaches a predetermined value, it is determined that learning is completed.

Next, an embodiment of the present invention will be described with reference to FIG. The basic idea is the same as that of FIG. 6, but is characterized by having a process for preventing excessive fusion. In this embodiment, during learning of the neural network, fusion between the intermediate layer elements is restricted to some extent. The intermediate layer elements of the neural network grow as the learning progresses, and their roles are differentiated. Even in an intermediate layer element having the same output value series (high correlation) in the initial stage of learning, the output value series may change as the learning progresses, and the correlation may decrease. Therefore, the fusion is limited to some extent during the learning, and the fusion is promoted when the role of the intermediate layer element becomes sufficiently clear after the learning is completed.

First, steps C1 to C3, C7 in FIG.
Are the same as B1 to B3 and B9 in FIG. In the processing of compact structuring 1 (compact structuring during learning) jumped from step C3 in FIG. 7, unnecessary intermediate layer elements with small variance and bias elements are merged, but intermediate layer elements having a high correlation are compared. Fusion is prohibited. In particular,
If the variance calculated in step C4 is equal to or smaller than the specified value, the intermediate layer element and the bias element are merged (steps C5 and C5).
6). In particular, over-fusion can be effectively prevented by adding a process that fuses with the bias element only when there are two or more unnecessary intermediate layer elements and always leaves one or more unnecessary intermediate layer elements.

In the processing of compact structuring 2 after the learning is completed, the unnecessary intermediate layer elements having small dispersion and the bias elements are integrated (Yes branch in step C9, C10, C13, C1).
4) and fusion of intermediate layer elements having high correlation (step C)
No. 9 branch, C11, C12, C10, C13, C1
Perform both of 4). When the variance is equal to or smaller than a specified value or the correlation coefficient is equal to or larger than a specified value, the neural network is stored by storing weights between elements of each layer of the neural network (step C10), and a learning error is calculated. The fusion of the intermediate layer elements is performed (step C1).
3, C14). The learning error is calculated by using the stored neural network to detect an error between a neural network output value, which is an actual output, and a learning target value. Thereafter, the learning error is calculated again (step C1).
5) The error is compared with the error before fusion to determine whether the error is worse (step C16). When the learning error becomes worse, the fusion is prohibited, and the neural network before the fusion is restored in step C17 (the weight between the elements of each layer is returned to the value before the fusion).

As described above, in the compact structuring 2, when the learning error after the fusion is worse than before the fusion, the neural network before the fusion is restored, and the fusion is performed only when the learning error is improved. By doing so, over-fusion in a compact structure is prevented. In the present embodiment, the threshold value for fusion (predetermined value to be compared with the variance or the correlation coefficient) may be changed for each intermediate layer element.
For example, by reducing the threshold value of the intermediate layer element that leads to loose coupling to suppress fusion, and by loosening the threshold value of the intermediate layer element that leads to full coupling to promote fusion, the total coupling portion is reduced. It is possible.

(5) Embodiment of Claim 5 According to the algorithms of claims 3 and 4, it is possible to eliminate unnecessary intermediate layer elements and unnecessary couplings generated during learning. Since there is no difference in the growth rate between the loosely coupled portion and the fully coupled portion, learning of the loosely coupled portion is not performed quickly, and the loosely coupled portion may not grow sufficiently.
If there are few loosely coupled parts and many fully coupled parts,
It is difficult to analyze like a conventional neural network. Therefore, in this embodiment, an evaluation function that accelerates the learning of the loosely coupled portion and promotes the growth is introduced.

FIG. 8 is a flowchart showing the processing of this embodiment. Compared with FIG. 5 which is the embodiment of the second aspect of the present invention, the addition of the step D3 for rearranging the intermediate layer elements as the 2-1 step and the third step D
4 is characterized in that the evaluation function for calculating the correction amount is changed.

In the rearrangement step D3 of the intermediate layer elements of the present embodiment, the neural network shown in FIG. 2 formed by removing any connection is rearranged, and the loosely coupled portions 12 are grouped as shown in FIG. A partial neuro 1 (group 1) is defined, and the whole including the loosely coupled portion 12 and the fully coupled portion 11 is defined as a partial neuro 2 (group 2).
In the case of FIG. 2, since the loosely coupled portions 12 are grouped on the left side, the rearrangement is not performed in FIG. 9, but in general, the loosely coupled portions are rearranged into one and the partial neurons 1 and 2 are combined. Must be defined.

Next, the evaluation function used in the third step D4 will be described. Assuming that the output of the partial neuro 1 is O ₁ , the output of the partial neuro 2 (normal neuro output) is O ₂ , and the teacher signal (learning target value) is t, the evaluation functions J ₁ and J of the partial neuros 1 and 2 are obtained. Let ₂ be Equation 11.

[0075]

J ₁ = １ ／ · (O ₁ −t) ² , J ₂ = １ ／ · (O ₂ −t) ² (ordinary evaluation function)

The actual evaluation functions are two evaluation functions J ₁ ,
Expression _{2 is obtained} by summing J2.

[0077]

[Expression 12] J = γ ₁ J ₁ + γ ₂ J ₂

The evaluation function J in Expression 12 is called a superposition energy function, and “reduction of redundancy of a multilayer perceptron by the superposition energy function” (Transactions of the Institute of Electronics, Information and Communication Engineers).
D-II, Vol.J80-D-II, No.9, pp.2532-2540, September 1997) and the like. Note that γ ₁ and γ ₂ in Expression 12 are evaluation functions J ₁ and J of partial neurons 1 and 2.
Indicates the weight of ₂ (partial energy function). When the superimposed energy function is minimized, it is possible to obtain a neural network in which unnecessary dispersion representation is suppressed and the neural networks are arranged in the order of importance of the intermediate layer elements. In the superposition energy function, the upper side of the intermediate layer element (the left side in FIG. 9)
There is a feature that learning progresses so as to generate a learning target value with a smaller number of elements as the combination of the elements becomes.

The above-mentioned distributed expression indicates that at least one connection is dispersed into a plurality of connections, that is, as a result, many connections are required. In many cases, the neural network structure becomes complicated and redundant. In the evaluation function J of Expression 12, J ₂ is an ordinary evaluation function including the fully-coupled portion 11 and the loosely-coupled portion 12, while J ₁ is an evaluation function of the loosely-coupled portion 12. Since the loosely coupled portion 12 is involved in both J ₁ and J ₂ , learning (calculation of the weight correction amount using the evaluation function J) is accelerated in order to reduce the error of the loosely coupled portion 12 quickly. As a result, the occurrence of the distributed representation is suppressed, and the neural network structure can be simplified.

(6) Embodiment of the Invention of Claim 6 The invention of claim 6 is an improvement of the invention of claim 5. According to the fifth aspect of the present invention, the intermediate layer elements of the loosely-coupled portion 12 learn and grow at the same speed, and grow.
The learning of the intermediate layer elements of the fully connected portion 11 proceeds at the same speed as each other and grows. If growing at the same rate, 1
One connection is stored in a distributed manner among a plurality of intermediate layer elements, and many intermediate layer elements may be required. In such a case, the weight of the intermediate layer element decreases, but does not decrease to the extent that it is determined that it is not necessary. That is, claim 5
According to the invention, the loosely-coupled portion grows faster when compared with the loosely-coupled portion 12 and the fully-coupled portion 11, but the difference in growth between the intermediate layer elements of the loosely-coupled portion 12 and the intermediate layer elements of the fully-coupled portion 11 is small. Therefore, there is a possibility that the number of intermediate layer elements cannot be sufficiently reduced.

An embodiment of the invention according to claim 6 will be described. In the embodiment of the sixth aspect of the invention, the rearrangement step D3 and the third
Step D4 is improved. In the step of rearranging the intermediate layer elements, as shown in FIG. 10, the intermediate layer elements in the loosely coupled portion and the intermediate layer elements in the fully coupled portion are alternately arranged to form H partial neurons (H = H in the example of FIG. 10). 6) is defined. Then, in the improved third step, H
The sum of the evaluation functions of the partial neuros is used as an evaluation function (superposition energy function) to calculate a weight correction amount.
In Equation 13, γ _i is the evaluation function J _{i of the} partial neuro.
Is shown.

[0082]

(Equation 13)

In the neural network of FIG. 10, the connection in the partial neuro 1 grows fastest, and the connection in the partial neuro 6 is relatively suppressed in growth. This means that, in the superimposed energy function, as the connection with the element on the upper side (left side in FIG. 10) of the intermediate layer elements, learning progresses and grows so as to generate a learning target value with a smaller number of elements. by. That is, in the present embodiment, the growth rate differs between the intermediate layer elements in the loosely coupled portion, and also between the intermediate layer element in the loosely coupled portion and the intermediate layer element in the fully coupled portion. In this way, by making the growth rates of the respective partial neurons different, it is possible to easily determine whether or not the intermediate layer element is necessary, and to quickly delete the unnecessary intermediate layer element.

(7) Embodiment of the Invention of Claim 7 The invention of claim 7 relates to the improvement of the evaluation function of the invention of claims 5 and 6. The invention of claims 5 and 6 is
In the learning process, unnecessary intermediate layer elements are suppressed by calculating the amount of weight correction using an evaluation function that makes the growth rate (learning speed) different for each intermediate layer element. That is, it works for each intermediate layer element. On the other hand, the invention of claim 2 or claim 3 corrects the weight using the correction amount calculated by the evaluation function, and acts on each individual weight. By fusing these two elements, it is possible to construct a neural network with very few unnecessary intermediate layer elements and connections, which can contribute to facilitation of analysis.

The evaluation function in the embodiment of the present invention is composed of the following two elements. Evaluation function = Evaluation function for making a difference in growth rate + Evaluation function for suppressing unnecessary intermediate layer elements A specific evaluation function is represented by Expression 14.

[0086]

[Equation 14]

The first term on the right side of Expression 14 is the same as Expression 13 and is substantially the same as Expression 12. That is, the first term has a function of promoting the growth of some intermediate layer elements. The second term on the right side of Expression 14 may use any one of Expressions 4 to 7 described in the embodiment of the third aspect of the present invention. In particular, if either Equation 6 or Equation 7 is applied, the generation of the fully-connected portion can be suppressed as described above, so that the growth of the loosely-coupled portion is relatively quick, and the neural network structure is easy to analyze.

(8) Embodiment of the Invention of Claim 8 The invention of claim 8 is a method of alternately performing the learning method of the invention of claims 5 and 6 a plurality of times. In the learning by the method according to claim 6, since the intermediate layer elements of the loosely coupled portion and the fully coupled portion are alternately arranged, even if the intermediate layer element of the fully coupled portion does not need to be used at all, the fully coupled portion can be used. May be built. Therefore, according to the eighth aspect of the present invention, by performing the re-learning with the fifth aspect of the invention after the learning according to the sixth aspect of the invention, unnecessary intermediate layer elements in all the coupling portions are suppressed. Further, by repeating this process a plurality of times, it is possible to make the structure closer to the optimal structure. Of course, by combining the inventions of claims 2 and 4, it is needless to say that a structure that is easier to analyze can be obtained.

FIG. 11 is a conceptual diagram showing learning according to the embodiment of the present invention and a change in a neural network structure constructed by the learning. Note that this neural network is for estimating an electric power demand amount as an example, and data such as electric power, weather, and unusual days are input to an input layer.

The stage (a) in FIG. 11 shows a neural network structure before learning is started, and each sub-network has two intermediate layer elements (eight in total). Among them, the intermediate layer elements 1 to 6 are loosely coupled portions, and the intermediate layer elements 7 and 8
Constitutes the entire binding portion (interaction component). (B) is learning (learning stage 1) according to the embodiment of the sixth aspect of the present invention, in which intermediate layer elements of loosely coupled portions and fully coupled portions are alternately arranged. Also, four partial neurons are formed.
(C) is learning (learning stage 2) according to the embodiment of the fifth aspect of the present invention. Since the learning of (b) has been performed, some intermediate layer elements have been reduced by growth (when the embodiments of the inventions of claims 3 and 4 are added). (C)
In the figure, the number of partial neurons is two, and the loosely coupled portion constitutes the partial neuro 1, and the loosely coupled portion and the fully coupled portion constitute the partial neuro 2.

According to the eighth aspect of the present invention, unnecessary intermediate layer elements can be reduced by repeating at least the learning of the above (b) and (c) alternately. In particular, it is more effective to alternately repeat the learning of (b) and (c) a plurality of times. Note that (d) is a result obtained by performing learning by the ordinary superposition energy function method after (c), and may be combined with another learning algorithm as described above. (E) is a neural network structure constructed after the learning is completed. Compared with the original structure, the number of intermediate layer elements is greatly reduced, and a network is constructed only with the necessary intermediate layer elements and weight connections.

(9) Embodiment of Claim 9 Next, an analysis method according to the invention of claim 9 will be described. The present invention is directed to a neural network which has been sufficiently learned by the method according to any one of claims 2 to 8, and 21 to 25, and is designed for a neural network from which unnecessary intermediate layer elements and connections are deleted. Is a method of analyzing the internal state from the coupling state (a path from the input layer element to the output layer element, etc.). A typical pattern is shown in Table 2.

[0093]

[Table 2]

As shown in Table 2, for example, when there is only one path from the input layer to the output layer, there is a high possibility that the input / output relationship is linear as shown by a one-dimensional linear function. When there are a plurality of paths, there is a high possibility that the input / output relationship is non-linear as indicated by a one-way quadratic function. Further, when the intermediate layer element has a coupling with a plurality of input layer elements, it is highly likely that the intermediate layer element is nonlinear having an interaction of input factors like a binary quadratic function. in this way,
An input factor such as linearity or non-linearity depending on a connection state between elements, for a neural network learned by the method according to claim 2. It is possible to roughly analyze the degree of influence of the data input to the element on the output.

(10) Embodiment of the Invention of Claim 10 The invention of claim 10 is a method of analyzing a neural network structure from an evaluation index of an input layer element / intermediate layer element. The analysis target is the neural network according to the first aspect of the present invention, which is a neural network having a loosely coupled portion. Here, the evaluation index is goodness
factor, effectiveness factor, variance, and inverse mapping. Hereinafter, these four evaluation indices are outlined,
The evaluation index itself is not the purpose of the present invention. Table 3 shows an analysis example using the evaluation index. The internal structure of the neural network can be analyzed by evaluating the input layer element and the intermediate layer element using one or a plurality of evaluation indices.

[0096]

[Table 3]

(1) Goodness factor (used for input layer element and intermediate layer element) This is a method using an input signal. This is the sum of input signals output from the intermediate layer element to the output layer element, and is an index indicating the effectiveness of the intermediate layer element. Smaller values have less effect on the output layer elements and are therefore considered "bad" elements.

(2) Effectiveness factor (used for input layer elements and intermediate layer elements) This is a method using weights. The effectiveness factor is expressed as the sum of squares of all weights connected to the intermediate layer element. This is because when the neural network converges,
If all the weights connected to an element are small, the validity of the element is low and the idea is that the element is a "bad" element. This method is basically characterized in that determination can be made only by weights, so that calculation of errors and outputs is not necessary as in the above-mentioned goodness factor, and the amount of calculation is small.

(3) Dispersion (Use for Intermediate Layer) This is a method utilizing the degree of activity of the intermediate layer element. The concept of unnecessary elements is that the output does not change even if the input pattern changes. A plurality of test patterns are input, and an element having a large output variance (a change in the output is large) is an effective element and an element having a small variance is an unnecessary element.

(4) Inverse Mapping (Used for Input Layer) This is a method of ignoring the sigmoid function and linearly approximating the neuron internal structure. The product of the weights from the input layer to the output layer is summed to obtain an approximate value of the sensitivity. This indicator is
As with the iveness factor, there is an advantage that the calculation amount is small.

FIG. 12 is a diagram for explaining the above-mentioned various evaluation indices. The number of input layer elements is I, the number of intermediate layer elements is J,
The case where the number of output layer elements is K is shown. The figure shows the evaluation indexes of goodness factor, effectiveness factor,
The calculation formulas for dispersion and inverse mapping are also shown. In addition, in the analysis example of Table 3 above, when the indices of the input layer elements x ₁ and x ₂ are all substantially the same as shown in the upper part of Table 3, or when the index is specified as shown in the middle part of Table 3 When the indices of the intermediate layer elements coupled to the input layer element are similar, it can be determined that the input layer element has a high possibility of having the target structure. Furthermore, as shown in the lower part of Table 3, when there is a difference in the magnitude of each index of the input layer element, it can be determined that there is a difference in the magnitude of the coefficient of the function indicating the input / output relationship.

(11) The eleventh embodiment of the present invention is to analyze the influence of the input factor on the output from the amount of information transmitted from the intermediate layer element to the output layer element when arbitrary data is input. How to The embodiment of the present invention is also directed to the neural network according to the first aspect of the present invention. Hereinafter, the concept of the present invention will be described. In FIG. 13, the information transmitted from the hidden layer to the output layer is the product sum of the hidden layer output O and the weight v. That is, this information is represented by Expression 15.

[0103]

(Equation 15)

Here, it can be said that an intermediate layer element which outputs the largest value of | v _i O _i | has the strongest influence on the output, and further has a strong influence of the input factor coupled to the intermediate layer element. . For example, in FIG.
When v ₁ O ₁ | is the largest, the effect of input 1 is strong,
When | v ₂ O ₂ | by the intermediate layer element 2 is the largest, it can be said that the interaction between input 1 and input 2 is strong. Therefore, in the embodiment of the present invention, by detecting the amount of information transmitted from each intermediate layer element to the output layer element, the intermediate layer element,
The strength of the connection between the output layer element and the input layer element can be known, and the influence of the input factor on the output in the neural network can be analyzed.

(12) Embodiment of the Invention of Claim 12 According to the invention of claim 12, when any data is input,
This is a method of analyzing the influence of an input factor on an output from a correlation between input data and information transmitted from an intermediate layer element to an output layer element. The present invention is also directed to the neural network according to the first aspect. Hereinafter, the concept of the present invention will be described. In the neural network shown in FIG.
Input 2) = Suppose that a plurality of data of (0,0) to (1,1) are input in increments of 0.2, and the amount of information transmitted to the output layer element at that time is as shown in FIG. That is, it is assumed that the information amount of the intermediate layer element 1 gradually increases, the information amount of the intermediate layer element 2 is almost constant, and the information amount of the intermediate layer element 3 gradually decreases. From this, the following can be understood.

The intermediate layer element 1 has a positive correlation, that is, the input 1 has a positive correlation with the output. The intermediate layer element 2 hardly affects the output. That is, there is almost no interaction between the input 1 and the input 2 coupled to the intermediate layer element 2. The intermediate layer element 3 has a negative correlation, that is, the input 2 has a negative correlation with the output. Further, the above is derived. To increase the output, a large value may be input to input 1 and a small value may be input to input 2. As described above, by detecting the behavior of the hidden layer element with respect to the input value (the amount of information transmitted from each hidden layer element to the output layer), the unknown data (x ₁ ,
even for x ₂ pattern), the output value for the operation of each input divisor and the intermediate layer elements are known can be easily estimated.

(13) Embodiment of the Invention of Claim 13 Next, an embodiment of the invention of claim 13 will be described. The neural network according to the first aspect of the present invention uses a sigmoid function represented by Expression 16 therein.

[0108]

## EQU16 ## y = 1 / {1 + exp (-x)}

Since the output range of the sigmoid function is 0 to 1, when it is actually used, it is 0 to 1 or 0.1 to 0.
It is often used after being normalized to a value of 9. For example, in a neural network for predicting temperature, -20 to +50
In a neural network that predicts the stock price by setting the temperature to 0 to 1, the daily fluctuation range of -2000 to +2000 yen is set to 0 to 1. However, there is no guarantee that a neural network constructed by learning outputs a value in an expected range, and often outputs only a value in a narrow range. Therefore, the present invention relates to a method for analyzing the actual upper and lower limits of the output of the neural network.

In the procedure, as in the twelfth embodiment, the relationship between input data and the amount of information transmitted from the intermediate layer element to the output layer element as shown in FIG. 14 is examined. In the example of FIG. 14, in order to minimize the amount of information transmitted to the output layer element, the input of input 1 coupled to the intermediate layer element 1 (the input factor is only input 1) is set to 0, It can be seen that the input of input 2 coupled to 3 (input factor is only input 2) should be set to 1. That is, the lower limit of the neural network output is obtained when (input 1, input 2) = (0, 1) is input. Incidentally, the upper limit value of the neural network output is obtained when (input 1, input 2) = (1, 0) is input.

(14) Embodiment of the Invention of Claim 14 According to the invention of claim 14, as a result of analyzing the neural network according to the invention of claims 9 to 12, it is determined that the neural network has fallen into an irrational learning state. The present invention relates to an abnormality determination method for determining. In addition, the analysis and
A diagnostic system is also disclosed. Claims 9 to 9 when the learning target of the neural network is known to some extent.
With the twelfth analysis method, it is possible to determine whether the learning state of the neural network is good or not. For example, when predicting the dam inflow by a neural network, the dam inflow has a very high correlation with the upstream flow difference (time difference of the upstream flow), and it is possible to judge that the learning is poor in the following example. When the connection with the upstream flow rate is not established. The difference between the existing prediction formula and the internal analysis result is large (for example, the flow rate increases, but the inflow rate decreases).

An embodiment of the present invention will be described with reference to FIG. (1) Reading of the trained neuro (Step E1) The neural of the neural network structure of claim 1 is read. Reading the neuro means reading the definition of the weight and the number of elements. (2) DB reading for abnormality determination (step E2) A DB for determining a learning failure is read. The criterion for judging abnormality is different for each phenomenon. For example, in the prediction of the dam inflow amount, the above-described determination items can be considered.
Specific abnormality determination criteria will be described later. (3) Neuro Analysis / Diagnosis (Step E3) The neural network is analyzed according to any one of claims 9 to 12. Then, the learning state is diagnosed by collating with the previously read abnormality determination DB. A specific abnormality determination method will be described later. (4) Diagnosis result display (Step E4) The result of analysis / diagnosis is displayed, and a warning sound is generated or transmitted to the outside if necessary.

Next, an analysis / diagnosis system for realizing the present invention is shown in FIG. In FIG. 16, the storage device 21 is a general term for an internal storage device such as an FDD, an HDD, an MO, a RAM, and a ROM and an external storage device. The storage device 21 stores a neural network weight and an abnormality determination DB. Further, the diagnosis result of the learning state is also stored in the storage device 21. Neuro reading module 22, abnormality determination DB reading module 23, analysis /
The operation of the diagnostic module 24 is as described in the flowchart of FIG. The results of the analysis and diagnosis are displayed (or printed) on a display device 26 such as a CRT or a printer in order to inform the operator. In a device without the display device 26, a warning is given by the buzzer 25 when an
It is also possible to transmit to another computer or the like via the N / telephone line 27.

(15) Embodiment of the Invention of Claim 15 According to the invention of claim 15, in the above-described abnormality determination method of claim 14, when the phenomenon to be learned is qualitatively known, the neural network is This is a method of performing an abnormality determination from the connection state. For example, in dam inflow prediction, it has been qualitatively found that the influence of the upstream flow rate difference is large as described above. That is, if the weight of the connection leading to the upstream flow rate difference as an input factor is equal to or smaller than a certain value compared to the weight of the other connections, it is a method of determining an abnormality (learning failure). Here, the connection that leads to the upstream flow difference means the connection between the input layer element (input of the upstream flow difference) and the intermediate layer element, or the connection between the intermediate layer element (the loosely-coupled middle layer element related to the upstream flow rate) and the output layer. Refers to the connection with the element.

As a specific method for realizing the above-described abnormality determination, for example, when the weight of the connection leading to the difference in the upstream flow rate is less than a certain value in the abnormality determination database, or less than a certain value than the weight of another connection (other (E.g., less than 80% of the weighted average) is registered so as to be determined to be abnormal.

(16) An embodiment of the invention of claim 16 The invention of claim 16 is a method of performing an abnormality judgment from a numerical index indicating an internal state of a neural network in the abnormality judgment method of the invention of claim 14. The difference from the fifteenth aspect is that the invention of the fifteenth aspect determines the weight itself by using the abnormality determination database, while
According to the invention, the goodness factor,
The point is that abnormality determination is performed using various evaluation indices indicating the effectiveness of the input layer element and the intermediate layer element, such as effectiveness factor, variance, and inverse mapping.

For example, the criterion for judging an abnormality in the prediction of the dam inflow amount is as follows. Goodness factor for upstream flow difference is lower than others Effectiveness factor for upstream flow difference is lower than others Dispersion for upstream flow difference is lower than others Positive / negative of inverse mapping for upstream flow difference is opposite (abnormal when negative)

(17) An embodiment of the seventeenth aspect of the present invention is the abnormality determination method according to the fourteenth aspect of the present invention, wherein when a mathematical expression representing a phenomenon to be learned exists, the mathematical expression and the neural network If the correlation coefficient with the internal state is less than or equal to a certain value, it is determined that the state is irrational due to learning failure and an abnormality is determined.

For example, in the above-described dam inflow prediction, it is assumed that the upstream flow rate difference has a linear relationship with the output. At this time, if the output of the intermediate layer element relating to the difference in the upstream flow rate shows nonlinearity, it is determined that the abnormality is due to learning failure. As another example, in the next-day power demand forecast, it has been found that the temperature and the power demand have a secondary relationship. Therefore, when the output of the intermediate layer element relating to the temperature is not secondary, it is determined that the abnormality is due to learning failure. The correlation coefficient is used for specific abnormality determination in this embodiment. The correlation coefficient is a statistical index output in the range of −1 to 1,
A formula set in advance by the operator in the abnormality determination database,
If the correlation coefficient between the neural network to be determined and the relational expression of the output of the neural network is less than a certain value, it may be determined that the abnormality is due to learning failure.

(18) An embodiment of the eighteenth aspect of the invention is the abnormality determination method according to the fourteenth aspect, wherein the upper limit value of the output of the neural network is smaller than the upper limit value of the learning data. Alternatively, when the lower limit value of the output of the neural network is larger than the lower limit value of the learning data, it is determined to be abnormal due to poor learning.

The neural network learns arbitrary data to obtain an internal structure that operates as learned data. However, when learning is poor, it may not work as expected. A typical example of learning failure is saturation of upper and lower limits. When constructing learning data from experimental data or natural phenomena, the data of the limit value (upper / lower limit value) is often insufficient. For example, in a neural network for predicting air temperature, data at 10 to 20 ° C. can be prepared abundantly, but since data around 40 ° C. is small, learning failures around 40 ° C. are likely to occur. Usually, a neural network has dozens of input factors, and it is difficult to test all input patterns. In the present invention, if the upper and lower limits of the neural network output analyzed by claim 13 are lower than the upper and lower limits of the learning data or a range narrower than the expected upper and lower limits, it is determined that the learning is defective and the abnormality is determined. I do.

(19) Embodiment of the Invention of Claim 19 According to the invention of claim 19, when it is determined by the above-described inventions 14 to 18 that the neural network has a learning failure, it is automatically re-created. On how to learn. FIG. 17 shows the processing of this embodiment. In FIG. 17, learning (step F2), analysis / diagnosis processing (step F3)
Is realized by each of the above-described inventions. Abnormal neural network learned by analysis / diagnosis (learning failure)
Is determined (step F4), the learning is performed again through the initialization (step F1). In the initialization (step F1), various information such as learning conditions and the structure of the neural network are input to the neural network. Usually, the conditions are changed from those in the previous learning. Note that the initialization processing can be omitted.

(20) An embodiment of the twentieth aspect of the invention according to the twelfth aspect of the present invention, in the invention of the seventeenth aspect, a range in which the learning state is poor is automatically specified, and the learning data in the range is increased to re-execute. Learn how to learn. For example, although it is generally said that there is a linear relationship between the upstream flow rate difference and the output, it is non-linear in FIG. 27 described below, and the input data is saturated when the input data is 0.8 or more or less than 0.2. I have. That is, when the input data is 0.8 or more and less than 0.2, the learning state is a bad range.

Normally, the learning failure of the neural network often indicates a state of being saturated at a certain value or more or less than the certain value. Therefore, the range of the learning failure can be easily specified by searching the saturation region. Is possible. The cause of learning failure is often extremely short of learning data in that range. Therefore, a good learning state can be achieved by increasing the learning data in the range where learning is poor and performing learning again.

[Embodiments] Embodiments of the invention according to claims 1 to 20 will be described below. The first embodiment mainly relates to the invention of claims 14, 4 and 8. Here, for simplicity, the function of two inputs and one output shown in Expression 17 was learned. The signs x ₁ and x ₂ have the meaning of both input layer elements and input data.

[0126]

Equation 17] _{y = x 1 + x 2 +} x 1 x 2 (x 1, x 2 = {0.0~1.0})

The learning algorithm is a method of performing three-stage learning as shown in FIG. 11, and the method of claim 3 or 4 is combined with the evaluation function for suppressing unnecessary intermediate layer elements. Used. The respective evaluation functions used in the first, second, and third stages are represented by Expressions 18 and 1
9 and Equation 20 respectively. In these equations, γ _i is the weight of the evaluation function of the partial neuro, ε ′ is the forgetting factor, and w is the weight of the connection.

[0128]

(Equation 18)

[0129]

[Equation 19]

[0130]

(Equation 20)

Table 4 shows the learning errors of this neural network. According to Table 4, the error is extremely small, and it can be seen that the learning is good.

[0132]

[Table 4]

FIG. 18A shows this neural network.
Learning data x used for learning ₁, X_Two, Y,
FIG. 18B shows the learning result. Also, FIG.
The structure of the neural network before learning
An input layer element, nine intermediate layer elements, and one output layer element
It consists of And the input layer element and the intermediate layer element
In the connection relationship of, the input layer element x₁Is only connected with 3
Loosely Coupled Part 12A Containing Intermediate Layer Elements and Input Layer Element
x_TwoLoosely Coupled Including Three Interlayer Elements Coupled Only to
Part 12B and all input layer elements x₁, X_TwoCoupled with
And a total coupling portion 11 including three intermediate layer elements.
Have been.

In the second embodiment, the internal structure of the neural network learned in the first embodiment is analyzed. This embodiment mainly relates to claims 9 to 11, 15, and 16. In Figure 20, the intermediate layer elements 1 of the first from the left is bonded only to x _1, constitutes a loose coupling portion. Because the information in this pathway is affected only to x _1, the function to be learned, suggesting that section only x ₁ is present. Similarly, the intermediate layer element 2 of the second from the left is bonded only to x _2, constitutes a loose coupling portion. Because the information in this pathway is affected only x _2, the function to be learned, and also suggest that section only x ₂ exists. These are analyzed by the invention of claim 9 in which the linearity or the like is determined from the connection state of the elements of the neural network. In FIG. 20, the weight of the coupling between the elements of the input layer, the intermediate layer, and the output layer is indicated by the thickness of the solid line, and the thin line is 0.1 to 1.0, and the thick line is 1.0 to 10
It is.

Next, with respect to the neural network of FIG. 20, evaluation indices of four intermediate layer elements are calculated, as shown in Table 5. In addition, two input layer elements (input factors)
Table 6 shows the evaluation indices for.

[0136]

[Table 5]

[0137]

[Table 6]

According to Table 5, the effectiveness factors, goodness factors,
Since the variances are substantially the same, it is determined that by applying the invention of claim 10, it is highly likely that the properties of x ₁ and x ₂ and the degree of contribution to the output are the same. Further, the third and fourth intermediate layer elements 3 and 4 are connected to both of the input layer elements x ₁ and x ₂ to form a fully connected portion. The interaction between the layer elements x ₁ and x ₂ or the existence of the nonlinear components x ₁ and x ₂ is expected. actually,
Target function independent claim and x ₁ of x _1, x ₂ of the formula 17,
x ₂ interaction term (third term), and x ₁ , x
₂ is nonlinearly coupled in the third term. Further, according to Table 6, the effectiveness of the input layer elements x ₁ and x ₂ is also shown.
It can be seen that the factor, goodness factor, and variance are comparable.

Intermediate layer elements when inputting representative values to the input layer elements of the neural network shown in FIG. 20 (loosely coupled intermediate layer elements 1 and 2, fully coupled intermediate layer elements 3 and 4, and bias element) Are shown in Table 7. According to Table 7, it can be seen that there is an intermediate layer element whose output does not become 0 even when 0 is inserted in the input data. Therefore, at the time of analysis,
The lower limit value of each hidden layer output is checked, and the target function y = x ₁ + x ₂
+ X ₁ fixed fraction not shown in _{x 2 (y = x 1 +} x 2 + x 1
It turns out that it is necessary to check how much d) in x ₂ + d.

[0140]

[Table 7]

FIGS. 21 to 24 are graphs schematically showing the degree of influence of input data together with the sigmoid function. FIG. 21 shows that when the input data x ₁ and x ₂ are (0, 0),
22 shows the case of (0.5, 0.5), FIG. 23 shows the case of (1, 1), and FIG. 24 shows the case of (1, 0). From these figures, it is possible to easily grasp the degree of influence on the output of each intermediate layer element and each input layer element according to the size of the input data. This means
This is due to the function of the eleventh invention. Furthermore, according to the invention of claim 15, there is a connection connected to the input elements x ₁ and x ₂ and the connection of the weights is almost the same, so that the learning is performed normally, and the abnormality due to the learning failure is abnormal. Can be determined not to exist. Also in the invention of claim 16, since the evaluation indices for x ₁ and x ₂ in Tables 5 and 6 are almost the same, it can be determined that good learning has been performed and no abnormality has occurred.

Next, a third embodiment will be described. This embodiment mainly relates to claims 9, 10, 12, 15 to 17. In this embodiment, a neural network for predicting the inflow of a dam is analyzed. The input factors used for the prediction are the flow rates of the three gauging stations, as shown in Table 8.
Flow rate difference (flow rate for the past 3 hours every 1 hour), flow rate of dam to be predicted, flow rate difference (flow rate for the past 2 hours every 1 hour), basin average rainfall (watershed for the past 20 hours every 1 hour) Average rainfall).

[0143]

[Table 8]

FIG. 25 is a schematic diagram of the upstream area of the dam to be predicted. A rain gauge is installed in each basin, and a water gauge is provided for each basin. FIG. 2 shows the structure of the neural network obtained by the same learning algorithm as in the first embodiment.
FIG. 27 shows information and output values which are shown in FIG.

Learning was started with the number of the initial intermediate layer elements before learning being 12, but according to the structure of FIG. 26 obtained by learning, the number of intermediate layer elements was reduced to three. The intermediate layer elements of the constructed neural network of FIG. 26 include “upstream flow difference”, “rainfall”, and “interaction (all connected parts)”.
No intermediate layer element was constructed, which led to only "upstream flow rate", "dam flow rate", and "dam flow rate difference".
In other words, from the invention of claim 9, "upstream flow difference", "rainfall", "interaction (total coupling portion)" are important factors for prediction, and "upstream flow", "dam flow", "dam flow" It can be seen that the “flow rate difference” is a factor of low importance. here,
The “interaction component (all binding portions)” indicates a complex component which is an effect of a plurality of input factors and cannot be represented by a single component. For example, corresponds to a portion of the x ₁ x ₂ in _{y = x 1 + x 2 +} x 1 x 2.

Table 9 shows the effe of intermediate layer elements 1 to 3 in FIG.
Shows ctiveness factor, goodness factor, and variance.

[0147]

[Table 9]

It can be seen from the goodness factor in Table 9 that the “upstream flow difference” subnetwork has a very important function. This is an analysis result according to the tenth aspect of the present invention. These are in good agreement with the operator's feelings, and can be determined to be good learning results. This is based on the abnormality determination method according to claims 15 and 16.

Further, from FIG. 27, although the component of “upstream flow rate difference” has a great influence on the neuro output, when the input value (input layer) is in the range of approximately 0.2 or less and 0.7 or more, Saturated. According to the twelfth aspect, it is understood that the output of the neural network hardly changes for input values in this range. Furthermore, according to the operator's feeling, there is a linear relationship between the upstream flow rate difference and the output. FIG. 28 shows the amount of information transmitted from the intermediate layer element 1 to the output layer, which leads to only the upstream flow rate difference. Since the upstream flow difference has a slightly smaller linearity correlation coefficient R ² with respect to the input value,
This indicates that learning is poor. This corresponds to claim 17
This is the result of the abnormality determination by the above. In particular, when the input value is approximately 0.
Learning data is poor for learning data in the range of 2 or less and 0.7 or more. According to the twelfth aspect, "rainfall"
It changes greatly only when the input value (input layer) is large.
This accurately represents the phenomenon that the rainfall is easy to be absorbed by the ground when the rainfall is small, and easily drained when the rainfall is large.
The fluctuation of the "interaction component" is very small and acts to the degree of correction. This is an analysis result according to the twelfth aspect.

Table 10 is an index for individually evaluating each input factor. When the goodness factor is a factor of 0.1 or more, it is a part of the flow rate difference and the rainfall, and almost matches the operator's feeling. This is according to the sixteenth invention. The factors of rainfall before 10 hours are often small. It has been found that the target dam has a high correlation with the rainfall up to 10 hours ago, which is consistent with the results.

[0151]

[Table 10]

Next, a fourth embodiment will be described. This embodiment mainly relates to the inventions of claims 11, 13 and 17. In the present embodiment, the analysis is performed using the next day's power demand prediction as an example. The input factors are as shown in Table 11, and the power (maximum power), the weather (highest temperature, the lowest temperature, the minimum humidity, the weather), and the special day flag (Saturday, holiday) are two days before (i) for each season. -2) or from i to 7 days before (i-7).

[0153]

[Table 11]

As a result of preparing and learning 12 intermediate layer elements, a neural network having the structure shown in FIG. 29 was obtained. FIG. 30 shows the input / output relationship of this neural network. The power demand has a positive correlation with the closest power record, and the temperature correlation has a positive correlation and a negative correlation. You can see that it is. This coincides with a phenomenon in which the demand for electric power increases in order to operate the heater when the temperature is low and to operate the cooler when the temperature is high. This is completely consistent with the operator's sense of power demand forecasting.

It is generally said that there is a quadratic relationship between temperature and power demand. From FIG. 31, the temperature (weather) is 2
Since the correlation with the following equation is very strong, it can be determined that the learning is well performed by the invention of claim 17.
The factor that lowers the power demand is that the temperature is moderate and the power demand in the immediate vicinity is small. Conversely, the factor that increases the power demand is whether the temperature is too high or too low. The bias is that the power demand in the immediate vicinity is large. In other words, it can be understood from the invention of claim 13 that the input data that gives the lower limit value of this neural network is when the input related to power is 0 and the input related to weather is 0.5. Table 12 shows specific performance data.

[0156]

[Table 12]

The reference values in Table 12 were created based on the input data that would minimize the power demand based on the results of analysis by the neural network in FIG. This is based on the analysis result according to the invention of claim 13. In fact,
Several values of actual data were input, and it was confirmed that values smaller than the lower limit according to the analysis result were not output.

The following is an example of analyzing the effect of the input factor on the output of the three types of neural networks for spring prediction by using the actual data according to the present invention. FIG. 32 (April 9) is a time when there is no correlation between weather and power, and the predicted value depends only on the actual power. In FIG. 33 (May 30), the correlation between the weather and the power starts to appear, and the predicted value is determined by two factors, the power and the weather. Figure 34 (June 25
Sun) is a season close to summer, when there is a strong correlation between temperature and electricity. As for the predicted value, the influence of electric power is decreasing and the influence of temperature is increasing.

The embodiments and examples of the first to twentieth aspects of the present invention have been described above. According to these inventions, it is compatible with the conventional neural network, and the internal analysis is possible. That is, by providing a neural network structure according to the analysis purpose at the time of learning, the relationship between an arbitrary input factor and an output can be easily analyzed. Usually, this type of analysis is performed with a purpose, and the invention described in claims 1 to 20 can satisfy the need. However, when the analysis purpose is not clear at the time of learning a neural network, an appropriate neural network It may not be possible to give the structure. Therefore, the following invention provides a learning method that improves the inventions of claims 1 to 20, and automatically analyzes the relationship between input and output data to construct a neural network having an appropriate structure. is there.

(21) Embodiment of the Invention of the Twenty-First Embodiment In the inventions of the first to twentieth aspects, it is necessary to prepare an intermediate layer element connected to the fully coupled part and the loosely coupled part before learning is started. Therefore, in the invention of claim 21, learning is performed only in the loosely coupled portion, and the fully coupled portion is added only when learning is not possible only in the loosely coupled portion. That is, according to the present invention, since the number of intermediate layer elements prepared first is small and the amount of calculation is small, the learning time can be reduced.

FIG. 35 is a flowchart showing the processing of the embodiment of the present invention. The first step G1 is a normal neural network initialization process. Specifically, an initial value is given by a small number of random numbers for all weights. This step is the same as step A1 in FIG. The second step G2 is a process for transforming the neural network structure into an analyzable neural network structure. Here, it is necessary to generate only loosely coupled portions without generating fully coupled portions. Third
Step G3 is a calculation of the weight correction amount of the ordinary neural network. Any of the evaluation functions described above may be used as the evaluation function for calculating the correction amount.

The fourth step G4 is a re-correction of the correction amount for the neuro structure that can be analyzed. Third step G3
May construct a bond that should have been deleted. In order to prevent this, the correction amount regarding the weight deleted in the second step is forcibly set to zero. The fifth step G5 is a weight correction process. The weight is corrected according to the calculated correction amount. The sixth step G6 is a learning end determination. This step is the same as step A6 in FIG. 5, and is different from the "sixth step" in claim 4. In this step G6, when the learning error becomes equal to or smaller than the specified value or when the specified number of times of learning is reached, it is determined that the learning is completed. Whether the learning error has become equal to or less than the specified value is determined based on an evaluation function or an error with respect to all learning data. Here, when the learning error is larger than the specified value, it is determined that the learning is not completed, and the process shifts to the seventh step G7.

The seventh step G7 is a process for adding an intermediate layer element of the entire connection portion. The fact that learning is not completed by step G6 means that learning cannot be performed only by the intermediate layer of the loosely coupled portion. Therefore, the process returns to the third step G3 after adding an arbitrary number of intermediate layer elements in all the coupling portions. Normally, the intermediate layer elements of the entire coupling portion are added one by one, but a plurality of them may be added at one time. Further, although omitted in the description of the above steps, the inventions of claims 1 to 20 have described a technique for removing unnecessary intermediate layer elements and couplings. Therefore, by applying these techniques, in the present embodiment, if there are too many intermediate layer elements added or unnecessary intermediate layer elements discovered during learning, these may be deleted. good.

(22) Embodiment of the Invention of Claim 22 Next, an embodiment of the invention of claim 22 will be described. In the invention of claims 1 to 21, at the start of learning of the neural network (the second
An arbitrary neural network structure must be given to the step of removing an arbitrary connection as a step) according to the purpose of analysis. In other words, when the purpose of the analysis is not clear, there is a possibility that an inappropriate structure may be given.
Therefore, in the present invention, a method of automatically constructing a neural network structure in a step at the start of learning (a step of deleting an arbitrary connection) (a method of automatically deleting a connection)
And provided an appropriate structure by automatically dividing input factors into multiple groups based on the input / output relationship of learning data. In this way, by combining input factors that are considered unnecessary in future analysis and separating input factors that are considered necessary for analysis, a neural network with a structure suitable for the expected analysis purpose can be constructed in advance. Can be.

The principle of grouping input factors will be described below. For simplicity, a description will be given of an example of a power demand prediction problem. The power demand prediction problem is a problem of estimating the maximum power of the next day in a certain area, and is positioned as an important task for a power company to formulate a plan for starting and stopping a generator. The amount of power demand can be predicted based on the day of the week, the temperature, the results of the latest power, and the like. Table 13 shows examples of input factors for performing power demand prediction.

[0166]

[Table 13]

In Table 13, input factors are electric power (maximum electric power), weather (highest temperature, lowest temperature, weather), and special day flag (Saturday, holiday) from the current day i to two days before (i-2) for each season. Or from the current day i to seven days before (i-7), and is predicted by a factor of several tens of items in total. Here, the temperature rarely changes extremely one day before or two days before, and the probability of obtaining similar data is high. In addition, it is easy to imagine that the temperature one day ago and the temperature two days ago show the same tendency with respect to the predicted value (neuro output), and it is not necessary to analyze them individually. In other words, similar data is not required to be analyzed separately, so they are grouped into one group, and dissimilar data are separately analyzed because they may be analyzed individually. Hereinafter, a method of grouping input factors will be described.

First Step The input / output relationship of learning data for learning by the neural network is analyzed. Here, those having similar input factors are divided into a plurality of groups. The method of grouping is
There are a method using a simple statistical method such as a maximum / minimum value, a standard deviation, and a correlation coefficient of each input factor, and an advanced statistical method such as a cluster analysis and a discriminant analysis.

Second Step A neural network is constructed according to the grouping obtained in the first step. FIG. 36 shows a four-input one-output neural network in which input 1 and input 2 are determined to be in the same group. Here, the number of intermediate layer elements differs depending on a learning condition defined in advance. Note that reference numeral 11 denotes a fully connected portion, and reference numeral 12 denotes a loosely connected portion.

(23) Embodiment of the Invention of Claim 23 The invention of claim 23 relates to the grouping of input factors in the step at the start of learning (the step of deleting arbitrary combinations), similarly to the invention of claim 22. Yes, this is a method of grouping input factors using simple statistical values such as maximum, minimum, average, and standard deviation of data. In particular, the maximum / minimum value of each input factor is a value that is always calculated when the neural network learns data, and the method using the maximum / minimum value is a method that hardly increases the amount of calculation. An example of a discriminant for performing grouping is shown below. Of course, grouping is also possible in a format other than these formulas. Here, input factors having the same evaluation value are defined as the same group.

Discriminant example 1 Evaluation value _i = int (log ₁₀ (abs (maximum value _i −minimum value _i )) + 0.5)
(Minimum value _i ≧ 0) Evaluation value _i = −int (log ₁₀ (abs (maximum value _i −minimum value _i )) + 0.
5) (a minimum value _i <0) discriminant Example 2 evaluation value _i = int (ln (standard deviation _i) +0.5) discriminant Example 3 evaluation value _i = int (log ₁₀ (mean _i) +0.5) However I: input factor number evaluation value _i : evaluation value of the _i- th input factor int: function for rounding down decimal places to an integer abs: function for calculating absolute value

(24) Embodiment of the Invention of the Twenty-fourth Embodiment The invention of the twenty-fourth aspect also relates to the grouping of input factors in the step at the start of learning (the step of deleting arbitrary combinations), similarly to the twenty-second aspect of the invention. This is a method of performing grouping using correlation coefficients between input factors. The correlation coefficient is a coefficient for calculating the similarity of data having values of −1 to 1. When there are three input factors, a total of three values of the correlation coefficient between input 1 and input 2, the correlation coefficient between input 2 and input 3, and the correlation coefficient between input 3 and input 1 are obtained. In the grouping, input factors having a high correlation coefficient are put together as the same group. Details will be described in Examples.

(25) Embodiment of the Invention of the Claim 25 In the invention of the claim 24, grouping is performed based on the correlation coefficient between input factors. However, this method has a disadvantage that the amount of calculation increases exponentially as the number of input factors increases. Therefore, as a simple method, grouping is performed by focusing only on the correlation coefficient between the input factor and the output factor. Specifically, input factors that have the same degree of correlation coefficient with respect to the output factor are put together in the same group. In the present invention, a step at the start of learning (a step of deleting an arbitrary connection)
, When there are four input factors, the correlation coefficient between input 1 and output, the correlation coefficient between input 2 and output, the correlation coefficient between input 3 and output, the correlation coefficient between input 4 and output It is only necessary to calculate a total of four correlation coefficients. Therefore, when the number of input factors is large as compared with the invention of claim 24, the amount of calculation is greatly reduced, which is effective. It is also easy to reduce the number to a specified value or less so that the number of groups does not become too large. Since the correlation coefficient takes a value of -1 to 1, when the correlation coefficient is to be divided into two groups or less, the correlation coefficient is -1 to 0,0.
When it is desired to divide into groups of １1 and た い 4, the correlation coefficients should be -1 to -0.5, -0.5 to 0,0.
What is necessary is just to divide into 0.5, 0.5-1 groups.

[Embodiment] Next, an embodiment of the present invention will be described. In this embodiment, a method for automatically classifying input factors is implemented. An example is a neural network for predicting the maximum power demand on the day after winter and predicting the dam inflow. Table 14 shows input factors and evaluation values in the next day maximum power demand prediction, and Table 15 shows input factors and evaluation values in dam inflow prediction. The evaluation values are all based on the following formulas. Evaluation value = int (log ₁₀ (abs (maximum value _i −minimum value _i )) + 0.5)
(Minimum value _i ≧ 0) Evaluation value = −int (log ₁₀ (abs (maximum value _i −minimum value _i )) + 0.5)
(Minimum value _i <0)

[0175]

[Table 14]

[0176]

[Table 15]

In the neural network for predicting the maximum power demand the next day, as shown in Table 14, power, temperature, flags, and weather were grouped. As shown in Table 15, in the dam inflow prediction, the flow rate was divided into two, the flow rate difference relation was classified into two, and the rainfall was grouped into five groups, one group. All of these results were almost the same as those of the group specified by the operator in the embodiments of the first to twentieth aspects of the invention, and good results were obtained. In the inventions of claims 1 to 20, it is necessary to manually group input factors. In the invention of claim 23, input factors can be automatically grouped as described above.

Next, embodiments of the present invention will be described. In this embodiment, a method for automatically classifying input factors is implemented. An example is prediction of the maximum power demand on the next day in winter. Table 16 shows the correlation coefficients between input and output factors. Here, the input factor numbers are the same as in the embodiment of the twenty-third aspect of the present invention.

[0179]

[Table 16]

First, an embodiment of the present invention will be described. In Table 16, the input factors whose correlation coefficient (the value excluding the output value column at the right end of Table 16) is 0.5 or more are the temperatures of No. 3 to No. 8, the flags of No. 9 and No. 12, the NO. .1
1, No. 14 flags. These three groups have very similar properties and are well grouped.
That is, in this example, a total of 15 input factors were classified into 7 groups.

Next, an embodiment of the twenty-fifth aspect of the present invention will be described. If the correlation coefficients between the input factors and the outputs (the rightmost column in Table 16) are grouped in increments of 0.2, they can be classified into six groups as follows. Group 1 (0.4 to 0.6): input factor 1 and 2 Group 2 (0.0 to 0.2): input factor 13 to 15 Group 3 (-0.0 to -0.2): input factor 11 Group 4 (-0.2 to -0.4): input Factors 3 to 7, 10 Group 5 (-0.4 to -0.6): input factor 8, 12 Group 6 (-0.6 to -0.8): input factor 9

[0182]

As described in detail above, according to the first to twelfth aspects of the present invention, it is possible to construct a neural network which is compatible with a conventional neural network and can be easily replaced. Further, similarly to the regression equation, the causal relationship between the output and the input can be easily grasped, and the output value can be easily estimated even for unknown data not included in the learning data. If an analysis result that is different from the operator's feeling is obtained (for example, a case where the upstream flow rate change saturates the output in the third embodiment), processing such as re-learning of the neural network is performed in advance. It is possible to do. In particular, in predicting the inflow of dams, it is important to ensure the safety of downstream areas during floods, and therefore accurate prediction of inflow is essential. According to the present invention, a neural network can be appropriately re-learned before a flood occurs. Also,
Even when re-learning is not in time, inappropriate prediction can be grasped. In the third embodiment, since a sudden upstream flow difference exceeding 0.7 or more in input data is saturated, the neural network tends to predict lower than the actual value. Can be easily grasped as being effective.

According to the learning method of the twenty-first aspect, the intermediate layer element is added only when the learning cannot be performed only by the intermediate layer element in the loosely coupled portion, so that the learning time can be reduced. In addition, as compared with the first to twentieth aspects of the present invention, the number of intermediate layer elements in the entire coupling portion can be reduced in many cases, and the analysis becomes easier. The inventions of claims 22 to 25 are related to the initialization at the start of learning. Claims 1 to 20
In the invention, the neural network structure must be defined in advance according to the purpose of analysis. That is,
Any input factors must be grouped. In this regard, in the invention of claims 22 to 25, the neural network structure can be automatically determined from the relationship between the learning data. As described in the embodiment, in the invention of claim 23, since the input factors are roughly grouped, it is relatively easy to obtain a structure which is relatively consistent with human senses. Almost the same structure as grouping was obtained). According to the invention of claim 24, since only considerably similar input factors are grouped (to be classified in detail), when analyzing unknown phenomena such as when it is not known how to analyze in the future, many input factors ( Input factor group) can be analyzed, and it is effective at the time of analysis. The invention of claim 25 is an improvement of the invention of claim 24. When the number of input factors increases, the grouping is performed by a simple method, and thus there is an advantage that the amount of calculation is small.

[Brief description of the drawings]

FIG. 1 is a diagram showing a hierarchical neural network structure according to an embodiment of the present invention.

FIG. 2 is a diagram showing a conventional hierarchical neural network structure.

FIG. 3 is a diagram showing coupling coefficients of a hierarchical neural network.

FIG. 4 is a configuration diagram of a conventional neural network device.

FIG. 5 is a flowchart showing a process according to an embodiment of the present invention.

FIG. 6 is a flowchart showing processing of a conventional compact structuring method.

FIG. 7 is a flowchart showing a process according to an embodiment of the present invention.

FIG. 8 is a flowchart showing a process according to an embodiment of the present invention.

FIG. 9 is a diagram showing a structure of a neural network rearranged according to the embodiment of the fifth invention.

FIG. 10 is a diagram showing a structure of a neural network rearranged according to the embodiment of the sixth invention.

FIG. 11 is a conceptual diagram showing learning and a change in a neural network structure constructed by the learning according to the embodiment of the eighth aspect of the present invention.

FIG. 12 is a diagram showing evaluation indices for judging the effectiveness of an input layer element and an intermediate layer element according to the embodiment of the tenth invention.

FIG. 13 is a diagram showing a structure of a neural network for explaining an embodiment of the invention of claim 11;

FIG. 14 is a diagram for explaining the operation of the embodiment of the invention according to claims 12 and 13;

FIG. 15 is a flowchart showing an embodiment of the invention according to claim 14;

FIG. 16 is a configuration diagram of an analysis / diagnosis system to which the embodiment of the invention of claim 14 is applied.

FIG. 17 is a flowchart showing an embodiment of the invention of claim 19;

FIG. 18 is an explanatory diagram of learning data and learning results in the first embodiment.

FIG. 19 is a diagram showing a structure of a neural network before learning in the first embodiment.

FIG. 20 is a diagram showing the structure of a neural network to be analyzed in the second embodiment.

FIG. 21 is a diagram for explaining the operation of the second embodiment.

FIG. 22 is a diagram for explaining the operation of the second embodiment.

FIG. 23 is a diagram for explaining the operation of the second embodiment.

FIG. 24 is a diagram for explaining the operation of the second embodiment.

FIG. 25 is a schematic diagram of an upstream area of a prediction target dam in the third embodiment.

FIG. 26 is an explanatory diagram of a neural network for flow prediction in the third embodiment.

FIG. 27 is a diagram showing the input / output relationship of the neural network in the third embodiment.

FIG. 28 is a diagram showing the linearity of the upstream flow rate difference in the third embodiment.

FIG. 29 is an explanatory diagram of the next day power demand forecasting neural network in the fourth embodiment.

FIG. 30 is a diagram showing the input / output relationship of the neural network in the fourth embodiment.

FIG. 31 is a diagram illustrating a relationship between input values and information transmitted to an output layer in a fourth embodiment.

FIG. 32 is a diagram illustrating an example in which the influence of an input factor on an output is analyzed using actual data in the fourth embodiment.

FIG. 33 is a diagram illustrating an example in which the influence of an input factor on an output is analyzed using actual data in the fourth embodiment.

FIG. 34 is a diagram illustrating an example of analyzing the influence of an input factor on an output using actual data in the fourth embodiment.

FIG. 35 is a flowchart showing a process according to an embodiment of the present invention.

FIG. 36 is a diagram showing the structure of a neural network constructed according to an embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 11 Fully connected part 12, 12A, 12B Loosely connected part 13 Storage device 14 Input data reading unit 15 Weight reading unit 16 Neuro calculation unit 17 Display / transmission device 18 Prediction / diagnosis storage unit 21 Storage device 22 Neuro reading module 23 Abnormality judgment Database reading module 24 analysis / diagnosis module 25 buzzer 26 display 27 LAN / telephone line

Claims (25)

[Claims] 1. A neural network having a hierarchical structure having a plurality of input layer elements and a plurality of intermediate layer elements, comprising a loosely coupled portion in which an intermediate layer element is coupled to a part of the plurality of input layer elements. A neural network, characterized in that: 2. The neural network learning method for learning a neural network according to claim 1, wherein all weights between an input layer element and an intermediate layer element are initialized. A second step of removing the coupling between the arbitrary input layer element and the intermediate layer element, and using the evaluation function for evaluating the learning error so that the evaluation function is reduced so that the input layer element and the intermediate layer element are reduced. A third step of calculating the correction amount of the weight between the above, a fourth step of setting the correction amount of the weight between the arbitrary input layer element and the intermediate layer element to 0,
And a fifth step of correcting the weight between the input layer element and the intermediate layer element using the final correction amount obtained through the step and the fourth step, until the learning error becomes equal to or less than a specified value. A method for learning a neural network, comprising repeating the processing of the third step and thereafter. 3. The third step according to claim 2, wherein the evaluation function includes a term for evaluating a learning error to reduce the learning error and an intermediate term unnecessary for simplifying the structure of the neural network. A neural network learning method characterized by having a term for reducing layer elements. 4. The method according to claim 2, wherein, as a sixth step, when the variance obtained by using the output value sequence of one element in the intermediate layer element is equal to or smaller than a specified value, this element is determined. Fused with the bias element, 2
When the correlation coefficient obtained by using the output value series of the two elements is equal to or larger than another specified value, the two intermediate layer elements are fused assuming that they have the same function in information transmission, thereby achieving compact structuring. A method for learning a neural network, comprising the steps of: 5. A method according to claim 2, wherein the intermediate layer elements are rearranged between the second step and the third step.
Performing a process of grouping the intermediate layer elements constituting the loosely coupled portion and the intermediate layer elements constituting the fully coupled portion formed by coupling the intermediate layer element to all of the plurality of input layer elements into separate groups. In addition, in the third step in claim 2, the weighting is performed using an evaluation function such that the coupling of the intermediate layer elements forming the loosely coupled portion grows faster than the coupling of the intermediate layer elements forming the fully coupled portion. A method for learning a neural network, comprising calculating a correction amount. 6. A partial neuro having an intermediate layer element constituting a loosely coupled part by rearranging intermediate layer elements between the second step and the third step in claim 2, and an intermediate layer constituting a fully coupled part A step of alternately arranging partial neurons having elements and an evaluation function in the third step according to claim 2, wherein a coupling of an arbitrary intermediate layer element grows faster than a coupling of other intermediate layer elements. A learning method for a neural network, wherein a weight correction amount is calculated by using. 7. The third step according to claim 2, wherein a term for growing a connection of an arbitrary intermediate layer element faster than a connection of other intermediate layer elements and an intermediate section unnecessary for simplifying the structure of a neural network. A neural network learning method, comprising calculating an amount of weight correction using an evaluation function having a term for reducing layer elements. 8. The learning method according to claim 5, and claim 6.
A learning method for a neural network, which alternately performs the learning method described in (1). 9. For a neural network constructed by the learning method according to any one of claims 2 to 8, a linear network is obtained from a mutual connection state of an input layer element, an intermediate layer element, and an output layer element. A method for analyzing a neural network, characterized by analyzing the degree of influence of input factors such as gender and nonlinearity on output. 10. The neural network according to claim 1, wherein a structure of the neural network is analyzed by using an evaluation index indicating the effectiveness of the input layer element and the intermediate layer element. How to analyze the network. 11. An input based on the magnitude of the amount of information transmitted from an intermediate layer element to an output layer element when arbitrary input data is input to the neural network described in claim 1 as a whole. A method for analyzing a neural network, comprising analyzing an influence of a factor on an output. 12. A correlation between the amount of information transmitted from an intermediate layer element to an output layer element and input data when arbitrary input data is input to the neural network according to claim 1. A method for analyzing a neural network, comprising: analyzing an influence of an input factor on an output based on the information. 13. A correlation between the amount of information transmitted from an intermediate layer element to an output layer element and input data when arbitrary input data is input to the neural network described in claim 1 as a whole. A method for analyzing a neural network, comprising analyzing upper and lower limits of a neural network output value on the basis of the above. 14. A method for analyzing a neural network by the analysis method according to claim 9, determining a learning failure of the neural network by comparing the neural network with an abnormality determination database, and outputting the result. Characteristic neural network abnormality determination method. 15. The abnormality determination method according to claim 14, wherein when the phenomenon to be learned is qualitatively known in advance, the learning failure of the neural network is determined based on a connection state between elements of the neural network. A neural network abnormality determination method. 16. The abnormality judging method according to claim 14, wherein when the phenomenon to be learned is qualitatively known, the validity of the input layer element and the intermediate layer element is used as a numerical index indicating the internal state of the neural network. A neural network abnormality determination method, comprising: determining a learning failure of a neural network based on an evaluation index indicating sexuality. 17. The abnormality determination method according to claim 14, wherein when a mathematical expression representing a phenomenon to be learned exists,
A method for determining an abnormality in a neural network, comprising determining a learning failure by comparing the expression with a correlation coefficient of an internal state of the neural network. 18. The abnormality determination method according to claim 14, wherein the upper limit of the output of the neural network is smaller than the upper limit of the learning data, or the lower limit of the output of the neural network is smaller than the lower limit of the learning data. A method for determining an abnormality in a neural network, wherein a learning failure is determined when the value is also large. 19. A learning method for a neural network according to claim 14, wherein when learning failure of the neural network is found by the abnormality determination method according to any one of claims 14 to 18, re-learning is performed as necessary. . 20. The neural network learning method according to claim 17, wherein a range in which the learning state is bad is automatically specified, and the learning data in the range is increased and re-learned. 21. The neural network learning method for learning a neural network according to claim 1, wherein a first step of initializing all weights between an input layer element and an intermediate layer element is provided. A second step of deleting only the coupling between the arbitrary input layer element and the intermediate layer element and generating only the intermediate layer part of the loosely coupled part without generating a fully coupled part, and an evaluation for evaluating a learning error A third step of calculating a correction amount of the weight between the input layer element and the intermediate layer element so as to reduce the evaluation function using the function, and calculating the weight of the weight between the arbitrary input layer element and the intermediate layer element. A fourth step of setting the correction amount to 0, a fifth step of correcting the weight between the input layer element and the intermediate layer element using the final correction amount obtained through the third step and the fourth step; , Learning error A sixth step of determining that the learning is completed when the value becomes equal to or less than a value, and a seventh step of adding an intermediate layer element to execute the processing of the third step and less again when the learning error is equal to or more than the specified value. A neural network learning method, comprising: 22. Any one of claims 1 to 8, 21
In the second step of the term, a plurality of input factors are classified into a plurality of groups based on an input / output relationship of learning data, and an intermediate layer portion of a loosely coupled portion is generated for each of the groups. Learning method. 23. Any one of claims 1 to 8, 21
In the second step in the term, the plurality of input factors are classified into a plurality of groups based on statistical indexes such as maximum, minimum, average, and standard deviation of the training data,
A method for learning a neural network, comprising generating an intermediate layer portion of a loosely coupled portion for each of these groups. 24. Any one of claims 1 to 8, 21
In the second step of the term, a plurality of input factors are classified into a plurality of groups based on a correlation coefficient between the input factors of the learning data, and an intermediate layer portion of a loosely coupled portion is generated for each of the groups. Neural network learning method. 25. Any one of claims 1 to 8, 21
In the second step of the term, the plurality of input factors are classified into a plurality of groups based on a correlation coefficient between the input and the output of the training data, and an intermediate layer portion of a loosely coupled portion is generated for each of the groups. A method for learning a neural network, comprising: