JPH10283334A - Input layer optimizing method for neural network - Google Patents

Abstract

PROBLEM TO BE SOLVED: To optimize the input layer of the neural network by evaluating the coefficient of coupling between the input layer and intermediate layer or output layer of the neural network that varies depending upon its learning, and detecting and removing unnecessary data in input data according to the result. SOLUTION: An ordinary learning process is performed by the neural network having the input layer for inputting pieces of data by using a necessary sufficient number of input variables, and the coefficient of coupling between the input layer and intermediate or output layer which varies depending upon the learning is evaluated (S21). Then it is decided whether or not there is unnecessary data according to the evaluation result and the process is ended when there is no unnecessary data (S3). When there is unnecessary data, the input variable is changed (S22) and learning is carried out again (S1); and unnecessary data detected by input evaluation (S21) is deleted. Consequently, the input layer of the neural network can be optimized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method for optimizing an input layer by removing unnecessary data from input data of a neural network used in various information processing fields.

[0002]

2. Description of the Related Art In various information processing fields, a neural network is used particularly for a system having nonlinearity. 5 to 7 show various known neural network models. FIG. 5 is a hierarchical model including an input layer, a middle layer and an output layer, and FIG. 6 is a Williams & Z as a recurrent network model including an input layer and an output layer.
FIG. 7 shows an Elman model as a recurrent network model including an input layer, a context layer, a hidden layer, and an output layer.

The Williams & Zipser model shown in FIG. 6 is a mutual connection type neural network, and all outputs are fed back to the input side. The input layer includes an external input and a feedback input, and performs learning using a learning algorithm called real-time recurrent learning (RTRL). In the Elman model of FIG. 7, the state of the first-order delay of the hidden layer is fed back to the context layer. The middle layer usually includes past information as the learning progresses, reflecting the state of the input / output layer. Therefore, the context layer sufficiently reflects the input / output history. These recurrent network models have a feedback connection inside, and it is possible to grasp the history of time-series data by having information one cycle before in the network.

[0004] These neural networks can be applied in various ways. Generally, neural networks are made to learn many past cases in advance.
Various predictions and controls are performed using a neural network that has been trained during use. Conventionally, input variables (input data) used for the learning and prediction have been determined by experts and experts in various fields based on experience and experimental results.

Specifically, the output is a target value itself such as a predicted value, a control amount, and a state, and is uniquely determined. The number of elements in the intermediate layer is determined by AI as an information amount reference.
Badness Factor and Goodness Facto, which are statistically determined using the information amount represented by C or are evaluation criteria limited to the intermediate layer of a hierarchical neural network as shown in FIG.
Determined based on criteria such as r and Effectiveness Factor. However, there are various types of input variables such as temperature and flow and cannot be uniquely determined. Conventionally, input variables selected by methods such as regression formulas, statistical processing, and intuition can be used as they are, or neuro learning can be performed many times, and the input variables that provide the best results can be selected. The method of determining was adopted.

[0006]

The problem here is that it is not always clear what input variables (input data) are needed to obtain the desired output. This is due to the nonlinear application field of the neural network. Generally, a sufficiently large amount of data is required to improve the accuracy of an output value. However, if you enter data that has nothing to do with the output,
That data can act as noise,
This may cause the prediction accuracy to deteriorate. Further, when the number of input data is large, the scale of the neural network becomes large and complicated, which leads to a long learning time and an increased risk of convergence to a local solution.

As a method of optimizing the input layer including the number of input data, there is no conceivable way to apply a sensitivity analysis, but it is generally difficult to apply the method and good results cannot be expected. Sensitivity analysis is a method to determine the effect of input errors, and measures the effect on output by fluctuating input data.Unfortunately, it is a powerful tool for analysis in a limited range of linear and nonlinear. However, a wide range of non-linear sensitivities cannot be obtained well. In a non-linear system, the sensitivity differs in the point of calculation, so it is difficult to make an objective and quantitative judgment. Therefore, it is difficult to automatically optimize the input layer by applying the sensitivity analysis.

The present invention has been made in view of the above points, and provides an input layer optimizing method in which unnecessary data is deleted and only appropriate input data is used irrespective of a method such as sensitivity analysis. It is assumed that.

[0009]

In order to solve the above-mentioned problems, the present invention relates to a neural network having an input layer to which a plurality of data are input, wherein the input layer and the intermediate layer or the output layer change by learning of the neural network. Is evaluated, and unnecessary data in the input data is detected and removed according to the result, thereby optimizing the input layer of the neural network.

[0010]

First, in the present invention, in learning a neural network, unnecessary data is detected from input data to optimize an input layer. That is, the present invention detects and removes unnecessary data during or after learning when unnecessary data that is not related to a desired output is input to the neural network.

FIG. 1 conceptually shows a neural network model to which an embodiment of the present invention is applied. Unnecessary data as well as necessary data are input to the external input of the network, but of the connection between the input layer and the output layer, the connection connected to the required data is strengthened, and the connection connected to the unnecessary data is weakened. Paying attention to this feature, unnecessary data is detected by an evaluation formula shown by the following Expression 1 (corresponding to Goodness Factor) or Expression 2 (corresponding to Effectiveness Factor).

[0012]

## EQU1 ## Good _i = Σ (w _ij · i _i ) ²

[0013]

[Equation 2] Effect _i = Σw _ij ²

In these equations, w _ij is the input element i
Is a coupling coefficient between the input element i and the output element j, and _ii is input data to the input element i. Good _i in Expression 1 is a sum of squares of information output from the input element i and is an evaluation expression based on the amount of information, and Effect _i in Expression 2 is an evaluation expression focusing on the network structure. Goo of Equation 1
Since d _i is the input data i _i into the right side evaluates to all training patterns, it is necessary to determine the like average or sum. On the other hand, Effect _i in Equation 2 is
Used when input data is normalized and its distribution is unbiased. This Effect _i can be calculated faster than Good _i because there is no input data i _i .

These Good _i and Effect _i are:
The concept of the evaluation criterion that has been conventionally applied to the hidden layer elimination algorithm is specially modified for the input layer optimization and applied, and is applicable to network types other than the feedforward hierarchical type.

FIG. 2 is a flowchart showing a series of processes for optimizing the input layer by deleting unnecessary data using the above evaluation formula. Compared with the related art, it can be realized only by adding a process of evaluating an input and changing an input variable according to the result. Hereinafter, the contents of each process in FIG. 2 will be described.

(1) Learning (S1) This is a normal learning process using a neural network.
When learning for the first time, a necessary and sufficient number of input variables (input data) are prepared. Here, there are various learning algorithms, such as the above-described real-time recurrent learning and back propagation (backpropagation), but the learning algorithm itself is not the gist of the present invention, and thus the description is omitted.

(2) Input evaluation (S21) This is performed after or in parallel with learning. More specifically, Good _i and Effect _i are obtained by using the coupling coefficient w _ij coming out of the input layer and the above-mentioned equation (1) or equation (2). Although various evaluation methods can be considered, for example, Good _i and E
There may be a method in which the input data when the value of f.sub.fect _i becomes equal to or greater than a certain set value is required data, and the other data is unnecessary data.

(3) Termination determination (S3) As a result of the input evaluation (S21), it is determined whether or not there is unnecessary data. If there is unnecessary data, the following input variables are changed (S22), and learning is performed again (S1).

(4) Change of input variables (S22) In this step, unnecessary data detected by the input evaluation (S21) is deleted. Conventionally, this processing is manually performed by an operator by trial and error. However, in the present embodiment, unnecessary data can be quantitatively and automatically evaluated and detected by the input evaluation (S21). It is possible to change input variables at high speed.

[0021]

EXAMPLE Table 1 shows the learning data. The input data has x1 (t) = sin (t) which has a large correlation with the output data y (t) = 2 sin (t), and the output data y
X2 (t) (= random noise), which has nothing to do with (t). Where t = 0, π / 18, 2π / 18,
.., 6π (108 data). The input layer optimization function of the present invention attempts to detect an element to which random noise as unnecessary data is input.

[0022]

[Table 1]

Table 2 shows the learning conditions. The target network model is Williams & Zipser shown in Figs.
Model, Elman model. Equation 2 is used as the evaluation equation.
Effectiveness Factor was used.

[0024]

[Table 2]

FIG. 3 (Williams & Zipser model), FIG. 4
(Elman model) shows the learning result. In any case, the Effectiveness Factor (the value of Expression 2) of the input element to which the input data x1 (t) = sin (t) is given increases as the number of times of learning increases. Effectiveness Factor remains almost 0 and does not change. In other words, as the learning progresses, the importance (Effectiveness Factor) of the input element of the data x1 (t) = sin (t) sin having a large correlation with the output data y (t) = 2sin (t) increases, but is unnecessary. The importance of the input element of random noise that is the input data x2 (t) remains low. From this, unnecessary input data having a low contribution to the output can be detected, and by deleting these unnecessary data, the input layer of the neural network can be optimized. 3 and 4 also show the absolute value error of the output data. It can be seen that this absolute value error decreases as the number of times of learning increases.

[0026]

As described above, according to the present invention, unnecessary data input to a neural network is objectively and
Quantitative evaluation can be performed, and the input layer of the network can be optimized and the learning time can be reduced. In addition, usually, neural network learning involves many trial-and-error input variable selection processes. This process can be greatly reduced.

For example, in a daily maximum power prediction for predicting the maximum value of the power demand on the next day using a neural network, the correlation between the weather and the power demand is strong, and the power demand is calculated by using the weather as input data. It is known that it can be predicted. However, it is not clear exactly which weather conditions and how much are required as input data. For example, there is an example in which the maximum and minimum temperatures are used, and an example in which the temperature at a specific time is used. In addition, the neural network normally inputs data for the past several days, but there is no clear standard as to how many days ago data needs to be input. Furthermore, the combination of input variables is enormous, and it is virtually impossible to verify all of them.
As a practical matter, research on daily maximum power has been conducted for the past several years. Such an example is a common problem in all fields to which a neural network is applied, such as dam inflow prediction, average stock price prediction, weld pool control, and robot control.

Therefore, according to the present invention, unnecessary input data can be easily detected and removed, thereby shortening the research period, improving learning accuracy and prediction accuracy by optimizing a network, and reducing processing time. Can be expected.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram of a neural network model to which an embodiment of the present invention is applied.

FIG. 2 is a flowchart showing a series of processes for optimizing an input layer in the embodiment of the present invention.

FIG. 3 is an Effectiveness Factor according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating an absolute value error of output data.

FIG. 4 is an Effectiveness Factor according to the embodiment of the present invention.
FIG. 7 is a diagram illustrating an absolute value error of output data.

FIG. 5 is a conceptual diagram of a hierarchical model.

FIG. 6 is a conceptual diagram of a Williams & Zipser model.

FIG. 7 is a conceptual diagram of an Elman model.

Claims (1)

[Claims] In a neural network having an input layer to which a plurality of data is input, a coupling coefficient between an input layer and an intermediate layer or an output layer, which is changed by learning of the neural network, is evaluated, and according to the result, An input layer optimization method for a neural network, comprising detecting and removing unnecessary data from input data.