JPH10283335A - Prediction method in recurrent network - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce an error appearing in the head of predicted time-sequential data, and to attain highly precise prediction by preventing the storage and increase of the error. SOLUTION: In this method, feedback connection is attained, and prediction by a recurrent network learning time-sequential data is operated. In this case, the network is initialized at first (S1). Then, prediction is operated by the network as a pre-processing until the continuous points of output data being a target, and the state of the network at the time of ending the prediction is preserved (S2A). Thus, the predication is operated by the network from the state of the preserved network (S2).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a prediction method using a recurrent network having a feedback structure, which is used in various information processing fields.

[0002]

2. Description of the Related Art In various information processing fields, a recurrent network is used when performing prediction using a neural network that has learned time-series data. The recurrent network model includes a Williams & Zipser model composed of an input layer and an output layer as shown in FIG. 10, and an Elman model composed of an input layer, a context layer, a middle layer and an output layer shown in FIG.

[0003] The Williams & Zipser model of FIG. 10 is an interconnected neural network, and all outputs are fed back to the input side. The input layer includes an external input and a feedback input, and performs learning using a learning algorithm called real-time recurrent learning (RTRL). In the Elman model in FIG. 11, the state of the first-order lag of the hidden layer is fed back to the context layer. Since the middle layer usually reflects the state of the input / output layer and contains past information as learning progresses,
The context layer reflects the input / output history sufficiently.

[0004] These recurrent network models have a feedback connection inside, and it is possible to grasp the history of time-series data by having information one cycle before in the network. The input information to the recurrent network is roughly classified into time-series data as an external input and data one cycle before as a feedback input from the inside of the network (also a part of the network output).

[0005]

The problem here is that the feedback input is used in the recurrent network as described above, so that the feedback input cannot be obtained without operating the network. For this reason, conventionally, as the time series head data at time t = 0, a value such as 0 or 0.5 is input instead of the feedback data from the inside.

After time t = 0, the output data usually includes an error. In this case, if the output data is directly fed back and input, prediction is performed using data having an error, and the prediction value of the next cycle includes more errors, and this error is accumulated for each prediction. Will go.

[0007] The present invention has been made in view of the above problems, and particularly focuses on handling feedback information of a recurrent network, appropriately handles feedback data that does not yet exist at time t = 0, and furthermore, handles time t = It is an object of the present invention to provide a prediction method using a recurrent network that can perform a highly accurate prediction by eliminating an accumulation error of feedback data after 0.

[0008]

In order to solve the above-mentioned problems, the invention according to claim 1 is a prediction method for performing prediction using a recurrent network having a feedback structure and learning time-series data, wherein a network is initialized. A first step of performing a prediction as a pre-process by a network up to a continuous point of target output data, and storing a state of the network at the end of the prediction; And a third step of making a prediction from the state by the network.

According to a second aspect of the present invention, there is provided a prediction method for performing prediction using a recurrent network having a feedback structure and learning time-series data, wherein a first step of inputting external data and feedback data to the network, , A third step of detecting an error in a predicted value output from the network, and a fourth step of correcting feedback input data of the network according to the error.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, an embodiment of the present invention described in claim 1 will be described. FIG. 1 shows a conventional prediction processing procedure using a recurrent network (FIG. 1A) and a processing procedure of the present embodiment (FIG. 1B). The present embodiment includes a first step (S1) for initializing a network, a second step (S2A) for performing prediction as preprocessing, and a third step (S2) for performing actual prediction (recall). And the pre-processing step (S2
It differs from the conventional processing procedure in the presence or absence of A).

Hereinafter, the processing contents of each step in the present embodiment will be described. (1) First Step (S1): Initialization This step is a step of reading the weight of the already learned network and initializing the input / output data and the internal potential. The network is made up of a connection of many neurons (brain cells), and initializes the input / output (stimulus / excitation degree) of each neuron to, for example, 0.5.

(2) Second Step (S2A): Preprocessing In this step, normal prediction is performed as preprocessing. For example, when predicting a periodic function such as a sine wave shown in FIG.
Is a discontinuity point. For such a periodic function,
After one cycle, after two cycles,..., The values are the same and are continuous points, so prediction is performed as a pre-process up to the continuous point of this data (for example, one cycle later). In addition, the state of the network, such as the weight, input / output data, and internal potential of the network for which prediction has been completed, is all stored.

(3) Third Step (S2): Prediction (Recall) This step is an original prediction process. Conventionally, all input / output data, internal potentials, and the like are initialized before prediction (step S1 in FIG. 1A), but the prediction in the third step (S2) in the present embodiment is performed without initializing in advance. The prediction is performed while maintaining the state of the network, such as the network weight, input / output data, internal potential, etc., stored at the end of the second step (S2A).

Thus, the discontinuous point generated at time t = 0 in FIG. 3A by the conventional method becomes a virtual continuous point as shown in FIG. 3B, and good prediction can be performed. . In other words, in the network of FIG. 4 (Williams & Zipser model) in which a sine wave is input and a sine wave is output, a feedback input at the time of initialization is a value such as 0 or 0.5 in the first step (S1) described above. In the present embodiment, a value such as −0.1 at the end of the second step at the time of the original prediction is given to the feedback input by the pre-processing of the second step (S2A), so that the time series Good prediction can be performed while maintaining the continuity of the leading data. Therefore, it is possible to avoid the prediction failure at the time t = 0.

Next, an embodiment of the present invention will be described. This embodiment is for avoiding accumulation of errors after the start of forecasting, and is particularly suitable for applications (such as power demand forecasting, average stock price forecasting, etc.) in which actual values are determined by the network each time forecasting is performed. It is.

FIG. 5 shows a case where the output value of the network is directly used as a feedback input according to the prior art (FIG. 5).
FIG. 5A shows the predicted value and the actual value when the output value is corrected as a feedback input according to the present embodiment without being used as the feedback input as it is (FIG. 5B). Output values y0 (t) and y0 (t +
1),... As feedback input as they are, the predicted values y0 (t), y0 (t +
1),... And the actual value (teacher value) T0 (t), T0 (t +
The errors between 1),... Are increasing.

However, as shown in FIG. 6, for example, the predicted values y0 (t + 1) and y1 (t +
1) Not the actual values but their actual values (teacher values) T0
(T + 1) and T1 (t + 1) are fed back to 1
If input as T0 (t) and T1 (t) before the cycle, the predicted values y0 (t) and y0 (t +) as shown in FIG.
1),... And the actual value (teacher value) T0 (t), T0 (t +
There is no possibility that the error between 1),.

FIG. 7 is a flowchart showing the processing procedure of this embodiment. That is, in the present embodiment, a first step (S21) of inputting data, a second step (S22) of actually performing prediction and outputting a predicted value, and a third step (S23) of determining whether there is an error. And a fourth step (S24) of correcting and optimizing the feedback input when there is an error.

The processing contents of each step in this embodiment will be described below. (1) First Step (S21): Data Input In this step, external data and feedback data are input to the input layer of the recurrent network. In this case, a predicted value output from the recurrent network is used as it is as feedback data.

(2) Second Step (S22): Output of Predicted Value In this step, the network is operated to output a predicted value.

(3) Third Step (S23): Determination of presence or absence of error It is determined whether or not there is an error between the predicted value and the actual value as shown in FIG. When there is an error (or exceeds the allowable value), the process proceeds to the next fourth step, and when there is no error (or below the allowable value), the process returns to the first step.

(4) Fourth Step (S24): Optimization of Feedback Input When it is determined that there is an error in the third step, the output value (predicted value) of the network is not directly used as the feedback input, but the actual value or the like is used. And optimize it as a feedback input.

As a result, the relationship between the predicted value and the actual value is shown in FIG.
As shown in FIG. 5B, as shown in FIG. 5A, the error is prevented from expanding every time prediction is performed, and highly accurate prediction can be performed.

[0024]

【Example】

(1) The embodiment of the invention according to claim 1 The learning data is as shown in Table 1, and the sine wave is
A learning algorithm for converting to s-wave was used. In Table 1, t = 0, π / 18, 2π / 18,.
(108 data).

[0025]

[Table 1]

The structure of the recurrent network used for learning and prediction is as shown in FIG. Table 2 shows the learning conditions.

[0027]

[Table 2]

FIG. 9 shows a prediction result according to this embodiment. FIG. 9A shows an output value, and FIG. 9B shows an error.
From these figures, it is apparent that a large error appears at the prediction start time in the related art, but according to the present embodiment, good prediction can be performed with a small error from the prediction start time.

(2) Embodiment of the Invention According to Claim 2 Table 3 shows learning data and Table 4 shows prediction data.
Since values such as experimental values usually include measurement errors and may differ from theoretical values, in this embodiment, the learning data includes a slight noise. Here, the Williams & Zipser model was used as the recurrent network model.

[0030]

[Table 3]

[0031]

[Table 4]

The maximum value (MAX), the minimum value (MIN), the average value (ABG) of the learning error, when the feedback input is not corrected by the conventional technique and when the feedback input is corrected (optimized) according to the present embodiment. Table 5 shows the moving average (ABSAVG) and the standard deviation (STD). One set of the number of learning data is 108 data.

[0033]

[Table 5]

As apparent from Table 5, one set (10
(8) data and 10 sets (108
According to the present embodiment, the prediction error can be reduced in any of the cases where the learning is performed using the (0) data.

[0035]

As described above, according to the first aspect of the present invention, it is possible to reduce an error that often appears at the beginning of time-series data. According to the second aspect of the present invention, it is possible to reduce errors that are enlarged and accumulated as prediction is repeated.

For example, when predicting the power demand by the recurrent network, the power demand in the spring of this year may be predicted by learning in advance using the spring data of the past several years. Conventionally, the predicted value was not stable during the first few days of the spring of this year, and high-precision prediction could not be performed. According to the invention of claim 1, the prediction is performed from the first day of the prediction period. be able to. In addition, when the invention of claim 2 is used for the prediction of the power demand, a highly accurate prediction can be made by using the actual value found every day for the feedback input instead of the predicted value having an error.

[Brief description of the drawings]

FIG. 1 is a flowchart showing a conventional prediction processing procedure using a recurrent network (FIG. 1A) and a processing procedure of an embodiment of the present invention (FIG. 1B).

FIG. 2 is an explanatory diagram of a predicted periodic function.

FIG. 3 is an explanatory diagram of a periodic function predicted according to the related art (FIG. 3A) and the embodiment (FIG. 3B) of the first embodiment of the present invention;

FIG. 4 is a diagram showing a structure of a recurrent network used in the first embodiment of the present invention;

FIG. 5 shows a case where the output value is directly used as a feedback input according to the prior art (FIG. 5 (a)) and a case where the output value is corrected and used as a feedback input according to the embodiment of the present invention (FIG. 5 (b)). It is a figure which shows the predicted value and actual value of)).

FIG. 6 is a conceptual diagram showing a process of correcting a feedback input in the embodiment of the second invention.

FIG. 7 is a flowchart showing a processing procedure in the embodiment of the second invention.

FIG. 8 is a diagram showing the structure of a recurrent network used in an embodiment of the present invention.

FIG. 9 is a diagram showing an output value (FIG. 9 (a)) and an error (FIG. 9 (b)) of the embodiment according to the first aspect of the present invention and a conventional technique, respectively.

FIG. 10 is a conceptual diagram of a Williams & Zipser model.

FIG. 11 is a conceptual diagram of an Elman model.

Claims (2)

[Claims] 1. A prediction method for performing prediction using a recurrent network having feedback coupling and learning time-series data, comprising: a first step of initializing the network; A second step of performing prediction by a network and storing the state of the network at the time of completion of the prediction, and a third step of performing prediction by the network from the state of the network stored in the second step. Forecasting method using a recurrent network. 2. A prediction method for performing prediction using a recurrent network having feedback coupling and learning time-series data, comprising: a first step of inputting external data and feedback data to the network; and a second step of performing prediction using the network. And a third step of detecting an error of a predicted value output from the network; and a fourth step of correcting feedback input data of the network according to the error. .