JPH06301661A - Leaning system for parallel connection type recurrent neural network - Google Patents

Abstract

PURPOSE:To provide a parallel connection type recurrent neural network(RNN) learning system capable of executing learning by an error backward propagation(BP) method. CONSTITUTION:Learning algorithm by the BP method based upon a complete connection type RNN status equation is introduced to the parallel connection type RNN. In the parallel connection type RNN parallel-connecting three units respectively consisting of a neuron 1 and an output neuron 2, a neuron 3 and an output neuron 4, and a neuron 5 and an output neuron 6 and completely connected by pair connection, the window width of teacher data (obtained by the learning of the 1st stage unit) relating to the 3rd stage unit and after is changed and transfer from each stage to the succeeding stage is executed in accordance with a prescribed evaluation function to attain the execution of learning based upon the BP method.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a learning method for a parallel connection type recurrent neural network (hereinafter referred to as a parallel connection type RNN) in which a plurality of units formed by completely connecting a plurality of neurons are connected in parallel. More specifically, the present invention relates to a learning method for a parallel-coupled RNN using the error back propagation method.

[0002]

2. Description of the Related Art Conventionally, units constituting a network of a recurrent neural network (RNN) are randomly connected as shown in FIG. This is because each unit is composed of a dynamic non-linear unit, and its operation forms a predetermined time series pattern represented by the following differential equation (state equation / Equation 1, Equation 2). .

[0003]

[Equation 1]

[0004]

[Equation 2]

Here, x _i , y _i , and X _i respectively represent the internal state, output, and external input of the i-th unit at time t, and w _i j is the i-th unit from the j-th unit. It represents the connection load to the unit. Also, τ _i is i
Is the time constant of the th unit, N is the total number of units. Further, the characteristic function f (x) is usually a nonlinear sigmoid function f (x) = tanh (x) or a linear function f (x).
= X is selected.

As shown in FIG. 8, there is a complete coupling type RNN in which the respective neuros are adjacent to each other and are coupled to each other. The operation of this fully-coupled RNN is
It is represented by the following general equation of state (Equations 3 and 4).

[0007]

[Equation 3]

[0008]

[Equation 4]

Here, x _i , y _i , and X _i respectively represent the internal activity, output value, and external input at the i-th neuro, and w _ij is the coupling coefficient from the j-th neuro to the i-th neuro. Represents Further, τ is a time constant of internal activity, and N is a total neuron number. Furthermore, the characteristic function f (x)
Here, again, a sigmoid function or a linear function is selected.

Since the state of the fully-coupled RNN is represented by a differential equation, it has excellent processing ability for time-series patterns and has the property of self-excited oscillation even when there is no external input. In particular, if the characteristic function is a linear function, it is possible to approximate a composite damped sine wave (Equation 5) having N frequency components if there are 2N neuros.

[0011]

[Equation 5]

On the other hand, as the RNN, there is a parallel coupling type in which a plurality of units formed by completely coupling a plurality of neuros are further connected in parallel. That is, the parallel-coupled RNN is
When M and N are natural numbers of 2 or more, a fully coupled RNN composed of N neuros is used in M stages.
For example, the one shown in FIG. 9 has a configuration in which three stages of completely coupled RNNs each composed of three neuros are used. 1 in the fully-coupled RNN in each stage in the parallel-coupled RNN
The frequency is extracted from the output unit, and the difference value between the teacher signal input at this time and the output signal is used as the teacher signal data for the next stage. Further, in the final stage, a combining operation is performed to approximate a composite damped sine wave having N frequency components.

[0013]

By the way, an error back-propagation method (hereinafter referred to as BP method) based on the steepest descent method is used for the learning rule of the fully coupled RNN. The basic equation of this BP method is represented by the following equation (Equation 6).

[0014]

[Equation 6]

Here, E [w (t)] is the evaluation function in the state of the coupling coefficient w (t) at time t, y (t) is the output value of the output unit at time t, and Y (t) is the time. t
Is a teacher signal in. Here, the BP method is to update the coupling coefficient w _ij so as to minimize the evaluation function E according to the following equation (Equation 7).

[0016]

[Equation 7]

In this equation, the initial value of the coupling coefficient is given by a small random number.

In the fully-coupled RNN, since all the other neurones are coupled to the output neuron, the error is propagated to all the neurones in the BP method. Therefore, the fully-coupled RNN can fully exert its ability by using the learning rule called the BP method.

Therefore, in the case of the parallel-coupled RNN, the BP method can be adopted as the learning rule of the fully-coupled RNN itself constituting each stage. According to the BP method, the output at t = 0 in the fully-coupled RNN is obtained. The output of the unit and the teacher data always match in the initial state, and the complete connection type RN from the second stage onwards
The output of the output unit at t = 0 in N is y (0) =
It will be 0. That is, in the case of the parallel connection type RNN, t
In the operating state of> 0, all the outputs of the output units of the second and subsequent stages become y (t) = 0, so that there is an inconvenience that learning by the BP method cannot be performed.

The present invention has been made to solve such a problem, and a technical problem thereof is to provide a learning method of a parallel connection type RNN capable of performing learning by the BP method.

[0021]

According to the present invention, a plurality of units including output neurons are completely connected to each other in parallel, and a plurality of units are connected in parallel. A learning method of a parallel-coupled RNN used, wherein the BP method is based on a state equation of a fully-coupled RNN, and in the initial state, an output signal of a first-stage unit among a plurality of units and an input teacher signal. And the window width of the teacher data referred to in the unit of the second stage is changed to form the initial point. A learning method of a parallel combination type RNN is obtained in which the transition is performed according to a predetermined evaluation function when the evaluation function is less than the predetermined evaluation value.

Further, according to the present invention, there is provided a parallel-coupling type RNN having a learning rule of the BP method, which is constituted by connecting in parallel a plurality of units each of which is formed by completely coupling a plurality of neurons including an output neuron. can get.

[0023]

EXAMPLES Examples are given below to illustrate the parallel connection type R of the present invention.
The NN learning method will be described in detail with reference to the drawings. FIG. 1 shows a basic configuration of a parallel coupling type RNN which is an embodiment of the present invention.

This parallel-coupled RNN has a neuro 1 and an output neuro 2, a neuro 3 and an output neuro 4,
It is configured by connecting three perfectly coupled RNNs (units) paired with a neuro 5 and an output neuro 6 in parallel. In this parallel-coupled RNN, the output neuro 2 of the first stage unit is connected to the data access terminal P1, the output neuro 4 of the second stage unit is connected to the data access terminal P2, and the output neuro 2 of the third stage unit is connected. The output neuro 5 is connected to the data access terminal P3. Further, teacher data (E) to be described later is provided between the data access terminals, that is, between P1 and P2 and P2 and P3.
They are respectively connected to transmit error 1 and error 2).

This parallel-coupling type RNN is configured to introduce the BP method. Therefore, in the initial state, based on the state equation of the fully-coupled RNN, it is obtained by the difference between the output signal related to the first-stage unit and the teacher signal input at that time, and the second-stage unit An initial point is obtained by changing the window width of the referenced teacher data and shifting the window width by a predetermined time. This method of changing the window width is such that when the teacher data obtained in the first stage unit is t> 0, first, d [Error1 (T)] / dt
The time when = 0 is set as the initial point of the second-stage unit. If the time (initial point) thus obtained is set to t = 0, the learning algorithm based on the BP method can be executed.
In addition, the second and subsequent units are operated from the initial point in the progressing state, and the transition to the next stage in each stage is performed when the evaluation function E is less than the predetermined evaluation value E1.
It should be noted that a different evaluation value E2 is obtained from the differentiation of the evaluation function E.
If, for example, the case of dE / dn <ε ₂ is further used in combination with the determination of E <ε ₁ described above as a guideline for the transition, the execution of learning can be further stabilized.

Then, a learning algorithm in the parallel-coupling type RNN according to the embodiment of the present invention will be described with reference to the flow chart shown in FIG. In this learning method,
First, in the state equation (formula 3) of the fully coupled RNN,
The required coupling coefficient is initialized (usually given by a small random number), and teacher data is extracted from the first-stage unit (step S1). Next, d [Error1 (T)] / dt in the case of t> 0 of this teacher data
The initial point is set (step S2) by setting the time that satisfies = 0 as the initial point.

Subsequently, the RNN learning routine (step S3) by the BP method is performed from the second stage unit by operating from the initial point. Next, the evaluation function E used in the RNN learning routine is E <ε ₁ or E ′ (differential of E)
It is determined whether or not <ε ₂ (step S4). As a result, if the condition is not satisfied, the routine returns before the RNN learning routine (step S3) to continue the learning, but if the condition is satisfied, As M = M + 1 (step S5), the process proceeds to the unit of the next stage (third stage). This allows
Teacher data (Error2) is transmitted to the unit of the third stage. At the time of shifting to the next stage, it is judged whether or not the final stage is reached (step S6). If the final stage is not reached, the RNN learning routine is performed so that the unit immediately before the final stage can be continuously learned. Although it returns before (step S3), if it has reached, a synthesis routine (step S7) is performed and then learning is completed.

FIGS. 3 to 5 show the relationship between the teacher signal and the output signal in each stage of the unit according to this learning method, and FIG. 3 relates to the first stage unit,
FIG. 4 relates to the second-stage unit, and FIG. 5 relates to the third-stage unit. FIG. 3 shows the time lag of t = 0 in the first-stage unit due to the setting of the initial point accompanying the change of the window width with respect to the teacher data. FIG. 4 shows that the composite damped sine wave is approximated based on the evaluation function by the operation of the initial point, and FIG. 5 shows that the composite damped sine wave is further approximated. There is. From these FIGS. 3 to 5, according to this learning method, even in the parallel-coupled RNN, the BP
It turns out that the law can be adopted as a learning rule.

FIG. 6 shows the relationship between the synthesized waveform and the original waveform, which is the final processing of this learning method. From FIG. 6, it can be seen that even the final synthesized waveform can be sufficiently learned.

In the embodiment, the example in which the neuron and the output neuron are fully coupled in a pair configuration and three stages are connected in parallel is shown. It is also applicable when M stages are used.

[0031]

As described above, according to the present invention, the window width of the teacher data relating to the second and subsequent units is changed in the parallel coupling type RNN in which a plurality of units of the completely coupling type RNN are connected in parallel. In addition, by making the transition to the next stage in each stage follow the evaluation function,
It is assumed that learning by law can be performed with high ability. In particular, in the case of a parallel connection type RNN in which a plurality of units in which the neuro and the output neuro are completely connected are connected in parallel, the number of connections between the respective neuros is small, which can contribute to a reduction in the number of memories for the computer to be used. As learning ability is guaranteed, it is expected to make a great contribution to the computer industry.

[Brief description of drawings]

FIG. 1 shows a basic configuration of a parallel coupling type RNN according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a learning algorithm related to the parallel combination type RNN shown in FIG.

FIG. 3 is a waveform diagram showing a relationship between a teacher signal and an output signal related to the first-stage unit of the parallel coupling type RNN shown in FIG.

FIG. 4 is a waveform diagram showing a relationship between a teacher signal and an output signal related to the second-stage unit of the parallel coupling type RNN shown in FIG.

5 is a waveform diagram showing a relationship between a teacher signal and an output signal related to the third-stage unit of the parallel-coupling type RNN shown in FIG.

6 is a diagram showing a relationship between a synthesized waveform and an original waveform according to the parallel-coupled RNN shown in FIG.

FIG. 7 is a diagram showing a unit configuration in a conventional RNN.

FIG. 8 is a diagram showing a basic configuration of a conventional fully-coupled RNN.

FIG. 9 is a diagram showing a basic configuration of a conventional fully-coupled RNN (also applicable to the present invention).

[Explanation of symbols]

 P1, P2, P3 data access terminals

Claims (2)

[Claims] 1. A learning of a parallel connection type recurrent neural network which is constructed by connecting a plurality of units including a plurality of output neurones to be completely connected in parallel in parallel and which uses an error backpropagation method as a learning rule. The error backpropagation method is based on a state equation of a fully-coupled recurrent neural network, and in the initial state, outputs an output signal of a first-stage unit of the plurality of units and an input teacher signal. And the window width of the teacher data referred to in the unit of the second stage is changed to form the initial point, and in the progressing state, it operates from the initial point and shifts to the next stage in each stage. According to a predetermined evaluation function is performed when the evaluation function is less than a predetermined evaluation value. Practice method. 2. A parallel-type recurrent neural network which is configured by connecting a plurality of units, each of which is a complete combination of a plurality of neurons including an output neuro, in parallel, and which uses an error back-propagation method as a learning rule. network.