JPH03122754A - Learning method for neural network model - Google Patents

Abstract

PURPOSE:To easily attain application control by regarding the weight of synapses as the expression of impulse response functions and allowing synapse groups having the same time relation which are the synapse groups corresponding to the same impulse response function to learn so as to be the value of the same weight. CONSTITUTION:The synapse groups A1,A2,A3..., B1,B2,B3..., C1,C2,C3..., corresponding to the same impulse response function are the synapse 3 group having the same time relation. Respective synapses 3 having the same time relation are repeatedly learned so as to have the same weight. Consequently, the weight of the synapses 3 can be automatically determined and a process model can easily be identified by a neural network model.

Description

[Detailed description of the invention]

〔table of contents〕 overview Industrial applications Conventional technology (Figures 7 to 9) Problems that the invention aims to solve Means to solve the problem (Figure 1) action Example (Figures 2 to 6) Effect of the invention

【overview〕

Regarding the learning method of neural network models, we will clarify the meaning of synapse weighting in neural networks, facilitate learning control of weighting, and
The purpose is to realize a neural network that enables processing of time series data and easy application control.The input layer consists of multiple input layer units each input with time series data, and is connected to each input layer unit by a synapse. A neural network consists of an intermediate layer consisting of multiple intermediate layer units to which intermediate time-series data is input, and an output layer unit connected to each intermediate layer unit by a synapse, and each synapse is connected based on causality. In the model learning method, synaptic weights are regarded as expressions of impulse response functions, and synaptic groups that correspond to the same impulse response function and have the same time relationship are trained so that they have the same weight value. Configure. [Industrial Application Field] The present invention relates to a learning method for neural network models, and more specifically, it is used in various fields of control and information processing. This invention relates to a learning method for neural network models that allows synaptic weighting to be easily performed through group learning. [Prior Art] Neural networks have traditionally been known as networks that imitate the human brain. The individual components of such a neural network have functions similar to those of human nerve cells (neurons). Nerve cells, that is, neurons are the basic unit, and these are tightly connected to form a neural network, and signal transmission from neuron to neuron is performed by synapses. Figure 7 is a diagram showing a conventional neural network (an example without a middle layer), where 1 is an input layer (input layer group), 2
indicates an output layer (output layer group), and 3 indicates a synapse. A large number of input signals (input signals) are input to the input layer 1, and are transmitted to the output layer 2 via the synapse 3 to become output signals (output signals). In this case, spatial data is given as the input signal, and predetermined processing (conversion) is performed by the connection of the synapse 3. Such a network without an intermediate layer is also called a bersebutron type network. Figure 8 shows a conventional neural network (with one middle layer).
(one example), and the same reference numerals as in FIG. 7 indicate the same things. In the figure, 4 indicates an intermediate layer (intermediate layer group). This neural network is a network in which 1M intermediate layers 4 are interposed between an input layer 1 and an output layer 2,
A synapse 3 is connected between the input layer 1 and the intermediate layer 4 and between the intermediate layer 4 and the output layer 2. In this case as well, the input signal (input signal) is input to the input layer 1, passes through the synapse 3, and is transmitted to the intermediate layer 4.
Furthermore, it is outputted to the output layer 2 through the synapse 3 and becomes an output signal (output pattern). Furthermore, spatial data is given as the input signal. FIG. 9 shows a conventional neural network (an example with two intermediate layers), and the same reference numerals as in FIG. 8 indicate the same parts. In this example, two intermediate layers 4 are provided between the input layer 1 and the output layer 2, respectively.
and the output layer 2 are connected by a synapse 3. Note that the above-mentioned intermediate layer 4 is not limited to 2N, but may include layers of more than 2N. In the case of the hierarchical network structure having the intermediate layer 4 as described above, each unit in the hierarchical network is connected in the direction from the input layer l to the intermediate layer 4 and from the intermediate layer 4 to the output layer 2 (forward), and within each layer and output layer 2
There are no connections from to input layer 1. The weighting of the connections between uni nodes in each layer is determined, for example, by learning such as the backpropagation method, and a certain conversion rule determined from this weight is applied to the input signal (input cover pattern), and the output layer 2 The converted output signal (output signal) is output. [Problems to be Solved by the Invention] The conventional devices as described above have the following drawbacks. (1) Conventional neural networks have been thought to be unable to adequately explain the learning results of synaptic weights (there are studies that create rule bases from synaptic weights). (2) Conventional neural networks handle only spatial data; none handle time-series data. That is, input data (input signals) are regarded as values in a multidimensional space and treated as spatial data. Then, this 0 space data is mapped to output data by a neural network, and the neural network is used to classify similar items. Even when dealing with time-series data, it is considered spatial data and processed as described above. For example, when 10 pieces of past data are used to determine conditions such as uphill, downhill, flat, etc., or when determining the amount of operation to achieve a control target value (this is , which corresponds to creating an inverse model of the controlled object using a neural network). The present invention eliminates these conventional drawbacks, clarifies the meaning of synapse weighting in neural networks, facilitates learning control of weighting, enables processing of time series data, and facilitates application control. The purpose of this study is to realize a neural network. [Means for Solving the Problems] FIG. 1 is a diagram showing the principle of the present invention, where l indicates an input layer, 2 indicates an output layer, 3 indicates a synapse (synapse group), and 4 indicates an intermediate layer. In addition, 1-1.1-2.1-3... is each unit of the input layer, 2-1.2-2- is each unit of the output layer 2, 4
-1.4-2.4-3・- is each unit of the middle layer, uz
(t-1), u1 (t-2). u l(t −3)− is time series input data, M t
(t-1), M+(t-2), M1(t-3)...
...- is the time series data of the middle layer, y s (t), y
*(t)・〜indicates output data. The neural network model shown in the figure has an intermediate layer 4 between an input layer 1 and an output layer 2, and each layer is a network connected by synapses 3, and each unit 1-1.1 of the input layer l. -2.1-3- respectively have time series data u1 (t1). u z (t-2), u + (t-3) ...- is done manually. Here, t indicates time, and u s (t −2) is u1(
t-1), 111 (t-3) indicates data older than ul(t-2), and M t (
The same relationship is shown for t −1), −=M t (t −2)−. In the present invention, the time series data u1(t-1)+us(
t-2), u 1(t-3) - are regarded as time series data (time data), and the weights of the neural network are regarded as impulse response functions of modern control theory, thereby creating an impulse response model. This is achieved using a neural network, and its learning method is as follows. That is, the synapses are connected based on the causal law in the impulse response of the control theory described above, and the weight of the synapse 3 of the neural network is regarded as an expression of the impulse response function, and is determined by learning. In this learning, groups of synapses corresponding to the same impulse response function (groups of synapses having the same time relationship) are trained to have the same weight value. For example, A in the figure! , At, A3- are synapses corresponding to the same impulse response function and have the same time relationship (ul(t-1) and M1(t-1), Ul(t-2) and M1(t-2) ...- is data at the same time and has the same time relationship). Furthermore, Bx and Bx- are also synapses corresponding to the same impulse response function and have the same time relationship (u-1(t-
2) and M s (t -1), when viewed from M1 (t 1), ul (t - 2) is the data at the previous time, and ur (
When viewed from M1 (t-2), ul (t-3) is data at the previous time, and they have the same time relationship). Therefore, A1. A s + A S...- are given the same weight, Bl, B2 . B3.- are given the same weight, and the synapses between units having the same time relationship are learned in the same way so that they have the same weight (average value, maximum value, etc.). [Operation] As described above, the present invention clarifies the content of learning by regarding the meaning of synaptic connections in a neural network model as an impulse response function. In this learning, the average value or maximum value of output data is determined for synapse groups corresponding to the same impulse response function, and learning is repeated so that all the synapses have the same weight value. The synapse group corresponding to the same impulse response function above is
A1. A synapse group consisting of At, As..., a synapse group consisting of Bx, Bx..., Ct. A synapse group consisting of C2...-, etc., and these are synapse groups having the same time relationship. Then, these synapses having the same time relationship are repeatedly trained so that they have the same weight. If learning is performed in this manner, synaptic weights can be automatically determined, and process models can be easily identified using a neural network model. [Example] Hereinafter, an example of the present invention will be described based on the drawings. FIG. 2 is an overall diagram of a neural network model according to an embodiment of the present invention, where l is an input layer, 2 is an output layer,
3 indicates a synapse, and 4 indicates an output layer. ' I + u g ---131 is a vector of time series data input to the input layer, Ml, , Ml t-
J@ is the vector of time series data of the middle layer 4, '/l
+'Iz-'/n indicates a vector of time-series data of the output layer 2. The above vector ' l + u z-134 is a plurality of time series data, u r (t-1), u 1 (t-
2), u 1 (t-3) + u 1 (t-4) - u 1
(t-1), u t(-2)+u 5(t-a)+u
*(t-4) −u s(t −1), u s (t
-2) ---uz (t -1), u (t -2)
−−−−, and the vectors MI+, MI
t −'PI/111m1Y11 Y z---'/n
The same applies to Between the input layer 1 and the middle layer 4, and between the middle layer 4 and the output layer 2,
They are connected by synapses 3, and these connections are made according to the law of cause and effect. FIG. 3 is a partially detailed diagram of the neural network model shown in FIG. 2. The vector u I + u Z shown in FIG. 2 is the time series data u 1 (t − IL ut (
t 2L ul(t-3Cu 1(t-4)-u +
(t-k), u 1 (t-k -1)---u s (
t −i), u 1 (t −1) + u 2 (t −2)
---u 1(t j ), vector Ml
l. Mlt is time series data M 1 (t −1), M t
(t2). Ms(t-3), M1(t4), --M1(t
+1). It consists of Ml(t-k), M1(t-1), -Ma(t-1). These time series data are treated as time series data as they are, and the synaptic weights of the neural network are determined by learning by considering them as impulse response functions (functions based on modern control theory). Moreover, the connection is determined based on the causal law in impulse response. As a result, as shown in the figure, synapses between units with the same time relationship are determined to have the same weight without being connected to the past ones when viewed from the input side. For example, between us (t-1) and M t (t-2), U
As between 1(t-2) and M1(t-3), from the input side, there is no connection to anything in the past in time (based on the middle layer, there is no connection to future input) . Also, between us(t-1) and M1(t-1), uIC
L-2) and M 1 (t-2) have the same time relationship, so they are given the same weight (all parallel synapses shown in the figure are given the same weight). Furthermore, between ur(t-2) and Ml(t-1), ul(-
3) and Ml(t2), etc. are also given the same weight. Note that the input time series data u1 is the time series data M of the middle layer.
On the other hand, if only time T is affected, t = +T, and ul(L-j) is calculated from Mx(t-j) (if j>k, from Mr(t-k)) Ml(t- It is sufficient to connect up to j+T) (however, if j-T≦0, up to Mx(tl)). Note that the present invention is applicable not only to the above example but also to cases where there are a plurality of intermediate layers. 4 to 6 are explanatory diagrams of a neural network learning method. In the present invention, as described above, time series data u x(t-t)+u 1(t-2Cu 1(t-
3), ux(t4)...- are treated as time series data as is.1. Furthermore, the weights of the neural network are considered as impulse response functions of modern control theory, and the impulse response model is realized by the neural network. That is, the synaptic weights of the neural network are regarded as expressions of impulse response functions, and are determined by learning. Furthermore, synaptic connections are made based on the causal law in the impulse response of the control theory described above. In such a neural network model, since it is possible to explain the learning content, the way synapses are connected is constrained (default) in correspondence with time-series data. First, in the neural network model, synapse groups corresponding to the same impulse response function are trained to have the same value during learning. For this learning, for example, a bank propagation method as shown in FIG. 4 is used. This method introduces error feedback into a hierarchical network consisting of an input layer, intermediate layer, and output layer as shown in the figure, and based on this error feedback (feeding back the error E between the output of the network and the teacher signal), It adjusts the synaptic weights (connection weights) so that the network learns the association between the input signal (input cover turn) and the output signal (output cover turn) and achieves appropriate data processing. be. In addition to the above-described backpropagation method, various known learning methods can be applied. When applying the above learning method, as shown in FIG. 5, synapses between the input layer and the intermediate layer that have the same time relationship are represented by the following symbols. (1) Between ul(t 1) and M l(t −1),
Between uI(t-2) and M+(t-2), u t (t -
3) and Ml(t-3), between ul(t-4) and Ml(t
-4) between u1(t-5) and M! The synapses between (t-5) and - are A 1 and A !, respectively. ,
A s, A 4. A s - (these are collectively referred to as A). (2) Between us(t-2) and Ml(t-1), ul(
between t-3) and M1 (t-2), between us (t-4) and M s
(-3), between ux4t-5) and Mx(t-4)
- synapses Bl, Bl, Bs, B4...
・(These are collectively referred to as B). (3) Between ul(t-3) and Ml(t-1), ul(
t4) and M t (t -2), between ux (t-5) and M
The synapses between s (-3) are CI, CI, and C3- (these are collectively referred to as C). In the same way, DI items that have the same time relationship. D2. Ds..., E plate * E *, '-. The above A (As, A +As・-), B (B t. 131, Bl---), C (Cs, C2, C3-)
, D (DI+ DI, Ds-), E (E 1.
E! , Es. A: u i (t-n)-*M i (t-n)
n xl, l'---1(. B: u 1(t-n)-+M 1(t-n+ 1)
n=2.3'=-k. C: u 1(t-n)-+Mi(t-n+2)n=3
．． 4.-k. D: u1(t-n)→Mi(t-n+3)n=4
+E to +5−−: u 1(t−n)−+Mi(t−
n+4) n=5.6---k. Similarly, F, G・- (however, i=1°2.
3-). For example, if i = 1 (first group of input layer), then A: ut (t - n) - M i (t - n)
n=1+2-k. B: u 1(t-n)-eM】(t-n+1) n
=2+3-k. C:ul(t-n)=Mx(t n+2)n=3+4
---, and if i=2, then A: u2(t-n)→M*(t-n) n=1
．． 2°−k. B: u1(t-n)-+M! I(t-n+1)n=2
+3-LC: us(t-n)-M*(t-n+2)n
"3.4-k." Figure 6 is an explanatory diagram of the impulse response function between the input layer and the output layer. The curve in the figure shows the impulse response function of the input Wiu and the intermediate layer M. , Illustrated (7) ft, fs, fs, f4...
- indicates each element of the impulse response function. As shown in the figure, synapses A, B, C, . . . (parallel lines in the figure) that have the same time relationship between the input layer, the intermediate layer, and each unit become elements of the same impulse response function. That is, A corresponds to fl, B corresponds to fl, and C corresponds to fs. Therefore, in the above neural network model,
When learning synapse groups corresponding to the same impulse function using the bank propagation method, etc., the above A (A
s, Ai, A3-), B (Bs. B included, B s-), C (Cs, C2, C3-)
) - are trained so that they each have the same value. In this case, synapses that correspond to the same impulse response function (with the same time relationship) are trained to have the same weight by using the average value or maximum value of the obtained values. . [Effects of the Invention] As explained above, the present invention has the following effects. (1) Once a process model is created using a neural network, the impulse response function, that is, the synaptic weights (parameters) are automatically determined through learning. (2) Learning content can be explained by considering the meaning of synaptic connections as an impulse response function (synapses can be explained). (3) Compared to conventional neural networks in which synapses are randomly connected, the convergence of learning can also be guaranteed (this can be done by modifying the qualitative model). (4) Process models can be easily identified using neural networks. (5) A neural network model that handles time-series data can be easily realized.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the principle of the neural network model learning method according to the present invention, FIGS. 2 to 6 are diagrams showing one embodiment of the present invention, and FIG. 2 is an overall diagram of the neural network model. Figure 3 is a partial detailed diagram of the neural network model, Figure 4 is an example of learning using the bank propagation method, Figure 5 is an explanatory diagram of the learning method (explanation of synapses), and Figure 6 is an explanatory diagram of the learning method. (Explanation of impulse response function). Figures 7 to 9 are diagrams showing conventional neural networks. Figure 7 is an example with no intermediate layer, Figure 8 is an example with one intermediate layer, and Figure 9 is an example with two intermediate layers. It is a figure showing a certain example. ■...Input layer 2-Output layer 3...Synapse 4-Middle layer

Claims (1)

[Claims] Time series data {u_1(t-1), u_1(t-2).
...) respectively input a plurality of input layer units (1
-1, 1-2, 1-3...) input layer (1)
are connected to the input layer units (1-1, 1-2...) through synapses (3), and intermediate time series data {M_1(t-1), M_1(t-2), M_1
(t-3)...} consists of a plurality of middle layer units (4-1, 4-2, 4-3...) input
4), and an output layer unit (4-1, 4-2, . . . ) connected to each of the intermediate layer units (4-1, 4-2, etc.) by a synapse (3).
2), in a learning method for a neural network model in which each of the synapses (3) above are connected based on the law of causality, the weight of the synapse (3) above is regarded as an expression of an impulse response function, and Corresponding synapse groups that have the same time relationship are given the same weight value ("A_1, A_2, A_3...", "B_1,
A method for learning a neural network model, characterized in that learning is performed by giving the same weight to each of "B_2, . . ." and "C_1 . . .".