JPH0554163A - Neural network and learning method thereof - Google Patents

Abstract

PURPOSE:To provide the efficient learning method by making the property of the neural network clear by analyzing the frequency of the neural network. CONSTITUTION:In the case of calculating constants ai, Wi and thetai so as to minimize the error of output data to teacher data f(x), first of all, Fourier transformation F(k) of the teacher data f(x) is calculated (S1). Next, a sampling interval T is decided, a sequence gn is generated from gn=nT.F(nt) (S2), and a sequence pi satisfying gn+p1gn-1+p2gn-2+...+pNgn-N=0 is calculated from the sequence gn (S3). Afterwards, a resolution Zi is calculated by resolving xN+p1xN<-1>+p2 xN<-2>+...+pN=0 from the sequence pi (S4) and next, the constants Wi and thetai are calculated from Wi=-T/log¦Zi¦ and thetai=arg(Zi)/log¦Zi¦ (S5). Then, tan<-1>(Wix+thetai) is calculated from the constants Wi and thetai by replacing the output of an intermediate layer, the constant ai is calculated by a method of least squares, and the processing is finished (S6).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network, and can be applied to all fields to which a neural network can be applied, such as voice recognition, voice synthesis, character recognition, robot control and stock price prediction.

[0002]

2. Description of the Related Art In recent years, studies on voice / image recognition and time series prediction using neural networks have been actively conducted, and their effectiveness has been confirmed. In addition, hardware that implements neural networks has begun to be sold, and it has been applied to various products and is about to be put into practical use. For example, Watanabe and Yoneyama et al.'S "Ultrasonic three-dimensional object recognition method using neural network" (Shingaku Giho, US90-29 (1990)) gave quite good results. Is under consideration.

The learning problem of the neural network is formulated as follows. That is, the learning problem of the three-layer perceptron type neural network is that the given real function f (x) called the teacher data is linearly summed with a predetermined monotonically increasing function σ (x).

[Equation 1] However, a _i : Synapse load of the output layer (i = 1,2, ..., N, N is the number of neurons in the intermediate layer) w _i : Synapse load of the intermediate layer θ _i : Threshold value of the intermediate layer ε (x): This is a problem that develops into error. That is, the constants a _i , w _i and θ _i are set so that the function ε (x) becomes the smallest under a predetermined criterion.
Is the problem of seeking. Here, the function σ (w _i x + θ _i )
Represents the output value of the i-th neuron in the hidden layer, and the function σ
As (x), the sigmoid function [1 / (1 + exp (-
x))] is used.

Best constant a for teacher data f (x)
Only the back-propagation method has been known so far as a method for obtaining _i , w _i and θ _i . In error backpropagation,
Calculate the error function E by the following formula

[Equation 2] Using this error function E, the steepest descent method Δa _i = −η∂E / ∂a _i Δw _i = −η∂E / ∂w _i Δθ _i = −η∂E / ∂θ _{i is used} to obtain a constant a _i , W _i , θ _i are obtained.

[0005]

The back-propagation method has made it possible to learn a neural network having three or more layers, which has been difficult so far. However, this method starts from a random initial value and uses the steepest descent method. Since learning is performed, the following inconveniences occur. It takes a huge amount of time to learn. Learning may fall to a local minimum and not progress. I do not understand what features are being learned, and I cannot interpret the learning results. The role of the intermediate layer is not clear and its number must be determined by trial and error. If the number is too small, learning will not proceed, and if too many, over learning will occur.

Therefore, several attempts have been made to improve the error backpropagation method in order to correct these disadvantages. However, none of these methods are essential improvements because they are performed without clarifying the mathematical properties of neural networks. It has also been considered difficult to clarify the mathematical properties of neural networks because of their non-linearity. An object of the present invention is to clarify the properties of a neural network by performing frequency analysis of the neural network and to provide an efficient learning method.

[0007]

SUMMARY OF THE INVENTION The present invention comprises an input layer,
An intermediate layer having a plurality of N units of neurons and an output layer are provided, and the synaptic weight from the input layer to the intermediate layer is w.
_i (where i = 1, 2, ..., N), the threshold value of each neuron in the intermediate layer is θ _i , and the synaptic weight from the intermediate layer to the output layer is a _i
Then, the first processing step for obtaining the Fourier transform F (k) of the teacher data f (x) and the arbitrary sampling interval T are determined and the sequence {g _n } is changed to “g _n = nT · F (nT)”. a second processing step of generating by "," g _n from the sequence {g _n} +
and a third processing step for _{obtaining a} sequence {p _i } satisfying the condition "p ₁ g _n-1 + p ₂ g _n-2 + ... + p _N g _nN = 0"
From the sequence {p _i }, the N-dimensional complex algebraic equation “x ^N + p ₁ x
^N-1 + p ₂ x ^N-2 + ... + p _N = 0 "to obtain a solution {Z _i } and the constants w _i and θ _i from the following equation.
And a fifth processing step for obtaining the following: w _i = −T / log | Z _i ｜ θ _i = arg (Z _i ) / log | Z _i ｜ The output of the intermediate layer is “tan ⁻¹ (w _i x + θ _i )”. Requested from
From the result, each constant w _i , θ _i and a _i is determined by the sixth processing step of obtaining the constant a _i by the least square method so that the error of the output data with respect to the teacher data f (x) is minimized. Is characterized by.

[0008]

According to the present invention, the structure of the neural network is clarified by performing frequency analysis of the neural network, and the learning problem of the neural network is solved by the direct method instead of the iterative method.
What becomes clear as a result of the frequency analysis is that the learning of the neural network results in the problem of linear prediction on the frequency axis. The problem of linear prediction has already been studied deeply in speech analysis methods that have been conventionally used in speech synthesis and the like, and the problem of neural networks can be solved by almost the same method.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a flow chart showing the processing procedure of the present invention. As shown in FIG. 2, the present invention is a three-layer perceptron consisting of an input layer 1, an intermediate layer 2 having a plurality of N neurons and an output layer 3, and a synaptic load from the input layer 1 to the intermediate layer 2 is w _i. (I = 1, 2, ...,
N), the threshold of each neuron in the hidden layer 2 is θ _i , and the hidden layer 2
Constants a _i , w _i , θ so that the error ε (x) of the output data with respect to the desired output value, the teacher data f (x), is minimized when the synaptic load from the output layer 3 to the output layer 3 is a _{i. i}
Is to ask. That is,

[Equation 3] In, the constants a _i , w _i and θ _i are obtained so that the function ε (x) becomes the smallest under a predetermined criterion.

In FIG. 1, first, the Fourier transform F (k) of the teacher data f (x) is obtained. Here, "F (k) ≡ ∫f
(X) exp (ikx) dx ”(step S1).

Then, an appropriate sampling interval T is determined and a sequence {g _n } is generated by "g _n = nTF (nT)". However, n is an integer (step S2).

[0012] Then, determine the sequence {p _i} that satisfies the obtained sequence _{{g n} g n + p} 1 g n-1 + p 2 g n-2 + ···· + p N g nN = 0 ( step S3). The determination of the order N corresponds to the determination of the number of neurons in the hidden layer. There are a covariance method and an autocorrelation method as a method for solving a sequence {p _i } as a linear prediction problem, which will be described later.

Then, an N-dimensional complex algebraic equation x ^N + p ₁ x ^N-1 + p ₂ x ^N-2 + ... + p _N = 0 is solved from the obtained sequence {p _i } to obtain a solution {Z _i }. Obtained (step S4). As a numerical solution for the N-dimensional complex algebraic equation, a fast, accurate and well-known DKA method can be used.

Next, the constants w _i and θ _i are obtained by the following equation (step S5). w _i = −T / log | Z _i | θ _i = arg (Z _i ) / log | Z _i | where “arg (x)” represents the argument of the complex number x.

Then, the output of the intermediate layer, "σ (w _i x +
theta _i) "and" _{^{tan -1 (w i x + θ}} i) "and replacing the constant previously determined w _i and theta" tan ^-1 (w from _{i i} x +
θ _i ) ”, and the constant a _i is calculated from this by the method of least squares (step S6), and the process ends.

In step S6, "σ (x) = tan ^-1 (x)
It is noted that the above learning problem of the neural network can be solved analytically by putting "." In this way, the constants a _i , w _i and θ _i can be directly obtained without repeating as in the conventional case. “Tan ⁻¹ (x)” has a shape very similar to the sigmoid function that is usually used as the output function of a neural network, and using this function does not change the capacity of the neural network.

Next, a series of these processes (steps S1 to S1)
The reason why the neural network can be learned by S6) will be explained by the following theorem. The Fourier transform described below means a Fourier transform as a slowly increasing superfunction. This is because "tan ^-1 (x)" and the sigmoid function cannot be Fourier transformed in the usual sense, but they can be Fourier transformed as a slowly increasing superfunction.

[Theorem] The following three propositions are equivalent. The function f (x) has constants a _i , w _i , θ _i ((w _i , θ
_i ) ≠ (w _i , θ _i )) is used to express as follows.

[Equation 4] The Fourier transform of the function f (x) is calculated as “F (k) ≡ ∫f (x) ex
"p (ikx) dx", "kF (k)" is a constant c _i ,
It is expressed as follows using d _i (d _i is a complex number, d _i ≠ d _j ).

[Equation 5] The Fourier transform of the function f (x) is calculated as “F (k) ≡ ∫f (x) ex
If p (ikx) dx ", then" g _n ≡nT · F (nT) "(where n is an integer and T is an arbitrary constant), then g _n + p ₁ g _n-1 + p ₂ g _{n There} are constants (p ₁ , p ₂ , ..., P _N ) such that _-2 + ... + P _N g _nN = 0 (5) holds. Here, the N-dimensional complex algebraic equation “x ^N + p ₁ x
^N-1 + p ₂ x ^N-2 + ... + p _N = 0 "has no multiple solution. [End theorem]

[Proof] <==>: The Fourier transform of “tan ⁻¹ (wx + θ)” is “j (π / 2)
^1/2 · exp [−k {(1 + jθ) / w}] / k ”.

==>: If “Z _i = exp (d _i T)” is set, then from equation (4),

[Equation 6] Is established. Next, the polynomial S (x) is considered, and its expansion coefficient is set as p _i . S (x) ≡ (xZ ₁ ) ・ (xZ ₂ )… (xZ _N ) ＝ x ^N ＋ p ₁ x ^N-1 ＋ p ₂ x ^N-2 ＋ ・ ・ ・ ・ ・ ・ ＋ p _N … (7) Then, the following value And calculate

[Equation 7] Using equation (6)

[Equation 8] Equation (5) is thus shown. From this proof, the coefficient p ₁ ,
The relationship between p ₂ , ..., P _N and d ₁ was also clarified.

==>: The number sequence satisfying the equation (5) has only one solution because the first N terms determine the remaining terms. Therefore, it suffices to show whether the sequence as shown in the equation (6) can be obtained (c _i can be obtained) by any method of giving the initial value. It has the matrix

[Equation 9] It suffices to show that has an inverse matrix. The determinant of this matrix is given by Π _{i> j} (Z _i −Z _j ), so (Vandermo
The determinant is not zero because the following equation x ^N + p ₁ x ^N-1 + p ₂ x ^N-2 + ... + p _N = 0 does not have multiple solutions. [End of certification]

[System] The constants appearing in the above theorem have the following relationship. d _i = − (1 + jθ _i ) / w _i Also, {exp (d _i T)} is a solution of the following equation x ^N + p ₁ x ^N-1 + p ₂ x ^N-2 + ... + p _N = 0. Is. [End of system]

From the above theorem, the following points regarding the learning of the neural network become clear, and as a result,
The learning algorithm described above can be constructed. Learning the neural network results in a linear prediction problem on the frequency axis. The number of hidden layers matches the order of linear prediction. The linear prediction coefficient {p _i } is determined only by the weights w _i and θ _i from the input layer to the intermediate layer. The weights a _i from the hidden layer to the output layer do not affect the linear prediction coefficients {p _i }. That is, what is obtained by linear prediction on the frequency axis is the weight from the input layer to the intermediate layer.

From the above, the following engineering point of view can be obtained. Whether or not the application of the neural network is suitable for the engineering problem to be solved can be judged by whether or not the linear prediction on the frequency axis is suitable. The number of hidden layers is determined by whether the residual of linear prediction is sufficiently small. The weight from the hidden layer to the output layer is
It is determined by the method of least squares, and the essence of the neural network lies in the weight determination from the input layer to the output layer.

By utilizing this fact, the number of intermediate layers can be automatically determined. That is, when the learning is performed with the number of hidden layers set to a certain value, and the residual due to linear prediction does not become sufficiently small, the number of hidden layers is insufficient, so the linear prediction is performed by increasing the number of hidden layers. Start over. Then, by repeating the operation until it becomes smaller than the preset residual value, the minimum number of intermediate layers required can be determined.

In the above description, the case where the input layer and the output layer are one unit and the intermediate layer is a plurality of N units has been described. Next, the case where the input layer is a plurality of units will be described. In this case, the problem is reduced to the one-dimensional case.

[Lemma]

[Equation 10] The Fourier transform F (k ₁ , k ₂ , ..., K _N ) of is given as follows.

[Equation 11] Here, the function ρ is the Fourier transform of the function σ. [End of lemma]

[System] Fourier transform of a function such as equation (9) is a combination of straight lines passing through the origin.

[Equation 12] It becomes 0 in all other cases. [System end] When the straight line of the equation (11) is obtained by using the Fourier transform of the given function, the ratio w _i1 : w _i2 : w _i3 : ...: w _iN ... (12) is obtained. It will be. This ratio does not depend on the shape of the function σ. The shape of the function σ is required to obtain the value of w _i1 , but the value of the ratio in Expression (12) is an invariant that does not depend on the function σ.

To obtain a straight line as shown in equation (11), the function |
F (k ₁ , k ₂ , ..., K _N ) | is regarded as an N-dimensional image, and all the straight lines passing through the origin may be obtained. On each straight line,

[Equation 13] Will result in a one-dimensional problem. Constant w _i1
And θ _i are calculated again by using “σ (x) = tan
^-1 (x) "can be used.

By the way, although it cannot be solved by the two-layer perceptron, there is an exclusive OR problem that it can be solved by three or more layers. When simulation was performed on this problem, the results shown in FIG. 3 were obtained. In the figure, (a) is teacher data, (b) is 1 by the error propagation method.
(C) is the learning result by the learning method of the present invention. The number of intermediate layers is five.

As is clear from this result, the learning method (c) according to the present invention captures the teacher data more faithfully than the conventional learning method (b). The learning time required for the conventional method was about 10 hours, whereas the learning method according to the present invention only required a few seconds, and the effectiveness was confirmed.

Next, a method of solving the linear prediction problem in step S3 described above will be described. Although this solution is a well-known matter, it will be briefly described in order to show that the above-described processing procedure can be realized.

First, a complex number sequence {x _t } _{t = 0,1, ...}
Consider the problem of linearly predicting _M-1 in the Nth order. x _t + α ₁ x _t-1 + ... + α _N x _tN = ε _t (14) Here, α _i and ε _t are complex numbers, respectively. The error E is defined as follows.

[Equation 14] (Where c _ij is self-adjoint, c _ij ^* = c _ji ),

[Equation 15] Is obtained. Minimize this value (α _i = p _i + sqrt
(−1) q _i ) can be characterized by ∂E / ∂p _i = ∂E / ∂q _i = 0, and “c _ij = a _ij + sqrt (−
1) b _ij ”, simultaneous equations

[Equation 16] Can be found by solving. The optimization range (t ₀ , t ₁ ) is chosen as (p _, M-1) and this equation is solved by the covariance method, and (-∞, + ∞) is chosen except (0, M-1). The method of setting the value of at 0 is called the autocorrelation method.

[0034]

According to the present invention, the learning method is selected by focusing on the fact that "y = tan ^-1 (x)" is selected as the non-linear function of the neural network and the learning results in a linear prediction problem on the frequency axis. Since the learning speed is nearly 1000 times faster than the error backpropagation method, it is possible to mathematically understand the essence of neural networks. As a result, significant progress can be expected in voice recognition, voice synthesis, character recognition, object recognition, stock price prediction, robot control, and the like.

[Brief description of drawings]

FIG. 1 is a flowchart showing a processing procedure of the present invention.

FIG. 2 is a block diagram of a three-layer perceptron.

FIG. 3 is a diagram showing a simulation result of an exclusive OR problem.

[Explanation of symbols]

 1 Input layer 2 Middle layer 3 Output layer

Claims (5)

[Claims] 1. An input layer, an intermediate layer having neurons of a plurality of N units, and an output layer, wherein synaptic weights from the input layer to the intermediate layer are w _i (where i = 1, 2,
, N), where the threshold of each neuron in the intermediate layer is θ _i , and the synapse weight from the intermediate layer to the output layer is a _i , the Fourier transform F (k) of the training data f (x) is obtained. The first processing step and an arbitrary sampling interval T are determined, and the sequence {g _n } is changed to "g
The second processing step generated by “ _n = nT · F (nT)” and the above sequence {g _n } to “g _n + p ₁ g _n-1 + p ₂ g _n-2”.
A third processing step for _{obtaining a} sequence {p _i } satisfying the condition “+ ... · + p _N g _nN = 0” and the N-dimensional complex algebraic equation “x ^N + p” from the above sequence {p _i }.
₁ x ^N-1 + p ₂ x ^N-2 + ... + p _N = 0 "to obtain a solution {Z _i } and a fifth process for obtaining the constants w _i and θ _i from the following equations And w _i = −T / log | Z _i ｜ θ _i = arg (Z _i ) / log | Z _i ｜ The output of the intermediate layer is calculated from “tan ⁻¹ (w _i x + θ _i )”, and the result is obtained. From the above, a sixth processing step of obtaining the constant a _i by the least square method so that the error of the output data with respect to the teacher data f (x) is minimized, and the constants w _i , θ _i and a _i are determined by Neural network characterized by. 2. An input layer, an intermediate layer having neurons of a plurality of N units, and an output layer, wherein a synaptic weight from the input layer to the intermediate layer is w _i (where i = 1, 2,
, N), the threshold of each neuron in the intermediate layer is θ _i , the synapse weight from the intermediate layer to the output layer is a _i , and a constant w that minimizes the error of the output data with respect to the teacher data f (x) _In the learning of the multi-layer perceptron for obtaining _i , θ _i , and a _i , the first processing step for obtaining the Fourier transform F (k) of the teacher data f (x) and an arbitrary sampling interval T are set and a sequence {g _n } To “g
The second processing step generated by “ _n = nT · F (nT)” and the above sequence {g _n } to “g _n + p ₁ g _n-1 + p ₂ g _n-2”.
A third processing step for _{obtaining a} sequence {p _i } satisfying the condition “+ ... · + p _N g _nN = 0” and the N-dimensional complex algebraic equation “x ^N + p” from the above sequence {p _i }.
₁ x ^N-1 + p ₂ x ^N-2 + ... + p _N = 0 "and a fifth process for obtaining a solution {Z _i } and a fifth process for obtaining the constants w _i and θ _i from the following equations And w _i = −T / log | Z _i ｜ θ _i = arg (Z _i ) / log | Z _i ｜ The output of the intermediate layer is calculated from “tan ⁻¹ (w _i x + θ _i )”, and the result is obtained. And a sixth processing step for obtaining the above-mentioned constants a _i by the least-squares method, and a learning method for a neural network characterized by: 3. The learning according to claim 2, wherein as a result of performing the learning, if the error is smaller than a predetermined value, the learning is ended, and if not smaller, the number of intermediate layers is increased and the learning is performed again. A learning method for a neural network, characterized in that the number of intermediate layers is determined by repeating the learning until the error becomes smaller than a predetermined value. 4. In learning of a multi-layer perceptron having three or more layers, a straight line passing through the origin is applied to the Fourier transform of a given function when the dimension of input data is two or more, and the result can be linearly predicted. A learning method and its learning method, characterized in that learning is performed by utilizing the fact that it is reduced to. 5. When approximating a given function using a neural network, the Fourier transform of the function is obtained, the function is approximated in frequency space, and then the approximate function is obtained by inverse Fourier transform. A function approximation method using a neural network.