JPH06149767A - Neural network - Google Patents

Abstract

PURPOSE:To provide the neural network which precisely outputs an analog output and also outputs its extremal value speedily. CONSTITUTION:The neural network consists of an input layer I consisting of neurons I1-Ii, an intermediate layer H consisting of neurons H1-Hj, and an output layer L consisting of a neuron L1, and a linear function is used as the response characteristics of the output layer L.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network (neural network) having a learning function, and in particular, an analog output can be obtained with high accuracy and an extreme value such as that found in the firing characteristics of neurons. The present invention relates to a hierarchical neural network that can quickly converge 1,0.

[0002]

2. Description of the Related Art Conventional neural networks
Then, it demonstrates based on FIG. This figure shows a hierarchical new
It shows a Ral network, where I is the input layer and H is the middle.
Layer, O is an output layer. Input layer I, middle layer H, output layer O
Respectively I₁~ I _i, H₁~ H_j, O₁~ O_kFrom
Is composed of neurons. Each neuron
Conveys information by a nonlinear sigmoid response function
And output O_K1~ O_KnIs getting

On the other hand, each neuron O in the output layer O₁~ O_k
Is a teacher signal (correct event) T₁~ T_kIs entered,
By the back propagation method (back propagation learning method)
As indicated by A, the signal travels in the opposite direction to the signal propagation. Middle
Coupling weight coefficient V from layer H to output layer O_jkAnd from input layer I
Coupling weight coefficient W to the intermediate layer H_ijIs the teacher signal T₁~ T _k
Output O each time is input_k1~ O_knAnd corresponding
Each teacher signal T₁~ T_kCorrected to minimize the error between
To be done.

As is well known, the sigmoid response function is a non-linear response function that simulates the human nerve response. In the operation of the neural network using this response function, the feature of the object to be recognized is extracted by the connection from the input layer to the intermediate layer, and the feature is expressed by the connection from the intermediate layer to the output layer.

This sigmoid response function has the characteristics shown in FIG. 1 (b) and is expressed by the following equation. f (X) = 1 / {1 + exp (−2X / u ₀ )} (1) u ₀ in the equation (1) is a positive parameter that determines the slope of the sigmoid function. Hereinafter, in order to facilitate understanding of the neural network of the present invention, the learning function of the neural network will be described. Differentiating the expression (1) with respect to X, the following expression is obtained. f ′ (X) = 2 · f (X) · {1-f (X)} / u ₀ (2) The outputs I _in , H _jn , and 0 _kn of each layer are expressed as follows. .

[0006]

[Equation 1]

(Where θ _j is the offset of neurons in the intermediate layer H and γ _k is the offset of neurons in the output layer O)

If the error between the output O _Kn of the output layer O and the teacher signal T _K is e _K (= T _K −O _Kn ), this mean square error E _K is expressed by the following equation. It E _K = 1 / _2 (T _K −O _Kn ) ² ……………… (5) The state in which the mean square error E _{k of the} neural network is the minimum is the optimal learning state. The mean squared error E _k is minimized with respect to the combination weighting factor V _jk . It is a learning action in the neural network to correct the coupling weight coefficient and the offset so that the mean square error E _k is minimized.

On the other hand, the function of the mean square error is a function of the output O _Kn of the output layer O, and the output O _Kn is a function of the output H _jn of the intermediate layer H. _Therefore , the following relation holds. ∂E _K / ∂V _jk = ∂E _K / ∂O _Kn・ ∂O _Kn / ∂V _jk ……… (6) ∂E _K / ∂O _Kn ＝ － (T _K −O _Kn ) ……………… (7) Here, the equation (4) is written as follows.

[0010]

[Equation 2]

When S _K is differentiated with respect to V _jk, it is expressed as ∂S _K / ∂V _jk = H _jn ............ (9).

A change in the output O _Kn with respect to a slight change in the coupling weight coefficient V _jk is expressed as follows. ∂O _Kn / ∂V _jk = ∂O _Kn / ∂S _K・ ∂S _K / ∂V _jk = f '(S _K ) ・ H _jn = 2 / u ₀・ O _Kn・ (1─O _Kn ) ・ H _jn …… (10)

Therefore, the change amount of the mean square error E _K with respect to the minute change of the coupling weight coefficient V _jk is expressed by the following formula from the formulas (7) and (10). ∂E _K / ∂V _jk = ∂E _K / ∂O _Kn・ ∂O _Kn / ∂V _jk = −H _jn・ 2 / u ₀・ (T _K −O _Kn ) ・ O _Kn・ (1─O _Kn ) = -Δ _K · H _jn ………………………… (11) (where δ _K = 2 / u ₀ · (T _K −O _Kn ) · O _Kn · (1 − O _Kn ). And)

The mean squared error E _K reaches a minimum value by repeating the learning for modifying the coupling weight coefficient V _jk in the direction in which the mean squared error E _K decreases. The correction amount ΔV _jk
Is expressed by the following equation. ΔV _jk = −η ₁ (∂E _K / ∂V _jk ) = η ₁ · δ _K · H _jn …………………… (12) (where η ₁ is a constant) Next, the same method The correction amount ΔW _ij is calculated by Here, the formula (3) is written as follows.

[0015]

[Equation 3]

The change amount of the mean square error E _K with respect to the minute change of the coupling weight coefficient W _ij is as follows.

[0017]

[Equation 4]

[0018] Therefore, the correction amount ΔW _ij of the coupling weight coefficient W _ij
Is expressed by the following equation.

[0019]

[Equation 5]

When the correction amounts Δγ _k and Δθ _j are _obtained by a similar method, they are expressed by the following equation. ∂E _K / ∂γ _k = ∂E _K / ∂O _Kn · ∂O _Kn / ∂γ _k = 2 / u ₀ · (T _k −O _kn ) · O _Kn · (1-O _Kn ) = − δ _K Therefore, the correction amount Δγ _k is Δγ _K = −η ₃ · (∂E _K / ∂γ _K ) = η ₃ · δ _K …………………… (16) (where η ₃ is a constant)

[0021]

[Equation 6]

The above η ₁ to η ₄ are learning coefficients for controlling the magnitude of the correction amount.

[0023]

In the neural network as described above, the sigmoid function as shown in the equation (1) is used as the response function of the neurons of the input layer I, the intermediate layer H and the output layer O. The sigmoid function is a response function based on an exponential function, and its output f (X) is 0 or 1
In order to satisfy the above, no output is made unless X = −∞ and + ∞. Therefore, there are disadvantages that the number of times the calculation is performed until the output f (X) reaches 0, 1 is extremely long, the convergence time is long, and the convergence is not easy. That is, when the sigmoid function is used as the response function of the neuron in each layer, there is a problem that it is difficult to output the extreme values 0 and 1 which are important in the firing characteristics of the neuron. This means that the extreme values 1 and 0 cannot be learned.

The sine function is learned by the conventional neural network, and the weighted addition output result is shown in FIG.
It is shown in (a). The solid line (a) in FIG. 4A shows the expected sine function curve, and the dotted curve (b) shows the output result based on 5000 times of learning. As is clear from this output result, despite the 5000 learnings, in the output result shown by the curve (b),
It has not converged to the extreme values 1 and 0. However, an information processing device or a control device that requires analog judgment using a neural network may frequently need the extreme values 1 and 0 as an information result or an output result. It is desired to develop a neural network that can quickly converge.

Further, since the response of the sigmoid function is non-linear, it is clear from the curve (b) of FIG. 4 (a) that a response characteristic having a bias with respect to the analog output in the range of [0, 1] is obtained. It is shown, which means that there is some "habit" in the output result. Due to these problems, the conventional neural network using only the sigmoid response function has a drawback that the versatility is lowered particularly when analog judgment is required.

The present invention has been made in view of the above-mentioned problems, and it is an object of the present invention to provide a neural network in which an analog output can be accurately obtained and the extreme values 1 and 0 thereof can be quickly output. It is intended.

[0027]

In order to solve the above problems, the neural network of the present invention has a learning function and uses a linear function as the response function of the output layer of the neural network. .

[0028]

In the neural network of the present invention, the learning function is maintained by using the neurons of the sigmoid response function in the input layer and the intermediate layer, and the linear function is used as the response function in the neurons of the output layer. A non-analog output can be obtained and quickly converged to the extreme value of 1,0 to enhance versatility, and the output layer is a simple linear function, which can reduce the calculation speed during learning. is there.

[0029]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A neural network of the present invention will be described below with reference to the drawings. FIG. 1A shows an embodiment of the neural network of the present invention. In FIG. 1A, the neural network includes neurons I ₁ ...
The input layer I is composed of I _{i, the} intermediate layer H is composed of neurons H _{1 to} H _j , and the output layer L is composed of neurons L ₁ . Information signals IN _{1 to} IN _n are input to the neurons I _{1 to} I _i of the input layer I, and output L _kn from the output layer L.
Is output. Further, at the time of learning, the teacher signal (t
_{1 to} t ₁₃ ) is input.

The neurons I _{1 to} I _i of the input layer I and the neurons H _{1 to} H _j of the intermediate layer H use the sigmoid response function of the equation (1) shown in FIG. In the neuron L ₁ , g (X) = shown in FIG.
A linear function having a linear response characteristic in the range of 1,0 is used. g (X) = aX + b (18) a in the equation (18) is a parameter that determines the slope of the linear function, and as shown in FIG. 1 (c), the slope depends on the magnitude of the parameter a. Change. In addition, b takes a value of 0.5.

FIG. 2 shows another embodiment of the neural network of the present invention, in which the intermediate layer H is a neural network composed of a plurality of layers. Also in this embodiment, the neurons L _{1 to} L _K of the output layer L are (18)
The linear function having the linear response characteristic shown in the equation is used, and the neurons I _{1 to} I _i of the input layer I and the neurons H _{j1 to} H _jn of the intermediate layer H have the sigmoid response function shown in the equation (1). Used. T _{1 to} T _n are teacher signals.

Below, the sine function is applied to the embodiment of FIG.
Learn and explain its learning function and its output result
It (A) Learning function In the neural network, the input layer I
Ron I₁~ I_nInformation signal IN such as angle θ₁~ IN_n
Is input, the neuron L of the output layer L₁The teacher signal
As the sine function value corresponding to each angle θ as
Data (t₁~ T ₁₃) Is entered. Teacher data (t₁~
t₁₃) Based on the backpropagation method,
The output result of FIG. 4A at each point of the X mark shown in FIG.
The learning is performed 5000 times as compared with (b). Also the teacher
Data (t₁~ T₁₃) Is a corner as shown in FIG.
Sign corresponding to the degree θ (0, 30, 50, ... 330, 360)
Neuron L whose function value is output layer L₁Entered in. Immediately
When the angle θ is 0 degree, the sine function value t₁As
[0.5000] is input as teacher data, and the angle θ
Is 30 degrees, the sine function value t₂As [0.750
0] will be input.

Each data of the teacher data (t _{1 to} t ₁₃ ) is
As indicated by the arrow 8, the signal is propagated in the direction opposite to the signal direction from the output layer L to the intermediate layer H and then to the input layer I, and the error from the teacher data (t _{1 to} t ₁₃ ) corresponding to the output L _k is minimal The respective connection weight coefficients W _ij and U _jk are automatically corrected so that Note that W _ij is a coupling weight coefficient from the input layer I to the intermediate layer H, and U _jk is a coupling weight coefficient from the intermediate layer H to the output layer L.

The correction amounts of the coupling weight coefficients W _ij and U _jk and the offsets θ _j0 and γ _k0 of the neurons in the intermediate layer H and the output layer will be described below. The outputs H _jn and L _kn of each layer are expressed by the following equation.

[0035]

[Equation 7]

If the error between the output L _Kn of the output layer L and the teacher signal T is δ _K0 (= T−L _Kn ), this average 2
The multiplication error E _K is expressed by the following equation. E _K = 1/2 ・ (T-L _Kn ) ² ……………… (21)

On the other hand, the function of the mean square error E _k is the output layer L
Is a function of the output L _K, the output L _K output H of the intermediate layer H
_Since it is a function with _jn , the following relation holds. ∂E _K / ∂U _jk = ∂E _K / ∂L _Kn · ∂L _Kn / ∂U _jk ……… (22) ∂E _K / ∂L _Kn ＝ － (T−L _Kn ) ……………… (23) Here, the equation (20) is written as follows.

[0038]

[Equation 8]

The influence on the output L _K with respect to the minute change of the coupling weight coefficient U _jk is expressed as follows. ∂L _Kn / ∂U _jk = ∂L _Kn / ∂S _K・ ∂S _K / ∂U _jk = g '(S _K ) ・ H _jn = a ・ H _jn …………………… (26)

Therefore, the change amount of the mean square error E _K with respect to the minute change of the coupling weight coefficient U _jk is expressed by the equation (23) and (2)
From the equation (6), the following equation is obtained. _{∂E K / ∂U jk = ∂E K} / ∂L Kn · ∂L Kn / ∂U jk = -a · (T-L Kn) · H jn = -δ k '· H jn .................. (27) (where δ _k ′ = a · (T−L _Kn )) Mean square error is repeated by repeating learning to correct the coupling weight coefficient U _jk in the direction of decreasing the mean square error E _K. Error E
_K reaches a local minimum. The correction amount ΔU _jk is expressed by the following equation. ΔU _jk = −η ₁ ′ (∂E _K / ∂U _jk ) = η ₁ ′ · δ _k ′ · H _jn …………………… (28) (However, η ₁ ′ is a constant)

Next, the correction amount Δ
When W _ij , _{Δγ k0} , and Δθ _j0 are obtained, they are expressed by the following equation. Note that γ _k0 and θ _j0 indicate the offsets of the neurons in the output layer L and the intermediate layer H. Hereinafter, the correction amount ΔW _ij will be obtained.

[0042]

[Equation 9]

Next, the correction amount Δγ _k0 is obtained. ∂E _K / ∂γ _K0 = ∂E _K / ∂L _Kn・ ∂L _Kn / ∂γ _K0 = −a (T−L _Kn ) = − δ _K ′ Δγ _K0 = −η ₃ ′ (∂E _K / ∂ γ _K0 ) = η ₃ ′ · δ _K ′ ……………… (30) (However, η ₃ ′ is a constant.)

[0044]

[Equation 10]

However, η ₁ ′ to η ₄ ′ are learning coefficients that control the magnitude of the correction amount. As is clear from the above results, even if a linear function is used as the response function of the output layer L of the neural network, there is no problem in the learning function.

(B) Output Result FIG. 4 (b) shows the output result of the neural network of the present invention, which is the sine function curve (a) shown by the solid line.
Indicates a sine function curve to be expected, and the curve (b) indicated by a dotted line indicates a sine function curve based on the output result of the embodiment. Information signals IN _{1 to} IN _n such as the angle θ are input to the neurons I _{1 to} I _i of the input layer I, and the output layer L
From the neuron L _{1 kn of} FIG. 4, the output L _kn weighted and added based on the connection weighting factors W _ij and U _jk obtained by the above learning is the sine sign as shown in the curve (b) of FIG. It is output as a function curve.

As is apparent from FIG. 4 (b), the output result based on the coupling weight coefficients W _ij and U _jk is substantially in agreement with the sine function curve of the solid line (a). Further, even at the extreme values 1 and 0, the agreement is better than that of the dotted line (b) in FIG. Further, the analog output in the range of [0, 1] does not have the bias as seen in FIG.

From this output result, it was proved that the neural network of the present invention converges to the extreme values 1, 0 by using the linear response function in the output layer L. In addition, the bias of analog output seen in the sigmoid response function is eliminated, and the convergence to extreme values 1 and 0 is extremely high as compared with the conventional neural network using only the sigmoid function, and it can be applied to various information processing devices and control devices. It was proved to be highly versatile.

Of course, it is obvious that the neural network of the present invention may be equivalently implemented by arithmetic processing by a computer, or may be implemented by a neurochip. In the latter case, it is extremely small and high. It is possible to provide a high-performance information processing device and control device.

[0050]

According to the neural network of the present invention, by using a linear function as the response function of the neuron in the output layer of the neural network, the extreme values 1 and 0 can be reasonably output as the output of the neural network. , [0, 1] range also has the advantage that it can be output with high accuracy without bias.

By using a linear function in the output layer,
Since the convergence is high, the number of calculations can be reduced, and the calculation efficiency is improved, so that the processing result can be obtained at high speed. Further, according to the neural network of the present invention, the extreme values 1 and 0 of the output can be used for the information processing device and the control device, and there is an advantage that the versatility of the neural network can be improved.

Of course, in the learning process as well, there is an advantage that the processing time for correcting the coupling weight coefficient between each layer can be shortened, and since the extreme values 1 and 0 can be learned relatively quickly, the number of times of learning can be extremely reduced. It is effective.

[Brief description of drawings]

1A is a diagram showing an embodiment of a neural network of the present invention, FIG. 1B is a diagram showing a sigmoid function,
(C) is a figure which shows a linear function.

FIG. 2 is a diagram showing another embodiment of the neural network of the present invention.

3A is a diagram for explaining a learning function, and FIG. 3B is a diagram showing teacher data of a sine function in a table.

4A is a diagram showing an output result of a conventional neural network, and FIG. 4B is a diagram showing an output result of the neural network of the present invention.

FIG. 5 is a diagram showing an example of a conventional neural network.

[Explanation of symbols]

I input layer H intermediate layer L output layer I _{1 to} I _i input layer I neuron H _{1 to} H _j intermediate layer H neuron L _{1 to} L _k output layer L neuron

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Koji Ueda 29-1, Mita, Miwa-cho, Kaifu-gun, Aichi Prefecture Mita Plant, Nagoya Electric Industry Co., Ltd. (72) Muneo Yamada, Shinoda, Miwa-cho, Kaifu-gun, Aichi Prefecture 29-1 Nagoya Electric Industry Co., Ltd. Miwa Plant

Claims (1)

[Claims] 1. A neural network, wherein the neural network has a learning function, and a linear function is used as a response function of its output layer.