JP2798793B2 - Neural network device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Application Field] The present invention simulates a nerve cell and a connection between the nerve cells, and performs, for example, memory, inference, pattern recognition, control, model estimation, and function approximation. The present invention relates to a neural network device for performing.

[Prior Art] FIG. 8 is a diagram illustrating, for example, a magazine (Dvaid E. RTumelhart, Geoff).
rey E. Hinton & Ronald J. Williams. “Learning repre
sentations by back-propagating erros ", Nauture, Vo
l.323, No. 9, 535 to 536, October, 1986) is an explanatory diagram showing the configuration of a conventional multilayer feed-forward type neural network device. For example, FIG. 1 shows a case where the input layer, the intermediate layer, and the output layer are each composed of one layer. The input layer has three neural elements, the intermediate element has four neural elements, and the output layer has one neural element. Is shown. In the figure, reference numeral (11) denotes an element that simulates a nerve cell (hereinafter, referred to as a neural element), and includes an input layer (11a), an intermediate layer (11b), and an output layer (11c). (12) is a neural element (11)
(Hereinafter, referred to as a coupling element) that simulates a synapse by connecting layers between the layers, and the strength of the coupling is referred to as a connection weight.

In the neural network device configured at this time, the neural element (11) is connected in a layered manner, and as shown in the arrow A, the input signal coming from the input layer (11a) is Propagated to the output layer (11c) via (11b).

It is quantitatively as follows. ▲ V ^P _1i ▼ the input layer (11
The i-th value of the p-th learning data in a), d
_KP is the k-th value of the p-th learning data in the output layer (11c), u _hj and V _hj are the internal state and the output value of the j-th neural element in the h-th layer, and W _hji is the output value of the h-th layer. The connection weight between the i-th neural element and the j-th neural element in the (h + 1) th layer. In this embodiment, in the input layer (11a), h = 1,
H = 2 in the intermediate layer (11b) and h = 3 in the output layer (11c). At this time, the relationship between the variables is expressed as in equations (1) and (2).

V _hj = g (u _hj ) (2) Here, the function g (*) may be any function that is differentiable and non-decreasing. For example, the function g (*) is represented by Expression (3). FIG. 9 shows u as the axis and g (u) as the vertical axis.

Further, the connection weight W is sequentially determined according to a learning rule shown in Expression (4). That is, the learning data d _1P in the output layer
(Desired signal) and a steepest descent method regarding a square error defined by a value actually selected by the neural network. Assuming that the number of layers of the neural network is H (= 3),
The square error is represented by equation (4).

In addition, the sequential change of the connection weight W can be executed by Expression (5) when α and β are used as appropriate parameters and the moment method is specified.

In actual use, if equation (5) is differentiated and the right side is described in more detail, learning is performed by adjusting the connection weight W while the output error is propagated from the output side to the input side as shown by the arrow B. Therefore, it is called a back propagation method or a back propagation method.

[Problem to be Solved by the Invention] Since the conventional neural network device is configured as described above, there has been a problem that a large amount of time is required for convergence of the repetitive operation in the learning algorithm. Also,
Whether the number of neural elements in the number and each of the layers of the intermediate layer is how necessary advance was unknown.

The present invention has been made in order to solve the above-described problems, and it is possible to speed up the convergence of the repetitive operation for learning so that learning can be performed at a high speed. Further, it is possible to determine the number of neural elements in the intermediate layer in advance. It is an object of the present invention to obtain a neural network device that can be used.

[Means for Solving the Problems] A neural network device according to the present invention provides a method for generating a pair of an input value at an irregular interval and an output value corresponding to the input data at a constant interval and an output value corresponding thereto at a constant interval. Assuming sampling data composed of pairs, the sampling function translated in parallel by the input value of the sampling data is stored in the neural element of the intermediate layer, and the output value of the sampling data is used as a connection weight for coupling the intermediate layer and the output layer. The output value of the sampling data is adjusted by the learning equation so that the error between the output obtained when the input value of the learning data is set and the output value of the learning data is locally minimized. is there.

[Operation] In the neural network device according to the present invention, sampling data sampled at regular intervals is assumed for learning data composed of pairs of input values and output values obtained at irregular intervals. The sampling function translated in parallel by the input value of the sampling data is stored in the neural element of the intermediate layer, and the output value of the sampling data is set to the connection weight between the input layer and the output layer. Then, when the input value of the learning data is given to the neural network device, learning is performed by adjusting the connection weight, which is the output value of the sampling data, so that the output of the neural network device becomes a desired output value. I have. Therefore, the convergence of the repetitive operation in the learning is fast, so that high-speed learning can be performed. Further, as in the sampling theorem, if the complexity of the information source that generated the learning data, that is, the maximum frequency is known, the number of required neural elements in the intermediate layer can be determined.

Embodiment FIG. 1 is an explanatory diagram showing a configuration of a neural network device according to an embodiment of the present invention. In the figure, (11) is a neural element, which is composed of an input layer (11a), an intermediate layer (11b), and an output layer (11c). (12) is the connection weight between the neural elements (11). FIG. 2 shows an example of the input / output characteristics of the neural element (11). The horizontal axis represents the input value, and the vertical axis represents the output value. M) is 21.

The function of a conventional multilayer feed-forward type neural network device will be described with reference to FIG. In the figure, (3
1) The original function that generated the learning data is a parabola, five black circles (32) are learning data, and (33) is an interpolated curve reproduced by a three-layer conventional neural network. In the figure, the horizontal axis represents input and the vertical axis represents output. Comparing the regenerated curve (33) with the original curve (31), the learning data (3
2) produces a value close to the output of the original curve (31),
In other parts, interpolation is performed and a curve is reproduced. As described above, the conventional device has a function of interpolating between an input and an output when they are given.

Since the function of the conventional neural network device is as described above, the relationship shown in FIG. 2 should be used instead of the so-called sigmoid function shown in FIG. 9 as the neural element (11). Thereby, a similar function can be realized.

Therefore, the Fourier expansion will be described first. A function with periodicity can be Fourier-expanded. Function y =
If f (x) is defined as a hypercube with a side length of 1, the definition or repetition in the other space may be regarded as being repeated at a period of 1. Therefore, it must have periodicity as a whole. become. FIG. 4 shows the definition or the two-dimensional case. Therefore, the function y = f (x) is Fourier-expanded as follows.

However, in the equation (7), the intensity of the Fourier series of the frequency k included in the function f (x), and N represents the number of dimensions of the input. K is the highest frequency of the function f (x), indicating that generally infinite frequencies must be taken into account to recover the unknown function.

In order for a function f (x) to be Fourier-expandable, it must be in its definition or in a hypercube. But,
Generally, the actual learning data x 'to be realized by the neural network device usually takes an arbitrary real value, which is contrary to the requirement of a hypercube. Therefore, the input data is subjected to appropriate normalization, and its value is stored in the hypercube as x.

Usually, it is customary that the output y of the neural network device is also in the hypercube, so that the actual output data, y ', is denormalized. In other words, the operation of the entire system including input / output normalization is as shown in FIG. That is, the actual learning data x 'which is an arbitrary real value is normalized in the block (41), the function is approximated by the neural network device in the block (42), and the output y is converted to the block (43)
To obtain the actual output data y '. As the normalization, for example, the following method can be considered.

Method of using sigmoid function Method of affine transformation In the case of the definition of actual variables or all real values, using the sigmoid function as shown in the following formula will convert nonlinearly into a hypercube.

If you know the definition of the data or the following in advance, The affine transformation may be performed on the region shown by the equation (10), and the region may be pushed into the hypercube.

The function to be estimated is f (x), and Equations (6) and (7)
Is transformed as follows using

Here, {*} approaches the delta function when the maximum frequency K is sufficiently large. That is, it can be interpreted that the function f (x) is constituted by the superposition of the delta functions having the height of f (x ') at x'.

Here, since learning data can be obtained only at irregular intervals, the integral of Expression (15) cannot be obtained with high accuracy unless there is a sufficient amount of data to fill in the definitions or the like. Therefore, for the sake of simplicity, it is assumed that learning data is regulated, and irregular learning data will be handled using this result. Now, the learning data is very convenient
Assuming that 2K _i + 1 = M _i are obtained, the integration of Expression (15) is performed by subtraction. First, And Representative point further integrated value which is equal M _i, Let's do it. Here, assuming that K is the highest frequency of the function f (x), Becomes However, M = 2K + 1 (22) In order to clarify the meaning of Expression (21), it is rewritten as follows.

However, Function [Phi (x) is close to the larger the delta function is M _i, the diagram shown in FIG. 2 in the case input and output of the M = 21 1-dimensional.

From the above, it can be seen that y = f (x) is expanded by the basis function Φ (x). In other words, it is a definition or a sampling theorem limited to a hypercube. Therefore, if the maximum frequency K is known, it is sufficient to obtain the learning data of at least M represented by the equation (22) necessary for reproducing the function.
No other training data is needed. This means that the complexity of the function, that is, the required number of training data can be known by knowing the highest frequency, and below that number, the function cannot be completely reproduced. FIG. 1 shows Expression (27) as a three-layer neural network. The intermediate layer (11b) in this embodiment may be composed of P = 21.

As described below, when 2K + 1 pieces of learning data are regularly obtained so as to equally divide a hypercube, the function can be completely restored in the same manner as the sampling theorem. However, here, it is assumed that the learning data can only be obtained irregularly. In such a case, regular data (hereinafter referred to as sampling data) is assumed, and the function restored from this data and the irregular data are assumed. Considering an error with respect to simple learning data (hereinafter simply referred to as learning data), the sampling data is adjusted so as to minimize this error locally.

Sampling data is (x _m , _m ) and learning data is (x
_p , y _p ), the error E is defined as follows.

However, Therefore, in order to minimize E, for example, the sampling data may be adjusted using the steepest descent method as follows.

However, Alternatively, the acceleration term can be considered by using the moment method as in the case of back propagation.

However, equation (33) is not a difference equation but a learning equation described by a differential equation.

At this time, a function f (x) that satisfies the learning data cannot be theoretically constructed unless sampling data having a frequency equal to or higher than the highest frequency of the function that generated the learning data is assumed. Therefore, when the sampling data is insufficient, it is necessary to increase the assumed maximum frequency to increase the number of sampling data.

The entire operation is shown in the flowchart of FIG. In step (51), a pair of learning data is prepared, and in step (52), the maximum frequency K and the initial value of the sampling data are set. Next, in step (53), the sampling data is adjusted so as to minimize the error E in the learning data. If the adjustment is insufficient in step (54), the sampling data is adjusted by assuming a higher maximum frequency. If the error is sufficiently small in step (55), the process ends.

The setting method of initial values of the sampling data in step (52) may be estimated for example by linear interpolation from nearby y _p. Regarding the method of increasing the sampling data when the maximum frequency is increased in step (54),
For example, there is a method of estimating new sampling data from old sampling data by linear interpolation.

FIG. 7 shows the convergence result by simulation. The function from which the learning data is generated (hereinafter referred to as the original function) is represented by (34) and has a mountain shape as shown by the line (61) in FIG. 7 (c).

FIG. 7A shows the number of repetitions on the horizontal axis, the square root of the error E on the vertical axis, FIG. 7B shows the number of repetitions on the horizontal axis, and the maximum value of the absolute value error on the vertical axis. FIG. 3C is a graph showing the input on the horizontal axis and the output on the vertical axis. FIG. 7 (a),
According to (b), the error increases when the highest frequency is switched, because the estimation by interpolation of new sampling data is not sufficient. In FIG. 7 (c), black circles (62) represent learning data, attenuated trigonometric functions (63) represent sampling functions, linear lines (61) connecting the learning data represent original functions, and winding lines (64). ) Is the reconstructed function. Comparing line (61) and line (64),
In each case, it can be seen that the reproduction is performed with high accuracy.

In addition, back propagation (B
The results (Table 1) of the speed comparison with P) are shown. The conditions are as follows.

 The original function is a mountain-shaped function shown in FIG.

 The computer used is a general-purpose workstation. The middle layer of the BP is 100. Irregular learning data is obtained randomly from the original function.

Assuming that the initial value of the highest frequency is 1, the calculation is repeatedly performed while increasing to 1, 3, 5, 7, and the like.

Table 1 shows a comparison result of the calculation time between the conventional example and the embodiment. In the table, indicates that no simulation was performed.

As can be seen from Table 1, since the repetition operation is performed,
If it becomes necessary to precisely reproduce the original function due to an increase in learning data, it takes a long learning time. Here, since the simulation is performed only once, the influence of the initial value of the random number is considerably affected, and this influence can be seen. Conventional backpropagation also depends on parameter settings, but requires a very long learning time and a large number of data.
Stop converging at 10. In any case, it can be seen that the neural network device according to this embodiment is much faster than the back propagation of the conventional device.

As described above, in the above embodiment, the convergence of the iterative operation for learning is fast, so that high-speed learning is possible.
Knowing the complexity of the information source that generated the learning data, that is, the maximum frequency, it is possible to determine the number of required neural elements in the intermediate layer.

In addition, the number and the number of layers of the neural element constituting the neural network are not limited to the above-described embodiment, and may be changed according to the application field.

[Effects of the Invention] As described above, according to the present invention, the input layer, the intermediate layer,
And a neural network device comprising a plurality of neural elements simulating nerve cells of a living body, and a connection weight for coupling between layers of the neural elements. Assuming sampling data consisting of pairs of input values at regular intervals and output values corresponding to the learning data, the sampling function translated in parallel by the input value of the sampling data is stored in the neural element in the intermediate groove Then, the output value of the sampling data is set as a connection weight for connecting the intermediate layer and the output layer, and the error between the output obtained when the input value of the learning data is given and the output value of the learning data is locally minimized. Since the output value of the sampling data is adjusted by the learning equation, the convergence of the iterative operation for learning is fast, so that learning can be performed at high speed. Complexity of the generated information sources, namely the construction of the neural network apparatus can determine the number of neural elements of the intermediate layer is highest frequency required knowing becomes possible.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram showing the configuration of an embodiment of a neural network device according to the present invention. FIG. 2 is a characteristic diagram showing input / output characteristics of the neural element according to this embodiment, FIG. 3 is a graph of a function in the operation of a conventional multilayer feedforward type neural network device, and FIG. FIG. 5 is an explanatory diagram showing the definition or definition of a function approximated by one embodiment. FIG. 5 is an operation diagram of the entire system including normalization of input variables and denormalization of output variables.
FIG. 6 is a flowchart showing the overall operation according to one embodiment, FIG. 7 (a) is a graph showing a reduction in error in learning convergence, and FIG. 7 (b) is a graph showing the absolute value of the error at the learning data point. A graph showing the maximum value, FIG. 7 (c) is a graph showing the original function and the function reproduced by the apparatus according to one embodiment,
FIG. 8 is an explanatory diagram showing a configuration of a conventional three-layer feedforward type neural network device, and FIG. 9 is a characteristic diagram showing input / output characteristics of a neural element according to the conventional device. (11), (11a), (11b), (11c) ... neural element, (12) ... connection weight.

──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Hiromi Ogi 1-4-10 Irifune, Chuo-ku, Tokyo Tokyo Electric Power Company System Research Laboratory (72) Inventor Yoshio Izui 8-1-1 Honcho Tsukaguchi, Amagasaki City, Hyogo Prefecture No. Mitsubishi Electric Corporation Industrial System Research Laboratory (72) Inventor Hisao Taoka 8-1-1 Tsukaguchi Honcho, Amagasaki City, Hyogo Prefecture Mitsubishi Electric Corporation Industrial System Research Laboratory (72) Inventor Toshiaki Sakaguchi 8 Tsukaguchi Honmachi, Amagasaki City, Hyogo Prefecture No. 1-1, Mitsubishi Electric Corporation Industrial System Research Laboratory (56) References Stinchcombe, H .; W white, "Universal Application Usage Using Feedforward Networks ks with Non-Signoid Hidden Layer Actiaction International Connection Insurance", IEEE Internet Information Communication Services. 1, P. ▲ I ▼ -613 ~ P. ▲ I ▼. 617 (1989) (58) Field surveyed (Int. Cl. ⁶ , DB name) G06F 15/18 JICST file (JOIS)

Claims (1)

(57) [Claims] An input layer, an intermediate layer, and an output layer;
A plurality of neural elements that simulate nerve cells of a living body, and a neural network device having connection weights for connecting layers of the neural elements, wherein learning data composed of pairs of input values at irregular intervals and output values corresponding to the input values are provided. On the other hand, assuming sampling data composed of pairs of input values and output values corresponding thereto at regular intervals, a sampling function translated in parallel by the input value of the sampling data is stored in the neural element of the intermediate layer, and the sampling is performed. An output value of data is set as a connection weight for connecting the intermediate layer and the output layer, and an error between an output obtained when an input value of the learning data is given and an output value of the learning data is locally minimized. Wherein the output value of the sampling data is adjusted by a learning equation.