JPH0713745A - Neural network type pattern learning method and pattern processor - Google Patents

Abstract

PURPOSE:To attain the learning despite a steep change of a learning pattern by calculating the differential value of an output pattern against an input pattern and adding the differential value to a learning cost function in response to the differential value to suppress the over-learning and changing the learning cost function based on a learning pattern distribution when the learning pattern is steeply changed to suppress the over-learning occurring in the progress of learning. CONSTITUTION:This processor is provided with a pattern input circuit 1, a square error calculating circuit 2, a differential value calculating circuit 3 which calculates the differential value of an output pattern against an input pattern, a coefficient calculating circuit 4, and a coupling coefficient correcting circuit 5. In such a constitution, it is possible to suppress the over-learning by calculating the differential value of the output pattern and adding this differential value to a learning cost function according to the differential value. Then the learning cost function is changed based on a learning pattern distribution in order to suppress the over-learning when a learning pattern has a sharp change. So that the learning is possible even in a steep change of the learning pattern.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network type pattern learning method and a pattern processing device for automatically learning by giving a desired output pattern corresponding to an input pattern.

[0002]

2. Description of the Related Art A conventional back propagation learning method is carried out as follows in a multilayer neural network as shown in FIG. It should be noted that FIG. 3 shows a three-layer neural network, in which circles represent non-linear threshold units, and straight lines represent weights between units.

Considering an m-layer multilayer neural network, the k-th i-th layer
The sum of the inputs to the th unit is i , The relation of these variables is as follows.

[0004]

[Equation 1] Consider the following as a cost function.

[0005]

[Equation 2] However, the sum for j is the number of units existing in the final layer m. Then, the weight change is as follows.

[0006]

[Equation 3]

[Equation 4] In this way, the weight is changed and learning is performed.

[0007]

When the learning pattern is distributed as shown in FIG. 4, for example, the cost function of the learning method described above has learned if it passes through the points of the given learning pattern. As the learning progresses, there is a problem that an over-learning state occurs as shown in FIG.

The present invention has been made in view of the above,
An object of the invention is to provide a neural network type pattern learning method and a pattern processing device capable of suppressing over-learning that occurs as learning progresses and capable of learning even when a learning pattern sharply changes.

[0009]

In order to achieve the above object, the neural network pattern learning method of the present invention is a neural network pattern learning method provided with a multilayer neural network composed of nonlinear threshold units. Therefore, when the differential value of the output pattern with respect to the input pattern is calculated and the differential value is added to the learning cost function according to the magnitude of the differential value, overlearning is suppressed, and when the learning pattern changes abruptly. The gist is to change the learning cost function based on the distribution of the learning pattern.

The neural network type pattern processing apparatus of the present invention is a neural network type pattern processing apparatus provided with a multilayered neural network composed of non-linear threshold units, the differential value of the output pattern with respect to the input pattern. And a means for calculating a ratio of adding the differential value to the learning cost function according to the magnitude of the differential value.

[0011]

In the neural network pattern learning method of the present invention,
Calculate the differential value of the output pattern with respect to the input pattern,
Over-learning is suppressed by adding the differential value to the learning cost function according to the magnitude of the differential value, and when the learning pattern changes abruptly, the learning cost function is changed based on the distribution of the learning pattern.

Further, in the neural network type pattern processing apparatus of the present invention, the differential value of the output pattern with respect to the input pattern is calculated, and the ratio of adding the differential value to the learning cost function is calculated according to the magnitude of the differential value.

[0013]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the arrangement of a neural network type pattern processing apparatus according to an embodiment of the present invention. The neural network type pattern processing apparatus shown in the figure includes a pattern input circuit 1 for inputting a pattern, a square error calculation circuit 2 for calculating a squared error, and a differential calculation circuit 3 for calculating a differential value of an output pattern with respect to an input pattern. , A coefficient calculation circuit 4 for calculating coefficients, and a coupling coefficient correction circuit 5 for correcting coupling coefficients.

Now, considering a multilayer neural network of m layers, the sum of inputs to the i-th unit of k layers is summed up. Then, the relation of these variables is as follows.

[0016]

[Equation 5] Consider the following squared error cost function as the cost function.

[0017]

[Equation 6] However, ε ₂ is a constant, and the sum for j is the number of units existing in the final layer m. It is a differentiation that makes the function realized by the circuit smooth and suppresses over-learning, but with the cost function with this penalty put, smoothness is suppressed where the learning pattern changes sharply. I can't. Therefore, how much the penalty is effective is determined by the term multiplied by the smoothness penalty of (1-ρ _j ). When ρ _j = 1, the learning pattern changes abruptly, and the penalty of smoothness is not included in the cost function. When ρ _j = 0, the smoothness penalty is included in the cost function.

The change of weight, which is learning, is as follows.

[0019]

[Equation 7]

[Equation 8]

[Equation 9] For example, the following method can be used to determine ρ _j .

Consider a Hopfield neural network that is commonly used to optimize certain constraints, and consider E'as its cost function.

[0021]

[Equation 10] V (ρ _j ) is the penalty when ρ _j = 1 and
Give as follows.

[0022]

[Equation 11] However, C ₁ , B ₁ , B ₂ , B ₃ , and B ₄ are constants. The first and second terms are the penalties for ρ _j = 1 and
The third term is a term for ρ to converge to 0 or 1. Here, G is the following function.

[0023]

[Equation 12] However, λ is a constant.

Here, the following equation is considered as an equation for determining ρ _j .

[0025]

[Equation 13] However, ε ₃ is a constant.

The time derivative of E'is always 0 or less according to the following equation.

[0027]

[Equation 14] When the time derivative of E ′ becomes 0, the following holds for all ρ _j , and ρ _j always converges.

Dρ _j / dt = 0 (27) Next, the pattern learning method will be described with reference to the flowchart shown in FIG. As described above, the differential value of the output with respect to the input is calculated by the differential calculation circuit 3 (step 110), and another Hopfield neural network is used to determine how effective the calculated differential value is in the cost function. Calculate using. That is, the coefficient by the Hopfield neural network is determined (step 120). Then, a multilayer neural network is trained with this cost function (step 130). By repeating the above procedure, it is possible to learn even while the learning pattern is changing sharply while suppressing over-learning.

[0029]

As described above, according to the present invention,
Calculate the differential value of the output pattern with respect to the input pattern,
Over-learning is suppressed by adding the differential value to the learning cost function according to the magnitude of the differential value, and when the learning pattern changes sharply, the learning cost function is changed based on the distribution of the learning pattern. In addition to suppressing the above, learning is possible even when the learning pattern changes sharply, pattern recognition that could not be realized by conventional methods,
Processing such as classification becomes possible.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a neural network type pattern processing apparatus according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a procedure of a neural network pattern learning method according to an embodiment of the present invention.

FIG. 3 is a diagram showing a three-layer neural network which is an example of a multilayer neural network.

FIG. 4 is a diagram showing points (both input and output are one-dimensional) of a learning pattern and a function realized by a neural network.

FIG. 5 is a diagram showing points (both input and output are one-dimensional) of a learning pattern and a function realized by an overlearned neural network.

[Explanation of symbols]

 1 pattern input circuit 2 square error calculation circuit 3 differential calculation circuit 4 coefficient calculation circuit 5 coupling coefficient correction circuit

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Noboru Sonehara 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corporation

Claims (2)

[Claims] 1. A neural network type pattern learning method comprising a multilayer neural network composed of non-linear threshold units, wherein a differential value of an output pattern with respect to an input pattern is calculated, and the differential value is calculated according to the magnitude of the differential value. Neural network characterized by suppressing the over-learning by adding the differential value to the learning cost function, and changing the learning cost function based on the distribution of the learning pattern when the learning pattern changes abruptly. Type pattern learning method. 2. A neural network pattern processing apparatus comprising a multilayer neural network composed of non-linear threshold units, comprising means for calculating a differential value of an output pattern with respect to an input pattern, and a magnitude of the differential value. And a means for calculating a ratio of adding the differential value to the learning cost function according to the above.