JP3412700B2 - Neural network type pattern learning method and pattern processing device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network type pattern learning method and a pattern processing device for automatically learning by giving a desired output pattern corresponding to an input pattern.

[0002]

2. Description of the Related Art A conventional back propagation learning method is carried out as follows in a multilayer neural network as shown in FIG. It should be noted that FIG. 3 shows a three-layer neural network, in which circles represent non-linear threshold units, and straight lines represent weights between units.

Considering an m-layer multilayer neural network, the k-th i-th layer
The sum of the inputs to the th unit is I,

[Outer 1] The relationship between these variables is as follows.

[0004]

[Equation 1] Consider the following as a cost function.

[0005]

[Equation 2] However, the sum for j is the number of units existing in the final layer m. Then, the weight change is as follows.

[0006]

[Equation 3]

[Equation 4] In this way, the weight is changed and learning is performed.

[0007]

When the learning pattern changes as shown in FIG. 4, for example, the cost function of the learning method described above has learned if it passes through the points of the given learning pattern. As the learning progresses, there is a problem that an over-learning state occurs as shown in FIG.

The present invention has been made in view of the above,
An object of the invention is to provide a neural network type pattern learning method and a pattern processing device capable of suppressing over-learning that occurs as learning progresses and capable of learning even when a learning pattern sharply changes.

[0009]

In order to achieve the above object, the invention according to claim 1 is a neural network type pattern learning method using a multilayer neural network composed of nonlinear threshold units, comprising: The differential value of the output pattern of the learning pattern with respect to the input pattern of the pattern is calculated, and the value obtained by multiplying the square of the differential value by the penalty of smoothness is added to the learning cost function, and the penalty of the smoothness is that the differential value is The neural network pattern learning method is characterized in that it is small when it is large, and it is large when the differential value is small, and overlearning is suppressed by changing the learning cost function.

According to a second aspect of the present invention, there is provided a neural network type pattern processing device using a multilayer neural network composed of non-linear threshold units, wherein the learning pattern output pattern corresponds to the learning pattern input pattern. A means for calculating a differential value, and means for adding to the learning cost function a value obtained by multiplying the square value of the differential value by a smoothness penalty that is small when the differential value is large and increases when the differential value is small. It is a neural network type pattern processing device characterized by having.

[0011]

In the neural network pattern learning method of the present invention,
The differential value of the output pattern with respect to the input pattern of the learning pattern is calculated, and the value obtained by multiplying the square of the differential value by the smoothness penalty is added to the learning cost function. The smoothness penalty is small when the differential value is large. When the differential value is small, it is increased and the learning cost function is changed to suppress overlearning.

Further, in the neural network type pattern processing apparatus of the present invention, the differential value of the output pattern with respect to the input pattern of the learning pattern is calculated, and the smoothness becomes small when the differential value is large and becomes large when the differential value is small. A value obtained by multiplying the square value of the differential value by the penalty of is added to the learning cost function.

[0013]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the arrangement of a neural network type pattern processing apparatus according to an embodiment of the present invention. The neural network type pattern processing apparatus shown in the figure includes a pattern input circuit 1 for inputting a pattern, a square error calculation circuit 2 for calculating a squared error, and a differential calculation circuit 3 for calculating a differential value of an output pattern with respect to an input pattern. , A coefficient calculation circuit 4 for calculating coefficients, and a coupling coefficient correction circuit 5 for correcting coupling coefficients.

Now, considering a multilayer neural network of m layers, the sum of inputs to the i-th unit of k layers is summed up.

[Outside 2] Then, the relation of these variables is as follows.

[0016]

[Equation 5] Consider the following squared error cost function as the cost function.

[0017]

[Equation 6] However, ε ₂ is a constant, and the sum for j is the number of units existing in the final layer m.

[Outside 3] It is a differentiation that makes the function realized by the circuit smooth and suppresses over-learning, but with the cost function with this penalty put, smoothness should be suppressed where the learning pattern changes sharply. I can't. Therefore, E ₂ determines how effective the penalty is with the term multiplied by the smoothness penalty of (1-ρ _j ). When ρ _j = 1, the learning pattern changes abruptly, and the penalty of smoothness is not included in the cost function. When ρ _j = 0,
Include a smoothness penalty in the cost function.

The change of weight, which is learning, is as follows.

[0019]

[Equation 7]

[Equation 8]

[Equation 9] For example, the following method can be used to determine ρ _j .

Consider a Hopfield neural network that is commonly used to optimize certain constraints, and consider E'as its cost function.

[0021]

[Equation 10] V (ρ _j ) is the penalty when ρ _j = 1 and
Give as follows.

[0022]

[Equation 11] However, C ₁ , B ₁ , B ₂ , B ₃ , and B ₄ are constants. The first and second terms are the penalties for ρ _j = 1 and
The third term is a term for ρ to converge to 0 or 1. Here, G is the following function.

[0023]

[Equation 12] However, λ is a constant.

Here, the following equation is considered as an equation for determining ρ _j .

[0025]

[Equation 13] However, ε ₃ is a constant.

The time derivative of E'is always 0 or less according to the following equation.

[0027]

[Equation 14] When the time derivative of E ′ becomes 0, the following holds for all ρ _j , and ρ _j always converges.

Dρ j / dt = 0 (27) Next, the pattern learning method will be described with reference to the flowchart shown in FIG. As described above, the differential value of the output with respect to the input is calculated by the differential calculation circuit 3 (step 110), and another Hopfield neural network is used to determine how effective the calculated differential value is in the cost function. Calculate using. That is, the coefficient by the Hopfield neural network is determined (step 120). Then, a multilayer neural network is trained with this cost function (step 130). By repeating the above procedure, it is possible to learn even while the learning pattern is changing sharply while suppressing over-learning.

[0029]

As described above, according to the present invention,
The differential value of the output pattern of the learning pattern with respect to the input pattern of the learning pattern is calculated, and the value obtained by multiplying the square of the differential value by the smoothness penalty is added to the learning cost function,
The smoothness penalty is small when the differential value is large, and is large when the differential value is small, and it is possible to suppress overlearning by changing the learning cost function. It becomes possible to perform pattern recognition, classification, and other processing that could not be realized.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a neural network type pattern processing apparatus according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a procedure of a neural network pattern learning method according to an embodiment of the present invention.

FIG. 3 is a diagram showing a three-layer neural network which is an example of a multilayer neural network.

FIG. 4 is a diagram showing points (both input and output are one-dimensional) of a learning pattern and a function realized by a neural network.

FIG. 5 is a diagram showing points (both input and output are one-dimensional) of a learning pattern and a function realized by an overlearned neural network.

[Explanation of symbols]

1 pattern input circuit 2 Square error calculation circuit 3 differential calculation circuit 4 coefficient calculation circuit 5 Coupling coefficient correction circuit

─────────────────────────────────────────────────── ─── Continuation of front page (72) Noboru Sonehara 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nippon Telegraph and Telephone Corporation (56) Reference JP-A-2-158860 (JP, A) (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06N 1/00-7/08 JISST file (JOIS)

Claims (2)

(57) [Claims] 1. A neural network type pattern learning method using a multilayer neural network composed of non-linear threshold units, wherein a differential value of an output pattern of a learning pattern with respect to an input pattern of the learning pattern is calculated, A value obtained by multiplying the square of the value by a smoothness penalty is added to the learning cost function, and the smoothness penalty is small when the differential value is large, and is large when the differential value is small. A neural network type pattern learning method characterized by suppressing over-learning by changing the. 2. A neural network type pattern processor using a multilayer neural network composed of non-linear threshold units, said means for calculating a differential value of an output pattern of a learning pattern with respect to an input pattern of a learning pattern, The smoothness penalty that is small when the differential value is large and is large when the differential value is small is 2 of the differential value.
And a means for adding a value multiplied by a multiplication value to a learning cost function.