JPH01271888A - Learning method for pattern recognition - Google Patents

Abstract

PURPOSE:To realize the title learning method with high efficiency and without deteriorating the resolution of identification by changing dynamically the parameter of a back-propagation learning method at learning and monitoring the output of a unit to change the weight vector. CONSTITUTION:For a structure model of a single unit of an intermediate layer, the inner product (s) is calculated by a multiplier 10 and an accumulator 11 at each unit between the input vectors (x1-x4) and the weight vectors (w1-w4). Then the output value (h) of each unit is calculated as h=1/[1+exp(-s/uO)] based on the sigmoid function 12 (uO: parameter). The product (s) is inputted by a controller 13 and the conditions ¦s¦<epsilon are decided to the set value epsilon (the positive value approximate to 0, e.g., 0.01) designated previously. Thus DELTAuO, i.e., the change component of uO is outputted. In this case, a constant, for example, can substitute for DELTAuO and the proportion to uO is also allowed as the function of uO.

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to a method for determining a circuit network and its parameters in order to recognize from input patterns the categories to which they belong.

(Prior Art) Conventionally, for example, there is a method called a pack propagation method regarding pattern recognition problems. This method is 1
In 986, MIT Publishing, “ParallelDist
"Rebutted Processing" Volume 2, R
Introduced by Umelhart and Hinton et al. The pack propagation method will be briefly explained below. Let the n-dimensional input pattern vector xll be xa=(Xal, X8□, xa3...l Xa
n). Let a be the category to which xa belongs. At this time, the S-dimensional category vector C is expressed as Ca ” (Cal+ Ca2+ Ca31 ””°l
Ca5), and it is expressed as Caa=1 Caj20 (j≠a). Performing recognition corresponds to constructing a function φ, Ca=Φ(X), which gives a category vector Ca for an arbitrary human cover turn vector Xa.In the pack propagation method, the following hierarchical circuit configuration is use

Here, we will proceed with the explanation using three layers as an example. The input layer has a number of units equal to the dimension of the input pattern vector, and the output value of each unit is made to correspond to each component of the vector.

The middle layer has m units. There is no particular limit to m here, but it is usually equal to or smaller than the number of categories or the dimension of the input button vector. For the output layer, prepare the number of units corresponding to the dimension of the category vector. A vector consisting of the output values of each unit of the intermediate layer is H1. A vector consisting of the output values of each unit of the output layer is Y.

H=(hl, h2. h3. ・・・・・hm)Y
= (yl, y2.y3....ys) Then, H,
Y is calculated using the following formula.

H=F(W,X) Y=F(V,H) Here, W and ■ are called weight (coefficient) matrices.

Note that F(1) is calculated using the following formula. A is m
When B is a matrix with rows and n columns, and B is an n-dimensional vector, B ” (b, b2. b3. ””・+ bn), F(1) is an m-dimensional vector F (A, B) =
(f(a□1*b1+a1□*b2+...+a1
n*bn).

f(a2□*b1 +a22*b2 + ・・”' +
a2n*bn) rf(aa, *b 1+ a32*b
2 + “”’ 1 asn*bn) 1 arr11*b
ENG+a1n2*b2+...ten an1n*bo))
It is. Here, f is a once-differentiable monotonically increasing function called a sigmoid function, and is a function expressed as f(x)=1/(1+exp(-x/uO)) with uO as a parameter.

To evaluate the recognition result, it is sufficient that the output vector Y correctly matches the category vector C for the human cover turn. The hack propagation method self-organizes the circuit parameter matrix W by pairing input button vectors X and category vectors C and learning them.
, ■.

First, if the vector H1 is calculated in the hidden layer and the vector Y is calculated in the output layer by the input of X, then Dy, = (c, -y
, )*y, *(1-y,), (i=1.-, s) calculates the error signal vector Dy1 of the output layer unit, and Dh defined as the error vector of the hidden layer unit.
,, Dh, = (Dyyo*v,, +-+ DyB*v8
□) *h, *(1-h,) (i=1....., m) Calculate.

Furthermore, according to the following equation, each element of the weighting coefficient matrix is modified as calculated by the following equation.

■1j←A*■1j1α*Dy, *h.

w,←γ*W1j+α*Dhi*XJ Here, γ and α are parameters for controlling the amount of correction, and are often set to γ~1 0 × α × 1.

The pack propagation (learning) method is such that if the above correction operation is repeated for various input buttons, ■, W converges and an output that matches the correct category vector C as Y. It is.

(Problems to be Solved by the Invention When constructing a discriminant function using the puncture propagation method, there are the following problems.

There are uO, α, and γ as parameters of the learning system,
These values should be set to optimal values, but no method has yet been found to find them. 1 for γ
．． It is known from experience that it is sufficient to set it to 0 or a slightly smaller value such as 0.99.

However, the values of uO and 0 are deeply related to the time it takes for learning to converge (convergence speed), and the larger uO is and the smaller a is, the slower the convergence speed is, and the learning There was a problem that it took a huge amount of time.

In addition, uO is a parameter for the decomposition ability to identify patterns, and there should be an optimal value depending on the statistical properties of the pattern group to be identified and the number of units in the middle layer, but a method for determining it has not yet been found. Not yet.

The purpose of the present invention is to dynamically change the parameter UO during learning in the pack propagation learning method, or change the weight vector by monitoring the output of the unit, without reducing the discrimination resolution ability. The purpose of this invention is to provide a method for efficient learning by speeding up the convergence of learning and adaptively increasing the number of intermediate layer units.

(Means for Solving the Problems) The first invention of the present application has weight values for excitatory connections and inhibitory connections having directionality between each unit, and the output value of the unit having the input connection and its weight value. an input unit group that receives input from the outside by a unit whose output value is a value obtained by applying a sigmoid function to the product-sum value X with the weight value corresponding to the combination; and an output unit group that outputs recognition results to the outside. ,
There is no human output directly to the outside, and the above-mentioned crabnit group,
Or, in a system composed of three types of unit groups, including the output unit group or an intermediate unit group that are themselves interconnected, the sigmoid function of the unit for the product-sum value %(1), the weight values are randomly initialized, and the weight values are learned self-organized using an input pattern and a teacher pattern. It is characterized in that when the IQ)d value of the sum value X is smaller than a predetermined value, U is increased, and when it is not, U is decreased.

Further, the second invention of the present application is the first invention, in which the output value of each unit is monitored for each pattern input, and the difference between the output value and the output value for the previous input pattern is equal to or less than a predetermined setting value. As long as the fixed coefficient is calculated by integrating the absolute value of the output value minus 0.5, and if the fixed coefficient exceeds another set value, all the weight values of that unit are Alternatively, it is characterized by randomly re-initializing a part.

Furthermore, the third invention of the present application is based on the first invention, in which the output value of each unit is monitored for each pattern input, and the difference between the output value and the output value for the previously input pattern is equal to or less than a pre-specified setting value. 0.5 from the output value as long as
Calculate the fixation coefficient by integrating the absolute value of the subtracted value,
If the fixed coefficient exceeds another setting value, a part of the weight value of that unit is set to zero, thereby creating a plurality of orthogonal weight values, and using a group of units using these as weight values, the above-mentioned It is characterized by replacing the unit.

(Example) Next, the present invention will be explained with reference to the drawings.

An embodiment in which the present invention is applied to a pattern recognition system will be described below.

FIG. 1 shows a network structure of units to which a three-layer backpropagation learning method can be applied. For simplicity, the input layer has 4 units, the middle layer has 3 units, and the output layer has 2 units. The input layer units are Xl, X2 . X3. X4, h-engine the middle layer unit,
h2. Let h3 and output layer unit be distinguished from 3'l+y2. Further, the weights of the connection from the input layer unit X to the hidden layer unit h1 are set as w and J, and similarly the weight of the connection from the hidden layer unit to the output layer unit yk is evaluated as 1. Furthermore, the teacher signal to the output unit y8 is represented by cl.

Next, a model of the structure of one unit of the intermediate layer in FIG. 1 will be shown, and the present invention will be explained in detail. Note that the exact same unit can be used even if there is an output layer unit or another intermediate layer. However, since the input layer unit actually only outputs to the intermediate layer unit, a structure that simply holds and outputs the value is sufficient.

Next, FIG. 3 shows a structural model of a unit conventionally used in the bank propagation method. In the following, each input value will be explained as an element of a vector.

In each unit, first input vectors (xl, x2 .
x3°X4) and the weight vector (w12w22w32
The inner product S with w4) is calculated by the multiplier 10 and the accumulator 11. Here, a weight vector is prepared for each unit. Furthermore, at this time, it is common practice to add a constant input. That is, as the fifth input, X5=1 (constant=1
) and the weight w5 for this is also added,
It can be treated in exactly the same way as if the dimension had increased.

The output value of this unit is calculated by the sigmoid function 12 as h=1/(1+exp(-s/uO)). Conventionally, uO has been set to the same value throughout the network.

FIG. 2 shows a structural model of one unit of the intermediate layer of the first invention. As shown in this model, in the first invention, the parameters of the sigmoid function are set independently for each unit, and these values are adaptively changed using the inner product value S. At this time, the increase or decrease in UO is increased if the value of S is sufficiently close to zero, and otherwise decreased. If the parameter UO (positive value) of the sigmoid function is made small, it will be removed from the equation, but it will asymptotically approach a so-called step function, a threshold open number. Also uO
If it is made large, it becomes close to linearity (h is proportional to S) near S=O.

From this, when the discrimination between the batons is good, in other words, when the absolute value of S is sufficiently large, uO is made small so that the batons with noise and deformation can be tolerated, and conversely, when there are similar battens, , S becomes small, so uO may be increased to make h proportional to S so that a slight difference in S values can be utilized. Therefore, in the present invention, the controller 13 inputs S and determines 1 x 8 conditions for a prespecified set value ε (a positive value close to O, such as 0.01), and The amount of change, ΔuO, is outputted. Further, here, ΔuO may be a constant, for example, or may be changed proportionally to UO as a function of uO. However, uO should be a positive value.

FIG. 4 shows a structural model of one unit of the intermediate layer regarding the second invention. This is a method of monitoring output values, finding a unit that always outputs the same fixed value, and effectively utilizing that unit.

That is, the output value of each two nits is monitored for each pattern input,
As long as the difference from the output value for the previous input pattern is less than or equal to the pre-specified setting value, the fixation coefficient is calculated by integrating the absolute value of the output value minus 0.5, and If the weighting factor exceeds another set value, all or some of the weight values for that unit are randomly initialized again.

In FIG. 4, the fixed coefficient calculation circuit 14 calculates the fixed coefficient q according to the following equation, for example. Here, t indicates the order in which learning patterns are presented.

q(t) = (q(t-1) ten years old (t) 0.51
)*δ(h(t) −h(t-1)) Here, the function δ can be, for example, a so-called delta function. In that case, cumulative calculation is performed as long as the output for the previous input pattern completely matches the output for the currently input pattern. The function δ may be a function that is 1 when (t)-h(t-1)A for the first set value A, and in this case, the so-called delta function is included as the case A=00. .

Next, the reinitialization circuit 15 reinitializes the weight vector W of the unit when the condition q(t)>B holds for the second set value B. Here, it is sufficient that the second set value B is 0.5*N, for example, for the number N of patterns included in 1 cent when learning patterns for all categories are 17 bits. .

In order for each unit to contribute to pattern identification, the output must change depending on the person cover turn. According to the present invention, a unit that happens to have a fixed output and become invalid due to poor initialization or convergence due to learning is effectively used again.

Next, a third invention will be described as another method of finding a unit that outputs a fixed value for a long period of time during learning and effectively utilizing that unit.

In the third invention, as in the second invention, the output value of each unit is monitored for each pattern input, and the difference between the output value and the output value in the column of the previous input pattern is less than or equal to a pre-specified setting value. As long as the output value is subtracted by 0.5, the absolute value is integrated to calculate the fixing coefficient ψ° (＼), and it is determined whether the fixing coefficient exceeds another set value. It's the same until you do it.

Note that in the third invention, the set value for determining whether the fixed coefficient exceeds the fixed coefficient is set smaller than that in the second invention.

Here, the third invention is to create a plurality of orthogonal weight vectors by setting a part of the weight values of the units to zero rather than random re-initialization, and to form a group of units using these as weight vectors, the Unino I I am trying to replace .

Next, the replacement of the unit will be explained using the $5 diagram. Figure 5 shows the four-dimensional weight vector W-(wl,
W2. W3. W4) From the two weight vectors I and W'=(w'□, w'2.w'3.ws, W"=
``,,w''22w1'32w''4) is shown.Here, W' and Wl+ are assumed to have an orthogonal relationship with an internal value of 0.

For example, if the number of middle layer units is insufficient and identification is not possible, the output of the middle layer will not change even if the input button is different. Therefore, it is necessary to detect this and increase the number of middle layer units. However, the weight vectors learned in this case are common among the patterns that you want to identify, so if you divide the learned weight vectors into orthogonal vectors, you can use the learning results so far. mid-tier units can be added efficiently.

In the present invention, different parts of vectors are set to zero in order to obtain orthogonal vectors, and therefore orthogonal vectors can be easily obtained. Note that it is of course possible to add an uncorrelated random component to the zeroed element.

(Effects of the Invention) As explained above, according to the present invention, when performing backpropagation learning method for backpropagation learning, the learning speed can be increased without sacrificing recognition accuracy, and the middle layer unit Effective use has the effect of efficiently learning weights. Moreover, the first, second, and third inventions can be used in combination.

[Brief explanation of the drawing]

FIG. 1 shows a network structure of units to which a three-layer backpropagation learning method can be applied. Second
The figure shows a structural model of one unit of the intermediate layer for the first invention. FIG. 3 shows a structural model of a unit conventionally used in the backpropagation method. FIG. 4 shows a structural model of one unit of the intermediate layer for the second invention. FIG. 5 is a diagram illustrating the replacement of units in the third invention. In the figure, 10... Multiplier, 11... Accumulator, 12... Sigmoid function, 13... Controller, 14... Fixed coefficient calculation circuit, 15... Re-initialization circuit.

Claims (3)

[Claims] (1) Each unit has weight values for excitatory connections and inhibitory connections with directionality, and the sum of products x of the output value of a unit with input connections and the weight value corresponding to that connection is a sigmoid. An input unit group that receives input from the outside by a unit whose output value is a value subjected to a function, an output unit group that outputs recognition results to the outside, and an input unit that does not have input/output directly to the outside. A system consisting of three types of unit groups, such as a group, or the output unit group, or an intermediate unit group that are themselves interconnected, and as a sigmoid function f(x) of the unit for the product-sum value x. In a pattern recognition learning method in which the weight value is randomly initialized using f(x)=1/(1+exp(-x/u)), and the weight value is learned self-organized by an input pattern and a teacher pattern. , u is prepared for each unit, and when the absolute value of the sum of products x is smaller than a predetermined value, u is increased; otherwise, u is
A pattern recognition learning method characterized by reducing . (2) In the pattern recognition learning method of claim (1),
The output value of each unit is monitored for each pattern input, and as long as the difference from the output value for the previous input pattern is less than or equal to a pre-specified setting value, 0.
A fixing coefficient is calculated by summing the absolute value of 5. If the fixing coefficient exceeds another set value, all or part of the weight values of that unit are randomly initialized again. A pattern recognition learning method characterized by (3) In the pattern recognition learning method for the unit according to claim (1), the output value of each unit is monitored for each pattern input, and the difference from the output value for the previous input pattern is a pre-specified set value. Calculate the fixing coefficient by integrating the absolute value of the output value minus 0.5, as long as the fixing coefficient exceeds another set value, then the unit's A pattern recognition learning method characterized in that a plurality of orthogonal weight values are created by setting a portion of the weight values to zero, and the units are replaced by a unit group having these weight values as weight values.