JP2013016006A - Learning apparatus for pattern classification - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a learning apparatus capable of enhancing accuracy of a pattern classifier.SOLUTION: A misclassification measure value D(x;Λ) of an input pattern is defined by formula (1). A value of each parameter is adjusted so that a value of a minimization target function is minimized.

Description

  The present invention relates to a pattern classification learning apparatus that classifies some measurement data into one of predetermined classes, and in particular, LGM-MCE (Large Geometric Margin Minimum Classification Error) learning is used to expect higher classification accuracy. The present invention relates to a learning device that can perform learning.

[Pattern recognition and learning]
Pattern recognition is an important technique in the interface between humans and machines. The pattern recognition technique is used in various aspects such as speaker identification, utterance content recognition, person identification by face image, and character recognition. In short, pattern recognition is an operation of classifying an observed value pattern obtained by observing some physical phenomenon into one of a plurality of classes. Although these tasks are relatively easy for humans, it is not easy to make them work. A device that performs such work is a pattern recognition device when called comprehensively. In order for the pattern recognition apparatus to perform pattern recognition, a preliminary operation called learning is required to obtain parameters necessary for classification by statistically processing learning data.

  As learning methods for such pattern classification, there are an LGM-MCE method disclosed in Non-Patent Document 1 and an MCE method disclosed in Non-Patent Document 2. Both methods employ classification decision rules based on discriminant functions. These will be described below.

Consider a classification task that assigns an input pattern (observed value) xεΧ to any one of J classes (classes) C ₁ ,..., C _J. Here, Χ represents the entire input pattern space. The LGM-MCE method (Non-Patent Document 1) employs the following classification decision rule based on a discriminant function, as in the early MCE method (Non-Patent Document 2).

Here g _j (x; Λ) is the discriminant function for a class C _j, represents the degree to which x is attributable to the class C _j. Λ represents a learning parameter (adjustment parameter) of the classifier, and g _j (x; Λ) (j = 1,..., J) is second-order differentiable with respect to x and Λ.

Next, focusing on the classification decision boundary formed by the classification decision rule of the above equation, the Euclidean distance r between x and the boundary is considered as a learning sample near the boundary where x is correctly classified. This r is nothing but the geometric margin, and by increasing this value, the possibility of accurate classification of unknown patterns that are easily misclassified increases. Assuming that xεC _y , the geometric margin is expressed by the following equation (generally approximately) from the result of Non-Patent Document 1.

Here, d _y (x; Λ) is a misclassification measure of the following equation defined by the initial MCE method (ψ> 0).

If ψ → ∞, d _y (x; Λ) is as follows.

Here, C _i is a best-incorrect class for x. In other words, the geometric margin is approximately equal to the positive / negative reversal of the misclassification measure (called the function margin) normalized by the norm of the gradient.

The LGM-MCE learning method adopts the following D _y (x; Λ) corresponding to the positive / negative inversion of this geometric margin as a new misclassification measure.

A positive value of D _y (x; Λ) corresponds to misclassification, and a negative value corresponds to positive classification. This property is in common with the misclassification measure d _y (x; Λ) in the early MCE method. Hereinafter, the conventional misclassification measure d _y (x; Λ) and the new misclassification measure D _y (x; Λ) are referred to as a function margin type misclassification measure and a geometric margin type misclassification measure, respectively.

  The ideal state of Λ is to minimize the classification error number risk (classification error probability for all patterns) of the following equation consisting of an infinite number of samples.

Here, p represents a probability density function, and 1 (A) is an indicator function that returns 1 if the proposition A is true and returns O if the proposition A is false. Therefore, 1 (D _y (x; Λ)> 0) represents a loss of classification error that returns 1 if it is a misclassification and O if it is a correct classification. This function is illustrated by the graph 22 in FIG. However, the classification error number loss is not differentiable with respect to Λ. In reality, only a finite number of learning samples can be used. Thus, the LGM-MCE method (similar to the initial MCE method) replaces the classification error number loss with a smooth (differentiable with respect to Λ) logistic function and minimizes this average for a finite learning sample. The logistic function is shown by the graph 32 in FIG. The smoothed classification error number loss for x∈C _y is defined by the following equation (α _y > 0).

The minimization target function aimed by the LGM-MCE learning method is as follows: Ω _N = {x _n , y _n } _{n = 1} ^N as a supervised learning sample set consisting of N samples, and an empirical average represented by the following equation Loss L (Λ).

The minimization of L (Λ) in the above equation not only directly aims at minimizing the number of classification errors for a finite learning sample, but as shown in FIG. 2, loss_l _y (D _y ) (character is underscore "_" was added immediately before, indicating that the character is drawn in italics in the formula.) it is because it is a monotonically increasing function of D _y, greatly increases the D _y in the negative direction Let Thereby, the positive / negative inversion of D _y , that is, the geometric margin (r in FIG. 2) increases.

Of course, L (Λ) in the above equation consisting only of a finite number of learning samples is only an approximation of the classification error number risk R (Λ) including all unknown patterns not included in the learning sample set, Λ that minimizes L (Λ) generally does not minimize R (Λ). However, by setting an appropriate finite value α _y (see equation (7)), the evaluation criterion L (Λ) becomes a smooth function and improves learning tolerance to unknown patterns not included in the learning sample set. . That is, this smoothing makes the loss sensitive not only to a given learning sample but also to the vicinity thereof, and the effect of increasing the number of learning samples can be obtained.

For minimizing L (Λ), not only a batch method such as steepest descent method, but also an adaptive learning method that adjusts Λ every time one sample (x, y) is extracted from Ω _N is widely used. It has been. The adjustment mechanism of Λ in the method is given by the following equation. Here, _l ^′ _y is a derivative of the loss function _l _y , and the learning coefficient ε is variable at each iteration step.

The above is the outline of the LGM-MCE method.

H. Watanabe et al., “Minimum Error Classification with Geometric Margin Control”, IEEE ICASSP Proceedings, pp. 2170-2173, March 2010 (H. Watanabe et al., Minimum error classification with geometric margin control. ”In Proc. IEEE ICASSP, pp. 2170-2173 Mar. 2010) B. -H. Juan and S. Katagiri, “Distinguishing Learning for Minimum Error Classification” IEEE Signal Processing Transactions, Vol. 40, No. 12, pp. 3043-3054, December 1992 (B.-H. Juang and S. Katagiri, “Discriminative learning for minimum error classification,” IEEE Trans. Signal Processing, vol.40, no.12, pp.3043-3054, Dec. 1992.)

  The boundary between each class in the conventionally implemented LGM-MCE learning method is given by a linear function. That is, the boundary between each class is defined by a straight line in the case of a two-dimensional space, a plane in the case of three dimensions, and a hyperplane of dimension-1 in the case of four or more dimensions.

  This is because a linear function is used as the discriminant function. In fact, the use of a linear function has the effect that the process for learning is relatively simple. However, on the contrary, there is a problem that the LGM-MCE learning method is difficult to apply only in a limited field. Furthermore, the LGM-MCE learning method using a linear discriminant function has a problem that it is difficult to increase the accuracy of classification.

  Therefore, an object of the present invention is to provide a learning device that can increase the accuracy of the obtained classifier in the learning device for a pattern classifier by LGM-MCE learning.

The classifier learning device according to the first aspect of the present invention is a classifier learning device that classifies an input pattern into one of J classes C _j (j is an integer from 1 to J). This apparatus uses a learning sample storage means for storing a learning sample set including N (N is a positive integer) supervised input pattern, and a learning parameter set Λ of a classifier is initialized by a predetermined setting method. Initializing means for converting to an initial value. Class C input pattern x of training samples in the set which belongs to _y is measure the degree to which misclassified other classes Geometric Margin type misclassification measure value D _y (x; lambda) is defined below.

However ψ is a positive real number, g _y (x; Λ) for each of J-number of class C _y, the input pattern x of training samples in the set to determine the degree of whether belonging to the class Therefore, d _y (x; Λ) is called a function margin-type misclassification measure.

For the vector λ = [λ ₁ ... Λ _k ] in which _k variables included in the learning parameter set Λ are arranged, the partial differentiation of the misclassification measure value D _y (x; Λ) by the vector λ is the function d _y (x ; Λ) gradient vector ∇ _x d _y is given by the following equation, where the superscript T represents the transpose of the matrix.

The learning apparatus further includes a bias of the misclassification measure value D _y (x; Λ) so that the value of the predetermined minimization target function L (Λ) with respect to the learning parameter set Λ is minimized with respect to the learning sample set. Parameter adjustment means for adaptively adjusting the value of each parameter included in the learning parameter set Λ using differentiation.

Preferably, the discriminant function for class C _j (j = 1,..., J) has M prototypes belonging to class C _j as p _{j, 1} ,..., P _{j, M} and positive definite corresponding to each prototype. The value matrix is given by the following equation as A _{j, 1} ,..., A _{j, M.}

However, p _j and A _j are distance distances defined by the following equation between the input pattern x and the prototype belonging to the class C _j.

P _j = p _{j, m (j)} and A _j = A _{j, m (j),} where _{m (j)} is the prototype index that minimizes. The function margin type misclassification scale d _y (x; Λ) is given by the following equation.

The geometric margin type misclassification measure D _y (x; Λ) and its partial derivative are given by the following equations.

More preferably, the positive definite matrix A _{j, 1} ,..., A _{j, M} is a diagonal matrix having a positive diagonal component as follows.

The parameters a _{j, 1} , ..., a _{j, D} are included in the learning parameter set Λ, and the parameters a _{y, d} and a _{i, d of the} geometric margin type misclassification measure D _y (x; Λ) The partial differentiation with respect to (d = 1, ..., D) is expressed by the following equation.

More preferably, the discriminant function for class C _j (j = 1,..., J) may be given as:

Where p _{j, 1} , ..., p _{j, M} are M prototypes belonging to class C _j , and w _{j, m} (m = 1, ..., M) is the mth prototype. It is a weight for the Euclidean distance. The learning parameter set Λ and the function margin type misclassification measure d _y (x; Λ) may be given by the following equations.

However Class C _y and C _i are the correct class and best-The net part class x respectively. The geometric margin type misclassification scale D _y (x; Λ) and its partial derivative are expressed by the following equations.

More preferably, the classifier is a three-layer feedforward neural network classifier including an input layer, an intermediate layer, and an output layer. The input layer includes D + 1 units. The intermediate layer includes M + 1 units. The m-th unit (m = 1,..., M) in the intermediate layer performs output by applying a non-linear function f _m to the weighted sum of the output from the input layer. The output layer includes J units. Each j-th unit (j = 1,..., J) outputs the weighted sum of the outputs from the intermediate layer as a discriminant function g _j of class C _j . The discriminant function for class C _j (j = 1,..., J) is given by

Where w _{m, d} (m = 1, ..., M; d = 0,1, ..., D) is for the coupling from the d-th unit of the input layer to the m-th unit of the intermediate layer. The weighting factor, v _{j, m} (j = 1, ..., J; m = 0,1, ..., M) is for the coupling from the mth unit in the middle layer to the jth unit in the output layer It is a weighting factor. The learning parameter set Λ includes weighting coefficients w _{m, d} (m = 1, ..., M; d = 0,1, ..., D) and v _{j, m} (j = 1, ..., J ; m = 0,1, ..., M). The geometric margin type misclassification scale D _y (x; Λ) and its partial differentiation are as follows.

  The computer program according to the second aspect of the present invention causes a computer to function as each means of the learning device for any one of the classifiers described above.

It is a graph of the classification error number loss function in the LGM-MCE learning method. It is a graph of the smoothing classification error number loss function by the logistic sigmoid function in a LGM-MCE learning method. It is a block diagram of the character recognition system using the classifier which concerns on one embodiment of this invention. It is a flowchart of the program for performing the learning of a classifier by the 1st Embodiment of this invention. It is a flowchart of the program for performing the learning of a classifier by the modification of the 1st Embodiment of this invention. It is a flowchart of the program for performing the learning of a classifier by the 2nd Embodiment of this invention. It is a figure which shows typically the structure of the neural network used with the system of the 3rd Embodiment of this invention. It is a flowchart of the program for performing the learning of a classifier by the 3rd Embodiment of this invention. It is a figure which shows the hardware external appearance of the general purpose computer system which implement | achieves embodiment of this invention. It is a block diagram of the internal structure of the computer system shown in FIG.

  Embodiments of the present invention will be described below. In the following description and drawings, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  In the following embodiment, a nonlinear function is used as the discriminant function. By using a non-linear function, conceptually, the classification boundary becomes a hypersurface in a sample space of four or more dimensions, and it can be expected that the classification accuracy is further improved.

[Partial differential calculation]
As described above, when a nonlinear function is employed as a discriminant function, there is a problem that partial differential calculation is difficult particularly in the case of a multidimensional space. In the following, the partial differential calculation of the geometric margin type misclassification measure D _y (x; Λ) for the nonlinear discriminant function will be generally discussed.

The LGM-MCE training for the general form of the discriminant function in order to perform in accordance with equation (9), a D-dimensional input pattern belonging to the class _{C y x = [x 1 ...} x D] T
As false Geometric Margin type nonlinear in an expression (9) Classification Scale D _y to derive;; (lambda x) below variable differential ∇ of (x Λ) Λ D _y. However, superscript T represents transposition of a matrix.

First _{A = || ∇ x d y (} x; Λ) is denoted by || ^2. In this _{case, 1 / || ∇ x d y} (x; Λ) || = A -1/2. When D _y (x; Λ) is partially differentiated by one variable λ included in Λ, the following is obtained.

Further, when A is partially differentiated by λ, the following is obtained.

Substituting this into equation (10) gives:

For the vector λ = [λ ₁ ... Λ _k ] ^{T in which} _k variables included in Λ are arranged, the partial differential vector is expressed by the following equation from Equation (11).

Incidentally, the left-hand side of equation (14) is a Jacobian matrix of the gradient vector ∇ _x d _y.

Eventually, from equations (10) and (13), the partial differentiation of the geometric margin type misclassification measure D _y (x; Λ) with respect to the vector value variable is given as follows.

[Application of LGM-MCE learning method to quadratic discriminant function classifier]
《Partial differentiation of discriminant function and geometric margin type misclassification measure》
The discriminant function for class C _j (j = 1,..., J) has M prototypes belonging to class C _j as p _{j, 1} ,..., P _{j, M} , and a positive definite matrix corresponding to each prototype as A _{j. , i} ,..., A _{j, M} are given by the following equation.

P _j and A _j in equation (16) are p _j = p, where m (j) is a prototype index that gives the minimum value of the distance Distance in the sense of the following equation among x prototypes belonging to class C _{j j, m (j)} and A _j = A _{j, m (j)} .

Alternatively, A _{j, 1} =... = A _{j, M} = A _j . The specific form of A _{j, m} does not matter. For example, it may be given as an inverse matrix of the covariance matrix of the learning sample set belonging to the m-th cluster in class C _j (determined by the K-means method, etc.) Alternatively, A _{j, m} may be given as an inverse matrix of a diagonal covariance matrix of learning sample sets belonging to the same cluster. The function margin type misclassification scale is given by the following equation.

However C _y and C _i are the correct class and best-The net part class x respectively. At this time, the differential of the function margin type misclassification scale is as follows.

The geometric margin type misclassification scale D _y (x; Λ) and its partial differentiation are expressed by the following equation from the equation (15).

<< First Embodiment: System Configuration >>
Referring to FIG. 3, a character recognition system 40 as an example using the classifier according to the present embodiment performs classification based on supervised sample data to classify character image data into character categories. , A learning unit 50 for learning by the classifier 80 by learning using digital character data with a teacher character category, a touch panel 52 for inputting a character image, and a signal output by the touch panel 52 Is recognized as a character category 56 using the classifier 80 learned by the learning unit 50.

  The learning unit 50 stores character image data with a teacher character category, and a predetermined character feature (position information, statistical moment, edge count, etc.) is extracted from the storage unit 70 by calculation. Using the character feature extraction module 74, the learning data storage unit 76 that stores the teacher data output from the character feature extraction module 74 as learning sample data, and the learning sample data stored in the learning data storage unit 76, And a learning module 78 that performs learning of the classifier 80 by a learning method to be described later. In the following description, it is assumed that the character feature amount is represented by a vector. That is, if the character feature vector is x and the character category as teacher data is y, each sample data can be expressed in the form (x, y).

  The character recognition unit 54 converts the output signal of the touch panel 52 into a digital signal, and the character output of the signal output from the binarization processor 90 by the same method as the character feature extraction module 74. Character extraction module 94 that extracts and outputs the character, and a decoder 96 that applies the classifier 80 to a series of character feature output from the character feature extraction module 94 and outputs an output character category 56. .

《Learning algorithm》
Hereinafter, an algorithm for performing parameter learning in the present embodiment will be described with reference to FIG. Referring to FIG. 4, a program for realizing this algorithm includes the following steps.

1. Initialization step 120. Here, the initial value p _{j, m} ⁽⁰⁾ of the prototype p _{j, m} and the positive definite matrix A _{j, m} are set (j = 1, ..., J; m = 1, ..., M). . In addition, an upper limit value E of the epoch count e is set.

  2. Step 122 of executing processing 124 including the following substeps (a) to (c) for the number of epochs e = 0, 1,.

(A) Sub-step 140. Here, if necessary, the optimum value α _y (y = 1,..., J) of the loss smoothness parameter is obtained. The value of the loss smoothness parameter α _y may be determined empirically, but is determined using the loss function smoothness automatic setting method described in Japanese Patent Application No. 2010-184334 filed earlier by the present inventor. It is more preferable to use the obtained loss smoothness parameter α _y .

(B) Sub-step 142. Here, the supervised learning sample (x, y) is taken out from the learning sample set Ω _N , and the processing 144 including the following sub-steps 160 to 174 is executed for each learning sample.

(B1) The discriminant function value g _j is calculated according to the following procedure (j = 1,..., J) (substep 160).

(B2) The index i of the best-incorrect class for x is obtained (substep 162).

(B3) A function margin type misclassification measure is calculated by d _y = −g _y + g _i (substep 164).

    (B4) A gradient vector for the input pattern of the function margin type misclassification measure is calculated (step 166).

(B5) gradient to calculate the norm || ∇ _x d _y || vector (substep 168).

(B6) The geometric margin type misclassification measure D _y (x; Λ) is calculated by D _y = d _y / || ∇ _x d _y || (substep 170).

(B7) The partial differentiation of the geometric margin type misclassification measure D _y (x; Λ) is calculated as follows (substep 172).

(B8) The parameter is updated by the following formula (substep 174).

The above sub-steps (b1) to (b8) are the contents of the process 144. The processing 144 is executed once for each learning sample in turn, and when the adjustment for all the samples is completed, a new prototype p _{j, m} ^{(e + 1)} (j = 1,..., J; m = 1, ..., M).

(C) Step 146. Here, the order of learning samples in the learning sample set Ω _N is shuffled.

  The above steps (a) to (c) are the contents of the process 124. After performing the process 124 for the epoch e, the epoch is advanced by 1 (e = e + 1) and the same process is repeated.

  Thus, when the number of epochs reaches the planned upper limit value E and the processing 124 is finished, a parameter set Λ after learning by the classifier 80 is obtained.

<< Modification of First Embodiment >>
Further, not only the prototype {p _{j, m} } but also the positive diagonal component may be learned by using A _j as a diagonal matrix. here

Adjust a _{j, d} (d = 1,..., D). The partial differential for each of the function margin type misclassification scales a _{y, d} , a _{i, d} (d = 1, ..., D) is as follows.

Here, p _j = [p _{j, 1} ... P _{j, D} ] ^T. The second-order variable derivative of d _y is

It becomes. However, δ _{i, j} = 1 (if i = j), 0 (if i ≠ j). The partial differentiation with respect to each a _{y, d} , a _{i, d} (d = 1,..., D) of the geometric margin type misclassification scale D _y is expressed by the following equation from Equation (12).

Regarding the learning coefficient ε in equation (9), ε for correction of p _{j, m} and ε for correction of a _{j, d} may be different from each other.

《Modification learning program》
Referring to FIG. 5, the program that realizes the algorithm of the modified example of the first embodiment includes the following steps.

1. Initialization step 220. Here, the initial values p _{j, m} ⁽⁰⁾ (j = 1, ..., J; m = 1, ..., M) of the prototype p _{j, m} and the variables corresponding to the positive definite matrix A _j Set initial values a _{j, d} ⁽⁰⁾ (j = 1, ..., J; d = 1, ..., D). In addition, an upper limit value E of the epoch count e is set.

  2. Step 222 of executing processing 224 including the following substeps (a) to (c) for the epoch counts e = 0, 1,.

(A) Sub-step 240. Here, if necessary, the optimum value α _y (y = 1,..., J) of the loss smoothness parameter is obtained.

(B) Substep 242. Here, the supervised learning sample (x, y) is taken out from the learning sample set Ω _N , and the processing 244 including the following sub-steps 260 to 274 is executed for each learning sample. “(E)” on the right shoulder of the variable represents an epoch number.

(B1) The discriminant function value g _j is calculated according to the following procedure (j = 1,..., J) (substep 260).

(B2) A best-incorrect class index i for x is obtained (substep 262).

(B3) a function margin type misclassification measure calculated by _{d y = -g y + g i} ( substep 264).

    (B4) A gradient vector for the input pattern of the function margin type misclassification measure is calculated (step 266).

(B5) gradient to calculate the norm || ∇ _x d _y || vector (substep 268).

(B6) The geometric margin type misclassification measure D _y (x; Λ) is calculated by D _y = d _y / || ∇ _x d _y || (substep 270).

(B7) The partial differentiation of the geometric margin type misclassification measure D _y (x; Λ) is calculated as follows (substep 272).

(B8) The parameter is updated by the following formula (substep 274).

The above substeps (b1) to (b8) are the contents of the processing 244. The processing 244 is executed once for each learning sample in turn, and when the adjustment for all the samples is completed, a new prototype p _{j, m} ^{(e + 1)} (j = 1,..., J; m = 1, ..., M) and matrix parameters a _{j, d} ^{(e + 1)} (j = 1, ..., J; d = 1, ..., D).

(C) Step 246. Here, the order of learning samples in the learning sample set Ω _N is shuffled.

  The above steps (a) to (c) are the contents of the process 224. After performing the process 224 on the epoch e, the epoch is advanced by 1 (e = e + 1) and the same process is repeated.

  Thus, when the number of epochs reaches the predetermined upper limit value E and the processing 224 is completed, a parameter set Λ after learning of the classifier 80 according to this modification is obtained.

[Second Embodiment: Weighted Prototype Classifier]
As an example using a nonlinear discriminant function, consider a weighted prototype type classifier. In this case, as in the first embodiment, the classifier can be learned as follows.

《Partial differentiation of discriminant function and geometric margin type classification measure》
The discriminant function for class C _j (j = 1,..., J) is given by

Where p _{j, 1} , ..., p _{j, M} are M prototypes belonging to class C _j , and w _{j, m} (m = 1, ..., M) is the m-th prototype. It is a weight for the Euclidean distance. The learning parameter set Λ and the function margin type misclassification measure are given below.

However, the class C _y and C _i are the correct class and best-The net part class x respectively. At this time,

(Where I is a unit matrix), the geometric margin type misclassification measure D _y (x; Λ) and its partial differentiation are expressed by the following equation from equation (15).

Regarding the learning coefficient ε in equation (9), ε for correction of p _{j, m} and ε for correction of w _{j, m} may be different from each other.

<< Learning program of the second embodiment >>
Referring to FIG. 6, the program for realizing the algorithm of the second embodiment includes the following steps.

1. Initialization step 320. Here, the initial value p _{j, m} ⁽⁰⁾ (j = 1, ..., J; m = 1, ..., M) of the prototype p _{j, m and} the initial value w of the weight coefficient w _{j, m} Set _{j, m} ⁽⁰⁾ (j = 1, ..., J; m = 1, ..., M). In addition, an upper limit value E of the epoch count e is set.

  2. Step 322. Here, a process 324 including the following substeps (a) to (c) is executed for the number of epochs e = 0, 1,. The following are the sub-steps constituting the process 324.

(A) Sub-step 340. Here, if necessary, the optimum value α _y (y = 1,..., J) of the loss smoothness parameter is obtained.

(B) Sub-step 342. Here, the supervised learning sample (x, y) is taken out from the learning sample set Ω _N , and processing 344 including the following sub-steps 360 to 374 is executed for each learning sample. “(E)” on the right shoulder of the variable represents an epoch number. The following (b1) to (b8) are sub-steps constituting the process 344.

(B1) The discriminant function value g _j is calculated (j = 1,..., J) (substep 360).

(B2) The index i of the best-incorrect class for x is obtained (substep 362).

(B3) a function margin type misclassification measure calculated by _{d y = -g y + g i} ( substep 364).

    (B4) A gradient vector for the input pattern of the function margin type misclassification measure is calculated (step 366).

(B5) gradient to calculate the norm || ∇ _x d _y || vector (substep 368).

(B6) The geometric margin type misclassification measure D _y (x; Λ) is calculated by D _y = d _y / || ∇ _x d _y || (substep 370).

(B7) The partial differential of the geometric margin type misclassification measure D _y (x; Λ) is calculated as follows (m = 1,..., M) (substep 372).

(B8) Parameter update is performed by the following equation (m = 1,..., M) (substep 374).

The above substeps (b1) to (b8) are the contents of the process 344. The process 344 is executed once for each learning sample in turn, and when the adjustment for all the samples is completed, a new prototype p _{j, m} ^{(e + 1)} (j = 1,..., J; m = 1, ..., M) and weight coefficients w _{j, m} ^{(e + 1)} (j = 1, ..., J; m = 1, ..., M).

(C) Step 346. Here, the order of learning samples in the learning sample set Ω _N is shuffled.

  The above is the content of the process 324. After performing the process 324 on the epoch e, the epoch is advanced by 1 (e = e + 1) and the same process is repeated.

  Thus, when the number of epochs reaches the planned upper limit value E and the processing 324 is finished, a parameter set Λ after learning of the classifier 80 according to the second embodiment is obtained.

[Third embodiment: three-layer feedforward neural network classifier]
As an example using a nonlinear discriminant function, consider a classifier consisting of a feedforward neural network. In this case, as in the first and second embodiments, the classifier can be learned as follows.

《Partial differentiation of discriminant function and geometric margin type classification measure》
A three-layer feedforward neural network classifier 400 is shown in FIG. The neural network classifier 400 includes an input layer 412, an intermediate layer 414, and an output layer 416.

  The input layer 412 includes D + 1 units (d = 0, 1,..., D). The 0th unit receives the value 1 and the other units receive each component of the D-dimensional input pattern x and output it as it is.

The intermediate layer 414 includes M + 1 units (m = 0, 1,..., M). The 0th unit receives nothing and outputs a value of 1. The other m-th unit (m = 1, ..., M ) performs a non-linear function f _m with respect to the weighted sum of the outputs from the input layer 412, and outputs the result.

The output layer 416 includes J units (j = 1,..., J). Each j-th unit (j = 1,..., J) outputs the weighted sum of the outputs from the intermediate layer 414 as the discriminant function g _j of class C _j .

  Note that in the implementation based on MCE learning, the output layer unit is not subjected to nonlinear function processing.

The discriminant function for class C _j (j = 1,..., J) is given by:

Here, w _{m, d} (m = 1,..., M; d = 0, 1,..., D) is a weighting coefficient for coupling from the input layer 412 to the intermediate layer 414, and v _{j, m} (J = 1,..., J; m = 0, 1,... M) are weighting coefficients for coupling from the intermediate layer 414 to the output layer 416. The learning parameter set Λ is a set of all the weighting factors. Non-linear function f _m may be any differentiable function, but here employs a sigmoid function of the following equation.

The partial differentiation for each weighting coefficient of the discriminant function of the kth class C _k (k = 1,..., J) is as follows.

Further, the partial differentiation with respect to the input of the pth dimension of the discriminant function of the kth class (k = 1,..., J) is as follows.

Further, the second derivative for the discriminant function of the kth class (k = 1,..., J) is given by

Here, in the case of the sigmoid nonlinear function of Expression (51), the first-order and second-order derivatives are respectively expressed by the following expressions.

Based on the above, the partial differential of 1 floor and second floor of the classification scale d _y = -g _y + g _i erroneous function margin type and thus given by the following equation. However, the class C _y and the class C _i are the correct class and the best-incorrect class of the vector x, respectively.

Then, the geometric margin type misclassification scale D _y (x; Λ) and its partial differentiation are expressed by the following equation from the equation (12).

Regarding the learning coefficient ε in equation (9), ε for the correction of v _{j, m} and ε for the correction of w _{m, d} may be different from each other.

<< Learning program of the third embodiment >>
Referring to FIG. 8, the program for realizing the algorithm of the third embodiment includes the following steps.

1. Initialization step 440. Weight coefficient {v _{j, m} } _{j = 1} ^J _{m = 0} ^M , {w _{m, d} } _{m = 1} ^M _{d = 0} Initial value of ^D {v _{j, m} ⁽⁰⁾ } _{j = 1} ^J _{m = 0} ^M , {w _{m, d} ⁽⁰⁾ } _{m = 1} ^M _{d = 0} ^D is set. In addition, an upper limit value E of the epoch count e is set.

  2. Step 442. Here, processing 444 including the following substeps is executed for the number of epochs e = 0, 1,. The following are the sub-steps constituting the process 444.

(A) Sub-step 460. Here, if necessary, the optimum value α _y (y = 1,..., J) of the loss smoothness parameter is obtained.

(B) Sub-step 462. Here, the supervised learning sample (x, y) is taken out from the learning sample set Ω _N , and processing 464 including the following sub-steps 480 to 502 is executed for each learning sample. “(E)” on the right shoulder of the variable represents an epoch number. The following (b1) to (b12) are sub-steps constituting the process 464.

    (B1) An input value to the intermediate layer 414 is calculated (m = 1,..., M) (step 480).

(B2) The discriminant function value g _j is calculated (j = 1,..., J) (substep 482).

However, f _m is given by, for example, equation (51).

    (B3) The index i of the best-incorrect class for x is obtained (substep 484).

    (B4) The partial differentiation of the discriminant function is calculated (k = y, i) (substep 486).

However f _'m is given by example equation (61).

    (B5) The partial differentiation related to the input of the discriminant function is calculated (k = y, i) (step 488).

(B6) Second-order partial differentiation with respect to the discriminant function is calculated (k = y, i) (step 490).

However f '' _m is given by example equation (62).

(B7) A function margin type misclassification measure is calculated by d _y = −g _y + g _i (step 492).

    (B8) First-order and second-order partial differentiation of the function margin type misclassification scale is calculated by the following equation (substep 494).

(B9) calculating a norm || ∇ _x d _y || of the gradient vector (substep 496).

(B10) The geometric margin type misclassification measure D _y (x; Λ) is calculated by D _y = d _y / || ∇ _x d _y || (substep 498).

(B11) The partial differentiation of the geometric margin type misclassification measure D _y (x; Λ) is calculated as follows (substep 500).

(B12) The parameter is updated by the following equation (substep 502).

The above substeps (b1) to (b12) are the contents of the process 464. The processing 464 is executed once for each learning sample in order, and when the adjustment for all the samples is completed, a new {v _{j, m} ^{(e + 1)} } _{j = 1} ^J _{m = 0} ^M , { w _{m, d} ^{(e + 1)} } _{m = 1} ^M _{d = 0} ^D is obtained.

(C) Step 466; Here, the order of learning samples in the learning sample set Ω _N is shuffled.

  The above is the content of the process 444. After performing the process 444 for the epoch e, the epoch is advanced by 1 (e = e + 1) and the same process is repeated.

  Thus, when the number of epochs reaches the planned upper limit value E and the processing 444 ends, a parameter set Λ after learning of the classifier 80 according to the third embodiment is obtained.

[Experimental result]
In order to verify the usefulness of the nonlinear discriminant function classifier by the LGM-MCE learning method according to the above embodiment, a conventional MCE learning method that employs a quadratic discriminant function as a nonlinear discriminant function and uses a function margin type misclassification measure A comparison was made between the (FM-MCE method) and the LGM-MCE method using the geometric margin type misclassification scale.

The secondary discriminant function in class C _j is given by equation (16). In this experiment, A _{j, 1} =... = A _{j, M} = A _j and a pair of learning sample sets belonging to class C _j as matrix A _j Fixed to the inverse of the angular covariance matrix. The prototype {p _{j, m} } _{j = 1} ^J _{m = 1} ^M is learned by both the FM-MCE and LGM-MCE methods, and the K-means method was used as initialization thereof. Furthermore, an experiment was conducted in which both FM-MCE and LGM-MCE methods were applied to a Euclidean distance discriminant function type classifier. Euclidean distance discriminant function is a typical example of (piecewise) linear discriminant function is given as to fix the A _j in matrix in equation (16).

  The experiment used the Letter Recognition data set provided by UCI Machine Learning Repository (http://archive.ics.uci.edu/ml/). This data set is a 26-class, 16-dimensional data set composed of 20,000 pieces of data extracted from a font character image of the English alphabet. Since this data has a large number of samples, the Holdout method for dividing the data set was used as an evaluation method. Of the 20,000 sample sets, 1,000 were used as learning sample sets, and the other 19,000 were used as unknown sample sets.

  Table 1 shows the unknown classification sample rate (%) of each discriminant function and each learning method. Figures in parentheses are learning sample classification rates. When the number of prototypes is 1, both the FM-MCE and LGM-MCE methods have a higher classification rate for the secondary discriminant function type classifier than the Euclidean distance type, and in the secondary discriminant function type classifier, the LGM-MCE The method gives a higher classification rate than the FM-MCE method. When the number of prototypes is 3, in the unknown sample classification rate, both the FM-MCE and LGM-MCE methods have a higher classification rate for the secondary discriminant function type classifier than the Euclidean distance type, and further a secondary discriminant function type In the classifier, the LGM-MCE method gives a higher classification rate than the FM-MCE method. From the above, it was confirmed that the LGM-MCE learning method gives higher classification accuracy than the conventional FM-MCE method not only in the linear discriminant function but also in the quadratic discriminant function.

[Realization by computer]
The classifier learning apparatus according to the embodiment described above can be realized by a general-purpose computer and a computer program executed thereon. FIG. 9 shows the external appearance of the computer system 550 used in the above embodiment, and FIG. 10 is a block diagram of the computer system 550. The computer system 550 shown here is merely an example, and other configurations can be used. The core part of this computer program has the control structure shown by the flowcharts of FIGS.

  Referring to FIG. 9, computer system 550 includes a computer 560, a monitor 562, a keyboard 566, a mouse 568, a speaker 558, and a microphone 590 that are all connected to computer 560. Further, the computer 560 includes a DVD-ROM (Digital Versatile Disk Read-Only-Memory) drive 570 and a semiconductor memory port 572.

  Referring to FIG. 10, computer 560 further stores a bus 586 connected to DVD-ROM drive 570 and semiconductor memory drive 572, a CPU 576 all connected to bus 586, and a boot-up program for computer 560. A ROM 578, a RAM 580 that provides a work area used by the CPU 576 and a storage area for programs executed by the CPU 576, a hard disk drive 574 for storing learning data, etc., and a connection to the network 552. A network interface 596 to be provided.

  The software that realizes the system of the above-described embodiment is distributed in the form of object code, script, or source program recorded on a computer-readable recording medium such as DVD-ROM 582 or semiconductor memory 584, and is a DVD-ROM drive. The data is provided to the computer 560 via a reading device such as 570 or the semiconductor memory port 572 and stored in the hard disk drive 574. When the source program is introduced into the computer 560, it is necessary to compile with a predetermined compiler to generate an object code. When the CPU 576 executes the program, the object program (or script) is read from the hard disk drive 574 and stored in the RAM 580. An instruction is fetched from an address designated by a program counter (not shown), and the instruction is executed. The CPU 576 reads data to be processed from the hard disk drive 574 or the RAM 580 and stores the processing result in the hard disk drive 574 or the RAM 580 as well. Although the speaker 558 and the microphone 590 are not used in this embodiment mode, the present invention can also be applied to speech recognition and speaker recognition, in which case they are necessary when preparing learning data about speech. It becomes. The microphone 590 also functions as an input device for inputting the voice to be processed when performing voice recognition on this computer.

  The learning data is collected in advance and includes a large number of sets of input patterns and classes to which the patterns belong. In the system shown in FIG. 3, this is a character feature amount extracted from each character image and a character category corresponding to the character image. The learning data is stored in the hard disk drive 574 (the storage unit 70 and the learning data storage unit 76 shown in FIG. 3). The classification parameter set Λ or the like calculated by the above processing is temporarily stored in the hard disk drive 574 or the like and further copied to the classifier via the network or the USB memory. The classifier classifies the input pattern into any class using the parameter set Λ for class classification.

  Since the general operation of computer system 550 is well known, detailed description will not be repeated here.

  Regarding the software distribution method, the software does not necessarily have to be fixed on a storage medium. For example, the software may be distributed from another computer connected to the network. A part of the software may be stored in the hard disk drive 574, and the remaining part of the software may be taken into the hard disk drive 574 from the network and integrated at the time of execution.

  Typically, modern computers serve the general functionality provided by the computer's operating system (OS), and the general or specific purpose provided by the scripting language's execution system if a scripting language is used. The functions along the line are utilized to achieve the functions in a controlled manner according to the desired purpose. Therefore, even if the program does not include a general function that can be provided from the OS or a third party and thus specifies a combination of execution order of functions provided by other systems, the program is desired as a whole. Obviously, the program is included in the scope of the present invention as long as it has a control structure that achieves the object.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

40 character recognition system 50 learning unit 54 character recognition unit 56 output character category 76 learning data storage unit 78 learning module 80 classifier 96 decoder 400 neural network classifier 412 input layer 414 intermediate layer 416 output layer 550 computer system 560 computer

Claims (6)

A classifier learning device that classifies an input pattern into one of J classes C _j (j is an integer of 1 to J),
Learning sample storage means for storing a learning sample set including N (N is a positive integer) supervised input pattern;
Initializing means for initializing the learning parameter set Λ of the classifier by a predetermined setting method,
Class C input pattern x of training samples in the set which belongs to _y is measure the degree to which misclassified other classes Geometric Margin type misclassification measure value D _y (x; lambda) is defined below,
However ψ is a positive real number, g _y (x; Λ) is said for each of the J Class C _y, determines the degree input pattern x of training samples in the set is whether belonging to the class Is a discriminant function that can be second-order differentiated with respect to x and the learning parameter set Λ, and d _y (x; Λ) is called a function margin type misclassification measure,
For a vector λ = [λ ₁ ... Λ _k ] in which _k variables included in the learning parameter set Λ are arranged, the partial differentiation of the misclassification measure value D _y (x; Λ) by the vector λ is the function d _y ( x; Λ) using the gradient vector ∇ _x d _y , where superscript T represents the transpose of the matrix,
Further, the partial differentiation of the misclassification measure value D _y (x; Λ) so that the value of the predetermined minimization target function L (Λ) with respect to the learning parameter set Λ is minimized with respect to the learning sample set. A classifier learning device including parameter adjustment means for adaptively adjusting the value of each parameter included in the learning parameter set Λ. The discriminant function for class C _j (j = 1,..., J) has M prototypes belonging to class C _j as p _{j, 1} ,..., P _{j, M} and a positive definite matrix corresponding to each prototype. A _{j, 1} ,..., A _{j, M} are given by
However, p _j and A _j are distance distances defined by the following equation between the input pattern x and the prototype belonging to the class C _j.
P _j = p _{j, m (j)} and A _j = A _{j, m (j)} , where _{m (j)} is the prototype index that minimizes
The function margin type misclassification scale d _y (x; Λ) is given by
The classifier learning device according to claim 1, wherein the geometric margin type misclassification measure D _y (x; Λ) and its partial differentiation are given by the following equations.
The positive definite matrix A _{j, 1} ,..., A _{j, M} is a diagonal matrix having a positive diagonal component as follows:
Parameters a _{j, 1} , ..., a _{j, D} are included in the learning parameter set Λ, and parameters a _{y, d} and a _{i, d} (of the geometric margin type misclassification measure D _y (x; Λ) The classifier learning device according to claim 2, wherein the partial differentiation with respect to d = 1,..., D) is represented by the following expression.
The discriminant function for class C _j (j = 1, ..., J) is given by
Where p _{j, 1} , ..., p _{j, M} are M prototypes belonging to class C _j , and w _{j, m} (m = 1, ..., M) is the mth prototype. A weight for the Euclidean distance,
The learning parameter set Λ and the function margin type misclassification measure d _y (x; Λ) are given by the following equations:
However Class C _y and C _i are correct class and best-The net part class x respectively,
The classifier learning apparatus according to claim 1, wherein the geometric margin type misclassification scale and the partial differentiation thereof are represented by the following equations.
The classifier is a three-layer feedforward neural network classifier including an input layer, an intermediate layer, and an output layer,
The input layer includes D + 1 units;
The intermediate layer comprises M + 1 single unit, m-th of the intermediate layer (m = 1, ..., M ) is the unit of performing a non-linear function f _m with respect to the weighted sum of the outputs from the input layer Output,
The output layer includes J units,
Each j-th unit (j = 1,..., J) outputs a weighted sum of outputs from the intermediate layer as a discriminant function g _j of class C _j ,
The discriminant function for class C _j (j = 1, ..., J) is given by
Where w _{m, d} (m = 1, ..., M; d = 0,1, ..., D) is for the coupling from the d-th unit of the input layer to the m-th unit of the intermediate layer. The weighting factor, v _{j, m} (j = 1, ..., J; m = 0,1, ..., M) is for the coupling from the mth unit in the middle layer to the jth unit in the output layer A weighting factor,
The learning parameter set Λ includes the weighting coefficients w _{m, d} (m = 1, ..., M; d = 0,1, ..., D) and v _{j, m} (j = 1, ... , J; m = 0,1, ..., M)
The classifier learning apparatus according to claim 1, wherein the geometric margin type misclassification scale and the partial differentiation thereof are as follows.
  A computer program that causes a computer to function as each unit of the learning device for a classifier according to any one of claims 1 to 5.