JP5527728B2 - Pattern classification learning device - Google Patents

Description

  The present invention relates to a pattern classification learning apparatus that classifies some measurement data into one of predetermined classes, and more particularly to a learning apparatus that can perform learning efficiently using MCE (minimum classification error learning).

[Pattern recognition and learning]
Pattern recognition is an important technique in the interface between humans and machines. The pattern recognition technique is used in various aspects such as speaker identification, utterance content recognition, person identification by face image, and character recognition. In short, pattern recognition can be said to be an operation of classifying observed value patterns obtained by observing some physical phenomenon into a plurality of classes. These tasks are relatively easy for humans, but it is not easy to make them work. A device that performs such work is a pattern recognition device if it is called comprehensively, but in order to make the pattern recognition device perform pattern recognition, the learning data is processed statistically, and the parameters necessary for classification A preliminary work called learning is required.

Consider a classification task that assigns an input pattern (observed value) xεΧ to any one of J classes (classes) C ₁ ,..., C _J. Here, Χ represents the entire input pattern space.

  The statistical approach for classifier learning is based on the concept of minimizing classification risk. The classification risk is an expected value for the entire pattern space of a loss imposed when classifying individual input patterns. The most natural and basic loss is the classification error number loss (0-1 loss). This loss is a loss that imposes a value of 1 for misclassification and a value of 0 for correct classification. This loss is defined by the following equation.

ここで＿ｌ（Ｃ_ｊ｜Ｃ_ｙ）（「＿」は直後の文字が筆記体であることを表す。）は、クラスＣ_ｙに属するパターンをクラスＣ_ｊに分類する際に課せられる分類誤り数損失である。

Here _{_l (C j | C y)} (. "_" As indicating that the character following it is a cursive) the classification error count imposed in classifying patterns belonging to the class C _y in class C _j It is a loss.

The ultimate goal of classifier learning is to realize a classification decision rule c: Χ → {C _j } _{j = 1} ^J that minimizes the classification error number risk R in the following equation.

ただしｐは確率密度関数を表す。Ｒは全入力空間Χに対する分類誤り確率に等しい。

However, p represents a probability density function. R is equal to the classification error probability for the entire input space Χ.

  Hereinafter, minimum classification error (MCE) learning (Non-Patent Document 1) particularly related to the present invention will be described.

[MCE learning]
<Classification decision rule based on discriminant function>
The MCE learning method employs a practical classification decision rule of the following equation using a discriminant function and aims at direct minimization of R.

ここでｇ_ｊ（ｘ、Λ）はクラスＣ_ｊに対する判別関数であり、任意の関数形において、ｘがＣ_ｊに帰属する度合いを測る。Λは分類器の学習パラメータ（調整パラメータ）セットを表す。ｇ_ｊ（ｘ、Λ）はΛに関して微分可能であるとする。
〈誤分類測度〉
式（３）の決定則は全ての判別関数値の計算とそれらの比較演算とからなる。比較を含む演算は、学習段階で要求されるパラメータ最適化のような数値演算には適さない。したがって、式（３）を、数値演算に適した形式に置換えなければならない。ＭＣＥ学習は、Ｌｐノルム形式の平滑な誤分類測度を学習パターンに適用することでそのような置換を実現する。クラスＣ_ｙに属するパターンｘに対する誤分類測度は次式で定義される。

Here, g _j (x, Λ) is a discriminant function for class C _j and measures the degree to which x belongs to C _j in an arbitrary function form. Λ represents a learning parameter (adjustment parameter) set of the classifier. Let g _j (x, Λ) be differentiable with respect to Λ.
<Misclassification measure>
The decision rule of equation (3) consists of calculating all discriminant function values and comparing them. Computations involving comparisons are not suitable for numerical computations such as parameter optimization required in the learning stage. Therefore, Equation (3) must be replaced with a format suitable for numerical operations. MCE learning implements such replacement by applying a smooth misclassification measure in Lp-norm format to the learning pattern. Classification measure erroneous with respect to the pattern x belonging to the class C _y is defined by the following equation.

ここでψは正の実数である。またｌｏｇを自然対数とする。ψ→∞とすることにより

Here, ψ is a positive real number. Log is a natural logarithm. By setting ψ → ∞

となることから確認できるように、十分大きいψにおいて、ｄ_ｙの正値は誤分類を、ｄ_ｙの負値は正分類を表す。ｄ_ｙの絶対値は分類決定の確信度を表す。加えて、ｄ_ｙはΛに関して微分可能であり、最も基本的な勾配探索型の最適化手法を学習に適用することが可能となる。

As can be seen from the fact that a, in large enough [psi, a positive value is misclassification of d _y, negative values of d _y represents a positive classification. the absolute value of d _y represents the confidence of the classification decision. In addition, _dy is differentiable with respect to Λ, and the most basic gradient search type optimization method can be applied to learning.

  MCE learning uses the misclassification measure to reformulate the number of classification error losses in equation (1) and the risk in equation (2), enabling efficient optimization techniques to be applied. Based on the definition of the misclassification measure, the classification error count loss can be rewritten as:

この関数のグラフ２０を図１に示す。ここで関数１（Ｐ）は、命題Ｐが真ならば１を、偽ならば０を、それぞれ返す指示関数である。リスクは次式のようなΛの関数として再定義される。

A graph 20 of this function is shown in FIG. Here, the function 1 (P) is an instruction function that returns 1 if the proposition P is true and 0 if it is false. Risk is redefined as a function of Λ as

〈平滑化分類誤り数損失〉
分類誤り数損失ｌ（ｄ_ｙ（ｘ，Λ）＞０）はΛに関して微分不可能である。この計算上の問題を克服するために、ＭＣＥ学習は微分可能な平滑化分類誤り数損失を定義して、これに式（１）を置換える。クラスＣ_ｙに属するパターンｘに対する平滑化分類誤り数損失として、ＭＣＥ学習では一般に、次式のロジスティックシグモイド関数が用いられる。

<Smooth classification error number loss>
The classification error number loss l (d _y (x, Λ)> 0) is not differentiable with respect to Λ. To overcome this computational problem, MCE learning defines a differentiating smoothed classification error number loss and replaces it with equation (1). As the smoothing classification error count loss for the class C _y belonging pattern x, the MCE learning Generally, the logistic sigmoid function of the following equation is used.

式（７）により表わされる関数のグラフ３０を図２に示す。ここで損失平滑度α_ｙは正の実数である。この＿ｌ_ｙ（ｄ_ｙ（ｘ，Λ））は誤分類測度ｄ_ｙ（ｘ，Λ）の単調増加関数であり、損失平滑度α_ｙが大きくなるにつれて傾きが大きく（急に）なり、損失平滑度α_ｙ→∞の極限で１（ｄ_ｙ（ｘ，Λ）＞０）に一致する。すなわち、平滑化分類誤り数損失は、誤分類カウントと直接的に結びついているだけでなく、Λに関して微分可能である。ＭＣＥ学習の最終的な学習目的は、式（６）における１（ｄ_ｙ（ｘ，Λ）＞０）を＿ｌ_ｙ（ｄ_ｙ（ｘ，Λ））に置換えた次式の期待損失を最小にするΛを求めることである。

FIG. 2 shows a graph 30 of the function represented by the equation (7). Here, the loss smoothness α _y is a positive real number. This _l _y (d _y (x, Λ)) is a monotonically increasing function of the misclassification measure d _y (x, Λ), and the slope increases (suddenly) as the loss smoothness α _y increases, and the loss smoothing It coincides with 1 (d _y (x, Λ)> 0) in the limit α _y → ∞. That is, the smoothed classification error number loss is not only directly related to the misclassification count, but is also differentiable with respect to Λ. The final learning purpose of MCE learning is to minimize the expected loss of the following equation by replacing 1 (d _y (x, Λ)> 0) in equation (6) with _l _y (d _y (x, Λ)) Is to find Λ.

損失平滑度α_ｙ→∞において、式（８）は式（６）と一致する。

In the loss smoothness α _y → ∞, the equation (8) matches the equation (6).

Equation (8) includes integration over an infinite number of input patterns. However, in reality, it is only possible to estimate Λ using a learning sample set Ω _N = {(x _n , y _n )} _{n = 1} ^N consisting of a finite number (N) of samples. Here, x _n ∈Χ is the nth learning pattern, that is, a learning sample, and y _n (= 1,..., J) is an index of a class to which the sample x _n belongs. Therefore, evaluation criteria of realistic MCE learning, the empirical average loss of the following equation that approximates the equation (8) in the finite learning sample set Ω _N.

ＭＣＥ学習では、この経験的平均損失＾Ｌ（Λ）（記号「＾」は、式中では直後の文字の直上に記載されている。）を最小にするようなパラメータΛを求める。有限個の学習標本のみから構成される上式の＾Ｌ（Λ）は、当然ながら、学習標本集合に含まれない全ての未知パターンをも含む分類誤り数リスク（分類誤り確率）Ｒ（Λ）の近似にすぎず、＾Ｌ（Λ）を最小にするΛがＲ（Λ）を最小にする保証はない。しかし、適度な有限値の損失平滑度α_ｙを設定することにより、評価基準＾Ｌ（Λ）が平滑な関数となり、学習標本集合に含まれない未知パターンに対する学習耐性を向上させる。すなわちこの平滑化により、与えられた学習標本のみならずその近傍に対しても損失が敏感となり、学習標本数を増やす効果が得られる。したがって適切な損失平滑度α_ｙを設定することが、汎化能力向上に対して極めて重要である。

In MCE learning, a parameter Λ that minimizes this empirical average loss ^ L (Λ) (the symbol “^” is described immediately above the character immediately after in the equation) is obtained. Of course, ^ L (Λ) in the above equation consisting only of a finite number of learning samples is, of course, the risk of classification error (classification error probability) R (Λ) including all unknown patterns not included in the learning sample set. Λ that minimizes L (Λ) is not guaranteed to minimize R (Λ). However, by setting an appropriate finite-value loss smoothness α _y , the evaluation criterion ^ L (Λ) becomes a smooth function and improves learning tolerance for unknown patterns not included in the learning sample set. That is, this smoothing makes the loss sensitive not only to a given learning sample but also to the vicinity thereof, and the effect of increasing the number of learning samples can be obtained. Therefore, setting an appropriate loss smoothness α _y is extremely important for improving the generalization ability.

B. -H. Juan and S. Katagiri, “Distinguishing Learning for Minimum Error Classification” IEEE Signal Processing Transactions, Vol. 40, No. 12, pp. 3043-3054, December 1992 (B.-H. Juang and S. Katagiri, “Discriminative learning for minimum error classification,” IEEE Trans. Signal Processing, vol.40, no.12, pp.3043-3054, Dec. 1992.) E. McDermat and S. Katagiri, “Deriving Minimum Classification Errors from Theoretical Classification Risk Using Parzen Estimation”, Computer Speech and Language, Vol. 18, pp. 107-122, April 2004 (E. McDermott and S. Katagiri, “A derivation of minimum classification error from the theoretical classification risk using Parzen estimation,” Computer Speech and Language, vol.18, pp.107-122, April 2004.) R. P. W. Duin, “Selecting a Smoothing Function for Parzen Estimation of the Probability Density Function”, IEEE Transaction of Computer, Vol. C-25, pp. 1175-1179, November 1976 (RPW Duin, “On the choice of smoothing parameters for Parzen estimators of probability density functions,” IEEE Trans. Comput., Vol. C-25, pp. 1175-1179, Nov. 1976. ) C. M.M. Bishop (translated by Hiroshi Motoda, Takio Kurita, Tomoyuki Higuchi, Yuji Matsumoto, Noboru Murata), Pattern Recognition and Machine Learning, Springer Japan, Tokyo, 2007.

Conventionally, a specific setting guideline for determining the above-described loss smoothness α _y has not been given. Therefore, there is a problem that the loss smoothness α _y must be set to ad hoc. Although the loss smoothness α _y obtained in this way is effective for learning data, there is a problem that it is unlikely that high recognition is obtained for unknown data. That is, the conventional learning device has a problem that it is difficult to increase the generalization ability of the obtained classifier.

  Accordingly, an object of the present invention is to provide a learning device capable of increasing the generalization ability of the obtained classifier in the learning device for a pattern classifier by MCE learning.

  Another object of the present invention is to provide a learning device that can calculate loss smoothness that can increase generalization ability with a specific setting guideline in a learning device for a pattern classifier by MCE learning. .

The learning device according to the first aspect of the present invention is a learning device for a classifier that classifies an input pattern into any of J classes C _j (j is an integer from 1 to J), each of which is an input pattern. And a learning sample storage means for storing N learning samples (N is a positive integer) including the class to which the class belongs and initialization for initializing the learning parameter Λ of the classifier by a predetermined setting method Means. A misclassification measure value d _y (x, Λ) in which the input pattern x of the learning sample belonging to the class C _y is misclassified into another class is defined as follows.

ただしψは正の実数であり、ｇ_ｙ（ｘ、Λ）はＪ個のクラスＣ_ｙの各々に対して、学習標本の入力パターンｘが当該クラスに属するか否かの度合いを判別するための、任意の形の判別関数である。この学習装置はさらに、Ｊ個のクラスＣ_ｙの各々について、当該クラスＣ_ｙに属する学習標本の各々に関する前記誤分類測度値を求め、当該クラスに属する標本を生成した真の確率分布を、誤分類測度空間における各誤分類測度値を中心とする、Ｐａｒｚｅｎ窓幅ｈ_ｙのＰａｒｚｅｎ分布であって、かつ誤分類測度値の関数として、交差確認型最尤推定により推定するためのＰａｒｚｅｎ分布推定手段を含む。
Ｐａｒｚｅｎ分布推定手段は、交差確認型最尤推定において、Ｐａｒｚｅｎ窓幅ｈ_ｙの関数としてＰａｒｚｅｎ分布の尤度を評価する。学習装置はさらに、Ｊ個のクラスＣ_ｙの各々について、Ｐａｒｚｅｎ分布推定手段による交差確認型最尤推定において最尤となるＰａｒｚｅｎ分布を与えるＰａｒｚｅｎ窓幅ｈ_ｙに対し、以下の関数

However the ψ is a positive real number, g y _(x, Λ) for each of J-number of class C _y, the input pattern x of training samples are used to determine the degree of whether belonging to the class A discriminant function of any form. The learning apparatus further, for each of the J Class C _y, determined the misclassification measure value for each of the training samples belonging to the class C _y, the true probability distribution that generated the sample belonging to the class, erroneous Parzen distribution estimator for parzen distribution with Parzen window width _hy , centered on each misclassified measure value in the classified measure space, and for estimating by cross-confirmation type maximum likelihood estimation as a function of the misclassified measure value including.
The Parzen distribution estimation means evaluates the likelihood of the Parzen distribution as a function of the Parzen window width _{hy in} the cross-confirmation type maximum likelihood estimation. Learning apparatus further comprises for each of the J Class _{C y,} to Parzen window width _{h y} giving the Parzen distribution as a maximum likelihood in cross-validation type maximum likelihood estimation by Parzen distribution estimating means, the following functions

によって、分類器のクラスＣ_ｙに対する損失平滑度の最適値α_ｙを算出するための最適損失平滑度算出手段と、学習標本集合から学習標本を１つずつ取出し、分類誤りリスクを最小化するよう、学習パラメータΛを逐次的に調整するための学習パラメータ調整手段と、Ｐａｒｚｅｎ分布推定手段、最適損失平滑度算出手段、及び学習パラメータ調整手段を、予め定める終了条件が成立するまで繰返し動作させ、終了条件が成立したときの学習パラメータΛを出力するための繰返し制御手段とを含む。

Accordingly, the optimum loss smoothness calculating means for calculating the optimum value alpha _y loss smoothness for the class C _y classifier, taken out from the learning sample set, one of the training samples, to minimize the classification error risk The learning parameter adjusting means for sequentially adjusting the learning parameter Λ, the Parzen distribution estimating means, the optimum loss smoothness calculating means, and the learning parameter adjusting means are repeatedly operated until a predetermined end condition is satisfied, Repetitive control means for outputting a learning parameter Λ when the condition is satisfied.

  Preferably, the learning device further includes shuffle means for shuffling the arrangement order of learning samples according to random numbers before repetition by the repetition control means and before adjustment of the learning parameter Λ by the learning parameter adjustment means.

  More preferably, the predetermined termination condition is a condition that the operations by the Parzen distribution estimation unit, the optimum loss smoothness calculation unit, and the learning parameter adjustment unit have been completed a predetermined number of times.

  More preferably, the iterative control means periodically omits the operations of the Parzen distribution estimation means and the optimum loss smoothness calculation means among the Parzen distribution estimation means, the optimum loss smoothness calculation means, and the learning parameter adjustment means.

A Parzen window Gaussian functions constitute the Parzen distribution, Parzen distribution estimating means removes one specimen from the specimen belonging to the class _{C y,} Parzen estimate distribution for constituting the Parzen estimate distribution in the rest of the sample a configuration unit, the expression for defining the Parzen estimate distribution arrangement means, mixing weighting factor 1 / (N'-1) ( N' the number of samples belonging to the class _{C y)} N'-1 single Gaussian mixture is It may be considered as a distribution probability density function, and may include a window width calculation means for calculating a Parzen distribution window width _hy that maximizes the mixed Gaussian distribution probability density function by an EM algorithm.

The computer program according to the second aspect of the present invention classifies an input pattern into any one of J classes C _j (j is an integer from 1 to J), and each of the computers belongs to the input pattern. Functions as learning sample storage means for storing N learning samples (N is a positive integer) including classes, and initialization means for initializing the learning parameters Λ of the classifier by a predetermined setting method It is a computer program to make it. A misclassification measure value d _y (x, Λ) in which the input pattern x of the learning sample belonging to the class C _y is misclassified into another class is defined as follows.

ただしψは正の実数であり、ｇ_ｙ（ｘ、Λ）はＪ個のクラスＣ_ｙの各々に対して、学習標本の入力パターンｘが当該クラスに属するか否かの度合いを判別するための、任意の形の判別関数である。このコンピュータプログラムは、コンピュータをさらに、Ｊ個のクラスＣ_ｙの各々について、当該クラスＣ_ｙに属する学習標本の各々に関する前記誤分類測度値を求め、当該クラスに属する標本を生成した真の確率分布を、誤分類測度空間における各誤分類測度値を中心とする、Ｐａｒｚｅｎ窓幅ｈ_ｙのＰａｒｚｅｎ分布であって、かつ誤分類測度値の関数として、交差確認型最尤推定により推定するためのＰａｒｚｅｎ分布推定手段として機能させる。
当該Ｐａｒｚｅｎ分布推定手段は、交差確認型最尤推定において、Ｐａｒｚｅｎ窓幅ｈ_ｙの関数としてＰａｒｚｅｎ分布の尤度を評価する。このコンピュータプログラムは、コンピュータをさらに、Ｊ個のクラスＣ_ｙの各々について、Ｐａｒｚｅｎ分布推定手段による交差確認型最尤推定において最尤となるＰａｒｚｅｎ分布を与えるＰａｒｚｅｎ窓幅ｈ_ｙに対し、以下の関数

However the ψ is a positive real number, g y _(x, Λ) for each of J-number of class C _y, the input pattern x of training samples are used to determine the degree of whether belonging to the class A discriminant function of any form. The computer program, the computer further, for each of the J Class C _y, determined the misclassification measure value for each of the training samples belonging to the class C _y, the true probability distribution that generated the sample belonging to the class Is a Parzen distribution with a Parzen window width _hy , centered on each misclassified measure value in the misclassified measure space, and Parzen for estimating by cross-confirmed maximum likelihood estimation as a function of the misclassified measure value It functions as a distribution estimation means.
The Parzen distribution estimation means evaluates the likelihood of the Parzen distribution as a function of the Parzen window width _{hy in} the cross-confirmation type maximum likelihood estimation. The computer program further computer, for each of the J Class _{C y,} to Parzen window width _{h y} giving the Parzen distribution as a maximum likelihood in cross-validation type maximum likelihood estimation by Parzen distribution estimating means, the following functions

によって、分類器のクラスＣ_ｙに対する損失平滑度の最適値α_ｙを算出するための最適損失平滑度算出手段と、学習標本集合から学習標本を１つずつ取出し、分類誤りリスクを最小化するよう、学習パラメータΛを逐次的に調整するための学習パラメータ調整手段と、Ｐａｒｚｅｎ分布推定手段、最適損失平滑度算出手段、及び学習パラメータ調整手段とを、予め定める終了条件が成立するまで繰返し動作させ、終了条件が成立したときの学習パラメータΛを出力するための繰返し制御手段として機能させる。

Accordingly, the optimum loss smoothness calculating means for calculating the optimum value alpha _y loss smoothness for the class C _y classifier, taken out from the learning sample set, one of the training samples, to minimize the classification error risk The learning parameter adjusting means for sequentially adjusting the learning parameter Λ, the Parzen distribution estimating means, the optimum loss smoothness calculating means, and the learning parameter adjusting means are repeatedly operated until a predetermined termination condition is satisfied, It functions as an iterative control means for outputting the learning parameter Λ when the end condition is satisfied.

It is a graph of classification error number loss function. It is a graph of the smoothing classification error number loss function by a logistic sigmoid function. It is a schematic graph for demonstrating Parzen estimation of the probability density function on a misclassification measure. ^{_{d y (x y k, Λ}} ) is a graph showing the Parzen window width _{h y} around the. ^{_{d y (x y k, Λ}} ) integral value for the positive region of a Gaussian Parzen window centered on ^{_{the, d y (x y k,}} Λ) of the logistic sigmoid function smoothing classification error count loss function values in It is a figure which shows typically that it is approximately equal to. It is a figure for demonstrating the relationship between the window width of Parzen window, and the smoothness of the smoothing classification error number loss function. It is a figure which shows Parzen estimated distribution comprised by the remaining samples which remove | excluded one sample _xn . It is a flowchart which shows the control structure of the program which implement | achieves MCE learning with automatic control of loss smoothness obtained by applying to Parzen estimation on misclassification measure space in 1st Embodiment. 10 is a flowchart showing a control structure of a program for obtaining an optimum Parzen window width by an EM algorithm, which is executed in step 156 of FIG. 8. It is a figure which shows the hardware external appearance of the general purpose computer system which implement | achieves 1 embodiment of this invention. It is a block diagram of the internal structure of the computer system shown in FIG. In the 2nd Embodiment of this invention, it is a flowchart which shows the control structure of the program which implement | achieves MCE learning accompanied by the automatic control of loss smoothness.

Embodiments of the present invention will be described below. In the description and drawings, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. In addition, regarding the minimization of ^ L (Λ), not only a batch method such as a steepest descent method, but also an adaptive adjustment of Λ every time one sample (x _n , y _n ) is extracted from Ω _N Learning methods are also widely used. The adjustment mechanism of Λ in the method is given by the following equation.

以下の実施の形態では、この適応的学習方法を採用することとする。
多くのＭＣＥ学習の実装において、式（７）に関し、全てのクラスＣ_ｙ（ｙ＝１，…，Ｊ）に対して共通の損失平滑度α_ｙが設定される。しかし本実施の形態では、後述するようにクラス毎に損失平滑度α_ｙの自動制御が行われるので、損失平滑度α_ｙは各クラス個別に設定されるものとする。

In the following embodiment, this adaptive learning method is adopted.
In many MCE learning implementations, with respect to equation (7), a common loss smoothness α _y is set for all classes C _y (y = 1,..., J). However, in this embodiment, the loss smoothness α _y is automatically controlled for each class as will be described later, and therefore the loss smoothness α _y is set for each class individually.

<Reformulation of MCE learning based on Parzen estimation>
The MCE learning method was originally formulated based on the concept of counting the number of classification errors using 0-1 loss as described in the previous section. Apart from that, there is also a formulation for directly estimating the classification error probability (Non-Patent Document 2). The estimation of the classification error probability is realized by applying Parzen estimation, which is one of the nonparametric probability density estimation methods, on the misclassification measure space. With this formulation, MCE learning does not target only a given set of learning samples, but performs optimization that takes into account unknown patterns that are predicted to exist in the vicinity of them. Can be caught. In the following, the formulation procedure will be described.

  First, the integration with the entire pattern space as the integration range in the equation (6) representing the classification error number risk is replaced with the integration with the entire subset as the integration range.

ここでΧ_ｙ（Λ）は以下の式で表される。

Here, _{ｙ y} (Λ) is expressed by the following equation.

Χ_ｙ（Λ）は判別関数｛ｇ_ｊ（ｘ，Λ）｝_ｊ＝１ ^Ｊによる分類結果が誤分類となるようなクラスＣ_ｙに属するパターンの集合である。

Chi _y (lambda) is the set of discriminant function _{{g j (x, Λ)} } j = 1 classification result by ^J belongs to the class _{C y} such that misclassification pattern.

Next, the integration in the input pattern space with Χ _y (Λ) as the integration range in Equation (11) is replaced with the integration in the misclassification measure space with the region where the value of the misclassification measure is positive as the integration range. The Here, since x and {g _j (x, Λ)} _{j = 1} ^J can be regarded as continuous random variables, the misclassification measure d _y (x, Λ) is also a continuous random variable. Then the integral in equation (11), in each class C _y, can be replaced as follows.

ここでｐ_Λ（ｔ｜Ｃ_ｙ）は、連続確率変数と考えた誤分類測度ｄ_ｙ（ｘ，Λ）の出現確率を表現する確率密度関数である。ｄ_ｙ（ｘ，Λ）がΛに依存するため、この確率密度関数もΛに依存する。このような置き換えを行った積分を用いた分類誤り数リスクは、以下のようになる。

Here, p _Λ (t | C _y ) is a probability density function expressing the appearance probability of the misclassification measure d _y (x, Λ) considered as a continuous random variable. Since d _y (x, Λ) depends on Λ, this probability density function also depends on Λ. The classification error number risk using the integration with such replacement is as follows.

注目すべき点は、積分が普段高次元となる元々のパターン空間ではなく１次元の誤分類測度の空間上で行われることと、１次元空間上の条件付確率密度ｐ_Λ（ｔ｜Ｃ_ｙ）のモデル化がパラメータ学習の新たなアプローチを示唆していることである。

It should be noted that the integration is performed in the space of the one-dimensional misclassification measure instead of the original pattern space, which is usually high-dimensional, and the conditional probability density p _Λ (t | C _{y in the} one-dimensional space. ) Suggests a new approach to parameter learning.

Based on this knowledge, the new MCE learning formulation is the following for approximating p _Λ (t | C _y ) with a finite number of learning samples {x ^y _k } _{k = 1} ^Ny in each class C _y . Introduce the Parzen estimated distribution of the equation (FIG. 3).

ここでｘ_ｋ ^ｙはクラスＣ_ｙに属する学習標本のうち、ｋ番目の学習標本、Ｎ_ｙはクラスＣ_ｙに属する学習標本の総数である。

Here x _k ^y among the training samples belonging to the class C _y, k-th training samples, N _y is the total number of training samples belonging to the class C _y.

は誤分類測度空間上に変換されたデータ点ｄ_ｙ（ｘ^ｋ _ｙ，Λ）を中心とした幅ｈ_ｙのＰａｒｚｅｎ窓５０である（図４）。

Is a Parzen window 50 is converted into the misclassification measure spatial data points ^{_{d y (x k y, Λ}} ) width _{h y} centered (Fig. 4).

In FIG. 3, the horizontal axis represents the misclassification measure, and the vertical axis represents the appearance probability distribution (probability density). x _k ^y is _{(N y} pieces in total) k-th learning pattern belonging to the y-th class ^{_{C y, dy (x k y}} , Λ) is the misclassification measure value. As described above, Λ represents a learning parameter set of the recognizer. Parzen window group 40 shown in the figure, a number of Parzen window, each centered ^{_{d y (x k y, Λ}} ) to _{(k = 1, ..., N} y) containing. By taking the arithmetic mean of the Parzen window for all of the learning patterns belonging to y-th class C _y, approximation of misclassification measure distribution of class C _y (approximate distribution 42) is obtained. Furthermore, the integral of this approximation distribution 42 in the positive region (hatched portion in the drawing) is an approximation of the probability that correct class misclassified pattern is C _y to a class other than C _y.

Referring to FIG. 4, the Parzen window 50 is generally a unimodal function whose value is positive with respect to the data point. P _lambda expression _{(13) (t | C y} ) the Parzen estimate distribution ^ p _lambda | By approximated by (t _{C y),} further approximates P a _{(C y)} in _N y / N, classification error count The estimated value based on the finite learning sample set Ω _N of the risk is expressed by the following equation.

ここで重要なことに、次式（１６）のように、損失関数＿ｌ_ｙ（ｄ_ｙ（ｘ，Λ））を各々のＰａｒｚｅｎ窓の正領域の積分として新たに定義することにより、式（１５）のＲ_Ｎ（Λ）が式（９）の経験的平均損失＾Ｌ（Λ）、すなわちＭＣＥ学習の評価基準と一致する。

Significantly, the loss function _l _y (d _y (x, Λ)) is newly defined as the integral of the positive region of each Parzen window, as shown in the following equation (16). ) of _{R N} (lambda) is the empirical average loss of the formula (9) ^ L (Λ) , that is consistent with the evaluation criteria MCE learning.

興味深いことに、重要な損失関数が馴染み深い窓関数から導かれる。例えば、ガウス関数型の窓関数

Interestingly, important loss functions are derived from familiar window functions. For example, a Gaussian window function

を採用した場合、式（１６）の＿ｌ_ｙ（ｄ_ｙ（ｘ，Λ））は式（７）のロジスティックシグモイド関数に似た損失関数となる。実際、

Is used, _l _y (d _y (x, Λ)) in equation (16) is a loss function similar to the logistic sigmoid function in equation (7). In fact,

と設定した場合、式（７）と式（１６）とは極めて近いものとなる（図５）。ただし両者が厳密には異なることには注意する必要がある。

Is set to be very close to (7) and (16) (FIG. 5). However, it should be noted that the two are strictly different.

Figure 5 and Figure 6 and as is apparent from a comparison of found the width _{h y} wide Parzen window 60, a smooth loss function narrower Parzen window 70 width _{h y.} That is, the window width h of the Parzen window expresses the smoothness of the loss function. The integration of the individual Parzen window 60 or Parzen window 70 in the positive region (the hatched portion on the left side of FIGS. 5 and 6) corresponds to the smoothed classification error loss used in MCE (curve 62 and FIG. 5). Curve 72 in FIG. These arithmetic averages are approximate values of misclassification probabilities, and at the same time, are evaluation criteria for minimization targets in MCE learning.

The Parzen estimation framework described above provides a new loss smoothing control mechanism for the original MCE formulation. The more closely the Parzen estimated distribution of equation (14) approximates the true probability distribution p _Λ (t | C _y ), the more empirical average loss of equation (15) or equation (9) becomes, Or it is a better approximation of the risk of classification error in equation (6). In other words, if the window width _hy is estimated so that the Parzen estimated distribution of Equation (14) is a good approximation of the true distribution, the evaluation criterion for MCE learning composed of a finite number of learning samples is unknown samples. It approaches the classification error probability for the entire pattern space including. At this time, the minimum state of the evaluation criteria for MCE learning approaches the minimum state of the classification error probability for the entire pattern space, and the generalization ability of MCE learning is improved.

<Parzen estimation based on cross-confirmed maximum likelihood estimation>
In this section and the next section, a discussion of general Parzen estimation that is not limited to loss smoothness control in MCE learning will be developed. Therefore, given (N ′) samples are _represented by x _k (k = 1, 2,..., N '), a Parzen window width is _h, treat the common problem of Parzen estimate of the true probability distribution that generated ^{_{{x k} k = 1 N'}} . For loss smoothness control in MCE learning, the following applies as N ′ = N _y and x _k = d _y (x _y ^k , Λ) (k = 1, 2,..., N _y ). This is performed for each class C _y (y = 1, 2,..., J).

  In order to determine the window width h such that the Parzen estimated distribution is a good approximation to the true distribution, consider estimating h by the maximum likelihood estimation method. However, unlike the parametric estimation method that uses a characteristic value such as mean or variance, Parzen estimation, which is non-parametric estimation, constructs an estimated distribution using all samples, so the same sample is used as the estimated distribution. The likelihood function cannot be maximized by substituting (the window width becomes zero). Therefore, in Non-Patent Document 3, cross-confirmation type maximum likelihood estimation is formulated by forming a Parzen estimated distribution with a sample set from which one sample is removed, and substituting the removed sample into the estimated distribution. . This outline will be described below.

Referring to FIG. 7, the n-th sample x _n is removed from the given N ′ samples x _k (k = 1, 2,..., N ′), and the Parzen estimated distribution 82 is obtained with the remaining samples. Configure. A Parzen window 80 represented by a broken line in FIG. 7 corresponds to the removed sample _xn .

取除いた標本ｘ_ｎを上式に代入し、全ての標本に対してこれの積をとった尤度関数を定める。

The removed sample _xn is substituted into the above equation, and a likelihood function is determined by taking the product of all the samples.

そして上式のＬ（ｈ）を最大化するｈを求める。以上が交差確認型最尤推定の概要である。
〈ＥＭアルゴリズムに基づくＰａｒｚｅｎ窓幅決定法＞
以下、本実施の形態の構成について説明する。式（２０）のＰａｒｚｅｎ窓幅ｈによる最大化は、一般的には多くの計算量を要する。本実施の形態に係る損失平滑度自動制御型ＭＣＥ学習（次節において説明する。）はＰａｒｚｅｎ窓幅ｈの最適化を多数回行なうため、この最適化を少ない計算量で実行するのが望ましい。そこで本実施の形態では、Ｐａｒｚｅｎ窓φが式（１７）のガウス型関数である場合において、ＥＭアルゴリズムによりＰａｒｚｅｎ窓幅ｈの最大化を効率的に行なう。以下、図９に示すフローチャートを参照しながら、このＥＭアルゴリズムについて説明する。

Then, h that maximizes L (h) in the above equation is obtained. The above is the outline of the intersection confirmation type maximum likelihood estimation.
<Parzen window width determination method based on EM algorithm>
Hereinafter, the configuration of the present embodiment will be described. The maximization by the Parzen window width h in the equation (20) generally requires a large amount of calculation. Since loss smoothness automatic control type MCE learning (described in the next section) according to the present embodiment optimizes the Parzen window width h many times, it is desirable to perform this optimization with a small amount of calculation. Therefore, in the present embodiment, when the Parzen window φ is a Gaussian function of Expression (17), the Parzen window width h is efficiently maximized by the EM algorithm. The EM algorithm will be described below with reference to the flowchart shown in FIG.

  Equation (19) is transformed into the following equation.

φが式（１７）のガウス型関数である場合、式（２２）中の次の項

If φ is a Gaussian function of equation (17), the next term in equation (22)

は平均ｙ_ｍ ^（ｎ）および分散ｈ^２のガウス分布確率密度関数と見なすこともできる。したがって、ｐ_−ｎ（ｔ｜ｈ）は、混合重み係数を均一の１／（Ｎ´−１）に固定した混合数Ｎ´−１の混合ガウス分布確率密度関数と形式的に見なされる。このとき、ｍを潜在変数と考えることもできるので、潜在変数を含む確率モデルの最尤推定に対する効率的な繰返し計算型アルゴリズムであるＥＭアルゴリズム（例えば、非特許文献４の第９章）を、式（２０）のＰａｒｚｅｎ窓幅ｈによる最大化に適用できる。

Can also be viewed as a Gaussian probability density function with mean y _m ⁽ⁿ⁾ and variance h ² . Therefore, p _−n (t | h) is formally regarded as a mixed Gaussian distribution probability density function of the number of mixing N′−1 in which the mixing weight coefficient is fixed to uniform 1 / (N′−1). At this time, since m can also be considered as a latent variable, an EM algorithm (for example, Chapter 9 of Non-Patent Document 4), which is an efficient iterative calculation algorithm for maximum likelihood estimation of a probability model including a latent variable, This can be applied to the maximization by the Parzen window width h in the equation (20).

  Assume that an estimated value ^ h of the Parzen window width h is obtained in the step immediately before the iterative calculation. The burden factor used in the EM algorithm is as follows.

ＥＭアルゴリズムで定義される補助関数（Ｑ関数）は次式となる。

The auxiliary function (Q function) defined by the EM algorithm is as follows.

ただし“Ｃｏｎｓｔ．”はｈに無関係の項を表す。上式をｓに関して微分し、更にこれが０となるｓを求めることにより、上式の補助関数を最小にするＰａｒｚｅｎ窓幅ｈの平方（ｈ^２）が以下で与えられる。

However, “Const.” Represents a term unrelated to h. By differentiating the above equation with respect to s and finding s where it becomes 0, the square (h ² ) of the Parzen window width h that minimizes the auxiliary function of the above equation is given below.

更にここで、ｑ´_ｍ，ｎ（ｍ＝１，…，Ｎ´；ｍ≠ｎ）を以下で定める。

Further, q ′ _{m, n} (m = 1,... _{, N} ′; m ≠ n) is defined as follows.

このときｑ´_ｍ，ｎおよび補助関数を最小にするｈ^２は以下となる。

At this time, q ′ _{m, n} and h ² that minimizes the auxiliary function are as follows.

アルゴリズム表記の煩雑さを避けるため、再びｑ´_ｍ，ｎをｑ_ｍ，ｎと置きなおす。結局、Ｐａｒｚｅｎ窓幅の最尤推定に対するＥＭアルゴリズムは以下でまとめられる。

In order to avoid the complexity of the algorithm notation, q ′ _{m, n} is replaced with q _{m, n} again. Finally, the EM algorithm for maximum likelihood estimation of the Parzen window width is summarized below.

(1) An initial value h ⁽⁰⁾ > 0 is given. Set _l = 0. (Step 220 in FIG. 9)
(2) Calculate q _{m, n} of the following equation (n = 1,..., N ′; m = 1,..., N ′, m ≠ n). (Steps 222, 224 and 226 in FIG. 9)

（３）以下のようにパラメータの再推定をする。（ステップ２２８）

(3) Re-estimate parameters as follows. (Step 228)

（４）ｈが収束条件を満たしていれば（ステップ２３０でＹＥＳ）終了、さもなくば（ステップ２３０でＮＯ）、＿ｌ←＿ｌ＋１として（ステップ２３２）、（２）へ。収束条件としては、本実施の形態では、次式の対数尤度の収束を採用する。

(4) If h satisfies the convergence condition (YES in step 230), the process ends. If not (NO in step 230), _l ← _l + 1 is set (step 232) to (2). As a convergence condition, the present embodiment employs logarithmic likelihood convergence of the following equation.

または、予め繰返し回数の上限Ｉを設定しておき、繰返し回数＿ｌがＩに達したかまたは上式の対数尤度が収束したと判断されたとき、繰返しを終了させても良い。初期値ｈ^（０）は、合理的な設定法であればどのような手法で設定しても良い。例えば以下のアルゴリズムを適用できる。
（１）各標本ｘ_ｎ（ｎ＝１，…，Ｎ´）に対して、自身以外で最も近いデータを割当てる。

Alternatively, the upper limit I of the number of repetitions may be set in advance, and the repetition may be terminated when it is determined that the number of repetitions_l has reached I or the log likelihood of the above equation has converged. The initial value h ⁽⁰⁾ may be set by any method as long as it is a reasonable setting method. For example, the following algorithm can be applied.
(1) The nearest data other than itself is assigned to each sample x _n (n = 1,..., N ′).

（２）以下のようにパラメータの初期値を与える。

(2) The initial value of the parameter is given as follows.

〈損失平滑度自動制御型ＭＣＥ学習〉
［第１の実施の形態］
前節で説明したＥＭアルゴリズムに基づくＰａｒｚｅｎ窓幅決定法を、誤分類測度空間上のＰａｒｚｅｎ推定に適用することにより、損失平滑度の自動制御を伴うＭＣＥ学習アルゴリズムが定型化できる。具体的には、以下のアルゴリズムとなる。図８を参照しながら説明する。

<Loss smoothness automatic control type MCE learning>
[First Embodiment]
By applying the Parzen window width determination method based on the EM algorithm described in the previous section to Parzen estimation in a misclassification measure space, an MCE learning algorithm with automatic loss smoothness control can be standardized. Specifically, the following algorithm is used. This will be described with reference to FIG.

(1) The initial value Λ ⁽⁰⁾ of the classifier learning parameter Λ is set. Set the upper limit E of the number of epochs e and set e = 0. (Step 140)
(2) The following step 180 is executed for e = 0, 1,. (Step 142) Step 180 includes the following sub-steps (a), (b) and (c).

(A) Repeat step 150 including the following sub-steps a1 to a3 for all classes y = 1,..., J.
(A1) Discriminant function values g _j (x _k ^y , Λ ^(e ) for all learning samples (this number is N _y ) belonging to class y for all classes j (j = 1..., J) ⁾ ) Is calculated (j = 1,..., J; k = 1,..., N _y ). (Steps 144, 146 and 148)
(A2) The following steps i to iii are executed for the class y. (Steps 152 to 158)
i. A misclassification measure value d _y (x _k ^y , Λ ^(e) ) is calculated (k = 1,..., N _y ). (Step 152)
ii. N ′ = N _y and x _k = d _y (x _k ^y , Λ ^(e) ) (k = 1,..., N _y ) (step 154), and execute the Parzen window width determination method based on the EM algorithm in the previous section And the optimum window width _hy is obtained. (Step 156)
iii. Equation (18) is executed to obtain the optimum loss smoothness α _y . (Step 158)
(B) The learning sample (x _n , y _n ) is taken out from the learning sample set Ω _N (step 162), and the classifier parameter Λ is adjusted by equation (10) (step 164). This is executed once for each learning sample in turn (step 160). When the adjustment for all samples is completed, a new classifier parameter Λ ^{(e + 1)} is obtained. (Step 166)
(C) to shuffle the order of the training samples in Ω _N. (Step 168)
The sub-step 2a in the above algorithm may be executed at every epoch e, but may be executed only when the execution interval E ′ is set and the epoch e is an integral multiple of the interval E ′.

<Experimental result>
The following experiment was performed using the apparatus using the Parzen window width determination method based on the EM algorithm according to the above-described embodiment.

  The device according to the above embodiment is originally applicable to various discriminant functions. Here, as an example, an experiment was performed using a classifier having a Euclidean distance from a prototype vector (abbreviated as “prototype”) as a discriminant function. “Prototype vector” refers to a vector representing each class. Multiple prototypes can be provided per class. In this experiment, the number of prototypes in each class was set to 8 in common. Due to the closeness between distance and probability, this classifier is highly versatile and can easily be applied to probability measure type discriminant functions such as hidden Markov models often used in speech recognition and the like.

The discriminant function in class C _j is given by

ここでｐ_ｊはＣ_ｊに属するプロトタイプの中でｘに最も近いものである。Λはすべてのプロトタイプの集合である。クラスＣ_ｙに属する学習標本ｘが与えられたとする。係数ψを∞にした式（５）の誤分類測度は、ｘに対するｂｅｓｔ‐ｉｎｃｏｒｒｅｃｔクラス（正解ではないが、正解に最も近い誤りクラス）をＣ_ｉとして

Here, p _j is the closest to x among the prototypes belonging to C _j . Λ is a set of all prototypes. _Assume that a learning sample x belonging to the class Cy is given. The misclassification measure of equation (5) with the coefficient ψ set to ∞ is the best-incorrect class for x (not the correct answer, but the error class closest to the correct answer) C _i

となる。

It becomes.

A Glass Identification data set provided by UCI Machine Learning Repository was used for the experiment. This data set is composed of 214 glass specimen patterns of 6 classes, and the contents of nine kinds of oxides contained in each glass specimen are given as 9-dimensional vector input patterns.
A recognition rate is obtained by removing one pattern from the data set as a recognition target, learning the classifier using the remaining patterns, and then recognizing the removed pattern for all 214 patterns. (Open data recognition rate) was calculated (Leave-One-Out method). In addition to the open data recognition rate calculation for the removed pattern as a recognition target, the recognition rate for the 213 patterns used for learning (closed data recognition rate) was also calculated. Since the calculation of the closed data recognition rate is performed 214 times, the average of them is used as the final closed data recognition rate.

Table 1 shows the recognition rate results for the conventional MCE learning method. In this method, MCE learning is performed with the loss smoothness α _y of the smoothed classification error loss in Expression (7) fixed to a predetermined value (loss smoothness α common to each class). That is, in the algorithm of the previous section, α ₁ =... = Α _J = α is fixed to a fixed value, and the sub-step 2a is omitted. In the table, the recognition rate for the loss smoothness α which is a plurality of types of fixed values is described. The highest open data recognition rate is obtained when α = 1.0 (75.23%).

テーブル２は、損失平滑度α_ｙを本発明技術により自動的に設定する新しいＭＣＥ学習法に対する認識率の結果である。

Table 2 shows the recognition rate results for a new MCE learning method in which the loss smoothness α _y is automatically set by the technique of the present invention.

オープン・データ認識率は、７４．３０％と、従来型のＭＣＥ学習法とほぼ同等の値が得られている。この値は、若干であるが、従来型のＭＣＥ学習法でのオープン・データ認識率の最大値より小さい。しかし、従来型のＭＣＥ学習法では、オープン・データ認識率を最大にする平滑化パラメータ（本実験では１．０）をアドホックに設定しなければならず、パターン認識器の学習に多大な労力を要することとなる。一方新しいＭＣＥ学習法では、平滑化パラメータがデータから自動的に算出され、学習の手間が大幅に削減されるとともに、従来型ＭＣＥ（の最大性能）と同等のオープン・データ認識率が得られている。

The open data recognition rate is 74.30%, which is almost the same value as the conventional MCE learning method. This value is slightly smaller than the maximum open data recognition rate in the conventional MCE learning method. However, in the conventional MCE learning method, the smoothing parameter (1.0 in this experiment) that maximizes the open data recognition rate must be set to ad hoc, and a great deal of effort is required to learn the pattern recognizer. It will be necessary. On the other hand, in the new MCE learning method, the smoothing parameter is automatically calculated from the data, and the labor of learning is greatly reduced, and the open data recognition rate equivalent to the conventional MCE (maximum performance) is obtained. Yes.

[Realization by computer]
The pattern classifier learning apparatus according to the first embodiment described above can be realized by a general-purpose computer and a computer program executed thereon. FIG. 10 shows the external appearance of a computer system 550 used in this embodiment, and FIG. 11 is a block diagram of the computer system 550. The computer system 550 shown here is merely an example, and other configurations can be used. The core part of this computer program has a control structure shown by the flowcharts of FIGS.

  Referring to FIG. 10, a computer system 550 includes a computer 560, a monitor 562, a keyboard 566, a mouse 568, a speaker 558, and a microphone 590 that are all connected to the computer 560. Further, the computer 560 includes a DVD-ROM (Digital Versatile Disk Read-Only-Memory) drive 570 and a semiconductor memory drive 572.

  Referring to FIG. 11, computer 560 further stores bus 586 connected to DVD-ROM drive 570 and semiconductor memory drive 572, CPU 576 all connected to bus 586, and a boot-up program for computer 560. ROM 578, RAM 580 which provides a work area used by CPU 576 and serves as a storage area for programs executed by CPU 576, hard disk drive 574 for storing observation data (learning data), and the like, network And a network interface 596 that provides a connection to 552.

  The software that realizes the system of the above-described embodiment is distributed in the form of object code, script, or source program recorded on a computer-readable recording medium such as DVD-ROM 582 or semiconductor memory 584, and is a DVD-ROM drive. The data is provided to the computer 560 via a reading device such as 570 or the semiconductor memory drive 572 and stored in the hard disk drive 574. When the source program is introduced into the computer 560, it is necessary to compile with a predetermined compiler to generate an object code. When the CPU 576 executes the program, the object program (or script) is read from the hard disk drive 574 and stored in the RAM 580. An instruction is fetched from an address designated by a program counter (not shown), and the instruction is executed. The CPU 576 reads data to be processed from the hard disk drive 574 and stores the processing result in the hard disk drive 574 as well. The speaker 558 and the microphone 590 are not directly related to the present invention, but the speaker 558 is necessary when reproducing sound. When learning data about speech is collected, a microphone 590 is required for recording speech data.

  The learning data is collected in advance and includes a large number of sets of input patterns and classes to which the patterns belong. The learning data is stored in the hard disk drive 574. The class classification parameter set Λ calculated by the above processing is temporarily stored in the hard disk drive 574 or the like, and further copied to the classifier via the network or USB memory. The classifier classifies the input pattern into any class using the parameter set Λ for class classification.

  Since the general operation of computer system 550 is well known, detailed description will not be repeated here.

  Regarding the software distribution method, the software does not necessarily have to be fixed on a storage medium. For example, the software may be distributed from another computer connected to the network. A part of the software may be stored in the hard disk drive 574, and the remaining part of the software may be taken into the hard disk drive 574 from the network and integrated at the time of execution.

Typically, modern computers utilize the general functions provided by a computer operating system (OS) to achieve functions in a controlled manner according to the desired purpose. Therefore, a control structure that does not include a general function that can be provided from the OS or a third party, and that achieves a desired purpose as a whole even if the program specifies only a combination of execution orders of the general functions. It is obvious that the program is included in the scope of the present invention.
[Second Embodiment]
A result equivalent to the result obtained by the algorithm shown in FIG. 8 of the first embodiment can be obtained by a different algorithm. FIG. 12 shows a flowchart of a program that realizes such an algorithm.
Referring to FIG. 12, the program for realizing MCE learning with automatic loss smoothness control according to the second embodiment includes steps 140 and 142 similar to those shown in FIG. However, in step 142, step 600 is repeated for all epochs e = 0,..., E instead of step 180 in FIG.
Step 600 includes the following processing steps.
(A) Compute discriminant function values g _j (x _n , Λ ^(e) ) for all learning samples and all classes (j = 1,..., J; n = 1,..., N). (Steps 610, 612 and 614)
(B) For each class y = 1,..., J, execute step 620 consisting of the following steps i to iii. (Step 616)
i. A misclassification measure value d _y (x _k ^y , Λ ^(e) ) is calculated (k = 1,..., N _y ). (Step 152)
ii. As N ′ = N _y and x _k = d _y (x _k ^y , Λ ^(e) ) (k = 1,..., N _y ) (step 154), Parzen based on the same EM algorithm as in the first embodiment The window width determination method is executed to obtain the optimum window width _hy . (Step 156)
iii. Equation (18) is executed to obtain the optimum loss smoothness α _y . (Step 158)
When step 620 is executed for all classes y = 1,..., J, control passes to step 160. The processing after step 160 is the same as that shown in FIG.
Even when the algorithm shown in FIG. 12 is used, the same result as that of the first embodiment shown in FIG. 8 can be obtained.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

20, 30 Graph 40 Parzen window group 42, 82 Function 50, 60, 70, 80 Parzen window of classification error number risk estimate

Claims (6)

A classifier learning device that classifies an input pattern into one of J classes C _j (j is an integer of 1 to J),
Learning sample storage means for storing N (N is a positive integer) learning samples each including an input pattern and a class to which the input pattern belongs;
Initializing means for initializing the learning parameter Λ of the classifier by a predetermined setting method,
Class C misclassification measure value measure the degree to which the input pattern x is misclassified other classes of training samples belonging to _y d _{y (x, Λ)} is defined below,

However ψ is a positive real number, g y _(x, Λ) is said for each of the J Class C _y, since the input pattern x of training samples to determine the degree of whether belonging to the class Is an arbitrary form of discriminant function,
For each of the J-number of class C _y, determined the misclassification measure value for each of the training samples belonging to the class C _y, the true probability distribution that generated the sample belonging to the class, each of misclassification measure space A Parzen distribution centered on the misclassification measure value and having a Parzen window width _hy , and including, as a function of the misclassification measure value, Parzen distribution estimation means for estimating by cross-validation maximum likelihood estimation,
The Parzen distribution estimation means evaluates the likelihood of the Parzen distribution as a function of the Parzen window width _{hy in} the intersection confirmation type maximum likelihood estimation,
For each of the J-number of class _{C y,} the Parzen in the cross-validation type maximum likelihood estimation by distribution estimating means to Parzen window width _{h y} giving the Parzen distribution as a maximum likelihood, the following function

The optimum loss smoothness calculating means for calculating the optimum value α _y of the loss smoothness for the class C _{y of the} classifier,
Learning parameter adjustment means for sequentially adjusting the learning parameter Λ so as to take out one learning sample from the learning sample set one by one and minimize the classification error risk;
The Parzen distribution estimating means, the optimum loss smoothness calculating means, and the learning parameter adjusting means are repeatedly operated until a predetermined end condition is satisfied, and the learning parameter Λ when the end condition is satisfied is output. A classifier learning device comprising: an iterative control means. The learning apparatus according to claim 1, further comprising: shuffling the order of the learning samples before repetition by the repetition control unit and before adjustment of the learning parameter Λ by the learning parameter adjustment unit. A learning apparatus including a shuffle means. 3. The learning apparatus according to claim 1, wherein the predetermined end condition is that the operation by the Parzen distribution estimation unit, the optimum loss smoothness calculation unit, and the learning parameter adjustment unit is completed a predetermined number of times. That is, the learning device. 4. The learning device according to claim 1, wherein the iterative control unit includes the Parzen among the Parzen distribution estimation unit, the optimum loss smoothness calculation unit, and the learning parameter adjustment unit. A learning apparatus that periodically omits the operations of the distribution estimation means and the optimum loss smoothness calculation means. The learning device according to any one of claims 1 to 4,
The Parzen window constituting the Parzen distribution is a Gaussian function,
The Parzen distribution estimating means removes one specimen from the specimen belonging to the class _{C y,} a Parzen estimated distribution configuration means for configuring the Parzen estimate distribution in the rest of the sample,
The expression for defining the Parzen estimate distribution arrangement means, mixing weighting factor 1 / (N'-1) ( N' Class _C belongs number of specimens _y) is N'-1 single Gaussian mixture probability density A learning apparatus comprising: a window width calculating means for calculating a Parzen distribution window width _hy that is regarded as a function and maximizes the mixed Gaussian distribution probability density function by an EM algorithm. In order to classify the input pattern into any of J classes C _j (j is an integer from 1 to J),
Learning sample storage means for storing N (N is a positive integer) learning samples each including an input pattern and a class to which the input pattern belongs;
A computer program that functions as an initialization unit for initializing the learning parameter Λ of the classifier by a predetermined setting method,
Class C misclassification measure value input pattern x of training samples belonging to _y is misclassified other classes d _{y (x,} lambda) is defined below,

However ψ is a positive real number, g y _(x, Λ) is said for each of the J Class C _y, since the input pattern x of training samples to determine the degree of whether belonging to the class Is an arbitrary form of discriminant function,
The computer program further includes the computer.
For each of the J-number of class C _y, determined the misclassification measure value for each of the training samples belonging to the class C _y, the true probability distribution that generated the sample belonging to the class, each of misclassification measure space A Parzen distribution with a Parzen window width _hy , centered on the misclassification measure value, and functioning as a Parzen distribution estimation means for estimating by cross-validation maximum likelihood estimation as a function of the misclassification measure value,
The Parzen distribution estimation means evaluates the likelihood of the Parzen distribution as a function of the Parzen window width _{hy in} the intersection confirmation type maximum likelihood estimation,
The computer program further includes the computer.
For each of the J-number of class _{C y,} the Parzen in the cross-validation type maximum likelihood estimation by distribution estimating means to Parzen window width _{h y} giving the Parzen distribution as a maximum likelihood, the following function

The optimum loss smoothness calculating means for calculating the optimum value α _y of the loss smoothness for the class C _{y of the} classifier,
Learning parameter adjustment means for sequentially adjusting the learning parameter Λ so as to take out one learning sample from the learning sample set one by one and minimize the classification error risk;
The Parzen distribution estimating means, the optimum loss smoothness calculating means, and the learning parameter adjusting means are repeatedly operated until a predetermined end condition is satisfied, and the learning parameter Λ when the end condition is satisfied is output. A computer program that functions as a repetitive control means.

Images