JP5684084B2 - Misclassification detection apparatus, method, and program - Google Patents

Description

  The present invention relates to a misclassification detection apparatus, method, and program, and more particularly to a misclassification detection apparatus, method, and program for detecting a sample of content classified into an incorrect category from a sample set.

  In many cases, content classification is performed manually. Alternatively, you can design a statistical classifier using some content manually categorized as training data, and use the statistical classifier to estimate new content categories. Do.

  However, there is always a risk of misclassification that classifies content into an incorrect category in manual classification work. In addition, content classified into an incorrect category causes a decrease in the automatic classification performance of the statistical classifier. Therefore, a misclassification detection technique for detecting a sample classified in a wrong category from given classified samples is important.

  In the conventional technique, in order to estimate misclassified samples from a set of classified samples, first, a statistical classifier trained using cross-validation with all the classified samples as training data is used. To estimate the sample category. Next, a sample whose estimated category does not match the classified category is detected as a misclassified sample. In order to improve the detection accuracy, the techniques of Non-Patent Documents 1 and 2 take the majority of the category estimation results obtained by a plurality of statistical classifiers, thereby reducing the bias of category estimation depending on the type of statistical classifier. Suppresses adverse effects. In the techniques of Non-Patent Documents 3 and 4, since there are only two types of categories, the category of another sample is estimated using a statistical classifier trained by replacing the category of one sample with a different category. . The accuracy of category estimation is improved by performing final determination from a plurality of estimation results collected by changing the sample for replacing the category.

Carla E. Brodley and Mark A. Friedl. Identifying mislabeled training data. Journal of Artificial Intelligence Research, 11 (11): 131-166, 1999. Sundara Venkataraman, Dimitris Metaxas, Dmitriy Fradkin, Casimir Kulikowski, and Ilya Muchnik.Distinguishing mislabeled data from correctly labeled data in classifier design.In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'04), pages 668-672 , 2004. Andrea Mallosini, Enrico Blanzieri, and Raymond T. Ng.Detecting potential labeling errors in microarrays by data perturbation.Bioinformatics, 22 (17): 2114-2121, 2006. Chen Zhang, Chunguo Wu, Enrico Blanzieri, You Zhou, Yan Wang, Wei Du, and Yanchun Liang.Methods for labeling error detection in microarrays based on the effect of data perturbation on the regression model.Bioinformatics, 25 (20): 2708− 2714, 2009.

  In the techniques of Non-Patent Documents 1 and 2 described above, a statistical classifier used for category estimation is learned using training data excluding samples to be estimated. The training data includes misclassified samples. In general, the performance of a statistical classifier trained with a training data set containing misclassified samples is inferior to that of a statistical classifier trained with exclusion of misclassified samples from the training data set . Therefore, in order to improve the accuracy of the statistical classifier, there is a need for a technique that suppresses adverse effects of misclassified samples on the learning of the statistical classifier.

  In the non-patent documents 3 and 4 described above, the category estimation accuracy is improved by performing learning by replacing the categories assigned to the samples in the training data. However, the techniques described in Non-Patent Documents 3 and 4 assume a problem only when there are two types of categories, and cannot be applied to a classification problem with a plurality of categories.

  The present invention has been made in view of the above circumstances, and suppresses adverse effects caused by misclassified samples in the learning of a statistical classifier used for detection, and is an error in a general classification problem having a plurality of categories. An object of the present invention is to provide a misclassification detection apparatus, method, and program capable of detecting a classified sample.

In order to achieve the above object, a misclassification detection apparatus according to the present invention includes a misclassification sample that is a content classified into an incorrect category, and includes a misclassification sample from a set of samples to which the category to which the content belongs is known. A misclassification detection device for detecting
Content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate ^ theta of parameters theta of, the joint probability model p (x based on one point out cross assay, y; A probability model generating means for calculating using a weight w _n set for each sample n so as to maximize the prediction likelihood of Θ )
Based on the estimated value { circumflex over (θ )} of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) calculated by the probability model generation unit, the content x _n is classified for each sample _n . A misclassification that calculates a predicted class posterior probability P (y _n | x _n ; ^ Θ ) and detects misclassified samples based on the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Sample detection means;
Only including,
The probability model generation means includes:
Based on a weight parameter estimate ^ W that defines a weight w _0n to set a large value for the incorrectly classified sample n and a weight w _1n to set a large value to the correctly classified sample n Using the estimated value Θ Θ _n of the parameter Θ based on the one-point exclusion cross-validation method, the correct or incorrect prediction probability P (z |) that gives a prediction of the latent variable z indicating whether each sample n is correctly classified x _n , y _n ; Θ _−n ), a correct / incorrect prediction probability calculation means;
Weight calculation means for calculating an estimated value ^ W of the weight parameter matrix using the correct / incorrect prediction probability P (z | x _n , y _n ; ^ Θ _-n ) calculated by the correct / incorrect prediction probability calculation means ;
The amount of change of the estimated value ^ W of the weight parameter matrix is calculated and the calculation by the correct / incorrect prediction probability calculating means until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is obtained or until a predetermined number of times is reached, and Convergence determining means for repeatedly performing calculation by weight calculating means;
Using the estimated value ^ W of the weight parameter matrix obtained by the iterative processing by the convergence determination means, a one-point exclusion cross-validation method for the parameter Θ of the joint probability model p (x, y; Θ) for each sample n And a parameter calculation means for calculating an estimated value ^ Θ _-n based on .

The misclassification detection method according to the present invention is a misclassification detection method for detecting a misclassification sample from a sample set including a category to which a content belongs, including a misclassification sample that is a content classified into an incorrect category. There,
By the probability model generation means, content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate of the parameter theta of ^ theta, the joint probability model based on a single point out cross assay p (x, y; Θ) the predicted likelihood of so as to maximize, calculating using the weight w _n set for each sample n,
The content x _n is classified for each sample n based on the estimated value ΘΘ of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) calculated by the probability model generation means by the misclassified sample detection means. Calculated predicted class posterior probability P (y _n | x _n ; ^ Θ ) of the category y _{n, and based on} the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Detecting a classification sample;
Only including,
Calculating the parameter value Θ of the joint probability model p (x _n , y _n ),
The weight parameter matrix defining the weight w _0n to set a large value for the sample n incorrectly classified and the weight w _1n to set a large value to the sample n correctly classified by the correct / incorrect prediction probability calculation means. Give a prediction of the latent variable z that represents whether each sample n is correctly classified, using the estimated value ^ Θ _-n of the parameter Θ based on the one-point exclusion cross-validation method based on the estimated value ^ W Calculating a correct / incorrect prediction probability P (z | x _n , y _n ; Θ _−n ) ;
The weight calculation means calculates the estimated value ^ W of the weight parameter matrix using the correct / incorrect prediction probability P (z | x _n , y _n ; ^ Θ _-n ) calculated by the correct / incorrect prediction probability calculation means. Steps,
By calculating the amount of change of the estimated value ^ W of the weight parameter matrix by the convergence determination means and calculating the correct or incorrect prediction probability until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is obtained or until a predetermined number of times is reached. Performing a calculation process by means and a repetition process of calculation by the weight calculation means;
By using the estimated value ^ W of the weight parameter matrix obtained by the iterative processing by the convergence determination means by the parameter calculation means, the parameter Θ of the joint probability model p (x, y; Θ) is calculated for each sample n. Calculating an estimate ^ Θ _-n based on a one-point exclusion cross-validation method ;
The characterized by containing Mukoto.

According to the present invention, the probability model generation means, content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate ^ theta of parameters theta of one point out cross assay the joint probability model p (x, y; Θ) based on the predicted likelihood of so as to maximize, calculated using the weights w _n set for each sample n.

Then, based on the estimated value ^ Θ of the parameter value Θ of the joint probability model p (x , y ; Θ ) calculated by the probability model generation means by the misclassified sample detection means, content x _n for each sample _n Calculate the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of the category y _n into which the categorization is classified, and based on the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Detect misclassified samples.

Maximum predicted likelihood Thus, the joint probability model p (x, y;; Θ ) estimate of the parameter theta of ^ theta a joint probability model p (Θ x, y) based on one point out cross assay The prediction class of each sample n is calculated using the weight set for each sample n and calculated based on the estimated value ΘΘ of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) By detecting the misclassified sample using the posterior probability P (y _n | x _n ; ^ Θ ), the adverse effect of the misclassified sample on the learning of the statistical classifier used for detection is suppressed, Samples misclassified by a general classification problem with multiple categories can be detected.

The probability model generation means according to the present invention provides a weight w _1n for setting a large value to the correctly classified sample n so as to maximize the sum of log likelihoods of each sample n based on the one-point exclusion cross-validation method. A weight calculation means for calculating the estimated value ^ W of the weight parameter matrix that prescribes the weight, and until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is calculated by calculating the amount of change of the estimated value ^ W of the weight parameter matrix Or until a predetermined number of times is reached, a convergence determination unit that repeats the calculation by the weight calculation unit, and an estimated value ^ W of the weight parameter matrix obtained by the repetition process by the convergence determination unit, for each n, before Symbol joint probability model p (x, y; Θ) and parameter calculation means for calculating an estimated value ^ theta _-n based on one point out cross assay parameters theta of, It can be made to contain.

  The program according to the present invention is a program for causing a computer to function as each unit of the misclassification detection apparatus.

As described above, misclassification detecting apparatus of the present invention, a method, and according to the program, the joint probability model p (x, y; Θ) an estimate ^ theta of parameters theta of, on one point out cross assay joint probability model p (x, y; Θ) based predictions likelihood so as to maximize, using the weight set for each sample n is calculated, the joint probability model p (x, y; Θ) parameter values A statistical class utilized for detection by detecting misclassified samples using the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n calculated based on the estimated value of Θ. It is possible to suppress the adverse effect of the misclassified sample on the learning of the classifier, and to detect the misclassified sample by the general classification problem having a plurality of categories.

It is the schematic which shows the structure of the misclassification detection apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the structure of the probability model production | generation part in the misclassification detection apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the misclassification detection process routine in the misclassification detection apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the probability model generation process routine in the misclassification detection apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the structure of the probability model production | generation part in the misclassification detection apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the probability model generation process routine in the misclassification detection apparatus which concerns on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Content consisting of text information such as articles, patents, online news data, e-mail, etc. included in the database, content consisting of text information and link information such as Web data, blog data, or content such as image data, etc. This is a misclassification detection device that detects samples classified into the wrong category from a set of samples classified into categories that represent categories such as sports, music, and mathematics. A case where the invention is applied will be described.

[First Embodiment]
<System configuration>
The misclassification detection apparatus 100 according to the first exemplary embodiment of the present invention receives a set of samples including content to which a label of the category to which the user belongs belongs, and selects an incorrect category from the set of input samples. Detect and output the sample with the label. This misclassification detection apparatus 100 is constituted by a computer including a CPU, a RAM, and a ROM storing a program for executing a misclassification detection processing routine to be described later, and is functionally configured as follows. Has been. As shown in FIG. 1, the misclassification detection apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 30.

The input unit 10 receives a set of samples including input content to which a category label to which the user belongs is assigned. Words or pixels, link or T = the feature space composed of combinations thereof, and the like, in the content _{{t 1, ..., t i} , ..., t V} when the characteristic of the content The vector x is expressed by x = {x ₁ ,..., X _i ,..., X _V ) ^T based on the frequency xi of ti included in the content. V represents the number of types of features that may be included in the content. For example, when the content is text data, V represents the total number of vocabularies that can appear in the content. Each sample in the sample set includes a content feature vector x and a label y of the category to which the sample belongs.

In addition, the input unit 10 inputs various parameters (hyperparameter vector η, prior probability P (y) of category, prior probability P (z) of right and wrong, and category y _{n of the nth} sample, which will be described later. Accept the class conditional probability P (y | x, z ₀ )) if it is incorrect.

  The calculation unit 20 includes a sample database 21, a probability model generation unit 22, a storage unit 23, and a misclassification sample detection unit 24.

The sample database 21 stores the sample set received by the input unit 10. Here, it is assumed that the sample set for misclassification detection is D = {(x _n , y _n )} ^N _{n = 1} .

The probability model generation unit 22 sets the parameter Θ = [of the joint probability model p (x, y; θ _y ) for the sample set D = {(x _n , y _n )} ^N _{n = 1} to be misclassified. θ ₁ ,..., θ _k ,..., θ _K ] are calculated based on an estimated value {^ Θ _−n } ^N _{n = 1} . The estimated value {^ Θ _−n } ^N _{n = 1} of the calculated probability model parameter is stored in the storage unit 23.

Here, n represents an ID number of a sample included in the sample set to be misclassified, x _n is a feature vector of the nth sample, y∈ {1,..., K,. Represents the category to which the sample belongs. ^ Θ _−n is a subset D _−n = {(x _{n ′} , y _{n ′} )} _{n ′ ≠ n} obtained by excluding the n th sample (x _n , y _n ) from the sample set D This is an estimated value of the parameter of the probability model to be calculated, and p represents the probability density.

The misclassified sample detection unit 24 uses the estimated value ^ Θ _−n of the parameter of the joint probability model to predict the posterior probability P (y _n | x _n ; ^ Θ _−n ) = p (x _n , y) of each sample. _n ; ^ θ _{yn, −n} ) / Σ ^K _{k = 1} p (x _n , k; ^ θ _{k, −n} ), and the sample with a small predicted class posterior probability is classified as an incorrect category Detect as a sample. Alternatively, the misclassified sample detection unit 24 calculates the ratio R _n = P (y _n | x) between the maximum value of the predicted posterior probability and the predicted class posterior probability for categories y ≠ y _n other than the category y _n in which the sample is classified. _{n; ^ Θ -n) / max} y ≠ yn P (y | x n; ^ Θ -n) is calculated and may be detected sample R _n is small. Here, P represents a probability value.

  The output unit 30 outputs the detection result of the misclassified sample to the user.

  As shown in FIG. 2, the probability model generation unit 22 includes an accuracy prediction probability calculation unit 31, a weight calculation unit 32, a first convergence determination unit 33, a second convergence determination unit 34, a parameter calculation unit 35, Is provided.

The correct / incorrect prediction probability calculation unit 31 reads the sample set D = {(x _n , y _n )} ^N _{n = 1} stored in the sample database 21 and sets the initial value W ⁽⁰⁾ of the weight parameter matrix or the second The weight parameter matrix W ^{(t) in the} middle of convergence input from the convergence determination unit 34, the hyperparameter vector η, the prior probability P (y) of the category, the prior probability P (z) of accuracy, and the nth sample And the class conditional probability P (y | x, z ₀ ) when the category y _n is incorrect, and the correct / incorrect prediction probability P (z | x _n , y _n ; ^ Θ ^{(t )} _-N ) is calculated. Here, z∈ {z ₁ , z ₀ } is a latent variable indicating whether or not the category y into which the sample content x is classified is correct. If z = z ₁ , the classification is correct. This means that if z = z ₀ , it is classified incorrectly. The weight parameter matrix W is a weight vector w ₁ = (w ₁₁ ,..., W _1n ,... Having weights w _1n that are given larger values as the n-th sample is more likely to be correctly classified. .,, w _1N ) ^T and weight vector w ₀ = (w ₀₁ ,..., with elements having weights w _0n that are given larger values as the nth sample is more likely to be misclassified. w _0n , _... , w _0N ) ^T and a matrix W = [w ₁ , w ₀ ]. a ^T represents a transposed vector of a.

The weight calculation unit 32 is input from the initial value W ⁽⁰⁾ of the weight parameter matrix, the weight parameter matrix W ^{(t) during} convergence input from the second convergence determination unit 34, or the first convergence determination unit 33. Using the weight parameter matrix W ^{(s) in the} middle of convergence and the correct / incorrect prediction probability P (z | x _n , y _n ; ^ Θ ^(t) _−n ) of each sample n, the updated value W ^{( s + 1)} is calculated.

The first convergence determination unit 33 calculates the change amount d (s) of the weight parameter matrix, and if the convergence condition d (s) <ε _s is satisfied, the weight is set as W ^{(t + 1)} ← W ^{(s + 1).} The parameter matrix estimation value W ^{(t + 1)} is output to the second convergence determination unit 34. If the convergence condition is not satisfied, the parameter learning step is updated as s ← s + 1, and the process of the weight calculation unit 32 is performed again. This process is repeated until the convergence condition is satisfied or s reaches a predetermined number of times s _max .

The second convergence determination unit 34 calculates the change d (t) of the weight parameter matrix, and if the convergence condition d (t) <ε is satisfied, the weight parameter matrix is estimated as ^ W ← W ^{(t + 1).} Outputs the value ^ W. If the convergence condition is not satisfied, the parameter learning step is updated as t ← t + 1, and a series of processing by the correct / incorrect prediction probability calculation unit 31, the weight calculation unit 32, and the first convergence determination unit 33 is performed again. carry out. This process is repeated until the convergence condition is satisfied or t reaches a predetermined number t _max .

The parameter calculation unit 35 calculates and outputs an estimated value {^ Θ _−n } ^N _{n = 1} based on the one-point exclusion cross validation method of the probability model using the estimated value ^ W of the weight parameter matrix.

  Here, the probability model in the present embodiment will be described. Hereinafter, a case where a Naive Bayes model (hereinafter referred to as an NB model) based on a multinomial distribution is used as the probability model p (x, y; θy) will be described as an example.

In the NB model based on the multinomial distribution, when the content is classified into the correct category, the joint probability model p (x, y; θ _y ) = p (x | y; θ _y ) P of category y and feature vector x p (x | y; θ _y ) of (y) is defined by the following equation (1), assuming that the appearance probability θy _i of each feature t _i in category y is independent.

Where θ _y = (θ _y1 , _... , Θ _yi , _... , Θ _yV ) ^T , θ _yi > 0 and || θ _y || ₁ = Σ ^V _{i = 1} θ _yi = 1 It is. Θ = [θ ₁ ,..., Θ _k ,..., Θ _K ] ^T represents a parameter matrix of the NB model. P (y)> 0 represents the appearance probability of the category y and satisfies Σ ^K _{k = 1} P (k) = 1.

Further, in the present embodiment, assuming that the probability probability θ _z0i of each feature t _i is independent of the probability model of the feature vector x of the sample classified into the wrong category, the following equation (2) Defined in

Where θ _z0 = (θ _z01 , _... , Θ _z0i , _... , Θ _z0V ) ^T , θ _z0i > 0 and || θ _z0 || ₁ = Σ ^V _{i = 1} θ _z0i = 1 It is.

<Operation of misclassification detection device>
Next, the operation of the misclassification detection apparatus 100 according to the first embodiment will be described. First, when a set of samples including content labeled with the class to which it belongs is input to the misclassification detection apparatus 100, the input sample set is stored in the sample database 21 by the misclassification detection apparatus 100. In addition, various parameters (hyper parameter vector η, prior probability P (y) of category, correct prior probability P (z), and class conditional probability P when n-th sample category y _n is incorrect. (y | x, z ₀ ))) is input to the misclassification detection apparatus 100, the misclassification detection apparatus 100 executes the misclassification detection processing routine shown in FIG.

First, in step S101, the probability model generation unit 22 reads the sample set D = {(x _n , y _n )} ^N _{n = 1} from the sample database 21 and sets the probability for each sample n. Calculate the estimated value ^ Θ- _n based on the model parameter one-point exclusion cross-validation method. Each element of ^ Θ _−n includes a weight parameter vector w ₁ that is an element of the input weight parameter matrix W ⁽⁰⁾ or the calculated weight parameter matrix W ^(t) , and an input hyperparameter value η _y , ∀ with _y, is calculated by the following equation (3).

Here, I _{y (y n ')} is, y _{n' 'and = 1, y n I y (} y n)' in the case of = y and _{I y (y n ') =} 0 in the case of ≠ y It is an indicator function. || x _{n ′} || ₁ represents the L1 norm of x _{n ′} .

Similarly, each element of the estimated value ^ θ _{z0, −n} based on the one-point exclusion cross- _validation of the parameter of the probability model p (x | z ₀ ; θ _z0 ) of the feature vector x of the sample classified into the wrong category Using the weight parameter vector w ₀ , which is an element of the input weight parameter matrix W ⁽⁰⁾ or the calculated weight parameter matrix W ^(t) , and the input _{hyperparameter} value η _z0 , Calculate according to equation (4).

The _{hyperparameter} vector η = (η ₁ , _... , Η _k , _... , Η _K , η _z0 ) is a constant value set in advance for parameter calculation.

Here, the principle of calculating the estimated value ^ Θ- _n based on the one-point exclusion cross-validation method of the parameters of the probability model will be described.

In the present embodiment, the value of the weight parameter matrix W is given by maximizing the prediction likelihood of the joint probability model p (x, y) for the sample set D based on the one-point exclusion cross validation method. Since the sample set includes samples classified in the wrong category, the joint probability model p of the content feature vector x, the category y, and the latent variable z∈ {z ₁ , z ₀ } representing the right or wrong Using (x, y, z) = p (x, y | z) P (z), the joint probability model p (x, y) = Σ ¹ _{j = 0} p (x, y | z _j ) P ( z _j ) is designed. When the class of the sample is correct (z = z ₁ ), p (x, y | z ₁ ) is given by the following equation (5).

Further, p (x, y | z ₀ ) when the sample class is incorrect (z = z ₀ ) is given by the following equation (6).

  At this time, the value of the weight parameter matrix W is given by maximization of the objective function shown in the following equation (7).

Note that L (W) in the above equation (7) corresponds to the prediction likelihood of the joint probability model p (x _n , y _n ) based on the one-point exclusion cross-validation method.

  The value of W that maximizes the objective function shown in the above equation (7) can be obtained by performing repeated calculations such as the expected value maximization (EM) algorithm twice. Regarding the EM algorithm, reference literature (AP Dempster, NMLaird, and DB Rubin: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39, 1-38 (1977)). Since it is described, detailed description is omitted.

When the estimated value of the W in the learning step (t) and ^{W (t), log b ≦} b-1 than ^{L (W) -L (W (} t)) ≧ Q (W, W (t)) - A Q function satisfying Q (W ^(t) , W ^(t) ) can be given by the following equation (8).

  The Q function shown in the above equation (8) is given by weighting and adding the log likelihood of each sample based on the one-point exclusion cross-validation method with the correct / incorrect prediction probability.

W that maximizes L (W) locally by repeatedly calculating the value that maximizes Q (W, W ^(t) ) as an estimated value W ^{(t + 1) in the} learning step (t + 1) Can be sought.

Then, based on the estimated value ^ W of the weight parameter matrix finally obtained, according to the above equation (3), the estimated value {^ Θ _−n } ^N _{n =} Calculate ₁

  The processing in step S101 is realized by a probability model generation processing routine shown in FIG.

In step S111, the correct / error prediction probability calculation unit 31 applies P (z | x _n , y _n ; ^ Θ ⁽ ) given by the above equations (9) and (10) for each sample n as follows. ^t) _-n ) is calculated.

First, a parameter value obtained by substituting the input estimated value W ^(t) of the weight parameter matrix into the above equations (3) and (4) is represented as ^ Θ ^(t) _−n = [^ θ ^(t) _{1 , −n} , ..., ^ θ ^(t) _{k, −n} , ..., ^ θ ^(t) _{K, −n} , ^ θ ^(t) _{z0, −n} ]. Next, p (x _n | y _n ; ^ θ _{yn, −n} (w ^(t) ₁ ) is obtained by substituting the values into θ _y in the above equation (1) and θ _z0 in the above equation (2). ) And p (x _n ; ^ θ _{z0, −n} (w ^(t) ₀ )). Further, P (z | x _n , y _n ; ^ Θ ^(t) _−n ) is obtained by substituting the value into the above equations (9) and (10) and calculating. In other words, the correct / incorrect prediction probability calculation unit 31 calculates the correct / incorrect prediction probability using a joint probability model designed for a sample classified into a correct category and a joint probability model designed for a misclassified sample. . Prior probabilities P (y) and P (z) and class conditional probabilities P (y | x, z ₀ ) included in the equations (9) and (10) may be estimated as unknown parameters. Although possible, in order to improve accuracy by simplifying the parameter calculation algorithm and adjusting these parameter values, in this embodiment, the parameter values are given from the outside.

Also, the value of W that maximizes Q (W, W ^(t) ) is Q (W, W ^(t) from log ≦ b−1, where W ^(s) is the estimated value in W learning step (s). W ^(t) ) −Q (W ^(s) , W ^(t) ) ≧ Q '(W, W ^(s) | W ^(t) ) −Q ′ (W ^(s) , W ^(s) | W ^{( A} Q function satisfying ^t) ) can be given by the following equation (11).

Therefore, the value that maximizes Q ′ (W, W ^(s) | W ^(t) ) in the learning step (s + ¹⁾ is repeatedly calculated as the estimated value W ^{(s + 1)} , so that Q (W, W W that maximizes ^(t) ) locally in the vicinity of W ^(t) can be obtained.

In step S112, the weight calculation unit 32 calculates a solution W ^{(s + 1)} of W that maximizes Q ′ (W, W ^(s) | W ^{(t) )} by the following equation (14), (15): Calculate according to the formula.

After calculating the learning step (s + 1) in the weight parameter matrix W ^{(s + 1),} the determination in step S113, the first convergence determining unit 33, for example, whether the convergence conditions are satisfied given by the following equation (16) To do.

Here, || w _{(s) j} || ₂ represents the L2 norm of vector w ^(s) _j . ε _s is a minute value given by the designer. If it is determined in step S113 that the convergence condition is satisfied, W ^{(s + 1)} is set as W ^{(t + 1)} , and the process proceeds to step S114. On the other hand, if it is determined that the convergence condition is not satisfied, the process from step S112 to step S113 is repeated as s ← s + 1.

After calculating the weighting parameter matrix W ^{(t + 1)} in the learning step (t + 1), in step S114, the second convergence determining unit 34, for example, whether the convergence conditions are satisfied given by the following equation (17) Determine.

Here, ε _t is a minute value given by the designer. If it is determined in step S114 that the convergence condition is satisfied, the process proceeds to step S115 with W ^{(t + 1)} as an estimated value ^ W of the weight parameter matrix. On the other hand, if the convergence condition is not satisfied, the process from step S111 to step S114 is repeated as t ← t + 1.

In step S115, the parameter calculation unit 35, the estimated value ^ W weight parameter matrices are substituted into equation (3), the estimated value of the parameter based on one point out cross assay probabilistic model for each sample n ^ theta _{- n} is calculated and stored in the storage unit 23, and the probability model generation processing routine is terminated.

  The parameter calculation algorithm described above is summarized and written as follows.

Step 1: Set various parameters.
1. Hyper parameter η, prior probabilities P (y), P (z), and class conditional probability P (y | x, z ₀ ) are set from the outside as predetermined values.
2. Set the convergence condition parameters ε _t , ε _s and the maximum number of iterations t _max , s _max .

Procedure 2: Set initial values of learning step t and weight parameter matrix.
1. Substitute 0 for t.
2. Set the value W ^(t) of the weight parameter matrix.

Step 3: Calculate the estimated value ^ W of the weight parameter matrix.
1. Using W ^(t) , the correct / incorrect prediction probability P (z | x _n , y of the sample for each sample n according to the above equations (1) to (4), (9), and (10). _n ; ^ Θ ^(t) _−n ) is calculated (step S111, FIG. 4).
2. Substitute 0 for s. Substitute W ^(t) for W ^(s) .
3. Calculate the weight parameter matrix value W ^{(t + 1)} that maximizes Q (W, W ^(t) ).
(a) Using W ^(s) , W ^{(s + 1)} is calculated by the above equations (12) to (15) (step S112, FIG. 4).
(b) Convergence determination processing is executed using the above equation (16) (step S113, FIG. 4).
4. Convergence determination processing is executed using the above equation (17) (step S114, FIG. 4).

Step 4: Substitute the estimated weight parameter matrix estimated value ^ W into the above equation (3), and calculate the estimated model parameter estimated value ^ Θ- _n based on the one-point exclusion cross validation method for each sample n (step S115, FIG. 4).

Step 5: Estimate the parameter {^ Θ _−n } ^N _{n = 1} is output to the misclassified sample detection unit 24.

In step S102 of the misclassification detection processing routine, the misclassification sample detection unit 24 uses the estimated value ^ Θ- _n of the parameter of the joint probability model to predict the posterior probability P (y _n | x _{n of} each sample _n. ; ^ Θ _−n ) = p (x _n | y _n ; ^ θ _{yn, −n} ) P (y _n ) / Σ ^K _{k = 1} p (x _n | k; ^ θ _{k, −n} ) P (k ) And the sample whose predicted class posterior probability is less than or equal to the threshold is detected as a sample that is suspected of being classified into the wrong category. Alternatively, the ratio between the maximum value of the predicted posterior probability and the predicted class posterior probability for a category y ≠ y _n other than the category y _n in which the sample is classified by the misclassified sample detection unit 24 R _n = P (y _n | x _n ; ^ Θ− _n ) / max _{y ≠ ynP} (y | _xn ; ^ Θ− _n ) may be calculated to detect samples in which R _n is equal to or less than a threshold value.

  The processing in step S102 can be easily realized by a general rearrangement algorithm, and thus further description is omitted.

As described above, according to the misclassification detection apparatus according to the first embodiment, the parameter value Θ of the joint probability model p (x _n , y _n ) is changed to the joint probability model p based on the one-point exclusion cross validation method. (x _n, y _n) so as to maximize the prediction likelihood of, using the weight set for each sample n is calculated, the joint probability model p (x _n, y _n) calculated based on the parameter value Θ of The misclassified samples are provided to the learning of the statistical classifier used for detection by detecting misclassified samples using the predicted class posterior probabilities P (y _n | x _n ) of each sample n It is possible to detect a sample misclassified by a general classification problem having a plurality of categories while suppressing adverse effects.

  Each sample included in the training data set was weighted, and the weighted training data set was used to learn the parameter value Θ of the statistical classifier based on the probability model. It makes it possible to reduce the adverse effects that the sample has on learning the probabilistic model. In addition, by setting the weight given to each sample included in the training data set to maximize the likelihood of the probability model obtained based on the one-point cross-validation method, misclassified samples can be accurately identified. A probability model is obtained that gives a higher predictive probability for a category of samples that are more accurately classified than misclassified samples if there are fewer than classified samples. This effect enhances the performance of the device that detects misclassified samples using the predicted probability of the category.

[Second Embodiment]
<System configuration>
Next, a second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The second embodiment is different from the first embodiment in that the correct / incorrect prediction probability calculation unit and the second convergence determination unit are omitted.

  As shown in FIG. 5, the probability model generation unit 222 of the misclassification detection apparatus according to the second embodiment includes a weight calculation unit 232, a first convergence determination unit 233, and a parameter calculation unit 35. .

The weight calculation unit 232 uses the initial value W ⁽⁰⁾ of the weight parameter matrix or the weight parameter matrix W ^(s) in the middle of convergence input from the first convergence determination unit 233 to update the weight parameter matrix W ^{(s +1)} is calculated.

The first convergence determination unit 233 calculates the change d (s) of the weight parameter matrix, and if the convergence condition d (s) <ε _s is satisfied, the weight parameter matrix is estimated as ^ W ← W ^{(s + 1).} Outputs the value ^ W. If the convergence condition is not satisfied, the parameter learning step is updated as s ← s + 1, and the process of the weight calculation unit 232 is performed again. This process is repeated until the convergence condition is satisfied or s reaches a predetermined number of times s _max .

Here, the principle of calculating the estimated value ^ Θ- _n based on the one-point exclusion cross-validation method of the parameters of the probability model will be described.

In the present embodiment, a joint probability model p (x, y, z) = p (x, y) of a content feature vector x, a category y, and a latent variable z∈ {z ₁ , z ₀ } representing correctness / incorrectness. | z) For P (z), P (z ₁ ) = 1 and P (z ₀ ) = 0.

The value of the weight parameter matrix W is given by maximization of the objective function shown in the following equation (18). Since P (z ₀ ) = 0, the weight vector w ₀ = (w ₀₁ , _... , W _0n , _... , W _0N ) ^T in the weight parameter matrix W is not calculated.

  The value of W that maximizes the objective function shown in the above equation (18) can be obtained by performing an iterative calculation such as an expected value maximization (EM) algorithm, as in the first embodiment. it can.

When the estimated value of the W in the learning step (s) and ^{W (s), log b ≦} b-1 than ^{L (W) -L (W (} s)) ≧ Q (W, W (s)) - A Q function satisfying Q (W ^(s) , W ^(s) ) can be given by the following equation (19).

  As described above, the value of the weight parameter matrix W is given by maximizing the sum of log likelihoods of each sample based on the one-point exclusion cross-validation method. Note that L (W) in the above equation (18) corresponds to the sum of log likelihoods of each sample n based on the one-point exclusion cross-validation method.

Then, based on the estimated value ^ W of the finally obtained weight parameter matrix, the estimated value {^ Θ _−n } ^N _{n = 1} based on the one-point exclusion cross-validation method of the probability model is calculated according to the above equation (3). calculate.

<Operation of misclassification detection device>
First, when a set of samples including content labeled with the class to which it belongs is input to the misclassification detection apparatus 100, the input sample set is stored in the sample database 21 by the misclassification detection apparatus 100. When the hyperparameter vector η is input to the misclassification detection apparatus 100, the misclassification detection process routine is executed by the misclassification detection apparatus 100 as in the first embodiment.

  A probability model generation processing routine according to the second embodiment will be described with reference to FIG. In addition, about the process similar to 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

First, in step S211, the weight calculation unit 232 calculates a solution W ^{(s + 1)} of W that maximizes the Q function of the above equation (19).

After calculating the learning step (s + 1) in the weight parameter matrix W ^{(s + 1),} at step S212, the the first convergence determining unit 233 determines whether or not a convergence condition is satisfied given by Equation (16).

If it is determined in step S212 that the convergence condition is satisfied, W ^{(s + 1)} is set as ^ W and the process proceeds to step S115. On the other hand, if it is determined that the convergence condition is not satisfied, the process of step S211 is repeated as s ← s + 1.

In step S115, the estimated value ^ W of the weight parameter matrix is substituted into the above equation (3) to calculate the estimated value ^ Θ- _n of the parameter based on the one-point exclusion cross validation method for each sample n, Store in the storage unit 23, and the probability model generation processing routine ends.

  Note that other configurations and operations of the misclassification detection apparatus according to the second embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

In the first embodiment, the estimated value ^ Θ _−n of the probability model parameter calculated as described above is obtained by calculating the correctness prior probability P (z) as P (z ₁ ) = 1, P (z ₀ ) Is equal to the estimated value ^ Θ- _n of the probability model parameter calculated when set to 0. Thus, the probability model generation unit can be designed in a simplified manner.

〔Example〕
Next, the results of experiments performed by applying the method according to the above embodiment will be described.

  An evaluation experiment was performed to detect document data classified into an incorrect subcategory in the problem of classifying document data belonging to a computer as a higher category into one of five subcategories. 20 newsgroups (20News, reference (K. Nigam, A. McCallum, S. Thrun, and T. Mitchell: Text classification from labeled and unlabeled documents using EM. Machine Learning, Vol. 39, pp. 103-134, 2000.)).

In order to create an evaluation data set, 1000 samples were randomly extracted from document data belonging to 5 subcategories. Then, r _m % samples were randomly selected from the 1000 samples, and misclassified samples were created by randomly changing the subcategory to which the document data belongs to one of the other four subcategories. A data set including misclassified samples obtained by this operation was used for performance evaluation as a sample set for misclassification detection. As a performance evaluation scale, the average precision (AP), which is often used to measure the ranking of samples in information retrieval tasks, was used. The average precision is calculated by the following equation (20), where M is the total number of misclassified samples.

  The average precision indicates that the larger the value, the higher the performance of ranking the samples.

Table 1 shows the average precision obtained when P (z ₁ ) = 0.5 (method 1) in the apparatus according to the present invention having the probability model generation unit 22 described in the first embodiment, and The average precision obtained with the apparatus (method 2) according to the present invention having the probability model generation unit 222 described in the second embodiment and a simple one-point exclusion cross-validation method without introducing a weight parameter matrix Shows the result of the average precision obtained by applying to the NB model (Method 3).

In the experiment, R _n = P (y _n | x _n ; ^ Θ _−n ) / max _{y− ≠ yn} P (using the estimated values {^ Θ _−n } ^N _{n = 1} obtained by each method. y | x _n ; ^ Θ _−n ) was calculated and detected as samples in which misclassification was suspected in ascending order. From the above Table 1, in all cases the experiments conducted by changing the value of r _m, Method 1, is average precision obtained by the process 2, higher than the average precision obtained in Method 3. From the above results, it was found that the apparatus according to the present invention is effective in detecting samples in descending order of risk of being classified incorrectly.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Input part 20 Calculation part 22, 222 Probability model production | generation part 24 Misclassification sample detection part 30 Output part 31 Correct / correct prediction probability calculation part 32, 232 Weight calculation part 33 1st convergence determination part 34 2nd convergence determination part 35 Parameter calculation part 100 Misclassification Detection Device 233 Convergence Determination Unit

Claims (7)

A misclassification detection device that detects misclassified samples from a set of samples to which a category to which content belongs is known, including misclassified samples that are contents classified into wrong categories,
Content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate ^ theta of parameters theta of, the joint probability model p (x based on one point out cross assay, y; A probability model generating means for calculating using a weight w _n set for each sample n so as to maximize the prediction likelihood of Θ )
Based on the estimated value { circumflex over (θ )} of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) calculated by the probability model generation unit, the content x _n is classified for each sample _n . A misclassification that calculates a predicted class posterior probability P (y _n | x _n ; ^ Θ ) and detects misclassified samples based on the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Sample detection means;
Only including,
The probability model generation means includes:
Based on a weight parameter estimate ^ W that defines a weight w _0n to set a large value for the incorrectly classified sample n and a weight w _1n to set a large value to the correctly classified sample n Using the estimated value Θ Θ _n of the parameter Θ based on the one-point exclusion cross-validation method, the correct or incorrect prediction probability P (z |) that gives a prediction of the latent variable z indicating whether each sample n is correctly classified x _n , y _n ; Θ _−n ), a correct / incorrect prediction probability calculation means;
Weight calculation means for calculating an estimated value ^ W of the weight parameter matrix using the correct / incorrect prediction probability P (z | x _n , y _n ; ^ Θ _-n ) calculated by the correct / incorrect prediction probability calculation means ;
The amount of change of the estimated value ^ W of the weight parameter matrix is calculated and the calculation by the correct / incorrect prediction probability calculating means until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is obtained or until a predetermined number of times is reached, and Convergence determining means for repeatedly performing calculation by weight calculating means;
Using the estimated value ^ W of the weight parameter matrix obtained by the iterative processing by the convergence determination means, a one-point exclusion cross-validation method for the parameter Θ of the joint probability model p (x, y; Θ) for each sample n A parameter calculation means for calculating an estimated value ^ Θ _-n based on
A misclassification detection device including: A misclassification detection device that detects misclassified samples from a set of samples to which a category to which content belongs is known, including misclassified samples that are contents classified into wrong categories,
Content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate ^ theta of parameters theta of, the joint probability model p (x based on one point out cross assay, y; A probability model generating means for calculating using a weight w _n set for each sample n so as to maximize the prediction likelihood of Θ )
Based on the estimated value { circumflex over (θ )} of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) calculated by the probability model generation unit, the content x _n is classified for each sample _n . A misclassification that calculates a predicted class posterior probability P (y _n | x _n ; ^ Θ ) and detects misclassified samples based on the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Sample detection means;
Only including,
The probability model generation means includes:
An estimate of the weight parameter matrix that defines the weight w _1n that should be set to a large value for the correctly classified sample n so as to maximize the sum of the log likelihoods of each sample n based on the one-point exclusion cross-validation method ^ A weight calculation means for calculating W;
The amount of change in the estimated value ^ W of the weight parameter matrix is calculated, and the calculation process by the weight calculation means is repeated until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is obtained or until a predetermined number of times is reached. A convergence determination means to perform;
Using an estimated value ^ W obtained the weight parameter matrix by repeating processing by the convergence determination unit, for each sample n, before Symbol joint probability model p (x, y; Θ) one point out cross test parameters theta of A parameter calculation means for calculating the estimated value ^ Θ _-n based on the method ;
A misclassification detection device including:   4. The joint probability model p (x, y; Θ) = p (x | y; Θ) P (y) of p (x | y; Θ) is given by a Naive Bayes model whose parameter is represented by Θ. The misclassification detection apparatus according to 1 or 2. A misclassification detection method for detecting a misclassified sample from a set of samples in which a category to which the content belongs is known, including a misclassified sample that is content classified into a wrong category,
By the probability model generation means, content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate of the parameter theta of ^ theta, the joint probability model based on a single point out cross assay p (x, y; Θ) the predicted likelihood of so as to maximize, calculating using the weight w _n set for each sample n,
The content x _n is classified for each sample n based on the estimated value ΘΘ of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) calculated by the probability model generation means by the misclassified sample detection means. Calculated predicted class posterior probability P (y _n | x _n ; ^ Θ ) of the category y _{n, and based on} the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Detecting a classification sample;
Only including,
Calculating the parameter value Θ of the joint probability model p (x _n , y _n ),
The weight parameter matrix defining the weight w _0n to set a large value for the sample n incorrectly classified and the weight w _1n to set a large value to the sample n correctly classified by the correct / incorrect prediction probability calculation means. Give a prediction of the latent variable z that represents whether each sample n is correctly classified, using the estimated value ^ Θ _-n of the parameter Θ based on the one-point exclusion cross-validation method based on the estimated value ^ W Calculating a correct / incorrect prediction probability P (z | x _n , y _n ; Θ _−n ) ;
The weight calculation means calculates the estimated value ^ W of the weight parameter matrix using the correct / incorrect prediction probability P (z | x _n , y _n ; ^ Θ _-n ) calculated by the correct / incorrect prediction probability calculation means. Steps,
By calculating the amount of change of the estimated value ^ W of the weight parameter matrix by the convergence determination means and calculating the correct or incorrect prediction probability until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is obtained or until a predetermined number of times is reached. Performing a calculation process by means and a repetition process of calculation by the weight calculation means;
By using the estimated value ^ W of the weight parameter matrix obtained by the iterative processing by the convergence determination means by the parameter calculation means, the parameter Θ of the joint probability model p (x, y; Θ) is calculated for each sample n. Calculating an estimate ^ Θ _-n based on a one-point exclusion cross-validation method ;
Misclassification detection method characterized by including Mukoto a. A misclassification detection method for detecting a misclassified sample from a set of samples in which a category to which the content belongs is known, including a misclassified sample that is content classified into a wrong category,
By the probability model generation means, content x and the joint probability model p (x, y; theta) of the sample represented by category y an estimate of the parameter theta of ^ theta, the joint probability model based on a single point out cross assay p (x, y; Θ) the predicted likelihood of so as to maximize, calculating using the weight w _n set for each sample n,
The content x _n is classified for each sample n based on the estimated value ΘΘ of the parameter value Θ of the joint probability model p (x 1 , y 2 Θ ) calculated by the probability model generation means by the misclassified sample detection means. Calculated predicted class posterior probability P (y _n | x _n ; ^ Θ ) of the category y _{n, and based on} the predicted class posterior probability P (y _n | x _n ; ^ Θ ) of each sample n Detecting a classification sample;
Only including,
Calculating the parameter value Θ of the joint probability model p (x _n , y _n ),
A weight parameter that defines a weight w _1n for which a large value should be set for a correctly classified sample n so that the sum of log likelihoods of each sample n based on the one-point exclusion cross-validation method is maximized by the weight calculation means Calculating the matrix estimate ^ W;
By the convergence determination means, the amount of change of the estimated value ^ W of the weight parameter matrix is calculated, and until the estimated value ^ W of the weight parameter matrix satisfying the convergence condition is obtained or until a predetermined number of times is reached, the weight calculation means A step to repeat the calculation;
By the parameter calculating means, using the estimated value ^ W obtained the weight parameter matrix by repeating processing by the convergence determination unit, for each sample n, before Symbol joint probability model p (x, y; Θ) parameters theta Calculating an estimate ^ Θ _-n based on the one-point exclusion cross validation method ,
Including misclassification detection methods.   4. The joint probability model p (x, y; Θ) = p (x | y; Θ) P (y) of p (x | y; Θ) is given by a Naive Bayes model whose parameter is represented by Θ. 4. The misclassification detection method according to 4 or 5.   The program for functioning a computer as each means of the misclassification detection apparatus of any one of Claims 1-3.

Images