JP2014085948A - Misclassification detection apparatus, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect misclassified samples with high accuracy by suppressing adverse effects to be given to learning of a statistical classifier which is utilized for misclassification detection by the misclassified samples.SOLUTION: A probability model generation unit 12 divides a misclassification detection object sample set D into two subsample sets Dand D, and calculates a parameter estimation value ^Ψ of a conditional class probability model P(y|x;Ψ) which is a statistical classifier adapted to both of one subsample set Dand a sample set Dwithout a label. A misclassification determination unit 16 calculates conditional class probability P(y|x;^Ψ) to the respective categories of the respective misclassification detection object samples (x, y) included in the subsample set Dby using the parameter estimation value ^Ψ, and determines whether or not the respective misclassification detection object samples (x, y) are misclassified samples.

Description

  The present invention relates to a misclassification detection apparatus, method, and program, and more particularly to a misclassification detection apparatus, method, and program for detecting a sample of content classified into an incorrect category from a sample set.

  In many cases, content classification is performed manually. Alternatively, by learning a statistical classifier using a plurality of contents manually classified into categories as training data, and using this statistical classifier, estimating the category of the content whose category is unknown, the content The category classification is automatically performed.

  However, there is always a risk of misclassification that classifies content into an incorrect category in manual classification work. Moreover, when a sample of content classified into an incorrect category (hereinafter referred to as “misclassified sample”) is used as training data for the statistical classifier, the automatic classification performance of the statistical classifier is degraded. Therefore, a misclassification detection technique for detecting misclassified samples from given classified samples is important.

  In the conventional technique, in order to estimate misclassified samples from a set of categorized samples, first, a statistical classifier trained using cross-validation is performed using all categorized samples as training data. To estimate the sample category. Next, a sample in which the estimated category and the classified category do not match is detected as a misclassified sample. In order to improve the detection accuracy, the techniques of Non-Patent Documents 1 and 2 take a majority vote of the category estimation results obtained by a plurality of statistical classifiers, thereby reducing the bias of category estimation depending on the type of statistical classifier. Suppresses adverse effects. In the technique of Non-Patent Document 3, misclassification is achieved by improving the classification performance of the statistical classifier by using not-labeled sample collections in which not only categorized samples but also samples with unknown categories are collected for learning. Increases sample detection accuracy.

Carla E. Brodley and Mark A. Friedl, "Identifying mislabeled training data", Journal of Artificial Intelligence Research, 11 (11): 131-166, 1999. Sundara Venkataraman, Dimitris Metaxas, Dmitriy Fradkin, Casimir Kulikowski, and Ilya Muchnik, "Distinguishing mislabeled data from correctly labeled data in classifier design", In Proceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'04), pages 668 −672, 2004. D. Guan, W. Yuan, Y.-K.Lee, and S. Lee, "Identifying mislabeled training data with the aid of unlabeleddata", Applied Intelligence, 35 (3): 345-358, 2010.

  In the techniques of Non-Patent Documents 1 and 2, a cross-validation method is applied to learn a statistical classifier from training data excluding samples that are targets of misclassification detection. However, since training data contains misclassified samples, there is no guarantee that a statistical classifier with high categorization performance can be obtained. Therefore, for many samples that are classified into the correct category, there is a risk that the statistical classifier misdetects the misclassified sample by misclassifying the category. On the other hand, there is a risk that a misclassified sample is not detected because a manually given category matches a category classified by a statistical classifier. Thus, when the reliability of misclassification detection by each statistical classifier is low, it is not expected to obtain a high detection accuracy of misclassified samples even if the final determination is made by majority decision of a plurality of statistical classifiers.

  In the technique of Non-Patent Document 3, a sample of content (hereinafter referred to as “unlabeled sample”) that is the same type of content as the sample indicated by the misclassification detection target and whose category belongs to is used for learning. Thus, the performance of the statistical classifier used to detect misclassified samples is improved. In this technique, first, a plurality of statistical classifiers are trained using samples with known categories to which content belongs (hereinafter referred to as “labeled samples”) including misclassified samples as training data. A classifier is used to predict the category to which each unlabeled sample belongs. Then, a set of unlabeled samples in which the prediction categories of all statistical classifiers match and the prediction categories are added to the training data as new labeled samples, and the statistical classifier is re-learned. This technique reduces the proportion of misclassified samples that were originally included in the training data by creating labeled samples from unlabeled samples and adding them to the training data. We expect that adverse effects on classifier learning will be suppressed.

  However, statistical classifiers trained with training data containing misclassified samples may not be highly capable of predicting the category, so there is a risk of giving incorrect category predictions for many unlabeled samples . If a large amount of unlabeled samples for which incorrect category prediction is performed due to the adverse effects of the misclassified samples that were originally included are added to the training data, the performance of the statistical classifier cannot be expected to improve even after re-learning.

  As described above, the conventional technique has a problem that a misclassification sample included in training data adversely affects learning of a statistical classifier used for misclassification detection.

  The present invention has been made in view of the above circumstances, and suppresses adverse effects caused by misclassified samples in the learning of a statistical classifier used for misclassification detection, and misclassified samples with high accuracy. An object of the present invention is to provide a misclassification detection apparatus, method, and program capable of being detected.

  In order to achieve the above object, the misclassification detection apparatus of the present invention includes a first sub-sample set that is a part of a sample set with a label to which the category to which the content belongs and an unlabeled sample with an unknown category to which the content belongs. A probability model generating means for generating a conditional class probability model indicating a probability that the content belongs to each category, wherein the generation model learned using both the set and the identification model are integrated, and the probability Based on the conditional class probability model generated by the model generation means, each labeled sample included in the second subsample set obtained by removing the first subsample set from the labeled sample set is classified into an incorrect category. Misclassification determining means for determining whether the content is a sample of the content.

  According to the misclassification detection apparatus of the present invention, the probability model generation means includes a first subsample set that is a part of a sample set with a label to which the category to which the content belongs and an unlabeled sample with an unknown category to which the content belongs. A conditional class probability model is generated in which a generation model learned using both sets and an identification model are integrated, and the probability that content belongs to each category is generated. The misclassification determining means, based on the conditional class probability model generated by the probability model generating means, each labeled sample included in the second subsample set excluding the first subsample set from the labeled sample set, It is determined whether or not it is a sample of content classified into an incorrect category.

  Thus, based on the conditional class probability model in which the generation model and the identification model learned using both the first sub-sample set and the unlabeled sample set that are part of the labeled sample set are integrated. In order to determine whether the labeled sample included in the second subsample set, which is the remaining part of the labeled sample set, is misclassified, it was misclassified in the learning of the statistical classifier used for misclassification detection It is possible to detect the misclassified sample with high accuracy by suppressing the adverse effect of the sample.

  In addition, the misclassification determining means has a category to which the content of each labeled sample included in the second subsample set belongs, and the probability obtained from the conditional class probability model for the labeled sample content is maximized. If the category does not match, the probability based on the conditional class probability model of the category to which the content of each labeled sample included in the second subsample set belongs is less than a predetermined first threshold, or the first The conditional class probability of the category to which the content of the labeled sample belongs with respect to the maximum value of the probability based on the conditional class probability model of each category other than the category to which the content of each labeled sample included in the two subsample sets belongs The probability ratio based on the model is a predetermined second If it is less than the value, it can be determined that the misclassified samples the labels located sample.

  The misclassification detection method of the present invention is a misclassification detection method in a misclassification detection apparatus including a probability model generation means and a misclassification determination means, wherein the probability model generation means knows the category to which the content belongs. This is a model in which the generation model and the identification model learned using both the first sub-sample set that is a part of the labeled sample set and the unlabeled sample set whose content category is unknown are integrated. Generating a conditional class probability model indicating the probability that the content belongs to each category, and the misclassification determining means is configured to generate the labeled sample set based on the conditional class probability model generated by the probability model generating means. Each labeled sample included in the second subsample set excluding the first subsample set from Is a method of determining whether a sample or not the content that is classified as Li.

  Moreover, the misclassification detection program of this invention is a program for functioning a computer as each means which comprises said misclassification detection apparatus.

  As described above, according to the misclassification detection apparatus, method, and program of the present invention, learning is performed using both the first sub-sample set and the unlabeled sample set that are part of the labeled sample set. Based on a conditional class probability model in which the generation model and the identification model are integrated, to determine whether or not the labeled sample included in the second subsample set that is the remaining part of the labeled sample set is misclassified Thus, it is possible to suppress the adverse effect of the misclassified sample on the learning of the statistical classifier used for misclassification detection and to detect the misclassified sample with high accuracy.

It is the schematic which shows the structure of the misclassification detection apparatus which concerns on this Embodiment. It is a flowchart which shows the content of the misclassification detection process routine in the misclassification detection apparatus which concerns on this Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, content consisting of text information such as papers, patents, online news data, e-mail, etc. included in the database, content consisting of text information and link information such as Web data, blog data, or images This is a misclassification detection device that detects misclassified samples from a set of samples in which content that can be expressed by feature vectors such as data content is classified into categories representing categories such as sports, music, and mathematics. A case where the invention is applied will be described.

<System configuration>
In the misclassification detection apparatus 10 according to the present embodiment, a set of content samples to which a category label belongs is input, and an incorrect category label is assigned from the input sample set. Detect and output misclassified samples. The misclassification detection apparatus 10 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a misclassification detection processing routine described later. As shown in FIG. 1, this computer can be functionally represented by a configuration including a probability model generation unit 12, a parameter storage unit 14, and a misclassification determination unit 16.

Misclassification detector 10 classifies detected sample set D erroneously as an input, and accepts no sample set D _u label.

The misclassification detection target sample set D is a set of content samples to which the category label is assigned, and is a set of samples that are targets of misclassification detection by this apparatus. Words or pixels, link or T = the feature space composed of combinations thereof, and the like, in the content _{{t 1, ..., t i} , ..., t V} when the characteristic of the content The vector x is expressed by x = {x ₁ ,..., X _i ,..., X _V ) ^T using the frequency x _{i of} t _i included in the content. V represents the number of types of features that may be included in the content. For example, when the content is text data, V represents the total number of vocabularies that can appear in the content. Each misclassification detection target sample included in the misclassification detection target sample set includes a content feature vector x and a label y of a category to which the content belongs. The misclassification detection target sample set D received here is D = {(x _n , y _n )} _{n = 1} ^N. n is misclassified detection target sample ID number included in the classification detection target sample set false, the total number of classification detection target sample N is erroneous, x _n is the feature vector of the n-th misclassified detected samples, y _n is the n th Represents the label of the category assigned to the false detection target sample, and yε {1,..., K,. K is the total number of classes in the category.

Each unlabeled sample included in the unlabeled sample set _Du is composed of only the content feature vector x. Assume that the unlabeled sample set D _{u received} here is D _u = {(x _m )} _{m = 1} ^M. m is unlabeled sample ID number included without sample set labels, M is the total number of samples unlabeled, is x _m represents the feature vector of m-th unlabeled sample.

Probabilistic model generation unit 12 uses the misclassified detection target sample set D and unlabeled sample set D _u, the conditional class probability model showing the probability of contents belonging to each category, mis of each misclassified detection target sample classification It is generated as a statistical classifier used for judgment.

Specifically, the probability model generation unit 12 divides the misclassification detection target sample set D into two subsample sets D ₁ and D ₂ , and one subsample set D ₁ and an unlabeled sample set D _u. Are used to calculate the parameter estimate {circumflex over (の)} of the conditional class probability model P (y | x; Ψ). The probability model generation unit 12 stores the calculated parameter estimated value ^ Ψ in the parameter storage unit 14.

The parameter Ψ is described in Non-Patent Document 4 (Akinori Fujino, Nobuyoshi Ueda, Masaaki Nagata, “Semi-supervised learning that is robust to selection bias of labeled data”, IPSJ Journal, Mathematical Modeling and Application (TOM), 4 (2), 31-42 (2011)), the parameters W, Θ, β of the statistical classifier are abbreviated as Ψ = [W, Θ, β]. The statistical classifier described in Non-Patent Document 4 directly uses a generation model designed according to the type of content in order to use unlabeled samples for learning, and classification boundaries of training data (labeled samples) directly. both the identification model to be learned, a statistical classifier integrated by learning using both the sub-sample set D ₁ and unlabeled sample set D _u. This suppresses overfitting of the statistical classifier to the misclassified samples included in the subsample set D ₁ , thereby adversely affecting the performance of the statistical classifier by the misclassified samples included in the subsample set D _1. Can be suppressed. Where W is the parameter of the identification model that constitutes the statistical classifier, Θ is the parameter of the generation model that constitutes the statistical classifier, and β is the weight for the generation model in the integration of the identification model and the generation model. .

Misclassification determination unit 16, the stored parameter estimates in the parameter storage unit 14 ^ with [psi, the misclassified detection target sample contained in a sub-sample set D ₂ which has not been used in the calculation of the parameter estimates ^ [psi The conditional class probability P (y | x _s ; ^ Ψ) of (x _s , y _s ) is calculated to determine whether each misclassification detection target sample is a misclassification sample.

For example, as shown in the following equation (1), if the category that gives the maximum probability P (y | x _s ; ^ Ψ) is the prediction category ^ y _s and y _s ≠ ^ y _s , the misclassification The detection target sample (x _s , y _s ) is determined to be a misclassified sample.

Further, the misclassification determination unit 16 sets a certain threshold value a (0 <a <1), and the misclassification detection target sample (x _s , y _s ) satisfying P (y _s | x _s ; ^ Ψ) <a. ) May be determined to be misclassified samples.

In addition, the misclassification determination unit 16 may determine that a misclassification detection target sample (x _s , y _s ) satisfying the following equation (2) is a misclassification sample by setting a certain real threshold value b. .

  The misclassification determination unit 16 determines whether or not all misclassification detection target samples included in the misclassification detection target sample set D are misclassification samples, and the misclassification detection target samples determined as misclassification samples. Is output as a misclassification detection result.

<Operation of misclassification detection device>
Next, the operation of the misclassification detection apparatus 10 according to the present embodiment will be described. When the misclassification detection target sample set D and the unlabeled sample set _Du are input to the misclassification detection apparatus 10, the misclassification detection apparatus 10 executes the misclassification detection processing routine shown in FIG.

First, in step 100, the probability model generation unit 12, the misclassified detection target sample set D accepted, divides the two to sub sample sets D ₁ and D _2.

Next, in step 102, the probability model generation unit 12 uses both the _one subsample set D1 divided in step 100 and the received unlabeled sample set _Du to generate a generation model and an identification model. The parameter estimation value {circumflex over (Ψ)} of the conditional class probability model P (y | x; Ψ), which is a statistical classifier integrated with In addition, the probability model generation unit 12 stores the calculated parameter estimation value Ψ in the parameter storage unit 14.

Next, in step 104, the misclassification determination unit 16 uses each parameter estimation value ^ Ψ stored in the parameter storage unit 14 to include each of the other subsample sets D ₂ divided in step 100. misclassified detection target sample _(x s, _{y s)} conditional class probability P for each category of _{(y | x s; ^ Ψ} ) is calculated, and each misclassification detection target sample _(x s, _{y s)} is false Determine whether it is a classification sample. Misclassification determination unit 16, a determination is made as to whether or classification sample or not false for all misclassified detection target sample contained in a sub-sample set D _2, misclassified samples determined to be the misclassified detection target sample ID The number is stored in a predetermined storage area.

Next, in step 106, it is determined whether the misclassification determination unit 16 has performed the above-described determination processing regarding whether or not all misclassification detection target samples included in the misclassification detection target sample set D are misclassification samples. judge. If there is an unprocessed misclassification detection target sample, the process returns to step 100 and the processes of steps 100 to 106 are repeated. In this case, among the misclassified detected sample set D, select the sub-sample set D ₂ from the misclassified detection target sample that has not been yet misclassification determination, sub-sample set the remaining misclassified detected sample set the process is repeated as D _1. Incidentally, the misclassification determination result obtained in the previous iteration does not affect the next iteration, the Kumigoto of each sub-sample set D ₁ and D _2, and the may be performed classification detection processing erroneous in parallel .

  On the other hand, when the misclassification determination process is completed for all misclassification detection target samples, the process proceeds to step 108, and the ID number of the misclassification sample stored in the predetermined area in step 104 is output as the misclassification detection result. Then, the misclassification detection processing routine ends.

  As described above, according to the misclassification detection apparatus according to the present embodiment, the statistical classifier (conditional class probability) that performs misclassification determination of each misclassification detection target sample included in the misclassification detection target sample set. In order to obtain a model, a statistical classifier is trained using a misclassification detection target sample set (a sample set with a label) and an unlabeled sample set excluding a misclassification detection target sample to be misclassified. This statistical classifier is a combination of a generation model and an identification model learned using both labeled and unlabeled sample sets, and misclassification that makes misclassification judgments with unlabeled sample sets. A learning technique adapted to both the misclassification detection target sample set excluding the detection target samples is used.

  For this reason, it is possible to detect the misclassified sample with high accuracy by suppressing the adverse effect of the misclassified sample in the learning of the statistical classifier used for misclassification detection.

<Experimental example>
Next, the results of experiments performed by applying the method according to the above embodiment will be described.

  An evaluation experiment was performed to detect document data classified into an incorrect subcategory in the problem of classifying document data belonging to a computer as a higher category into one of five subcategories. 20 newsgroups frequently used for performance evaluation in text classification problems (Non-Patent Document 5 “K. Nigam, J. Lafferty, and A. McCallum,“ Using maximum entropy for text classification ”, In IJCAI-99 Workshop on Machine Learning for Information Filtering, 61-67, 1999 ”).

In order to create an evaluation data set, 1000 samples were randomly extracted from document data belonging to five subcategories. A sample of r _m % was randomly selected from 1000 samples, and a miscategorized sample was created by randomly changing the subcategory to which the document data belongs to one of the other four subcategories. Further, in order to create an unlabeled sample set, 2000 samples different from the sample selected above were randomly extracted from the document data. A data set including misclassified samples obtained by this operation was used as a misclassification detection target sample set for performance evaluation. As a scale for performance evaluation, an F value often used in an information retrieval task or the like was used. The F value is calculated from the number N _c of misclassified samples detected by the device, the number N _{r of} samples detected by the device, and the number N _n of misclassified samples included in the misclassification detection target sample set as follows: 3) Calculated by the equation. The F value indicates that the larger the value, the higher the detection performance of misclassified samples.

  Table 1 shows a misclassification detection apparatus (method 1) according to the present embodiment and TSVM (Non-Patent Document 6 “R. Collobert, F. Sinz”, which is a representative statistical classifier that uses unlabeled samples for learning. , J. Weston, and L. Bottou, "Large scale transductive SVMs", Journal of Machine Learning Research, 7: 1687-1712, 2006.)) and 5-way cross validation (method 2) An example of the experimental result of the obtained F value is shown. In addition, in the unlabeled data utilization method described in Non-Patent Document 3, the statistical classification that gives the maximum F value when three statistical classifiers of the naive Bayes model, the 1-neighbor method, and the maximum entropy model are applied. An example of an experimental result of a method using a vessel (method 3) is shown. In Non-Patent Document 3, after learning a plurality of statistical classifiers using unlabeled data, misclassification determination is performed by majority decision based on prediction results of these statistical classifiers or unanimous agreement. However, in the method 3 of this experiment, in order to compare the superiority or inferiority of the usage of unlabeled data when learning a statistical classifier, before performing a decision by majority decision or unanimous agreement by a plurality of learned statistical classifiers. The results are shown. Methods 1 and 2 can be used as one of a plurality of statistical classifiers when configuring a misclassification detection apparatus that performs determination by majority vote or unanimous decision.

From Table 1, in the case of all experiments performed with different values of r _m, the F value obtained by the method 1, was equal to or higher than the F value obtained by the method 2, and 3. From the above results, it can be seen that the misclassification detection apparatus according to the present invention is effective in detecting misclassified samples.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Misclassification detection apparatus 12 Probability model production | generation part 14 Parameter storage part 16 Misclassification determination part

Claims (4)

A generation model and an identification model learned using both the first sub-sample set in which the category to which the content belongs is a part of the sample set with a label and the unlabeled sample set to which the category to which the content belongs is unknown Probability model generation means for generating a conditional class probability model that is an integrated model and indicates the probability that content belongs to each category;
Based on the conditional class probability model generated by the probability model generation means, each labeled sample included in the second subsample set obtained by removing the first subsample set from the labeled sample set is an incorrect category. Misclassification judging means for judging whether or not the content is classified as a sample,
A misclassification detection device including: The misclassification determining means includes
When the category to which the content of each labeled sample included in the second subsample set belongs does not match the category having the maximum probability obtained from the conditional class probability model for the labeled sample content,
The probability based on the conditional class probability model of the category to which the content of each labeled sample included in the second subsample set belongs is less than a predetermined first threshold, or included in the second subsample set The ratio of the probability based on the conditional class probability model of the category to which the labeled sample content belongs to the maximum probability based on the conditional class probability model of each category other than the category to which the labeled sample content belongs. , If less than a predetermined second threshold,
The misclassification detection apparatus according to claim 1, wherein the labeled sample is determined as a misclassification sample. A misclassification detection method in a misclassification detection device including a probability model generation means and a misclassification determination means,
The probability model generation means is learned using both the first sub-sample set in which the category to which the content belongs is a part of the sample set with a label and the unlabeled sample set to which the category to which the content belongs is unknown. A model in which the generation model and the identification model are integrated, and a conditional class probability model indicating the probability that the content belongs to each category is generated,
Each label included in the second subsample set obtained by removing the first subsample set from the labeled sample set based on the conditional class probability model generated by the probability model generating unit A misclassification detection method that determines whether or not a sample is a sample of content classified into an incorrect category.   The misclassification detection program for functioning a computer as each means which comprises the misclassification detection apparatus of Claim 1 or Claim 2.

Images