JP2019148566A - Teacher data creation support method and device - Google Patents

Abstract

To provide a technique for efficiently extracting data subjected to teaching in which there is possibility of having a teaching error, out of a plurality of pieces of data subjected to teaching in which one class is taught in advance.SOLUTION: A teacher data creation support method comprises: a step (step S22) for constructing a group of sub data subjected to teaching formed of data subjected to teaching, selected by one or more pieces from each of a plurality of classes in a group of data 90 subjected to teaching; a step (step S23) for setting the group of teh sub data subjected to teaching to teacher data for generating a sub sorter; and a step (step S24) for acquiring a classification achievement (correct answer ratio, for example) of the sub sorter by a re-substitution method. The method comprises a step S26 for selecting a sub sorter having the best classification achievement out of a prescribed number of sub sorters obtained by repeating the steps S22-S24, and a step S27 for extracting incompatible data subjected to teaching whose classification destination class does not match a teaching class, when the sub sorter classifies the group of data 90 subjected to teaching.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a technique for supporting creation of teacher data used for learning of a classifier.

  In manufacturing a semiconductor substrate, a glass substrate, a printed circuit board, and the like, an appearance inspection is performed using an optical microscope, a scanning electron microscope, or the like in order to inspect defects such as foreign matters, scratches, and etching defects. Conventionally, the cause of the defect has been identified by performing further detailed analysis on the defect detected in such an inspection process, and countermeasures against the defect have been taken.

  In recent years, as the pattern on the substrate becomes more complex and finer, the types and quantities of detected defects tend to increase, and automatic classification has been proposed to automatically classify defects detected in the inspection process. Yes. It is possible to quickly and efficiently analyze defects by automatic classification, and it is possible to preferentially take measures by paying attention to the types of defects that occur frequently.

  In automatic classification, a classifier using a neural network, a decision tree, discriminant analysis, or the like is used when classifying defects. In order for the classifier to perform automatic classification, it is necessary to prepare the teacher data including a signal indicating a defect image (or a feature amount of the defect image) and a category that is the type of the defect image, and learn the classifier.

  In Patent Document 1, an operator observes a teaching defect image displayed on a monitor, and creates teacher data by adding a corresponding defect category to a teaching defect image from a list of defect categories. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for creating high-quality teacher data by determining whether a category (teaching class) given in advance to a defect image is a category to which the defect image belongs. Specifically, a feature amount range including a representative value of the feature amount is set for the type based on the distribution of the feature amount of each type of the plurality of defect images belonging to each category, A process of voting for a category that includes a type of feature quantity in the type of feature quantity range is performed for all types of feature quantities. Then, when the category with the largest number of votes out of a plurality of categories is different from the category to which the defect image belongs, this is output.

JP 2010-91401 A

  However, the determination of a category by statistical processing in Patent Document 1 may be considered that the distribution of each type of feature amount of a plurality of defect images (teacher images) belonging to each category follows a normal distribution (or follows a normal distribution). If it does not follow the normal distribution, such as the distribution of feature values is multimodal, it is appropriate to detect mistakes in the pre-assigned category (class) (teaching errors). There was a problem that difficult support was difficult.

  Therefore, an object of the present invention is to provide a technique for efficiently extracting taught data with a possibility of teaching error from a plurality of taught data in which classes are taught in advance.

  In order to solve the above-mentioned problem, a first aspect is a teacher data creation support method for supporting creation of teacher data used for generating a classifier that classifies data based on its feature value, and (a1) a plurality of Preparing a taught data group composed of a plurality of taught data in which any one of the classes is taught as a teaching class; and (b1) one or more of each of the plurality of classes in the taught data group (C1) generating a sub-classifier that classifies the taught data using the sub-taught data group as teacher data. (D1) obtaining a classification result of the sub-classifier by classifying all or part of the taught data group with the sub-classifier generated by the step (c1); e1) (b1), classification results when all or part of the taught data group is classified from among the plurality of sub-classifiers obtained by repeating the step (c1) and the step (d1) multiple times Selecting a sub-classifier satisfying a predetermined classification performance criterion as a typical classifier, and (f1) the typical classifier selected in the step (e1), wherein all or one of the taught data groups is selected. And extracting the non-conforming teaching data in which the class to be classified does not match the teaching class when the part is classified.

  The second mode is the teacher data creation support method according to the first mode, wherein the step (e1) selects one sub-classifier having the best classification result from the plurality of sub-classifiers. It is a process to select as a container.

  A third aspect is a teacher data creation support method that supports creation of teacher data used to generate a classifier that classifies data based on its feature value, and (a1) any one of a plurality of classes Preparing a taught data group consisting of a plurality of taught data taught as a teaching class, and (b1) the taught data selected one or more from each of the plurality of classes in the taught data group (C1) generating a sub-classifier that classifies the taught data using the sub-taught data group as teacher data, and (d1) Classifying the plurality of taught data by the sub classifier generated by the step (c1) to obtain a classification result of the sub classifier, and (e2) the step (b1), the step ( c1) and the step (d1) One or more sub-classifiers in which the classification result when all or part of the taught data group is classified from among the plurality of sub-classifiers obtained by repeating several times satisfies a predetermined classification result criterion And (f2-1) classifying all or part of the plurality of taught data groups in each of the one or more sub-classifiers selected in the step (e2), A step of generating a typical classifier using matching taught data whose classification destination class matches the teaching class as teacher data, and (f2-2) the typical classifier generated in the step (f2-1). A step of extracting non-conforming taught data whose classification destination class does not match the taught class when all or part of the taught data group is classified.

  A fourth aspect is the teacher data creation support method according to the third aspect, wherein the step (e2) includes at least two sub-classifiers satisfying a predetermined classification result criterion among the plurality of sub-classifiers. This is a process of selecting.

  A fifth aspect is the teacher data creation support method according to the third aspect, wherein the step (e2) selects one sub-classifier having the best classification result from the plurality of sub-classifiers. It is a process to select as a container.

  A sixth aspect is the teacher data creation support method according to any one of the first to fifth aspects, wherein the classification result is classified by the teaching class and the sub-classifier out of the total number of the taught data. This is the ratio of the total number of taught data that matches the previous class.

  A seventh aspect is a teacher data creation support device that supports creation of teacher data used to generate a classifier that classifies data based on its feature quantity, wherein any one of a plurality of classes is a teaching class A storage unit for storing a taught data group composed of a plurality of taught data taught as: and by selecting one or more of each of the plurality of classes in the taught data group, A sub-teached data group constructing unit to construct, a classifier generating unit for generating a sub-classifier that classifies data using the sub-taught data group as teacher data, and all of the taught data group in the sub-classifier Alternatively, a classification result acquisition unit that acquires a classification result of the sub-classifier when a part is classified, a sub-teached data group construction unit, the classifier generation unit, and the classification result acquisition A repetition control unit that obtains a classification result of each of the plurality of sub-classifiers generated from the plurality of sub-taught data groups, and a plurality of the sub-classifiers, When a sub-classifier that satisfies the classification performance criteria is selected as a typical classifier, and when part or all of the taught data group is classified by the typical classifier, the classification destination class does not match the taught class A data extraction unit for extracting the taught data.

  According to the teacher data creation support method of the first aspect, a plurality of taught data used for generating a sub-classifier that satisfies a predetermined classification result criterion is teacher data suitable for generating a classifier having a good classification result. It is believed that there is. For this reason, based on a typical classifier obtained from the plurality of taught data as teacher data, by extracting non-conforming teaching data whose classification destination class does not match the teaching class, taught data with a high possibility of teaching error Can be extracted efficiently. Moreover, since the sub classifier is used as a typical classifier as it is, it is not necessary to regenerate the classifier. For this reason, it is possible to reduce the amount of calculation processing related to classifier generation.

  According to the teacher data creation support method of the second aspect, since the sub classifier with the best classification result is the typical classifier, the quantity of the non-conforming taught data can be minimized.

  According to the teacher data creation support method of the third aspect, the matched taught data in which the sub-classifier satisfying the predetermined classification result criteria in the taught data group is classified into the same class as the teaching class is It can be teacher data suitable for generation. Therefore, by generating a typical classifier using the matched taught data as teacher data and classifying all or a part of the taught data group, it is possible to efficiently extract taught data with a high possibility of teaching error. .

  According to the teacher data creation support method of the fourth aspect, a typical classifier is generated using a plurality of taught data used for generating two or more sub-classifiers as teacher data. Based on this typical classifier, the non-conforming taught data is extracted, so that it is possible to efficiently extract the taught data having a high possibility of teaching mistake.

  According to the teacher data creation support method of the fifth aspect, since the typical classifier is generated based on the sub-classifier having the best classification result, the quantity of the non-conforming taught data can be minimized.

  According to the teacher data creation support method of the sixth aspect, the sub-classifier can be evaluated based on a general accuracy rate (Accuracy) as an index of the classification performance of the classifier.

  According to the teacher data creation support device of the seventh aspect, it is considered that the matched taught data in which the classification destination class matches the teaching class by the resubstitution method is suitable for generating a classifier having excellent classification results. For this reason, by extracting the non-conforming taught data based on the typical classifier that can obtain the conforming taught data as the teacher data, it is possible to efficiently extract the taught data having a high possibility of teaching mistake.

FIG. 1 is a diagram illustrating a schematic configuration of an image classification device 1 according to the embodiment. FIG. 2 is a diagram illustrating a flow of defect image classification by the image classification apparatus 1 according to the embodiment. FIG. 3 is a block diagram showing the configuration of the host computer 5. FIG. 4 is a block diagram showing a functional configuration of the host computer 5 for generating the classifier 422 of the inspection / classification apparatus 4. FIG. 5 is a block diagram showing a functional configuration of the teacher data creation support unit 61 of the host computer 5. FIG. 6 is a diagram showing the flow of the first teacher data creation support process. FIG. 7 is a diagram showing the flow of the second teacher data creation support process. FIG. 8 is a diagram illustrating a configuration of the classifier generation unit 615. FIG. 9 is a diagram illustrating a flow of sub-classifier generation by the classifier generation unit 615. FIG. 10 is a diagram illustrating a frequency distribution table of a plurality of classes on the first feature amount axis indicated by the frequency distribution data 82. FIG. 11 is a diagram illustrating a histogram for each class on the first feature amount axis indicated by the frequency distribution data 82. FIG. 12 is a diagram showing a frequency distribution table of a plurality of classes on the second feature amount axis indicated by the frequency distribution data 82. FIG. 13 is a diagram illustrating a histogram for each class on the second feature amount axis indicated by the frequency distribution data 82. FIG. 14 is a diagram illustrating an example of the classification result of the taught data 90 by the classifier 330. FIG. 15 is a diagram for explaining a modification example of the frequency distribution data 82. FIG. 16 is a diagram for explaining another modification example of the frequency distribution data 82. FIG. 17 is a graph showing the relationship between the number of taught data (the number of teacher data) and the correct answer rate of the classifier 330. FIG. 18 is another graph showing the relationship between the number of taught data and the correct answer rate of the classifier 330.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the component described in this embodiment is an illustration to the last, and is not a thing of the meaning which limits the scope of the present invention only to them. In the drawings, the size and number of each part may be exaggerated or simplified as necessary for easy understanding. Further, in this application, the expression “comprising”, “including”, or “having” one or more constituent elements is not an exclusive expression for excluding the presence of other constituent elements unless otherwise specified.

<1. First Embodiment>
FIG. 1 is a diagram illustrating a schematic configuration of an image classification device 1 according to the embodiment. In the image classification device 1, a defect image indicating a pattern defect on the semiconductor substrate 9 is acquired, and the defect image is classified. The image classification device 1 includes an imaging device 2, an inspection / classification device 4, and a host computer 5. In this embodiment, the case where the classification target is a defect image obtained by imaging a semiconductor substrate will be described as an example. However, the present invention is not limited to the defect image, and for example, obtained by imaging a cell. Cell images and the like may be classified.

  The imaging device 2 images the inspection target area on the semiconductor substrate 9. The inspection / classification device 4 performs defect inspection based on the image data acquired by the imaging device 2. When a defect is detected, the inspection / classification device 4 classifies the defect for each defect type (class). The class of pattern defects present on the semiconductor substrate 9 may include defects, protrusions, disconnections, shorts, foreign matter, and the like. The host computer 5 controls the overall operation of the image classification device 1 and generates a classifier 422 used for defect classification in the inspection / classification device 4.

  The imaging device 2 can be incorporated into a production line for the semiconductor substrate 9 and the image classification device 1 can be a so-called inline system. The image classification apparatus 1 is an apparatus in which a function of automatic defect classification is added to a defect inspection apparatus.

  The imaging device 2 includes an imaging unit 21, a stage 22, and a stage driving unit 23. The imaging unit 21 images the inspection area of the semiconductor substrate 9. The stage 22 holds the semiconductor substrate 9. The stage driving unit 23 moves the stage 22 relative to the imaging unit 21 in a direction parallel to the surface of the semiconductor substrate 9.

  The imaging unit 21 includes an illumination unit 211, an optical system 212, and an imaging device 213. The optical system 212 guides illumination light to the semiconductor substrate 9. The light reflected by the semiconductor substrate 9 enters the optical system 212 again. The imaging device 213 converts the image of the semiconductor substrate 9 formed by the optical system 212 into an electrical signal.

  The stage driving unit 23 includes a ball screw, a guide rail, a motor, and the like. The host computer 5 controls the stage drive unit 23 and the imaging unit 21 so that the inspection target area on the semiconductor substrate 9 is imaged.

  The inspection / classification apparatus 4 includes a defect detection unit 41 and an automatic classification unit 42. The defect detection unit 41 detects defects while processing the image data of the inspection target area. Specifically, the defect detection unit 41 has a dedicated electric circuit that processes image data of the inspection target area at high speed, and compares an image obtained by imaging with a reference image (an image without a defect) A defect inspection of the inspection target area is performed by image processing. The automatic classification unit 42 classifies the defect images detected by the defect detection unit 41. Specifically, the automatic classification unit 42 includes a CPU that performs various arithmetic processes, a memory that stores various information, and the like. The automatic classification unit 42 includes a feature amount calculation unit 421 and a classifier 422. The classifier 422 executes defect classification, that is, defect image classification, using a neural network, a decision tree, discriminant analysis, or the like.

  FIG. 2 is a diagram illustrating a flow of defect image classification by the image classification apparatus 1 according to the embodiment. First, when the imaging apparatus 2 shown in FIG. 1 images the semiconductor substrate 9, the defect detection unit 41 of the inspection / classification apparatus 4 acquires image data (step S11).

  Subsequently, the defect detection unit 41 detects a defect by performing a defect inspection on the inspection target region (step S12). If a defect is detected in step S12 (YES in step S12), the image of the defective portion (that is, the defect image) is transmitted to the automatic classification unit 42. If no defect is detected (NO in step S12), image data acquisition in step S11 is performed.

  When receiving the defect image, the automatic classification unit 42 calculates a feature amount that is an array of a plurality of types of feature amounts of the defect image (step S13). The calculated feature amount is input to the classifier 422, and classification is performed by the classifier 422 (step S14). That is, the classifier 422 classifies the defect image into one of a plurality of classes. In the image classification device 1, every time a defect is detected by the defect detection unit 41, the feature amount is calculated in real time, and a large number of defect images are automatically classified at high speed.

  Next, learning of the classifier 422 by the host computer 5 will be described. FIG. 3 is a block diagram showing the configuration of the host computer 5.

  The host computer 5 has a CPU 51, ROM 52 and RAM 53. The CPU 51 includes an arithmetic circuit that performs various arithmetic processes. The ROM 52 stores a basic program. The RAM 53 is a volatile main storage device that stores various information. The host computer 5 has a general computer system configuration in which a CPU 51, a ROM 52, and a RAM 53 are connected by a bus line 501.

  The host computer 5 includes a fixed disk 54, a display 55, an input unit 56, a reading device 57, and a communication unit 58. These elements are connected to the bus line 501 through an interface (I / F) as appropriate.

  The fixed disk 54 is an auxiliary storage device that stores information. The display 55 is a display unit that displays various types of information such as images. The input unit 56 is an input device including a keyboard 56a and a mouse 56b. The reading device 57 reads information from a computer-readable recording medium 8 such as an optical disk, a magnetic disk, or a magneto-optical disk. The communication unit 58 transmits and receives signals to and from other elements of the image classification device 1.

  The host computer 5 reads the program 80 from the recording medium 8 via the reading device 57 and records it on the fixed disk 54. The program 80 is copied to the RAM 53. The CPU 51 executes arithmetic processing according to the program 80 stored in the RAM 53.

  FIG. 4 is a block diagram showing a functional configuration of the host computer 5 for generating the classifier 422 of the inspection / classification apparatus 4. The functions of the host computer 5 are realized by the CPU 51, the ROM 52, the RAM 53, the fixed disk 54, and the like of the host computer 5. In FIG. 4, the inspection / classification apparatus 4 is also shown. The host computer 5 includes a teacher data creation support unit 61 and a learning unit 63. The teacher data creation support unit 61 creates teacher data used for learning of the classifier. The learning unit 63 learns the classifier using the teacher data.

  The teacher data includes teacher image data that is a defect image, a feature value of the teacher image, and a teaching class that is information indicating a defect class. As the feature amount of the teacher image, for example, the area of the defect, the brightness average, the peripheral length, the flatness, the inclination of the long axis when the defect is approximated to an ellipse, and the like can be adopted.

  In the learning unit 63, the feature value of the teacher image read from the teacher data is input to a classifier (not shown) in the host computer 5. Then, learning is performed so that the output of the classifier becomes the same as the teaching class, and the learning result, that is, the classifier 422 after learning (more precisely, information indicating the structure of the classifier 422 and the value of the variable) is obtained. It is transferred to the automatic classification unit 42. Thus, the classifier 422 is generated using the teacher data. Generation of a classifier means generation of a classifier by assigning a value to a parameter included in the classifier or determining a structure.

  FIG. 5 is a block diagram showing a functional configuration of the teacher data creation support unit 61 of the host computer 5. The teacher data creation support unit 61 includes a data calculation unit 610, a display 55, and an input unit 56. The data calculation unit 610 includes a storage unit 611, a sub-taught data group construction unit 613, a classifier generation unit 615, a classification result acquisition unit 617, a repetition control unit 618, a data extraction unit 619, and a display control unit 620. Details of the processing of the data calculation unit 610 will be described later. It should be noted that the function of the data operation unit 610 (and the learning unit 63) may be constructed by a dedicated electric circuit or may be partially used.

  FIG. 6 is a diagram showing the flow of the first teacher data creation support process. Each operation described below is executed by the data calculation unit 610 unless otherwise specified.

  In the first teacher data creation support process, first, a plurality of taught data 90 are prepared (step S21). Hereinafter, the plurality of taught data 90 are also referred to as “taught data 90 group”. The taught data 90 is data indicating a defect image in which one of a plurality of defect classes is previously taught as a teaching class. The taught data 90 group may include taught data 90 in which the wrong class is taught. In the teacher data creation support process executed by the teacher data creation support unit 61, high-quality teacher data is generated by efficiently finding the taught data 90 of teaching errors and teaching the correct class. The quantity of the taught data 90 included in the taught data 90 group is not particularly limited, but is assumed to be about 1000 to 100,000, for example. Moreover, the teaching class of each taught data 90 may be taught based on the classification result of an arbitrary classifier, or the operator taught based on the operator visually confirming each defect image. It may be a thing. The prepared taught data 90 group is stored in the storage unit 611.

  After the taught data 90 group is prepared, the sub-taught data group construction unit 613 constructs a sub-taught data group (step S22). The sub-teached data group is a set of taught data 90 that is randomly selected from one or more equal numbers of each of a plurality of classes among all the taught data 90 stored in the storage unit 611.

  When the sub-taught data group is constructed, the classifier generation unit 615 generates a sub classifier (step S23). Specifically, the classifier generation unit 615 generates a sub classifier that classifies data by machine learning using a plurality of taught data 90 belonging to the sub taught data group. The sub classifier classifies the defect class into any one of a plurality of defect classes based on the feature amount of the defect image of the taught data 90.

  Specifically, in the classifier generation unit 615, the feature amount of the defect image read from the taught data 90 is input to a classifier (not shown) in the host computer 5. Then, learning is performed so that the output of the classifier coincides with the teaching class, and the learning result, that is, the sub-classifier after learning (more precisely, information indicating the structure of the sub-classifier and the value of the variable) Is acquired. In this way, a sub classifier is generated using the taught data 90 of the sub taught data group. A detailed example of the construction method of the sub classifier will be described later.

  When the sub classifier is generated, the classification result acquisition unit 617 acquires the classification result of the sub classifier (step S24). Specifically, the classification result when all or part of the taught data 90 group stored in the storage unit 611 (including taught data 90 other than the sub-taught data group) is classified by the sub-classifier. To get. As an index of classification results, for example, a correct answer rate (Accuracy) can be adopted. The correct answer rate is a ratio of the total number of taught data 90 in which the class classified by the sub classifier and the taught class match among the total number of taught data 90 classified by the sub classifier.

  In addition, as an index of the classification result, a recall (Recall) or a precision (Precision) may be adopted instead of the correct answer rate. The recall is the ratio of the taught data 90 correctly classified into the teaching class by the sub-classifier out of the taught data 90 of the specific teaching class. The relevance ratio is a ratio of taught data 90 in which the teaching class matches the specific class out of the taught data 90 classified into a specific class by the sub classifier.

  When the classification result of the sub classifier is acquired, the repetition control unit 618 determines whether a predetermined number of sub classifiers have been generated (step S25). When the predetermined number of sub classifiers has not been generated (NO in step S25), repetition control unit 618 controls sub-taught data group construction unit 613, classifier generation unit 615, and classification result acquisition unit 617. Then, step S22 to step S24 are executed again. Thus, in this example, steps S22 to S24 are repeatedly performed until a predetermined number of sub-classifiers are generated.

  When iterative control unit 618 determines that a predetermined number of sub-classifiers have been generated (YES in step S25), data operation unit 610 selects the sub-class with the best classification result from the predetermined number of sub-classifiers. A classifier is selected (step S26). Specifically, when the “correct answer rate” is adopted as an index of classification results, the sub classifier having the highest correct answer rate is selected. In step S26, it is not essential to select a sub-classifier having the best classification result. One sub-classifier that satisfies a predetermined classification performance criterion (for example, “the correct answer rate exceeds a predetermined threshold”, etc.) may be selected.

  After one sub classifier is selected, the data extraction unit 619 sets the one sub classifier as a typical classifier. Then, the data extraction unit 619 uses the typical classifier to classify all or part of the taught data 90 group, and the taught data 90 (hereinafter referred to as “nonconforming”) whose classification destination class does not match the original teaching class. (Also referred to as “taught data”) is extracted (step S27). In this example, the classification result by the typical classification is obtained when the classification result of the sub classifier corresponding to the typical classifier is acquired in step S24. For this reason, in this example, it is not essential to perform classification again in step S27.

  After the non-conforming taught data is extracted, the display control unit 620 displays a defect image of the non-conforming taught data on the display 55 (step S28). Thereby, the taught data 90 with the possibility of teaching mistake is presented to the operator. At this time, the original teaching class and the classification destination class may be displayed together with the defect image for the non-conforming teaching completed data. As the taught data 90 is displayed, the data calculation unit 610 may function as a re-teaching accepting unit that accepts teaching of a new class. Specifically, the operator visually checks each taught data 90 to determine the validity of the teaching class. When the operator determines that class re-teaching is necessary, an input for designating a class is performed via the input unit 56. The data calculation unit 610 may correct the teaching class of the taught data 90 stored in the storage unit 611 by receiving the input.

  According to the first teacher data creation support process, a sub classifier satisfying a predetermined classification result criterion is selected as a typical classifier, and a class to be classified by the typical classifier is selected from the taught data 90 group as a teaching class. Incompatible teaching data that does not match is extracted. For this reason, by appropriately setting the standard of classification results, it is possible to efficiently extract taught data with a high possibility of teaching error. Moreover, since the sub classifier is used as a typical classifier as it is, it is not necessary to regenerate the classifier. For this reason, it is possible to reduce the amount of calculation processing related to classifier generation. In particular, the quantity of non-conforming taught data can be minimized by extracting non-conforming taught data based on the sub-classifier having the best classification result.

  Of the taught data 90 group, taught data 90 in which the classification class of the sub classifier does not match the teaching class (hereinafter also referred to as non-matched taught data) reduces the classification result of the sub classifier. . For this reason, the coincidence taught data (excluding the mismatched taught data from the taught data 90 group) can be teacher data suitable for generating a classifier with good classification results. Therefore, by using only the matched taught data among the taught data 90 group as the teacher data, a typical classifier having better classification results than the case where the entire taught data 90 group is used as the teacher data can be generated. . Therefore, based on the typical classifier, the taught data 90 with the possibility of teaching error can be effectively and efficiently extracted from the taught data 90 group.

  FIG. 7 is a diagram showing the flow of the second teacher data creation support process. Of the teacher data creation support process shown in FIG. 7, steps S21 to S25 are common to the teacher data creation support process shown in FIG.

  In the second teacher data creation support process, when the data operation unit 610 determines that a predetermined number of sub-classifiers have been generated in step S25 (YES in step S25), the data operation unit 610 determines that the predetermined number of sub-classifiers have been generated. A plurality of sub classifiers whose classification results satisfy a predetermined criterion are selected from the classifiers (step S261). As an index of classification results, for example, the correct answer rate can be adopted as in step S24 of the first teacher data creation support process. In this case, in step S261, a plurality of sub-classifiers whose correct answer rate exceeds a predetermined value may be selected. In addition, the recall rate or the matching rate for a specific class or all classes can be adopted instead of the correct answer rate.

  After a plurality of sub classifiers are selected, the classifier generation unit 615 generates a typical classifier that classifies data (step S262). Specifically, the classifier generating unit 615 classifies all or a part of the taught data 90 group in each of the plurality of sub-classifiers selected in step S261, and teaches that the classification destination class matches the teaching class. Collected data (hereinafter also referred to as “coincidence taught data”). Then, the classifier generation unit 615 generates a typical classifier by performing machine learning using the collected matched taught data as teacher data.

  After the typical classifier is generated, the data extracting unit 619 classifies all or part of the taught data 90 of the taught data 90 group stored in the storage unit 611 by the typical classifier. Then, the data extraction unit 619 extracts the taught data 90 in which the class to be classified by the typical classifier does not match the original teaching class from the plurality of classified taught data 90 as non-conforming taught data. (Step S27). Then, the display control unit 620 displays the nonconforming teaching data on the display 55 (step S28).

  The sub-taught data group used to generate a plurality of sub-classifiers that satisfy a predetermined classification performance criterion is considered to be a set of typical data suitable for generating a high-precision classifier. Therefore, a typical classifier having excellent classification results is generated by generating a typical classifier using each sub-taught data group of the taught data 90 used for generating the sub-classifier selected in step S261 as teacher data. Can be generated. Therefore, based on the generated typical classifier, by extracting the taught data 90 in which the class to be classified does not match the teaching class, data that may possibly cause a teaching error is efficiently presented to the operator. be able to.

  In the second teacher data creation support process, a typical classifier is generated based on a plurality of sub-classifiers in steps S261 and S262. However, a typical classifier may be generated based on one sub-classifier. For example, in step S261, the classifier generation unit 615 selects one sub classifier having the best classification result. In step S262, when the classifier generation unit 615 classifies all or part of the taught data 90 group by the one sub-classifier, the matching taught data whose classification destination class matches the teaching class is obtained. collect. Then, the classifier generation unit 615 may generate a typical classifier by performing machine learning using the collected matched taught data as teacher data.

<Classifier generation process>
As described above, the classifier generation unit 615 performs sub-classifier generation processing (FIG. 6: step S25) in the first and second teacher data support processing. The classifier generation unit 615 performs a typical classifier generation process (FIG. 7: step S262) in the second teacher data support process. Here, the classifier generation processing by the classifier generation unit 615 will be described. In the following, the generation process of the sub classifier will be mainly described, but the typical classifier can be generated in the same manner. FIG. 8 is a diagram illustrating a configuration of the classifier generation unit 615. FIG. 9 is a diagram illustrating a flow of sub-classifier generation by the classifier generation unit 615.

  As shown in FIG. 8, the classifier generation unit 615 includes a teacher data storage unit 6150, a frequency distribution data generation unit 6151, a classifier construction unit 6152, a frequency distribution data correction unit 6153, and a repetition control unit 6154.

  The teacher data storage unit 6150 stores teacher data (in this case, a plurality of taught data 90) for constructing a classifier. When the classifier generation unit 615 generates a sub classifier, the teacher data storage unit 6150 stores a plurality of taught data 90 belonging to the sub taught data group constructed by the sub taught data group construction unit 613. The data is stored (FIG. 9: Step S31).

  When each taught data 90 of the sub taught data group is prepared, the frequency distribution data generation unit 6151 generates frequency distribution data 82 indicating the frequency distribution for each feature amount axis based on the plurality of taught data 90. (FIG. 9: Step S32). The frequency distribution data 82 is data indicating a frequency distribution using each taught data 90 as a sample. For each feature amount axis, the frequency (appearance frequency) in each section obtained by discretizing the value of each feature amount axis is expressed as follows. This data is shown by class.

  Specifically, the frequency distribution data generation unit 6151 specifies a maximum value and a minimum value from the feature value values included in each taught data 90 for each feature value axis, thereby obtaining a value for each feature value axis. Get the distribution range. Then, the frequency distribution data generation unit 6151 equally divides (discretizes) the distribution range into an appropriate number of sections. The number of divisions of the distribution range may be, for example, 2 to the 1st power (ie, 2) or more and 2 to the 10th power or less. Then, the frequency (appearance frequency) for each class in each discretized section (discrete section) is obtained. Specifically, for each feature quantity axis, for each taught data 90, only one frequency of the corresponding discrete section corresponding to the feature quantity value of the taught data 90 is added.

  FIG. 10 is a diagram illustrating a frequency distribution table of a plurality of classes on the first feature amount axis indicated by the frequency distribution data 82. FIG. 11 is a diagram illustrating a histogram for each class on the first feature amount axis indicated by the frequency distribution data 82. FIG. 12 is a diagram showing a frequency distribution table of a plurality of classes on the second feature amount axis indicated by the frequency distribution data 82. FIG. 13 is a diagram illustrating a histogram for each class on the second feature amount axis indicated by the frequency distribution data 82.

  In FIG. 10 and FIG. 12, a plurality of (here, three types) classes are represented as “Class 1”, “Class 2”, and “Class 3”, respectively, and the section on the feature amount axis is 0 in the row representing “bin” in the heading. It is indicated by a number of ~ 15 (the same applies hereinafter).

  In FIG. 10 to FIG. 13, the number of divisions of the distribution range of the values of each feature amount axis is 14 (section (1) to section (14)), and the section (0) having a value smaller than the distribution range and a large value are included. Section (15) is also provided. The section (0) and the section (15) are used when the value indicated by the unknown data whose class is not taught is outside the distribution range based on the taught data 90. As will be described later, the frequency distribution by class for each feature amount axis is used for data classification (that is, class discrimination). Therefore, the frequency distribution data 82 can be said to be “discrimination information” for class discrimination.

  The number of divisions of the distribution range may be different for each feature amount axis, or may be the same for all feature amount axes. For example, among the number of divisions in which there is only one section having a frequency of 1 in all sections, the smallest one can be set as the upper limit of the number of divisions. Thereby, it is suppressed that the section where the frequency becomes 1 or more becomes discontinuous. In addition, in the feature amount space defined by the plurality of feature amount axes, the number of divisions is such that the number of regions (cells) expressed by the sections of the plurality of feature amount axes is sufficiently larger than the total number of teacher images. It is preferable that the lower limit of is set.

  The classifier construction unit 6152 generates an initial classifier 330 that classifies data (that is, classifies) based on the value of each feature quantity (FIG. 9: step S33). The initial classifier 330 is configured to classify data to be classified based on the appearance ratio of each class in each section indicated by the frequency distribution data 82.

  Here, the basic structure of the classifier 330 is determined in advance, and the classifier 330 includes a plurality of weak classifiers that perform computation for each of a plurality of feature amount axes. Each weak classifier refers to the feature value of the image to be classified, and the probability that the image from which the value was acquired belongs to each of a plurality of classes (only the feature value axis corresponding to each weak classifier Is obtained as a class evaluation result.

Here, the total number of each taught data 90 in the sub-teached data group is N, the number of classes is n, and the class C _i (i = 1, 2,..., N) belongs to (that is, the taught class is C _i. The total number of taught data 90 is N _i . _Ni is the same number for any feature amount axis. Therefore, the sum total of all classes of the total number N _i of the taught data 90 belonging to the class C _i is the total number N of each taught data 90 in the sub-taught already-trained data group as shown in Expression (1).

Further, the total number of feature amount axes is m, the number of divisions when the value of the feature amount axis Dj (j = 1, 2,..., M) is discretized is K _j, and the section k ( k = 1, 2, · · ·, when the number of teaching-data-90 belonging to the class _{C i} in _{K j)} expressed by F _ij (k), the total number _{N i} of the teaching-data-90 belonging to the class _{C i} is It is expressed as equation (2).

On the other hand, when attention is paid to only one feature amount axis D _j , the appearance ratio of the taught data 90 belonging to the class C _i in the section k (the total number of taught data 90 belonging to the class C _i of the number F _ij (k)) The ratio with respect to N _i ) can be considered as the probability that an image whose value of the feature amount axis D _j belongs to the section k belongs to the class C _i . Hereinafter, the probability is expressed as P _jk (C _i ). This probability P _jk (C _i ) is expressed as shown in Equation (3).

For example, when the probability P _jk (C _i ) is obtained for the ninth section (9) of the first feature amount axis (feature amount axis D ₁ ), for example, the probability P _{1,9 of} class 1 (C ₁ ) (C ₁ ) is 0.031 (= 49/1578). The probability P _1,9 (C ₂ ) of class 2 (C ₂ ) is 0.171 (= 486/2849). Further, the probability P _1,9 (C ₃ ) of class 3 (C ₃ ) is 0.013 (= 9/688). On one feature axis, the probability P _jk (C _i ) is obtained by n (number of classes), but the sum of the probabilities P _jk (C _i ) of all classes is not 1.

Each of the plurality of weak classifiers constituting the classifier 330 refers to the frequency distribution data 82 from the value of the corresponding feature amount axis D _j , thereby classifying the appearance ratio (probability P _jk (C _i )) for each class. As a result. In the weak classifier, it can be said that an evaluation value indicating validity (confidence) when an image from which the value of the feature amount axis D _j is acquired belongs to a specific class C _i is obtained as a class evaluation value. .

  By the way, in the frequency distribution by class on many feature amount axes, there are many overlapping portions as shown in the histograms of FIGS. 11 and 13. The prediction accuracy is not very high (although it can be said that the accuracy is higher than predicting the class randomly). Therefore, the classifier 330 adopts the concept of ensemble learning, and the classifier 330 is configured as a strong classifier that determines a classification class based on the class evaluation results of a plurality of weak classifiers for a plurality of feature amount axes. Is done.

  In the classifier 330, evaluation values (probabilities) of a plurality of classes are obtained as class evaluation results for each feature amount axis. For example, the class evaluation result is obtained by assigning 1 to the class having the maximum evaluation value, You may give 0 to a class. In this case, the process is to vote for the class having the largest evaluation value on each feature amount axis, and the class having the largest number of votes on the plurality of feature amount axes is determined as the classification class.

  As shown in FIG. 8, the frequency distribution data correction unit 6153 corrects the frequency distribution data 82 generated by the frequency distribution data generation unit 6151 and sent to the classifier construction unit 6152. Hereinafter, a flow of processing in which the frequency distribution data correction unit 6153 corrects the frequency distribution data 82 will be described. Specifically, the frequency distribution data correction unit 6153 classifies each taught data 90 of the sub-taught data group using the classifier 330 that performs data classification based on the frequency distribution data 82 to be corrected (FIG. 9). : Step S34).

In this processing example, the frequency distribution data correction unit 6153 uses the taught data 90 used when the initial classifier 330 is generated as the taught data 90 (that is, when the initial frequency distribution data 82 is generated). All are classified by the classifier 330. Note that the frequency distribution data correction unit 6153 may select only a part of all the taught data 90 and cause the classifier 330 to classify it. The classification of the teaching-data-90, identified the value of the characteristic quantity axis D _j of each teaching-data-90, class C _i different appearance ratio (probability P _jk (C _i)) is obtained as a class evaluation result .

  In the classification by the classifier 330, a representative value (for example, an average value, a median value, a weighted average value, etc.) of the appearance ratio of each feature amount axis is further obtained. Then, the class having the maximum representative value among all classes is determined as the class to be classified in the taught data 90. The classification class of each taught data 90 is stored in the frequency distribution data correction unit 6153. In the preferred processing by the classifier 330, (1) when the maximum representative value is less than the predetermined threshold SH1, or (2) the difference between the maximum representative value and the second largest representative value (or The additional class indicating that the class to be classified is unknown is determined as the classification class, for example, when the ratio is less than another predetermined threshold value SH2. In the following description, it is assumed that the teacher images of the teacher image group are classified into the additional class in the cases (1) and (2).

  FIG. 14 is a diagram illustrating an example of the classification result of the taught data 90 by the classifier 330. FIG. 14 is a confusion matrix (confusion matrix) that summarizes the classification results. In FIG. 14, three types of teaching classes are described in the row headings, and four types of classification classes including unknown are described in the column headings. Of each taught data 90 whose teaching class is “A”, the number of taught data 90 classified into class “B” is indicated in the cell where the row “A” and the column “B” intersect. It is. For example, among each taught data 90 whose teaching class is “Class 1”, the number of taught data 90 classified as “Class 2” is 143. In addition, a cell in which a “Precision” (accuracy) row and a “Recall” (recall) column intersect is classified by the classifier 330 out of the total number of taught data 90 classified by the classifier 330. The ratio of the total number of taught data 90 in which the class and the teaching class match (accuracy rate: Accuracy). When all the taught data 90 is a classification target, the classification result of FIG. 9 is a result of so-called resubstitution method evaluation.

  Subsequently, the frequency distribution data correcting unit 6153 has taught data 90 in which the classified class is different from the taught class among the taught data 90 (hereinafter, this data is also referred to as “mismatched taught data”). ) Exists, the frequency distribution data 82 is corrected based on each of the mismatched taught data (FIG. 9: step S35).

  In the correction of the frequency distribution data 82, by considering the mismatched taught data, for each feature amount axis, a section (corresponding section) corresponding to the feature value of the mismatched taught data is specified. The frequency of the teaching class of the mismatched taught data in the corresponding section is increased by a predetermined positive value (for example, “1”). That is, the frequency distribution data 82 that is counted twice is generated in the frequency distribution data 82 that indicates the frequency distribution for each class for each feature amount axis for the mismatched taught data whose classification destination class is different from the teaching class. It becomes. This process can also be regarded as a change in the weight of the mismatched taught data.

FIG. 15 is a diagram for explaining a modification example of the frequency distribution data 82. Here, it is assumed that the teaching class of the mismatched taught data is “Class 2”, and the corresponding section of the feature amount value of the feature amount axis D _j possessed by the mismatched taught data is the section k. Then, in this case, as shown in FIG. 15, the frequency distribution data correction unit 6153 increases the frequency (number F _2j (k)) of “Class 2” in the interval k by “1”. As a result, the appearance ratio (probability P _jk (C ₂ )) of “Class 2” in the interval k apparently increases. In this manner, the frequency distribution data correction unit 6153 corrects the frequency distribution data 82 by increasing the frequency of the corresponding section for all feature amount axes of the mismatched taught data.

  Subsequently, the classifier construction unit 6152 updates the classifier 330 so as to classify the data based on the corrected frequency distribution data 82 (FIG. 9: Step S36). In the new frequency distribution data 82, the appearance ratio of the corresponding section corresponding to each feature amount axis is increased for the mismatched taught data. For this reason, the updated classifier 330 has a higher probability of correctly classifying the mismatched taught data into the teaching class than before the update.

  Subsequently, the frequency distribution data correction unit 6153 uses the classifier 330 based on the corrected frequency distribution data 82 to classify each taught data 90 of the sub-taught data group (FIG. 9: Step S37). Here, all of the taught data 90 of the sub-taught data group prepared in step S31 may be targeted, or a part of them may be targeted.

  The repetition control unit 6154 causes the frequency distribution data correction unit 6153 to correct the frequency distribution data 82 until the classification result of the taught data 90 of the classifier 330 satisfies a predetermined standard. Specifically, the repetition control unit 6154 determines whether or not the correct answer rate of class discrimination of each taught data 90 by the classifier 330 satisfies a predetermined standard based on the classification result of step S17 of FIG. FIG. 9: Step S38). For example, it is conceivable that the predetermined standard is “the correct answer rate exceeds a predetermined value”. If the correct answer rate does not satisfy the predetermined standard (NO in step S38), the process returns to step S15, and the frequency distribution data correction unit 6153 further corrects the frequency distribution data 82 based on the mismatched taught data in step S37. I do. As described above, steps S35 to S37 are repeatedly performed until the correct answer rate of class discrimination of each taught data 90 by the classifier 330 becomes a predetermined value.

  If a classifier 330 that satisfies the predetermined criteria is obtained in step S38 (YES in step S38), the classifier 330 is registered as a sub-classifier (step S39).

  Thus, in the classifier generation unit 615, the classifier 330 classifies each taught data 90 of the subclass based on the frequency distribution data 82, and the frequency distribution data correction unit 6153 is classified into a class different from the teaching class. The frequency distribution data 82 is corrected based on the mismatched taught data. At this time, the frequency distribution data correction unit 6153 corrects the frequency distribution data 82 so that the frequency of the corresponding section corresponding to the feature value of the mismatched taught data increases for each feature value axis. Thereby, the frequency distribution data 82 in which the appearance ratio of the section to which the value of each feature amount of each of the mismatched taught data belongs apparently increases is generated. Then, since the classifier 330 is updated to perform data classification based on the corrected frequency distribution data 82, the probability of correctly classifying the mismatched taught data into the teaching class increases. Therefore, by repeatedly correcting the frequency distribution data 82, it is possible to obtain a classifier 330 that classifies the data into a class that matches the teaching class with high probability.

  FIG. 16 is a diagram for explaining another modification example of the frequency distribution data 82. In the example shown in FIG. 15, when there is one mismatched taught data, the frequency of only the corresponding section k corresponding to a certain feature value of the mismatched taught data is increased by “1”. However, as shown in FIG. 16, the frequency may be increased not only in the corresponding section k, but also in the sections k + 1 and k−1 (adjacent sections) adjacent to the section k. In the example illustrated in FIG. 16, the increase number in the section k is “2”, and the increase number in the sections k + 1 and k−1 is “1”. In this way, the increase number of the corresponding section and the increase number of the adjacent section may be made different or may be matched.

  Further, the section for increasing the frequency is not limited to the corresponding section and the adjacent sections on both sides thereof. For example, the frequency may be increased for the corresponding section k and a plurality of sections on both sides thereof, or for the corresponding section k and a plurality of sections on one side thereof. Further, the frequency increase number in each section on both sides (or one side) may be determined in accordance with a Gaussian distribution centering on the corresponding section k (specifically, a Gaussian distribution converted into an integer).

  By increasing the frequency of the corresponding section and the adjacent section to which the feature quantity of the mismatched taught data belongs, the frequency distribution data 82 in which the appearance ratio of those sections is apparently increased can be generated. Here, the feature amount of the data belonging to the teaching class of the mismatched taught data can be normally distributed in a specific corresponding section and an adjacent section adjacent thereto. Therefore, it is possible to increase the probability that the updated classifier 330 classifies the data having the feature amount of the corresponding section or the adjacent section into the teaching class of the mismatched taught data.

  Further, when the increase number of the corresponding section is larger than the increase number of the adjacent section, the appearance ratio of the corresponding section can be increased relatively larger than the appearance ratio of the adjacent section. Therefore, the updated classifier 330 has the data to be classified into the teaching class of the mismatched taught data, that is, the feature amount of the corresponding section, but the probability of being classified into the teaching class can be increased.

  Here, the sub-classifier generation process (FIGS. 6 and 7: step S23) in the first and second teacher data creation support processes has been described. However, the typical classifier generation process (FIG. 7: step S262) in the second teacher data creation support process can also be generated in the same procedure as the sub-classifier generation process.

  FIG. 17 is a graph showing the relationship between the number of taught data (the number of teacher data) and the correct answer rate of the classifier 330. In this graph, the horizontal axis indicates the number of taught data (the number of teacher data for which a class is taught), and the vertical axis indicates the correct answer rate of the classifier 330 generated by the number of taught data.

  The graph of FIG. 17 shows an example in which the classifier 330 classifies the defect data (the number of feature amount axes is 174 dimensions) having a total data number of 5112 into three classes. In order to calculate the correct answer rate, all 5112 data are given class labels in advance, but are not referred to during the classification process of the classifier 330. For example, when the classifier 330 is generated, if one taught data 90 is used for one class, the number of taught data is 3. When two taught data 90 are used for one class, the number of taught data in the sub-taught data group is 6. Selection of the taught data 90 is performed at random.

  In the example shown in FIG. 17, when the classifier 330 is acquired, when the frequency distribution data correction unit 6153 corrects the frequency distribution data 82 (see FIG. 9: step S35), the corresponding section k (attention zone) is set. “3”, the frequency is increased by “1” in each of the sections k−1, k + 1 on both sides of the corresponding section k ({+1, + 3, + 1} (total number 5, standard deviation 1)) . In addition, the correct answer rate is evaluated 100 times independently for any number of taught data. In the graph, the points indicated by black circles indicate the average value of the correct answer rate obtained by 100 evaluations, and the upper and lower “beards” indicate the distribution range (maximum and minimum) of the correct answer rate. For comparison, the correct answer rate of the classifier 330 generated by a simple method of increasing the corresponding interval by +1 when correcting the frequency distribution data 82 is indicated by a broken line.

  As shown in FIG. 17, the correct answer rate of the classifier 330 is improved by increasing the number of taught data. Further, the classifier 330 having a higher correct answer rate can be obtained when the frequency is increased in the corresponding section and its adjacent sections than when the frequency is increased by 1 in the corresponding section. This tendency is more remarkable as the number of taught data is smaller.

  FIG. 18 is another graph showing the relationship between the number of taught data and the correct answer rate of the classifier 330. In this graph, the horizontal axis indicates the number of taught data, and the vertical axis indicates the correct answer rate of the classifier 330 generated by the classifier construction device 33. This graph shows a case where a total of 5000 cell images (the number of feature axes is 11 dimensions) is classified into 3 classes using a classifier 330 that is generated not by a defect image but by a cell image as teacher data. The correct answer rate is shown.

  In the example shown in FIG. 18, when the classifier 330 is generated, when the frequency distribution data correction unit 6153 corrects the frequency distribution data 82 (see FIG. 9: step S35), the corresponding section k (attention zone) is set. “14”, “1”, “2”, “3”, “5”, “8”, “11”, “13” in each of the eight negative sections (sections k−8 to k−1) centering on the corresponding section k , The frequency of “13” “11” “8” “5” “3” “2” “1” is increased in each of the eight positive sections (sections k + 1 to k + 8) ({1, 2, 3, 5, 8, 11, 13, 14, 13, 11, 8, 5, 3, 2, 1} (total number 100, standard deviation 3)). In the graph, the points indicated by black circles indicate the average value of the correct answer rate obtained by 100 evaluations, and the upper and lower “beards” indicate the distribution range (maximum and minimum) of the correct answer rate. For comparison, the correct answer rate of the classifier 330 generated by a simple method of increasing the corresponding interval by +1 when correcting the frequency distribution data 82 is indicated by a broken line.

  Also in the example shown in FIG. 18, as in the example shown in FIG. 17, the correct answer rate of the classifier 330 is improved by increasing the number of taught data. Further, with respect to the mismatched taught data, it is possible to obtain the classifier 330 having a higher correct answer rate when the frequency is increased in the corresponding section and its surrounding sections than when the frequency is increased by 1 in the corresponding section. is made of. This tendency is more remarkable as the number of teachings is smaller.

  As shown in FIGS. 17 and 18, even if the number of taught data is the same, the correct answer rate varies depending on how taught data is selected. For example, as shown in FIG. 17, when the number of taught data is 3, the average correct answer rate is about 50%, but depending on the combination of taught data 90, there is a possibility that a correct answer rate of 65% or more may be obtained. There is. That is, in the plurality of sub-classifiers generated by the first and second teacher data creation support processes shown in FIGS. 6 and 7, the correct answer rate can vary greatly. In particular, in the first teacher data creation support process, in the typical classifier selection process (FIG. 6: step S26), the sub-classifier having the best classification result is selected from the plurality of sub-classifiers as the typical classifier. To do. For this reason, the effect of minimizing the quantity of non-conforming taught data to be presented to the operator in step S28, that is, the number of taught data for which human judgment is required becomes significant.

  Note that it is not essential for the classifier generation unit 615 to generate a classifier according to the procedure shown in FIG. 9, and other techniques (for example, known techniques such as linear discriminant analysis and SVM) are also prevented from being adopted. Absent.

<2. Modification>
Although the embodiment has been described above, the present invention is not limited to the above, and various modifications are possible.

  For example, in step S22, the quantity of the taught data 90 selected from each class by the sub-taught data group construction unit 613 is not limited to the same number, and may be different for each class. For example, the quantity selected from each class may be proportional to the total number of taught data 90 belonging to each class. That is, when there are the first class and the second class, and the total number of the first classes is L times the total number of the second classes, the quantity selected from the first class is set to the second class. It is good also as L times the quantity selected from a class. However, the sub-classifier is generated by machine learning based on the appearance frequency for each class of teacher data. For this reason, when the quantity to be selected is different for each class, the classification result of the sub-classifier tends to be biased. For this reason, it is desirable that the quantity of the taught data 90 selected from each class as the sub-teached data group is the same number between classes.

  Although the present invention has been described in detail, the above description is illustrative in all aspects, and the present invention is not limited thereto. It is understood that countless variations that are not illustrated can be envisaged without departing from the scope of the present invention. The configurations described in the above embodiments and modifications can be appropriately combined or omitted as long as they do not contradict each other.

DESCRIPTION OF SYMBOLS 1 Image classification device 5 Host computer 55 Display 56 Input part 61 Teacher data creation assistance part 610 Data operation part 611 Storage part 613 Sub teaching data group construction part 615 Classifier generation part 6150 Teacher data storage part 6151 Frequency distribution data generation part 6152 Classifier construction unit 6153 Frequency distribution data correction unit 6154 Repeat control unit 617 Classification result acquisition unit 618 Repeat control unit 619 Data extraction unit 620 Display control unit 63 Learning unit 90 Taught data

Claims (7)

A teacher data creation support method for supporting creation of teacher data used to generate a classifier that classifies data based on its feature amount,
(a1) preparing a taught data group composed of a plurality of taught data in which any one of a plurality of classes is taught as a teaching class;
(b1) in the taught data group, constructing a sub-taught data group composed of the taught data selected one or more from each of the plurality of classes;
(c1) generating a sub classifier that classifies the taught data using the sub taught data group as teacher data;
(d1) obtaining a classification result of the sub-classifier by classifying all or part of the taught data group in the sub-classifier generated by the step (c1);
(e1) Classifying all or part of the taught data group from a plurality of the sub-classifiers obtained by repeating the step (b1), the step (c1) and the step (d1) a plurality of times. Selecting a sub-classifier as a typical classifier whose classification results satisfy the predetermined classification criteria,
(f1) When the typical classifier selected in the step (e1) classifies all or part of the taught data group, non-conforming taught data whose classification destination class does not match the taught class Extracting, and
Including teacher data creation support method. A teacher data creation support method according to claim 1,
The teacher data creation support method, wherein the step (e1) is a step of selecting, as the typical classifier, one sub classifier having the best classification result from the plurality of sub classifiers. A teacher data creation support method for supporting creation of teacher data used to generate a classifier that classifies data based on its feature amount,
(a1) preparing a taught data group composed of a plurality of taught data in which any one of a plurality of classes is taught as a teaching class;
(b1) in the taught data group, constructing a sub-taught data group composed of the taught data selected one or more from each of the plurality of classes;
(c1) generating a sub classifier that classifies the taught data using the sub taught data group as teacher data;
(d1) obtaining a classification result of the sub-classifier by classifying a plurality of the taught data in the sub-classifier generated by the step (c1);
(e2) Classifying all or part of the taught data group from the plurality of sub-classifiers obtained by repeating the step (b1), the step (c1) and the step (d1) a plurality of times. Selecting one or more sub-classifiers whose classification results satisfy a predetermined classification criteria,
(f2-1) When all or some of the plurality of taught data groups are classified in each of the one or more sub-classifiers selected in the step (e2), the classification destination class is the teaching class. Generating a typical classifier using matched taught data that matches a class as teacher data;
(f2-2) Non-conformity in which the classification class does not match the taught class when all or part of the taught data group is classified by the typical classifier generated in the step (f2-1) Extracting the taught data; and
Including teacher data creation support method. A teacher data creation support method according to claim 3,
The teacher data creation support method, wherein the step (e2) is a step of selecting two or more sub-classifiers satisfying a predetermined classification result criterion from the plurality of sub-classifiers. A teacher data creation support method according to claim 3,
The teacher data creation support method, wherein the step (e2) is a step of selecting one sub-classifier having the best classification result as the typical classifier from the plurality of sub-classifiers. A teacher data creation support method according to any one of claims 1 to 5,
The teacher data creation support method, wherein the classification result is a ratio of a total number of taught data in which the teaching class matches a class to be classified by the sub-classifier out of a total number of the taught data. A teacher data creation support device that supports creation of teacher data used to generate a classifier that classifies data based on its feature amount,
A storage unit for storing a taught data group composed of a plurality of taught data in which any one of a plurality of classes is taught as a teaching class;
In the taught data group, a sub-taught data group constructing unit that constructs a sub-taught data group by selecting one or more from each of the plurality of classes;
A classifier generation unit that generates a sub classifier for classifying data using the sub-taught data group as teacher data;
A classification result obtaining unit for obtaining a classification result of the sub-classifier when all or part of the taught data group is classified by the sub-classifier;
The classification results of each of the plurality of sub classifiers generated from the plurality of sub taught data groups are acquired by controlling the sub taught data group construction unit, the classifier generation unit, and the classification result acquisition unit. A repeat control unit to
When a sub-classifier satisfying a predetermined classification performance criterion is selected as a typical classifier from the plurality of sub-classifiers, and when part or all of the taught data group is classified by the typical classifier A data extraction unit that extracts non-conforming teaching data whose classification destination class does not match the teaching class;
A teacher data creation support device.

Images