JP7083721B2 - Classifier generation method and classifier generator - Google Patents

Description

The present invention relates to a technique for classifying the data according to a plurality of types of features of the data.

In the manufacture of semiconductor substrates, glass substrates, printed wiring substrates, etc., visual inspection is performed using an optical microscope, scanning electron microscope, or the like in order to inspect defects such as foreign matter, scratches, and etching defects. Further, by performing a detailed analysis on the defects detected in such an inspection step, the cause of the occurrence of the defects is identified, and countermeasures against the defects are taken.

In recent years, as the pattern on the substrate becomes more complicated and miniaturized, the types and quantities of defects detected tend to increase, and automatic classification that automatically classifies the defects detected in the inspection process is also used. Defect analysis can be performed quickly and efficiently by automatic classification. In automatic classification, a classifier using a neural network, decision tree, discriminant analysis, etc. is used. In order for the classifier to perform automatic classification, it is necessary to prepare teacher data including signals indicating the defect image and its category (that is, the type (class) of the defect image) to train the classifier (classifier). For generation, see, for example, Patent Document 1). Typically, the teacher data is created by the operator determining the category of each defect image.

If the operator's determination of the category is ambiguous in the creation of teacher data, the performance of the classifier may deteriorate due to the inclusion of inconsistent data in the training data of the classifier. Therefore, Patent Documents 2 to 5 disclose a neural network learning device that detects a learning input pattern that hinders learning during or after learning, removes or changes it, and then performs re-learning or the like.

Patent Document 6 discloses a technique for extracting the above-mentioned inconsistent data. In the defect classification method, a discriminant function is obtained by learning using the teacher data, and the teacher data is classified into each category using the discriminant function. Then, the teacher data whose classification category does not match the initial category is presented to the operator. Accordingly, the operator modifies the initial category of the teacher data or excludes the data from the teacher data.

Patent Document 7 describes a teacher data creating means for creating teacher data including information on the correct answer category and the certainty level by inputting the correct answer category to which the defect region extracted from the inspection image should be classified with the certainty level. A classification device including a classification means for classifying teacher data having a high confidence level as more significant data than teacher data having a low confidence level is disclosed.

Japanese Patent Application Laid-Open No. 2003-317803 Japanese Unexamined Patent Publication No. 6-83792 Japanese Unexamined Patent Publication No. 7-168799 Japanese Unexamined Patent Publication No. 9-81190 Japanese Unexamined Patent Publication No. 10-49509 Japanese Unexamined Patent Publication No. 11-344450 Japanese Unexamined Patent Publication No. 2006-189915

In the prior art, the operator may reconfirm whether the relevant teacher data is truly inappropriate based on the results presented by the classifier. For this reason, the work load of the operator has increased. In addition, when inappropriate learning data is deleted, the teacher data is reduced. Furthermore, in a classifier generated by re-learning that ignores some existing data, when encountering data similar to the ignored data, there is a risk that the classification result for it will be left to the discretion.

Also, in the field of machine learning, the ability to give correct classification results (generalization ability) even for data that was not used for learning is emphasized, but it is a sample created by extracting data evenly from the population. It is assumed that the learning is based on the distribution. Therefore, when learning by intentionally excluding some data, it is difficult to give correct classification results even for inconvenient data.

An object of the present invention is to provide a technique for producing a classifier having excellent generalization ability.

In order to solve the above problems, the first aspect is a classifier generation method for generating a classifier that classifies data according to feature quantities, and (a) a plurality of types of features for each of a plurality of teaching data. A step of calculating the amount, and (b) a step of generating a teacher data group including a plurality of teacher data by teaching one of different positive classes for each of the plurality of teaching data, and ( c) a step of generating a classifier using the teacher data group, (d) a step of classifying the teacher data group with the classifier generated by the step (c), and (e) the teacher data group. Of these, for teacher data whose classification destination is different from the teaching in the step (d), the teaching class is changed to a subclass which is a pair of the original positive class, and (f) the teaching is performed by the step (e). A step of generating a new classifier using a teacher data group containing teacher data whose class has been changed to the subclass, and (g) the teacher data group being generated by the classifier generated by the step (f). Regarding the step of classifying and (h) the teacher data whose classification destination is different from the teaching in the step (g), if the teaching is a positive class, the teaching class is changed to a subclass which is a pair thereof, and the teaching is performed. Is a subclass and the classification destination is a positive class that is a pair of the subclass, the step of changing the teaching class to the positive class and (i) the teaching class is changed in the step (h). It includes a step of generating a new classifier using a teacher data group including teacher data.

The second aspect is the method for generating a classifier of the first aspect, further comprising (j) a step of repeating the steps (g) to (i).

The third aspect is the classifier generation method of the second aspect, in which the step (j) is a step of performing the steps (g) to the step (i) twice or more until a predetermined end condition is satisfied. Is.

The fourth aspect is the classifier generation method of the third aspect, in which the termination condition is that the teaching class does not change in the step (h).

A fifth aspect is the method for generating a classifier according to the third aspect, wherein the end condition is that the correct answer rate for the positive class exceeds a predetermined threshold value in the classification result of the step (g).

The sixth aspect is a classifier generator that generates a classifier that classifies data according to features, and generates a classifier using a teacher data group taught by one of different classes. A unit and a teaching class changing unit that changes the teaching class of the teacher data between a normal class and a subclass that is a pair thereof according to the classification destination of the teacher data by the classifier. When the teacher data group is classified by the classifier, the teaching class change unit sets the teaching class to the teaching class when the teaching is a positive class for the teacher data whose classification destination by the classifier is different from the teaching. If the teaching is a subclass and the classification destination is a positive class that is a pair of the subclass, the teaching class is changed to the positive class and the classifier is generated. The unit generates a new classifier using the teacher data group including the teacher data whose teaching class has been changed by the teaching class changing unit.

According to the classifier generation method of the first aspect, the teacher data for which the taught class was inappropriate is automatically changed to a subclass assuming that the teaching is unreliable, and the subclass is also included. A classifier is generated by re-learning. Since the re-learning is performed without removing the teacher data in this way, it is possible to obtain a classifier having excellent generalization ability. Further, it is possible to obtain a classifier having a high degree of certainty for the result classified into the normal class and having a low degree of certainty for the result classified into the subclass. Therefore, it is possible to obtain a classifier in which information on the degree of certainty is added to the classification result.

According to the classifier generation method of the second aspect, the accuracy of the information of the degree of certainty given to the classification result can be improved by repeatedly generating the classifier including the subclass paired with the positive class.

According to the classifier generation method of the third aspect, the accuracy of the certainty information given to the classification result is determined by repeatedly generating the classifier by re-learning including the subclass until the end condition is satisfied. Can be improved.

According to the classifier generation method of the fourth aspect, the accuracy of the certainty information given to the classification result can be improved by repeatedly generating the classifier by re-learning until the teaching class is not changed.

According to the classifier generation method of the fifth aspect, the confidence given to the classification result between the positive classes by repeatedly generating the classifier by re-learning until the correct answer rate of the positive class exceeds a predetermined threshold value. The accuracy of the information of the degree can be improved.

According to the classifier generator of the sixth aspect, the teacher data for which the taught class was inappropriate is automatically changed to a subclass assuming that the teaching is unreliable, and the subclass is also included. A classifier is generated by re-learning. Since the re-learning is performed without removing the teacher data in this way, it is possible to obtain a classifier having excellent generalization ability. Further, it is possible to obtain a classifier having a high degree of certainty for the result classified into the normal class and having a low degree of certainty for the result classified into the subclass. Therefore, it is possible to obtain a classifier in which information on the degree of certainty is added to the classification result.

It is a figure which shows the schematic structure of the image classification apparatus 1 of an embodiment. It is a figure which shows the flow of classification of the defect image by the image classification apparatus 1 of an embodiment. It is a block diagram which shows the structure of the host computer 5 of embodiment. It is a block diagram which shows the functional structure of the host computer 5 for generating a classifier 422. It is a figure which shows an example of a teacher data group 82G. It is a figure which shows the flow of generating the classifier 84 in the host computer 5 of an embodiment. It is a figure which conceptually shows the flow of generating the classifier 84 in the host computer 5 of an embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the components described in this embodiment are merely examples, and the scope of the present invention is not limited to them. In the drawings, the dimensions and numbers of each part may be exaggerated or simplified as necessary for easy understanding.

Expressions indicating equal states (eg, "same", "equal", "homogeneous", etc.) not only represent quantitatively exactly equal states, but also provide tolerance or similar functionality, unless otherwise noted. It shall also represent the state in which there is a difference.

<1. Embodiment>
FIG. 1 is a diagram showing a schematic configuration of the image classification device 1 of the embodiment. The image classification device 1 acquires a defect image showing a pattern defect on the semiconductor substrate 9 and classifies the defect image. The image classification device 1 includes an image pickup device 2, an inspection / classification device 4, and a host computer 5.

The image pickup apparatus 2 takes an image of an inspection target area on the semiconductor substrate 9. The inspection / classification device 4 performs a defect inspection based on the image data acquired by the image pickup device 2. When a defect is detected, the inspection / classification device 4 classifies the defect according to the type of the defect. Hereinafter, this type is also referred to as a "class". The class of pattern defects existing on the semiconductor substrate 9 is, for example, "defect", "projection", "disconnection", "short circuit", "foreign matter" and the like. The host computer 5 controls the overall operation of the image classification device 1 and generates a classifier 422 used for classifying defects in the inspection / classification device 4. The host computer 5 is an example of a classifier generator.

The image pickup apparatus 2 may be incorporated in the production line of the semiconductor substrate 9. The image classification device 1 may be a so-called inline type system. The image classification device 1 is a device in which a function of automatic defect classification is added to a defect inspection device.

The image pickup device 2 includes an image pickup unit 21, a stage 22, and a stage drive unit 23. The image pickup unit 21 takes an image of an inspection target region on the surface of the semiconductor substrate 9. The stage 22 holds the semiconductor substrate 9 in a horizontal posture. The "horizontal posture" means a state in which the semiconductor substrate 9 is parallel to the horizontal plane. The stage drive unit 23 moves the stage 22 relative to the image pickup unit 21 in a direction parallel to the surface of the semiconductor substrate 9.

The image pickup unit 21 includes an illumination unit 211, an optical system 212, and an image pickup device 213. The optical system 212 guides the illumination light from the illumination unit 211 to the semiconductor substrate 9. The light reflected on the surface of the semiconductor substrate 9 is incident on the optical system 212 again. The image pickup device 213 converts the image of the semiconductor substrate 9 imaged by the optical system 212 into an electric signal.

The stage drive unit 23 includes a ball screw, a guide rail, a motor, and the like. The host computer 5 controls the stage drive unit 23 and the image pickup unit 21, so that the inspection target area on the semiconductor substrate 9 is imaged.

The inspection / classification device 4 includes a defect detection unit 41 and a classification control unit 42. The defect detection unit 41 detects defects while processing the image data of the inspection target area. Specifically, the defect detection unit 41 has a dedicated electrical circuit that processes image data in the inspection target area at high speed, and compares an image obtained by imaging with a reference image (an image without defects). Defect inspection of the inspection target area is performed by image processing. The classification control unit 42 classifies the defect image detected by the defect detection unit 41. The classification control unit 42 is composed of a CPU that performs various arithmetic processes, a memory that stores various information, and the like, and has a feature amount calculation unit 421 and a classifier 422. The classifier 422 performs defect classification, that is, classification of defect images, using neural networks, decision trees, discriminant analysis, and the like.

FIG. 2 is a diagram showing a flow of classification of defective images by the image classification device 1 of the embodiment. When the image pickup device 2 takes an image of the surface of the semiconductor substrate 9, the defect detection unit 41 of the inspection / classification device 4 acquires image data (step S11). Subsequently, the defect detection unit 41 detects the defect by inspecting the defect in the inspection target area (step S12). When a defect is detected in step S12 (YES in step S12), the data of the image of the defect portion (that is, the defect image) is transmitted to the classification control unit 42. If no defect is detected (NO in step S12), the process returns to step S11 to acquire image data.

Upon receiving the defect image, the classification control unit 42 calculates a plurality of types of feature quantities of the defect image (step S13). An array of a plurality of types of features is also referred to as a feature vector. The calculated feature amount vector is input to the classifier 422, and classification is performed by the classifier 422 (step S14). That is, the classifier 422 classifies the defect image into one of a plurality of classes. In the image classification device 1, each time a defect is detected by the defect detection unit 41, the feature amount vector is calculated in real time, and a large number of defect images are automatically classified at high speed.

FIG. 3 is a block diagram showing the configuration of the host computer 5 of the embodiment. The host computer 5 is an example of a classifier generator that generates a classifier 422 by machine learning.

The host computer 5 includes a CPU 51, a ROM 52, and a RAM 53. The CPU 51 includes an arithmetic circuit that performs various arithmetic processing. The ROM 52 stores this program. The RAM 53 is a volatile main storage device that stores various types of information. The host computer 5 includes a general computer system having a CPU 51, a ROM 52, and a RAM 53 connected to each other by a bus line 501.

The host computer 5 includes a fixed disk 54, a display 55, an input unit 56, a reading device 57, and a communication unit 58. Each of these elements is connected to bus line 501 via an interface (I / F).

The fixed disk 54 is an auxiliary storage device that stores information. The display 55 is an output device that displays various information such as images. The input unit 56 is an input device including a keyboard 56a, a mouse 56b, and the like. The reading device 57 is a device that reads information from a transient recording medium 8 having portability. The recording medium 8 is, for example, an optical disk, a magnetic disk, a magneto-optical disk, or a semiconductor memory. The communication unit 58 transmits / receives signals to / from other elements of the image classification device 1.

The host computer 5 reads the program 80 from the recording medium 8 via the reading device 57. The read program 80 is recorded on the fixed disk 54. The program 80 is appropriately stored in the RAM 53. Then, the CPU 51 performs various arithmetic processes according to the program 80 stored in the RAM 53.

FIG. 4 is a block diagram showing a functional configuration of the host computer 5 for generating the classifier 422. The host computer 5 includes a control unit 61 and a storage unit 63. When the CPU 51 of the host computer 5 operates according to the program 80, the control unit 61 functions as a feature amount calculation unit 611, a classifier generation unit 613, a teaching class change unit 615, and an end determination unit 617.

The feature amount calculation unit 611 calculates the feature amount vector from the defect image. The feature quantity items included in the feature quantity vector include, for example, the area of the defect portion, the average brightness, the perimeter, the flatness, or the slope of the long axis when the defect portion is approximated to an ellipse.

The classifier generation unit 613 generates a classifier 84 for classifying defective images by performing machine learning using a teacher data group including a plurality of teacher data 82. Machine learning means to calculate a discriminant function for discriminating between assigned classes for the teacher data group 82G. Examples of the method for calculating the discriminant function include a neural network, a decision tree, and discriminant analysis.

FIG. 5 is a diagram showing an example of the teacher data group 82G. The teacher data 82 includes a feature amount vector 822 extracted from the defect image and a class label which is information indicating a class (hereinafter, also referred to as a teaching class 824) taught by the operator in advance. In teaching (assigning a class) a class for each defect image, a plurality of positive classes are defined in advance. The positive class is, for example, a class defined for each type of defect assumed by the operator. The teacher data 82 is generated by the operator teaching one of the plurality of positive classes to each defect image.

In the present embodiment, as will be described later, new classifiers 84 are generated one after another by repeatedly performing re-learning. In the process of generating the classifier 84 by the re-learning, a subclass may be defined. The subclass is a class that is paired with the primary class defined at the beginning, and has a one-to-one relationship with the primary class. That is, for one primary class, only one subclass is defined. It should be noted that the subclass is defined as needed and is not necessarily defined for all the positive classes. Further, in the process of generating the classifier 84 by re-learning, unnecessary subclasses may be appropriately removed.

The teaching class changing unit 615 changes the teaching class 822 in the teacher data 82 according to a predetermined change rule. This change rule will be described in detail later. When changing the teaching class 824 of the teacher data 82, the teaching class changing unit 615 changes the teaching class 824 between the primary class taught first and the subclass that is a pair of the positive classes. That is, the teaching class 824 of each teacher data 82 is changed only between one positive class and the subclass corresponding to the positive class.

The end determination unit 617 determines whether or not to end the generation process of the classifier 84 by determining whether or not the end condition is satisfied. As will be described later, the control unit 61 stops re-learning based on the determination result of the end determination unit 617, and ends the generation of the new classifier 84. The classifier 84 generated in the host computer 5 is sent to the inspection / classification device 4 and used as the classifier 422.

<About the generation of the classifier 84 by re-learning>
FIG. 6 is a diagram showing a flow of generating a classifier 84 in the host computer 5 of the embodiment. FIG. 7 is a diagram conceptually showing the flow of generating the classifier 84 in the host computer 5 of the embodiment. Unless otherwise specified, the operation of the host computer 5 shown in FIGS. 6 and 7 shall be performed by the control unit 61.

In order to generate the classifier 84 by machine learning, the teacher data group 82G is generated (FIG. 6: step S21). Specifically, the feature amount calculation unit 611 calculates the feature amount vector 822 which is a plurality of types of feature amounts for each of the plurality of defect images which are the data used as the teacher data 82. Further, the operator assigns one positive class from a plurality of predefined positive classes as the teaching class 824 for each defect image. As shown in FIG. 5, a plurality of teacher data 82 including the feature amount vector 822 and the teaching class 824 are generated from each defect image.

In the example shown in FIG. 7, three positive classes "a", "b", and "c" are defined, and six teacher data 82 shown in a circular shape are prepared for each of these classes a-c. .. Further, for each of these positive classes ac, the paired subclasses are "a?", "B?", And "c?". “A”, “b” and “c” described in each teacher data 82 indicate teaching class 824. In the following, when the teaching classes (positive classes ac) are distinguished for a plurality of teacher data 82, the classes are written in parentheses at the end of the code. For example, the teacher data 82 of the regular class a is written as "teacher data 82 (a)", the teacher data 82 of the regular class b is written as "teacher data 82 (b)", and the teacher data 82 of the regular class c is written. It is written as "teacher data 82 (c)".

When the teacher data group 82G is generated in step S21, the classifier generation unit 613 generates the first-stage classifier 84 by machine learning using the teacher data group 82G (FIG. 6: step S22). The first-stage classifier 84 has a function of classifying the defect image among a plurality of positive classes according to the feature amount vector 822 calculated from the defect image.

When the first-stage classifier 84 is generated, the classifier 84 classifies the teacher data group 82G (FIG. 6: step S23). The process of step S23 is a so-called reassignment method. As a result, the class to be classified is determined for each teacher data 82. In the example shown in FIG. 7, among the teacher data 82 groups, some teacher data 82 are classified into a class different from the teaching. For example, in the regular class A, five teacher data 82 are classified, one is the teacher data 82 (b) and the other is the teacher data 82 (c) whose teaching is the regular class c. be.

Subsequently, the teaching class changing unit 615 changes the teaching class 824 of the teacher data 82 according to the class of the classification destination determined in step S23 according to the default change rule (FIG. 6: step S24). In this step S24, the teaching class changing unit 615 changes the teaching class according to the following change rule 1.

(Change rule 1)
As a result of the re-imputation method, for the teacher data 82 whose classification destination is different from the teaching, the teaching class 824 is changed from the original positive class to the subclass which is a pair of the positive class.

That is, as a result of the reassignment method in step S23, when the teacher data 82 whose classification destination is different from the teaching exists, the teaching class changing unit 615 pairs the positive class which is the original teaching class 822 of the teacher data 82. Define a subclass that becomes. Then, the teaching class changing unit 615 changes the teaching class 822 of the teacher data 82 whose classification destination is different from the teaching to the newly defined subclass.

In the example shown in FIG. 7, for the teacher data 82 (b) and 82 (c) whose classification destination is the positive class a, the respective teaching classes 822 (normal classes b and c) are subclass b? , C? Will be changed to. For the two teacher data 82 (c) whose classification destination was the positive class b, the teaching class 822 (normal class c) is the subclass c? Will be changed to. Even for one teacher data 82 (a) whose classification destination was the positive class c, the teaching class 822 (normal class a) is the subclass a? Will be changed to.

In step S24, the teaching class 824 is not changed for the teacher data 82 that does not correspond to the change rule, and the teaching class is maintained as it is.

Returning to FIG. 6, when the change processing of the teaching class 824 in step S24 is completed, the classifier generation unit 613 generates the second stage classifier 84 by performing re-learning using the teacher data group 82G. (FIG. 6: Step S25). The teacher data group 82G used for this re-learning includes the teacher data 82 in which the teaching class 824 is changed in step S24, and the teacher data 82 in which the teaching class 824 is not changed. Therefore, the second-stage classifier 84 has a function of classifying defective images not only among a plurality of positive classes but also among classes including a subclass defined in step S24.

When the second-stage classifier 84 is generated, the classifier 84 classifies the teacher data group 82G (FIG. 6: step S26). The process of step S26 is also a so-called reassignment method as in step S23. As a result, the class to be classified is determined for each teacher data.

Subsequently, the teaching class changing unit 615 changes the teaching class 824 of the teacher data 82 according to the classification destination class determined in step S25 (FIG. 7: step S27). In this step S27, the teaching class changing unit 615 changes the teaching class according to the following change rules 2 and 3.

(Change rule 2)
As a result of the re-imputation method, when the teaching is a positive class for the teacher data 82 whose classification destination is different from the teaching, the teaching class 824 is changed to the subclass which is the pair thereof. This change rule 2 has almost the same contents as the change rule 1.

(Change rule 3)
As a result of the re-imputation method, for teacher data 82 whose classification destination is different from the teaching, if the teaching is a subclass and the classification destination is a positive class that is a pair of the subclass, the teaching class 824 is set to the positive class. change.

Regarding the change rule 2, in the example shown in FIG. 7, one of the teacher data 82 (a) in which the teaching class 824 was the positive class a is classified into the positive class c. Therefore, the teaching class of the teacher data 82 (a) changes from the normal class a to the subclass a in accordance with the change rule 2. Will be changed to.

Regarding the change rule 3, in the example shown in FIG. 7, the teaching class 824 is a subclass a? The teacher data 82 (a?) That was was the subclass a? It is classified into the positive class a, which is a pair of. Therefore, according to the change rule 3, the teaching class 824 (subclass a?) Of the teacher data 82 (a?) Is changed to the regular class a. Similarly, the teaching class 824 is the subclass c? The teacher data 82 (c?) Was classified into the original positive class c. Therefore, the teaching class 824 (subclass c?) Of the teacher data 82 (c?) Is changed to the positive class c.

In step S27, the teaching class 824 is not changed for the teacher data 82 that does not correspond to the change rule, and the teaching class is maintained as it is.

Returning to FIG. 6, when the change process of the teaching class 824 in step S27 is completed, the classifier generation unit 613 generates the classifier 84 of the third stage by performing re-learning using the teacher data group 82G. (FIG. 6: Step S28). The teacher data group 82G used for this re-learning includes teacher data 82 in which the teaching class 824 is changed in step S27 and teacher data 82 in which the teaching class 824 is not changed.

Subsequently, the end determination unit 617 determines whether or not the end condition is satisfied. As a result, it is determined whether or not the termination criterion is satisfied (FIG. 6: step S29). For example, the following two can be considered as the termination conditions.

(End condition 1)
As a result of the reassignment method in step S26, the teaching class does not change in step S27.

(End condition 2)
In the classification result of step S26, the accuracy of the correct answer exceeds a predetermined threshold value. Here, the accuracy rate means the ratio of the total number of teacher data 82 in which the classification destination and the teaching match among the total number of teacher data 82 classified by the classifier 84. Further, the predetermined threshold value may be set to the correct answer rate required in operation. For example, if a correct answer rate of 95% is required, the threshold value may be set to 0.95. Further, the correct answer rate smaller than the correct answer rate may be set as the threshold value based on the correct answer rate when the end condition 1 is satisfied (that is, when the change of the teaching class does not occur). For example, assuming that the correct answer rate when the end condition 1 is satisfied is 95%, 92% (= 0.92), which is smaller than the 95%, may be set as the threshold value.

When the end determination unit 617 determines in step S29 that any one of the end conditions 1 and 2 is satisfied (YES in step S29), the control unit 61 ends the generation process of the classifier 84. As a result, the classifier 84 generated in the immediately preceding step S28 is stored in the storage unit 63. When the end determination unit 617 determines that neither of the above end conditions 1 and 2 is satisfied (NO in step S29), the control unit 61 returns to step S26 and continues the process. That is, steps S26 to S28 are repeated until the end condition 1 or 2 is satisfied.

In the host computer 5 of the present embodiment, a paired subclass is defined as necessary for each of the positive classes defined in advance by the operator. Then, when the reassignment method is performed, the teacher data that is not classified as a positive class is made into a subclass to which the teaching class is a pair. As a result of the re-learning, if the teacher data to which the subclass is taught is classified into the original positive class by the re-imputation method, the teaching class of the teacher data is changed to the positive class again. In this way, even if the teacher data 82 is not correctly classified into the positive class taught first, the classifier 84 is generated by performing re-learning without removing the teacher data 82. As a result, a classifier 84 having excellent generalization ability can be obtained.

According to the classifier 84 generated in the present embodiment, if the class label given by the operator is inappropriate, it is automatically classified into a subclass on the premise that the teaching is unreliable. After that, machine learning including its subclass is performed. Since the teaching class of the teacher data is not changed between the regular classes, the generalization ability can be kept high between the regular classes with high teaching reliability. On the other hand, unreliable teacher data is likely to be classified into subclasses by a classifier. Therefore, the teacher data classified into the subclass has low teaching reliability. In this way, it is possible to generate a classifier 84 to which information on the degree of certainty is added to the classification result.

The classifier 84 is transmitted to the inspection / classification device 4 and registered as the classifier 422. When the classifier 422 classifies data whose class is unknown (that is, the type of defect is unknown), the data classified into the positive class may be the result with high certainty. Further, those classified into subclasses may be classified into the positive class of the tentatively, but the result may be low in certainty. It may be a pair of positive class defects. In addition, those classified into any of the subclasses may be collectively classified as unclassifiable, and the classification destination may not be a specific class.

For example, by actively incorporating the part where two positive classes (A and B) are mixed and distributed in the feature space into learning as a subclass, the automatically generated teacher data of the subclass is reproduced. The classification accuracy of the positive class can be kept high without confirmation. Between the two positive classes A and B, the distribution area of the subclass serves as a margin width, for example, in SVM (Support Vector Machine). Therefore, the generalization ability when a two-class classifier is used is improved.

In the above description, the subclass is defined as needed. However, at the time of step S21, a subclass may be defined in advance for each of all the positive classes. Then, at the stage of step S22, a classifier 84 that classifies between a plurality of positive classes and a plurality of subclasses corresponding to the positive classes may be generated. Further, in the above description, when the classifier 84 is generated, the subclass that is no longer needed due to the disappearance of the teacher data to which the classifier 84 belongs is excluded from the definition. However, this is not essential, and a new classifier 84 may be generated by re-learning while maintaining unnecessary subclasses.

(Example 1)
Table 1 is an example of the results of evaluating the performance of a classifier that classifies images of defects found in defect inspection of semiconductor wafers into three types (Class 1/2/3) by linear discriminant analysis by the re-imputation method. be. Table 1 is a confusion matrix (also referred to as a classification table or a confusion comparison table) showing the classification results by the classifier. In this example, it can be considered that the reliability of the teaching given by the operator is sufficiently high.

In this Table 1, three types of classes (Class 1/2/3) taught in advance are described in the row headings, and the classes classified by the classifier are described in the column headings. For example, the teacher data taught as Class 1 shows that 1496 were classified into Class 1, 81 were classified into Class 2, and 1 was classified into Class 3.

Further, in Table 1, the row described as "Sum" in the heading indicates the total number of teacher data classified into each class by the classifier, and the column described as "Sum" in the heading indicates the total number of teacher data taught. There is.

The line marked "Precision" in the heading is the ratio (adaptation rate) of the correctly classified teacher data among the teacher data classified into a specific class by the classifier. The column marked "Recall" in the heading is the percentage (recall rate) of the teacher data correctly classified into the specific class by the classifier among all the teacher data taught to be in the specific class.

The cell where the row of "Precision" and the column of "Recall" intersect is "Accuracy", and the class classified by the classifier and the taught class match the total number of teacher data of the teacher data. It is the ratio of the total number (correct answer rate).

When the method shown in FIG. 6 was applied to this data set, the results shown in Table 2 were obtained by repeating the teaching change and re-learning 266 times.

In the example shown in Table 2, three positive classes (Class 1/2/3) and three subclasses (Class 1? / 2? / 3?) That are pairs of these positive classes are defined. A classifier 84 for classifying into 6 types of classes has been obtained.

The recall (reproducibility) of Class 1/2/3, which is a positive class, is close to 100%. That is, when the classifier 84 answers the classification results as Class 1, Class 2, and Class 3, it can be said that the classifier 84 can be trusted with an accuracy of 93.25%, 96.03%, and 98.32%, respectively. And this Recall is significantly superior to the Recall of the early classifiers shown in Table 1.

Also, among the Precision of Class 1? / 2? / 3? Which is a subclass, only the value of Class2? Is not inferior to that of the positive class. In this case, the primary / subclass of Class2 may be integrated and determined as Class2. Further, for Class 1? / 3? With a relatively low Precision, these may be integrated and judged as "unclassifiable". Furthermore, only Class 3 ?, whose Precision is smaller than Class 1 ?, may be determined as "unclassifiable". In this case, if the classification destination is Class 1 ?, a determination result with a low certainty such as "may be Class 1" may be output.

The height of the Precision of Class 2? Suggests that a group of images that are correctly classified as Class 2? By teaching Class 2? (That is, the teaching is changed) may form a new class Class 4. .. Therefore, it is conceivable to newly define a class with high Precision as a positive class. For example, a subclass in which Precision exceeds a predetermined threshold value may be defined as a new positive class, and the teaching class of the teacher data 82 classified into the subclass may be changed to a new positive class. Then, the generation process of the classifier 84 (step S22 in FIG. 6) may be performed using the teacher data group 82G including the changed teacher data 82. By regenerating the classifier 84 in this way, a higher performance classifier 84 may be obtained.

(Example 2)
Table 3 shows an example in which the images of cell masses (spheroids / organoids) taken during the efficacy test of a new drug are classified into two types, "Alive" and "Dead", by the linear kernel SVM. In this example, the reliability of the teaching is not so high. This is because all cell masses in the wells to which no drug was added to the culture medium are regarded as alive, and conversely, all cell masses in the wells to which a high concentration drug is added are regarded as dead. This is because the actual life and death of the cell is given without confirmation. In this example, it is not the number of the classification result itself, but the problem is that the image of the cell mass judged to be dead by the classifier contains something that is clearly alive.

When the method shown in FIG. 6 was applied to this data set, the following results were obtained by repeating the teaching change and re-learning 40 times.

As shown in Table 4, the recall of Alive / Dead, which is a positive class, is 100%, and the Precision of Alive is also a high value of 99.95%. On the other hand, 115 of the data whose teachings have been changed to the subclass Alive? Are classified into the positive class of Dead, but the important thing is that there is no data that is clearly alive. be.

Before applying this method, as shown in Table 3, data with 282 Alive labels were mixed in Dead, 60 to 70% of them were truly misclassified, and at least 10 were "conspicuous". It was a thing. On the other hand, in this method, less than half of the above 115 misclassifications are conspicuous, and it is considered that they are working effectively.

Although the invention has been described in detail, the above description is exemplary in all aspects and the invention is not limited thereto. It is understood that innumerable variations not illustrated can be assumed without departing from the scope of the present invention. Each configuration described in each of the above-described embodiments and modifications can be appropriately combined or omitted as long as they do not conflict with each other.

5 Host computer (classifier generator)
9 Semiconductor substrate 82 Teacher data 82G Teacher data group 84 Classifier 611 Feature amount calculation unit 613 Classifier generation unit 615 Teaching class change unit 617 End judgment unit 822 Teaching class 822 Feature quantity vector 824 Teaching class

Claims (6)

It is a classifier generation method that generates a classifier that classifies data according to features.
(A) A step of calculating a plurality of types of features for each of a plurality of teaching data, and
(B) A step of generating a teacher data group including a plurality of teacher data by teaching one of different positive classes for each of the plurality of teaching data.
(C) A step of generating a classifier using the teacher data group and
(D) A step of classifying the teacher data group with the classifier generated by the step (c), and a step of classifying the teacher data group.
(E) Of the teacher data group, for teacher data whose classification destination is different from the teaching in the step (d), a step of changing the teaching class to a subclass which is a pair of the original positive class.
(F) A step of generating a new classifier using a teacher data group including teacher data in which the teaching class is changed to the subclass by the step (e).
(G) A step of classifying the teacher data group with the classifier generated by the step (f), and a step of classifying the teacher data group.
(H) For the teacher data whose classification destination is different from the teaching in the step (g), if the teaching is a positive class, the teaching class is changed to a subclass which is a pair thereof, and the teaching is a subclass. If the classification destination is a positive class that is a pair of its subclasses, the process of changing the teaching class to the positive class, and
(I) A step of generating a new classifier using a teacher data group including teacher data whose teaching class is changed in the step (h).
How to generate a classifier, including. The method for generating a classifier according to claim 1.
(J) A step of re-performing the steps (g) to (i).
A method for generating a classifier. The method for generating a classifier according to claim 2.
The step (j) is
A method for generating a classifier, which is a step of performing the steps (g) to the step (i) twice or more until a predetermined end condition is satisfied. The method for generating a classifier according to claim 3.
A method for generating a classifier, wherein the termination condition is that the teaching class does not change in the step (h). The method for generating a classifier according to claim 3.
A method for generating a classifier, wherein the end condition is that the correct answer rate for the positive class exceeds a predetermined threshold value in the classification result of the step (g). A classifier generator that generates a classifier that classifies data according to features.
A classifier generator that generates a classifier using the teacher data group taught by one of the different classes,
A teaching class changing unit that changes the teaching class of the teacher data between a normal class and a subclass that is a pair thereof according to the classification destination of the teacher data by the classifier.
Equipped with
The teaching class change part is
When the teacher data group is classified by the classifier, the teacher data whose classification destination by the classifier is different from the teaching is
If the teaching is a positive class, change the teaching class to a subclass that is a companion to the positive class.
If the teaching is a subclass and the classification destination is a positive class that is a pair of the subclass, the teaching class is changed to the positive class.
The classifier generator is a classifier generator that generates a new classifier using a teacher data group including teacher data whose teaching class has been changed by the teaching class changing unit.

Images