JP6063756B2 - Teacher data creation support device, teacher data creation device, image classification device, teacher data creation support method, teacher data creation method, and image classification method - Google Patents

Description

  The present invention relates to a technique for supporting creation of teacher data used for learning of a classifier that classifies images, a technique for creating teacher data using the technique, and a technique for classifying images.

  In the manufacture of semiconductor substrates, glass substrates, printed wiring boards, and the like, appearance inspection is performed using an optical microscope, a scanning electron microscope, or the like in order to inspect defects such as foreign matters, scratches, and etching defects. In addition, the cause of the defect is specified by performing detailed analysis on the defect detected in such an inspection process, and countermeasures against the defect are taken. In recent years, as the pattern on the substrate becomes more complex and finer, the types and quantities of detected defects tend to increase, and automatic classification that automatically classifies defects detected in the inspection process is also used. Automatic classification enables defect analysis to be performed quickly and efficiently. In automatic classification, a classifier using a neural network, a decision tree, discriminant analysis, or the like is used. In order for the classifier to perform automatic classification, it is necessary to prepare the teacher data including a signal indicating the defect image and its category (that is, the type of the defect image) and to learn the classifier ( For example, see Patent Document 1). Typically, teacher data is created by the operator determining the category of each defect image.

  If the teacher data includes, for example, a defect image to which the wrong category is assigned, the classifier is generated so as to classify the defect image into the wrong category, which may reduce the classification performance. . Therefore, cross-validation is performed on the teacher data, and the validity of the teaching of the teacher data is evaluated by predicting the performance of the classifier generated from the teacher data. For example, in Patent Document 2, one of N groups obtained by dividing a plurality of previously classified defects (defect images) is selected as a classification group, and the remaining N-1 groups are selected as teaching groups. A classification standard is set based on the defects belonging to, and the defect belonging to the classification group is classified according to the set classification standard. The above processing is performed N times while changing the classification group to another group, and classification results of all defects are obtained (so-called K-division cross validation is performed). Thereby, the classification | category accuracy rate at the time of using the said several defect is obtained.

JP 2003-317083 A JP 2004-295879 A

  By the way, according to the method of Patent Document 2 (K-division cross-validation), each group needs to include a certain number of defect images in order to avoid a bias in the types of defect images when dividing into N groups. However, when the number of defect images serving as a population is small, the number of groups N is limited, and the classification performance (the validity of teaching of teaching data) cannot be accurately predicted. In addition, in order to generate a high-performance classifier, it is important to handle a defect image classified into an incorrect category in the examination of teacher data. Therefore, there is a need for a new technique for easily and appropriately creating teacher data while taking the above contents into consideration.

  The present invention has been made in view of the above problems, and an object thereof is to easily and appropriately create teacher data.

The invention according to claim 1 is a teacher data creation support device that supports creation of teacher data used for learning of a classifier that classifies images, each of which is assigned to one of a plurality of initial categories. A storage unit for storing a plurality of images and initial data indicating an initial category of the plurality of images, and selecting a predetermined ratio of the plurality of images as a test image group, and learning using the remaining images The selection classification process while changing the image included in the test image group, with the process of acquiring the classification category of each image included in the test image group by the test classifier generated by performing the selection as the selection classification process and a test classifying unit repeated multiple times, an input unit that receives a combination of the input selection of one initial category and a class category, assigned to the one of the initial category It is, and, and a display control unit for displaying at least one image is classified at least once to the classification categories of the one in the test classification unit on the display unit, the display control unit, of the at least one image for each, Ru displayable der frequency of misclassification on the display unit and the classification category by the selected sorting process and the initial category are different.

The invention according to claim 2 is the teacher data creation support apparatus according to claim 1 , wherein the display control unit determines the frequency of each of the plurality of classification categories for each of the at least one image. It can be displayed on the display unit.

A third aspect of the present invention is the teacher data creation support apparatus according to the first or second aspect , wherein the set of images assigned to the same initial category in the plurality of images is used as the image set, and the test classification is performed. The test image group is configured by selecting the predetermined ratio of images from each of a plurality of image sets for the plurality of initial categories.

The invention according to claim 4 is a teacher data creation device for creating teacher data used for learning of a classifier that classifies images, and the teacher data creation support device according to any one of claims 1 to 3 And an initial data correction unit for correcting the initial data in accordance with an input from the operator.

The invention according to claim 5 is an image classification device for classifying images, and learning is performed using the teacher data creation device according to claim 4 and the teacher data created by the teacher data creation device. And a classifier for classifying the images.

The invention according to claim 6 is a teacher data creation support method for supporting creation of teacher data used for learning of a classifier that classifies images, and a) each of which is assigned to one of a plurality of initial categories. Preparing a plurality of images and initial data indicating an initial category of the plurality of images; b) selecting a predetermined percentage of the plurality of images as a test image group, and selecting the remaining images The process of acquiring the classification category of each image included in the test image group by the test classifier generated by performing learning using as a selection classification process, while changing the image included in the test image group, the selection sorting process comprising the steps of repeating multiple times, a step of accepting input of selection of the combination of the initial category and a class categories of c) one, d) on the one of the initial category Ri is swung, and the b) and a step of displaying at least one image is classified at least once to the classification categories of the one on the display unit in step, in the step d), of the at least one image for each frequency of misclassification of the classification categories according to the selected sorting process and the initial category is differences Ru displayable der on the display unit.

The invention according to claim 7 is the teacher data creation support method according to claim 6 , wherein, in the step d), the frequency of each of the plurality of classification categories is set for each of the at least one image. It can be displayed on the display unit.

The invention according to claim 8 is the teacher data creation support method according to claim 6 or 7 , wherein a set of images assigned to the same initial category in the plurality of images is used as the image set, and b) In the step, the test image group is configured by selecting the predetermined ratio of images from each of the plurality of image sets for the plurality of initial categories.

The invention according to claim 9 is a teacher data creation method for creating teacher data used for learning of a classifier that classifies images, and the teacher data creation support method according to any one of claims 6 to 8 And a step of correcting the initial data.

The invention according to claim 10 is an image classification method for classifying images, the step of learning a classifier using the teacher data created by the teacher data creation method according to claim 9 , and the classifier And a step of classifying the images.

  According to the present invention, it is possible to accurately predict the classification performance in the initial data. In addition, by displaying an image related to a combination of one initial category and one classification category, it is possible to easily determine the suitability of the initial category of the image, eligibility as a teacher image, and the like. As a result, teacher data can be created easily and appropriately.

It is a figure which shows the structure of an image classification device. It is a figure which shows the flow of a classification | category of a defect image. It is a figure which shows the structure of a host computer. It is a figure which shows the function structure of a host computer. It is a figure which shows the function structure of a teacher data creation part. It is a figure which shows the flow of the process which produces teacher data and learns a classifier. It is a figure which shows the flow of the process which produces teacher data and learns a classifier. It is a figure which shows the window displayed on a display. It is a figure which shows the window displayed on a display by the method of a comparative example.

  FIG. 1 is a diagram showing a schematic configuration of an image classification apparatus 1 according to an embodiment of the present invention. In the image classification apparatus 1, a defect image indicating a defect on a semiconductor substrate 9 (hereinafter simply referred to as “substrate 9”) is acquired, and the defect image is classified. The image classification device 1 picks up a defect into a category to which the defect should belong when the defect is detected based on the image data from the image pickup device 2 and the image pickup device 2 that picks up the inspection target area on the substrate 9. It has an inspection / classification device 4 for automatic classification, and a host computer 5 that controls the overall operation of the image classification device 1 and generates a classifier 421 used for defect classification in the inspection / classification device 4. The types (categories) of defects present on the substrate 9 are, for example, defects, protrusions, disconnections, shorts, and foreign matters. In addition, the imaging device 2 is incorporated in the production line of the substrate 9, and the image classification device 1 is a so-called inline system. The image classification device 1 can also be regarded as a device in which a function of automatic defect classification is added to a defect inspection device.

  The imaging device 2 captures an inspection target area on the substrate 9 to acquire image data, a stage 22 that holds the substrate 9, and a stage that moves the stage 22 relative to the imaging unit 21. A drive unit 23 is included. The imaging unit 21 electrically outputs an illumination unit 211 that emits illumination light, an optical system 212 that guides the illumination light to the substrate 9 and receives light from the substrate 9, and an image of the substrate 9 formed by the optical system 212. An imaging device 213 that converts the signal into a signal is included. The stage driving unit 23 includes a ball screw, a guide rail, a motor, and the like, and the inspection area on the substrate 9 is imaged by the host computer 5 controlling the stage driving unit 23 and the imaging unit 21.

  The inspection / classification apparatus 4 includes a defect detection unit 41 that detects defects while processing image data of an inspection target area, and an automatic defect classification unit 42 that classifies defect images. The defect detection unit 41 has a dedicated electric circuit for processing image data of the inspection target area at high speed, and performs defect inspection of the inspection target area by comparing the captured image with a reference image having no defect or by image processing. Do. The automatic defect classification unit 42 includes a CPU that performs various arithmetic processes, a memory that stores various information, and the like, and classifies defects (that is, defect images) using a classifier 421 that uses a neural network, a decision tree, discriminant analysis, and the like. Classification).

  FIG. 2 is a diagram showing a flow of defect image classification by the image classification apparatus 1. First, when the imaging device 2 shown in FIG. 1 images the substrate 9, the defect detection unit 41 of the inspection / classification device 4 acquires image data (step S11). Next, when the defect detection unit 41 performs the defect inspection of the inspection target region and a defect is detected (step S12), the image of the defective portion (that is, the defect image) is transmitted to the automatic defect classification unit 42. The The automatic defect classification unit 42 calculates values of a plurality of types of feature values of the defect image (step S13), and the feature value values of the defect image are input to the classifier 421 of the automatic defect classification unit 42 and the classification result is output. The That is, the defect image is classified into one of a plurality of categories by the classifier 421 (step S14). In the image classification device 1, each time a defect is detected by the defect detection unit 41, the feature value is calculated in real time, and a large number of defect images are automatically classified at high speed.

  Next, classifier learning by the host computer 5 will be described. FIG. 3 is a diagram showing the configuration of the host computer 5. The host computer 5 has a general computer system configuration in which a CPU 51 that performs various arithmetic processes, a ROM 52 that stores basic programs, and a RAM 53 that stores various information are connected to a bus line. The bus line further includes a fixed disk 54 that stores information, a display 55 that is a display unit that displays various types of information such as images, a keyboard 56a and a mouse 56b that accept input from an operator (hereinafter referred to as "input unit 56"). A signal is transmitted / received between the reading device 57 for reading information from the computer-readable recording medium 8 such as an optical disk, a magnetic disk, a magneto-optical disk, and the other configuration of the image classification device 1. The communication unit 58 to be connected is appropriately connected through an interface (I / F) or the like.

  In the host computer 5, the program 80 is read in advance from the recording medium 8 via the reading device 57 and stored in the fixed disk 54. Further, the program 80 is copied to the RAM 53 and the CPU 51 follows the program in the RAM 53. Arithmetic processing is executed.

  FIG. 4 is a block diagram showing a functional configuration for learning the classifier realized by the CPU 51, ROM 52, RAM 53, fixed disk 54, etc. of the host computer 5, and also shows the inspection / classification device 4. The host computer 5 includes a teacher data creating unit 61 that creates teacher data used for learning the classifier, and a learning unit 62 that learns the classifier using the teacher data.

  The teacher data includes teacher image data that is a defect image, a feature value of the teacher image, and a teaching signal that is information indicating the category of the defect. Examples of the feature amount of the teacher image include, for example, the defect area and brightness. The average, circumference, flatness, inclination of the major axis when the defect is approximated to an ellipse, and the like are employed. In the learning unit 62, the feature value of the teacher image read from the teacher data is input to a classifier (not shown) in the host computer 5, and the output of the classifier is the same as the teaching signal indicating the defect category. Learning is performed, and the learning result, that is, the learned classifier 421 (more precisely, information indicating the structure of the classifier 421 and the value of the variable) is transferred to the automatic defect classification unit 42.

  FIG. 5 is a block diagram showing a functional configuration of the teacher data creation unit 61 of the host computer 5, and also shows a learning unit 62. The teacher data creation unit 61 includes a data calculation unit 610, a display 55, and an input unit 56. The data calculation unit 610 includes a test classification unit 611, a display control unit 612, a storage unit 613, an initial data correction unit 614, and a feature amount calculation unit 615. Details of the processing of the data calculation unit 610 will be described later. Note that the functions of the data calculation unit 610 and the learning unit 62 may be constructed by a dedicated electric circuit, or a dedicated electric circuit may be partially used.

  6 and 7 are diagrams showing a flow of processing for creating teacher data and learning a classifier. In the teacher data creation unit 61, first, data of a large number of teacher images (for example, thousands of teacher images), which are defect images for creating teacher data, are stored and prepared in the storage unit 613 shown in FIG. Step S21). The teacher image may be acquired using the imaging device 2 and the defect detection unit 41 illustrated in FIG. 1 or may be separately prepared.

  The feature amount calculation unit 615 calculates a plurality of types (for example, 100 to 200 types) of feature amounts of all the teacher images (step S22). The calculated feature value is stored in the storage unit 613. Further, each teacher image is displayed on the display 55 and a display prompting the user to input a category of the teacher image is displayed, and the input of the category by the operator is accepted by the input unit 56 (step S23). In the following description, the category input (teached) by the operator for each teacher image in the process of step S23 is referred to as “initial category”. Note that the initial category of each teacher image may be automatically determined using a plurality of types of feature amounts by a predetermined algorithm.

  In this way, a plurality of teacher images each assigned to one of a plurality of initial categories and initial data 710 indicating the initial categories of the plurality of teacher images are prepared and stored in the storage unit 613. . In the present embodiment, the initial data 710 also includes a plurality of types of feature amounts in each teacher image. Also, a number is assigned to each of a plurality of types of categories, and the value of the categorical variable associated with the teacher image is changed to the number of the initial category by determining the initial category for each teacher image.

  When the initial data 710 is prepared, the test classification unit 611 has a predetermined ratio (for example, 30%, hereinafter referred to as “test image ratio”) of a plurality of teacher images included in the initial data 710. Is selected as a test image group (step S24). At this time, a set of images assigned to the same initial category in the plurality of teacher images is used as an image set, and an image having the test image ratio is randomly selected from each of the plurality of image sets for the plurality of initial categories. Thus, a test image group is configured. The test image ratio is, for example, 1/3 or less.

  The remaining images of the plurality of teacher images are set as a training image group, and the test classification unit 611 generates a test classifier by performing learning using a plurality of types of feature amounts of the training image group (step S25). ). The generation of the test classifier is the same as the generation of the classifier 421 described later in the learning unit 62, so that the output of the test classifier for each teacher image in the training image group is the same as the initial category of the teacher image. Learning is performed and a test classifier is generated. The test classifier may be generated by the learning unit 62. In this case, the learning unit 62 may be regarded as a part of the test classification unit 611.

  Subsequently, each teacher image included in the test image group is classified into any category by the test classifier (step S26). That is, a classification category of a plurality of teacher images included in the test image group is acquired with the category classified by the test classifier as a classification category. The classification category for each teacher image in the test image group is stored in the storage unit 613. A category indicating unclassified (or training image group) is assigned to each teacher image in the training image group.

  The selection of the test image group (random selection), the generation of the test classifier, and the classification of the test image group in steps S24 to S26 are repeated a number of times (for example, 100 times) designated in advance by the operator (for example, 100 times). Step S27). At this time, every time the test image group is classified, the classification category for each teacher image is stored.

  As described above, the test classification unit 611 selects a test image group from a plurality of teacher images, and selects a process for acquiring a classification category of the test image group using a test classifier generated using the remaining teacher images. As the classification process, the selection classification process is repeated many times while changing the images included in the test image group. The number of repetitions of the selection classification process is preferably the number of times that the average classification number of one teacher image is 2 or more (more preferably 10 or more, and still more preferably 30 or more).

  When the repetition of the designated number of times of the selected classification process is completed (step S27), the display control unit 612 displays the confusion matrix (also referred to as a classification table or a confusion comparison table) of Table 1 indicating the classification result on the display 55 ( Step S28).

  In Table 1, three types of initial categories are described in the row header as “initial category A”, “initial category B”, and “initial category C”, and the three types of classification categories are “classification category A” and “classification category B”. , “Category category C” is written in the column heading. In Table 1, for example, among the teacher images taught by the operator as the initial category A, the total number of teacher images classified as the classification category A by the test classifier in the above-described many selection classification processes is 45359. It is shown that. The row marked “Sum” in the heading indicates the total number of teacher images classified into each classification category, and the row marked “Purity” in the heading represents the total number of teacher images classified in each classification category. The ratio of the total number of teacher images whose initial category matches the classification category is indicated (the same applies to the column labeled “Sum” and the column labeled “Accuracy”). Further, the cell where the “Purity” row and the “Accuracy” column intersect is the ratio of the total number of teacher images in which the initial category and the classification category match among the total number of classified teacher images (that is, This is the correct answer rate and the classification performance in the initial data.

  Subsequently, the input unit 56 receives an input for selecting a combination of one initial category and one classification category (FIG. 7: step S29). The input for selecting the combination is performed by, for example, selecting a cell in which a row of one initial category and a column of one classification category intersect in the confusion matrix displayed on the display 55. In the following description, the initial category and the classification category in the selected combination are referred to as “selected initial category” and “selected classification category”, respectively.

  When input of the selection initial category and the selection classification category is received, a teacher image (hereinafter referred to as “target image”) assigned to the selection initial category and classified into the selection classification category at least once in the multiple selection classification processing. Is displayed on the display 55 by the display control unit 612 (step S30).

  FIG. 8 is a diagram showing a window displayed on the display 55. A confusion matrix is displayed in the table display area 71 in the window, and it is shown that the one cell is selected by making the color of one cell different from the color of other cells in the confusion matrix. An image display area 72 is provided above the table display area 71, and a plurality of target images 721 are displayed in the image display area 72.

  In FIG. 8, the initial category A and the classification category B are selected as the selection initial category and the selection classification category, respectively, and the image display area 72 is set to 1 by the test classifier although the initial category A is taught. The teacher image that has been erroneously classified into the classification category B is displayed as the target image 721 in the image display area 72.

  Subsequently, when an input for selecting one target image 721 from the plurality of target images 721 is received by the input unit 56 (step S31), the selected target image 721 (hereinafter referred to as “selection target image 721”). On the other hand, the respective frequencies of the plurality of classification categories acquired by the multiple selection classification processes are displayed on the display 55 (step S32). In the example of FIG. 8, a classification information display area 73 is provided below the table display area 71, and the frequency of a plurality of classification categories in the selection target image 721 (that is, how many times the categories are classified) Along with the value, it is displayed in the classification information display area 73 by a bar graph. As described above, since the selection target image 721 is a teacher image that teaches the initial category A, the total frequency of the classification category B and the classification category C in the classification information display area 73 is the sum of the initial category and the number of times described above. The frequency of misclassification that differs from the classification category by the selected classification process is shown. That is, the classification information display area 73 substantially displays the frequency of misclassification in the selection target image 721.

  The frequency of each classification category in the target image 721 displayed in the image display area 72 and the selection target image 721 displayed in the classification information display area 73 is confirmed by the operator, and the initial category of the target image 721 needs to be changed. In this case (step S33), the input unit 56 receives an input specifying the target image 721 and indicating the changed initial category. In the initial data correction unit 614, the initial category of the target image 721 in the initial data 710 is changed (that is, the value of the categorical variable is changed) according to the input of the operator, and the initial data 710 is corrected (step S34). In the process of step S <b> 34, for example, the target image 721 in which the frequency of two different classification categories is relatively high (that is, the category is misleading) may be excluded from the initial data 710.

  As described above, the initial data 710 is changed as necessary by the operator while changing the combination of the selection initial category and the selection classification category for displaying the target image 721 and the selection target image 721 for displaying the frequency of misclassification. It is corrected (steps S35, S29 to S34). Thereby, a final category is determined for each teacher image.

  When the final category of each teacher image is determined (step S35), the data of all the teacher images and the final initial data 710 including information on these feature amounts and categories are stored as teacher data. The transfer is transferred from 613 to the learning unit 62, and the learning unit 62 learns the classifier using the teacher data (step S36). That is, the values of variables constituting the classifier (see FIG. 4) are determined, and the structure is determined to generate the classifier 421. Thereby, the processing of FIGS. 6 and 7 for creating the teacher data and learning the classifier is completed.

  Here, a method of a comparative example in evaluating the validity of the teaching of the initial data (that is, the classification performance in the initial data) will be described. In the method of the comparative example, a test classifier is generated using all the teacher images, and all the teacher images are classified by the test classifier. Therefore, the method of the comparative example is hereinafter referred to as “all-number learning—all-number classification”. Call it. Table 2 shows a confusion matrix which is a classification result when exhaustive learning-exhaustive classification is used for a plurality of teacher images included in the same initial data as when the confusion matrix of Table 1 is acquired.

  As shown in Table 2, among the 1578 teacher images taught by the operator as the initial category A, the number of teacher images classified as the classification category A is 1498, and the number of teacher images classified as the classification category B 79, the number of teacher images classified as classification category C is 1, and the correct answer rate (accuracy of classification) for the 1578 teacher images is 94.9%. In addition, the 1744 teacher images classified as the classification category A by the test classifier in exhaustive learning and exhaustive classification are 246 taught as the initial category B in addition to the 1498 teacher images taught as the initial category A by the operator. The teacher images are included, and the correct answer rate (reliability) for the 1744 teacher images is 85.9%.

  In exhaustive learning-exhaustive classification, when a plurality of teacher images in the initial data include all types of defects, the classification results of the defective images (classification performance) when the classifier is generated using the initial data as the teacher data Can be accurately predicted. However, it is not easy to include all kinds of defects in the initial data, and the classification performance in the initial data may not be accurately predicted by exhaustive learning- exhaustive classification.

  Further, when the validity of teaching of initial data is evaluated using K-division cross validation, the number of groups to be divided is limited when the number of teacher images as a population (the number of samples) is small. It becomes impossible to predict the performance (the validity of the teaching of the initial data) with high accuracy. Furthermore, in a method (leave-one-out cross-validation (LOOCV)) in which only one teacher image is extracted from initial data and used as a test image, and the rest is a training image group, the number of teacher images included in the initial data and It is necessary to repeat the generation of the test classifier the same number of times, which takes a lot of time.

  On the other hand, the test classification unit 611 of the image classification device 1 selects a predetermined ratio of teacher images among a plurality of teacher images as a test image group, and the test classifier generated using the remaining teacher images performs the test. The process of acquiring the classification category of each teacher image included in the image group is the selection classification process, and the selection classification process is repeated many times while changing the image included in the test image group. Then, by acquiring the confusion matrix indicating the classification result, the classification performance (correct answer rate) of the defect image when the classifier is generated using the initial data as the teacher data, that is, the classification performance in the initial data It is possible to predict accurately and easily without being affected by the number of teacher images in the data. In addition, it is possible to grasp a tendency such as which classification category the teacher image of each initial category is easily misclassified.

  Here, in the exhaustive learning-exhaustive classification, each teacher image is classified only once by the test classifier. Therefore, as shown in FIG. 9, for example, although the initial category A is taught, the test classifier Thus, it is possible to easily display the teacher image misclassified into the classification category B. The same applies when using K-division cross-validation or LOOCV.

  On the other hand, when the selection classification process is repeated many times, the number of classifications by the test classifier differs for each teacher image, and each teacher image is not necessarily classified into the same classification category every time. Therefore, when assuming a device in which the display control unit 612 is omitted, in order to identify the teacher image misclassified as described above in such a device, each classification result of the multiple selection classification processing is performed. Are stored, and the operator himself / herself has to confirm the classification results in order, which requires extremely complicated work.

  On the other hand, in the image classification device 1, when the input unit 56 receives an input for selecting one initial category and one classification category combination, the image classification device 1 is assigned to the initial category and the test classification unit 611. The display control unit 612 displays the target image 721 classified at least once in the classification category in FIG. Then, by referring to the target image 721, the operator can easily determine whether the initial category of the target image 721 is appropriate, eligibility as a teacher image, and the like.

  Further, unlike the exhaustive learning- exhaustive classification, K-division cross validation, and LOOCV in which each teacher image is classified only once by the test classifier, the image classification apparatus 1 repeats the selection classification process many times, Various misclassification examples (for example, misclassification that a person cannot understand) are acquired, and the teacher image is displayed. Therefore, the initial data can be corrected so as to correspond to the various misclassification examples such as providing a new category, and as a result, the teacher data can be created easily and appropriately.

  Further, the image classification apparatus 1 can display the frequency of misclassification in which the initial category and the classification category are different on the display 55 for each target image 721. The frequency of misclassification (that is, whether misclassification is frequent, rare, or not misclassified at all) can be said that a teacher image taught in a certain initial category clearly belongs to that category. It is important information for optimizing teaching because it indicates whether it is a thing or an approximation to the feature of another category. Therefore, by displaying the frequency of misclassification, it is possible to more easily and appropriately determine whether or not the initial category of the target image 721 is appropriate. Further, even if the operator mistakenly teaches the initial category of the target image 721 in the process of step S23 in FIG. 6, by confirming the frequency of misclassification, the initial category of the target image 721 is determined. Can be easily corrected. In exhaustive learning-exhaustive classification, K-division cross-validation, and LOOCV, each teacher image is classified only once by the test classifier, so the frequency of misclassification cannot be acquired.

  Furthermore, the frequency of each of a plurality of classification categories can be displayed for each target image 721. As a result, in which category the target image 721 is likely to be misclassified (whether the category of the teacher image is similar to the feature amount), or how misleading the target image 721 is, etc. It is possible to intuitively present information about the to the operator. This makes it possible to efficiently perform the work of optimizing the category of each teacher image. In addition, since it becomes difficult for teacher data to include teacher images associated with contradictory categories in the feature amount space, it is expected that a learning result (classifier) using such teacher data has high classification performance.

  The image classification apparatus 1 described above can be variously modified. If, in principle, the combination of teacher images included in the test image group is different every time in the multiple selection and classification processes in the test classification unit 611, a predetermined rule is determined from a plurality of teacher images included in the initial data 710. The test image group may be selected in (for example, selecting a number of teacher images that are relatively prime to the number of the plurality of teacher images in turn as the test image group). In order to appropriately generate the test classifier, it is preferable that a predetermined ratio of images is selected from each of a plurality of image sets for a plurality of initial categories.

  On the display 55, all the target images 721 corresponding to the selected initial category and the selected classification category are not necessarily displayed, and only one target image 721 may be displayed. In this case, the target image 721 to be displayed is sequentially switched by the operator's input. As described above, in the process of step S <b> 30 of FIG. 7, at least one target image 721 is displayed on the display 55.

  Information indicating the frequency of misclassification may be displayed in the image information area 722 immediately below the target image 721 in the image display area 72 of FIG. In this case, it is preferable that the frequency of misclassification of the target image 721 is displayed in each image information area 722 of all target images 721. Further, when the mouse cursor is held over one target image 721 in the image display area 72, the classification information display area may be displayed in a pop-up (see the area denoted by reference numeral 74 in FIG. 8). Furthermore, the frequency of misclassification may be substantially presented by displaying only the value of the wrong answer rate (or correct answer rate).

  In the image classification device 1, feature value values of a large number of teacher images used for teacher data may be calculated by the inspection / classification device 4. Further, the feature value may be omitted from the teacher data, and the learning unit 62 may obtain the feature value based on the teacher image data.

  In the above-described embodiment, the image classification apparatus 1 in which the function of automatic defect classification is added to the defect inspection apparatus has been described. However, an observation apparatus that observes defects on the substrate detected by other defect inspection apparatuses (both review apparatuses) The teacher data creation unit 61 may be used in a device in which a function of automatic defect classification is added to (referred to as an image classification device). In such an image classification device, an image with a higher resolution than that of the image pickup device 2 in FIG. 1 is acquired in order to analyze defects more highly.

  In the image classification apparatus 1 of FIG. 1, inspection of a glass substrate (for example, a glass substrate for a flat display device), a printed wiring board, or a mask substrate used for exposure of the substrate may be performed instead of the semiconductor substrate.

  Further, the image classification device 1 may be used for the purpose of classifying a cell image obtained by imaging cells in a predetermined liquid such as blood or a culture solution. As described above, the image classification device 1 can be used to classify images showing various objects. Furthermore, in the image classification device 1, in addition to images captured with visible light, images captured with an electron beam, an X-ray, or the like may be classified.

  In the above embodiment, the teacher data creation support apparatus that appropriately supports the creation of teacher data includes the test classification unit 611, the display control unit 612, the storage unit 613, and the input unit 56 provided in the teacher data creation unit 61. However, the teacher data creation support device may be realized as a dedicated device separated from the image classification device 1. The same applies to the teacher data creation device that mainly includes the teacher data creation support device and the initial data correction unit 614.

  The configurations in the above-described embodiments and modifications may be combined as appropriate as long as they do not contradict each other.

DESCRIPTION OF SYMBOLS 1 Image classification device 55 Display 56 Input part 421 Classifier 611 Test classification part 612 Display control part 613 Storage part 614 Initial data correction part 710 Initial data 721 Target image S14, S21-S36 Step

Claims (10)

A teacher data creation support device that supports creation of teacher data used for learning of a classifier that classifies images,
A plurality of images each assigned to one of a plurality of initial categories, and a storage unit for storing initial data indicating initial categories of the plurality of images;
A predetermined category of the plurality of images is selected as a test image group, and the classification category of each image included in the test image group is determined by a test classifier generated by performing learning using the remaining images. as the selection classification process the process of acquiring, said while changing the images included in the test image group, the test classification unit to repeat multiple times the selection classification process,
An input unit that accepts input of selection of a combination of one initial category and one classification category;
A display control unit that displays on the display unit at least one image that is assigned to the one initial category and is classified into the one classification category at least once by the test classification unit;
Equipped with a,
Wherein the display control unit, wherein for each of the at least one image to the classification category by the selected sorting process and the initial category, wherein displayable der Rukoto on the display unit the frequency of misclassification of differences Teacher data creation support device. The teacher data creation support device according to claim 1 ,
The teacher data creation support apparatus, wherein the display control unit can display the frequency of each of a plurality of classification categories on the display unit for each of the at least one image. The teacher data creation support device according to claim 1 or 2 ,
A set of images assigned to the same initial category in the plurality of images is set as an image set, and the test classification unit selects the predetermined ratio of images from each of the plurality of image sets for the plurality of initial categories. Thus, the test data group is constituted by the teacher data creation support apparatus. A teacher data creation device for creating teacher data used for learning of a classifier that classifies images,
A teacher data creation support device according to any one of claims 1 to 3 ,
An initial data correction unit that corrects the initial data according to an operator's input;
A teacher data creation device comprising: An image classification device for classifying images,
A teacher data creation device according to claim 4 ,
Learning is performed using the teacher data created by the teacher data creation device, and a classifier that classifies images;
An image classification apparatus comprising: A teacher data creation support method for supporting creation of teacher data used for learning of a classifier that classifies images,
a) preparing a plurality of images each assigned to one of a plurality of initial categories, and initial data indicating an initial category of the plurality of images;
b) Classification of each image included in the test image group by a test classifier generated by selecting a predetermined ratio of the plurality of images as a test image group and performing learning using the remaining images. as the selection classification process the process for acquiring the category, while changing the image included in the test images, a step of repeating multiple times the selection classification process,
c) receiving input of selection of a combination of one initial category and one classification category;
d) displaying on the display unit at least one image assigned to the one initial category and classified at least once in the one classification category in the step b);
Equipped with a,
In the d) step, for each of the at least one image, the frequency of misclassification of the classification categories according to the selected sorting process and the initial category differs is characterized displayable der Rukoto on the display unit Teacher data creation support method. The teacher data creation support method according to claim 6 ,
In the step d), for each of the at least one image, the frequency of each of a plurality of classification categories can be displayed on the display unit. The teacher data creation support method according to claim 6 or 7 ,
A set of images assigned to the same initial category in the plurality of images is defined as an image set, and the predetermined ratio of images is selected from each of the plurality of image sets for the plurality of initial categories in the step b). The teacher image creation support method, wherein the test image group is configured by: A teacher data creation method for creating teacher data used for learning a classifier that classifies images,
A teacher data creation support method according to any one of claims 6 to 8 ,
Modifying the initial data;
A method for creating teacher data, comprising: An image classification method for classifying images,
Learning a classifier using the teacher data created by the teacher data creation method according to claim 9 ;
Classifying images by the classifier;
An image classification method comprising:

Images