JP6584250B2 - Image classification method, classifier configuration method, and image classification apparatus - Google Patents

Description

  The present invention relates to a technique for configuring a classifier by machine learning using a teacher image and automatically classifying images.

  As a technique for automatically classifying a plurality of images into a plurality of categories according to the degree of similarity between images, there is a technique using supervised learning. In this technique, the classifier is constructed by estimating the feature quantity distribution of each classification category from the feature quantity distribution of the teacher image collected in advance and obtaining the boundary between categories that minimizes misclassification. As typical examples, a linear discrimination method, a neural network, a support vector machine (SVM), and the like are known. For example, the technique described in Patent Document 1 automatically classifies defects such as semiconductor substrates and printed boards, and performs defect classification using a discriminant function generated by machine learning based on teacher data prepared in advance.

  In such an automatic classification technique, it is difficult to completely eliminate misclassification, and various techniques for improving the classification accuracy have been proposed.

JP 2003-317083 A

  An example of a technique for improving the classification accuracy is ensemble learning in which a plurality of weak classifiers are combined. For example, the AdaBoost algorithm is well known. In AdaBoost, generalization performance can be improved by adaptively updating the weights of cases used for learning. However, there are only a limited number of conventional learning algorithms that can weight such cases. It cannot be applied to any machine learning algorithm according to the purpose of classification.

  The present invention has been made in view of the above problems, and when a classifier using a teacher image for automatically classifying an image is configured, it is applied to an arbitrary machine learning algorithm to configure a classifier having excellent performance. It aims at providing the technology that can do.

  One aspect of the present invention is based on the result of machine learning using teacher data including information related to a teacher image represented by a feature vector of P (P is a natural number) dimension and a classification category assigned to the teacher image. An image classification method for automatically classifying images, and in order to achieve the above object, a machine using the teacher data corresponding to a plurality of the teacher images by assigning a unique identification number to each of a plurality of category categories A first step of configuring a classifier by executing a learning algorithm; a second step of classifying each of the teacher images into a plurality of classification categories by the classifier configured in the first step; and For each, the classification category assigned in the second step is compared with the classification category assigned in advance in the teacher data, and a numerical value corresponding to the result is determined as a new feature amount. The third step of increasing the number of dimensions of the feature vector of the teacher image by 1 is executed M (M is an integer of 2 or more) times in this order, and the dimension of the input vector is 1 M different classifiers are prepared, and in the third step, when the classification category assigned in the second step is different from the classification category given in advance in the teacher data, When the identification number of the classification category assigned in the second step is the new feature amount, and the classification category assigned in the second step matches the classification category assigned in advance in the teacher data Then, an irregular numerical value within a predetermined numerical range centering on the identification number of the classification category is set as the new feature amount, and P types of P-type feature vectors are formed for the classification target image. A fourth step of obtaining the collected amount and classifying the feature vector of the image into any of the classification categories by the classifier corresponding to the input vector of the same dimension as the feature vector, and the classification in the fourth step By adding the identification number of the classification category as a new feature quantity to the feature vector, the fifth step of increasing the number of dimensions of the feature vector of the image by 1 is executed M times in this order, and M sets of classification A result is acquired, and one classification result selected from the M classification results is output as a classification result of the image.

  According to another aspect of the present invention, there is provided a classifier configuration method for automatically classifying an image into a plurality of classification categories. In order to achieve the above object, each of the plurality of classification categories has a unique identification. Assigns a number, obtains teacher data including information on a plurality of teacher images represented by P (P is a natural number) -dimensional feature vector and a classification category assigned to each teacher image, and corresponds to the teacher image A first step of configuring a classifier by executing a machine learning algorithm using the teacher data and a classifier configured to classify each of the teacher images into a plurality of classification categories by the classifier configured in the first step. For each of the two teacher images, the classification category assigned in the second step is compared with the classification category assigned in advance in the teacher data, and according to the result By adding the value to the feature vector as a new feature quantity, the third step of increasing the number of dimensions of the feature vector of the teacher image by one is executed M (M is an integer of 2 or more) times in this order. , When M classifiers having different input vector dimensions are formed, and the classification category assigned in the second step is different from the classification category assigned in advance in the teacher data in the third step Includes the identification number of the classification category assigned in the second step as the new feature amount, and the classification category assigned in the second step matches the classification category assigned in advance in the teacher data. In this case, an irregular numerical value within a predetermined numerical range centering on the identification number of the classification category is set as the new feature amount.

  In the invention configured as described above, the teacher image itself is classified by the classifier learned using the teacher data, and the teacher image that is different from the previously assigned classification category, that is, the erroneous classification result is obtained. An identification number indicating an incorrect classification category is added to the feature vector as a new feature amount. On the other hand, if the classification result matches a classification category assigned in advance, an irregular numerical value within a predetermined numerical range centering on the numerical value of the identification number indicating the classification category is added to the feature vector as a new feature amount. Is done.

  When creating a new feature quantity and increasing the dimension of the feature vector, if the classification result is incorrect, the incorrect classification result is used as a new feature value as it is. The inventor of the present application has found that the result of misclassification effectively functions as a feature amount contributing to improvement of the classification performance by giving a slight fluctuation to the result to be a new feature amount value. While misclassification results are clearly propagated to later learning, correct classification results are propagated in a more ambiguous manner, and it is considered that the effect of weighting cases indirectly is obtained.

  As described above, the method of repeatedly performing learning while adding a new feature amount corresponding to the classification result to the feature vector of the teacher data can be applied to any learning algorithm. And by executing machine learning in this way, it becomes possible to configure a classifier with excellent classification performance, more precisely a generalization ability, and by using the classifier thus obtained, Classification accuracy can also be improved for unknown images.

  According to still another aspect of the present invention, there are provided an image acquisition means for acquiring a plurality of images to be a teacher image and an image to be classified, a classification means for classifying an image by executing the image classification method described above, or the above-described An image classification apparatus comprising: a classification unit that classifies images using a classifier configured by a classifier configuration method; and a notification unit that notifies a user of an output classification result. In the invention configured as described above, it is possible to classify various images with higher classification accuracy by classifying images by using the classifier having improved classification performance compared to the conventional method by the above method.

  As described above, according to the present invention, teacher data is classified using a classifier obtained by supervised learning, and a feature amount corresponding to the classification result is added to each teacher data to further perform learning. Thus, the classification performance of the configured classifier can be improved. This method can be applied to any machine learning algorithm, and can improve classification accuracy in automatic classification using the algorithm.

1 is a diagram showing a schematic configuration of an inspection system to which the present invention can be preferably applied. It is a flowchart which shows operation | movement of a defect detection apparatus. It is a flowchart which shows operation | movement of a defect classification device. It is a flowchart which shows the prior learning process performed by the defect classification device. It is a flowchart which shows the feature vector update process. It is a flowchart which shows the classification process with respect to an unclassified defect image. It is a figure which illustrates the effect of this embodiment.

  FIG. 1 is a diagram showing a schematic configuration of an inspection system to which the present invention can be preferably applied. This inspection system 1 is an inspection system that performs inspection of defects such as pinholes and foreign matters appearing on the appearance of a semiconductor substrate S to be inspected, and automatically classifies detected defects. The inspection system 1 includes a defect detection device 2 that detects a defect from an inspection target region on the substrate S, a defect classification device 3 that analyzes the defect detected by the defect detection device 2 in more detail and identifies the type of defect, and It has. The defect detection device 2 is incorporated in the production line of the substrate S, and the inspection system 1 is a so-called in-line system.

  The defect detection apparatus 2 includes an image acquisition unit 20 that acquires an image of an inspection target area on the substrate S, and a detection unit 21 that analyzes the image data from the image acquisition unit 20 and detects a defect. The image acquisition unit 20 has an imaging element such as a CCD sensor or a CMOS sensor, for example, and acquires an image data by acquiring an image of an inspection target area on the substrate S, a stage 202 that holds the substrate S, and A stage driving unit 203 that moves the stage 202 relative to the imaging unit 201 is provided. The stage driving unit 203 includes a ball screw, a guide rail, a motor, and the like. The imaging unit 201 and the stage driving unit 203 are controlled by a mechanism control unit 211 provided in the detection unit 21, and operate according to a control command from the mechanism control unit 211, so that an inspection target region on the substrate S is changed. Take an image.

  The defect detection unit 212 performs appropriate data processing on the image data obtained by imaging by the imaging unit 201 to detect defects. The defect detection unit 212 performs appropriate data processing such as noise removal processing or smoothing processing on the image data given from the imaging unit 201, and performs defect detection based on the image data thus processed. More specifically, when an image captured on the actual substrate S is compared with an image of a defect-free substrate stored in advance in the storage unit 213, and the difference exceeds a predetermined reference It is determined that the image includes a defect. The defect detection unit 212 transmits the detected defect image data to the defect classification device 3 via the interface (I / F) 214.

  The defect classification device 3 determines the type of the detected defect based on the defect data given from the defect detection device 2. That is, the defect classification device 3 has a function of automatically classifying defects (ADC: Automatic Defect Classification) into classification categories to which the defect belongs based on the defect image data given from the defect detection device 2. This defect classification device 3 is a device on which an embodiment of an image classification device according to the present invention is mounted. As described below, the function as the defect classification device 3 can be realized by executing a control program describing each process according to the present invention using hardware resources of a general personal computer. Is possible.

  The defect classification device 3 executes each control block shown in FIG. 1 by hardware and software by executing a control program read in advance, and executes various processes described later. The defect classification device 3 includes an interface (I / F) 31 for receiving defect image data from the defect detection device 2, and a defect classification unit (ADC) 32 that determines the type of the defect by automatic classification based on the defect image data. It has. The interface 31 can communicate with other external devices via, for example, a LAN (Local Area Network).

  The image data given from the defect detection device 2 and various processing data generated in the process in the defect classification unit 32 are stored in the storage unit 33 as needed. The defect classifying unit 32 classifies the defect image into a plurality of classifications set in advance corresponding to the defect type based on the values of the plurality of types of feature amounts calculated by the feature amount calculating unit 34 based on the given defect image data. Sort into one of the categories.

  Furthermore, the defect classification device 3 includes an input receiving unit 35 such as a keyboard and a mouse that receives an operation input from a user, and a display unit 36 that displays visual information for the user such as an operation procedure and a processing result.

  The defect classification unit 32 executes a process of classifying detected defects using an appropriate learning algorithm such as SVM (Support Vector Machine), neural network, decision tree, discriminant analysis, logistic regression, and the like in software. To do. Specifically, based on the defect image data given from the defect inspection apparatus 2 or another external apparatus and stored in the storage unit 33, and teaching information from the user regarding the classification category into which the defect image should be classified. Machine learning using an appropriate learning algorithm in advance. When new image data corresponding to an unclassified defect image is given, automatic classification based on the learning result is performed to classify the new defect image into any one of the classification categories.

  FIG. 2 is a flowchart showing the operation of the defect detection apparatus. The defect detection apparatus 2 images the inspection target substrate S, and executes the following defect detection processing as an operation for detecting defects from the imaging result. When the inspection target substrate S is first loaded on the stage 202 of the image acquisition unit 20 (step S101), the mechanism control unit 211 controls the imaging unit 201 and the stage driving unit 203 to perform a predetermined inspection target on the substrate S. The position is imaged and an image is acquired (step S102). For the image data obtained in this way, the defect detection unit 212 performs processes such as brightness adjustment and noise removal as necessary. Further, the image data of a portion corresponding to the inspection target position in the image of the defect-free substrate (reference image) stored in advance in the storage unit 213 is read and pixel unit with the image data obtained by imaging Is calculated (step S103).

  If the captured image does not include a defect, the difference from the previously stored defect-free image is small, but if a defect is included, a large difference appears. By utilizing this fact, it is possible to determine the presence or absence of a defect by obtaining the difference between the two. Specifically, appropriate smoothing processing and binarization processing are performed on the obtained difference image (step S104), and if there is a region where a significant difference is recognized, that portion is extracted as a defective portion ( Step S105). For an image in which a defect is detected (step S106), the image is transmitted as a defect image to the defect classification device 3 via the interface 214 (step S107). By performing the above processing on the entire substrate S while sequentially changing the inspection target position (step S108), the inspection of the substrate S is completed.

  The defect detection device 2 inspects each position of the inspection target substrate S to determine whether there is a defect. On the other hand, the defect classification device 3 determines the type of detected defect, analyzes the tendency of occurrence of the defect, and feeds back the result to the production line, or adjusts the processing content of the defect detection processing by the defect detection device 2 Used to do By analyzing the detected defect by the defect classification device 3, the defect detection device 2 can execute a process specialized for defect detection, so that a defect can be detected at high speed.

  FIG. 3 is a flowchart showing the operation of the defect classification apparatus. As described above, the defect classification device 3 classifies the defect image output from the defect detection device 2 into a classification category corresponding to the defect type. More specifically, when the defect classification unit 32 performs pre-learning (described later) based on the defect image data collected in advance (step S201), an unclassified defect image is newly given from the defect detection device 2. (Step S202), the feature quantity calculation unit 34 calculates a plurality of feature quantity values that characterize the defect image (Step S203). Then, based on the calculated feature value, the defect classification unit 32 classifies the defect image into one of a plurality of classification categories using a classifier configured by prior learning (step S204).

  A plurality of types of classification categories are set in advance corresponding to a plurality of types of defects that may occur in the substrate S. For example, foreign materials, scratches, bubbles, pinholes, pattern disconnections, short circuits, and the like are representative. However, it is not limited to these. In the following processing, the classification category is not specified by the above-described defect type name, but by a category number assigned in advance in a one-to-one correspondence with the classification category. A serial number in an integer sequence is used as the category number. For example, the category numbers “1”, “2”, “3”,... Are assigned to the classification categories corresponding to the defect types “foreign matter”, “scratch”, “bubble”,. Is done.

  FIG. 4 is a flowchart showing a pre-learning process executed by the defect classification apparatus. The pre-learning process is performed based on a teacher image obtained by imaging various defects and teaching information indicating which classification category the teacher image should be classified, which is taught by the user. The more teacher images that show the typical appearance of the defect, the better the learning effect and the higher the classification accuracy.

  First, a plurality of defect images used as teacher images are collected (step S301). Here, the defect images output from the defect detection device 2 are collected by sequentially storing them as teacher images in the storage unit 33. For example, the defect images are collected in an image library prepared in advance or in an external storage. It may be an aspect in which teacher image data is acquired. Of course, more defect images are required than the number of categories to be classified. In the following, the total number of collected teacher images is N, and the teacher images are distinguished from each other by adding symbols T (1), T (2),..., T (N).

  Next, for each collected teacher image, a teaching input for specifying a classification category in which the teacher image is to be classified is received from the user (step S302). Input from the user is received by the input receiving unit 35. Thereby, teacher data in which the teacher image is associated with the taught classification category is created.

  When teaching input is made for all teacher images, the feature vector of each teacher image is calculated (step S303). Specifically, a plurality of types of feature amounts representing appearance features of the teacher image are calculated, and a feature vector composed of the plurality of feature amounts is obtained. Hereinafter, a feature vector representing the teacher image T (n) is represented by a symbol V (n) (n = 1, 2,..., N). The dimension of the feature vector at this time is a code P (P is an integer of 2 or more). That is, each teacher image is represented by P types of feature amounts. It should be noted that the processing order for receiving the teaching input and calculating the feature vector may be reversed.

  Subsequently, the value of the internal parameter i for loop processing is set to 0 (step S304), and the loop processing of steps S305 to S309 is executed. First, case learning using teacher data is executed by an appropriate learning algorithm such as SVM, neural network, decision tree, discriminant analysis, logistic regression, etc., and a classifier is configured. This classifier is represented by code C (i) (step S305). In the first loop, a classifier C (0) is configured.

  Next, the teacher image used to configure the classifier C (i) itself is classified by the configured classifier C (i) (step S306). Ideally, each teacher image should be classified into a classification category given by the user by teaching input. However, the classification result is not necessarily so depending on the suitability of the collected teacher image, the teaching input by the user, the selection of the feature amount used for classification, and the like. That is, the correct answer rate of the classification result by the classifier is generally less than 100%. That is, the possibility that an optimal classifier can be obtained by one case study is not necessarily high.

  With respect to the loop processing in steps S305 to S309, an exit condition for exiting the loop is determined in advance (step S307). The escape conditions include, for example, that the correct answer rate of the classification result by the classifier for the class taught in advance is equal to or higher than a predetermined value, and that the classification result does not change significantly even when the loop process is executed (for example, the change in the number of incorrect answers) That the number of executions of the loop processing has reached a preset upper limit number of times is applicable, and when any of these requirements is satisfied, the escape condition is satisfied. The escape conditions are not limited to these.

  When the escape condition is satisfied (YES in step S307), the loop process ends. As described above, it is considered that the escape condition is hardly satisfied in the first process. If the escape condition is not satisfied (NO in step S307), the feature vector of each teacher image is updated by a method described later based on the classification result (step S308). Then, the parameter i is incremented by 1 (step S309), and the process returns to step S305.

  The above loop processing is repeatedly executed until the escape condition is satisfied. When the escape condition is satisfied in step S307, a value obtained by subtracting 1 from the parameter i at that time is stored as the parameter M (step S310), and the pre-learning process is completed.

  The value of the parameter i incremented each time the loop process is executed with 0 as an initial value is a value obtained by adding 1 to the number of times step S305 is executed. Therefore, the value of the parameter M obtained by subtracting 1 from this corresponds to the number of executions of step S305. Since the classifier C (i) is configured every time step S305 is executed, M sets of classifiers represented by codes C (0) to C (M−1) are configured as a whole. If the teacher data is the same, these classifiers are all the same. However, since the feature vector included in the teacher data is updated each time, the classifier C (i) (i = 0, 1,..., M− 1) are different from each other.

  FIG. 5 is a flowchart showing the feature vector update process. This feature vector update process is a process corresponding to step S308 in the flowchart of FIG. 4, and the parameter i can take various values depending on the number of executions of the loop process. As is apparent from FIG. In the series of processes shown in FIG. 8, the value of the parameter i is fixed. In the following, the category number corresponding to the classification category classified by the classifier C (i) is represented by the symbol L (i).

  First, the threshold used for processing is appropriately set within a range of 0 or more and less than 1 (step S401). An appropriate method for determining the threshold will be described later. Next, the value of the internal parameter n for specifying the teacher image is set to 1 (step S402), and one image T (n) is selected from the collected teacher images (step S403). At this time, among the N teacher images, the one assigned with the code T (1) is selected. For the selected teacher image T (n), the classification category previously given by the user is compared with the result of automatic classification using the classifier C (i) (step S404).

  If the classification category assigned by automatic classification does not match the classification category assigned by the user, that is, if the classification result is incorrect (NO in step S404), the category number L of the classification category output by the classifier C (i). (I) is set as a new feature value (step S412), and this feature value is added as a new element to the feature vector V (n) of the teacher image T (n) (step S409). As a result, the dimension of the feature vector V (n) increases by one.

  On the other hand, when the classification category assigned by automatic classification matches the classification category assigned by the user, that is, the classification result is a correct answer (YES in step S404), a random number between 0 and 1 is generated (step S405). ), And the random number is compared with a predetermined threshold (step S406). If the generated random number is smaller than the threshold value (YES in step S406), the category number L (i) of the classification category output by the classifier C (i) is a new feature as in the case of misclassification. The amount is set as a value (step S411), and is added as a new element to the feature vector V (n) of the teacher image T (n) (step S409).

  If the generated random number is greater than or equal to the threshold (NO in step S406), a random number greater than or equal to (−1) and less than (+1) is generated (step S407), and the classification category output by classifier C (i) The value obtained by adding the random number to the category number L (i) of the first is used as a new feature value (step S408), and the feature is added as a new element of the feature vector V (n) (step S409). ). The value of the feature amount added at this time is a section [L (i) -1, L (i) +1) centered on the category number value L (i) of the classification category output by the classifier C (i). It becomes the random number in.

  The processes in steps S401 and S405 to S406 are for selectively applying two rules for determining a new feature value with a certain probability. That is, when the classification result by the classifier C (i) is a correct answer, as a rule for determining a new feature value to be added to the feature vector, the output category number Li is directly used as the feature value. Or a random number within a predetermined range centered on the output category number Li. Since one rule is selected in accordance with the magnitude relationship between the random number generated in step S405 and the threshold value, the probability that these two rules will be applied is determined by the set value of the threshold value. .

  That is, although a threshold value is set as a parameter for adjusting the application probability of the two rules and a random number is used, the magnitude relationship between the threshold value and the random number has no physical meaning. Therefore, “YES” and “NO” in step S406 may be reversed, and the two rules are selectively applied with a certain probability as described above, and the probability can be changed. If there are, the specific method is arbitrary.

  In this way, the feature vector V (n) representing the teacher image T (n) is added with a new feature amount having a value corresponding to the classification result output from the classifier C (i) as an element. The dimension is updated to a new feature vector increased by one.

  Until the parameter n reaches the total number N of teacher images (step S410), the parameter n is incremented by 1 (step S412), and the loop processing of steps S403 to S409 is repeated to obtain N teacher images T ( Feature vectors V (1) to V (n) corresponding to 1) to T (N) are updated.

  The threshold value can be corrected as necessary (step S413). If the threshold value needs to be corrected, the process returns to step S401 and the above process is re-executed. At this time, the update result of the feature vector executed based on the previous threshold setting is discarded, and a new update process is performed on the feature vector having the original number of dimensions.

  As described above, when the feature vector update process is executed, the dimension of the feature vector V (n) increases by one. Therefore, in the loop process shown in FIG. 4, the dimension of the input vector given to the case learning in step S305 is one different between the i-th process and the (i + 1) -th process. Therefore, the classifier C (i) and the classifier C (i + 1) are different classifiers corresponding to input vectors of different dimensions.

  Compared with the classifier C (i), the classifier C (i + 1) has one more corresponding input vector element, and the increased element is the classification result (category number) output by the classifier C (i). Is a feature quantity having a value based on. That is, the feature vector given as the teacher data to the classifier C (i + 1) includes information on the classification result by the classifier C (i) configured immediately before as a feature amount. Therefore, in the case study in step S305, the classification results executed so far are propagated as feature quantities.

  The case study for configuring the classifier C (i + 1) includes a feature amount reflecting the classification result of each of the teacher images T (n) in the classifiers C (0) to C (i) configured in the past. The feature vector V (n) is used as teacher data. The feature value added based on the classification result is a category number indicating the incorrect classification result itself if it is an incorrect answer different from the instruction from the user, while it is classified if it is a correct answer. It is a value having a certain fluctuation around the category number.

  For this reason, an incorrect classification result propagates as a strong feature quantity with respect to subsequent case learning, while a correct classification result propagates as a weak feature quantity. As a result, the effect of weighting each case is indirectly obtained when repeating case learning, and therefore the performance of the classifier is expected to be improved. According to the experiment of the present inventor, how to give fluctuation to the value of the feature amount when the classification result is a correct answer, more specifically, setting the threshold value in step S401 (FIG. 5) is appropriate. Thus, it has been found that it is possible to obtain a classifier having excellent classification performance.

  In principle, the threshold can be optimized as follows. After classifying the teacher image by the classifier C (i), the feature vector update process (FIG. 5) is executed, and class learning is performed using the updated feature vector to obtain the classifier C (i + 1). The teacher image is classified using the obtained classifier C (i + 1), and the classification result is evaluated. Specifically, the ratio (correct answer rate) of N teacher images from which the classifier C (i + 1) outputs a classification result that matches the user teaching is obtained. Such a process is repeatedly executed while variously changing the threshold setting value in the feature vector updating process, and the threshold setting value with the best classification result of the classifier C (i + 1) is determined.

  In addition, it turns out that the classification performance close | similar to the best can be obtained more simply by doing as follows. That is, according to the knowledge of the present inventor, in the feature vector update process shown in FIG. 5, the number of teacher images determined as “NO” in step S404 and the number of teacher images determined as “YES” in step S406. When the number is almost equal, the classification result of the classifier C (i + 1) tends to be the best. That is, among the teacher images in which the classification result of the classifier C (i) is correct, the classified category number itself is the feature value, and the classification result of the classifier C (i) is incorrect. What is necessary is just to make it the same number as the case where an incorrect category number is made into the value of a feature-value.

  When the classification result of the classifier C (i) is correct, the classified category number itself is used as the feature value, and the value obtained by adding a random number to the category number is used as the feature value. The ratio depends on the threshold value set in step S401. Therefore, the threshold value can be optimized by adjusting the threshold value so that the case where the classified category number is the value of the feature amount is approximately equal to the number of erroneous answers.

  The actual operation is as follows, for example. In the feature vector update process of FIG. 5, the threshold value is temporarily set to an appropriate value in step S401, and steps S402 to S412 are executed. At this time, out of the number of cases in which the classification result is determined not to match the teaching in step S403, that is, the number of wrong answers and the correct answer in which the classification result matches the teaching, the case in which the random number is determined to be smaller than the threshold in step S406. And count the number of each.

  In step S413, the number of both cases is compared, and if the difference (or ratio) is within a predetermined allowable range, it is determined that the threshold value need not be corrected (NO in step S413). On the other hand, if there is a difference exceeding the allowable range, it is determined that the threshold needs to be corrected (YES in step S413), the process returns to step S401, the threshold value is changed, and the process is executed again. The threshold value is changed to a smaller value if the number of cases judged “YES” in step S406 is larger than the number of erroneous answers, and to a larger value if the number is less than the number of erroneous answers. By doing in this way, the number of cases in which the random number is determined to be smaller than the threshold value in step S406 out of the correct answers whose classification results match the teachings substantially matches the number of erroneous answers, and the threshold value is optimized.

  In the case learning that is subsequently performed, a classifier C (i + 1) with improved classification performance over the classifier C (i) can be obtained. By repeating such an improvement in the loop processing of FIG. 4, it is possible to obtain a classifier with better classification performance. However, as the number of executions of the loop process increases, the time required for the process becomes longer. On the other hand, the degree of improvement in the classification performance generally reaches its peak gradually. For this reason, the escape condition in step S307 is determined as described above so that the loop process is completed when a certain improvement effect is obtained.

  As described above, in the pre-learning process of the present embodiment, M sets of classifiers C (0) to C (M-1) having different input vector dimensions by 1 are configured. Among these, the classifier C (0) corresponds to the same P-dimensional input vector as the feature vector of the teacher image obtained in step S303 (FIG. 4), and the other classifiers C (1) and C (2). ,..., C (M−1) correspond to (P + 1), (P + 2),..., (P + M−1) -dimensional input vectors, respectively. Next, a method for classifying unclassified defect images using the M sets of classifiers configured as described above will be described.

  FIG. 6 is a flowchart showing a classification process for an unclassified defect image. The process shown in FIG. 6 corresponds to steps S202 to S204 in FIG. First, an unclassified defect image to be classified is acquired (step S501). This processing step corresponds to step S202 in FIG. Hereinafter, the acquired classification target image is represented by reference numeral Ix. For example, an image including a defect detected on the inspection target substrate S by the defect detection apparatus 2 is set as the classification target image Ix.

  Next, a feature vector Vx of the classification target image Ix is obtained (step S502). This processing step corresponds to step S203 in FIG. 3, and the same P types of feature amounts as the teacher image are calculated for the classification target image Ix, and the P-dimensional feature vector Vx is specified.

  Subsequent steps S503 to S509 correspond to step S204 in FIG. In step S503, the value of the internal parameter j for selecting one from the M classifiers is set to an initial value 0 (step S503), and the classifier C (j) identified by this is set. In this case, The classification target image Ix is classified using the classifier C (0) (step S504). Then, the classification result output from the classifier C (j), that is, the category number corresponding to the classification category to which the classification target image Ix should be classified is added as a new element to the feature vector Vx representing the classification target image Ix. (Step S505). As a result, the feature vector Vx is updated, and its dimension increases by one.

  While incrementing the value of parameter j by 1 (step S506), steps S504 and S505 are repeated until the value of parameter j reaches the number M of classifiers (step S507). Each time classification is performed by the classifier C (j), the result is added as a new feature amount to the (P + j) -dimensional feature vector Vx, and the dimension of the feature vector is increased by 1 to become (P + j + 1) -dimensional. Then, it is input to the classifier C (J + 1) corresponding to the (P + j + 1) -dimensional input vector and classified again.

  By repeating such processing, M sets of classification results output from the M sets of classifiers are obtained. Since the classification performance of each classifier is different, these M sets of classification results are not necessarily the same. One of these classification results is selected as the final classification result (step S508), and a notification is output to the user (step S509). For example, the final result can be selected as follows.

  First, it is conceivable that the classification result output by the last classifier C (M−1) is the final result. As described above, the classification performance of the classifier is gradually improved by configuring a new classifier while updating the feature vector according to the classification result. Therefore, it is generally expected that the classifier configured later has superior classification performance. From this, it can be considered that the classification category classified by the last classifier C (M-1) is the final result.

  Secondly, it is conceivable that the classification category having the highest appearance frequency in the M classification results is the final result. That is, the final result is determined by taking the majority of the classification results by the M classifiers. This is because that a specific classification category appears many times as a classification result indicates the validity of the classification category.

  Third, it is conceivable that an appropriate weight is given to each of the M classification results, and the classification category having the largest weight is the final result. Specifically, the correct answer rate with respect to the teacher data for each of the M classifiers, that is, the ratio of the N teacher images to which the classification result that matches the teaching by the user is output is evaluated. Then, a weight that increases as the classifier having a higher correct answer rate is given to each classifier, and the weights of the classifiers that output the classification category as a classification result are totaled for each classification category. The classification category having the largest total value is the final classification result. Since the result output by the classifier having a high correct answer rate is estimated to be relatively reliable, it can be said that it is reasonable to select the final result by the method of summing the weights.

  According to the knowledge of the inventor of the present application, if the threshold value is appropriately set in the feature vector update process, it is possible to obtain a sufficiently accurate classification result by the above-described first method, as will be exemplified later. is there.

  In addition, although the manner of the notification output in step S509 is arbitrary, for example, by displaying the defect type corresponding to the final classification result (category number) on the display unit 36 together with the classification target image, the classification result is displayed to the user. Can be notified.

  FIG. 7 is a diagram illustrating the effect of this embodiment. FIG. 7A to FIG. 7C are tables comparing and comparing the classification results obtained by the classifier constituted by the case learning with the classification categories taught in advance by the user, and are also called a confusion matrix. . The inventor of the present application reclassified the teacher image using a classifier obtained by performing linear discriminant analysis using 5115 defect images as the teacher images. The classifier configured at this time corresponds to the classifier C (0) in the above embodiment. FIG. 7A shows the result of reclassification as a confusion matrix.

  The row heading described as “teach” at the left end of the table indicates the category number taught by the user, and the column heading described as “ADC” at the top indicates the category number classified by the classifier. The numbers “1”, “2”, and “3” are category numbers assigned to the three types of classification categories. There are 1578 images taught by the user as category number “1”, and the number of correct answers correctly classified by the classifier C (0) as taught by the user is 1427. Therefore, the recall (Recall) of the classifier C (0) for the category number “1” taught by the user is 90.4%. The classifier C (0) misclassified 150 of the other images as category number “2” and 1 as category number “3”.

  Similarly, the number of correct answers by the classifier C (0) out of 2849 images assigned by the user with the category number “2” is 2354, and the recall is 82.6%. The number of correct answers by the classifier C (0) is 644 out of 688 images to which the user has assigned the category number “3”, and the recall is 93.9%. These can be seen by looking at the table horizontally.

  On the other hand, looking at the classification result output by the classifier C (0) when the table is viewed in the vertical direction, there are 1692 images classified by the classifier C (0) into the category number “1”, of which match the user teaching. It was 1427 sheets. Therefore, the precision for the category number “1” is 84.3%. Similarly, the accuracy for category number “2” and category number “3” is 92.5% and 73.7%, respectively. Of the total 5115 images, the number of correct answers that matched the user teaching and the classification result is 4427, and the accuracy is 86.5%.

  Next, a new feature quantity is added to the feature vector of each teacher image by the feature vector update process (FIG. 5) according to the correctness of the classification result (category number) output by the classifier C (0). A new classifier was constructed by linear discriminant analysis as teacher data. This classifier corresponds to the classifier C (1) in the above embodiment. FIG. 7B shows the result of reclassifying the teacher image using the classifier C (1). Although there are items that decrease when looking at the recall and accuracy, the overall improvement is better than that in FIG. 7A, and the accuracy is improved by 0.4 points from the result shown in FIG. 7A. It is done.

  Furthermore, a feature vector update process is executed based on the classification result (category number) output by the classifier C (1) to add a new feature amount to the feature vector of each teacher image, and this is linearly used as teacher data. A new classifier was constructed by discriminant analysis. This classifier corresponds to the classifier C (2) in the above embodiment. FIG. 7C shows the result of classifying the teacher image using the classifier C (2). Compared with the result shown in FIG. 7B, the accuracy is further improved by 0.4 points.

  In this way, by adopting a classifier configuration method in which teacher data is updated by adding a new feature amount based on the result classified by the classifier, and a new classifier is configured by case learning based on it. It can be seen that a classifier with higher classification performance can be obtained. Since it is not necessary to change the learning algorithm itself when adopting such a method, this method can be applied to various machine learning algorithms.

  As described above, in the above embodiment, steps S305, S306, and S308 in FIG. 4 correspond to the “first step”, “second step”, and “third step” of the present invention, respectively. Steps S504 and S505 in FIG. 6 correspond to the “fourth step” and “fifth step” of the present invention, respectively. The category number representing the classification category corresponds to the “identification number” of the present invention.

  The inspection system 1 of the above embodiment corresponds to the “image classification device” of the present invention, and the defect detection device 2 and the defect classification device 3 function as the “image acquisition unit” and “classification unit” of the present invention, respectively. is doing. Further, the defect classification device 3 alone has the function as the “image classification device” of the present invention. In this case, the interface 31 functions as the “image acquisition means” of the present invention, and the defect classification unit 32 is It functions as the “classification means” of the invention. The display unit 36 functions as the “notification unit” of the present invention.

  The present invention is not limited to the above-described embodiment, and various modifications other than those described above can be made without departing from the spirit of the present invention. For example, in the above-described embodiment, the present invention is applied to classification of defect images obtained by imaging a semiconductor substrate. However, an image to be classified based on the method of the present invention shows such defects on the substrate. It is arbitrary without being limited to what was imaged. For example, the present invention can be applied to classification of images obtained by imaging cells or bacteria cultured in a medium.

  Further, for example, in the inspection system 1 of the above embodiment, the defect detection device 2 and the defect classification device 3 are configured as separate bodies, but these may be integrated. In the defect classification device 3 of the above embodiment, case learning based on a teacher image and classification of an unknown image based on the result are performed by the same device, but these are performed by individual devices. Also good. In this case, the classifier constituted by the case learning performed in one apparatus is mounted in another apparatus, and the image is classified.

  Further, for example, the inspection system of the above embodiment includes the defect detection apparatus 2 having an imaging function, but the present invention is also applied to an apparatus that does not have an imaging function itself and exclusively receives image data from outside and performs processing. It is possible.

  As described above, the specific embodiment has been described by way of example. In the image classification method according to the present invention, for example, the Mth classification result is obtained from the M classification results obtained by using the M classifiers. The classified result can be output. It is expected that the classification performance of the classifier will gradually improve by repeating the case learning while adding the previous classification result to the feature quantity. Therefore, it is possible to obtain a good classification accuracy by outputting the finally obtained classifier result as the final classification result.

  Further, for example, the classification result determined by the majority of the M classification results may be output. Even if the classification results of the classifiers are low in reliability, the output is determined by their majority vote, so that a more accurate classification result can be stably obtained.

  Further, for example, a numerical value that is continuous in an integer string may be assigned as an identification number to each of a plurality of classification categories, and the range of the numerical value range may be set to be smaller than 2. When consecutive integers are used as identification numbers, the difference between adjacent identification numbers is 1. In this case, if the range of fluctuation added to the correct classification result as the feature quantity is larger than 2, it may not be understood what the original classification result was and may not function properly as the feature quantity. By setting the fluctuation range to be smaller than 2, information on the original classification result is propagated to later learning without being lost, which can contribute to improvement of classification performance.

  Further, for example, in the third step, when the classification category assigned in the second step matches the classification category assigned in advance in the teacher data, the same numerical value as the identification number of the classification category falls within the numerical range described above. A value obtained by adding the generated random number may be used as a new feature amount. By doing so, it is possible to give an appropriate and irregular fluctuation to the value of the feature value, and to improve the classification performance of the classifier.

  Also, for example, in the third step, when the classification category assigned in the second step matches the classification category assigned in advance in the teacher data, a new value that is the same as the identification number of the classification category is newly added with a predetermined probability. It may be configured to have various feature amounts. According to the knowledge of the present inventor, even when the classification category obtained by the classifier and the taught classification category match, the identification number corresponding to the classification category is a new feature value as it is at a certain ratio. By doing so, it is possible to effectively improve the classification performance.

  Also, for example, when the number of images that have a classification result different from the classification category assigned in advance in the second step is referred to as the number of erroneous answers, the number of erroneous answers in the second step is executed last time. If it becomes the same as the number of wrong answers in the second step, the repetition of the first to third steps may be terminated. Repeated execution of the first to third steps can improve the classification performance, but the processing time also becomes longer. By ending the repetition with the change in the number of erroneous answers eliminated, it is possible to obtain a classifier with a certain level of classification performance without unnecessarily increasing the processing time.

  In the image classification device according to the present invention, for example, the image acquisition means may have an imaging function. According to such a structure, it is possible to implement | achieve from acquisition of an image to classification | category with an integrated apparatus.

  The present invention can be suitably applied to a technical field in which a classifier is configured by machine learning using a teacher image and an unknown image is classified using the classifier, and any learning algorithm to be applied can be applied. Things can be used.

1 Inspection system (image classification device)
2 Defect detection device (image acquisition means)
3. Defect classification device (classification means, image classification device)
31 Interface (Image acquisition means)
32 Defect classification part (classification means)
36 Display section (notification means)
S305 (first step)
S306 (second step)
S308 (third step)
S504 (4th process)
S505 (5th process)

Claims (11)

An image classification method for automatically classifying images based on the result of machine learning using teacher data including information on a teacher image represented by a feature vector of P (P is a natural number) dimension and a classification category assigned to the teacher image In
A unique identification number is assigned to each of a plurality of classification categories,
A first step of configuring a classifier by executing a machine learning algorithm using the teacher data corresponding to a plurality of the teacher images;
A second step of classifying each of the teacher images into a plurality of classification categories by the classifier configured in the first step;
For each of the teacher images, the classification category assigned in the second step is compared with the classification category assigned in advance in the teacher data, and a numerical value corresponding to the result is added as a new feature amount to the feature vector. Thus, the third step of increasing the number of dimensions of the feature vector of the teacher image by 1 is executed M (M is an integer of 2 or more) times in this order, and M sets of input vectors whose dimensions are different by 1 Prepare a classifier,
Here, in the third step, when the classification category given in the second step is different from the classification category given in advance in the teacher data, the classification category given in the second step When the classification number assigned in the second step and the classification category assigned in advance in the teacher data match with the identification number as the new feature amount, the identification number of the classification category is the center. An irregular numerical value within a predetermined numerical value range as the new feature amount,
For the images to be classified, P type feature amounts forming a P-dimensional feature vector are obtained,
A fourth step of classifying the feature vector of the image into one of the classification categories by the classifier corresponding to an input vector of the same dimension as the feature vector;
By adding the identification number of the classification category classified in the fourth step to the feature vector as a new feature amount, the fifth step of increasing the number of dimensions of the feature vector of the image by M is performed M times in this order. Execute to get the classification results of M sets,
An image classification method for outputting one classification result selected from the M classification results as the classification result of the image.   The image classification method according to claim 1, wherein a classification result acquired for the Mth time among the M sets of classification results is output.   The image classification method according to claim 1, wherein a classification result determined by a majority vote of the M sets of classification results is output.   The image classification method according to any one of claims 1 to 3, wherein each of the plurality of classification categories is assigned a numerical value that is continuous in an integer string as the identification number, and a range of the numerical value range is smaller than two.   In the third step, when the classification category assigned in the second step matches the classification category assigned in advance in the teacher data, the same numerical value as the identification number of the classification category falls within the numerical value range. The image classification method according to claim 1, wherein a value obtained by adding the generated random number is used as the new feature amount.   In the third step, when the classification category assigned in the second step matches the classification category assigned in advance in the teacher data, the same number as the identification number of the classification category is obtained with a predetermined probability. The image classification method according to claim 1, wherein the new feature amount is used. In the second step, when the number of images resulting in a classification result different from the classification category assigned in advance among the teacher images is referred to as the number of incorrect answers,
The repetition of said 1st thru | or 3rd process is complete | finished if the number of wrong answers in the said 2nd process becomes the same as the number of wrong answers in the said 2nd process performed last time. The image classification method described. In a method of configuring a classifier for automatically classifying an image into a plurality of classification categories,
A unique identification number is assigned to each of the plurality of classification categories,
Obtaining teacher data including information on a plurality of teacher images represented by a feature vector of P (P is a natural number) dimension and a classification category assigned to each teacher image;
A first step of configuring a classifier by executing a machine learning algorithm using the teacher data corresponding to the teacher image;
A second step of classifying each of the teacher images into a plurality of classification categories by the classifier configured in the first step;
For each of the teacher images, the classification category assigned in the second step is compared with the classification category assigned in advance in the teacher data, and a numerical value corresponding to the result is added as a new feature amount to the feature vector. Thus, the third step of increasing the number of dimensions of the feature vector of the teacher image by one is executed M (M is an integer of 2 or more) times in this order, and M sets whose input vector dimensions are different by one Configure the classifier
In the third step, when the classification category assigned in the second step is different from the classification category assigned in advance in the teacher data, the identification number of the classification category assigned in the second step is set. When the classification category given in the second step matches the classification category given in advance in the teacher data as the new feature amount, a predetermined centered on the identification number of the classification category A classifier configuration method in which an irregular numerical value within a numerical value range is used as the new feature amount. Image acquisition means for acquiring a plurality of images to be teacher images and images to be classified;
Classifying means for classifying an image by executing the image classification method according to any one of claims 1 to 7,
An image classification device comprising notification means for notifying a user of an output classification result. Image acquisition means for acquiring images to be classified;
Classifying means for classifying an image using a classifier configured by the classifier configuration method according to claim 8;
An image classification device comprising notification means for notifying a user of an output classification result.   The image classification device according to claim 9 or 10, wherein the image acquisition unit has an imaging function.

Images