TW202125405A - Classifying device and image classifying system - Google Patents

Abstract

Provided is a classifying device with which it is possible to improve the performance of a classifier. More specifically, in this classifying device 30, a classifier learning means 31a is configured so that, when the category of a defective image Pd of data D2 for evaluation that has been classified by a classification execution means 31b and the category of a defective image Pd previously classified by a user do not match, a classifier 32 is retrained on the basis of training data D11 in a state where a category selection image Pd3, in which the category classified by the classification execution means 31b is selected as the category of the defective image Pd for which the aforementioned categories do not match, is included in training data D1.

Description

Classification device and image classification system

The invention relates to a classification device and an image classification system.

Heretofore, there is known a classifier that classifies images into any one of a plurality of classes (for example, refer to Patent Document 1).

The aforementioned Patent Document 1 discloses a classifier construction method for classifying an image into any one of a plurality of categories (categories). In this classifier construction method, firstly, a teacher image of a category taught by the user in advance and teacher data containing a value of each of a plurality of feature quantity axes based on the teacher image are prepared. Furthermore, the image includes the image of the defect formed on the substrate. Secondly, based on the prepared teacher data, the frequency distribution data (frequency of appearance) in each interval in which the value of the feature quantity axis is discretized is generated for each of the plurality of feature quantity axes. For example, the area, circumference, position of the center of gravity, and the amount of moment of the feature axis defect. In addition, the frequency distribution data is formed as follows. First, specify the maximum value and minimum value of the characteristic value axis, and obtain the distribution range of the value of the characteristic quantity axis. Then, the distribution range is divided (discretized) into an appropriate number of intervals. Then, find the frequency (frequency of appearance) of each type in each interval of discretization.

Secondly, based on the occurrence ratio of each feature axis in each interval shown by the frequency distribution data by category, a classifier (a plurality of weak classifiers) for classifying the image is generated. For example, for a certain feature quantity, the probability that an image belonging to a certain interval belongs to a certain other is calculated. Then, the weak classifier refers to the frequency distribution data respectively according to the value of the corresponding feature quantity axis, thereby obtaining an evaluation of the validity (certainty) of the image that is regarded as the acquired feature quantity axis value belongs to a specific category The value is used as the category evaluation value.

Secondly, use the classifier to classify the teacher information. Then, correct the frequency distribution data for the misclassified teacher data in the teacher data classified by the classifier. Specifically, the frequency of the category corresponding to the interval (the interval of the frequency distribution data) corresponding to the value of the feature quantity of the teacher data whose category is misclassified is increased. Then, the classifier is updated based on the corrected frequency distribution data, whereby the above-mentioned misclassified teacher data is more likely to be classified into the correct category from the updated classifier. [Prior technical literature] [Patent literature]

[Patent Document 1] Japanese Patent Laid-Open No. 2019-57024

[The problem to be solved by the invention]

In the method for constructing a classifier described in Patent Document 1, it is generated based on a teacher image of a category taught in advance by a user, and teacher data including a value of each of a plurality of feature quantity axes based on the teacher image Classifier. Here, the types of teacher images are taught by the user. On the other hand, when the number of teacher images is relatively large, etc., it is considered that the concentration of the user may be reduced due to fatigue, etc., and the types are shown. Teaching mistakes. In this case, since the classifier is generated (evaluated) based on the type (category) of the wrong teaching, there is a problem that it is difficult to improve the classification accuracy of the classifier.

The present invention was completed in order to solve the above-mentioned problems. One of the objectives of the present invention is to provide a classification device and an image classification system that can improve the performance of the classifier. [Technical means to solve the problem]

In order to achieve the above object, the classification device of the first aspect of the present invention classifies unknown image categories, and includes: a classifier learning mechanism that learns the classifier by machine learning based on teacher data. Including a plurality of images pre-classified by the user into any one of the plurality of categories; and a classification execution mechanism, which, based on the learned classifier, classifies the evaluation data including the plurality of images into a plurality of categories Any category; the classifier learning mechanism is structured as follows, that is, when the category of the image of the evaluation data classified by the classification implementing agency is inconsistent with the category of the image pre-classified by the user, it will be classified by the classification implementing agency The category of the image or the category re-determined by the user is selected as the category of an image with inconsistent categories. When the category selection image is included in the teacher data, the classifier is relearned based on the teacher data.

In the classification device of the first aspect of the present invention, as described above, the classifier learning mechanism is configured as follows, that is, when the classification of the image of the evaluation data classified by the classification execution mechanism and the image classified in advance by the user When the categories are inconsistent, if the category classified by the classification agency or the category re-determined by the user is selected as the category of the inconsistent image category, the category selection image is included in the teacher data, and the classifier is based on the teacher data study again. In this way, even when the category of the image is incorrectly classified, the category of the incorrectly classified image can be changed to the correct category. In addition, when the category selection image after the incorrectly classified category is changed to the correct category is included in the teacher data, the classifier is relearned based on the teacher data, so that the performance of the classifier can be improved.

In the classification device of the first aspect described above, it is preferable that the classification of the classification selection image is selected based on the estimated classification probability, which is an index of the degree of certainty of classification performed by the classification execution mechanism. With this configuration, by selecting the category of the image with a relatively high estimated probability of classification, it is possible to suppress the wrong category selection. In addition, by not selecting the category of the image with a relatively low estimated probability of classification, it is also possible to suppress the wrong category selection.

In this case, it is preferable that the estimated classification probabilities include at least one of the ratio of the majority of the classification results of a plurality of classifiers to the classification results and the estimated classification probabilities output from a single classifier. If constructed in this way, the ratio of the majority of the classification results of the plural classifiers to the category (the estimated probability of classification output from a single classifier) is the value output from the general learning algorithm. Therefore, it can be easily based on these values Choose a category.

In the above-mentioned classification device selected based on the estimated classification probability, it is preferable to further include a display unit that displays the evaluation of the classification execution agency classification related to the image in which the classification execution agency classification is inconsistent with the user classification category Use the type of the image of the data, the type of the image pre-classified by the user, and the estimated probability of classification. If configured in this way, the user can easily see whether there is a situation in which the category of the image classified by the classification execution mechanism is inconsistent with the category of the image pre-classified by the user.

In this case, it is preferable to be configured as follows, that is, when the category of the image of the evaluation data classified by the classification executing agency is inconsistent with the category of the image pre-classified by the user, the display unit accepts the user's classification The input of the selection, and the classifier learning mechanism is configured as follows, that is, in the state that the category selection image of the category selected by the user is included in the teacher data, the classifier is relearned based on the teacher data. With this configuration, the user can select the category of the image while confirming the estimated probability of the classification displayed on the display unit. That is, the estimated classification probability serves as an indicator of whether or not to select the image category, so that the user can easily select the category.

In the classification device that accepts the input of the user's selection of the category, it is preferably configured as follows: When the categories are inconsistent and the estimated probability of classification is higher than a certain threshold, the display unit accepts the input of the user’s selection of the category, and the classifier learning mechanism is structured as follows: In the state of teacher data, the classifier is relearned based on the teacher data. If configured in this way, the user's input for category selection will be accepted only when the estimated classification probability is higher than a certain threshold. Therefore, in situations where category selection is not required, the control of accepting category selection input can be omitted.

In the classification device provided with the above-mentioned display unit, it is preferable to be configured as follows, that is, when the type of the image of the evaluation data classified by the classification execution mechanism is not consistent with the type of the image pre-classified by the user, the display unit The automatic selection of categories is accepted, and the classifier learning organization is structured as follows, that is, when the automatic selection has been accepted, the category of the image is automatically selected, and the category selection image of the automatically selected category is included in the teacher data In the state, the classifier is relearned based on the teacher's information. If configured in this way, the user does not need to select the category (manually), but automatically selects the category. Therefore, the user's time and effort can be saved.

In the classification device that accepts the automatic selection of the above categories, it is preferable to be configured as follows. That is, when the category of the image of the evaluation data classified by the classification executing agency is inconsistent with the category of the image pre-classified by the user, then When the estimated probability of classification is higher than a certain threshold, the display unit accepts automatic selection of the category, and the classifier learning mechanism is structured as follows The category selection image of the category is included in the state of the teacher's data, and the classifier is relearned based on the teacher's data. If configured in this way, the automatic selection of the category will be accepted only when the estimated probability of the classification is above a certain threshold. Therefore, when there is no need to select the category, the control of the input of the accepted category selection can be omitted.

In the classification device of the first aspect described above, it is preferable that the evaluation data include teacher data, and the classification execution mechanism is configured to classify the categories of the images contained in the teacher data based on the classifier after learning, and the classifier learning mechanism The structure is as follows, that is, after updating the teacher information in a way that includes the category selected by the classification execution agency as the category selection image, the classifier is re-learned based on the teacher information. If configured in this way, even when the category of the image included in the teacher data is misclassified, since the teacher data is updated, the performance degradation of the classifier due to the user's misclassification of the category can be suppressed.

In the classification device of the first aspect described above, it is preferable that the evaluation data include images different from the teacher data, and the classification execution mechanism is configured to be based on the classifier after learning, and the images included in the evaluation data different from the teacher data The image category is classified, and the classifier learning mechanism is structured as follows, that is, when the category selected by the classification execution agency is selected as the category, the category selection image is included in the teacher data, and the classifier is re-learned based on the teacher data . Here, in order to improve the performance of the temporarily generated classifier, the classification of the evaluation data including the image different from the teacher data is sometimes classified. In this case, the category of the image included in the evaluation data is also pre-classified by the user. In addition, the user may misclassify the category due to fatigue or the like. Therefore, with the above-mentioned configuration, even when the category of the image included in the evaluation data is misclassified, the performance of the classifier can be improved by selecting (correcting) the category.

The image classification system according to the second aspect of the present invention includes: an imaging device for capturing images; and a classification device that classifies the category of unknown images captured by the imaging unit; the classification device includes: a classifier A learning institution that allows the classifier to learn by performing machine learning based on teacher data. The teacher data includes a plurality of images pre-classified by a user into any one of a plurality of categories; and a classification execution institution, which is based on learning The latter classifier classifies the evaluation data including a plurality of images into any one of the plurality of categories; and the classifier learning mechanism is constructed as follows, that is, when the image of the evaluation data classified by the classification execution agency When the category is inconsistent with the category of the image pre-classified by the user, the category selected by the classification agency or the category re-determined by the user is selected as the category of the image with the inconsistent category. The category selection image is included in the teacher In the state of the data, the classifier is relearned based on the teacher's data.

In the image classification system of the second aspect of the present invention, as described above, the classifier learning mechanism is configured as follows, that is, when the image category of the evaluation data classified by the classification execution mechanism is the same as that of the image classified by the user in advance When the image categories are inconsistent, when the category classified by the classification agency or the category re-determined by the user is selected as the category of the inconsistent image category, the category selection image is included in the teacher information, and the image is based on the teacher information. The classifier relearns. Thereby, even when the category of the image is incorrectly classified, the category of the incorrectly classified image can be changed to the correct category. In addition, when the wrongly classified category has been changed to the correct category, the category selection image is included in the teacher data, and the classifier is relearned based on the teacher data. Therefore, an image that can improve the performance of the classifier can be provided Classification system. [Effects of the invention]

According to the present invention, as described above, the performance of the classifier can be improved.

Hereinafter, an embodiment that embodies the present invention will be described based on the drawings.

[This embodiment] Referring to FIGS. 1 to 14, the configuration of the image classification system 100 (classification device 30) of this embodiment will be described.

As shown in FIG. 1, the image classification system 100 includes an imaging device 10. The imaging device 10 includes an illumination unit 11, an optical system 12, and an imaging unit 13. The light emitted from the illuminating unit 11 is irradiated to the semiconductor substrate 200 via the optical system 12. In addition, the imaging unit 13 captures the light reflected by the semiconductor substrate 200. In this way, the surface of the semiconductor substrate 200 is photographed.

In addition, the imaging device 10 includes a stage 14 and a stage drive unit 15. The semiconductor substrate 200 is placed on the surface of the stage 14. The stage driving unit 15 moves the stage 14 on which the semiconductor substrate 200 is placed in a horizontal plane. In addition, by moving the stage 14 by the stage driving unit 15, the imaging unit 13 captures a desired area of the surface of the semiconductor substrate 200.

In addition, the image classification system 100 includes an inspection device 20. In addition, the inspection device 20 includes a defect detection unit 21. The defect detection unit 21 detects defects d (black defects db and white defects dw described below) in the image Ps (refer to FIG. 2) of the semiconductor substrate 200 captured by the imaging unit 13. For example, the inspection device 20 acquires an image of a certain region of the image Ps of the semiconductor substrate 200 captured by the imaging unit 13 (hereinafter referred to as the inspection target region image) and the image corresponding to the inspection target region and does not contain defects d The difference image of the image (reference image, not shown) of the semiconductor substrate 200. And, based on the acquired difference image, the defect d is detected. Furthermore, the so-called defect d refers to defects, protrusions, foreign objects, etc. on the semiconductor substrate 200.

In addition, the image classification system 100 includes a classification device 30. The classification device 30 is configured to classify the category of the unknown defect image Pd (see FIG. 4, etc.). Furthermore, in this embodiment, the defect image Pd includes the defect image Pd of the semiconductor substrate 200, and the plurality of categories includes the types of the plurality of defects d included in the semiconductor substrate 200. Furthermore, the defective image Pd is an example of the "image" in the scope of the patent application. Hereinafter, a specific description will be given.

The classification device 30 includes a computer. As shown in FIG. 3, the classification device 30 (computer) includes a control unit 31 such as a CPU (Central Processing Unit). In addition, the classification device 30 (control unit 31) includes a classifier learning mechanism 31a. The classifier learning mechanism 31a is configured to perform machine learning based on teacher data D1 (refer to FIG. 4) including a plurality of defective images Pd pre-classified by the user into any one of a plurality of categories, thereby, Let the classifier 32 learn.

Specifically, as shown in FIG. 4, a plurality of defect images Pd in which the defect d is captured are prepared. For example, the defect d includes a black defect db and a white defect dw. In addition, the user manually classifies the categories of the plurality of defect images Pd into black defects db and white defects dw. In addition, plural defect images Pd classified into black defects db and white defects dw are used as teacher data D1. Furthermore, when the number of defective images Pd is large, the user may misclassify the category due to user fatigue or the like. In FIG. 4, the defect image Pd1 of the black defect db is classified as the category of the white defect dw, and the defect image Pd2 of the white defect dw is classified as the category of the black defect db. Furthermore, the black defect db and the white defect dw are examples of "defects" in the scope of the patent application.

In addition, the classifier learning mechanism 31a performs machine learning based on the teacher data D1 including a plurality of defective images Pd shown in FIG. 4. Furthermore, as machine learning (supervised machine learning algorithm), for example, linear discriminant method, support vector machine, neural network, deep learning and decision tree are used. In addition, in the machine learning, learning is performed based on the respective types (black defects db or white defects dw) of the plurality of defect images Pd and the feature values of the plurality of defect images Pd. The characteristic quantity is, for example, the maximum brightness, minimum brightness, and brightness range in the defect image Pd, and the area ratio when the brightness of the defect image Pd is binarized.

Moreover, as shown in FIG. 3, the classification apparatus 30 (control part 31) is equipped with the classification execution mechanism 31b. The classification execution unit 31b is configured to classify the evaluation data D2 (refer to FIGS. 4 and 5) into any one of a plurality of categories based on the classifier 32 after learning. Furthermore, the evaluation data D2 includes a plurality of defect images Pd, and the plurality of defect images Pd are used to evaluate the performance of the classifier 32, and are pre-classified into any one of a plurality of categories by the user.

In addition, in this embodiment, as shown in FIG. 4, the evaluation data D2 includes the teacher data D1. Hereinafter, the evaluation data D2 composed of the teacher data D1 is referred to as the first evaluation data D21. Specifically, the first evaluation data D21 is the teacher data D1 itself. In addition, the classification execution unit 31b is configured to classify the category of the defective image Pd included in the first evaluation data D21 (teacher data D1) based on the classifier 32 after learning.

Furthermore, in this embodiment, as shown in FIG. 5, the evaluation data D2 includes a defect image Pd that is different from the teacher data D1. Hereinafter, the evaluation data D2 composed of the defect image Pd different from the teacher data D1 is referred to as the second evaluation data D22. In addition, the classification execution unit 31b is configured to classify the category of the defect image Pd included in the second evaluation data D22 that is different from the teacher data D1 based on the classifier 32 after learning.

In addition, in the present embodiment, the classifier learning mechanism 31a is configured as follows. That is, when the type of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b is divided by the defect image Pd classified in advance by the user When the categories are inconsistent, when the defective image Pd (hereinafter referred to as the category selection image Pd3) is included in the teacher profile D1 (refer to Figures 7 and 8), based on the teacher profile (updated teacher profile D11 and D12) The classifier 32 is re-learned, and among the above-mentioned defective images Pd, the class classified by the classification executing mechanism 31b is selected as the class of the defective image Pd with inconsistent classes. Hereinafter, the classification of the defect image Pd by the user in advance is referred to as MDC (Manual Defect Classification). In addition, the classification of the defect image Pd by the classification execution mechanism 31b is called ADC (Automatic Defect Classification). Moreover, in this embodiment, the category selection image Pd3 is changed to the category classified by the classification execution mechanism 31b.

Specifically, in the present embodiment, the category of the category selection image Pd3 is selected based on the estimated classification probability, which is an index of the degree of certainty of the category classification performed by the classification execution mechanism 31b. Furthermore, the estimated classification probabilities include at least one of the ratio of the majority of the classification results of the plurality of classifiers 32 to the classification results, and the estimated classification probabilities output from a single classifier 32. In detail, when the algorithm of machine learning is a decision tree, etc., the estimated classification probability is the ratio of the majority of the classification results of the plurality of classifiers 32 (decision trees) to the classification. For example, the estimated classification probability when it is classified as a black defect db is calculated as (the number of classifiers 32 classified as a black defect db)/(the number of all classifiers 32)×100. In addition, when the algorithm of machine learning is a support vector machine, etc., the classifier 32 after learning with the support vector machine outputs the estimated classification probability. Hereinafter, the selection of the category of the defective image Pd will be specifically described.

In this embodiment, as shown in FIG. 1, the classification device 30 includes a display unit 33. As shown in FIG. 6, the display unit 33 displays the category of the defective image Pd of the evaluation data D2 classified by the classification executing agency 31b, which is related to the defect image Pd classified by the classification executing agency 31b and the category of the user classification is inconsistent, The category of the defect image Pd pre-classified by the user, and the estimated probability of the classification. For example, in FIG. 6, the types of defect images Pd of No. 1 to No. 3 in which ADC and MDC are not consistent are displayed, and the estimated probability of classification. In addition, in FIG. 6, the defect image Pd of No. 1 is selected (emphasized). For example, in the defect image Pd of No. 1, it is classified as a black defect db by MDC, and on the other hand, it is classified as a white defect dw by ADC. In addition, the estimated probability of classification is 90%.

In addition, in the present embodiment, as shown in FIG. 6, the structure is as follows, namely, the defect map of the evaluation data D2 (the first evaluation data D21, the second evaluation data D22) classified by the classification executing agency 31b When the category (ADC) of the image Pd does not match the category (MDC) of the defect image Pd pre-classified by the user, the display unit 33 accepts the user's input for the selection of the category. Specifically, the structure is as follows, that is, when the type (ADC) of the defect image Pd of the evaluation data D2 (the first evaluation data D21, the second evaluation data D22) classified by the classification executing agency 31b and the used When the category (MDC) of the defect image Pd classified in advance by the user does not match, and the estimated classification probability is higher than a certain threshold, the display unit 33 accepts the user's input for the selection of the category.

For example, in the screen 33a displayed on the display unit 33, select "manual feedback" (zero symbol) with a mouse or the like. In addition, a defect image Pd (for example, the defect image Pd of No. 1) is also displayed on the screen 33a. While observing the displayed defect image Pd and referring to the displayed classification estimation probability, the user judges whether the category of the defect image Pd is a black defect db or a white defect dw. And, when it is determined that the category (MDC) of the defect image Pd pre-classified by the user is wrong, select the "category" of the "re-determination" of the screen 33a (for example, select the white defect dw), and press the "update" button . As a result, the MDC category of the defect image Pd of No. 1 is changed to a white defect dw.

In addition, a "learning file name" (name of the classifier 32), a button of "re-learning", and a button of "re-learning with another name" are displayed on the screen 33a of the display unit 33. Moreover, the classifier 32 is re-learned by the user pressing the button of "re-learning" or the button of "re-learning under another name". Furthermore, when the user presses the "re-learning" button, the classifier 32 after the re-learning is overwritten to the classifier 32 (file) before the re-learning. In addition, when the user presses the button of "re-learn with another name", the re-learned classifier 32 is generated as a new classifier 32 (file).

In addition, in the present embodiment, the classifier learning mechanism 31a is configured as follows, that is, based on the teacher data (teacher data D11 or teacher Data D12) makes the classifier 32 relearn. Here, when the evaluation data D2 is the first evaluation data D21, the classifier learning mechanism 31a is configured as follows, that is, in a state after the teacher data D1 is updated to include the category selection image Pd3, based on The teacher information (updated teacher information D11, refer to FIG. 7) causes the classifier 32 to relearn. In the above-mentioned category selection image Pd3, the category classified by the classification executing agency 31b is selected as the category. Specifically, the teacher data D1 is updated by changing the type of the defective image Pd in which the MDC and ADC of the plurality of defective images Pd included in the teacher data D1 are not consistent. And, based on the updated teacher data D1 (teacher data D11), the classifier 32 is relearned.

In addition, in the present embodiment, when the evaluation data D2 is the second evaluation data D22, the classifier learning mechanism 31a is configured as follows, that is, in a state where the category selection image Pd3 is included in the teacher data D1, Based on the teacher data (updated teacher data D12, refer to FIG. 8), the classifier 32 is re-learned. In the above-mentioned category selection image Pd3, the category classified by the classification executing agency 31b is selected as the category. Specifically, the defective image Pd (type selection image Pd3) after the category change that is inconsistent with the MDC and the ADC is added to the teacher profile D1 (or the updated teacher profile D11). Furthermore, based on the teacher profile D12 to which the category selection image Pd3 is added, the classifier 32 is relearned. In addition, the second evaluation data D22 (the second evaluation data D23, see figure 9).

In addition, in the present embodiment, as shown in FIG. 6, the configuration is as follows, namely, when the category of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b and the defect image Pd pre-classified by the user When the categories are inconsistent, the display unit 33 accepts automatic selection of the category. Specifically, the configuration is as follows, that is, when the category of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b does not match the category of the defect image Pd pre-classified by the user, the classification estimated probability is specified When it exceeds the threshold, the display unit 33 accepts automatic selection of the category.

For example, on the screen 33a displayed on the display unit 33, select "automatic feedback" (square sign) with a mouse or the like. In addition, the screen 33a is configured to allow input of the estimated probability of classification to make the ADC accurate (correct classification). For example, the estimated probability of being inputted is "85%".

In addition, in this embodiment, the classifier learning mechanism 31a is configured to automatically select the category of the defective image Pd when the automatic selection has been accepted, and select the image Pd3 in the category of the automatically selected category. In the state included in the teacher data D1, the classifier 32 is relearned based on the teacher data D1. In the example shown in FIG. 6, the category of the defect image Pd with the estimated classification probability of 85% or more is automatically changed to the category classified by the ADC. That is, all categories of defect images Pd of No. 1 to No. 3 are automatically changed to categories classified by ADC. Furthermore, the estimated probability of classification to make ADC accurate is an example of the "specified threshold" in the scope of the patent application. Here, similar to the aforementioned "manual feedback", when the evaluation data D2 is the first evaluation data D21, the classifier 32 is relearned based on the updated teacher data D11 (refer to FIG. 7). In addition, when the evaluation data D2 is the second evaluation data D22, the classifier 32 is relearned based on the teacher data D12 (refer to FIG. 8) to which the category selection image Pd3 is added. In addition, similar to the aforementioned "manual feedback", the classifier 32 is re-learned by pressing the button of "re-learning" or the button of "re-learning under another name".

Next, the procedure of the relearning of the classifier 32 will be explained.

First, referring to FIG. 10, the operation on the side of the imaging device 10 and the inspection device 20 will be described. As shown in FIG. 10, in step S1, the imaging device 10 is used to photograph the surface of the semiconductor substrate 200. The image Ps captured by the imaging device 10 is sent to the inspection device 20.

Then, in step S2, the inspection device 20 (defect detection unit 21) detects the defect image Pd including the defect d from the image Ps of the semiconductor substrate 200. Then, the detected defect image Pd is stored in the storage unit 22 provided in the inspection device 20.

Next, referring to FIG. 11, the operation of the classification device 30 will be described. As shown in FIG. 11, in step S11, the required plurality of defect images Pd are selected from the plurality of defect images Pd stored in the inspection device 20. Furthermore, the selection of the required defect image Pd is performed by the user, for example.

Then, in step S12, the user classifies (MDC) the defect d of the selected defect images Pd. For example, the defect image Pd is displayed on the display unit 33, and input of the type (black defect db or white defect dw) of the defect image Pd is accepted.

Then, in step S13, the feature quantities (maximum brightness, minimum brightness, brightness range, etc.) are calculated for the selected plurality of defective images Pd, respectively.

Then, in step S14, a plurality of defect images Pd corresponding to the category and the feature quantity are integrated into the teacher data D1. In addition, a plurality of defect images Pd corresponding to the category and the feature amount are stored, for example, in one folder of the storage unit 34.

Then, in step S15, the classifier learning mechanism 31a makes the classifier 32 learn (generate) based on the teacher profile D1.

Then, in step S16, the learned classifier 32 is stored in the storage unit 34.

[Evaluation of the classifier based on the first evaluation data] (Manual feedback) Next, referring to FIG. 12, the evaluation procedure of the classifier 32 based on the first evaluation data D21 (teacher data D1 itself) will be described. In addition, in FIG. 12, the case where "manual feedback" is selected will be described.

First, in step S21, the classification execution mechanism 31b classifies the respective categories of the defective images Pd included in the first evaluation data D21 (the teacher data D1 itself) based on the classifier 32 learned by the classifier learning mechanism 31a. That is, the respective categories of the defective images Pd included in the teacher data D1 used for the classifier 32 to learn (generate) are classified.

Then, in step S22, the classification result is verified. Specifically, in step S22, it is determined whether the performance of the classifier 32 is sufficient (good) or not. In step S22, if it is YES, the evaluation action of the classifier 32 ends. In the case of No in step S22, proceed to step S23. Furthermore, the performance of the classifier 32 is determined based on accuracy, precision, recall, and so on. In addition, the judgment of whether the performance of the classifier 32 is sufficient may be performed by the user, or may be performed automatically by the control unit 31.

Then, in step S23, the type of the defect image Pd of the first evaluation data D21 classified by the classification actuator 31b (ADC) is not consistent with the type of the defect image Pd classified by the user in advance (MDC) The image Pd, the category of ADC classification, the category of MDC classification, and the estimated classification probability are displayed on the display unit 33 (refer to FIG. 6).

Then, in step S24, when the category of the defect image Pd of the first evaluation data D21 classified by the classification actuator 31b (ADC) is not consistent with the category of the defect image Pd pre-classified (MDC) by the user, then When the estimated classification probability is greater than or equal to a certain threshold (for example, 85%), the display unit 33 accepts the user's input for the selection of the classification. Then, the user selects the category of the defect image Pd based on the defect image Pd displayed on the display unit 33 and the estimated classification probability.

Then, in step S25, the teacher profile D1 is updated. That is, the teacher data D1 (teacher data D11, refer to FIG. 7) is updated by changing the type of the defective image Pd inconsistent with the MDC and the ADC.

Then, in step S26, the user clicks the "re-learning" button (or the "re-learning with another name" button) on the screen 33a of the display unit 33 with a mouse or the like, thereby accepting the re-learning. Then, it returns to step S21. Then, the operations of steps S21 to S26 are repeated until it is determined that the performance of the classifier 32 is sufficient.

(Automatic Feedback) Next, referring to FIG. 13, the case where the first evaluation data D21 (the teacher data D1 itself) is used as the evaluation data D2 and "automatic feedback" is selected will be described.

Furthermore, the operations of step S31 to step S33 are the same as the operations of step S21 to step S23 described above, respectively.

Then, in step S34, when the category of the defect image Pd of the first evaluation data D21 classified by the classification actuator 31b (ADC) is inconsistent with the category of the defect image Pd pre-classified (MDC) by the user, then When the estimated classification probability is greater than or equal to a certain threshold (for example, 85%), the display unit 33 accepts automatic selection of the classification. Then, the classifier learning mechanism 31a automatically selects the category of the defective image Pd when the automatic selection has been accepted.

Furthermore, the operations of step S35 and step S36 are the same as those of step S25 and step S26, respectively.

[Evaluation of the classifier based on the second evaluation data] (Manual feedback) Next, referring to FIG. 14, the evaluation procedure of the classifier 32 based on the second evaluation data D22 will be described. In addition, in FIG. 14, the case where "manual feedback" is selected will be described.

First, similar to the above-mentioned steps S1 and S2 (refer to FIG. 10), the imaging device 10 photographs the surface of the semiconductor substrate 200, the inspection device 20 (defect detection unit 21) detects the defect image Pd, and the defect map Like the preservation of Pd. Furthermore, the semiconductor substrate 200 photographed by the imaging device 10 is a different substrate (or a different part) from the semiconductor substrate 200 photographed (step S1) for the preparation of the teacher data D1.

Then, as shown in FIG. 14, in steps S41 to S44, the same operations as the above-mentioned steps S11 to S14 are performed, respectively. That is, the selection of the stored defect image Pd, the calculation of the MDC of the selected defect image Pd, the feature amount, and the integration of the defect image Pd of the MDC into the second evaluation data D22 are performed.

Then, in step S45, the classification execution unit 31b classifies the respective categories of the defective images Pd included in the second evaluation data D22 based on the classifier 32 learned by the classifier learning unit 31a.

Furthermore, the operations of steps S46 to S48 are the same as the operations of steps S22 to S24 described above.

Then, in step S49, the teacher data D1 and the second evaluation data D22 are updated. That is, the defect image Pd (referred to as the category selection image Pd3) that has been changed to the category classified by the classification executing agency 31b is added to the teacher profile D1 to update the teacher profile D1 and generate the teacher profile D12 (refer to FIG. 8 ). In addition, the second evaluation data D22 is updated by changing the type of the defect image Pd in which the MDC and ADC do not match in the second evaluation data D22, and the second evaluation data D23 is generated (see FIG. 9).

Then, in step S50, based on the teacher data D12 obtained by adding the defect image Pd of the category selected (changed) by the user to the teacher data D1, the learning is restarted.

(Automatic feedback) In the case of selecting "Automatic feedback", after step S46, perform the same actions as the above-mentioned steps S33 to S36.

(Effects of this embodiment) Next, the effects of this embodiment will be described.

In the present embodiment, as described above, the classifier learning mechanism 31a is configured as follows, that is, when the category of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b and the defect image pre-classified by the user When the categories of Pd are inconsistent, when the category selection image Pd3 is included in the teacher data D1, the classifier 32 is re-learned based on the teacher data D1. In the above category selection image Pd3, the category classified by the classification executing agency 31b is selected It is the category of the defective image Pd with inconsistent categories. Thereby, even when the category of the defect image Pd is incorrectly classified, the category of the incorrectly classified defect image Pd can be changed to the correct category. In addition, in a state where the category selection image Pd3 of the incorrectly classified category has been changed to the correct category is included in the teacher data D1, the classifier 32 is relearned based on the teacher data D1 (teacher data D11 and/or teacher data D12), Therefore, the performance of the classifier 32 can be improved.

In addition, in this embodiment, as described above, the category of the category selection image Pd3 is configured to be selected based on the estimated classification probability, which is an index of the degree of certainty of the category classification performed by the classification execution unit 31b. In this way, by selecting the category of the defective image Pd having a relatively high estimated probability of classification, it is possible to suppress erroneous selection of the category. In addition, by not selecting the category of the defective image Pd with a relatively low estimated probability of classification, it is also possible to suppress the wrong category selection.

Furthermore, in this embodiment, as described above, the estimated classification probability includes at least one of the ratio of the majority of the classification results of the plurality of classifiers 32 to the classification and the estimated classification probability output from a single classifier 32. Thereby, the majority ratio of the classification results of the plural classifiers 32 to the category (the estimated probability of classification output from a single classifier 32) is the value output from the general learning algorithm. Therefore, it can be easily based on these values. Choose a category.

In addition, in this embodiment, as described above, a display unit 33 is provided. The display unit 33 displays the defective images Pd classified by the classification actuator 31b and the classification of the classification actuator 31b is not consistent with the user classification. The category of the defect image Pd of the evaluation data D2, the category of the defect image Pd pre-classified by the user, and the estimated probability of classification. Thereby, the user can easily see whether there is a situation in which the category of the defect image Pd classified by the classification executing mechanism 31b is not consistent with the category of the defect image Pd pre-classified by the user.

In addition, in the present embodiment, as described above, the configuration is as follows, namely, when the category of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b and the category of the defect image Pd pre-classified by the user When they do not match, the display unit 33 accepts the input of the user's selection of the category, and the classifier learning mechanism 31a is configured as follows, that is, in the state where the category selection image Pd3 of the category selected by the user is included in the teacher data D1, Based on the teacher data D1 (the teacher data D11 and/or the teacher data D12), the classifier 32 is relearned. Thereby, the user can select the category of the defect image Pd while confirming the estimated probability of classification displayed on the display portion 33. That is, the estimated classification probability serves as an index of whether to select the category of the defective image Pd, and therefore, the user can easily select the category.

In addition, in the present embodiment, as described above, the configuration is as follows, namely, when the category of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b and the category of the defect image Pd pre-classified by the user If they do not match, and the estimated probability of classification is greater than or equal to the specified threshold, the display unit 33 accepts the user's input for the selection of the classification. In this way, the user's input for category selection is accepted only when the estimated probability of classification is above a certain threshold. Therefore, when the category does not need to be selected, the control of accepting the input of category selection can be omitted.

In addition, in the present embodiment, as described above, the configuration is as follows, namely, when the category of the defect image Pd of the evaluation data D2 classified by the classification execution mechanism 31b and the category of the defect image Pd pre-classified by the user When there is a mismatch, the display unit 33 accepts the automatic selection of the category, and the classifier learning mechanism 31a is configured as follows, that is, when the automatic selection has been accepted, the category of the defective image Pd is automatically selected, and the category of the automatically selected category is automatically selected. When the category selection image Pd3 is included in the teacher data D1, the classifier 32 is relearned based on the teacher data D1 (the teacher data D11 and/or the teacher data D12). In this way, the user does not need to select the category (manually), and the category is automatically selected, so that the user's time and effort can be saved.

Furthermore, in this embodiment, as described above, the display unit 33 is configured as follows, namely, when the type of the defect image Pd of the evaluation data D2 classified by the classification executing mechanism 31b and the defect image classified by the user in advance When the categories of Pd are inconsistent, and the estimated probability of classification is higher than a certain threshold, the automatic selection of the category is accepted. In this way, only when the estimated classification probability is above a certain threshold, the automatic selection of the category is accepted. Therefore, when the category does not need to be selected, the control of the input of the accepted category selection can be omitted.

In addition, in this embodiment, as described above, the evaluation data D2 includes the teacher data D1, and the classification execution mechanism 31b is configured to classify the defect image Pd included in the teacher data D1 based on the classifier 32 after learning. , And the classifier learning mechanism 31a is configured as follows, that is, in a state after the teacher data D1 is updated to include the category selection image Pd3, the classifier 32 is re-learned based on the teacher data D1 (teacher data D11). In the selection image Pd3, the category classified by the classification execution mechanism 31b is selected as the category. Thereby, even when the category of the defective image Pd included in the teacher data D1 is misclassified, since the teacher data D1 is updated, the performance degradation of the classifier 32 caused by the user's misclassification of the category can be suppressed .

Furthermore, in this embodiment, as described above, the evaluation data D2 (the second evaluation data D22) includes a defect image Pd different from the teacher data D1, and the classification execution mechanism 31b is configured based on the classifier 32 after learning. The category of the defective image Pd included in the evaluation data D2 that is different from the teacher data D1 is classified, and the classifier learning mechanism 31a is configured as follows, that is, in a state where the category selection image Pd3 is included in the teacher data D1, The classifier 32 is relearned based on the teacher data D1 (teacher data D12), and in the above-mentioned category selection image Pd3, the category classified by the classification executing agency 31b is selected as the category. Here, in order to improve the performance of the temporarily generated classifier 32, the category of the evaluation data D2 including the defective image Pd different from the teacher data D1 is sometimes classified. In this case, the category of the defect image Pd included in the evaluation data D2 is also classified by the user in advance. In addition, the user may misclassify the category due to fatigue or the like. Therefore, with the above-mentioned configuration, even when the category of the defect image Pd included in the assessment data D2 (the second assessment data D22) is misclassified, the category can be selected (corrected). The performance of the classifier 32 is improved.

[Variations] Furthermore, it should be considered that the implementations and examples disclosed this time are illustrative and not restrictive in all respects. The scope of the present invention is indicated by the scope of the patent application rather than the description of the above-mentioned embodiments and examples, and further includes all changes (variations) within the meaning and scope equivalent to the scope of the patent application.

For example, in the above embodiment, the following example is shown, that is, when the category of the image classified by ADC is inconsistent with the category of MDC classification, the category of the image with the inconsistent category is changed to the category of ADC classification, but the present invention does not Limited to this. For example, when the category of the image classified by the ADC is inconsistent with the category of the MDC classification, the category of the image with the inconsistent category can also be changed to the category after the user's re-determination. For example, when there are more than 3 categories (A, B, C,...), when the initial MDC classification is A, and the ADC classification is B, it can also be re-determined by the user Change the category to C.

In addition, in the above-mentioned embodiment, an example is shown in which the classifier is configured to classify the types (classes) of defects of the semiconductor substrate, but the present invention is not limited to this. For example, the classifier may be configured to classify objects (for example, cells, etc.) that are different from defects of the semiconductor substrate. In addition, the classifier may be configured to classify the semiconductor substrate with or without defects.

In addition, in the above-mentioned embodiment, an example is shown in which when the category of the ADC classification does not match the category of the MDC classification, the category is changed based on the estimated probability of the classification, but the present invention is not limited to this. For example, when the category of ADC classification is inconsistent with the category of MDC classification, all of them can be changed to the category of ADC classification.

In addition, in the above-mentioned embodiment, an example is shown in which the category of MDC classification is changed to the category of ADC classification when the estimated classification probability is greater than or equal to a certain threshold. However, the present invention is not limited to this. For example, it is also possible to change the category of MDC classification to the category of ADC classification based on indicators other than the estimated probability of classification.

In addition, in the above embodiment, an example of the configuration is shown. That is, when the category of the defect image classified by ADC does not match the category of the defect image classified by MDC, and the estimated classification probability is greater than or equal to a certain threshold, the display unit The input of the user's selection of the category (automatic selection of the category) is accepted, but the present invention is not limited to this. For example, it may be configured such that even when the estimated classification probability is not greater than a specific threshold, the display unit accepts the input of the user's selection of the category (automatic selection of the category).

In addition, in the above-mentioned embodiment, an example is shown in which the evaluation data includes the first evaluation data (teacher data itself) and the second evaluation data (data different from the teacher data), but the present invention is not limited to this. For example, the evaluation data may include only one of the first evaluation data and the second evaluation data. In addition, there may be a mixture of teacher data and data different from the teacher data in the evaluation data.

In addition, in the above-mentioned embodiment, the following example is shown, that is, when the category of the defect image classified by ADC does not match the category of the defect image classified by MDC, the category of MDC classification is changed to the category of ADC classification. The invention is not limited to this. In the present invention, when the category of the defect image classified by ADC is inconsistent with the category of the defect image classified by MDC, the category of the defect image classified by ADC is inconsistent with the category of the defect image classified by MDC (below , Called the inconsistent image) category, can also be re-determined by the user to select the MDC category. That is, when the teacher data and the evaluation data are different, when the ADC classification is inconsistent with the MDC classification, and the MDC is re-determined (confirmed) by the user and the MDC is accurate, it will be re-determined by the MDC The category is added to the teacher's profile as the category of the inconsistent image, and the learning is restarted. For example, in the example of Figure 15 below, the defect image of No.1 of the category selected by the ADC classification and the defect image of No.4 of the category selected by the MDC classification are included in the teacher data, and the process is repeated learn.

In addition, in the above-mentioned embodiment, an example is shown in which the category of MDC classification is automatically changed to the category of ADC classification when the estimated classification probability is greater than or equal to a certain threshold (85%), but the present invention is not limited to this. For example, when it is lower than a relatively low threshold (20%, etc.) that is different from the above-mentioned specific threshold, the category of the MDC classification can be included in the teacher data without changing, and the learning can be restarted. Thereby, when the automatic selection of the category (automatic feedback) has been accepted, and the estimated probability of the classification has not reached a relatively low threshold, the defective image with the category of the MDC classification can be added during the relearning. For example, when the threshold of the estimated probability of classification to make ADC accurate is set to 85%, and the threshold of estimated probability of classification to make MDC accurate is set to 20% or less, in the example of Figure 16 below, set No. 1 The defect images of No.2, No.4, and No.5 are included in the teacher's profile, and the study should be restarted.

10: Imaging device 11: Illumination part 12: Optical system 13: Imaging part 14: Stage 15: Stage driving part 20: Inspection device 21: Defect detection part 22: Memory part 30: Classification device 31: Control part 31a: Classification Machine learning mechanism 31b: Classification execution mechanism 32: Classifier 33: Display section 33a: Screen 34: Memory section 100: Image classification system 200: Semiconductor substrate d: Defect db: Black defect (defect) dw: White defect (defect) D1, D11, D12: Teacher data D2, D21, D22, D23: Evaluation data Pd: Defect image (image) Pd1: Defect image Pd2: Defect image Pd3: Category selection image Ps: image

Figure 1 is a block diagram to illustrate the image classification system. Figure 2 is a diagram showing an image obtained by photographing a semiconductor substrate. Figure 3 is a block diagram used to illustrate the classification device. Figure 4 is a diagram used to illustrate teacher information. Figure 5 is a diagram used to illustrate the evaluation data. Figure 6 is a diagram showing the screen of the display part of the classification device. Figure 7 is used to illustrate the updated teacher information (1). Figure 8 is used to illustrate the updated teacher information (2). Figure 9 is a diagram used to illustrate the updated evaluation data. Fig. 10 is a flowchart for explaining the actions of the camera device and the inspection device side. Figure 11 is a flowchart for explaining the learning (generation) of the classifier. Figure 12 is a flowchart used to describe the process for evaluating the performance of the classifier based on the first evaluation data (in the case of manual feedback). Figure 13 is a flowchart used to describe the process for evaluating the performance of the classifier based on the first evaluation data (in the case of automatic feedback). Figure 14 is a flowchart used to describe the process for evaluating the performance of the classifier based on the second evaluation data (in the case of manual feedback). Figure 15 is a diagram (1) used to illustrate the re-judgment of the types of changes. Figure 16 is a diagram (2) used to illustrate the re-judgment of the types of changes.

30: Sorting device

31: Control Department

31a: Classifier learning institution

31b: Classification implementing agency

32: classifier

33: Display

34: Memory Department

Claims (8)

A classification device that classifies the category of unknown images and has: A classifier learning mechanism for learning the above-mentioned classifier by performing machine learning based on teacher data, the above-mentioned teacher data including a plurality of images pre-classified by a user into any one of a plurality of categories; and A classification execution agency, which, based on the above-mentioned learned classifier, classifies the evaluation data including a plurality of images into any of the above-mentioned plural categories; The above-mentioned classifier learning mechanism is constituted as follows. That is, when the category of the image of the evaluation data classified by the above-mentioned classification execution mechanism is inconsistent with the category of the image pre-classified by the user, the classification execution mechanism will The category of the classification or the category re-determined by the user is selected as the category of the above-mentioned image with inconsistent categories. The category-selected image is included in the above-mentioned teacher data, and the above-mentioned classifier is re-learned based on the above-mentioned teacher data, The above-mentioned categories constituting the above-mentioned category selection image are selected based on the estimated classification probability, which is an indicator of the degree of certainty of the above-mentioned category classification performed by the above-mentioned classification enforcement agency, Furthermore, it is provided with a display unit that displays the category of the image of the evaluation data classified by the classification implementing agency and the category of the image of the evaluation data classified by the category implementing agency and the category of the image of the evaluation data classified by the user The category of the pre-classified image and the estimated probability of the above classification, The configuration is as follows, that is, when the category of the image of the evaluation data classified by the above-mentioned classification enforcement agency does not match the category of the image pre-classified by the user, the above-mentioned display unit accepts the automatic selection of the above-mentioned category, and The above-mentioned classifier learning mechanism is configured as follows, that is, when the above-mentioned automatic selection situation has been accepted, the above-mentioned category of the above-mentioned image is automatically selected, and the above-mentioned category selection image of the above-mentioned category has been automatically selected is included in the above-mentioned teacher information In the state, relearn the above-mentioned classifier based on the above-mentioned teacher information. The classification device of claim 1, wherein the estimated classification probabilities include at least one of the majority ratio of the classification results of the plurality of the classifiers to the classification results, and the estimated classification probabilities output from a single classifier. For example, the classification device of claim 1 or 2, which is constituted as follows, that is, when the type of the image of the evaluation data classified by the above-mentioned classification executing agency is inconsistent with the type of the image pre-classified by the user, the above-mentioned The display unit accepts the input of the above-mentioned user's selection of the above-mentioned category, and The classifier learning mechanism is configured to relearn the classifier based on the teacher data in a state where the category selection image of the category selected by the user is included in the teacher data. For example, the classification device of claim 3, which is constituted as follows, that is, when the category of the image of the evaluation data classified by the above-mentioned classification execution agency is inconsistent with the category of the image pre-classified by the user, the above-mentioned classification is inferred When the probability is higher than a certain threshold, the display unit accepts the input of the user's selection of the category, and The classifier learning mechanism is configured to relearn the classifier based on the teacher data in a state where the category selection image of the category selected by the user is included in the teacher data. For example, the classification device of any one of claims 1 to 4, which is constituted as follows, that is, when the category of the image of the evaluation data classified by the above-mentioned classification executing agency and the category of the image pre-classified by the above-mentioned user Inconsistent, and the estimated probability of the classification is greater than a certain threshold, the display unit accepts the automatic selection of the classification, and The above-mentioned classifier learning mechanism is configured as follows, that is, when the above-mentioned automatic selection situation has been accepted, the above-mentioned category of the above-mentioned image is automatically selected, and the above-mentioned category selection image of the above-mentioned category has been automatically selected is included in the above-mentioned teacher information In the state, relearn the above-mentioned classifier based on the above-mentioned teacher information. Such as the classification device of any one of request items 1 to 5, wherein the above-mentioned evaluation data includes the above-mentioned teacher data, The above-mentioned classification execution mechanism is configured to classify the above-mentioned categories of the above-mentioned images contained in the above-mentioned teacher information based on the above-mentioned classifier after learning, and The above-mentioned classifier learning mechanism is constituted as follows, that is, after updating the above-mentioned teacher information in a state that includes the above-mentioned category selection image in which the category classified by the above-mentioned classification execution mechanism is selected as the above-mentioned category, the above-mentioned The classifier relearns. For example, the classification device of any one of claim 1 to 6, wherein the above-mentioned evaluation data includes images different from the above-mentioned teacher data, The above-mentioned classification execution mechanism is configured to classify the categories of images contained in the above-mentioned evaluation materials that are different from the above-mentioned teacher materials based on the above-mentioned classifier after learning, and The above-mentioned classifier learning mechanism is configured as follows, that is, in a state where the above-mentioned category selection image of the category classified by the above-mentioned classification execution mechanism is selected as the above-mentioned category is included in the above-mentioned teacher data, the above-mentioned classifier is re-learned based on the above-mentioned teacher data . An image classification system with: A camera device for taking images; and A classification device, which classifies the category of the unknown image captured by the above-mentioned camera unit; The above classification device includes: A classifier learning mechanism for learning the above-mentioned classifier by performing machine learning based on teacher data, the above-mentioned teacher data including a plurality of images pre-classified by a user into any one of a plurality of categories; and A classification execution agency, which, based on the above-mentioned learned classifier, classifies the evaluation data including a plurality of images into any of the above-mentioned plural categories; The above-mentioned classifier learning mechanism is structured as follows, that is, when the category of the image of the evaluation data classified by the above-mentioned classification implementing agency is inconsistent with the category of the image pre-classified by the above-mentioned user, the classification is performed by the above-mentioned classification implementing agency. The category or the category re-determined by the user is selected as the category of the above-mentioned image whose category is inconsistent. When the category-selected image is included in the above-mentioned teacher data, the above-mentioned classifier is relearned based on the above-mentioned teacher data, The structure is as follows, that is, the category of the category selection image is selected based on the estimated probability of the category, which is an index of the degree of certainty of the category classification performed by the category enforcement agency, Furthermore, it is provided with a display unit that displays the categories of the images of the evaluation data classified by the classification implementing agencies and the categories of the images of the evaluation data classified by the category implementing agencies related to the images in which the category classified by the classification implementing agency and the category of the user category are inconsistent. The category of the pre-classified image and the estimated probability of the above classification, The configuration is as follows, that is, when the category of the image of the evaluation data classified by the above-mentioned classification enforcement agency does not match the category of the image pre-classified by the user, the above-mentioned display unit accepts the automatic selection of the above-mentioned category, and The above-mentioned classifier learning mechanism is structured as follows, that is, when the above-mentioned automatic selection situation has been accepted, the above-mentioned category of the above-mentioned image is automatically selected, and the above-mentioned category selection image of the above-mentioned category has been automatically selected is included in the above-mentioned teacher information In the state, relearn the above-mentioned classifier based on the above-mentioned teacher information.

Images