TW201413239A - Training data verification apparatus, training data generation apparatus, image classification apparatus, training data verification method, training data generation method, and image classification method - Google Patents

Abstract

Training data showing a plurality of training images and their initial categories is stored in a storage part (614), and with respect to each of a plurality of feature value types, an evaluation value showing validity in the case where each training image is assumed to belong to each of a plurality of categories is obtained by an evaluation value obtaining part (611). In a category candidate specifying part (612), a representative value of evaluation values of the plurality of feature value types obtained for each training image is calculated for each category, and a category candidate to which the training image should belong is determined on the basis of a plurality of representative values obtained for the plurality of categories. It is therefore possible to easily specify the appropriate category candidate based on feature values of the plurality of feature value types. A category non-coincidence image which is a training image in which the initial category is not coincident with the category candidate is displayed together with the category candidate on a display (55). This makes it possible to easily generate a highly-reliable training data.

Description

Guidance data verification device, guidance data production device, image classification device, guidance data verification method, guidance data production method, and image classification method

The present invention relates to a technique for verifying training data for learning a classifier for classifying images, a technique for producing guidance materials using the technology, and a technique for classifying images.

In the production of a semiconductor substrate, a glass substrate, a printed wiring board, or the like, an appearance inspection is performed using an optical microscope, a scanning electron microscope, or the like in order to examine defects such as foreign matter, damage, or etching failure. Further, for the defect detected in the inspection step, the cause of the defect is specified by performing detailed analysis, and countermeasures against the defect are implemented. In recent years, as the pattern on the substrate is complicated and miniaturized, the type and number of defects detected are increased, and automatic classification that automatically detects the defects detected in the inspection step is also used. By automatic classification, the analysis of defects can be performed quickly and efficiently.

In the automatic classification, a classifier using a neural network or a decision tree, discriminant analysis, or the like is used. In order for the classifier to perform automatic classification, it is necessary to prepare a signal containing the defect image and the type indicating the type (ie, the type of the defect image). The guidance material allows the classifier to learn. Japanese Laid-Open Patent Publication No. 2010-91401 (Document 1) discloses a method of producing a higher quality guide material by judging whether or not a category to which a defective image is given in advance is a category to which a defective image should be subordinated. . Specifically, all kinds of feature amounts are processed by setting a feature amount range including a representative value of the feature amount for the type based on the dispersion of various feature amounts of the plurality of defect images belonging to each category, A feature quantity of each defect image is included in a category within the range of the feature quantity of the category for voting. Then, when the category having the largest number of votes in the plurality of categories is different from the category to which the defective image belongs, the point is output.

Further, a method for calculating a plurality of feature amounts for a plurality of indication defect images and calculating a feature space based on feature amounts for each indication defect is disclosed in Japanese Laid-Open Patent Publication No. H11-344450 (Document 2). In the classification parameter indicated by the discriminant function between the categories, the statistical significance indicating the indication data indicating the correspondence between the plurality of indication defects and the category corresponding to the type of the defect is determined, and there is an indication that the performance may be lowered. The defect image is displayed on the screen to correct the indication data.

Furthermore, the category determination by the statistical processing in Document 1 is particularly the case where the distribution of various feature quantities of a plurality of defect images (guide images) belonging to each category follows a normal distribution (or can be regarded as following a normal distribution). It is effective, but when the distribution of the feature amount shows that the multimodality or the like does not follow the normal distribution, there is a case where the result (category) in the document 1 does not coincide with the judgment of the operator, or the operator cannot reasonably interpret the result. In this case, there is a problem in the production of guidance materials with high reliability. Further, in the method of Document 2, when the type of the guidance image is presented to the operator, it is necessary to calculate the discriminant function between the categories by complicated processing. Therefore, a novel method for easily specifying an appropriate category candidate based on a plurality of types of feature amounts for each guidance image is sought.

The present invention is directed to a guidance material verification device for verifying guidance materials for learning a classifier for classifying images, and aims to easily specify an appropriate category candidate for a plurality of types of feature amounts for each guidance image.

The guidance material verification device of the present invention includes: a storage unit that stores a plurality of guidance images each assigned to any one of a plurality of categories, and guidance materials indicating categories of the plurality of guidance images; and an evaluation value acquisition unit Each of the plurality of types of the feature amount is obtained as an evaluation value indicating the validity of each of the plurality of categories when the guidance image belongs to each of the plurality of categories; the category candidate specifying unit is configured as described above In each of the plurality of types, the representative value of the evaluation value of the feature quantity in the plurality of types is obtained, and the plurality of representative values in the plurality of categories are determined as The category to which the respective guidance images belong is specified as a category candidate; and the display control unit uses the guidance image of the category indicated by the guidance material and the candidate of the category as a category-specific image. And each of the at least one type of dissimilar image is displayed on the display unit together with the category candidate.

According to the present invention, it is possible to easily specify an appropriate category candidate based on a plurality of types of feature amounts for each guidance image. Further, by displaying the category difference image together with the category candidate on the display unit, it is possible to easily create guidance materials having high reliability.

In a preferred aspect of the present invention, the evaluation value acquisition unit generates a histogram of the feature amount of the guidance image assigned to each of the plurality of categories for each type of the feature amount, and includes the above-described respective The section of the feature quantity of the guidance image is obtained as the attention section, and each of the plurality of categories of the respective guidance images is acquired based on the frequency of the attention range in the plurality of histograms of the plurality of categories. The above evaluation value.

In this case, it is preferable that the evaluation value acquisition unit is based on the above The evaluation value of each of the plurality of categories of the respective guidance images is obtained by the frequency of the plurality of sections adjacent to the both of the attention periods in the plurality of histograms. Thereby, even when the number of guidance images is small, the category candidates can be appropriately determined.

Further, in the plurality of histograms, the frequency of the attention range in the histogram for one category is a specific number or less, and the above-mentioned attention range in the histogram for each of the other categories When the frequency is 0, the evaluation value acquisition unit does not acquire the evaluation value. Thereby, the type (the evaluation value) of the inappropriate feature amount which unreasonably raises the representative value of the evaluation value can be excluded.

More preferably, the display control unit displays each of the at least one different type of image on the display unit together with the category indicated by the guidance material and the category candidate.

The present invention is also directed to a guidance material production device having a guidance data verification device, an image classification device having a guidance material production device, a guidance data verification method for verifying guidance materials, a guide data production method including a guide data verification method, and use An image classification method for guiding materials for guiding the production of data.

The above and other objects, features, aspects and advantages of the present invention will be apparent from the accompanying drawings.

1‧‧‧Image classification device

2‧‧‧ camera device

4‧‧‧Check. Sorting device

5‧‧‧Main computer

8‧‧‧Recording media

9‧‧‧Semiconductor substrate

21‧‧‧Photography Department

22‧‧‧ platform

23‧‧‧ Platform Drive Department

41‧‧‧Defect Detection Department

42‧‧‧Automatic Defect Classification Department

51‧‧‧CPU

52‧‧‧ROM

53‧‧‧RAM

54‧‧‧Fixed discs

55‧‧‧ display

56‧‧‧ Input Department

56a‧‧‧ keyboard

56b‧‧‧mouse

57‧‧‧Reading device

58‧‧‧Communication Department

61‧‧‧Guidance Materials Production Department

62‧‧‧Learning Department

71a~71d‧‧‧ guidance image

72‧‧‧Category display area

80‧‧‧ program

211‧‧‧Lighting Department

212‧‧‧Optical system

213‧‧‧Photographic components

421‧‧‧ classifier

610‧‧‧Data Computing Department

611‧‧‧ Evaluation Value Acquisition Department

612‧‧‧ Category Specific Department

613‧‧‧Display Control Department

614‧‧‧Memory Department

615‧‧‧Characteristic calculation unit

710‧‧‧Guide materials

A, B, C‧‧‧Focus

H1~H4‧‧‧Histogram

S11~S14, S21~S29‧‧‧ steps

Z0‧‧‧Circular area

Z1~Z4‧‧‧ring area

Fig. 1 is a view showing the configuration of an image classifying device.

Fig. 2 is a view showing a classification process of a defective image.

Fig. 3 is a view showing the configuration of a host computer.

Fig. 4 is a view showing the functional configuration of the host computer.

Fig. 5 is a view showing the functional configuration of the guidance data creation unit.

Fig. 6 is a view showing a processing flow for creating a guide material and learning the classifier.

Figure 7 is a diagram showing a plurality of guide images displayed on a display.

Figure 8 is a diagram showing a plurality of histograms for a plurality of categories.

Figure 9 is a diagram showing a plurality of guide images displayed on a display.

Figure 10 is a diagram for explaining the degree of heterogeneity.

Figure 11 is a diagram showing a plurality of guidance images.

Figure 12 is a diagram showing a plurality of guidance images.

Fig. 1 is a view showing a schematic configuration of an image classification device 1 according to an embodiment of the present invention. In the image classification device 1, a defect image indicating a defect on the semiconductor substrate 9 (hereinafter simply referred to as "substrate 9") is acquired, and the defect image is classified. The image classification device 1 has an imaging device 2 that captures an inspection target area on the substrate 9; The sorting device 4 automatically classifies the defect into a category to which the defect should be subordinated when the defect is detected based on the defect inspection from the image data of the image pickup device 2; and the host computer 5 controls the image classification device 1 The overall action, and generate checks. A classifier 421 for classifying defects in the sorting device 4. The type (category) of the defect existing on the substrate 9 is, for example, a defect, a protrusion, a disconnection, a short circuit, or a foreign matter. Further, the imaging device 2 is incorporated into the production line of the substrate 9, and the image classification device 1 is a so-called inline type system. The image classification device 1 can also be understood as a device that adds a function of automatic defect classification to the defect inspection device.

The imaging device 2 includes an imaging unit 21 that captures an inspection target area on the substrate 9 to acquire image data, a stage 22 that holds the substrate 9 , and a platform driving unit 23 that relatively aligns the stage 22 with respect to the imaging unit 21 mobile. The imaging unit 21 has an illumination unit 211 that emits illumination light, an optical system 212 that guides illumination light to the substrate 9, and The light from the substrate 9 is incident; and the imaging element 213 converts the image of the substrate 9 imaged by the optical system 212 into an electrical signal. The platform drive unit 23 includes a ball screw, a guide rail, a motor, and the like, and the main computer 5 controls the stage drive unit 23 and the imaging unit 21 to take an inspection target area on the substrate 9.

an examination. The sorting device 4 has a defect detecting unit 41 that detects a defect while processing image data of the inspection target region, and an automatic defect classifying unit 42 that classifies the defect image. The defect detecting unit 41 has a dedicated circuit for processing image data of the inspection target region at a high speed, and performs defect inspection of the inspection target region by comparison of the captured image with a reference image having no defect or image processing. The automatic defect classification unit 42 includes a central processing unit (CPU) that performs various types of arithmetic processing, a memory that stores various kinds of information, and the like, and performs a defect using a classifier 421 using a neural network, a decision tree, a discriminant analysis, or the like. Classification (ie classification of defect images).

FIG. 2 is a view showing a flow of classification of defective images by the image classifying device 1. First, the substrate 9 is photographed by the image pickup device 2 shown in FIG. The defect detecting unit 41 of the sorting device 4 acquires image data (step S11). When the defect detecting unit 41 detects the defect in the inspection target region and detects the defect (step S12), the data of the image of the defective portion (that is, the defect image) is transmitted to the automatic defect classifying unit 42. The automatic defect classification unit 42 calculates a plurality of types of feature amounts of the defect image (step S13), and inputs the feature amount of the defect image to the classifier 421 of the automatic defect classification unit 42 to output the classification result. That is, the defect image is classified by the classifier 421 to any of a plurality of categories (step S14). In the image classification device 1, when the defect detection unit 41 detects a defect, the feature amount is calculated immediately, and the plurality of defect images are automatically classified at high speed.

Next, the learning of the classifier using the host computer 5 will be described. Fig. 3 is a view showing the configuration of the host computer 5. The main computer 5 becomes a variety of arithmetic processing The CPU 51, a ROM (Read Only Memory) 52 for storing a basic program, and a random access memory (RAM) 53 for storing various kinds of information are connected to a general computer system of a bus bar. In the bus line, the fixed disk 54 for information storage, the display 55 as a display unit for displaying various information such as images, and the input from the operator are connected via an interface (I/F) or the like as appropriate. The keyboard 56a and the mouse 56b (hereinafter collectively referred to as "input unit 56"), a reading device 57 for reading information from a computer-readable recording medium 8 such as a compact disc, a magnetic disk, a magneto-optical disc, and the like A communication unit 58 that transmits and receives signals between other components of the sorting device 1.

In the host computer 5, the program 80 is read from the recording medium 8 via the reading device 57 and stored in the fixed disk 54, and the program 80 is copied to the RAM 53, and the CPU 51 executes the arithmetic processing in accordance with the program in the RAM 53. .

4 is a block diagram showing the functions of the classifier learning by the CPU 51, the ROM 52, the RAM 53, the fixed disc 54, and the like of the host computer 5, and also shows the check. Classification device 4. The host computer 5 has a guidance material creation unit 61 that creates guidance materials for learning the classifier, and a learning unit 62 that learns the classifier using the guidance materials.

The instructional material includes information on the guidance image of the defect image, the feature quantity of the guidance image, and the information indicating the category of the defect, that is, the instruction signal, as the feature quantity of the guidance image, for example, the area of the defect, the average brightness, and the week The long axis, the flatness, the slope of the long axis when the defect is similar to the ellipse, and the like. In the learning unit 62, the feature amount of the guidance image read from the instruction material is input to the classifier (not shown) in the host computer 5 so that the output of the classifier is the same as the indication signal indicating the type of the defect. In the manner of learning, the learning result, that is, the classifier 421 after learning (accurately, information indicating the value of the structure or variable of the classifier 421) is transferred to the automatic defect classifying unit 42.

Fig. 5 is a view showing the functional configuration of the instruction material creating unit 61 of the host computer 5. The block diagram also shows the learning unit 62. The guidance material creation unit 61 includes a data calculation unit 610, a display 55, and an input unit 56. The data calculation unit 610 includes an evaluation value acquisition unit 611, a category candidate specifying unit 612, a display control unit 613, a storage unit 614, and a feature amount calculation unit 615. Details of the processing of the data calculation unit 610 will be described below. Furthermore, the functions of the data calculation unit 610 and the learning unit 62 can be constructed by a dedicated circuit, or a dedicated circuit can be used locally.

Fig. 6 is a view showing a processing flow for creating a guide material and learning the classifier. In the guidance material creation unit 61, first, the data of a plurality of guidance images (for example, thousands of guidance images) which are the defect images for guiding the material production are stored in the guidance material creation unit 61 shown in FIG. The memory unit 614 prepares (step S21). Further, the guidance image can be obtained by the imaging device 2 and the defect detecting unit 41 shown in FIG. 1, and can be separately prepared.

The feature amount calculation unit 615 calculates a plurality of feature numbers (for example, 100 to 200 types) of all the guidance images (step S22). The calculated feature amount is stored in the memory unit 614. Further, on the display 55, each guidance image is displayed, and display for prompting the input of the type of the guidance image is performed, and the input unit 56 accepts the input of the category by the operator (step S23). In the following description, the category input by the operator for each guidance image in the process of step S23 is referred to as an "initial category". Furthermore, the initial category of each guide image can also be automatically determined by a specific algorithm.

FIG. 7 is a diagram illustrating a plurality of guidance images displayed on the display 55. A category display area 72 is provided below each of the guidance images 71a, 71b, 71c, and 71d. In the guidance images 71a to 71d of FIG. 7, the initial categories (names) are respectively referred to as "large defect" and "medium defect". ", "protrusion", "middle protrusion". In this manner, a plurality of guidance images each of which is assigned to any of a plurality of categories, and a plurality of guidance images are prepared The guidance material 710 of the category is stored in the memory unit 614. In the present embodiment, a plurality of feature quantities in each guidance image are also included in the instruction material 710. Further, each of the plurality of types is assigned a number, and the value of the category variable associated with the guidance image is changed to the initial category number based on the determination of the initial type of each guidance image.

When the guidance material 710 is prepared, the evaluation value acquisition unit 611 generates a histogram of the feature amounts of the guidance images assigned to each of the plurality of categories in each of the feature types. Fig. 8 is a view showing a plurality of histograms for a plurality of categories among one type of feature quantities. In the example of FIG. 8, 5348 guidance images are assigned to any of the 4 categories of numbers 1 to 4, the category 1 number contains 1578 guide images, and the number 2 category contains 2849 guide images. The number 3 category contains 688 guide images, and the number 4 category contains 133 guide images. Moreover, in FIG. 8, the feature quantity of each guidance image is normalized, and it is quantized into 100 sections, and shows the frequency of each section. In FIG. 8, symbols H1, H2, H3, and H4 are attached to the histograms of the categories of numbers 1 to 4, respectively. In addition, the histogram of the feature amount in the evaluation value acquisition unit 611 may be a display of the frequency distribution, and the graph indicating the frequency distribution is substantially equivalent to the histogram.

The evaluation value acquisition unit 611 acquires an evaluation value indicating the validity (may also be understood as a probability) when the respective guidance images belong to each of the plurality of categories for each type of the feature amount (step S24). For example, the image guide as a guide attention image characteristic amount with respect to one type, comprising an image characteristic amount of the guidance section as notable attention interval in attention interval, the category number of the image guide 1 x ₁ exists There are x ₂ guide images of category ₂ , x ₃ guide images of category ₃ , and 4 guide images of category ₄ (ie, histograms H1, H2, H3) The frequencies of the attention intervals in H4 are x ₁ , x ₂ , x ₃ , and x ₄ , respectively. Furthermore, when it is assumed that the initial category of the attention guidance image is the category of the number 2, one of the x ₂ is the attention guidance image. In this case, it is assumed that the evaluation value P _n of the case where the attention instruction image belongs to the category of the number n (here, n is any one of 1 to 4) is ((x _n /( The form of x ₁ + x ₂ + x ₃ + x ₄ )) × 100) [%] is obtained.

In particular, attention to the guidance image feature amount included in Figure 8 is attached to the symbol interval labeled A, the histogram H1, H2, H3, frequency of section A of attention are a ₁ th, a ₂ th, A _{In the case of three} (the frequency in the histogram H4 is 0), the evaluation value P ₁ indicating the validity of the case where the attention guidance image belongs to the category of the number ₁ is ((a ₁ /(a ₁ + a _{2 )} +a ₃ )) × 100) [%]. Further, the evaluation value P ₂ indicating the validity of the case where the attention guidance image belongs to the category of the number 2 is ((a ₂ /(a ₁ + a ₂ + a ₃ )) × 100) [%], indicating the assumption The evaluation value P ₃ of the case of the case of the category of the number 3 is ((a ₃ /(a ₁ + a ₂ + a ₃ )) × 100) [%]. In addition, the evaluation value P ₄ indicating the validity of the case where the attention guidance image belongs to the category of the number ₄ is 0%. Furthermore, in Fig. 8, the frequency of the point is indicated by a symbol attached to the black dot.

Further, on the image feature amount of the attention guidance included in Figure 8 is attached to a standard symbol interval B, the frequency of the histogram H1, H2, H3 of attention in section B were respectively th b _1, b ₂ th, b ₃ th In the case where the frequency in the histogram H4 is 0, the evaluation value P ₁ indicating the validity of the case where the attention guidance image belongs to the category ₁ is (b ₁ /(b ₁ +b ₂ +b) ₃ )) × 100) [%]. In addition, the evaluation value P ₂ indicating the validity of the case where the attention guidance image belongs to the category of number 2 is ((b ₂ /(b ₁ +b ₂ +b ₃ ))×100)[%], indicating the assumption The evaluation value P ₃ of the case of the case of the category of the number 3 is ((b ₃ /(b ₁ +b ₂ +b ₃ ))×100)[%]. In addition, the evaluation value P ₄ indicating the validity of the case where the attention guidance image belongs to the category of the number ₄ is 0%.

Further, the guidance image to the feature amount of the attention in Figure 8 is attached symbol included in the section labeled C, the histogram H3, H4 in the frequency range C of attention are th c _3, c ₄ th sum (histogram H1, When the frequency in H2 is 0), the evaluation value P ₃ indicating the validity of the case where the attention guidance image belongs to the category of number ₃ is ((c ₃ /(c ₃ +c ₄ ))×100) [ %]. Further, the evaluation value P ₄ indicating the validity of the case where the attention instruction image belongs to the category of No. 4 is ((c ₄ /(c ₃ + c ₄ )) × 100) [%]. In addition, the evaluation value P ₂ indicating the validity of the case where the attention instruction image belongs to the category number 1 and the evaluation value P 2 when the assumption is the category of the number ₂ are 0%.

As described above, the evaluation value acquisition unit 611 sets the section including the feature amount of each guidance image as the attention section, and acquires the guidance map based on the frequency of the attention section in the plurality of histograms for the plurality of categories. The evaluation value of each of the plural categories. In this way, in each of the guidance images, a combination of four evaluation values among the four types of categories is obtained for one type of the feature amount (for example, "the category of the number 1 is 25%, the category of the number 2 is 72%," When the number of the number 3 is 3% and the category of the number 4 is 0%, the evaluation value group is N. When there are N types of the feature quantity, N evaluation value groups are obtained in principle. The above processing is performed for all the guidance images, and N evaluation value groups are acquired (in principle) in all the guidance images. In the above processing, even when the histogram has a multimodality or the like, an evaluation value corresponding to the unevenness of the histogram can be obtained. In addition, it is assumed that the evaluation value of the validity when each of the guidance images belongs to each of the plurality of categories can be obtained by an operation other than the above.

On the other hand, for example, in the right side region in the horizontal axis of Fig. 8, only the histogram H4 of the category of No. 4 is distributed. In this case, even if the frequency of the histogram H4 in the attention interval is extremely low as 1 or 2, the evaluation value of the category of the number 4 is necessarily 100%, which is unreasonably improved by the following processing. The representative value of the evaluation value of the category of No. 4. Therefore, the frequency of the attention interval in the histogram for a category in a plurality of histograms of a plurality of categories of one type of feature quantity is a specific number (for example, 1 or 2). In the case where the frequency of the attention section in the histogram of each of the other categories is 0, the evaluation value is not obtained for the type of the feature amount.

Then, in the category candidate specifying unit 612, the representative value of the evaluation value among the plurality of types of feature amounts (accurately, all types of the evaluation values are obtained) is obtained for each of the guidance images (steps) S25). Here, the representative value of the evaluation value indicates a value near the center of the range in which the evaluation value is distributed, and is an average value or a central value. The representative value of the evaluation value of each category in each guidance image is a value indicating the validity of considering a plurality of (all) feature amounts when the guidance image belongs to the category.

When a plurality of representative values in the plurality of categories are obtained for each guidance image, the category determined to be the dependent person of the guidance image based on the plurality of representative values is specified as the category candidate (step S26). Specifically, in each of the guidance images, a category in which the maximum value among the plurality of representative values is specifically obtained is selected as the category candidate. However, when the maximum value is less than a specific threshold (for example, 50%), it is preferable to assign "unclassifiable" to the guidance image as a category candidate.

In the display control unit 613, the guidance image having the initial category and the category candidate indicated by the guidance material 710 is displayed as a category-specific image, and a plurality of different types of images are displayed on the display 55 (step S27). FIG. 9 is a view showing a plurality of guidance images displayed on the display 55. In the present embodiment, the guidance images 71b and 71c other than the category-specific images 71a and 71d are also displayed on the display 55. In the category display area 72 for the guidance images 71a to 71d in FIG. 9, the category candidates (names) are respectively referred to as "unclassifiable", "medium defect", "middle protrusion", and "unclassifiable". Further, in the category display area 72, both the initial category and the category candidates are displayed. Further, the background color of the category display area 72 for the category-specific images 71a, 71d is different from the background color of the category display area 72 for the other guide images 71b, 71c of the image different from the category, and the emphasized categories are different. Images 71a, 71d.

Then, the operator determines the final category by considering the initial category and the category candidate for each type of dissimilar image and referring to the dissimilar image of the category (step S28). Then, the input unit 56 accepts the input from the final category of the operator, and changes or maintains the category (the value of the category variable) associated with the guidance image that is the image of the different type. In this manner, the guidance material 710 is updated. Of course, it is also possible to input the final category of the guidance image other than the category-specific image. However, it is preferable to maintain the current category (initial category) for the guidance image in which the initial category and the category candidate are identical.

When the final category of each guidance image is determined, the updated guidance material 710 including the information of all the guidance images and the information on the feature quantity and the category of the guidance image is transferred from the storage unit 614 to the learning unit 62, and the learning unit 62 The guidance material 710 is used to perform the learning of the classifier (step S29). That is, the value of the variable constituting the classifier (see FIG. 4) is determined, or the structure is determined to generate the classifier 421. Thereby, the processing of FIG. 6 in which the guidance material is created and the classifier is learned is completed.

Here, in the processing of FIG. 6, for each type of dissimilar image, a category candidate having a higher degree of validity after considering all kinds of feature amounts is presented. Therefore, it has been decided that the guidance material of the final category does not easily contain a guidance image associated with the category that contradicts the feature quantity space. That is, the reliability of the guidance material is improved. Thus, it is expected that the learning result (classifier) using such guidance material has a high classification performance.

Furthermore, if the category dissimilar image is removed from the original instruction material and the following processing of the processing of steps S24 to S26 is repeated several times, all the guidance images assigned to a certain category as the initial category can be used. The number of repetitions of the processing before being distinguished by the category-specific image is correspondingly divided into a plurality of groups. Each group can be understood as a collection of guidance images that are approximately equal in degree of heterogeneity for the category, and the degree of heterogeneity differs from one another in a plurality of groups. In Figure 10, the plural of the degree of heterogeneity is abstractly represented For example, a guidance image that is specified as a different type of image in the first processing can be understood as a "heterogeneous person" included in the outermost annular region Z1. The guidance images specified as the different types of images in the second to fourth processings are respectively included in the annular regions Z2, Z3, and Z4 whose degrees of heterogeneity are gradually reduced. The guidance image that is not specified as a different type of image in the fourth processing is included in the central circular area Z0, and can be understood as a typical guidance image of the category.

Next, the superiority of the above-described processing of the specific category candidate will be described. In the image classification device 1, in the process of FIG. 6, the class difference image can be removed from the original guidance material, and the processes of steps S24 to S26 are repeated several times, where the self-guided material is removed. The result of the case where the difference image of the type is repeated and the processing of steps S24 to S26 is repeated four times (hereinafter referred to as "the result of this processing example") is compared with the result of the processing of the comparative example. Here, the processing of the comparative example is the same as the method of the Japanese Patent Laid-Open Publication No. 2010-91401 (Document 1), and the processing of all kinds of feature quantities is performed as follows, that is, according to a plurality of guide figures belonging to each category For the dispersion of various feature amounts, a feature amount range including a representative value of the feature amount is set for the type, and a feature amount of each guidance image is included in the category of the feature amount range of the type to vote. Then, the category with the largest number of votes in the plurality of categories is set as the category candidate. In addition, when the difference between the number of votes having the largest number of votes and the number of votes having the second largest number of votes is equal to or less than a specific value, such a guidance image is determined as an inappropriate guidance image.

Fig. 11 is a view showing the results of the present treatment example, and Fig. 12 is a view showing the results of the comparative examples. In FIGS. 11 and 12, the same plurality of guidance images 71a to 71d are shown together with the category candidates. Further, in the category display area 72 for the class-specific image in FIGS. 11 and 12, the more the number of repetitions of the processing before the category-specific image is specified, the narrower the diagonal line is attached (actually , the background hue of the category display area 72 different). Actually, in the results of the present processing example, the guidance images 71a and 71d shown in FIG. 11 are specified as the class-specific images in the first processing of steps S24 to S26, and the guidance image 71c is in the step. The fourth processing of S24 to S26 is specified as a class-specific image, and the guidance image 71b is not specified as a class-specific image in any of the first to fourth processes (that is, the initial category is correct). . Further, in the results of the comparative example, the guidance image 71b shown in FIG. 12 is specified as a class-specific image in the processing of the first comparative example, and the guidance image 71c is processed in the second comparative example. The middle image is specified as a class-contrast image, and the guidance images 71a and 71d are not specified as class-dissimilar images in the processing of any of the first to fourth comparative examples.

In the processing of the comparative example, as shown in FIG. 12, the "large defect" is determined as the category candidate in the same manner as the initial category for the guidance image 71a indicating the disconnection of the pattern. In other words, the "large defect" and the "broken line" which are considered to have large differences in the feature amount space are not indicated as a "large defect". On the other hand, in the present processing example, as shown in FIG. 11, the "unclassifiable" which is different from the "large defect" of the initial category is determined as the category candidate for the guidance image 71a indicating the disconnection of the pattern. Point out the mixture of "large defect" and "broken line". Further, in the present processing example, the "unclassifiable" which is different from the "middle protrusion" of the initial category is determined as the category candidate for the guidance image 71d, and it is pointed out that the indistinguishable guidance image 71d is "middle protrusion" or " Which of the big protrusions? As apparent from the above, the results of the present treatment examples are more in line with the feelings of the comparative examples.

As described above, in the image classification device 1, a plurality of guidance images and guidance materials indicating the types of the plurality of guidance images are stored in the storage unit 614, and each of the plurality of types of the feature amount is used. The evaluation value acquisition unit 611 acquires an evaluation value indicating the validity when each of the guidance images belongs to each of the plurality of categories. Then, in the category candidate specifying unit 612, representative values of the evaluation values in the plurality of types of the feature amounts are obtained for each of the types of guidance images, and further, the plurality of classes are specified. The plurality of representative values are determined to be the category candidates of the respective dependent images. Thereby, it is possible to easily specify an appropriate category candidate based on the feature amounts of a plurality of types (all types) by simple calculation for each guidance image.

Further, the guidance image different in the initial category from the category candidate, that is, the category difference image, is displayed on the display 55 together with the category candidate by the display control unit 613. Then, the operator refers to the category-specific image and the category candidate on the display 55, and inputs the final category, whereby the highly reliable guidance material can be easily produced. As a result, in the image classification device 1, the defect image can be accurately classified by the classifier 421 that uses the guidance material for learning.

Further, for each type of dissimilar image, both the initial category and the category candidate are displayed in parallel on the display 55, whereby the operator can easily interpret the dissimilar image. Further, in the image classification device 1, the frequency of the attention interval in the histogram of one category in the plurality of histograms of the various types of the feature amount is (1 or more) the specific number or less, and is for all other categories. When the frequency of the attention interval in the histogram of each of the cases is 0, the evaluation value acquisition unit 611 does not acquire the evaluation value of the type of the feature amount. Thereby, it is not necessary to perform the process of excluding the type of the feature quantity that is not required by the dispersion analysis (ie, analysis of variance), and it is possible to eliminate the inappropriate increase of the representative value of the evaluation value unreasonably. The type of feature quantity (evaluation value).

Furthermore, when the number of guidance images is small, many guidance images exist in isolation in any interval (in an isolated state in the feature amount space), and represent a histogram of the category of number 4 in FIG. H4 is like a low frequency and flat distribution in most intervals. In this case, it is preferable that in the histogram of the category of the number n, not only the frequency of the attention interval in the attention instruction image but also the frequency of the plurality of intervals adjacent to both sides of the attention interval is obtained. The value of x _n is set, and the evaluation value P _{n is} obtained by ((x _n / (x ₁ + x ₂ + x ₃ + x ₄ )) × 100) [%]. In the evaluation value acquisition unit 611, based on the frequency of the attention area in the plurality of histograms and the plurality of sections adjacent to both sides of the attention section, the evaluation is performed for each of the plurality of categories of the guidance images. Therefore, when the number of guidance images is small, the category candidates can be appropriately determined.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made.

In the above embodiment, the superiority of the processing of FIG. 6 of the specific category candidate is performed by referring to the result of the process of removing the category dissimilar image from the instruction material and repeating the processing of steps S24 to S26 of FIG. 6 several times. Note, but the final guidance material may also be generated by removing the category dissimilar image from the instructional material and repeating the processing of steps S24 to S26 several times. As explained with reference to Figure 10, such guidance material can be understood to include only typical guidance images for each category.

In the display 55, it is not necessary to display a plurality of different types of images, and only one type of different images may be displayed. Further, it is also possible to display only the category candidates and the different types of images. As described above, in the processing of step S27 of FIG. 6, each of at least one type of dissimilar image is displayed on the display 55 together with the category candidate.

In the image classification device 1, it can also be checked by inspection. The sorting means 4 calculates the feature amount of the plurality of guidance images for guiding the material. Further, the feature amount may be omitted from the guidance material, and the feature amount may be obtained from the learning unit 62 based on the data of the guidance image.

In the above embodiment, the image classifying device 1 that adds the function of automatic defect classification to the defect inspection device, and other observation devices (also referred to as inspection devices) that observe defects on the substrate detected by the defect inspection device are described. In the device to which the function of automatic defect classification is added (which can be similarly understood as an image classifying device), the above-described guidance material creating unit 61 can also be used. In the image pickup device of such an image sorting device, in order to analyze the lack of height more In the trap, an image having a higher resolution than the image pickup device 2 of FIG. 1 is obtained.

In the image classification device 1 of FIG. 1, an inspection of a glass substrate (for example, a glass substrate for a flat display device), a printed wiring substrate, or a mask substrate for exposure of a substrate may be performed instead of the semiconductor substrate.

Further, the image classification device 1 can also be used for the purpose of classifying cell images of cells in a specific liquid such as blood or a culture solution. As such, the image classification device 1 can be used to indicate the classification of images of various objects. Further, in the image classification device 1, in addition to images captured by visible light, images captured by an electron beam or X-rays or the like can be classified.

In the above-described embodiment, the guidance material verification device that verifies the guidance material for each guidance image specific category candidate is provided in the memory unit 614, the evaluation value acquisition unit 611, and the category candidate specific unit provided in the guidance material creation unit 61. Although the 612 and the display control unit 613 are realized, the instruction material verification device can be realized as a device separate from the guidance material creation unit 61. Further, the guidance material creation unit 61 as the guidance material creation device can also be realized as a device separate from the image classification device 1.

The configurations in the above-described embodiments and modifications are appropriately combined as long as they do not contradict each other.

The invention has been described and illustrated in detail, but the foregoing description is illustrative and not restrictive. Therefore, various modifications or aspects can be realized without departing from the scope of the invention.

55‧‧‧ display

56‧‧‧ Input Department

61‧‧‧Guidance Materials Production Department

62‧‧‧Learning Department

610‧‧‧Data Computing Department

611‧‧‧ Evaluation Value Acquisition Department

612‧‧‧ Category Specific Department

613‧‧‧Display Control Department

614‧‧‧Memory Department

615‧‧‧Characteristic calculation unit

710‧‧‧Guide materials

Claims (14)

A guidance material verification device is a verification guide information, and the guidance material is used for learning a classifier that classifies images; and has: a memory portion, each of which is assigned to any one of a plurality of categories a plurality of guidance images and guidance materials indicating types of the plurality of guidance images; and an evaluation value acquisition unit that acquires, for each of a plurality of types of feature quantities, that each of the guidance images belongs to the above The evaluation value of the validity of each of the plurality of categories; the category candidate specifying unit is configured to determine the feature quantity in each of the plurality of categories in each of the plurality of categories a representative value of the evaluation value, and determining, based on the plurality of representative values in the plurality of categories, a category to which each of the guidance images belongs is specified as a category candidate; and a display control unit that The guidance image shown in the instructional material is different from the above-mentioned category candidate as a different image of the category, and each of the at least one different type of image is associated with the above category. The candidate is displayed on the display unit together. The guidance data verification device according to the first aspect of the invention, wherein the evaluation value acquisition unit generates a histogram of the feature amount of the guidance image assigned to each of the plurality of categories with respect to each type of the feature amount. And a section including the feature quantity of each of the guidance images is used as a target section, and the plurality of the plurality of guidance images are obtained based on a frequency of the attention range in the plurality of histograms of the plurality of categories. The above evaluation values of each of the categories. The guidance data verification device according to the second aspect of the invention, wherein the evaluation value acquisition unit is based on a frequency of the plurality of sections in the plurality of histograms and the plurality of sections adjacent to both sides of the attention section. The evaluation values for each of the plurality of categories of the respective guidance images are obtained. The guidance material verification device of claim 2, wherein in the plurality of histograms, the frequency of the above-mentioned attention interval in the histogram for one category is a specific number or less and for each of all other categories When the frequency of the above-mentioned attention section in the histogram is 0, the evaluation value acquisition unit does not acquire the evaluation value. The guidance data verification device according to any one of claims 1 to 4, wherein the display control unit displays each of the at least one different type of image and the category indicated by the guidance material and the candidate of the category Also displayed on the display unit described above. A guidance material production device for producing a guide material for use in learning a classifier for classifying an image; and comprising: guidance material for applying for any one of claims 1 to 5 a verification device; and an input unit that accepts input of a final category for each of the at least one different type of image. An image classification device that classifies an image; and includes: a guidance material production device of claim 6; and a classifier that uses the guidance material produced by the guidance material production device Learn and classify images. A method for guiding data verification, which is to verify the guidance material, and the guidance material is used for learning the classifier for classifying images; and has the following steps: a) preparing each of the plurality of categories to be assigned a plurality of guidance images and guidance materials indicating categories of the plurality of guidance images; b) each of a plurality of types of feature quantities is obtained for indicating that each guidance image belongs to each of the plurality of categories The evaluation value of the validity of the case; c) obtaining, in each of the plurality of categories, the representative value of the evaluation value of the feature quantity in the plurality of types in each of the guidance images; d) specifying, as the category candidate, the category to which the respective guidance images belong, based on the plurality of representative values in the plurality of categories; and e) classifying the category indicated by the guidance material and the category candidate Each of the at least one type of dissimilar image is displayed on the display unit together with the category candidate as the different dissimilar image. The method for verifying the guidance data of the eighth aspect of the patent application, wherein in the step b), a histogram of the feature quantity of the guidance image assigned to each of the plurality of categories is generated for each type of the feature quantity And the section including the feature quantity of each of the guidance images is used as the attention section, and the complex number for each of the guidance images is obtained based on the frequency of the attention range in the plurality of histograms of the plurality of categories The above evaluation values for each of the categories. The method for verifying the guidance data according to claim 9 wherein, in the step b), the frequency of the plurality of intervals in the plurality of histograms and the plurality of intervals adjacent to the two sides of the attention range are The evaluation values for each of the plurality of categories of the respective guidance images are obtained. The method for verifying the guidance data of claim 9 wherein, in the step b), in the plurality of histograms, the frequency of the above-mentioned attention interval in the histogram for one category is a specific number or less When the frequency of the above-mentioned attention interval in the histogram of each of the other categories is 0, the above evaluation value is not obtained. The method for verifying the guidance data according to any one of the items 8 to 11, wherein in the step e), each of the at least one different type of image and the category indicated by the guidance material are The above-mentioned category candidates are displayed together on the display unit. A method for producing a guidance material, which is for the production of a guide material, and the guide material is used for learning a classifier that classifies images, and has: A method of verifying a guidance material according to any one of claims 8 to 12, and a step of determining a final category for each of the at least one of the different types of images. An image classification method for classifying an image; and having the following steps: learning the classifier by using the guidance material produced by the guidance material production method of claim 13; and by the above The classifier classifies the images.