JP2014178229A - Teacher data creation method, image classification method and image classification device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide technology for effectively performing learning even when the sufficient number of teacher images cannot be prepared for all of a plurality of classification categories.SOLUTION: A teacher data creation method comprises: a teaching step of associating one classification category into which each of a plurality of teacher images should be classified among a plurality of classification categories with each of the plurality of teacher images; a primary creation step of using at least one of the plurality of classification categories as a target category and making a set of values of a plurality of kinds of feature quantities calculated for the teacher image associated with the target category into teacher data associated with the target category; and a secondary creation step of generating a set of values of new feature quantities corresponding to a point at which distance from a point occupied by the teacher data is within a predetermined distance in a feature quantity space consisting of the plurality of kinds of feature quantities based on one piece of teacher data and making the values of feature quantities as new teacher data associated with the target category.

Description

  The present invention relates to an image classification technique based on a plurality of teacher images, and more particularly to a technique for creating teacher data based on teacher images.

  For example, in the field of manufacturing technology such as semiconductors and printed circuit boards, in order to detect defects contained in products and analyze / evaluate them, the object to be evaluated is imaged through a microscope, etc. Research has been conducted to calculate the amount and perform automatic classification (see, for example, Patent Document 1). In this type of technology, each of the defect images (teacher images) collected in advance is associated with one of a plurality of classification categories set in advance corresponding to the type of defect by the operator. (Teaching) is performed. Based on the values of the plurality of types of feature values obtained for each teacher image and the classification category associated with the teacher image, an appropriate learning method such as discriminant analysis or logistic regression analysis, neural network, genetic programming or Machine learning is performed by a support vector machine or the like, and a discrimination tree or a classifier is configured. An unknown defect image is classified using the discriminant tree and classifier constructed in this way.

  In this type of technique, the classification accuracy improves statistically as the number of teacher images used for learning increases. Therefore, it is required to collect as many teacher images as possible for each type of defect. However, there may be a case where a sufficient number of teacher images cannot be prepared for all defect types due to, for example, a difference in the occurrence frequency of defects. When machine learning is performed using a small number of teacher images, an over-matching phenomenon called over-learning may occur, and defect images may not be properly classified.

  On the other hand, there are technologies described in Patent Documents 2 and 3, for example, to deal with such problems. In the technique described in Patent Document 2, when an unknown defect image that cannot be classified appears in the defect classification process, the defect image is newly added to the teacher image, or a new classification category is created and re-created as necessary. Learning is done. In the technique described in Patent Document 3, a pseudo defect image is generated by performing some processing on an image obtained by imaging a non-defective non-defective inspection object. Then, by using this as a teacher image together with the actually obtained defect image, the number of teacher images is increased and used for learning, thereby improving the effect of machine learning.

Japanese Patent No. 4155497 (for example, FIG. 5) JP 2000-057349 A (for example, paragraph 0052) JP 2011-214903 A (for example, paragraph 0037)

  As described above, in the image classification technique based on machine learning, it is desirable to prepare many teacher images for each of various defects, but this may not always be achieved. On the other hand, with the technique described in Patent Document 2, the above problem cannot be solved until a considerable number of defect images suitable as teacher images appear. In addition, since the pseudo defect image in the technique described in Patent Document 3 does not naturally represent an actual defect, an image having characteristics that cannot be actually generated is learned as a teacher image depending on the creation method. There is a risk of being served. That is, there remains a problem that the classification accuracy based on the learning result is lowered by using such an inappropriate image as a teacher image.

  The present invention has been made in view of the above problems, and provides a technique capable of effectively learning even when a sufficient number of teacher images cannot be prepared for all of a plurality of classification categories. For the purpose.

  In order to achieve the above object, the teaching data creation method according to the present invention relates to each of a plurality of teacher images, a teaching step for associating one classification category in which the teacher image should be classified among a plurality of classification categories; , Using at least one of the plurality of classification categories as a target category, a set of a plurality of feature value values obtained for the teacher image associated with the target category is used as teacher data associated with the target category. Based on the primary creation step and the one teacher data, a new feature amount corresponding to a point that is within a predetermined distance from a point occupied by the teacher data in the feature amount space composed of the plurality of types of feature amounts. A secondary generation step of generating a set of values and associating the feature value with the target category to form new teacher data

  In the invention configured as described above, teacher data is obtained from an actual teacher image, and a set of feature amounts corresponding to a point within a predetermined distance from a point in the feature amount space corresponding to the teacher data is obtained. The new teacher data is associated with the same classification category as the original teacher data.

  When a pseudo defect image is generated by performing some processing on the image as in the above prior art and the teacher image is supplemented, the true defect image and the pseudo defect image may be largely separated in the feature amount space. In particular, when the dimension of the feature amount space is increased, even a slight difference in appearance may be a large difference in the feature amount space. Including the pseudo defect image generated at a position far from the original defect image in the feature amount space in the teacher image in this way makes the boundary between the classification categories in the feature amount space unclear and causes a reduction in classification accuracy. .

  On the other hand, in the present invention, teacher data is supplemented by generating new points around a point corresponding to one existing teacher data in the feature amount space, instead of generating a teacher image in a pseudo manner. To do. In this way, a necessary and sufficient number that does not obscure the boundaries between classification categories in the feature amount space by newly generating teacher data in which the distance in the feature amount space from the original teacher data is defined Can be supplemented with teacher data. By using these teacher data for learning, even when a sufficient number of teacher images cannot be prepared for each classification category, effective learning can be performed.

  In this invention, when there are a plurality of teacher images associated with the target category, for example, the secondary creation step may be executed for each of the teacher data corresponding to them. By doing so, it is possible to avoid biased supplementation of teacher data based only on specific teacher data, and it is possible to perform appropriate learning using the supplemented teacher data.

  Further, for example, a secondary creation process based on at least one of new teacher data created by the secondary creation process may be further executed. There may be a case where the number of teacher data is insufficient only by supplementing teacher data based on teacher data corresponding to an actual teacher image. In such a case, it is desirable to replenish more teacher data, but it was created from the teacher data corresponding to the teacher image rather than creating a lot of teacher data only around the teacher data corresponding to the teacher image. It is preferable to create new teacher data around the teacher data because more effective learning is possible.

  Further, for example, the new teacher data generated in the secondary creation step may correspond to the vertices of the hypercube in the feature amount space whose center is the point corresponding to the base teacher data. Various methods can be considered as a method for identifying a point within a predetermined distance in the feature amount space from the base teacher data. Among these, in the method of using the point corresponding to the teacher data as a vertex of the hypercube, a new value is obtained only by adding a predetermined value to the value in at least one of various feature quantities representing the teacher data. Since the teacher data can be generated, the teacher data can be supplemented by a simple process.

  Further, for example, among the category categories associated with at least one teacher image, the category having the smallest number of associated teacher images may be set as one of the target categories. Even if the absolute number of teacher images is sufficiently large, problems such as over-learning due to the shortage of the number of teacher images still remain if the number of teacher images associated with the classification category is small in some classification categories. In the field of defect classification, such a deviation in the number of teacher images tends to occur due to a difference in the frequency of occurrence of defects. By replenishing teacher data with a classification category having a small number of teacher images as a target category, it is possible to effectively avoid inconveniences such as over-learning due to an imbalance in the number of teacher data. is there.

  Further, in order to achieve the above object, the image classification method according to the present invention corresponds to each of the teacher data created by any one of the teacher data creation methods described above and the teacher image associated with a classification category other than the target category. A classifier is configured by performing learning based on a plurality of teacher data composed of a plurality of feature value sets, and the classification target image is classified into one of the classification categories using the configured classifier. Process.

  In the invention configured as described above, learning can be performed effectively by using the teacher data created by the teacher data creation method configured as described above. Therefore, the classifier configured by learning is used. The classification target images can be classified with high accuracy.

  In order to achieve the above object, an image classification apparatus according to the present invention classifies an image acquisition unit that acquires a plurality of teacher images, and the teacher image among a plurality of classification categories for each of the teacher images. Receiving means for receiving teaching information for associating one classification category to be obtained; feature quantity calculating means for calculating a plurality of types of feature quantities for each of the teacher images; and teacher data creating means for creating teacher data based on the teacher images A classifier configured to perform learning based on the teaching information and the teacher data, and classify a classification target image using the classifier, wherein the teacher data creating unit includes the plurality of classifiers. Associating the plurality of types of feature value values obtained for each of the teacher images with the classification category to which the teacher image is associated On the other hand, based on at least one of the teacher data, a new distance corresponding to a point that the teacher data occupies in the feature amount space composed of the plurality of types of feature amounts is within a predetermined distance. A set of feature value values is generated, and the set of feature value values is associated with the target category as one piece of the teacher data.

  In the invention configured as described above, supplementation of teacher data based on the above-described principle is possible, and an image based on a learning result using teacher data corresponding to an existing teacher image and teacher data supplemented in this way. By performing the classification, it is possible to classify the classification target images with high accuracy even when a sufficient number of teacher images cannot be prepared for a plurality of classification categories. Also in this invention, it is desirable that supplementation of teacher data is performed for classification categories with a small number of teacher images.

  According to the present invention, even when a sufficient number of teacher images cannot be prepared for each classification category, it is possible to effectively supplement the teacher data and effectively perform learning for image classification. Become.

1 is a diagram showing a schematic configuration of an inspection system to which the present invention can be preferably applied. It is a flowchart which shows operation | movement of a defect detection apparatus. It is a flowchart which shows operation | movement of a defect classification device. It is a flowchart which shows the prior learning process performed by the defect classification device. It is a figure which shows an example of distribution of the defect image in feature-value space. It is a figure for demonstrating the principle of the teacher data supplement method in this embodiment. It is a figure which illustrates the result of teacher data supplement.

  FIG. 1 is a diagram showing a schematic configuration of an inspection system to which the present invention can be preferably applied. This inspection system 1 is an inspection system that performs inspection of defects such as pinholes and foreign matters appearing on the appearance of a semiconductor substrate S to be inspected, and automatically classifies detected defects. The inspection system 1 includes a defect detection device 2 that detects a defect from an inspection target region on the substrate S, a defect classification device 3 that analyzes the defect detected by the defect detection device 2 in more detail and identifies the type of defect, and It has. The defect detection device 2 is incorporated in the production line of the substrate S, and the inspection system 1 is a so-called in-line system.

  The defect detection apparatus 2 includes an image acquisition unit 20 that acquires an image of an inspection target area on the substrate S, and a detection unit 21 that analyzes the image data from the image acquisition unit 20 and detects a defect. The image acquisition unit 20 has an imaging element such as a CCD sensor or a CMOS sensor, for example, and acquires an image data by acquiring an image of an inspection target area on the substrate S, a stage 202 that holds the substrate S, and A stage driving unit 203 that moves the stage 202 relative to the imaging unit 201 is provided. The stage driving unit 203 includes a ball screw, a guide rail, and a motor. The imaging unit 201 and the stage driving unit 203 are controlled by a mechanism control unit 211 provided in the detection unit 21, and operate according to a control command from the mechanism control unit 211, so that an inspection target region on the substrate S is changed. Take an image.

  The defect detection unit 212 performs appropriate data processing on the image data obtained by imaging by the imaging unit 201 to detect defects. The defect detection unit 212 performs appropriate data processing such as noise removal processing or smoothing processing on the image data given from the imaging unit 201, and performs defect detection based on the image data thus processed. More specifically, when an image captured on the actual substrate S is compared with an image of a defect-free substrate stored in advance in the storage unit 213, and the difference exceeds a predetermined reference It is determined that the image includes a defect. The defect detection unit 212 transmits the detected defect image data to the defect classification device 3 via the interface (I / F) 214.

  The defect classification device 3 determines the type of the detected defect based on the defect data given from the defect detection device 2. That is, the defect classification device 3 has a function of automatically classifying defects (ADC: Automatic Defect Classification) into classification categories to which the defect belongs based on the defect image data given from the defect detection device 2. The defect classification device 3 is an embodiment of the image classification device according to the present invention, and is a device capable of executing the teacher data creation method and the image classification method according to the present invention.

  The defect classification device 3 implements each functional block shown in FIG. 1 by dedicated hardware and software by executing a control program read in advance. The defect classification device 3 includes an interface (I / F) 31 for receiving defect image data from the defect detection device 2, and a defect classification unit (ADC) 32 that determines the type of the defect by automatic classification based on the defect image data. It has. The interface 31 can communicate with other external devices via, for example, a LAN (Local Area Network).

  The image data given from the defect detection device 2 and various processing data generated in the process in the defect classification unit 32 are stored in the storage unit 33 as needed. The defect classifying unit 32 classifies the defect image into a plurality of classifications set in advance corresponding to the defect type based on the values of the plurality of types of feature amounts calculated by the feature amount calculating unit 34 based on the given defect image data. Sort into one of the categories.

  Furthermore, the defect classification device 3 includes an input receiving unit 35 such as a keyboard and a mouse that receives an operation input from a user, and a display unit 36 that displays visual information for the user such as an operation procedure and a processing result.

  The defect classification unit 32 executes a process of classifying the detected defect using a learning algorithm such as SVM (Support Vector Machine), neural network, decision tree, discriminant analysis, and the like in software. Specifically, appropriate learning is performed in advance based on defect image data given from the defect inspection apparatus 2 and stored and stored in the storage unit 33, and teaching information from a user regarding a classification category in which the defect image should be classified. Perform machine learning using an algorithm. When new image data corresponding to an unclassified defect image is given, automatic classification based on the learning result is performed to classify the new defect image into any one of the classification categories.

  FIG. 2 is a flowchart showing the operation of the defect detection apparatus. The defect detection apparatus 2 images the inspection target substrate S, and executes the following defect detection processing as an operation for detecting defects from the imaging result. When the inspection target substrate S is first loaded on the stage 202 of the image acquisition unit 20 (step S101), the mechanism control unit 211 controls the imaging unit 201 and the stage driving unit 203 to perform a predetermined inspection target on the substrate S. The position is imaged and an image is acquired (step S202). For the image data obtained in this way, the defect detection unit 212 performs processes such as brightness adjustment and noise removal as necessary. Further, the image data corresponding to the inspection target position is read out from the image data of the defect-free substrate stored in the storage unit 213 in advance, and the difference in pixel units from the image data obtained by imaging is calculated. (Step S103).

  If the captured image does not include a defect, the difference from the previously stored defect-free image is small, but if a defect is included, a large difference appears. By utilizing this fact, it is possible to determine the presence or absence of a defect by obtaining the difference between the two. Specifically, appropriate smoothing processing and binarization processing are performed on the obtained difference image (step S104), and if there is a region where a significant difference is recognized, that portion is extracted as a defective portion ( Step S105). For an image in which a defect is detected (step S106), the image is transmitted as a defect image to the defect classification device 3 via the interface 214 (step S107). By performing the above processing on the entire substrate S while sequentially changing the inspection target position (step S108), the inspection of the substrate S is completed.

  The defect detection device 2 inspects each position of the inspection target substrate S to determine whether there is a defect. On the other hand, the defect classification device 3 determines the type of detected defect, analyzes the tendency of occurrence of the defect, and feeds back the result to the production line, or adjusts the processing content of the defect detection processing by the defect detection device 2 Used to do By analyzing the detected defect by the defect classification device 3, the defect detection device 2 can execute a process specialized for defect detection, so that a defect can be detected at high speed.

  FIG. 3 is a flowchart showing the operation of the defect classification apparatus. As described above, the defect classification device 3 classifies the defect image output from the defect detection device 2 into a classification category corresponding to the defect type. More specifically, when the defect classification unit 32 performs pre-learning (described later) based on the defect image data collected in advance (step S201), an unclassified defect image is newly given from the defect detection device 2. (Step S202), the feature quantity calculation unit 34 calculates a plurality of feature quantity values that characterize the defect image (Step S203). Then, based on the calculated feature value, the defect classification unit 32 classifies the defect image into one of a plurality of classification categories using a classifier configured by prior learning (step S204).

  A plurality of types of classification categories are set in advance corresponding to a plurality of types of defects that may occur in the substrate S, and for example, foreign materials, scratches, pattern disconnections, short circuits, and the like are representative, but are not limited thereto. It is not something.

  FIG. 4 is a flowchart showing a pre-learning process executed by the defect classification apparatus. The pre-learning process is performed based on a teacher image obtained by imaging various defects and teaching information indicating which classification category the teacher image should be classified, which is taught by the user. Specifically, teacher images are collected first. Here, the defect images output from the defect detection apparatus 2 are collected as teacher images, which are sequentially stored in the storage unit 33 (step S301). For example, an image library prepared in advance or an external storage is used. It is also possible to obtain the teacher image data collected in the above.

  For each of the many teacher images collected in this way, the feature amount calculation unit 34 calculates a feature amount (step S302). The types of feature quantities calculated here are the same as those used when classifying unknown defect images. The calculated feature value is stored and saved in the storage unit 33 as part of the teacher data corresponding to each teacher image.

  Next, a classification category in which the teacher image is to be classified is associated with each teacher image. More specifically, the collected teacher images are sequentially displayed on the display unit 36, and a teaching input is received from the user via the input receiving unit 35 regarding which defect type the teacher image belongs to. Then, the teacher image is associated with the classification category corresponding to the taught defect type (step S303), and the feature value of the teacher image and the user who designates the classification category associated with the teacher image are specified. The teaching information is set as a set and stored in the storage unit 33 as teacher data corresponding to the teacher image.

  Subsequently, based on the teacher data prepared in this way, that is, based on the set of feature value values representing the teacher image and the teaching information, the defect classification unit 32 performs learning using an appropriate learning algorithm to configure a classifier (step S304). ). Then, in order to verify the learning result, the teacher data itself is classified using the configured classifier (step S305), and the classification result is calculated (step S306). The classification results can be expressed using an appropriate performance evaluation scale, and can be expressed by, for example, purity and accuracy for each classification category. The classification result calculated in this way is displayed on the display unit 36, so that the user can evaluate the appropriateness of learning.

  Statistically, the greater the number of prepared teacher images, the more effective learning can be performed, but the frequency of defects varies depending on the type of defect, and a sufficient number of defect images are collected for each defect type. It is not always possible. In particular, immediately after a new manufacturing process is started, even if many defect images are collected for a certain defect type, only a few defect images may be obtained for other defect types.

  FIG. 5 is a diagram illustrating an example of a defect image distribution in the feature amount space. Various types of feature values are known for use in defect image classification, and many types of feature values are used in actual defect classification. For this reason, the feature amount space in which each of a plurality of types of feature amounts is used as one coordinate axis is also a multidimensional space. Here, in order to facilitate understanding of the principle, two types of feature amounts X1 and X2 are used. Consider the following two-dimensional feature space. When the collected defect image (teacher image) is plotted in the feature amount space according to the feature value, for example, as shown in FIG. 5, defect images having similar features gather to some extent to form several clusters. To do. Each point in FIG. 5 represents a point having these values as coordinate values in the feature amount space when the defect image is represented by a feature amount, and each point corresponds to one defect image. Since defect images classified into the same defect type naturally have similar features on the images, it is considered that they are close to each other in the feature amount space. That is, defect images belonging to the same defect type form one cluster in the feature amount space.

  If the spread of clusters in the feature amount space for each defect type is known, when an unknown defect image is given, the defect type of the defect image is determined from the position of the feature amount space occupied by the point corresponding to the defect image. Can be determined. In order to make this possible, learning based on a teacher image is performed.

  Here, for example, relatively many teacher images are associated with the defect types A and B illustrated in FIG. 5, and a cluster is formed by many points corresponding to these. Therefore, it is considered that many types of the defect types are available, and the spread range of the cluster can be estimated with relatively high accuracy. Therefore, it is possible to determine that an unknown defect image that occupies a position within the spread range in the feature amount space belongs to the defect type with high accuracy. Further, it is easy to specify the boundary between the defect types A and B, and the probability of misclassification between them can be reduced.

  On the other hand, when the number of associated teacher images is small, such as the defect type C shown in FIG. 5, it is difficult to specify a range in which the defect type is distributed in the feature amount space. That is, for example, whether it is a relatively small spread as indicated by a one-dot chain line or a larger spread as indicated by a two-dot chain line cannot be accurately determined only by an existing teacher image. . Therefore, it becomes difficult to specify the boundary between the defect types A and C and the boundary between the defect types B and C in the feature amount space, and as a result, misclassification increases. Such a phenomenon is due to an overfitting phenomenon of learning called overlearning, and becomes prominent when there is a large imbalance in the number of teacher images between classification categories. In order to cope with such a case, the pre-learning process of this embodiment is configured such that teacher data can be supplemented and re-learning can be performed as necessary.

  Applying the above-described prior art technical idea to solve this problem, it is possible to artificially create an image simulating a defect of a defect type with a small number of teacher images, and to supplement the image as a teacher image and perform learning It becomes. However, even if the defect generation mechanism is imitated, it is not an actual defect, so even a slight difference in the image is a large difference in the multidimensional feature space, and the defect belongs to each defect type. The spread of the clusters created by the image may be obscured.

  Therefore, in this embodiment, teacher data is replenished by deriving a point corresponding to new teacher data created by replenishment in the vicinity of a point occupied by already existing teacher data in the feature amount space. That is, an image imitating a defect image is not created as a real image. For this reason, the distance from the existing teacher data to the new teacher data in the feature amount space can be limited.

  FIG. 6 is a view for explaining the principle of the teacher data supplementing method in this embodiment. As shown in FIG. 6A, a point P (v1, v2) in the feature amount space corresponding to existing teacher data is considered. The coordinate values v1 and v2 of the point P correspond to the values of the feature amounts X1 and X2 obtained for the teacher image corresponding to this point, respectively. Then, a new point Q is set within a predetermined distance in the feature amount space from the point P, and the feature value corresponding to the coordinate value of the point Q and the classification category of the point P that is the basis of the creation Are used as new teacher data.

  In principle, for example, one point on a hypersphere (a circle indicated by a broken line in the two-dimensional feature space) having a predetermined radius R with the point P as the center can be set as the point Q. More simply, the point Q can be set as one vertex of a hypercube (a square indicated by a dotted line in the two-dimensional feature amount space) in the feature amount space having the point P as a body center. Since the hypersphere in the multi-dimensional feature quantity space is represented by a large number of feature quantity functions, the calculation for obtaining the coordinates of the points on the hypersphere is complicated. On the other hand, the coordinates of the vertices of the hypercube in the feature amount space can be obtained only by adding a constant value to the coordinate values of each coordinate axis if the sides of the hypercube are parallel to the direction of each coordinate axis. .

  In the example of FIG. 6A showing the two-dimensional feature amount space, a value obtained by adding or subtracting a predetermined value Δv1 to the X1 coordinate value v1 of the base point P can be used as the X1 coordinate value of the point Q. . A value obtained by adding or subtracting the predetermined value Δv2 to the X2 coordinate value v2 of the point P can be used as the X2 coordinate value of the point Q. In FIG. 6A, the point Q (v1 + Δv1, v2 + Δv2) is created at the upper right vertex with respect to the point P, but a new point may be created at another vertex. Two or more vertices may be used as new points.

In addition, since each feature-value is digitized based on each separate calculation standard, naturally the scale of a numerical value may differ for every feature-value. In view of this point, in order to make the hypersphere or hypercube in the feature quantity space significant, it is preferable that the value of each feature quantity in this case is normalized. The feature amount normalization calculation can be performed, for example, by dividing the feature amount value in the teacher data corresponding to the point P by the maximum value of the feature amount among all the teacher data. For example, for the X1 coordinate value v1 of the point P, when the value of the feature quantity X1 in the teacher data corresponding to the point P is v0 and the maximum value among all the teacher data of the feature quantity X1 is vmax,
v1 = v0 / vmax (Formula 1)
It can ask for.

Further, for example, it can be performed by dividing the feature value in the teacher data corresponding to the point P by the difference between the maximum value and the minimum value of the feature value among all the teacher data. For example, for the X1 coordinate value v1 of the point P, the value of the feature quantity X1 in the teacher data corresponding to the point P is v0, the maximum value among all the teacher data of the feature quantity X1 is vmax, and the minimum value is vmin. When
v1 = v0 / (vmax−vmin) (Expression 2)
It can ask for. Since the purpose is not to obtain the numerical value of the feature value as a result, any calculation method of (Expression 1) or (Expression 2) may be used as long as the calculation rule is unified among the feature values.

  In the following description, the point Q that is derived from the point P as described above may be expressed as a “child” of the point P. In addition, the base point P may be expressed as the “parent” of the point Q. The teacher data corresponding to the point P may be expressed as “parent data”, and the teacher data corresponding to the point Q may be expressed as “child data”.

  Returning to FIG. 4, the description of the pre-learning process will be continued. When the user determines that the classification result obtained in step S306 and displayed on the display unit 36 is insufficient, it is possible to instruct supplementation of teacher data via the input receiving unit 35. When it is necessary to supplement teacher data (YES in step S307), a defect type for which the number of teacher data is insufficient is designated by user input (step S308). Further, the defect type to be supplemented with data may be automatically selected according to the number of teacher data for each defect type. For example, since the effect of supplementing teacher data is particularly noticeable for defect types with a small number of original teacher data, supplementation can be performed in order from a defect type with a small number of teacher data. In this case, for example, the user may be allowed to specify how many defect types are replenished.

  Then, teacher data is supplemented for the designated defect type. As described above, the supplementation of data is performed by deriving new teacher data near the existing teacher data in the feature amount space. At this time, in order to prevent new teacher data from being generated only around specific teacher data, new child data is generated by setting only the teacher data having no “child” as “parent” (step S309). ). By deriving child data for all teacher data having no children, the number of teacher data increases. When N child data are generated from one parent data, the number of teacher data is N times the original data.

  There are many vertices of the hypercube in the multi-dimensional feature amount space, and if a child is generated for all of them, a huge number of teacher data is generated, and the number of teacher data for each defect type is unbalanced. Preferably, the number of children to be generated from one parent is determined in advance, and a number of vertices corresponding to the number of generations are selected from a large number of vertices, and children are generated at those vertices. The selection rule is arbitrary, and can be determined at random using, for example, a random number. However, among the vertices, those that already have a point corresponding to the teacher data at or near that position are preferably excluded from selection. In this way, uneven distribution of teacher data can be prevented.

  Whether or not there is other teacher data in the vicinity of the point where the teacher data is to be created can be determined, for example, from the Euclidean distance in the feature amount space. However, it is not practical to perform calculations on all existing teacher data because the calculation amount is enormous. Instead of this, for example, it is possible to use a method of searching for other points included in a hypercube having an appropriate size including the point inside or having the point as a body center. In this case, it is possible to search for neighboring points by mutual comparison of numerical values for each coordinate axis, that is, for each feature amount, so that it is possible to greatly reduce the necessary calculation amount.

  The teacher data newly generated as described above is treated as being belonging to the same classification category (defect type) as the original teacher data without being distinguished from the original teacher data. Subsequently, performance evaluation is performed using the newly generated teacher data and the original teacher data (step S310). This performance evaluation is the same as the processing in steps S304 to S306 described above. At this time, if there is no change in the classification result (NO in step S311), it is considered that the supplementation of the teacher data is not sufficient, and the process returns to step S309 and the data supplement is performed again.

  At this time, the teacher data that became the parent in the previous data supplementation is not used as the parent again, and the data newly supplemented by using the child newly generated in the previous data supplementation as the new parent. That is, as shown in FIG. 6B, when the child Q1 is generated with the point P as the parent in the previous data supplementation, the child Q2 is generated with the point Q1 as the parent in the next data supplementation. Further, in this case, a child Q2 is generated at a vertex different from the point P among the vertices of the hypercube having the point Q1 as a body center.

  Thus, when supplementing teacher data, a new child is generated from the existing teacher data that is not a parent in the past data supplement, that is, teacher data having no children. Also, the teacher data is supplemented to the point where the teacher data does not exist at or near the vertex of the hypercube having the parent as the body. By doing so, it is possible to supplement teacher data with no bias around the teacher data corresponding to the collected teacher image, and to expand the space occupied by the defect type in the feature amount space.

  When there is a change in the classification result in the performance evaluation after data supplementation (YES in step S311 in FIG. 4), the result is presented to the user to determine whether further supplementation of teacher data is necessary (step S307). ). Then, by repeating the above processing until it is determined that supplementation is not necessary, supplementation of teacher data is completed.

  FIG. 7 is a diagram illustrating the result of teacher data supplementation. When teacher data is supplemented for defect type C, teacher data (indicated by white circles) supplemented around the teacher data (indicated by black circles) corresponding to the collected teacher image in the feature space. ) Are distributed evenly. Thereby, the extension of the cluster belonging to the defect type C becomes clearer than the state before replenishment shown in FIG. As a result, learning based on these teacher data is effectively performed, and a classifier with few misclassifications can be configured.

  As described above, in this embodiment, learning is performed using the teacher data supplemented together with the teacher data based on the teacher image, so that the adverse effects of over-learning can be suppressed and the probability of misclassification can be reduced. The teacher data is supplemented by a method of creating new teacher data in the vicinity of the existing teacher data in the feature amount space. In the method of supplementing the teacher image in a pseudo manner, there is a concern that the variation in the feature amount space is increased, but in this embodiment, teacher data whose position in the feature amount space is managed is supplemented. Therefore, it is possible to effectively improve the learning accuracy.

  As described above, in this embodiment, the defect classification device 3 functions as the “image classification device” of the present invention, and the defect detection device 2 and the interface 31 of the defect classification device 3 that receives defect image data from the defect detection device 2 are the present. It functions as the “image acquisition means” of the invention. Further, the input receiving unit 35 functions as the “receiving unit” of the present invention. The feature quantity calculation unit 34 functions as the “feature quantity calculation unit” of the present invention, while the defect classification unit 32 functions as the “classification unit” of the present invention. The defect classification unit 32 and the feature amount calculation unit 34 function as a “teacher data creation unit” of the present invention.

  In the flowchart of FIG. 4, step S303 corresponds to the “teaching process” of the present invention, and steps S302 and S303 correspond to the “primary creation process” of the present invention. On the other hand, step S309 corresponds to the “secondary creation process” of the present invention. In the above embodiment, the defect types A, B, and C all correspond to the “classification category” of the present invention, and among these, the defect type C that has the smallest number of teacher images and is the object of data supplementation is “ It corresponds to “Target category”.

  The present invention is not limited to the above-described embodiment, and various modifications other than those described above can be made without departing from the spirit of the present invention. For example, in the above embodiment, learning is performed based on the created teacher data after the teacher data is created. This is to verify the validity of the teacher data, but the learning process is not an essential requirement in the teacher data creation method of the present invention. For example, after the collection of teacher images and the teaching work by the user is completed, it is determined whether or not data supplement is necessary from the number of teacher images for each defect type, and data supplement is performed for the required defect types. Also good. In this case, teacher data is created without performing learning work. Then, learning is performed after the supplementation of teacher data is completed, so that unknown defect images can be classified.

  In the above embodiment, the user determines whether or not it is necessary to supplement teacher data. For example, teacher data can be created without intervention by the user by performing the following, for example. In other words, the required number of teacher data is determined in advance, and if there is a classification category in which the number of teacher images does not reach the required number as a result of the teaching work, this is used as a target category until the number of teacher data exceeds the required number. Replenishment may be performed.

  In the above embodiment, the vertex of the hypercube centered on the point on the feature space corresponding to the parent teacher data is the point corresponding to the new teacher data. It is not limited to. For example, as described above, a hypersphere in the feature amount space may be assumed, and a new point may be derived inside such a hypercube or hypersphere. Thus, the main technical idea of the present invention is to derive the child at a position where the distance from the parent is limited in the feature amount space.

  Although not supported in the above embodiment, the user may change the content of the teaching work in the pre-learning process. In other words, the defect type information once given to each teacher image may be changed afterwards by the user as a result of verifying the classification result. In such a case, the teacher data derived based on the teacher image can be continuously used as effective teacher data by changing the associated defect type.

  Moreover, although the said embodiment is the inspection system 1 provided with the defect detection apparatus 2 and the defect classification apparatus 3, this invention is applied also to the apparatus which integrated the defect detection function and its classification function. It is possible. Moreover, the apparatus which is not provided with an imaging function and specialized in only defect classification may be used. Also, for example, the teacher data creation method according to the present invention is realized as software on an arithmetic processing device such as a personal computer, and the arithmetic processing device creates teacher data using a defect image acquired by the inspection device as a teacher image, The teacher data may be given to the inspection apparatus and defect classification may be performed by the inspection apparatus.

  Moreover, although the said embodiment is an image classification apparatus which test | inspects and classifies the defect of a semiconductor substrate, the image classification apparatus which is an application object of this invention is not only the apparatus which test | inspects a semiconductor substrate but other objects, for example, a print It may be a device for inspecting a substrate, a glass substrate or the like, or a surface inspection device for inspecting the surface state of various materials.

  In the above embodiment, the present invention is applied to an inspection system that detects and classifies defects from a semiconductor substrate or the like, and classification categories are defined corresponding to defect types. However, the present invention is not limited to such a defect image, and the present invention can be applied to all techniques for handling various images evaluated based on feature amounts.

  The present invention is suitable for a technique for handling an image represented by a plurality of feature amounts, and can be suitably applied particularly to a technical field involving machine learning based on teacher data.

DESCRIPTION OF SYMBOLS 1 Inspection system 2 Defect detection apparatus (image acquisition means)
3. Defect classification device (image classification device)
31 Interface (Image acquisition means)
32 Defect classification unit (classification means, teacher data creation means)
34 feature amount calculation unit (feature amount calculation means, teacher data creation means)
35 Input reception part (reception means)
S302 Primary creation step S303 Teaching step, Primary creation step S309 Secondary creation step

Claims (8)

A teaching step of associating each of the plurality of teacher images with one classification category in which the teacher image is to be classified among the plurality of classification categories;
First, using at least one of the plurality of classification categories as a target category, a set of feature value values obtained for the teacher image associated with the target category is used as teacher data associated with the target category. Creation process,
Based on the one teacher data, a new set of feature value values corresponding to a point within a predetermined distance from a point occupied by the teacher data in the feature amount space composed of the plurality of types of feature amounts is generated. And a secondary creation step of associating the feature value and the target category as new teacher data.   The teacher data creation method according to claim 1, wherein the secondary creation step is executed for each of the teacher data corresponding to the plurality of teacher images associated with the target category.   The teacher data creation method according to claim 1 or 2, further comprising executing the secondary creation step based on at least one of the new teacher data created by the secondary creation step.   The new teacher data generated in the secondary creation step corresponds to a vertex of a hypercube in the feature amount space having a point corresponding to the teacher data as a body as a body. Teacher data creation method described in 1.   The teacher data creation according to any one of claims 1 to 4, wherein among the classification categories associated with at least one of the teacher images, the one with the smallest number of the associated teacher images is one of the target categories. Method. A plurality of feature value values corresponding to each of the teacher data created by the teacher data creation method according to claim 1 and the teacher image associated with the classification category other than the target category. A step of configuring a classifier by performing learning based on a plurality of teacher data composed of
A method for classifying images to be classified into any of the classification categories using the classifier. Image acquisition means for acquiring a plurality of teacher images;
Receiving means for receiving teaching information that associates one of the plurality of classification categories with which one of the teacher images is to be classified;
Feature quantity calculating means for calculating a plurality of types of feature quantities for each of the teacher images;
Teacher data creating means for creating teacher data based on the teacher image;
A classifier is configured by performing learning based on the teaching information and the teacher data, and includes a classifying unit that classifies a classification target image using the classifier,
The teacher data creating means associates the plurality of types of feature value values obtained for each of the plurality of teacher images with each of the teacher data in association with the classification category to which the teacher image is associated. On the other hand, based on at least one of the teacher data, a new value of the feature amount corresponding to a point that is less than a predetermined distance from a point occupied by the teacher data in the feature amount space including the plurality of types of feature amounts. An image classification apparatus characterized by generating a set of feature values and associating the set of feature value values with the target category as one piece of the teacher data.   The teacher data creation means includes the teacher data associated with at least one of the classification categories including the one with the smallest number of the associated teacher images among the classification categories associated with at least one of the teacher images. The image classification apparatus according to claim 7, wherein a new set of feature amount values is generated based on the image.

Images