Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/28—Databases characterised by their database models, e.g. relational or object models
  • G06F16/284—Relational databases
  • G06F16/285—Clustering or classification
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
  • G06N20/20—Ensemble learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N5/00—Computing arrangements using knowledge-based models
  • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N7/00—Computing arrangements based on specific mathematical models
  • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

• the presently disclosed subject mater relates, in general, to the field of examination of a specimen, and more specifically, to methods and systems capable of automated detection of defects to be classified to a monitor) class of defects.
• the tenn“specimen” used in this specification should be expansively construed to cover any kind of wafer, mask, and other structures, combinations and/or parts thereof used for manufacturing semiconductor integrated circuits, magnetic heads, flat panel displays, and other semiconductor-fabricated articles.
• Hie term "defect" used in this specification should be expansively construed to cover any kind of abnormality or undesirable feature formed on or within a specimen.
• the complex manufacturing process of specimens is not error-free and such errors may cause faults in manufactured devices.
• faults may include defects that can harm operation of the device, and nuisances, which may be defects, but do not cause any harm or malfunction of the manufactured device.
• defects may be caused during the manufacturing process, due to faults in the raw' material, mechanical, electrical or optical errors, human errors or others.
• defects may be caused by spatio-temporal factors, such as temperature changes of the wafer occurring after one or more manufacturing stages during the examination process, which may cause some deformations of the wafer.
• Hie examination process can also introduce further alleged errors, for example due to optical, mechanical or electrical problems in the examination equipment or process, which thus provide imperfect captures. Such errors may produce false positive findings, which may seem to contain a defect, but no actual defect exists at the area.
• a defect In many applications, the type, or class, of a defect is of importance. For example, a defect may be classified into one of a number of classes, such as a particle, a scratch, process, or the like.
• a class may be considered a majority class if at least a predetermined number of defects of the class occur when examining a specimen, at least a predetermined part of the defects detected when examining a specimen belong to the class, the number of encountered defects belonging to the class is sufficient for training a classifier to recognize a defect as belonging to the class, or other criteria.
• Idle term "minority class" used in tins specification should be expansively construed to cover any kind or class of defects which do not occur frequently, and thus insufficient statistics are available for generating a reliable classification model by training a classifier to classify defects to this class.
• a class may he considered a minority class if less than a predetermined number of defects of the class occur when examining a specimen, less than a predetemiined part of the defects detected when examining a specimen belong to the class, the number of encountered defects belonging to the class is insufficient for training a classifier to recognize a defect as belonging to the class, or the like.
• the term “examination” used in this specification should be expansively construed to cover any kind of detection and/or classification of defects in an object.
• Examination is provided by rising non-destructive examination tools during or after manufacture of the object to be examined.
• the examination process can include scanning (m a single or in multiple scans), sampling, reviewing, measuring, classifying and/or other operations provided with regard to the object or parts thereof, using one or more examination tools.
• examination can be provided prior to manufacture of the object to be examined and can include, for example, generating an examination recipe(s).
• the term“examination” or its derivatives used in this specification is not limited with respect to the size of the inspected area(s), to the speed or resolution of the scanning, or to the type of examination tools.
• a variety of non-destructive examination tools includes, by way of non-limiting example, optical tools, scanning electron microscopes, atomic force microscopes, etc.
• the examination process can include a plurality of examination steps. During the manufacturing process, the examination steps can be performed a multiplicity of times, for example after the manufacturing or processing of certain layers, or the like. Additionally or alternatively, each examination step can be repeated multiple times, for example for different specimen locations or for the same specimen locations with different examination setings.
• run-time examination can employ a two-step procedure, e.g. inspection of a specimen followed by review of sampled defects.
• the surface of a specimen or a part thereof e.g. areas of interest, hot spots, etc.
• the captured inspection image is analyzed in order to detect defects and obtain locations and other inspection attributes thereof.
• the images of at least some of the defects detected during the inspection phase are, typically, captured at relatively low speed and/or high-resolution, thereby enabling classification and, optionally, other analyses of the at least some of tire defects.
• both phases can be implemented by the same inspection tool, and, in some other cases, these two phases are implemented by different inspection tools.
• a system to classify defects in a specimen into a plurality of classes comprise a processor and memor ' circuitry' (PMC) configured to: receive a plurality of defects classified by a first classifier into a majority class, wherein one or more defects from the plurality of defects belong to one or more minority classes and at least part of the defects from the plurality of defects belong to the majority class, and wherein each defect from the plurality of defects is characterized by plurality of attributes each having a respective value; select a subset of the plurality of attributes and define a differentiator for each attri bute of the subset, such that a second classifi er configured to classify defects in accordance with the subset and the differentiator for each respective attribute, classifies correctly to the one or more minority classes and the majority class at least a predefined share of defects from the plurality of defects; generate a temporal ⁇ ' training set, the temporary ' training set comprising:
• the PMC can be further configured to generate additional defects classifiable by the second classifier to the one or more minority classes and including the additional defects into the temporary training set.
• At least one of die additional defects can be a synthetic defect generated as a plurality of first v alues, each first value associated with an atribute from the subset in accordance with a differentiator for each attribute of the subset.
• a first value associated with a given attribute can be determined as a value between a value of the given attribute of a first defect belonging to the one or more minority classes and a value of the given attribute of a second defect neighboring the first defect and belonging to the majority class, as a value of the given attribute of a defect belonging to the one or more minority classes, etc.
• the PMC is further configured to: identify at least part of the output defects as belonging to the one or more minority classes; update the first classifier by training the first classifier on a trai ning set including at least some d effectss from the plurality of defects that belong to the one or more minority classes, at least some defects from the plurality of defects that belong to the majority class, and the output defects identified as belonging to the one or more minority classes: and update the plurality of defects classified by the first classifier to the majority class and output the defects appeared upon updating and for which the engine obtains a confidence level below the threshold.
• a method of classifying defects executed by a computing platform comprising a processor and memory circuitry (PMC).
• the method comprises: receiving by the PMC a plurality of defects classified by a first classifier into a majority class, wherein one or more defects from the plurality of defects belong to one or more minority classes and at least part of the defects from the plurality of defects belong to the majority class, and wherein each defect from the plurality of defects is characterized by plurality of attributes each having a respective value; selecting by the PMC a subset of the plurality of attributes and define a differentiator for each attribute of the subset, such that a second classifier configured to classify defects in accordance with the subset and the differentiator for each respective attribute, classifies correctly to the one or more minority classes and the majority class at least a predefined share of defects from the plurality of defects; generating by the PMC a temporary training set, the temporary training set comprising: at least part of the
• the method can further comprise generating additional defects classifiable by the second classifier to the one or more minority classes and including the additional defects into the temporary training set.
• At least one of the additional defects can be a synthetic defect generated as a plurality of first values, each first value associated with an attribute from the subset in accordance with a differentiator for each attribute of the subset
• the method can further comprise: identifying at least part of the output defects as belonging to tlie one or more minority classes; updating die first classifier by training the first classifier on a training set including at least some defects from the plurality of defects that belong to the one or more minority classes, at least some defects from the plurality of defects that belong to the majority class, and the output defects identified as belonging to the one or more minority classes; and updating the plurality of defects classified by the first classifier to the majority ciass and outputting the defects appeared upon updating and for which the engine obtains a confidence level below the threshold.
• success criteria related to automated defect classification.
• Some non-limiting examples of success criteria include classification purity of one or more classes, classification accuracy of one or more classes, and extraction
• FIG. 1 illustrates a generalized block diagram of an examination system in accordance with certain embodiments of the presently disclosed subject matter
• FIG. 2 illustrates a generalized flow-chart of a method of identifying defects that may belong to a minority class, in accordance with certain embodiments of the presently disclosed subject matter
• FIG. 3 illustrates a confusion matrix of defect classification, in accordance with certain embodiments of the presently disclosed subject matter
• Fig. 4 illustrates exemplar)' ⁇ two-dimensional defects upon which additional defects can be generated, in accordance with certain embodiments of the presently- disclosed subject matter.
• FIG. 5 illustrates a generalized flow-chart of a method of reviewing a specimen in accordance with certain embodiments of the presently disclosed subject matter.
• non-transitory memory and“non-transitory storage medium” used herein should be expansively construed to cover any volatile or non-volatile computer memory suitable to the presently disclosed subject matter.
• defect a specimen used in this specification should be expansively construed to cover any kind of abnormality or undesirable feature formed on or within a specimen.
• classification used in this specification should be expansively construed to cover any assignment of a defect to one of a plurality of predetermined classes. Typically, a classifier is trained upon defects known to belong to each of the required classes. Once training is done, the classifier can be used for classifying defects into any of the classes, whether it is known which class the defect belongs to or not.
• a given defect can belong to a majority class or a minority class.
• a classifier When applying a classifier to a set of detects, most of the defects will be automatically classified by the classifier to majority classes. However, due to an insufficient number of training defects which are known to belong to minority classes, many minority defects will also be erroneously classified to majority classes. Since only a fraction of the defects detected during inspection are further examined during review, the probability of a minority defect classified to a majority class to be reviewed is low.
• the stability of the examination process may be low: as smaller populations are more vulnerable to changes when defects are misclassified, a small number of defects, or even a single defect, may significantly affect the standard deviation of the process evaluation. It is thus hard to stably evaluate the examination process
• defects of type“particle” may be significantly different in metal and in silicone. By not identifying and training specific minority classes, important information related to particular defects may be lost.
• FIG. 1 illustrating a generalized block diagram of an examination system in accordance with certain embodiments of the presently disclosed subject matter.
• Examination system 100 illustrated in Fig. 1 can be used for examination of a specimen (e.g. of a wafer and/or parts thereof) as a part of specimen fabrication. Examination can be part of the object fabrication, and can be carried out during manufacturing of the object, or afterwards.
• the examination system can comprise a variety of examination tools, for example, one or more inspection examination tools 101 configured to capture inspection images (typically, at relatively high speed and/or low resolution), and one or more review examination tools 102 configured to capture review images of at least part of defects detected by inspection tools 101 (typically, at relatively low speed and/or high resolution).
• the illustrated examination system 100 further comprises a computer-based automated defect classifying tool (referred to hereinafter also as a classifier) 103 capable of automatically classifying defects into a plurality of classes in accordance with tire defects type.
• a computer-based automated defect classifying tool referred to hereinafter also as a classifier
• classification may have different purposes, and the classification results can be used to identify specific defects of interest (Dol), filter false defects from true defects, establish Pareto in order to identify excursions m statistical process control (SPC), and/or otherwise, in accordance with classification memeposes.
• Dol specific defects of interest
• SPC statistical process control
• GUI 108 may be configured to enable the user (and/or management system) to set and monitor success criteria or measures related to automated defect classification, such as purity, accuracy, extraction, or other parameters separately for each class, and to optimize the classification results in accordance with user requirements.
• Classifier 103 can be operatively connected to one or more inspection tools 101 and/or one or more review tools 102.
• classifier 103 can be fully or partly- integrated with one or more inspection tools 101 or with one or more review' tools 102.
• Classifier 103 can be further operatively connected to design server 110 and data repository 109.
• a specimen can be examined by inspection tool 101 (e.g. an optical inspection system, low-resolution SEM, etc.).
• the resulting images and/or derivatives can be processed (optionally together with other data as, for example, design data and/or defect classification data) to select defects for review.
• a subset of potential defect locations selected for review can be reviewed by review tool 102 (e.g a scanning electron microscope (SEM) or Atomic Force Microscopy (AFM), etc.)
• review data 122 informative of review images and/or derivatives thereof and respective associated metadata can be transmitted - directly or via one or more intermediate systems - to classifier 103 It is noted that review data can include data generated by the one or more intermediate systems as derivatives of review images
• Classifier 103 comprises a processor and memory' circuitry (PMC) 104 operatively connected to a hardware-based input interface 105 and to a hardware-based output interface 106.
• PMC 104 is configured to provide processing necessary for operating the classifier as further detailed with reference to Fig. 2, and comprises a processor and a memory' (not shown separately within PMC 104). Operation of classifier 103 and PMC 104 will be further detailed with reference to Figs. 2-4.
• Processing results of Classifier 103 output by output interface 106 may be provided as inspection instruction related data 123 or review instruction related data 124 to inspection examination tools 101 or review examination tools 102, respectively, in order to change their operation, for example by changing values of operation parameters.
• the examination system illustrated in Fig. 1 can be implemented in a distributed computing environment, in which the aforementioned functional modules shown in Fig. 1 can be distributed over several local and/or remote devices, and can be linked through a communication network. It is further noted that in other embodiments at least part of the examination tools 101 and/or 102, data repositories 109, storage system 107, design server 110 and/or GUI 108 can be external to the examination system 100 and operate in data communication with classifier 103 via input interface 105 and output interface 106. Classifier 103 can be implemented as a stand-alone computer(s) to be used in conjunction with one or more examination tools.
• classifier 103 can operate on pre-acquired review data stored in data repositories 109 and/or storage system 107.
• the respective functions of classifier 103 can, at least partly, be integrated with one or more examination tools, process control tools, recipe generation tools, systems for automatic defects review and/or classification, and/or other systems related to examination.
• PMC 104 can comprise minority defect identification engine 117, training engine 120 and one or more classification engines 121.
• Minority defect identification engine 117 can be used for identifying minority defects which can be used by training engine 120 to tra one or more classification engines 121
• Minority defect identification engine 117 can comprise initial minority defects obtaining engine 118 for identifying minority defects for which ground truth exists, and which are erroneously classified by a classification engine 121 to majority classes. Minority defect identification engine 117 can further comprise additional minority defects obtaining engine 119 configured to identify defects that are likely to be minority defects, but for which no ground truth exists. A user can then review these defects, and determine whether any one or more of the defects is indeed a minority defect. Once a sufficient number of minority class defects are available, training engine 12Q can be used for training a classification engine, also referred to as classifier 121.
• PMC 1Q4 can comprise training engine 12Q configured to receive a multiplicity of defects each with a corresponding classification, and output a classifier adapted to receive a defect and provide a class to which the defect is classified.
• PMC 104 can comprise one or more classification engines 120, also referred to as classifiers, which may be trained by training engine 118 or by another training engine, or received from an external source. As detailed below, one such engine can he an initial engine trained with an insufficient number of minority defects, and another engine can be trained with further defects.
• PMC 104 is configured to execute the respective operations detailed below in accordance with computer-readable instructions implemented on a non-transitory computer-readable storage medium.
• table 300 of Fig. 3 shows human vs. machine classification of a total of 250 defects, detected for example during inspection of an exemplary specimen, each defect belonging to one of classes A, B, C or D.
• Table 300 comprises rows 304, 308, 312 and 316, each related to defects of each class, as classified by a user, therefore this assignment to classes is referred to as "ground troth".
• the number of defects classified by the user are as shown in "total" column 332: 60 of Class A, 44 of Class B, 119 of Class C and 27 of Class D.
• a default or any other initial classifier is also used, which classifies the defects as shown in columns 320, 324 and 328 of table 300: for example, 52 defects of Class A have been indeed classified to Class A, 2 defects of Class B have been classified as class C, and so on.
• Classes A, B and C are classified correctly, as shown on the first three entries of the main diagonal of table 400: 52 out of 60 for Class A, 38 out of 44 for Class B, and 110 of 119 for Class C.
• classes A, B, and C are considered majority classes.
• no defect, of any class has been classified to Class D. This is likely because there were not enough defects of Class D when the classifier was trained, thus the classifier could not classify defects to this class.
• Class D is referred to as a minority class, and defects of this class are classified to the majority classes. It is therefore required to find more Class D defects, in order to train a classifier accordingly.
• PMC 104 receives (200) a plurality of defects which are classified by a first classifier into a first majority class.
• the received defects have been pre-dassified into predefined classes (e.g. by a human expert and/or another classifier and/or a previous classifier’s version, etc ), such that this pre-classification is considered ground truth.
• predefined classes e.g. by a human expert and/or another classifier and/or a previous classifier’s version, etc.
• PMC 104 may receive the defects assigned by the first classifier to one class, for example Class A shown on col. 32Q, and which are known to belong to class A or to class D. Defects classified to class A but belonging to other majority classes, for example the 4 defects belonging to Class B and the 3 defects belonging to Class C, are ignored. Therefore, it is required to differentiate between the 52 defects of Class A rightfully classified to Class A and the 3 defects of Class D erroneously classified to Class A.
• Each defect may be described as a collection, for example a vector, of values assigned to various attributes. Typically, each defect may be described as values assigned to hundreds or more atributes, related for example to its type, color, size, location, cause, or the like.
• PMC 1Q4 can then select (204) a subset of the attributes and define a differentiator for each attribute of the subset, in order to differentiate correctly between the defects belonging to the majority class and the defects belonging to the minority class.
• the atribute subset selection may be referred to as dimensionality reduction.
• dimensionality reduction When there are more dimensions (i e , attributes) than defects, it is mathematically possible to make such differentiation .
• reducing the number of dimensions may be useful in enhancing the generalization, e.g., the differentiation stability, such that additional defects can be classified as well to either class.
• Differentiation may be described as a temporary classifier based only on the selected attribute subset, and classifying the defects into the majority and minority classes. It will be appreciated that differentiating correctly between the defects assigned to the majority class and the defects assigned to the minority class, may relate to at least a predetermined part of the majority class defects being classified to the majority class, and at least a (possibly different) predetermined part of the minority class defects being classified to the minority class. A predetermined part may relate to a predetermined percentage of the respective defects, at least a predetermined number of the respective defects, or the like.
• the differentiator may be a cutoff point, such that defects having values exceeding the threshold are associated with one class, while defects having values belo the threshold are associated with die odier class. If die attribute may receive one of two possible values, for example 0 or 1 , the differentiator may be defined such that defects having the first value for the attribute are associated with one class, while defects having the second value for the attribute are associated with the otiier class.
• Selecting the attribute subset may use the Random Forest method, which operates by constructing a plurality of decision trees, receiving a classification result from each such decision tree, and taking a decision complying with the decision taken by a majority of the trees.
• a first tree is constructed: the first attribute may be selected in accordance with a Gini Impurity index, which may be regarded as a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset. For example, assume nine defects are provided, each belonging to one of two classes: "X", and "0".
• the nine defects have the following values for an attribute: j 1, 1.5, 2, 3, 4, 5.5, 6, 8, 10]
• the defect vector is: [X X 0 X X 0 0 X 0], meaning that the first, second, fourth, fifth and eighth defects are of class "X", and the others are of class "0".
• a Gini Impurity index for class "X”, defined as Pi *(1-Pi), for this vector, wherein Pi is the probability of correctly classified "X" is - * ⁇ -7. It will be appreciated that the optimal value for the Gini Impurity index is 0, which indicates perfect differentiation by an attribute (where Pi is either 0 or 1).
• the vector would be as follows: [X X j 0 X X 0 0 X 0] meaning that the defects to the left of the "j" symbol are classified as "0" and the defects to the right are of the "j" symbol are classified as "X”,
• Gini Impurity index for class "X” is: Similarly, the Gini Impurity 2 12 7
• the attribute and differentiator selection that provide the best weighted average may be selected as a first level of a decision tree. On a next level, a different atribute, or the same attribute with a different differentiating value, may be selected.
• the proportion of the defects erroneously classified to the majority class that may be selected for constructing the random forest may he higher than its proportion in the defects classified to the majority class. For example, if 100 defects have been classified to the majority class, out of which two belong to the minority class and 98 to the majority class, 20 defects may be selected for the random forest, winch include the two defects belonging to tire minority class, and 18 defects belonging to the majority class.
• a multiplicity of such trees may be constructed, each based on a different set of defects and a different selection of attributes and differentiators.
• an attribute subset may be selected.
• the attribute subset may be selected based on each attribute participation in the trees: an attribute participating in more trees, or higher in one or more of the trees, may be assigned a higher grade than an atribute participating in fewer trees, or is lower in the hierarchy in one or more of the trees.
• Idle Random Forest method may eliminate overfitting the atribute set to the available training set, wherein overfitting relates to a situation in which the attribute subset is designed to differentiate the training set well, but is therefore less effective for other defect collections.
• PMC 104 can then generate (212) a temporary training set, comprising: a plurality of defects of the majority class that have been classified correctly; and a plurality of defects of the minority class that have been classified incorrectly to the majority class in some embodiments, not all the defects classified to the majority class are selected. Moreover, the proportion of the defects of the minority class that have been classified incorrectly to the majority class may be higher than their respective part in all defects classified as majority defects.
• the temporary training set may also comprise additional defects.
• the additional defects can be generated by PMC 104.
• Tire additional defects are defects which a classifier based on the attribute subset, such as tire random forest classifier above, will classify to the minority class. Generating the additional defects may also be referred to as oversampling. It will be appreciated that generating a defect may be performed by assigning values to the attributes in the attribute subset.
• oversampling may relate to generating one or more synthetic defects, i.e. one or more collections of values, each of which is relatively close to an existing minority defect.
• Fig. 4 show's three minority defects, 400, 404 and 412 denoted by "Q", and a number of majority defects such as 408 or 416, denoted by "X".
• Q minority defects
• X majority defects
• more synthetic defects may be generated which are closer to minority defects having neighbors of the majority class.
• a synthetic defect may be generated close to defects 404 and 412, but not to defect 400, which has no close neighbors of class "X”.
• Generating synthetic defects may be performed using the Synthetic Minority
• SMOTE Oversampling Technique
• the distance to the nearest majority defect may be calculated. The distance calculation can be performed using any required metrics, such as (possible weighted) sum of squares over the selected attribute subset.
• a predetermined number of minority defects may be selected having the smallest distances to the nearest neighbor of the majority class. In some embodiments, the same minority defect may be selected multiple times. Thus, minority defects 4Q4 and 412 may be selected.
• a synthetic defect may be created, having values for the attribute in the range between the attribute value of the minority defect and the attribute value of the closest majority defect. In the example of Fig. 4, a synthetic defect may be created along the line connecting minority defect 404 and majority defect 408.
• the synthetic defects may be generated closer to the minority defect, for example the value of each attribute may be calculated as 0.8 * the value of attribute in minority defect + 0.2 * value of attribute in majority defect, or any other weighted combination thereof.
• a second synthetic defect may be generated along the line connecting minority defect 412 and majority defect 416.
• PMC 104 can then train (216) a confidence level engine upon the temporary training set, the attribute subset, and the differentiator for each attribute of the subset.
• the confidence level engine may be configured to receive a defect and a class to which the defect is classified, and to output a confidence level that the defect indeed belongs to the class.
• the confidence level engine may also be implemented as a tree ensemble, wherein the trees of the tree ensemble may be generated using gradient boosting. Generating the decision trees may start with a default classification tree, operating in accordance with the selected attribute subset, and classifying defects to the majority- class and the minority class.
• An enhanced Gini Impurity index may take into account a weight assigned to each class: WA*PA*(1-WAPA), wherein WA is the weight assigned to class A and PA is the probability of a defect to be correctly classified to class A
• WA is the weight assigned to class A
• PA is the probability of a defect to be correctly classified to class A
• the weight of the minority class may be higher than the proportion of the minority defects within all defects classified to the majority class. Further trees may then be generated, in winch a higher weight is assigned to those minority defects that were incorrectly classified by previously generated trees, for example as follows:
• the output of the confidence level engine generation is thus a collection of trees, each tree associated with a class and weight.
• the weight may be obtained based on the error rate associated with each tree.
• the error rate may be defined as the sum of the weights of the defects for which the tree outputs a result different from the ground truth.
• PMC 104 can apply (220) the confidence level engine to further defects classified to the majority class, wherein no ground truth is available for these defects. Each such defect classified to the majority- class is thus provided to the confidence level engine, and a confidence level is obtained.
• the confidence level indicates the confidence that the defect is indeed of the majority class and was not erroneously classified to the majority class.
• Obtaining the confidence level may be performed by classifying the defect by all trees in the tree ensemble generated at step 216, and summing the weights of all trees that classify the defect to the majority class, thus obtaining a confidence level between 0 and 1.
• PMC 1Q4 can then output (224) defects classified to the majority class, for which the confidence level is low, for example below a predetermined threshold. Alternatively, a predetermined number of defects, or a predetermined percentage of all defects classified to the majority class, that have the lowest confidence le vels, may be output. A low confidence level may indicate that there is a high probability that the defect is not a majority defect, but rather a minority defect.
• FIG. 5 there is illustrated a generalized flow -chart of a method of reviewing a specimen in accordance w ith certain embodiments of the presently disclosed subject matter.
• PMC 104 can be configured to execute the respective operations detailed below m accordance with computer-readable instructions implemented on a non-transitory computer-readable storage medium.
• a classifier is trained (500) upon available training data comprising training defects classified by a user.
• Classification (S04) is then performed for further defects, for which no classification is known.
• the classification results may be monitored and rejected (508) as described in association w ith Fig. 2 above .
• Monitoring relates to obtaining a confidence level for the majority defects, for example as detailed in steps 200, 204, 212, 216 and 220 of Fig. 2, and rejection may relate to outputting (224) defects for which the classification confidence level is low 7 .
• the output defects can be reviewed, for example by a user, and ground truth classification may be received (512) for these defects
• the defects indicated by a user as minority defects may be collected (516) and saved in a storage device or a data structure, referred to as a“container”.
• a number of minority defects sufficient for training is accumulated in a container, for example at least a predetermined number, such as 30, 40, 100 or the like, a new classifier can be trained in which this class is also a majority class, and which may thus classify correctly also the minority defects. This classifier may thus replace the first classifier
• Training (500) can then be repeated, which may also be referred to as re-training or updating the classifier, with the available ground truth classification.
• the iterations of Fig, 5 may continue for a predetermined number of times, for a predetermined period of time, until the rejection rate drops below a first predetermined threshold, until the number of rejections drops below a second predetermined threshold, or until another stopping criteria is met
• the system according to the invention may be, at least partly, implemented on a suitably programmed computer.
• the invention contemplates a computer program being readable by a computer for executing the method of the invention.
• the invention further contemplates a non-transitory computer- readable memory tangibly embodying a program of instructions executable by the computer for executing the method of the invention.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Computing Systems (AREA)
• Mathematical Physics (AREA)
• Databases & Information Systems (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Medical Informatics (AREA)
• Computational Mathematics (AREA)
• Pure & Applied Mathematics (AREA)
• Mathematical Optimization (AREA)
• Mathematical Analysis (AREA)
• Computational Linguistics (AREA)
• Probability & Statistics with Applications (AREA)
• Algebra (AREA)
• Life Sciences & Earth Sciences (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Evolutionary Biology (AREA)
• Bioinformatics & Computational Biology (AREA)
• Testing Or Measuring Of Semiconductors Or The Like (AREA)
• Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)
• Analysing Materials By The Use Of Radiation (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=71097693

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (4)

Citations (3)

Family Cites Families (19)

• 2018
  • 2018-12-20 US US16/228,676 patent/US11321633B2/en active Active
• 2019
  • 2019-11-24 CN CN201980065723.0A patent/CN112805719B/zh active Active
  • 2019-11-24 WO PCT/IL2019/051284 patent/WO2020129041A1/en not_active Ceased
  • 2019-11-24 KR KR1020217015166A patent/KR102530950B1/ko active Active
  • 2019-11-24 JP JP2021524054A patent/JP7254921B2/ja active Active
  • 2019-12-17 TW TW108146199A patent/TWI791930B/zh active

Patent Citations (3)

Also Published As

Similar Documents

Legal Events