TWI752100B - Systems, non-transitory computer-readable media, and computer-implemented method for training inspection-related algorithms - Google Patents

Abstract

Methods and systems for training an inspection-related algorithm are provided. One system includes one or more computer subsystems configured for performing an initial training of an inspection-related algorithm with a labeled set of defects thereby generating an initial version of the inspection-related algorithm and applying the initial version of the inspection-related algorithm to an unlabeled set of defects. The computer subsystem(s) are also configured for altering the labeled set of defects based on results of the applying. The computer subsystem(s) may then iteratively re-train the inspection-related algorithm and alter the labeled set of defects until one or more differences between results produced by a most recent version and a previous version of the algorithm meet one or more criteria. When the one or more differences meet the one or more criteria, the most recent version of the inspection-related algorithm is outputted as the trained algorithm.

Description

System, non-transitory computer-readable medium, and computer-implemented method for training inspection-related algorithms

The present invention generally relates to methods and systems for optimizing training sets for setting inspection correlation algorithms.

In this section, the following descriptions and examples are not considered prior art by virtue of their inclusion.

During a semiconductor manufacturing process, inspection processes are used at various steps to detect defects on wafers, thereby promoting higher yields and thus higher profits in the manufacturing process. Inspection has always been an important part of manufacturing semiconductor devices. However, as the size of semiconductor devices decreases, inspection becomes more important to the successful manufacture of acceptable semiconductor devices, since smaller defects can lead to device failure.

When defects are detected on a sample, such as a wafer, some type of algorithm is typically applied to the detected defects to equally divide them into different types of defects (or separate defects from non-defects). One way to accomplish this is to apply a defect classifier to the detected defects, which divides the detected defects into different types or classes of defects. Defect classifiers typically use one or more parameters of defects and/or defect images (eg, acquired in the vicinity of such defects commonly referred to as "block" images or "blocks") as input to determine the type of defect or grade. The defect classifier then assigns certain types of identifiers or IDs to each defect to indicate the determined type or class. Another way to separate detected defects is to separate actual defects from nuisances or noise. A "nuisance" defect is generally defined as a defect that the user does not care about and/or a defect that is detected as a defect but is not actually a defect. These algorithms are often referred to as defect filters and/or nuisance filters. The most widely used classifiers/nuisance filters on optical inspection tools are based on manually constructed decision trees. The tuning method used for these decision trees utilizes the experience and domain knowledge incorporated into the best known method (BKM) for tree construction. This typically results in the decision tree being initially constructed using BKM "templates", defect clusters, and substantially coarser defect markers (using blocks). After the structure of the tree has been obtained, the tree is then diversely sampled using diversity sampling, where there is a smart sample distribution across the leaf nodes on the tree. Next, a scanning electron microscope (SEM) inspects, classifies the sampled defects and uses them, etc., to finally adjust the decision cut lines (the boundaries separating different types of defects). Given a training set, other classifiers based on machine learning algorithms (eg nearest neighbor type classifiers) will automatically find decision boundaries, but currently there is no way to obtain a training set that will maximize their equivalent performance. However, currently used methods for setting and tuning defect classifiers suffer from a number of drawbacks. For example, existing methods are labor-intensive, require a lot of expertise, and will produce inconsistent results that rely on human experts. Building a classifier by a human expert is error-prone and expensive and time-consuming. Each defect has a relatively large number of features, making it nearly impossible to properly visualize the features for classification. Thus, due to lack of knowledge about the underlying multidimensional distribution, a human expert can make significant errors in constructing the classification boundaries. Even if there is no significant error, the probability of artificially creating a non-optimal classifier is substantially high. Accordingly, it would be advantageous to develop a system and/or method for optimizing an algorithm for setting check correlation that does not have one or more of the disadvantages described above.

The following descriptions of various embodiments are not to be construed in any way as limiting the scope of the appended claims. One embodiment relates to a system configured to train an inspection correlation algorithm. The system includes an inspection subsystem including at least one energy source and a detector. The energy source is configured to generate energy directed to a sample. The detector is configured to detect energy from the sample and generate an output in response to the detected energy. The system also includes one or more computer subsystems. One or more computer subsystems are configured to perform an initial training of an inspection correlation algorithm using a set of marked defects, thereby generating an initial version of the inspection correlation algorithm. The computer subsystem(s) are also configured to apply an initial version of the inspection-related algorithm to a set of unmarked defects and to alter the set of marked defects based on the results of the application. Additionally, the computer subsystem(s) are configured to retrain the inspection correlation algorithm using the changed set of flagged defects, thereby producing a newer version of the inspection correlation algorithm. The computer subsystem(s) are further configured to apply a newer version of the inspection-related algorithm to another unmarked defect group. In addition, the computer subsystem(s) are configured to determine one or more differences between the results of applying a newer version of the check-related algorithm and the results of applying an initial version or an older version of the check-related algorithm . The computer subsystem(s) are also configured to repeatedly change the marked defect set, retrain the inspection-related algorithm, apply a newer version of the inspection-related algorithm, and determine the one or more differences until the one or more The difference meets one or more criteria. When the one or more differences meet the one or more criteria, the subsystem(s) are configured to output an up-to-date version of the check correlation algorithm as a trained check correlation algorithm for checking other samples. The system can be further configured as described herein. Another embodiment relates to a computer-implemented method for training an inspection correlation algorithm. The method includes the steps of each of the functions of one or more of the computer subsystems described above. The steps of the method are performed by one or more computer systems. The method can be performed as further described herein. Additionally, the method may comprise any other step(s) of any other method(s) described herein. Furthermore, the method can be performed by any of the systems described herein. An additional embodiment relates to a non-transitory computer-readable medium storing program instructions executed on a computer system for performing a computer-implemented method of training an inspection correlation algorithm. The computer-implemented method comprises the steps of the method described above. The computer-readable medium can be further configured as described herein. The steps of the computer-implemented method can be performed as further described herein. In addition, the computer-implemented method for which the program instructions may be executed may be included in any other step(s) of any other method(s) described herein.

Turning now to the diagrams, it should be noted that the diagrams are not drawn to scale. In particular, the proportions of some of the elements in the figures are greatly exaggerated to emphasize the characteristics of the elements. It should also be noted that the figures are not drawn to the same scale. Elements shown in one of the above figures that can be configured similarly have been designated using the same reference numerals. Unless otherwise specified herein, any of the elements described and shown may comprise any suitable commercially available elements. One embodiment relates to a system configured to train an inspection correlation algorithm. In general, the embodiments described herein provide methods and systems for obtaining a training set of minimum size for classifying defects captured by optical tools and other tools, or for other inspection-related functions. In addition, the embodiments described herein can be advantageously used to find the smallest subset of the most instructive defects to build the classifiers described herein and other inspection-related algorithms for use with the defects described herein The purpose of classification and other inspection-related functions. Traditionally, the process of tuning sample inspection (eg, optical wafer inspection) for optimal performance has been almost entirely manual. The tuning procedure generally relies on the best known method (BKM) and the experience and skill of the human expert performing the tuning. Therefore, these methods are not expected to be used to set up production monitoring systems, not only because they are equally costly (energy and labor), but also because the adjustment results are subjective and lack consistency. However, despite these obvious shortcomings of the current check-tuning method, attempts to automate this procedure are not widely accepted in this production environment. The main reason is that this automation relies on algorithms whose performance derives from the data (referred to as a training set) used to train them. Therefore, unless training data is obtained in a systematic manner, the performance of these algorithms is uncertain. In other words, these automated solutions have all the problems of manual methods in the absence of finding a reliable way to optimize the performance of the algorithms. In particular, these solutions are not consistent, and no matter how good the underlying algorithm is, there is no guarantee that the performance of the underlying algorithm will match the performance of the manual method. Additionally, diagnosing performance issues and resolving them after finding them is often very difficult, if not impossible, in nature. So, so far, machine learning methods (as they are now called) have not been successful. Embodiments described herein provide an integrated tuning method for any machine learning algorithm that can be used for inspection-related functions such as classification and filtering. (Even though these embodiments can also be applied to detection algorithm tuning, the embodiments described herein are particularly useful for nuisance filters and classifiers.) These embodiments are based on targeting inspections for obtaining training sets The method can be advantageously fully integrated with the algorithmic tuning itself. The two are interconnected, and they should not be separated from each other to provide consistent behavior. The basic reasons for this interdependence are as follows. Inspections such as optical inspection are adjusted using thermal scans (high defectivity scans with a substantially higher rate of obstruction). The adjustment itself requires flagging defects (ie, classification defects that are typically classified by a human expert). This classification is performed on Scanning Electron Microscope (SEM) images acquired by a SEM viewing tool. The embodiments described herein would not be required if all defects detected in the thermal scan could be viewed and classified. However, since this inspection/sorting process is substantially labor and tool time consuming, this is not practically possible. Therefore, it is absolutely necessary to identify a suitable subset of defects that can yield the best performance of a classifier or other check-related algorithm, and it is highly desirable to find the smallest group that achieves this. Embodiments described herein provide systems and methods for optimizing the selection of defective training sets by learning iterations in which relevant algorithm (eg, classifier model) learning data is examined and the required data is requested to improve its efficiency. Embodiments described herein will also advantageously provide methods and systems for determining the point in time at which learning has reached an endpoint. In one embodiment, the sample includes a wafer. In another embodiment, the sample includes a reticle. The wafer and the reticle may comprise any wafer and reticle known in the art. An embodiment of such a system is shown in FIG. 1 . The system includes an inspection subsystem including at least one energy source and a detector. The energy source is configured to generate energy directed to a sample. The detector is configured to detect energy from the sample and generate an output in response to the detected energy. In one embodiment, the energy directed to the sample includes light, and the energy detected from the sample includes light. For example, in the embodiment of the system shown in FIG. 1 , inspection subsystem 10 includes an illumination subsystem configured to direct light to sample 14 . The lighting subsystem includes at least one light source. For example, as shown in FIG. 1 , the lighting subsystem includes a light source 16 . In one embodiment, the illumination subsystem is configured to direct light to the sample at one or more angles of incidence, which may include one or more oblique angles and/or one or more normal angles. For example, as shown in FIG. 1, light from light source 16 is directed through optical element 18 and then through lens 20 to beam splitter 21, which directs the light at a normal angle of incidence to Sample 14. The angle of incidence may include any suitable angle of incidence, which may vary depending, for example, on the characteristics of the sample and the defects to be detected on the sample. The illumination subsystem can be configured to direct light to the sample at different times and at different angles of incidence. For example, the inspection subsystem can be configured to alter one or more characteristics of one or more elements of the illumination subsystem such that light can be directed to the sample at an angle of incidence other than that shown in FIG. 1 . In one such example, the inspection subsystem can be configured to move the light source 16, optical element 18, and lens 20 so that light is directed to the sample at a different angle of incidence. In some instances, the inspection subsystem can be configured to simultaneously direct light to the sample at more than one angle of incidence. For example, an illumination subsystem may include more than one illumination channel, one of which may include a light source 16, an optical element 18, and a lens 20 as shown in FIG. 1 and the other of the illumination channels (not shown in the figure) shown) may include similar elements, which may be configured differently or the same, or may include at least one light source and may include one or more components, such as those described further herein. If this light is directed to the sample at the same time as another light, one or more properties (eg wavelength, polarization, etc.) of the light directed to the sample at different angles of incidence may be different such that at different angles of incidence The light generated by the illumination of the sample can be distinguished from each other at the detector(s). In another example, the lighting subsystem may include only one light source (eg, source 16 shown in FIG. 1 ) and the light from the lighting subsystem may be converted by one or more optical elements (not shown) of the lighting subsystem The light of the light source is split into different optical paths (eg, based on wavelength, polarization, etc.). The light in each of these different optical paths can then be directed to the sample. Multiple illumination channels can be configured to direct light to the sample simultaneously or at different times (eg, when different illumination channels are used to sequentially illuminate the sample). In another example, the same illumination channel can be configured to direct light with different characteristics to the sample at different times. For example, in some instances, optical element 18 may be configured as a spectral filter and the properties of the spectral filter may be changed in various ways (eg, by swapping the spectral filter) such that different The wavelength of light is directed to the sample. The illumination subsystem may have any other suitable configuration known in the art for sequentially or simultaneously directing light with different or the same characteristics to the sample at different or the same angle of incidence. In one embodiment, the light source 16 may comprise a broadband plasma (BBP) light source. In this way, the light generated by the light source and directed to the sample may comprise broadband light. However, the light source may comprise any other suitable light source such as a laser. The laser may comprise any suitable laser known in the art and may be configured to generate light at any suitable wavelength or wavelengths known in the art. Additionally, lasers can be configured to produce monochromatic or near-monochromatic light. In this way, the laser can be a narrowband laser. The light source may also include a polychromatic light source that produces light at a plurality of discrete wavelengths or wavelength bands. Light from optical element 18 can be focused by lens 20 onto beam splitter 21 . Although lens 20 is shown in FIG. 1 as a single refractive optical element, it should be understood that lens 20 may actually include several refractive and/or reflective optical elements that together focus light from the optical element to the sample. The illumination subsystem shown in Figure 1 and described herein may include any other suitable optical elements (not shown in the figure). Examples of such optical elements include, but are not limited to, polarizer(s), spectral filter(s), spatial filter(s), reflective optical element(s), apodizer(s), beam splitter(s) , aperture(s) and the like, which may contain any such suitable optical elements known in the art. Additionally, the system can be configured to alter one or more of the elements of the lighting subsystem based on the type of lighting to be used for the inspection. The inspection subsystem may also include a scanning subsystem configured to cause the light to scan the sample. For example, the inspection subsystem may include a stage 22 on which samples 14 are placed during inspection. The scanning subsystem can include any suitable mechanical and/or robotic assembly (including stage 22) that can be configured to move the sample so that light can scan the sample. Additionally or alternatively, the inspection subsystem may be configured such that one or more optical elements of the inspection subsystem perform some kind of scanning of the sample with light. The light can be made to scan the sample in any suitable manner. The inspection subsystem further includes one or more detection channels. At least one of the one or more detection channels includes a detector configured to detect light from the sample due to illumination of the sample by the inspection subsystem and to generate an output in response to the detected light. For example, the inspection subsystem shown in FIG. 1 includes two detection channels: one detection channel is formed by concentrator 24, element 26, and detector 28 and the other detection channel is formed by concentrator 30, element 32 And the detector 34 is formed. As shown in Figure 1, the two detection channels are configured to collect and detect light at different collection angles. In some examples, one detection channel is configured to detect specularly reflected light, and the other detection channel is configured to detect light that is not specularly reflected (eg, scattered, diffracted, etc.) from the sample. However, two or more of the detection channels can be configured to detect the same type of light (eg, specularly reflected light) from the sample. Although FIG. 1 shows one embodiment of an inspection subsystem that includes two detection channels, the inspection subsystem may include a different number of detection channels (eg, only one detection channel or two or more detection channels) ). Although each of the light collectors is shown in FIG. 1 as a single refractive optical element, it should be understood that each of the light collectors may include one or more refractive optical elements and/or one or more reflective optical elements. The one or more detection channels may comprise any suitable detector known in the art. For example, detectors may include photomultiplier tubes (PMTs), charge-coupled devices (CCDs), and time-lapse integration (TDI) cameras. The detector may also include any other suitable detector known in the art. Detectors may also include non-imaging detectors or imaging detectors. In this way, if the detectors are non-imaging detectors, each of the detectors can be configured to detect a particular characteristic of scattered light, such as intensity, but cannot be configured to detect dependent imaging planes The characteristic that changes with the location within. Thus, the output produced by each of the detectors included in each of the detection channels of the inspection subsystem may be a signal or data, but not an image signal or image data. In these examples, a computer subsystem, such as the computer subsystem 36 of the system, may be configured to generate images of the samples from the non-imaging outputs of the detectors. However, in other examples, the detector may be configured as an imaging detector configured to generate image signals or image data. Accordingly, the system can be configured to produce the output described herein in a number of ways. It should be noted that FIG. 1 is provided herein to generally illustrate one configuration of an inspection subsystem. Obviously, the inspection subsystem described herein can be altered to optimize the performance of a commercial inspection system as it normally performs when designing the system. Additionally, this document may be implemented using an existing system (eg, by adding the functionality described herein to an existing inspection system) such as the 28xx and 29xx series of tools available from KLA-Tencor, Milpitas, Calif. system described in . For some of these systems, the methods described herein may be provided as optional functionality of the system (eg, in addition to other functionality of the system). Alternatively, the systems described herein may be designed "from the ground up" to provide an entirely new system. The computer subsystem 36 of the system may be coupled to the detectors of the inspection subsystem in any suitable manner (eg, via one or more transmission media, which may include "wired" and/or "wireless" transmission media) such that during the sample During the scan, the computer subsystem can receive the output generated by the detector. Computer subsystem 36 may be configured to use the outputs of the detectors described herein to perform several functions and any other functions described further herein. This computer system can be further configured as described herein. This computer subsystem (and other computer subsystems described herein) may also be referred to herein as computer system(s). Each of the computer subsystem(s) or systems described herein may take various forms, including a personal computer system, video computer, mainframe computer system, workstation, network device, Internet device, or other device. In general, the term "computer system" can be broadly defined to encompass any device having one or more processors that execute instructions from a memory medium. The computer subsystem or system(s) may also include any suitable processor known in the art, such as a parallel processor. In addition, the computer subsystem(s) or systems may include a computer platform with high-speed processing and software as a stand-alone or networked tool. If the system includes more than one computer subsystem, the different computer subsystems may be coupled to each other such that images, data, information, instructions, etc. may be sent between the computer subsystems, as described further herein. For example, computer subsystem 36 may be coupled to computer subsystem(s) 102 by any suitable transmission medium, which may include any suitable wired and/or wireless transmission medium known in the art, as shown in FIG. shown by the dotted line. Two or more of these computer subsystems can also be operatively coupled by a common computer-readable storage medium (not shown). Although the inspection subsystem is described above as an optical or light based inspection subsystem, the inspection subsystem may be an electron beam based inspection subsystem. For example, in one embodiment, the energy directed to the sample includes electrons, and the energy detected from the sample includes electrons. In this manner, the energy source may be an electron beam source. In one such embodiment shown in FIG. 2 , the inspection subsystem includes an electronic column 122 coupled to a computer subsystem 124 . As also shown in FIG. 2 , the electron column includes an electron beam source 126 that is configured to generate electrons focused by one or more elements 130 to the sample 128 . The electron beam source can include, for example, a cathode source or emitter tip, and the one or more elements 130 can include, for example, a gun lens, an anode, a beam-limiting aperture, a gate valve, a beam-current-selective aperture , an objective lens, and a scanning subsystem, all of which may include any such suitable elements known in the art. Electrons (eg, secondary electrons) retroreflected from the sample may be focused by one or more elements 132 to detector 134 . One or more elements 132 may include, for example, a scanning subsystem, which may be the same scanning subsystem included in element(s) 130 . The electron column may comprise any other suitable element known in the art. Additionally, the electron column can be further configured as described in: US Patent No. 8,664,594, issued April 4, 2014 to Jiang et al., US Patent No. 8,692,204, issued April 8, 2014 to Kojima et al. , US Patent No. 8,698,093, issued to Gubbens et al. on April 15, 2014, and US Patent No. 8,716,662, issued to MacDonald et al. on May 6, 2014, which are incorporated by reference as if set forth in their entirety. into this article. Although the electron column is shown in FIG. 2 configured such that electrons are directed to the sample at one oblique angle of incidence and scattered from the sample at another oblique angle, it should be understood that the electron beam may be directed to the sample at any suitable angle and free from Sample scattering. Additionally, electron beam-based subsystems can be configured to use multiple modes to generate images of the sample (eg, at different illumination angles, collection angles, etc.). The modes of the electron beam-based subsystem may differ in any of the image generation parameters of the subsystem. Computer subsystem 124 may be coupled to detector 134, as described above. The detector can detect electrons retroreflected from the surface of the sample, thereby forming an electron beam image of the sample. The electron beam image may comprise any suitable electron beam image. Computer subsystem 124 may be configured to perform any of the functions described herein using the detector output and/or electron beam images. Computer subsystem 124 may be configured to perform any additional step(s) described herein. A system including the inspection subsystem shown in FIG. 2 can be further configured as described herein. It should be noted that FIG. 2 is provided herein to generally illustrate one configuration of an electron beam-based inspection subsystem that may be included in the embodiments described herein. As with the optical inspection systems described above, the e-beam-based inspection subsystem configurations described herein can be altered to optimize the performance of the inspection subsystem as normally performed when designing a commercial inspection system. Additionally, the systems described herein may be implemented using an existing inspection system (eg, by adding functionality described herein to an existing system). For some of these systems, the methods described herein may be provided as optional functionality of the system (eg, in addition to other functionality of the system). Alternatively, the systems described herein may be designed "from the ground up" to provide an entirely new system. Although the inspection subsystem is described above as a light or electron beam based inspection subsystem, the inspection subsystem may be an ion beam based inspection subsystem. Such an inspection subsystem can be configured as shown in FIG. 2, except that the electron beam source can be replaced with any suitable ion beam source known in the art. Additionally, the inspection subsystem may be any other suitable ion beam-based subsystem, such as those included in commercially available Focused Ion Beam (FIB) systems, Helium Ion Microscopy (HIM) systems, and Secondary Ion Mass Spectrometry (SIMS) systems. One or more computer subsystems described further herein may be coupled to the inspection subsystem that performs the sample inspection. For example, in one embodiment, one or more computer subsystems are configured to detect defects on the sample based on the output generated by the detector. Alternatively, one or more other computer subsystems may be coupled to the examination subsystem that performs the examination of the sample. The computer subsystem(s) may be configured as described further herein. In any event, one or more computer subsystems coupled to the inspection subsystem are configured to detect defects on the sample based on outputs generated by one or more detectors of the inspection subsystem. The output can be identified in any suitable manner (for example, by applying a threshold value to the output and identifying an output with one or more values above the threshold value as a defect and not treating it with a value below the threshold value). The output of one or more values is identified as a defect) to detect defects on the sample. Defects detected on the sample may include any defect known in the art. However, it is not necessary for the computer subsystem(s) included in the systems described herein to detect defects on the samples. For example, the computer subsystem(s) may be configured to obtain the results of a sample inspection that includes information on defects detected on the sample. The computer subsystem(s) described herein may come directly from the system performing the inspection (eg, a computer subsystem of the self-inspection system) or since the inspection results have been stored in a storage medium (such as a fab data) library) to obtain the results of the sample inspection. As mentioned above, the inspection subsystem is configured for scanning a physical version of the sample with energy (eg, light or electrons), thereby producing an actual image of the physical version of the sample. In this way, the inspection subsystem can be configured as a "real" tool rather than a "virtual" tool. For example, a storage medium (not shown) and computer subsystem(s) 102 shown in Figure 1 may be configured as a "virtual" tool. In particular, the storage medium and computer subsystem(s) are not part of the inspection subsystem 10 and do not have any ability to handle physical versions of the samples. In other words, in a tool configured as a virtual tool, the output of one or more of its "detectors" may be the outputs previously generated by one or more detectors of an actual tool and stored in the virtual tool, And during a "scan", the virtual tool can replay the stored output as if the sample was scanned. In this way, scanning a sample with a virtual tool may appear to be the same as scanning a physical sample with an actual tool, but in fact, "scanning" simply involves replaying the output of the sample in the same manner as a scannable sample. Systems and methods configured as "virtual" inspection tools are described in commonly assigned US Pat. Nos. 8,126,255, Bhaskar et al., issued Feb. 28, 2012, and Duffy et al., issued Dec. 29, 2015 US Patent No. 9,222,895, issued today, both of which are hereby incorporated by reference as if set forth in their entirety. The embodiments described herein can be further configured as described in these patents. For example, one or more of the computer subsystems described herein may be further configured as described in these patents. Additionally, configuring one or more virtual systems as a central computing and storage (CCS) system may be performed as described in the Duffy patent cited above. The persistent storage mechanisms described herein may have decentralized computation and storage, such as a CCS architecture, but the embodiments described herein are not limited to this architecture. As mentioned further above, the inspection subsystem can be configured to use multiple modes to generate the output of the samples. In general, a "mode" can be defined by the parameter values of the inspection subsystem used to generate the output of a sample. Thus, different modes may have different values for at least one of the imaging parameters of the inspection subsystem. For example, in one embodiment of an optical-based inspection subsystem, at least one of the plurality of modes uses at least one wavelength of illumination light that is different from at least one wavelength of illumination light used for at least another of the plurality of modes . Modes may have different illumination wavelengths for different modes, as described further herein (eg, by using different light sources, different spectral filters, etc.). In another embodiment, at least one of the modes uses an illumination channel of the inspection subsystem that is different from an illumination channel of the inspection subsystem used by at least another of the modes. For example, as mentioned above, the inspection subsystem may include more than one illumination channel. Thus, different illumination channels can be used for different modes. For example, the optical and electron beam subsystems described herein can be configured as inspection subsystems. However, the optical and e-beam subsystems described herein can be configured as other types of tools, such as defect inspection subsystems. In particular, the embodiments of the inspection subsystem described herein and shown in FIGS. 1 and 2 may modify one or more parameters to provide different imaging capabilities depending on the application in which it will be used. In one such example, if the inspection subsystem shown in FIG. 2 is used for defect inspection rather than inspection, it can be configured to have a higher resolution. In other words, the embodiments of inspection subsystems shown in FIGS. 1 and 2 describe some general and various configurations for an optical or e-beam subsystem that can be adapted in a number of ways apparent to those skilled in the art to produce a nearly suitable Different subsystems with different imaging capabilities for different applications. One or more computer subsystems may be configured for obtaining the output of samples generated by one of the inspection subsystems described herein. Obtaining output can be performed using one of the inspection subsystems described herein (eg, by directing light or an electron beam to the sample and detecting the light or an electron beam, respectively, from the sample). In this way, the acquisition output can be performed using the physical sample itself and some imaging hardware. However, acquiring output does not necessarily include imaging the sample using imaging hardware. For example, another system and/or method may generate output and may store the generated output in one or more of the storage media described herein, such as a virtual detection system, or another storage medium described herein. Thus, obtaining the output may include obtaining the output from a storage medium in which the output has been stored. In one embodiment, the inspection correlation algorithm is a defect classifier. For example, an algorithm may classify defects detected on a sample into different types or classes of defects. The defect classifier may have any suitable configuration such as a decision tree or a nearest neighbor type configuration. In another embodiment, the inspection correlation algorithm is a defect filter. The defect filter can be configured as a nuisance filter in that it can be configured to separate actual defects from nuisances (which can be defined as further described herein) and other noise, and then eliminated from inspection results (and thereby filter out) nuisances and noise. The defect filter may also have any suitable configuration such as a decision tree or a nearest neighbor type configuration. In an additional embodiment, the inspection correlation algorithm is a defect detection algorithm. The defect detection algorithm may be configured to perform defect detection as further described herein and/or in any other suitable manner known in the art. In an additional embodiment, the checking correlation algorithm is a machine learning algorithm. The inspection-related algorithms described herein can be configured as machine learning algorithms. For example, defect classifiers, defect filters, and defect detection algorithms may have machine learning algorithm configurations. Additionally, the machine learning algorithm can be configured as described in US Patent Application Publication No. 2017/0148226, Zhang et al., published May 25, 2017, Zhang et al., June 2017 No. 2017/0193680 issued on 6th, No. 2017/0194126 issued by Bhaskar et al. on June 6, 2017, No. 2017/0200260 issued by Bhaskar et al. on July 13, 2017 and Bhaskar 2017/0200265, published Jul. 13, 2017, and US Patent Application Ser. No. 15/603,249, Zhang et al., filed May 23, 2017, which are incorporated by reference as if set forth in their entirety Incorporated herein. The inspection-related algorithms described herein can have any of the configurations described in these publications. One or more computer subsystems are configured to perform an initial training of an inspection correlation algorithm using a marked defect set, thereby generating an initial version of the inspection correlation algorithm. In some embodiments, the computer subsystem(s) may be configured to generate a set of marked defects for performing initial training. For example, as shown in FIG. 3 , the computer subsystem(s) may select a first batch of defects as shown in step 300 . The first batch of defects can be selected as described further herein. Additionally, the computer subsystem(s) may classify the selected defects as shown in step 302 . (While FIG. 3 describes the steps with respect to a defect classifier, the steps shown in FIG. 3 and described herein may be performed for one of the different inspection-related algorithms described herein.) Computer Subsystem(s) The selected defects can be classified and/or a classification of the selected defects can be obtained as further described herein. Next, the computer subsystem(s) may train the classifier as shown in step 304 . Thus, the training performed in step 304 may be the initial training described herein. Initial training can be performed in any suitable manner known in the art. For example, information on the defects, such as attributes and/or images (or other detector outputs), can be input to a defect classifier, which can then classify and mark the defects. Next, one or more parameters of the defect classifier may be modified until the classifications produced by the defect classifier for the defects match the labels assigned to the defects. While the defects can be flagged as described herein, defect attributes and defect blocks (eg, optical attributes and/or optical blocks) can be used as input data for checking related algorithms. The computer subsystem(s) are also further configured to apply the initial version of the inspection-related algorithm to an unmarked defect group. For example, once an inspection-related algorithm is initially trained using marked defects, an initial version of the inspection-related algorithm can be applied to unmarked remaining defects (and latent defects) detected by inspection of a specimen (on a wafer One thermal inspection, which can contain thousands of defects). In this way, as described above, although the defects may be marked as described herein, the attribute(s) and/or block images or other detector outputs are input to the inspection correlation algorithm for initial train. After initial training on the marked set (eg, using the defect attribute(s) and/or block or other detector output), an initial version of the inspection correlation algorithm may be applied to the unmarked defect set. The initial version of applying the inspection correlation algorithm may be performed by inputting all or (some) of the information available for the unmarked defect group into the inspection correlation algorithm. Unmarked defect groups may be configured as described further herein. The computer subsystem(s) are further configured to alter the set of marked defects based on the results of the application. For example, when an initial version of an inspection correlation algorithm is applied to unmarked defects, the inspection correlation algorithm may output not only results for each of the unmarked defects (eg, a defect classification), but also its decisions (eg, about the classification) reliability. This confidence can then be used in the next iteration of the defect selection procedure. Defects selected in the defect selection process can be marked as further described herein, and then added to the marked defect group, thereby altering the marked defect group. Altering the marked defect group may be performed as described further herein. In one embodiment, the marked defect group and the unmarked defect group may be included in the same inspection result. For example, as described further herein, a set of marked defects and a set of unmarked defects may be generated by scanning one or more samples. This scan can be performed as a thermal scan to thereby capture as many defects or defect types as possible. When a scan includes a thermal scan, due to the amount of defects detected by this scan, only one thermal scan of a single sample can produce enough defects for all of the steps described herein. Some of the defects detected by this scan may be marked as described herein to thereby generate a set of marked defects (ie, a training set of defects). An unmarked defect group may be the remaining defects detected by this scan as an unmarked defect group. Thus, the owner of one or more thermal scan detected defects may form all of the defects used by the embodiments described herein, some of which are flagged and used for one or more of the defects described herein steps, and others are unlabeled and used for one or more of the other steps described herein. In another embodiment, altering the marked defect group includes marking one or more of the defects in the unmarked group and adding the one or more of the marked defects to the marked group. For example, one or more of the selection defects in the unmarked set can be selected as described herein, and the one or more defects can then be marked in any suitable manner. In one such example, the one or more selected defects may be imaged by an image acquisition subsystem having a resolution higher than the resolution of the inspection subsystem to thereby generate one of the one or more defects. Higher resolution images. The higher resolution defect images can then be provided to a user who assigns markings to one of the defects. However, as described further herein, selected defects can be flagged by an automatic defect classifier (ADC). Therefore, these higher resolution defect images can also be provided to the user or ADCs operating on these higher resolution images. The indicia assigned by the user may include any of the indicia described herein (such as defects, nuisances, noise, defect classification codes, etc.). The flags assigned by the user may vary depending on the configuration of the check-related algorithm. In some instances, the computer subsystem(s) may provide the user with several possible indicia (eg, defect, non-defect, defect level code x, defect level code y, etc.). Additionally, the computer subsystem(s) may allow a user to enter a new flag, such as a new defect level code, which may then be used to modify the configuration of the inspection correlation algorithm (eg, when an inspection correlation algorithm is when a new defect marker creates a new node, storage area, definition, etc.). One or more of the flagged defects may be added to the defect flag set in any manner (eg, by appending the information of the newly flagged defect to a file or other data structure in which the information of the previously flagged defect is stored) . As further described herein, in one embodiment, one or more computer subsystems are configured to detect defects on a sample based on output generated by a detector, and the detected defects on the sample Contains marked defect groups and unmarked defect groups. For example, the defects used by the computer subsystem(s) described herein may all be detected on a sample or samples by performing thermal scan(s) on the sample(s). In particular, for inspections such as optical inspections, the results of thermal scans are often used to train nuisance filters and other inspection-related algorithms (ie, sample inspections that produce results containing thousands of defects). A "thermal scan" can generally be defined as an inspection performed on a sample in which thresholds for detection of potential defects and defects are intentionally set at or substantially close to the noise floor of the output produced by the scan place. A "thermal scan" is typically performed to detect as many potential defects and defects as possible to ensure that most or all of the defects of interest are captured for inspection program settings and the like. Therefore, the thermal scan results can be used to train nuisance filters and other inspection related algorithms. To train an inspection correlation algorithm such as a nuisance filter or defect classifier, a relatively small subset of defects detected on a sample can be marked. By marking it is meant to "classify" such defects. "Classification" of these defects may vary depending on the inspection-related algorithm(s) trained or generated by the computer subsystem(s). For example, if the inspection correlation algorithm is a defect detection algorithm, classification may involve marking detected defects as actual defects and non-actual defects (eg, noise). In another example, if the inspection-related algorithm is a nuisance filter, the classification may involve marking detected defects as actual defects and nuisance defects (which may be generally defined as noise that the user does not actually care about and / or actual defects). In a further example, if the inspection correlation algorithm is a defect classifier, the classification may involve using defect IDs (eg, indicating different types of defects (such as bridges, particles, scratches, missing features, roughness, etc.) grade code) to mark detected defects. Such defect classification or marking may generally involve first obtaining a substantially higher resolution image of the defects. These high-resolution images can be generated using a SEM or high-resolution optical imaging. In one embodiment, the set of marked defects used for initial training includes a predetermined minimum number of defects selected from all defects detected on the sample. For example, as further described herein, one of the advantages of embodiments is that labeling defects in the training set can be minimized without sacrificing the quality of the trained inspection correlation algorithm. Thus, the predetermined minimum number of flagged defects for initial training may be the minimum number of defects required to generate a roughly trained initial version of the inspection correlation algorithm. The minimum number of flagged defects may be predetermined heuristically or based on past experience and knowledge (eg, as to how many flagged defects are required to train an inspection correlation algorithm). Additionally, the predetermined minimum number of marked defects may vary depending on the inspection-related algorithm. For example, for a defect classifier, the predetermined minimum number of flagged defects may be a small number (eg, 2 or 3) of defects of each defect type expected on the sample and/or for which the classifier is configured. For a different inspection-related algorithm, such as a defect detection algorithm or a nuisance filter, the predetermined minimum number of flagged defects may be many or dozens of defects and non-defects (eg, 10 to 50 of each). The predetermined minimum number of defects may be randomly selected from defects usable in the embodiments described herein and/or defects detected on the sample (eg, unmarked defects in thermal scan results). These randomly selected defects can then be marked as described herein. Next, the flagging defects can be analyzed to determine whether the predetermined minimum number of flagging defects is sufficient for initial training. If sufficient defects of a particular type are not selected and marked, the steps described above can be repeated until the sample of marked defects contains the desired number of the desired marked defects. Embodiments described herein provide an iterative approach for finding defects near the boundaries of the potential distribution. In addition, the embodiments described herein combine training set selection and defect marking and adjustment procedures by having an inspection correlation algorithm drive the selection procedure (which is believed to be a novel idea particularly suitable for optical inspection). For example, in a further embodiment, altering the set of flagged defects includes determining the certainty of results produced for defects in the unflagged set by applying an initial version of an inspection-related algorithm, selecting the set with the lowest certainty in the unflagged set defects, get the flag of the selected defect, and add the flag of the selected defect and its etc. to the flag defect group. For example, as shown in FIG. 3 , the computer subsystem(s) may be configured to calculate the uncertainty of the model (ie, check the correlation algorithm) for each defect as shown in step 306 . Additionally, the computer subsystem(s) may be configured to find a new group of defects with the lowest certainty in the test data as shown in step 308 . The computer subsystem(s) may be further configured to classify the new group as shown in step 310 . The computer subsystem(s) may also be configured to add this new group to the training group as shown in step 312 . In this manner, in these steps, after initially training the inspection correlation algorithm using a substantially smaller set of marked defects, the certainty of the inspection correlation algorithm (eg, classifier) with respect to each defect can be measured. Certainty may be determined in any suitable manner. For example, an inspection correlation algorithm can be configured to generate a confidence associated with each result it produces (eg, a confidence associated with each defect classification). This confidence can be used to determine certainty. The check correlation algorithm can also be configured to automatically generate a certainty for each result produced by the check correlation algorithm. Therefore, the defect group for which the inspection-related algorithm is most uncertain is selected and marked. Defect marking (classification) for the training set can be done manually using optical images (eg blocks) or SEM images. Labeling can also be performed automatically using a pre-trained SEM automatic defect classifier (ADC). For reliable SEM ADCs, this approach will fully automate the training process and further accelerate the protocol tuning process on top of the main construction ideas described herein. This new batch of marked defects is added to the previously marked defects and used to retrain (or correct) the inspection correlation algorithm. Each of these steps may be performed as further described herein. In one such embodiment, selecting the defect with the lowest certainty in the unmarked group includes selecting a predetermined minimum number of defects in the unmarked group with one of the lowest certainty. For example, the defects in the unmarked set may be selected from the defect with the lowest certainty to the defect with the second lowest certainty and so on until the predetermined minimum number has been reached. The reservation of the defects in the selected unlabeled set may be predetermined as described herein (eg, heuristically or based on previous experiments and history to determine the minimum number of defects required to achieve adequate training of the inspection correlation algorithm). Minimum number. In another such embodiment, selecting the defect with the lowest certainty in the unlabeled set is performed independently of the diversity of one or more characteristics of the defect in the unlabeled set. For example, the embodiments described herein may be based on examining correlation algorithms without considering the diversity of a first characteristic of a defect, a diversity of a second characteristic of a defect, or the diversity of any other characteristic of a defect Defects are selected based on the uncertainty of the assigned marks. In this way, selection of defects based on the uncertainty of examining the results produced by correlation algorithms for such defects differs from diversity sampling. In addition, selecting the defects with the lowest certainty in the unmarked set may be performed without considering any other attributes or information about the defects. However, when the inspection correlation algorithm is configured to assign different marks to different previously unmarked defects, the defect with the least certainty may include the defect assigned a first mark with the lowest certainty, the defect assigned a second Defects marked with minimum certainty. In other words, selecting the defect with the lowest certainty in the unmarked set without considering the diversity of one or more characteristics of the defect may be performed based on (or relying on) the tags assigned by the inspection correlation algorithm. However, the selection has not been performed based on the diversity of any one or more of the characteristics of the defects themselves. For example, a defect that is assigned a different label and has the lowest certainty need not have relatively diverse values for any characteristic of the defect. In fact, it is the similarity, not the diversity, of any one of the characteristics of the defects that makes it difficult to flag defects in an initial, preliminary, or intermediate version of an inspection-related algorithm. In some embodiments, altering the set of flagged defects includes determining the certainty of results produced for defects in the unflagged set by applying an initial version of an inspection-related algorithm, selecting the defect in the unflagged set with the lowest certainty group, select a defective subgroup in the group with the greatest diversity of one of the characteristics of the defects in the subgroup, obtain the flag of the defective subgroup and add the flag of the selected defective subgroup and the like to Mark the defect group. For example, embodiments described herein may combine uncertainty with diversity to make sampling more efficient. The first priority is to sample the defects for which the relevant algorithm is least confident, since these are known to be at the classification boundary, and providing a fact of these defects will best improve the quality of the inspection-related algorithm. However, when there are many "low-confidence" defects, try to ensure that the computer subsystem(s) do not choose defects that will all have the same confidence that appear to be essentially the same, but instead choose between many different low-confidence defects The deficit of diversification can be beneficial. In this way, the computer subsystem(s) can select the most diverse group around the classification boundary, as opposed to only selecting defects that are located in only one part of the boundary. (In principle, the classification boundaries are complex, unknown, and hyperplanes that can be hyperplanes in a multi-dimensional space, and which are used to obtain a sufficiently trained inspection correlation algorithm with a minimal number of labeled defects, the computer subsystem(s) compared to Defects are preferably carefully selected around the entire boundary. In other words, the computer subsystem(s) preferably do not choose to be relatively far from the classification boundary (i.e., with relatively high confidence) or in the same portion of the boundary (i.e., not significantly Diverse) defects.) (several) computer subsystems are also configured to retrain the inspection correlation algorithm using the changed set of flagged defects, thereby producing a newer version of the inspection correlation algorithm. For example, as shown in FIG. 3 , the computer subsystem(s) may be configured to retrain (or calibrate) the classifier as shown in step 314 . Retraining may be performed relative to initial training as described further herein. However, in the retraining step, retraining may begin by examining the most previous version of the correlation algorithm (eg, by examining the parameters of the correlation algorithm produced by the initial training) or by examining the first version of the correlation algorithm (eg, with the initial Check related algorithm version before training parameters). In general, when retraining a classifier after labeling a new batch of defects and adding them to the training set, in most cases retraining starts from scratch, although retraining can be started from a previous version of the classifier. (While either possibility can be performed, only each new training set is used to train a new classifier.) In this way, retraining can involve training the inspection correlation algorithm essentially from scratch using an initial pre-training version of one of the inspection correlation algorithms method, or retraining to check the previous version of the relevant algorithm by adjusting and possibly fine-tuning one or more parameters of the previous version. Additionally, the computer subsystem(s) are configured to apply a newer version of the inspection-related algorithm to another unmarked defect group. The set of unmarked defects applying newer versions of inspection-related algorithms may include any and/or owners of the remaining unmarked defects detected in the embodiments described herein and/or on samples or samples . In this way, the unmarked set applying the newer version differs from the unmarked set applying the original version (or a previous version) because one or more defects in the unmarked set were selected, marked and added to the Mark the defect group. Thus, the set of unmarked defects applying a newer version may contain fewer defects than the set of unmarked defects applying the original (or a previous) version. However, in some instances, if the number of remaining unmarked defects is not large enough, additional unmarked defects may be used to augment the set of unmarked defects remaining after some have been selected, marked, and the like added to the marked set . Amplifying the unlabeled set may be performed in any suitable manner, such as performing another thermal scan on another sample and/or obtaining additional examination results from a storage medium, virtual system, etc. In general, the scans described herein will provide sufficient unmarked defects for the functions/steps described herein. Thus, if there are no such defects sufficient to thereby increase the number of defects, the more commonly performed amplification will be the amplification of the marker set. Newer versions of inspection-related algorithms can be applied to other unmarked defect groups as described herein. For example, the information for all or at least some of the defects in the other unmarked groups can be input to check the latest version of the relevant algorithm, which will then generate the latest version for each or at least some of the unmarked defects in the group result. The computer subsystem(s) are also configured to determine one or more differences between the results of applying a newer version of the inspection-related algorithm and the results of applying an initial version or an older version of the inspection-related algorithm. The initial version of the check correlation algorithm will be used to determine the difference only if the newer version is the second version of the check correlation algorithm produced (the version produced immediately after the initial version). In all other instances, the older version of the check correlation algorithm used to determine the difference in this step may be the check correlation algorithm generated immediately before the newer version. In this way, a determination can be made to examine the difference between the latest generated version of the relevant algorithm and the version generated immediately before that version. In other words, in this step, the difference between the nth version of the check correlation algorithm and the n-1 th version of the check correlation algorithm can be determined. This difference is then used to determine whether to converge the program as further described herein. For example, as further described herein, the program executed by the computer subsystem(s) may be determined to converge when the change in classification (or other outcome) between iterations becomes relatively small. Due to statistical fluctuations in the training procedure, these changes may not be strictly zero. In other words, when training with the same training set multiple times, the exact same classification (or other result) may not be produced for the same defect. These small fluctuations can be estimated, and when the variation between iterations becomes as small as this estimated value, the program executed by the computer subsystem(s) can be stopped - it has converged. Furthermore, when this criterion has been met, it is checked that the relevant algorithm has achieved its maximum performance. The computer subsystem(s) are further configured to repeatedly change the marked defect set, retrain the inspection-related algorithm, apply a newer version of the inspection-related algorithm, and determine the one or more differences until the one or more differences meet one or more criteria. Thus, the one or more criteria define stopping criteria for terminating iterations of flagging defects and other steps described herein. For example, as described above, one or more differences may be determined when they are equal to or less than an estimate of the relatively small fluctuations that will occur between training sessions (regardless of checking the performance of the associated algorithm) meet the one or more criteria. In addition, the different results produced by the checking related algorithms may have different criteria. For example, one or more criteria for the difference of the results produced by one defect classification may not be used for the one or more criteria of the difference of the results produced by another defect classification. In such instances, the steps described above may be repeated until the owner of the one or more criteria has been met. In other instances, owners who examine different results produced by related algorithms may have the same criteria. For example, one or more criteria for the difference between the results produced by different defect classifications may be the same. However, in these instances, the steps described above may also be repeated until the owner of the one or more criteria has been met. For example, even if two defect classifications are required to meet the same one or more criteria, the results generated for one defect classification may meet the one or more criteria faster than the results generated for the other defect. In one such example, as shown in FIG. 3 , the computer subsystem(s) may be configured to determine whether convergence criteria have been met as shown in step 316 . If the convergence criteria are not met, then as shown in FIG. 3, the computer subsystem(s) may return to step 306 and compute the model (check the associated algorithm) for the uncertainty for each defect. The computer subsystem(s) may also repeat steps 308, 310, 312, and 314 shown in FIG. 3 until it is determined that the convergence criterion is met. The reliance on data-driven convergence criteria for the embodiments described herein is believed to be novel. In other words, as further described herein, a batch of unmarked defects may be selected for inspection for which the relevant algorithm (eg, a classifier) is least confident. Next, the selected defects can be marked as described herein. Newly marked defects can be added to the training set, and the altered training set can be used to train a new inspection-related algorithm. These steps can be repeated until convergence has been satisfied. In one embodiment, the one or more criteria define a boundary between: a) indicating that a newer version of an inspection-related algorithm is negligibly different from an initial or older version of an inspection-related algorithm One or more differences and b) one or more differences indicating that the newer inspection correlation algorithm is significantly different from the initial or older version of the inspection correlation algorithm. The one or more differences are differences determined as described above (eg, the difference between the nth version of the check correlation algorithm and the n-1 th version of the check correlation algorithm). In this way, the computer subsystem(s) can track the history of checking the correlation algorithm after each iteration, and if the change in the results produced by the checking correlation algorithm is small enough, it will terminate the iteration. The term "negligibly different" as used herein may vary by examining the relevant algorithm. However, "negligibly different" as used herein can be defined as being small enough to indicate that from checking one version of the correlation algorithm to another version, checking the correlation algorithm does not change significantly. Thus, the difference(s) that can be said to be "negligibly different" define the stopping criteria for the embodiments described herein. Thus, the "negligibly different" value of the one or more differences may be predetermined and defined by a user (based on their acceptable stopping criteria), and/or by the computer subsystem(s) or another method Or the system pre-orders based on general or specific information about the repeatability of the particular inspection-related algorithm being trained and/or the type of inspection-related algorithm being trained. As used herein, the term "significantly different" one or more differences may be any and all differences other than the "negligibly different" value of such differences. In this way, the one or more differences can have two different ranges of values: 1) differences that are "negligibly different" as defined herein; and 2) differences that are "significantly different" (except " all differences except negligibly different"). If the change from the previous iteration to the current iteration is zero (or very small), then the computer subsystem(s) determine that there are no new defects worth flagging because the inspection correlation algorithm is so certain about the defect. In a particular example, the computer subsystem(s) may use the history of changes in the predicted level codes for defects in the last test data set. However, several other measures of convergence are contemplated for use in the embodiments described herein. The owner of the convergence measure can monitor some aspect of the classifier performance and/or the content of the training set that varies depending on the training iteration. For example, the computer subsystem(s) can monitor and check the performance of the associated algorithm itself by tracking the accuracy that changes from iteration to iteration. Another approach relies on monitoring the improvement of the receiver operating curve (ROC) as a function of iteration. An ROC is basically a measure of the performance of a binary classifier across the entire range of operating points (eg, different nuisance rates). In addition, under certain circumstances or for some specific purposes, the computer subsystem(s) can monitor how different defect types become a training set through iterations, for example, the computer subsystem(s) can be used when the computer subsystem is no longer Stop when making the defect of interest (DOI) a training set. When the one or more differences meet the one or more criteria, the subsystem(s) are configured to output an up-to-date version of the check correlation algorithm as a trained check correlation algorithm for checking other samples. Outputting the latest version of the check-related algorithm may include (if necessary) outputting the latest training parameters of the check-related algorithm, which may have the general configuration of the check-related algorithm. Outputting the latest version of the inspection-related algorithm may also include storing the latest version in a storage medium (such as one of the storage media described herein) and/or in a inspection scheme such that the inspection scheme is executed when the inspection scheme is executed. Check related algorithms. (The term "scheme" as used herein may generally be defined as a set of instructions that may be used by a system to execute a program.) In one embodiment, one or more computer subsystems are configured to determine A separability measure of the different results produced by the latest version of the relevant algorithm is checked, and output is performed only after the determined separability measure is above a predetermined threshold. For example, applying an inspection-related algorithm (eg, a nuisance filter (classifier)) to having a variety of data between data corresponding to different things (eg, defect vs. degree of separability data. When the separation of the data is moderate or poor, the check-correlation algorithm is preferably not performed, and no matter what has been done, it will usually be left with a large number of relatively low-confidence results. Therefore, the convergence criteria are not based on any measure of reliability or performance. Thus, the computer subsystem(s) may only monitor when the inspection correlation algorithm stops improving, and at this point the best inspection correlation algorithm for this data has been generated. Thus, a separability measure of the latest version of the check correlation algorithm can be determined to determine whether the best check correlation algorithm that has been generated using the available training data actually performs well enough for other samples. If it is determined that the separability measure is insufficient, other options, such as other data generated using other output generation parameters of the inspection subsystem, may be explored as alternative inputs to inspection-related algorithms as described further herein. In one such example, as shown in FIG. 3 , once it is determined in step 316 that the convergence criteria have been met, the computer subsystem(s) may determine whether the data is separable as shown in step 318 . If it is determined in step 318 that the data is separable, then in step 320 the computer subsystem(s) may determine that the inspection-related algorithm is ready (ie, ready for inspection of other samples, ready for production) monitoring, etc.). In this way, the embodiments can use a measure to ensure checking the correctness of the associated algorithm. To ensure that the checking algorithm can separate the data correctly, the separability of the data can be measured. This measurement indicates whether the data is separable. For checking the correlation algorithm for one of the defect classifiers, measurements can indicate whether the data is separable and whether the classifier can classify each defect level better than a random guess. If the data in the training set are separable, a correct classifier can be declared to have been constructed. If the accuracy of the classifier for each class code is above a certain threshold (for example, some value above 50% (this is because an accuracy of 50% for a balanced training set implies completely random classification, i.e., no Separability)), the data in the training group can be considered separable. If the computer subsystem(s) determine in step 318 of FIG. 3 that the data are not separable, then as shown in FIG. 3 , the computer subsystem(s) may change the inspection parameters as shown in step 322 . For example, if the data is not separable, in the case of a defect classifier, the data is not classifiable. In this case, the computer subsystem(s) may decide that one or more parameters of the inspection subsystem should be changed. For example, the computer subsystem(s) may decide that the inspection mode should be changed. Then, the computer subsystem(s) may execute to adjust one or more parameters of the inspection subsystem or simply provide an instruction to another subsystem (computer or other) performing the adjustment. Adjustment or change of one or more parameters of the inspection subsystem may be performed in any suitable manner. The output generated using the adjusted or altered parameters of the inspection subsystem can then be used to generate a set of marked defects and a set of unmarked defects, which, in turn, can be used to perform the step(s) described herein to generate a trained check correlation algorithm. In this way, an inspection correlation algorithm can be generated for the new parameter training of the inspection subsystem. Embodiments described herein provide several advantages for training-check correlation algorithms. For example, combining inspection-related algorithm tuning and training set acquisition into a single method would provide a significant advantage over existing methods because the inspection-related algorithm tuning and training set acquisition described herein will maximize labeling defects for Check the validity of the performance of the relevant algorithms. (Labeling defects is the most instructive defect for training purposes, and therefore it is always best to check the performance of the associated algorithm for a given data.) In addition, in terms of tool time and labor, labeling defects (eg, manual sorting defects) are substantially expensive. Identifying the training set acquisition and checking the convergence criteria for the associated algorithm tuning procedure will minimize the training set size and thus provide an advantage. Furthermore, the combination of training set selection and defect labeling and tuning procedures is a novel realization that is absolutely necessary to apply any machine learning algorithm for optical inspection nuisance filters and classifiers. (The need to combine training set selection and defect marking with the adjustment procedure is because the training data has thousands of defects, most of which are a hindrance.) The embodiments described herein also ensure that the inspection scheme is consistent, i.e. , nuisance filter adjustment no longer depends on experience and skill. Each of the embodiments of the systems described herein may be combined with any other embodiments of the systems described herein. Another embodiment relates to a computer-implemented method for training an inspection correlation algorithm. The method comprises the steps of each of the functions of the computer subsystem(s) described above. In particular, the method includes performing an initial training of an inspection correlation algorithm using a set of marked defects, thereby generating an initial version of the inspection correlation algorithm. The method also includes applying the initial version of the inspection correlation algorithm to an unmarked defect group and altering the marked defect group based on the results of the application. Additionally, the method includes retraining the inspection correlation algorithm using the changed set of flagged defects, thereby generating a newer version of the inspection correlation algorithm. The method further includes applying a newer version of the inspection-related algorithm to another set of unmarked defects. The method also includes determining one or more differences between the results of applying the inspection of the newer version of the relevant algorithm and the results of applying the inspection of the initial version or an older version of the relevant algorithm. Additionally, the method includes repeatedly changing the marked defect group, retraining the inspection-related algorithm, applying a newer version of the inspection-related algorithm, and determining the one or more differences until the one or more differences meet one or more criteria . When the one or more differences meet the one or more criteria, the method includes outputting an up-to-date version of the check correlation algorithm as a trained check correlation algorithm for checking other samples. Each of the steps of the method can be performed as described further herein. The method may also include any other step(s) that may be performed by the inspection subsystem(s) and/or the computer subsystem(s) and/or the system(s) described herein. The steps of the method may be performed by one or more computer systems configured according to any of the embodiments described herein. Additionally, the methods described above can be performed by any of the system embodiments described herein. An additional embodiment relates to a non-transitory computer-readable medium storing program instructions executed on a computer system for performing a computer-implemented method of training an inspection correlation algorithm. One such embodiment is shown in FIG. 4 . In particular, as shown in FIG. 4 , non-transitory computer-readable medium 400 includes program instructions 402 executable on computer system 404 . The computer-implemented method may comprise any step(s) of any method(s) described herein. Program instructions 402 implementing methods such as those described herein may be stored on computer-readable medium 400 . The computer-readable medium may be a storage medium, such as a magnetic or optical disk, a magnetic tape, or any other suitable non-transitory computer-readable medium known in the art. Program instructions may be implemented in any of a variety of ways, including program-based techniques, component-based techniques, and/or object-oriented techniques, among others. For example, program instructions may be implemented using ActiveX controls, C++ objects, JavaBeans, Microsoft Foundation Classes ("MFC"), SSE (Streaming SIMD Extensions), or other technologies or methods as desired. Computer system 404 may be configured according to any of the embodiments described herein. The owner of the methods described herein may include storing the results of one or more steps of an embodiment of the method in a computer-readable storage medium. The results can include any of the results described herein and can be stored in any manner known in the art. The storage medium may include any storage medium described herein or any other suitable storage medium known in the art. After the results are stored, the results can be accessed in a storage medium and used by any of the method or system embodiments described herein, can be formatted for display to a user, can be used by another software Modules, methods or systems, etc. are used. For example, a trained inspection correlation algorithm may be used to perform inspection(s) on other sample(s) (the inspections may be performed as described herein). The results from the examination(s) may be used to perform one or more functions of the other sample(s) or to perform the procedures used to form the other sample(s). For example, results from inspection(s) performed using inspection-related algorithms trained as described herein may be used to alter one or more parameters of one or more procedures used to form other sample(s). Additionally or alternatively, results from inspections performed using inspection-related algorithms trained as described herein may be used to alter one or more parameters of one or more procedures to be performed at ( to form additional features or materials on the other sample(s) or to correct defects on the other sample(s), thereby altering the other sample(s) themselves. In view of this description, further modifications and alternative embodiments of various aspects of the invention will be apparent to those skilled in the art. For example, the present invention provides methods and systems for training an inspection correlation algorithm. Accordingly, this description is to be construed as illustrative only and for the purpose of teaching those skilled in the art of the general manner in which the invention may be practiced. It is to be understood that the forms of the invention shown and described herein are to be taken as presently preferred embodiments. As will be apparent to those skilled in the art having the benefit of this description of the invention, elements and materials may be substituted for those shown and described herein, parts and procedures may be reversed and the features of the invention may be utilized independently certain characteristics. Changes may be made in elements described herein without departing from the spirit and scope of the invention as described in the following claims.

10‧‧‧Inspection Subsystem 14‧‧‧Sample 16‧‧‧Light Source 18‧‧‧Optics 20‧‧‧Lens 21‧‧‧Beam Splitter 22‧‧‧Stand24‧‧‧Concentrator 26‧ ‧‧Component 28‧‧‧Detector 30‧‧‧Light collector 32‧‧‧Component 34‧‧‧Detector 36‧‧‧Computer subsystem 102‧‧‧Computer subsystem 122‧‧‧Electronic column 124 ‧‧‧Computer Subsystem 126‧‧‧Electron Beam Source 128‧‧‧Sample 130‧‧‧Component 132‧‧‧Component 134‧‧‧Detector 300‧‧‧Step 302‧‧‧Step 304‧‧‧Step 306‧‧‧Step 308‧‧‧Step 310‧‧‧Step 312‧‧‧Step 314‧‧‧Step 316‧‧‧Step 318‧‧‧Step 320‧‧‧Step 322‧‧‧Step 400‧‧‧Not Transient computer readable media 402‧‧‧Program instructions 404‧‧‧Computer system

Other objects and advantages of the present invention will become apparent upon reading the following detailed description and upon reference to the accompanying drawings, in which: Figures 1 and 2 depict side views of embodiments of a system configured as described herein Schematic diagram; FIG. 3 is a flowchart showing one embodiment of the steps that may be performed by the embodiments described herein; FIG. 4 is stored on a computer system for execution of the computer-implemented method described herein. A block diagram of one embodiment of a non-transitory computer-readable medium of one or more program instructions. While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the drawings and detailed description are not intended to limit the invention to the particular form disclosed, but on the contrary, this invention covers all modifications within the spirit and scope of the invention as defined by the appended claims , equivalents and alternatives.

10‧‧‧Check subsystem

14‧‧‧Sample

16‧‧‧Light source

18‧‧‧Optics

20‧‧‧Lens

21‧‧‧Beam Splitter

22‧‧‧ Shelf

24‧‧‧Concentrator

26‧‧‧Components

28‧‧‧Detector

30‧‧‧Concentrator

32‧‧‧Components

34‧‧‧Detector

36‧‧‧Computer Subsystem

102‧‧‧Computer subsystem

Claims (20)

A system configured to train an inspection correlation algorithm, comprising: an inspection subsystem including at least one energy source and a detector, wherein the energy source is configured to generate energy directed to a sample, and wherein the detector is configured to detect energy from the sample and to generate an output in response to the detected energy; and one or more computer subsystems, etc. configured to: use a marker performing an initial training of an inspection-related algorithm on a defect group, thereby generating an initial version of the inspection-related algorithm; applying the initial version of the inspection-related algorithm to an unmarked defect group; based on the results of the application to change the set of flagged defects; retrain the inspection-related algorithm using the changed set of flagged defects, thereby generating a newer version of the inspection-related algorithm; apply the newer version of the inspection-related algorithm to Another set of unmarked defects; determine one or more differences between the results of applying the newer version of the inspection-related algorithm and the results of applying the initial version or an older version of the inspection-related algorithm ; repeatedly changing the marked defect group, retraining the inspection-related algorithm, applying the newer version of the inspection-related algorithm, and determining the one or more differences until the one or more differences meet one or more criteria; and when the one or more differences meet the one or more criteria, outputting a newest version of the inspection correlation algorithm as a trained inspection correlation algorithm for inspecting other samples. The system of claim 1, wherein the inspection correlation algorithm is a defect classifier. The system of claim 1, wherein the inspection correlation algorithm is a defect filter. The system of claim 1, wherein the inspection-related algorithm is a defect detection algorithm. The system of claim 1, wherein the checking correlation algorithm is a machine learning algorithm. The system of claim 1, wherein the set of marked defects and the set of unmarked defects are included in the same result. The system of claim 1, wherein changing the marked defect group includes marking one or more of the defects in the unmarked defect group, and adding the marked defect one or more to the marked defect group. The system of claim 1, wherein the one or more computer subsystems are further configured to detect defects on the sample based on the output generated by the detector, and wherein the detected defect on the sample The defects include the marked defect group and the unmarked defect group. The system of claim 1, wherein the set of marked defects includes a predetermined minimum number of defects selected from all defects detected on the sample. The system of claim 1, wherein changing the set of flagged defects includes applying the inspection phase by the application determine the certainty of the results for the defects in the unmarked defect group on the initial version of the algorithm, select the defects with the lowest certainty in the unmarked defect group, obtain the waits for the flags of the selected defects, and adds the flags of the selected defects and their equivalents to the flagged defect group. The system of claim 10, wherein selecting the defects in the unmarked defect group with the lowest certainty includes selecting a predetermined minimum number of the defects in the unmarked defect group with one of the lowest certainty. The system of claim 10, wherein selecting the defects in the set of unmarked defects with the lowest certainty is performed independently of the diversity of one or more characteristics of the defects in the set of unmarked defects. The system of claim 1, wherein altering the set of flagged defects includes determining certainty, selection of the results for the defects in the set of flagged defects by applying the initial version of the inspection-related algorithm In the unmarked defect group, a group of the defects with the lowest certainty is selected, a defect subgroup in the group with the greatest diversity as one of the characteristics of the defects is selected, and the mark of the defect subgroup is obtained, and adding the selected defect subgroup and its equivalent flags to the flagged defect group. The system of claim 1, wherein the one or more criteria define a boundary between: a) indicating that the newer version of the inspection-related algorithm is negligibly different from the initial version of the inspection-related algorithm version or the one or more differences of the older version, consistent with b) indicating the newer check The correlation algorithm is significantly different from the one or more differences of the initial version or the older version of the check correlation algorithm. The system of claim 1, wherein the one or more computer subsystems are further configured to determine a measure of separability of different results produced by the latest version of the inspection-related algorithm, and only if determined The output is performed after the separability measure is above a predetermined threshold. The system of claim 1, wherein the sample comprises a wafer. The system of claim 1, wherein the energy directed to the sample comprises light, and wherein the energy detected from the sample comprises light. The system of claim 1, wherein the energy directed to the sample comprises electrons, and wherein the energy detected from the sample comprises electrons. A non-transitory computer-readable medium storing program instructions executed on a computer system for executing a computer-implemented method of training an inspection correlation algorithm, wherein the computer-implemented method includes: executing a an initial training of an inspection-related algorithm, thereby generating an initial version of the inspection-related algorithm; applying the initial version of the inspection-related algorithm to an unmarked defect group; altering the marked defect based on the results of the application set; retrain the inspection-related algorithm using the changed set of flagged defects, thereby generating the Check for a newer version of one of the relevant algorithms; apply the newer version of the check-related algorithm to another unmarked defect group; determine that the results of applying the newer version of the check-related algorithm are relevant to the application of the check One or more differences between the results of the initial version of the algorithm or an older version; repeatedly changing the flagged defect set, retraining the inspection-related algorithm, applying the newer version of the inspection-related algorithm and determine the one or more differences until the one or more differences meet one or more standards; and when the one or more differences meet the one or more standards, output a latest version of the check-related algorithm as a training an inspection-related algorithm for inspecting other samples, wherein the initial training is performed, the initial version is applied, the marked defect group is changed, the inspection-related algorithm is retrained, the newer version is applied, the one or more differences are determined , the repetition and the output are performed by the computer system. A computer-implemented method for training an inspection-related algorithm, comprising: using a marked defect group to perform an initial training of an inspection-related algorithm, thereby generating an initial version of the inspection-related algorithm; the inspection applying the initial version of the correlation algorithm to a set of unmarked defects; altering the set of marked defects based on the results of the application; retraining the inspection correlation algorithm using the changed set of marked defects, thereby generating the inspection correlation algorithm a newer version of the inspection-related algorithm; apply the newer version of the inspection-related algorithm to another unmarked defect group; determine the result of applying the newer version of the inspection-related algorithm to the result of applying the inspection-related algorithm one or more differences between the results of the original version or an older version; repeating changing the marked defect group, retraining the inspection-related algorithm, applying the newer version of the inspection-related algorithm, and determining the one or more differences until the one or more differences meet one or more criteria; and when When the one or more differences meet the one or more criteria, output a newest version of the inspection-related algorithm as a trained inspection-related algorithm for inspecting other samples, wherein the initial training is performed, the initial version is applied, the change is made The marking of defect groups, retraining the inspection-related algorithm, applying the newer version, determining the one or more differences, the repetition, and the output are performed by one or more computer systems.

Images