JP6833027B2 - Optimization of training sets used to set up inspection-related algorithms - Google Patents

Description

The present invention relates to methods and systems for optimizing training sets used to set up inspection-related algorithms.

The following description and examples, albeit in this section, are not considered to be prior art.

The inspection process is used in various steps of the semiconductor manufacturing process, and by detecting defects on the wafer, it is possible to improve the yield in the manufacturing process and thus increase the profit. Inspection has always been an important part of semiconductor device manufacturing. However, as smaller defects can cause devices to fail, inspection has become even more important for the successful manufacture of acceptable semiconductor devices as the dimensions of semiconductor devices have shrunk.

When defects are detected on a sample, eg, a wafer, certain algorithms are often applied to the detected defects to separate them into several types of defects (or to separate the defects from those that are not defects). ). One of the execution methods is to apply a defect classifier to the detected defects to classify the detected defects into several defect types or classes. Defect classifiers are usually one or more of those defects and / or defect images (eg, relatively small images captured around the defect, commonly referred to as "patch" images or "patches"). The attribute is used as an input to determine the defect type or class. The defect classifier then assigns a certain identifier or ID to each defect to reveal its identified type or class. Another detection defect separation method is to separate actual defects from nuisance or noise. A "nuisance" defect is generally defined as a defect that the user does not care about and / or that is detected as a defect but is not actually a defect. Such algorithms are commonly known as defect filters and / or nuisance filters.

The most widely used classifier / nuisance filters on optical inspection tools rely on manually constructed decision trees. The method of tuning these decision trees can leverage the experience and domain knowledge incorporated into the Best Knowing Method for Tree Construction (BKM). This typically results in the initial construction of a decision tree using BKM "templates", defect clustering and substantially coarse defect labeling (using patches). Then, after the structure of the tree is obtained, the tree is sampled in various ways by using diversity sampling with a smart sample distribution over the leaf nodes of the tree. The sampled defects are then reviewed by a scanning electron microscope (SEM), classified, and used for final tuning of the determined cut line (heterogeneous defect separation boundary). Other machine learning algorithm-based classifiers (eg, nearest neighbor classifiers) also automatically find decision boundaries given a training set, but there is currently a way to obtain a training set that is likely to maximize their performance. Not.

U.S. Pat. No. 8664594 U.S. Pat. No. 8,692,204 U.S. Pat. No. 8698093 U.S. Pat. No. 87166662 U.S. Pat. No. 8126255 U.S. Pat. No. 9222895 U.S. Patent Application Publication No. 2017/0148226 U.S. Patent Application Publication No. 2017/01/93680 U.S. Patent Application Publication No. 2017/0194126 U.S. Patent Application Publication No. 2017/0200260 U.S. Patent Application Publication No. 2017/0200265

However, the defect classifier setup and tuning methods currently in use have some drawbacks. For example, existing methods are labor-intensive, require a high degree of expertise, and produce inconsistent results that depend on human experts. Building a classifier by a human expert is often error-prone, expensive, and time-consuming. Since each defect has a relatively large number of features, it is almost impossible to properly visualize those features for classification. Therefore, due to the lack of knowledge about the underlying multidimensional distribution, human experts can make serious mistakes when constructing classification boundaries. Even if there are no serious mistakes, the rate of manual generation of non-optimal classifiers is quite high.

Therefore, it is beneficial to develop a system and / or method for optimizing the training set used to set up the inspection-related algorithm, which does not have one or more of the above-mentioned disadvantages.

The following statements regarding the embodiments should not be construed as limiting the subject matter of the appended claims in any way.

One embodiment relates to a system configured to train inspection-related algorithms. The system comprises an inspection subsystem that has at least an energy source and a detector. The energy source is configured to generate energy towards the sample. The detector is configured to detect energy from the sample and generate an output in response to the detected energy. The system also includes one or more computer subsystems. The one or more computer subsystems are configured to generate an initial version of an inspection-related algorithm by performing initial training on the inspection-related algorithm with a labeled defect set. The computer subsystem (s) is also configured to apply the first edition of the inspection-related algorithm to the unlabeled defect set and modify the labeled defect set based on the result of the application. In addition, the computer subsystem (s) will generate a more recent version of the inspection-related algorithm by retraining the inspection-related algorithm with its modified labeled defect set. It is composed. In addition, the computer subsystem (s) is configured to apply updated inspection-related algorithms to another set of unlabeled defects. In addition, the computer subsystem (s) is one or more differences between the application of the updated inspection-related algorithm and the application of the first or less recent version of the inspection-related algorithm. Is configured to determine. The computer subsystem (s) also modifies labeled defect sets, retrains inspection-related algorithms, applies updated inspection-related algorithms, and discriminates one or more differences. It is configured to repeat until a plurality of differences meet one or more criteria. When the one or more differences meet the one or more criteria, the computer subsystem (group) uses the latest version of the inspection-related algorithm for inspection of other samples. It is configured to output as a trained inspection-related algorithm used. Further, the system can be configured as described in the present application.

Another embodiment relates to a computer implementation method for training inspection-related algorithms. The method has steps for each of the functions of the one or more computer subsystems described above. The steps of the method are performed by one or more computer systems. This method can be performed as detailed in the present application. In addition, the method may include any other step (group) of any of the other methods (groups) described in the present application. Further, the method can be performed on any of the systems described in the present application.

A further embodiment is a non-temporary computer-readable medium in which a program instruction is stored, in which the program instruction is executed on a computer system to execute a computer implementation method for training an inspection-related algorithm. Regarding. The computer implementation method has the steps of the method described above. Further, the computer-readable medium may be configured as described in the present application. The steps of the computer implementation method can be performed as detailed in this application. In addition, the computer implementation method is capable of executing program instructions and may include any other step (group) of any other method (group) described in the present application.

Other objects and advantages of the present invention will be clarified by referring to the description below and the accompanying drawings as described below.

It is a schematic diagram which depicts the aspect of the Embodiment of the system configured as described in this application. It is a schematic diagram which depicts the aspect of the Embodiment of the system configured as described in this application. It is a flowchart which depicts one Embodiment of the steps which can be performed by the various embodiments described in this application. FIG. 5 is a block diagram depicting an embodiment of a non-temporary computer-readable medium in which program instructions executed on a computer system to execute one or more of the computer implementation methods described in the present application are stored.

The present invention may include various modifications and alternatives, the specific embodiments thereof being exemplified in the drawings and detailed in the present application. However, as will be understood, the intent of the drawings and details about them is not to limit the invention to a particular form of disclosure, but rather, to the contrary, the intent is the scope of the claims. It is intended to cover all modifications, equivalents and alternatives that fall within the essence and technical scope of the invention as defined by.

It should be noted in the drawings that they are not drawn at equal scale. In particular, some scales of the elements in the figure are greatly exaggerated to emphasize the characteristics of those elements. Also, the figures are not drawn to the same scale. Elements shown in multiple drawings that may be similarly constructed are shown using the same reference numerals. Unless otherwise noted in the present application, any of the illustrated elements may include any suitable commercially available element.

One embodiment relates to a system configured to train inspection-related algorithms. Overall, according to the methods and systems provided by the embodiments described herein, the smallest training set is obtained to classify defects captured by optics or other tools and to serve other inspection-related functions. be able to. In addition, by making useful use of the embodiments described in the present application, the minimum set of the most teaching defects is found, thereby constructing a classifier and other inspection-related algorithms described in the present application, and using them as defect classification and other inspection-related defects described in the present application. It can be useful for the function.

The process of tuning a sample inspection (eg, optical wafer inspection) for optimum performance has traditionally been almost entirely manual. The tuning process generally relies on the best known method (BKM) and the experience and skill of the human experts who perform the tuning. Such an approach is not suitable for setting up a production monitoring system, as it is extremely costly (in terms of labor and man-hours) and the tuning outcomes are subjective and inconsistent. However, despite these obvious shortcomings of current inspection tuning methodologies, attempts to automate the process do not appear to have gained citizenship in the production environment. The main reason for this is that such automation relies on algorithms, which derive its performance from its basic training data (what is called a training set). Therefore, the performance of these algorithms becomes uncertain unless the training data is captured in a systematic manner. In other words, without finding training sets and optimizing the performance of those algorithms in a reliable way, such automation measures would suffer all the problems of the manual approach. Above all, such measures are inconsistent, and no matter how good the underlying algorithms are, their performance is not guaranteed to be consistent with that of the manual method. In addition, diagnosing performance problems and, if found, correcting them are often quite difficult, if not impossible. So it's no surprise that machine learning methods (these approaches are so known today) haven't been successful so far.

The comprehensive tuning methodologies provided by the embodiments described herein are for any machine learning algorithm and can be used for inspection-related functions such as classification and filtering (these embodiments are also for detection algorithm tuning). Although applicable, the embodiments described in the present application are particularly useful for nuisance filters and classifiers). The underlying perception of these embodiments is that in the case of inspection (eg, optical inspection), the training set capture technique can be beneficially fully integrated with the algorithm tuning itself. The two should be linked to each other so that they do not separate from each other, which can result in consistent behavior. The basic reasons for this interdependence are as follows.

Inspections, such as optical inspections, are tuned using hot scans (extremely incomplete scans with fairly high nuance rates). Tuning itself requires labeled defects (ie, classified defects, typically those classified by human experts). This classification is performed on scanning electron microscope (SEM) images captured by the SEM review tool. If all the defects detected by the hot scan can be reviewed and classified, the embodiments described in the present application will not be necessary. However, the review / classification process is quite expensive in terms of man-hours and tool time, making it virtually impossible to do. Therefore, it is imperative to identify a suitable subset of defects so that it can exhibit the optimum performance of classifiers and other inspection-related algorithms, and to find the minimum set to achieve it. Strongly desired.

In the methods and systems provided by the embodiments described in the present application, learning iterations optimize the selection of training sets for defects, and the learning iterations allow inspection-related algorithms (eg, classifier models) to learn data and perform themselves. Ask what is needed to improve. Further, in the methods and systems provided by the embodiments described in the present application, the time when the learning is reached is preferably determined.

In some embodiments, the sample includes a wafer. In another embodiment the sample contains a reticle. These wafers and reticle may include any wafer and reticle known in the art of the present application.

An embodiment of such a system is shown in FIG. The system has an inspection subsystem, which at least has an energy source and a detector. The energy source is configured to generate energy towards the sample. The detector is configured to detect energy from the sample and generate an output in response to the detected energy.

In certain embodiments, light-containing energy is directed toward the sample, and light-containing energy is detected from the sample. For example, in the system of the embodiment shown in FIG. 1, the inspection subsystem 10 has a lighting subsystem and is configured to direct light to the sample 14. The lighting subsystem has at least one light source. For example, the lighting subsystem shown in FIG. 1 has a light source 16. In certain embodiments, the lighting subsystem directs light to the sample, including one or more incident angles, such as one or more oblique angles and / or one or more orthogonal angles. It is composed. For example, as shown in FIG. 1, the light from the light source 16 is directed to the beam splitter 21 via the optical element 18 and further to the lens 20, and the light is directed to the sample 14 from there at an orthogonal incident angle. The angle of incidence may be any angle of incidence as desired, but may depend, for example, on the characteristics of the sample and the defects to be detected on the sample.

The illumination subsystem may be configured to direct light to the sample at different time points and at different angles of incidence. For example, by modifying the characteristics of one or more elements of the lighting subsystem, it is possible to direct the light to the sample at an incident angle different from that shown in FIG. Subsystems can be configured. As an example, the inspection subsystem can be configured so that the light source 16, the optical element 18, and the lens 20 can be moved to direct light to the sample at different angles of incidence.

In some cases, the inspection subsystem may be configured to direct light at the sample at multiple angles of incidence at the same time. For example, the lighting subsystem is provided with a plurality of lighting channels, one of which is provided with a light source 16, an optical element 18 and a lens 20 as shown in FIG. 1, and another of the lighting channels. One (not shown) may be provided with a group of similar elements, and these similar elements may be configured differently, similarly configured, or at least having a light source. It may have one or a plurality of other members, for example, those described in detail in the present application. When directing the light to the sample at the same time as other light, different incidents are caused by different characteristics (eg, wavelength, deflection, etc.) of the light directed to the sample at different angles of incidence. The light produced by the illumination of the sample at the corner can be discriminated from each other by the detector (group).

In some cases, the lighting subsystem has only one light source (eg, the light source 16 shown in FIG. 1), and one or more optical elements (not shown) provided in the lighting subsystem can be used. Light from a light source may be split into separate optical paths (eg, based on wavelength, deflection, etc.). Then, the light existing in each of the separate optical paths may be directed to the sample. A plurality of illumination channels may be configured to direct light toward the sample at the same time, or may be configured to direct light at different time points (eg, several illumination channels are used to sequentially illuminate the sample. When). In some cases, the same illumination channel may be configured to direct light to the sample at different times with different properties. For example, in some cases, the optical element 18 may be configured as a spectroscopic filter, and by changing the characteristics of the spectroscopic filter in various ways (eg, by replacing the spectroscopic filter), light of different wavelengths can be separated. At this point it can be directed to the sample. The lighting subsystem may be equipped with any other configuration known and suitable in the art to direct light with different or same properties in sequence or simultaneously at different or same angles of incidence.

According to certain embodiments, the light source 16 can have a broadband plasma (BBP) light source. As a result, the light generated by the light source and directed toward the sample can include wideband light. However, the light source may have some other suitable light source, such as a laser. The laser may include any laser known and suitable in the art of the present application, even if the laser is configured to produce light at any wavelength or group of wavelengths known and suitable in the art of the present application. Good. In addition, the laser may be configured to produce monochromatic or near monochromatic light. In this way, the laser can be a narrow band laser. Further, the light source may have a multicolor light source, whereby light may be generated at a plurality of discrete wavelengths or wave bands.

The light from the optical element 18 can be focused on the beam splitter 21 by the lens 20. Although the lens 20 is shown as a single refractive optical element in FIG. 1, in reality, the lens 20 has a plurality of refractive and / or reflective optical elements. The light from the optical element may be focused on the sample by their cooperation. The lighting subsystem shown in FIG. 1 and described in the present application may be equipped with any suitable other optical element (not shown). Examples of such optics are, but are not limited to, deflection optics (group), spectroscopic filters (group), spatial filters (group), reflective optics (group), apodizers (group), beam splitters. There are (groups), apertures (groups) and the like, which may include all such optical elements known and suitable in the art of the present application. In addition, the system may be configured to modify one or more of the components of the lighting subsystem based on the type of lighting used for inspection.

The inspection subsystem may have a scanning subsystem configured to scan the sample with light. For example, the test subsystem may have a stage 22 on which the sample 14 may be placed during the test. Even if the scanning subsystem has some suitable mechanical and / or robotic assembly (with stage 22) that can be configured to allow light to scan over the sample by moving the sample. Good. In addition to or in place of this, the inspection subsystem may be configured such that some optical scanning of the sample is performed by one or more optical elements included in the inspection subsystem. The scanning of the sample with the light may be performed in any suitable manner.

In addition, the inspection subsystem is provided with one or more detection channels. A detector provided in at least one of the one or more detection channels detects the light from the sample derived from the sample illumination by the inspection subsystem, and outputs according to the detected light. Configured to generate. For example, the inspection subsystem shown in FIG. 1 has two detection channels, one of which is formed by a condenser 24, an element 26 and a detector 28, and the other is a condenser 30, an element. It is formed of 32 and a detector 34. As shown in FIG. 1, these two detection channels are configured to collect and detect light at different focusing angles. In some cases, one detection channel is configured to detect specularly reflected light, and the other detection channel is configured to detect non-specularly reflected light (eg, scattered light, diffracted light, etc.) from the sample. However, two or more of the detection channels may be configured to detect the same type of light from the sample (eg, specularly reflected light). FIG. 1 shows an embodiment in which the inspection subsystem has two detection channels, but the inspection subsystem has a different number of detection channels (eg, a single detection channel or two or more detection channels). ) May have. In FIG. 1, each concentrator is shown as a single refractive optic, but as will be understood, each concentrator is one or more refractive optics and / or one or more. May have a reflective optical element of.

The detector provided in the one or more detection channels may be any detector known and suitable in the art of the present application. Examples of such detectors are photomultiplier tubes (PMTs), charge-shifting devices (CCDs) and time-delayed integral (TDI) cameras. These detectors may include any other detector known and suitable in the art of the present application. These detectors may include a non-imaging detector or an imaging detector. If the detectors are non-imaging detectors, each detector may be configured to detect certain characteristics of scattered light, such as intensity, but to detect that characteristic as a function of position within the imaging plane. Does not configure. Therefore, the output generated by the individual detectors in each detection channel of the inspection subsystem can be a signal or data, but not an image signal or image data. In such a case, the computer subsystem of this system, for example, the computer subsystem 36, may be configured to generate an image of the sample from the non-imaging output of those detectors. On the other hand, in some cases, these detectors may be configured as an imaging type detector configured to generate an image signal or image data. As such, the system can be configured in a variety of ways to produce the outputs described in the present application.

It should be noted that FIG. 1 is provided in the present application in order to roughly depict the configuration of the inspection subsystem that may be provided in the system embodiments described in the present application. Obviously, the inspection subsystem configuration described in the present application may be modified to optimize the performance of the system, which is common practice when designing commercial inspection systems. In addition, in implementing the system described in the present application, it is commercially available from existing inspection systems (eg, by adding the functions described in the present application to the existing inspection system), for example, KLA-Tencor located in Milpitas, California, USA. 28xx and 29xx series tools may be used. In some of such systems, the methods described herein can be provided as optional features of the system (eg, in addition to other features of the system). Alternatively, a completely new system may be provided by designing the system described in the present application "from the beginning".

The computer subsystem 36 of the system is a detector of the inspection subsystem in some suitable manner (eg, via one or more transmission media including one or more "wired" and / or "wireless" transmission media). And thus the output produced by those detectors during sample scanning can be received by the computer subsystem. The computer subsystem 36 may be configured to perform any of the functions detailed in the present application, as well as a number of functions using the outputs of those detectors as described in the present application. Further, this computer subsystem can be configured as described in the present application.

In the present application, this computer subsystem (as well as other computer subsystems described in the present application) may also be referred to as a computer system (group). The computer subsystems (groups) or systems (groups) described in the present application may take various forms, including personal computer systems, image computers, mainframe computer systems, workstations, network devices, Internet devices and other devices, respectively. .. As a whole, the term "computer system" can be broadly defined to include all devices having one or more processors that execute instructions obtained from a storage medium. Also, the computer subsystem (s) or system (s) may be equipped with any processor known and suitable in the art of the present application, such as a parallel processor. In addition, the computer subsystem (s) or system (s) may be equipped with high-speed processing platforms and software, whether stand-alone or networking tools.

If the system is equipped with multiple computer subsystems, the heterogeneous subsystems can be interconnected so that images, data, information, instructions, etc. can be sent between the computer subsystems as described in detail in the present application. be able to. For example, the computer subsystem 36 may be coupled to the computer subsystem (group) 102 by any suitable transmission medium (as shown by a broken line in FIG. 1), and the transmission medium is known in the art of the present application. It may include any wired and / or wireless transmission medium suitable for the above. Also, two or more of such computer subsystems may be substantially coupled by a shared computer readable storage medium (not shown).

Although the inspection subsystem has been described above as an optical or optical inspection subsystem, the inspection subsystem may be an electron beam inspection subsystem. For example, in some embodiments, electron-containing energy is directed toward the sample, and electron-containing energy is detected from the sample. In this way, the energy source may be used as an electron beam source. In such an embodiment shown in FIG. 2, the electronic column 122 provided in the inspection subsystem is coupled to the computer subsystem 124.

As shown in FIG. 2, the electron column has an electron beam source 126, the electron beam source is configured to generate electrons, and the electrons are sampled by one or more elements 130. It is focused on 128. Examples of the electron beam source include a cathode source, an emitter chip, and the like, and examples of one or more elements 130 include a gun lens, an anode, a beam limiting aperture, a gate valve, a beam flow selection aperture, an objective lens, and scanning. There are subsystems and the like, each of which may include any type of element known and suitable in the art of the present application.

The electrons returned from the sample (eg, secondary electrons) may be focused on the detector 134 by one or more elements 132. An example of one or more elements 132 is a scanning subsystem, which can be the same scanning subsystem included in element (group) 130.

The electronic column may be equipped with any other element known and suitable in the art of the present application. In addition, the electronic column is described in Jiang et al. Patent Document 1, Kojima et al., Dated April 4, 2014 in the name. Patent Document 2, dated April 8, 2014, in the name of Gubbens et al. Patent Document 3 dated April 15, 2014 in the name, and MacDonald et al. It may be configured as described in Patent Document 4 dated May 6, 2014 in the name, and these documents shall be incorporated with reference to these documents as if they were fully described in the present application.

The electron column shown in FIG. 2 is configured such that electrons are directed toward the sample at one oblique angle of incidence and scattered from the sample at another oblique angle, but as you can see, the electron beam is The angle toward the sample and scattered from the sample may be any angle as desired. In addition, the electron beam subsystem may be configured to generate images of the sample using multiple modes (eg, with different irradiation angles, collection angles, etc.). The difference in the plurality of modes of these electron beam subsystems is the difference in the image generation parameters of any of the subsystems.

The computer subsystem 124 can be coupled to the detector 134 as described above. By detecting the electrons returned from the surface of the sample with the detector, an electron beam image of the sample can be formed. The electron beam images may include any suitable electron beam image. The computer subsystem 124 may be configured to perform any of the functions described in the present application using the output of the detector and / or their electron beam images. The computer subsystem 124 may be configured to perform any of the additional steps (s) described herein. The system having the inspection subsystem shown in FIG. 2 can be further configured as described in the present application.

It should be noted that FIG. 2 is provided in the present application in order to roughly depict the configuration of the electron beam inspection subsystem that may be provided in the embodiments described in the present application. Similar to the optical inspection subsystem described above, the electron beam inspection subsystem configuration described in the present application may be modified to optimize the performance of the inspection subsystem, which is commonly done when designing a commercial inspection system. That is what you are doing. In addition, in implementing the system described in the present application, the existing inspection system may be used (eg, by adding the functions described in the present application to the existing inspection system). In some of such systems, the methods described herein can be provided as optional features of the system (eg, in addition to other features of the system). Alternatively, the system described in the present application may be designed "from the beginning" to provide a completely new system.

Although the inspection subsystem has been described above as an optical or electron beam inspection subsystem, the inspection subsystem may be an ion beam inspection subsystem. Such an inspection subsystem can be configured as shown in FIG. 2, except that the electron beam source is replaced by any ion beam source known and suitable in the art of the present application. In addition, its inspection subsystems include any other suitable ion beam subsystems such as commercially available focused ion beam (FIB) systems, helium ion microscopic (HIM) systems and secondary ion mass spectrometry (SIMS). ) It may be included in the system.

One or more computer subsystems detailed in the present application may be combined with a test subsystem that performs test of a sample. For example, in one embodiment, the one or more computer subsystems are configured to detect defects on the sample based on the output generated by the detector. Alternatively, one or more other computer subsystems may be coupled to the inspection subsystem that performs the inspection of the sample. Such computer subsystems (groups) can be configured as detailed in the present application. In any case, by appropriately configuring one or more computer subsystems coupled to the inspection subsystem, it is based on the output generated by the one or more detectors provided in the inspection subsystem. Defects on the sample can be detected. Those defects are identified in any preferred manner (eg, applying a threshold to the output and identifying an output with one or more values above the threshold as a defect, while identifying the defect. It may be detected on the sample (by not identifying an output with one or more values below the threshold as a defect). Defects detected on a sample can include any defects known in the art of the present application.

However, the computer subsystem (group) provided in the system described in the present application does not necessarily detect defects on the sample. For example, the computer subsystem (s) may be configured to acquire test results for a sample that include information about defects detected on the sample. The test results of the sample may be obtained directly from the system performing the test (eg, from the computer subsystem of the test system) by the computer subsystem (group) described in the present application, or the test results are stored. It may be obtained from a stored medium such as a fab database.

As mentioned above, the inspection subsystem is configured to scan a tangible version of a sample with energy (eg, light or electrons), thereby producing a real image of the tangible version of the sample. .. As such, the inspection subsystem is configured as a "real" tool and cannot be configured as a "virtual" tool. Possible "virtual" tools can be configured, for example, a storage medium (not shown) and a computer subsystem (group) 102 shown in FIG. In particular, these storage media and computer subsystems (s) are not part of the testing subsystem 10 and have no capacity to handle tangible samples. In other words, in a tool configured as a virtual tool, the output of one or more "detectors" provided in it can be generated prior to one or more detectors in a real tool. It is an output stored in the virtual tool, and at the time of "scanning", the stored output can be reproduced by the virtual tool as if the sample is being scanned. In this way, scanning a sample with a virtual tool can appear to be similar to scanning a tangible sample with a real tool, but in reality, in that "scan", the sample is scanned. The output is simply regenerated in the same manner as in the case of. For systems and methods configured as "virtual" inspection tools, see Bhaska et al., With the applicant of the present application as the assignee. Patent Document 5 dated February 28, 2012 and Duffy et al. Since there is a description in Patent Document 6 dated December 29, 2015 in the name, both are incorporated with reference to this as if there is a full description in the present application. Further, the embodiments described in the present application may be configured as described in those patent documents. For example, one or more computer subsystems described in the present application may be further configured as described in these patent documents. In addition, the construction of the one or more virtual systems as a central calculation storage (CCS) system may be performed as described in Patent Document 6 described above. The persistent storage mechanism described in the present application may include distributed information processing and storage, such as having a CCS architecture, but the embodiments described in the present application are not limited to such an architecture.

As mentioned above, the test subsystem may be configured to generate output for the sample in multiple modes. In general, a "mode" can be defined by the value of the parameters of the test subsystem used to generate the output for the sample. Therefore, if the mode is different, the value will be different for at least one of the imaging parameters of the inspection subsystem. For example, in an optical inspection subsystem according to an embodiment, at least one illumination light wavelength used in at least one of the plurality of modes and at least one other of the plurality of modes. It is assumed that the wavelength of at least one illumination light used is different. The illumination wavelengths of the various modes may be different depending on the mode (eg, by using different light sources, different spectroscopic filters, etc.) as described in detail in the present application. In yet another embodiment, the lighting channel in the inspection subsystem used in at least one of the plurality of modes and the lighting in the inspection subsystem used in at least one of the plurality of modes. The channel is different. For example, as described above, the inspection subsystem may be provided with a plurality of illumination channels. Then, different lighting channels may be used in different modes.

As described in detail in the present application, the optical and electron beam subsystems described in the present application may be configured as an inspection subsystem. However, the optical and electron beam subsystems described in the present application may be configured as other types of tools such as defect review subsystems. In particular, in the inspection subsystems of the embodiments described in the present application and shown in FIGS. 1 and 2, by modifying one or more parameters, the imaging ability differs depending on the application in which they are used. It may be provided. In some such examples, the inspection subsystem shown in FIG. 2 may be configured to exhibit higher resolution if one wishes to use it for defect review rather than inspection. In other words, the inspection subsystems of the embodiments shown in FIGS. 1 and 2 speak for some common and diverse configurations with respect to optical or electron beam subsystems and are proficient in the art of the present application. It is possible to provide various subsystems that have different imaging capabilities and are more or less suitable for different applications by installing them in various ways that may be obvious to a person (so-called a person skilled in the art).

The one or more computer subsystems may be configured to obtain the output produced for the sample by the test subsystem described in the present application. The output acquisition may be performed using one of the inspection subsystems described in the present application (eg, by directing a light or electron beam toward the sample and detecting the light or electron beam from the sample, respectively). In this way, the output acquisition can be performed using the tangible sample itself and some type of imaging hardware. However, in order to obtain the output, the sample is not necessarily imaged using the imaging hardware. For example, the output may be generated by another system and / or method, and the generated output is stored in one or more storage media, for example, the virtual inspection system described in the present application or another storage medium described in the present application. You may. As a result, when acquiring the output, the output may be acquired from the storage medium in which they are stored.

The inspection-related algorithm according to one embodiment is a defect classifier. For example, the algorithm can classify defects detected on a sample into various defect types or classes. The defect classifier can have any suitable configuration, such as a decision tree or nearest neighbor configuration. An inspection-related algorithm according to another embodiment is a defect filter. The defect filter may be configured as a nuisance filter, and the actual defects may be separated from the nuisance (which can be defined as detailed in the present application) and other noises, and the nuances and noises are inspected. It may be configured to be excluded from (filtering by excluding) from. This defect filter can also have any suitable configuration, such as a decision tree or nearest neighbor configuration. The inspection-related algorithm according to the additional embodiment is a defect detection algorithm. By properly configuring the defect detection algorithm, defect detection can be performed as described in detail in the present application and / or in any other manner known and suitable in the art of the present application. The inspection-related algorithm according to a further embodiment is a machine learning algorithm. The various inspection-related algorithms described in the present application can be configured as a machine learning algorithm. For example, the configuration adopted by the defect classifier, the defect filter, and the defect detection algorithm may be a machine learning algorithm type configuration. In addition, these machine learning algorithms are described in Zhang et al. Patent Document 7, named May 25, 2017, Zhang et al. Patent Document 8, Bhaska et al., Dated July 6, 2017 in the name. Patent Document 9, Bhaska et al., Dated July 6, 2017 in the name. Patent Document 10 dated July 13, 2017 in the name of Bhaska et al. Patent Document 11 dated July 13, 2017 and Zhang et al. It may be configured as described in U.S. Patent Application No. 15/603249 dated May 23, 2017 in the name, and by this reference, they shall be incorporated as fully described herein. The inspection-related algorithms described in the present application may have any of the configurations described in these documents or applications.

The one or more computer subsystems are configured to generate an initial version of an inspection-related algorithm by performing initial training on the inspection-related algorithm with a labeled defect set. According to some embodiments, the computer subsystem (s) can be configured to generate a labeled defect set for performing initial training. For example, as shown in FIG. 3, the defects of the first batch may be selected according to the computer subsystem (group) as shown in step 300. The defects in the first batch may be selected as described in detail in the present application. In addition, the computer subsystems (groups) may classify those selected defects as shown in step 302 (FIG. 3 describes the steps involved in the defect classifier, which is shown in FIG. The steps described in may be performed with respect to another inspection-related algorithm described in the present application). The computer subsystem (group) can classify the selected defects and / or obtain the classification related to the selected defects as described in detail in the present application. The computer subsystem (group) may then train the classifier as shown in step 304. Therefore, the training performed in step 304 is the initial training described in the present application. This initial training may be performed in any manner known and suitable in the art of the present application. Information about these defects, such as attributes and / or images (or other detector outputs), may be entered into the defect classifier to classify those labeled defects. Then, one or more parameters of the defect classifier (eg, cut lines, defect attributes, etc.) are modified until the classification obtained by the defect classifier for those defects matches the label assigned to those defects. Just go. Defects can be labeled as described herein, and defect attributes and defect patches (eg, optical attributes and / or optical patches) can be used as input data for their inspection-related algorithms.

The computer subsystem (s) are also configured to apply the first edition of inspection-related algorithms to unlabeled defect sets. For example, after an inspection-related algorithm has been initially trained for labeled defects, applying the first edition of the inspection-related algorithm to the remaining defects (and potential defects), i.e., tens of thousands of defects in a hot inspection of a wafer. Can be applied to defects detected by inspection (which may have) but not labeled.

As mentioned above, where defects can be labeled as described herein, attributes (groups) and / or patch images (groups) and other detector outputs can be input to inspection-related algorithms for initial training. After initial training on labeled sets (eg, using defect attributes (groups) and / or patch groups and other detector outputs), the first edition of inspection-related algorithms can be applied to unlabeled defect sets. The application of the first edition of the inspection-related algorithm may be performed by inputting all (or part) of the available information about the unlabeled defect set into the inspection-related algorithm. The unlabeled defect set may be configured as described in detail in the present application.

The computer subsystem (s) are further configured to modify the labeled defect set based on the results of the application. For example, when applying the first edition of the inspection-related algorithm to unlabeled defects, the inspection-related algorithm not only gives the results for each unlabeled defect (eg, defect classification), but also the confidence in the decision (eg, classification It is sufficient to output the thing). Then, the confidence level may be used to repeat the defect selection process next time. The defects selected in the defect selection process may be labeled as detailed in the present application, and the labeled defect set may be modified by adding them to the labeled defect set. Modification of the labeled defect set may be performed as described in detail in the present application.

In some embodiments, labeled and unlabeled defect sets coexist in the same inspection result. For example, as described in detail in the present application, a labeled defect set and an unlabeled defect set may be created by scanning one or a plurality of samples. By executing such a scan as a hot scan, it is preferable to capture as many defects or defect types as possible. If the scan includes a hot scan, a large number of defects will be detected in the scan so that a single hot scan of a single sample is sufficient to obtain sufficient defects for all steps described in the present application. Some of the defects detected by the scan may be labeled as described in the present application, whereby a labeled defect set (that is, a defect training set) can be created. The remaining defects detected by the scan but not the labeled defect set may be used as an unlabeled defect set. Therefore, all of the defects detected by one or more hot scans can form the whole of the defects used by the embodiments described in the present application, some of which are labeled and one described in the present application. Alternatively, it may be used in multiple steps, the others unlabeled and used in the other one or more steps described in the present application.

In another embodiment, when modifying a labeled defect set, one or more of the defects in the unlabeled set are labeled, and one or more of those labeled defects are the labeled set. Will be added to. For example, one or more of the selected defects in the unlabeled set may be selected as described in the present application, and then the one or more defects may be labeled in any suitable manner. According to one example of that kind, the one or more selected defects are imaged by the imaging subsystem, and the resolution is made higher than that of the inspection subsystem. A high resolution image of selected defects can be generated. Then, the high-resolution defect image may be presented to the user, and the user may assign a label to the defect. However, as detailed in the present application, these selected defects may be labeled with an automatic defect classifier (ADC). Therefore, a high resolution defective image may be provided to the user or ADC and the high resolution image may be manipulated there. The label assigned by the user may include any of the labels described in the present application, such as defects, nuances, noise, defect classification codes, and the like. The label assigned by the user may vary depending on the configuration of the inspection-related algorithm. In some cases, the computer subsystem (s) may present the user with a large number of label candidates (eg, defects, non-defects, defect class code x, defect class code y, etc.). In addition, the computer subsystem (s) may allow the user to enter a new label, eg, a new defect class code, which may be used to modify the configuration of the inspection-related algorithm (eg). When inspection-related algorithms generate new codes, bins, definitions, etc. for new defect labels). The labeled one or more defects store information about those newly labeled defects (groups) in any preferred manner (eg, information about the previously labeled defects). It can be added to the labeled defect set (by adding it to the file or other data structure that is being labeled).

As detailed in the present application, in certain embodiments, the one or more computer subsystems are configured to detect defects on a sample based on the output generated by the detector and are detected on those samples. A labeled defect set and an unlabeled defect set coexist in the defect. For example, all defects used by the computer subsystem (group) described in the present application may be detected on a sample or sample group by performing one or more hot scans on the sample (group). Among other things, in inspections such as optical inspections, nuisance filters and other inspection-related algorithms are typically trained on the results of hot scans (ie, sample inspections that result in tens of thousands of defects). A "hot scan" is generally defined as a test performed on a sample in which latent defects and defect detection thresholds are intentionally set at or near the noise floor of the output generated by the scan. , Can be defined. A "hot scan" is typically performed to detect as many potential defects and defects as possible and ensure that most or all of the interesting defects are captured, for purposes such as inspection recipe setup. Therefore, nuisance filters and other inspection-related algorithms can be trained using hot scan results.

To train inspection-related algorithms such as nuance filters or defect classifiers, label a relatively small set of defects detected on the sample. The meaning of "labeling" is to "classify" those defects. The meaning of "classifying" defects can vary depending on the inspection-related algorithms trained or generated by the computer subsystem (s). For example, if the inspection-related algorithm is a defect detection algorithm, the detected defects may be labeled as real defects and non-real defects (eg, noise) when classifying. Also, for example, if the inspection-related algorithm is a nuance filter, the detected defects include real defects and nuance defects (which are noise and / or real defects that the user does not really care about) when classifying them. Can be generally defined as). Further, for example, if the inspection-related algorithm is a defect classifier, the detected defects are labeled with a defect ID, eg, class code, and thus various defect types, eg, bridges, when classifying. Particles, scratches, loss of shape features, rough parts, etc. may be indicated. In this defect classification or labeling, generally, a fairly high resolution image of those defects will be initially captured. These high-resolution images may be generated using SEM or high-resolution optical imaging.

In one embodiment, the initial training labeled defect set comprises a predetermined minimum number of defects selected from all the defects detected on the sample. For example, as detailed in the present application, one of the advantages of the embodiments is that labeled defects within the training set can be minimized without compromising the quality of the trained inspection-related algorithms. For that purpose, the predetermined minimum number of labeled defects for initial training may be the minimum number of defects required to generate a coarsely trained first edition inspection-related algorithm. The minimum number of labeled defects may be predetermined empirically or based on past experience and knowledge (eg, how many labeled defects are needed to train inspection-related algorithms). it can. In addition, the predetermined minimum number of labeled defects may vary depending on the inspection-related algorithm. For example, in a defect classifier, the predetermined minimum number of labeled defects may be several (eg, two or three) for each defect type that is likely to appear on the sample and / or is a constituent of the classifier. In another inspection-related algorithm, such as a defect detection algorithm or a nuance filter, the predetermined minimum number of labeled defects may be several or tens (eg, 10 to 50 each) for defects and non-defects. The predetermined minimum number of defects may be randomly selected from all the defects that can be used in the embodiments described in the present application and / or detected on the sample (eg, unlabeled defects in the hot scan result). .. Then, those randomly selected defects may be labeled as described in the present application. Then, by analyzing these labeled defects, it is sufficient to determine whether or not the predetermined minimum number of labeled defects is sufficient for the initial training. If a particular type of defect is not sufficiently selected and labeled, the steps described above may be repeated until the sample of labeled defect contains the desired number of labeled defects.

In the embodiments described in the present application, defects near the boundary of the underlying distribution are searched for in an iterative manner. In addition, in the embodiments described herein, training set selection and defect labeling are performed in combination with a tuning process, which is facilitated by the inspection-related algorithm, especially. It is a new idea for optical inspection. For example, in a further embodiment, when modifying a labeled defect set, the application of the first edition of the inspection-related algorithm determines the certainty of the results produced for the unlabeled set of defects and has the lowest certainty. Defects in the unlabeled set are selected, labels are obtained for those selected defects, and those selected defects and their labels are added to the labeled defect set. For example, as shown in FIG. 3, the computer subsystem (group) may be configured to calculate the uncertainty of its model, i.e. the inspection-related algorithm, for each defect as shown in step 306. In addition, the computer subsystem (group) may be configured to look for a new set of defects with the lowest certainty in the test data as shown in step 308. Further, the computer subsystem (group) may be configured to classify the new set as shown in step 310. Further, the computer subsystem (group) may be configured to add the new set to the training set as shown in step 312. Thus, since these steps are after the inspection-related algorithm has been initially trained with a sufficiently small labeled defect set, the certainty of the inspection-related algorithm (eg, classifier) for each defect. Can be measured. Its certainty can be determined in any suitable way. For example, the inspection-related algorithm may be configured to generate confidence (eg, confidence related to each defect classification) in association with each result generated by itself. Certainty may be determined using the degree of confidence. In addition, the inspection-related algorithm may be configured to automatically derive certainty for each result generated by the inspection-related algorithm. Then, the set of defects that the inspection-related algorithm is most uncertain about is selected and labeled. Labeling (classification) of defects related to the training set may be performed manually using an optical image (eg, patch) or SEM image. The labeling can also be performed automatically using a pre-trained SEM automatic defect classifier (ADC). With reliable SEMADC, the training process could be fully automated and the recipe tuning process could be further accelerated beyond the main configuration concept described in the present application. The labeled defects in this new batch can be used in addition to the previously labeled defects to retrain (or correct) the inspection-related algorithms. Each of these steps may be performed as described in detail in the present application.

In some of these embodiments, when selecting the unlabeled in-set defects with the lowest certainty, a predetermined minimum number of unlabeled in-set defects with the lowest certainty is selected. For example, the defects in the unlabeled set may be selected from the defects having the lowest certainty first, the defects having the next lowest certainty, and the like so as to satisfy the predetermined minimum number. The predetermined minimum number of defects to be selected in the unlabeled set may be predetermined as described in the present application (eg, empirically or based on previous experience and history, appropriate training of the inspection-related algorithm. It is sufficient to determine the minimum number of defects required to achieve this).

In another of this type of embodiment, the selection of unlabeled intra-aggregate defects with the least certainty is the diversity in one or more properties of those unlabeled intra-aggregate defects. It runs independently. For example, according to the embodiments described in the present application, defects can be selected based on the uncertainty of the label assigned by the inspection-related algorithm, and the diversity in the first characteristic of the defect and the second characteristic of the defect can be selected. Diversity and other diversity in any property of the defect are not relevant. Thus, defect selection based on the uncertainty of the results provided by inspection-related algorithms with respect to those defects is separate from diversity sampling. In addition, the selection of unlabeled in-set defects with the lowest certainty can be performed independently of any other attributes or defect-related information. Nevertheless, if inspection-related algorithms are configured to assign different labels to different unlabeled defects, among those lowest certainty defects, the lowest certainty defect to which the first label is assigned and the second It may include a label-assigned minimum certainty defect. In other words, the lowest certainty defect selection in an unlabeled set that is independent of the diversity in one or more characteristics of the defect is performed based on (or relying on) the label assigned by the inspection-related algorithm. Can be done. Yet, the choice is still not made based on the variety of properties of any one or more of the defects themselves. For example, the lowest certainty defects assigned different labels do not necessarily have relatively diverse values for any one of the properties of those defects. In fact, because similarity, rather than diversity, resides in one of the properties of those defects, labels to them by inspection-related algorithms in the first, preliminary, or intermediate version. Labeling can be difficult.

In some embodiments, when modifying a labeled defect set, the certainty of the results provided by the application of the first edition of the inspection-related algorithm for defects in the unlabeled set is determined and is the lowest within the unlabeled set. A group of defects with certainty is selected, a subset of the defects within that subset that exhibits the greatest variety in the properties of the defects within that subset, and a label is obtained for the defects in that subset. And the defects in the selected subset and their labels are added to the labeled defect set. For example, according to the embodiments described in the present application, sampling can be made more efficient by combining uncertainty and diversity. The top priority is to sample defects whose inspection-related algorithms are almost unreliable, because they are defects that are near the classification boundaries and by providing ground truth for those defects, that inspection-related. This is because the quality of the algorithm is the most improved. However, when a large number of "low reliability" defects are present, the computer subsystem (s) may select defects of essentially the same appearance, that is, defects that would all exhibit the same reliability. Instead, it can be beneficial to try to diversify the defects of choice among many different low-reliability defects. This allows the computer subsystem (group) to select the most diverse set near the classification boundary, as opposed to simply selecting many defects that are located only in part of the boundary ( Since classification boundaries are complex in principle, unknown, and probably hyperplanes in multidimensional space, to obtain a properly trained inspection-related algorithm with the minimum number of labeled defects, It is desirable to carefully select defects across the boundary by the computer subsystem (group), in other words, by the computer subsystem (group), defects that are relatively far from the classification boundary, that is, relatively high reliability. It is desirable that the defect group exhibiting the above, or the defect group in the same part of the boundary, that is, the defect group that is not particularly diverse) is not selected).

The computer subsystem (s) are also configured to generate an updated version of the inspection-related algorithm by retraining the inspection-related algorithm with a modified labeled defect set. For example, as shown in FIG. 3, the computer subsystem (group) may be configured to retrain (or correct) the classifier as shown in step 314. The retraining may be performed as detailed in this application in the context of the initial training. However, the retraining in this retraining step may be the oldest version of the inspection-related algorithm (eg, parameters of the inspection-related algorithm brought about by the initial training) or the first version of the inspection-related algorithm (eg, before the initial training). Let's start with a version of the inspection-related algorithm with parameters). In general, when retraining a classifier after a new batch of defects has been labeled and added to the training set, the retraining may start with the old version of the classifier, but usually the retraining starts from the beginning. (The new classifier is simply trained for each new training set, but any means can be implemented). Thus, retraining essentially involves training the test-related algorithm from the beginning using the pre-initial-training version of the test-related algorithm, or one or more of the previous versions. Retraining the immediately prior version of the inspection-related algorithms can be included by tuning the parameters and, if possible, fine tuning.

In addition, the computer subsystem (s) is configured to apply an updated inspection-related algorithm to another unlabeled defect set. An unlabeled defect set to which the updated inspection-related algorithm is applied remains among the unlabeled defects available and / or detected on their samples for use in the embodiments described herein. It may include some and / or all of the labels. In this way, the unlabeled set to which the updated version is applied differs from the unlabeled set to which the first edition (or the old version) is applied because one or more unlabeled set defects are selected and labeled. This is because it has been added to the labeled defect set. Therefore, the unlabeled defect set to which the updated version is applied may contain a smaller number of defects than the unlabeled defect set to which the first edition (or the old version) is applied. However, in some cases, the remaining unlabeled defect set, that is, after some defects have been selected, labeled and added to the labeled set, when the number of remaining unlabeled defects is not large enough. It may be reinforced with additional unlabeled defects. Reinforcement of the unlabeled assembly may be performed in any preferred manner, for example by performing another hot scan on another sample and / or obtaining additional test results from storage media, virtual systems, etc. .. Overall, the scans described in the present application provide sufficient unlabeled defects for the functions / steps described in the present application. Therefore, a more widely performed reinforcement is the reinforcement of the labeled defect set when such defects are not sufficiently present, and thus the increase in their number. The updated inspection-related algorithm may be applied to other unlabeled defect sets as described in the present application. For example, after inputting information on all or at least a part of the defects in the other unlabeled set into the updated inspection-related algorithm, the result of each or at least a part of the unlabeled defects in the set is input. It should be generated there.

The computer subsystem (s) is also configured to discriminate one or more differences between the application of the updated inspection-related algorithm and the application of the first or previous inspection-related algorithm. Will be done. The first edition of the inspection-related algorithm is used to determine the difference when the updated version is only the generated second edition of the inspection-related algorithm (the version generated immediately after the first edition). In any other case, the previous version of the inspection-related algorithm used for difference determination in this step may be the inspection-related algorithm generated immediately before the updated version. In this case, the difference between the recently generated version of the inspection-related algorithm and the version generated immediately before that version may be determined. In other words, in this step, the difference between the nth version of the inspection-related algorithm and the n-1th version of the inspection-related algorithm may be determined.

After that, using these differences, it is determined whether or not the process has converged as described in detail in the present application. For example, as detailed in the present application, it may be determined that the process being executed by the computer subsystem (group) has converged when the change between iterations of the classification (or other result) becomes relatively small. This change cannot be exactly zero due to statistical fluctuations in the training process. In other words, repeating training for the same training set multiple times does not result in exactly the same classification (other results) for the same defect. Since these small fluctuations can be estimated, the process running by the computer subsystem (group) can be stopped when the change between iterations becomes smaller than the estimate. The process has converged. Moreover, when this standard is reached, the inspection-related algorithm achieves its maximum performance.

The computer subsystem (s) further modifies the labeled defect set, retrains the inspection-related algorithm, applies an updated version of the inspection-related algorithm, and discriminates the one or more differences. Alternatively, it is configured to repeat until a plurality of differences meet one or more criteria. Therefore, the one or more criteria determine the stopping criteria for labeling defects and for ending the iterations of the steps described in the present application. For example, as described above, when the one or more differences are less than or equal to the estimate of the relatively small fluctuations that will occur with each training regardless of the performance of the inspection-related algorithm. It may be determined that a plurality of differences satisfy the one or a plurality of criteria. In addition, the various results provided by the test-related algorithm may have different criteria. For example, one or more criteria for differences in results resulting from one defect classification may differ from one or more criteria for differences in results resulting from another defect classification. In such a case, the above steps may be repeated until all of the one or more criteria are met. Otherwise, all the various results produced by the test-related algorithm may have the same criteria. For example, one or more criteria for differences in results resulting from different defect classifications may be the same. However, even in such a case, the above-mentioned steps can be repeated until all the one or more criteria are satisfied. For example, if both defect classifications are required to meet the same one or more criteria, the result for one defect classification is earlier than the result for another defect classification. It can meet one or more criteria.

According to one example of that kind, as shown in FIG. 3, the computer subsystem (group) can be configured to determine whether or not the convergence criteria have been met, as shown in step 316. If the convergence criteria are not met, the computer subsystem (group) may return to step 306 and calculate the uncertainty of the model (check-related algorithm) for each defect, as shown in FIG. .. Further, steps 308, 310, 312 and 314 shown in FIG. 3 may be repeated until it is determined that the convergence criterion is satisfied by the computer subsystem (group). The reliance on the embodiments described in the present application for data-driven convergence criteria is novel. In other words, as detailed in the present application, the inspection-related algorithm (eg, classifier) can select unlabeled defects in batches that are almost unreliable. The selected defects can then be labeled as described in the present application. These newly labeled defects can be added to the training set and the modified training set can be used to train new inspection-related algorithms. These steps can be repeated until convergence is satisfied.

In certain embodiments, according to the one or more criteria a) the one or more differences, the updated version of the inspection-related algorithm is negligible with respect to the first or previous version of the inspection-related algorithm. Boundary between what indicates no difference and b) the above-mentioned one or more differences, which indicate that the updated version of the inspection-related algorithm is significantly different from the first or previous version of the inspection-related algorithm. Is decided. One or more differences are defined as described above (eg, differences between the nth edition of the inspection-related algorithm and the n-1th edition of the inspection-related algorithm). In this way, the computer subsystem (s) may track the history of the test-related algorithm after each iteration and terminate the iteration if the change in results produced by the test-related algorithm is small enough. ..

The term "negligibly different" used in this application is different for each inspection-related algorithm. Nonetheless, the "negligible difference" used in this application means that some difference was small enough that the inspection-related algorithm did not change significantly from one version of the inspection-related algorithm to the next. Can be defined as indicating. Therefore, the suspension criteria according to the embodiments described in the present application are determined by one or a plurality of differences that are qualitatively appropriate for "the difference is negligible". Therefore, the user predetermines and defines (based on what their permissible suspension criteria) the "negligibly different" values for the one or more differences, and / or The computer subsystem (group) or other method or system can be pre-determined based on the reproducibility of a particular training subject inspection-related algorithm and / or general or specific information about the type of training subject inspection-related algorithm. A single or multiple "significantly different" difference in terms used herein can be any difference other than a "negligibly different" value difference. Thus, the one or more differences are in two ranges, i.e., 1) "negligibly different" as defined as described in the present application, and 2) "negligibly different". It can have all but "significantly different" things.

If the change from the previous iteration to the current iteration is zero (or small), then the computer subsystem (s) has no new defects worth labeling, because the test It is determined that they cannot be confirmed by the related algorithm. According to one embodiment, the computer subsystem (group) can use the history of class code changes predicted for defects in the final test data set. However, there are many other convergence indicators that can be considered for use in the embodiments described in the present application. With any of these convergence indicators, certain aspects of classifier performance and / or the content of the training set can be monitored as a function of training iterations. For example, by tracking the accuracy of a function of iteration, the computer subsystem (s) can monitor the performance of the inspection-related algorithm itself. Another method relies on monitoring the improvement of the receiver operating characteristic curve (ROC) as a function of iteration. ROC is basically an index of the performance of a binary classifier over the entire operating point (eg, various nuance rates). In addition, under certain circumstances, or for some specific purpose, what kind of defects are being put into the training set with each iteration by the computer subsystem (s). Since it can be monitored, for example, the computer subsystem (group) can be stopped when the computer subsystem (group) no longer has the attention defect (DOI) in the training set.

When the one or more differences meet the one or more criteria, the computer subsystem (group) uses the latest version of the inspection-related algorithm for the inspection of other samples. It is configured to output as a related algorithm. When outputting the latest version of the inspection-related algorithm, the recently trained parameters of the inspection-related algorithm may be output as necessary together with the outline configuration of the inspection-related algorithm, if possible. When outputting the latest version of the inspection-related algorithm, for example, by storing the latest version in the storage medium and / or the inspection recipe, which is one of the storage media described in the present application, the inspection-related algorithm is executed when the inspection recipe is executed. (The term "recipe" used in this application can be defined roughly as a set of instructions that can be used in the system to execute a process).

In certain embodiments, the one or more computer subsystems are configured to determine the separability indicators of the various results provided by the latest version of the inspection-related algorithm, and the determined separability indicators are predetermined. The output is executed only after the threshold of is exceeded. For example, data to which an inspection-related algorithm such as a nuisance filter (classifier) is applied has varying degrees of separability between data, corresponding to various things such as defects vs. nuisance, certain defects vs. different types of defects, and so on. Present. When the data separation is substantially good, the inspection-related algorithms generally achieve relatively good performance with relatively high reliability. When the separation in the data is mediocre or bad, the inspection-related algorithm is also not performed, so typically a significant amount of relatively unreliable results will remain no matter what. Therefore, the convergence criterion is not based on any reliability or performance index. Therefore, the computer subsystem (s) simply monitors when the test-related algorithm stops improving, at which point the best test-related algorithm is provided for the data. Therefore, discriminating the separability index for the latest version of the test-related algorithm actually works well enough that the best test-related algorithm provided by the available training data can be fully used for other samples. It is possible to determine whether or not to do so. If the separability index is determined to be inadequate, examine other data provided by other options, such as other output generation parameters of the inspection subsystem, and the inspection-related as detailed in this application. It can be an alternative input to the algorithm.

According to an example of that kind, as shown in FIG. 3, when it is determined in step 316 that the convergence criterion is satisfied, whether or not the data can be separated by the computer subsystem (group). Can be determined as shown in step 318. If it is determined in step 318 that the data is separable, then in step 320 the computer subsystem (group) is ready for the test-related algorithm (ie, ready for use in testing other samples). It can be determined that it has been completed, ready to be used for production monitoring, etc.). In this way, in various embodiments, a guarantee index for the correctness of the inspection-related algorithm can be used. In order to be able to correctly separate data by the inspection-related algorithm, the separability of the data should be measured. This indicator tells whether the data is separable. If the inspection-related algorithm is a defect classifier, the index can tell whether the data is separable and whether the classifier can better classify each defect class than to guess. If the data in the training set is separable, it can be said that the correct classifier has been constructed. An accuracy threshold for each class code of the classifier (eg, somewhere above 50% (because 50% accuracy for the equilibrium training set is a completely random classification and not separable). If it exceeds)), the data in the training set can be considered separable.

If the computer subsystem (group) determines in step 318 of FIG. 3 that the data is not separable, the computer subsystem (group) shows the inspection parameters in step 322 as shown in FIG. You can change it as follows. For example, if the data is not separable, then in the case of a defective classifier, the data cannot be classified. In this case, the computer subsystem (group) may determine that one or more parameters of the inspection subsystem should be changed. For example, the computer subsystem (group) may determine that the inspection mode should be changed. The computer subsystem (s) then tunes one or more parameters of the inspection subsystem, or simply to another subsystem (computer or the like) to perform the tuning. All you have to do is issue an order. Tuning or modification of one or more parameters of the inspection subsystem may be performed in any suitable manner. You can then use the tuning or modified parameters (groups) of the inspection subsystem to generate an output, and then use that output to generate a labeled and unlabeled defect set, and then use it. By executing the steps (groups) described in the present application, it is possible to generate a trained inspection-related algorithm. In this way, a trained inspection-related algorithm may be generated for the new parameters (groups) of the inspection subsystem.

The embodiments described herein offer a number of advantages with respect to training of inspection-related algorithms. For example, since the inspection-related algorithm tuning and acquisition of the training set described in the present application enhance the effectiveness of labeled defects related to the performance of the inspection-related algorithm, a single combination of inspection-related algorithm tuning and training set acquisition is used. The methodology offers enormous advantages over existing methods (these labeled defects are the most teaching defects for training purposes, so the performance of their inspection-related algorithms is always optimal for a given data. ). In addition, labeling defects (eg, manually classifying them) is quite expensive in terms of tool time and effort. Identifying the convergence criteria for the training set acquisition and inspection-related algorithm tuning process reduces the size of the training set, which is an advantage. Moreover, the recognition that the process of combining training set selection and defect labeling with its tuning process is absolutely essential for the application of any machine learning algorithm for optical inspection nuance filters and classifiers is new. (It is necessary to combine training set selection and defect labeling with the tuning process because the training data contains tens of thousands of defects, most of which are nuisances). Also, in the embodiments described in the present application, the consistency of the inspection recipe is guaranteed, so that the nuisance filter tuning no longer depends on experience and skill.

Each embodiment of the system described in the present application can be combined with any other embodiment of the system described in the present application.

Another embodiment relates to a computer implementation method for training inspection-related algorithms. The method has steps relating to each function of the computer subsystem (s) described above. In particular, in this method, the first version of the inspection-related algorithm is generated by performing the initial training of the inspection-related algorithm with the labeled defect set. Further, in this method, the inspection-related algorithm of the first edition is applied to the unlabeled defect set, and the labeled defect set is modified based on the result of the application. In addition, in this method, the inspection-related algorithm is retrained with the modified labeled defect set to generate an updated version of the inspection-related algorithm. Further, in this method, an updated inspection-related algorithm is applied to another unlabeled defect set. Further, in this method, one or a plurality of differences between the application result of the updated version of the inspection-related algorithm and the application result of the inspection-related algorithm of the first edition or the previous version are discriminated. In addition, in this method, modification of those labeled defect sets, retraining of inspection-related algorithms, application of updated inspection-related algorithms, and determination of one or more differences can be performed in one or more ways. Is repeated until the difference meets one or more criteria. When the one or more differences meet the one or more criteria, in this method, the latest version of the inspection-related algorithm is output as a trained inspection-related algorithm used for inspection of other samples. The algorithm.

Each step of the method can be performed as detailed in this application. The method may also include the inspection subsystem described in the present application and / or any other step (group) that can be performed by the computer subsystem (group) or system (group). One or more computer systems that perform the steps of the method may be configured according to any of the embodiments described in the present application. In addition, the above method may be performed by any of the system embodiments described in the present application.

A further embodiment is a non-temporary computer-readable medium in which a program instruction is stored, in which the program instruction is executed on a computer system to execute a computer-implemented method for training an inspection-related algorithm. It concerns temporary computer-readable media. One such embodiment is shown in FIG. In particular, within the non-transitory computer-readable medium 400 shown in FIG. 4, there is a program instruction 402 that can be executed on the computer system 404. The computer implementation method may include every step (group) of any method (group) described in the present application.

Method For example, the program instructions 402 that embody those described in the present application may be stored on the computer-readable medium 400. The computer-readable medium may be a storage medium such as a magnetic or optical disk, a magnetic tape, or any other non-temporary computer-readable medium known and suitable in the art of the present application.

These program instructions may be implemented according to any of a variety of methods, including procedure-based technology, element-based technology, and / or object-oriented technology. For example, use ActiveX® controls, C ++ objects, JavaBeans®, Microsoft® Foundation Classes (MFC), SSE (Streaming SIMD Extensions) and other technologies or methodologies to implement those program instructions. Just do it.

The computer system 404 may be configured according to any of the embodiments described in the present application.

In any of the methods described in the present application, the results of one or more steps of those method embodiments may be stored in a computer-readable storage medium. These results may include any of the results described in the present application, and may be stored in any manner known in the art. The storage medium may include any storage medium described in the present application, or any other storage medium known and suitable in the art of the present application. After the results are stored, the results in the storage medium can be accessed and used by any of the methods or system embodiments described in the present application, formatted for the user, and another software module. , Can be used by method or system, etc. For example, a trained test-related algorithm can be used to perform one or more tests on other samples (groups), and it can be performed as described in the present application. The results produced by that or those tests can be used to perform one or more functions with respect to the other sample (group) or to the process used to form the other sample (group). .. For example, one or more used to form another sample (group) using the results generated by performing one or more tests using a trained test-related algorithm as described in the present application. One or more parameters of the process can be modified. In addition to or in place of this, it will be performed on other samples (groups) using the results generated by performing one or more tests using the test-related algorithms trained as described in the present application. By modifying one or more parameters of one or more processes, additional features or materials can be formed on the other sample (group) or defects on the other sample (group). The other sample (group) itself can be modified by correction.

Those skilled in the art can conceive further modifications and alternative embodiments of aspects of the invention by reference to this specification. For example, methods and systems for training inspection-related algorithms are presented. Therefore, this specification should be understood exclusively as an illustration and is intended to teach so-called one of ordinary skill in the art the general procedure for carrying out the present invention. As will be appreciated, the embodiments illustrated and described herein should be considered as preferred embodiments. It is the present invention that elements and materials can be replaced with those illustrated and described herein, that parts and processes can be reordered, and that certain features of the invention can be used independently. It will be understood by those skilled in the art by referring to this description of. The elements described in the present application may be modified without being separated from the essence and technical scope of the invention described in the claims of another paragraph.

Claims (20)

A system configured to train inspection-related algorithms
An inspection subsystem that includes at least an energy source and a detector, the energy source being configured to generate energy towards the sample, detecting energy from the sample, and generating output in response to the detected energy. With the inspection subsystem in which the detector is configured to
One or more computer subsystems
Generate the first edition of the inspection-related algorithm by performing the initial training of the inspection-related algorithm with the labeled defect set.
Applying the first edition inspection-related algorithm to the unlabeled defect set,
Based on the result of the application, the labeled defect set is modified.
By retraining the inspection-related algorithm with the modified labeled defect set, an updated version of the inspection-related algorithm is generated.
Applying the updated inspection-related algorithm to another unlabeled defect set,
Determine one or more differences between the application result of the inspection-related algorithm of the updated version and the application result of the inspection-related algorithm of the first edition or the previous version.
Modification of the labeled defect set, retraining of the inspection-related algorithm, application of the updated version of the inspection-related algorithm, and determination of the one or more differences are all performed by the one or more differences. Or repeat until it meets multiple criteria,
When the one or more differences meet the one or more criteria, the latest version of the inspection-related algorithm is configured to be output as a trained inspection-related algorithm used for inspection of other samples. Computer subsystem and
System with. The system according to claim 1.
A system in which the inspection-related algorithm is a defect classifier. The system according to claim 1.
A system in which the inspection-related algorithm is a defect filter. The system according to claim 1.
A system in which the inspection-related algorithm is a defect detection algorithm. The system according to claim 1.
A system in which the inspection-related algorithm is a machine learning algorithm. The system according to claim 1.
A system in which the labeled defect set and the unlabeled defect set are included in the same inspection result. The system according to claim 1.
Modification of the labeled defect set
The step of labeling one or more of the defects in the unlabeled set,
The step of adding the labeled defect to the labeled set, and
System including. The system according to claim 1.
The one or more computer subsystems are further configured to detect defects on the sample based on the output generated by the detector, and the defects detected on the sample are labeled. Defect set and the system included in the unlabeled defect set. The system according to claim 1.
A system in which the labeled defect set contains a predetermined minimum number of defects selected from all the defects detected on the sample. The system according to claim 1.
Modification of the labeled defect set
Steps to determine the certainty of the results produced for defects in the unlabeled set by the application of the first edition inspection-related algorithm.
The step of selecting the defect in the unlabeled set with the lowest certainty, and
Steps to get a label for the selected defect,
The steps of adding the selected defects and their labels to the labeled defect set, and
System including. The system according to claim 10.
A system in which selection of the least certainty of defects in the unlabeled set comprises selecting a predetermined minimum number of defects with the lowest certainty of the defects in the unlabeled set. The system according to claim 10.
A system in which the selection of the defects in the unlabeled set with the lowest certainty is performed independently of the diversity in one or more characteristics of the defects in the unlabeled set. The system according to claim 1.
Modification of the labeled defect set
Steps to determine the certainty of the results produced for defects in the unlabeled set by the application of the first edition inspection-related algorithm.
The step of selecting the defect group having the lowest certainty among the defects in the unlabeled set, and
The steps to select a subset of defects within that group that exhibits the greatest variety in the properties of the defects within that subset.
The step of getting a label for a defect in that subset,
The steps of adding the defects of the selected subset and their labels to the labeled defect set, and
System including. The system according to claim 1.
The above-mentioned one or more criteria
a) One or more of the differences, indicating that the updated version of the inspection-related algorithm is negligibly different from the first or previous version of the inspection-related algorithm.
b) It defines the boundary between the above-mentioned one or more differences, which indicate that the inspection-related algorithm of the updated version is significantly different from the inspection-related algorithm of the first edition or the previous version. A system. The system according to claim 1.
The one or more computer subsystems are further configured to determine the separability index of various results provided by the latest version of the inspection-related algorithm, and the determined separability index is predetermined. A system in which the output is performed only after the threshold of is exceeded. The system according to claim 1.
A system in which a wafer is included in the sample. The system according to claim 1.
A system in which light is included in the energy directed toward the sample, and light is included in the energy detected from the sample. The system according to claim 1.
A system in which the energy directed toward the sample contains electrons, and the energy detected from the sample contains electrons. A non-temporary computer-readable medium in which program instructions are stored, and by executing the program instructions on a computer system, a computer-implemented method for training inspection-related algorithms is executed. There is a computer implementation method,
The steps to generate the first edition of an inspection-related algorithm by performing initial training on the inspection-related algorithm with a labeled defect set, and
Steps to apply the first edition inspection-related algorithm to an unlabeled defect set,
The step of modifying the labeled defect set based on the result of the application, and
A step of generating an updated version of the inspection-related algorithm by retraining the inspection-related algorithm with the modified labeled defect set.
Steps to apply the updated inspection-related algorithm to another unlabeled defect set,
A step of determining one or more differences between the application result of the updated version of the inspection-related algorithm and the application result of the first-edition or previous version of the inspection-related algorithm.
Modification of the labeled defect set, retraining of the inspection-related algorithm, application of the updated version of the inspection-related algorithm, and determination of the one or more differences are all performed by the one or more differences. Or a step that repeats until it meets multiple criteria,
When the one or more differences meet the one or more criteria, the step of outputting the latest version of the inspection-related algorithm as a trained inspection-related algorithm used for inspection of other samples, and
The execution of the initial training, the application of the first edition, the modification of the labeled set, the retraining of the inspection-related algorithm, the application of the updated version, the determination of one or more differences, the repetition, And a non-transitory computer-readable medium whose output is executed by the computer system. A computer practice method for training inspection-related algorithms
The steps to generate the first edition of an inspection-related algorithm by performing initial training on the inspection-related algorithm with a labeled defect set, and
Steps to apply the first edition inspection-related algorithm to an unlabeled defect set,
The step of modifying the labeled defect set based on the result of the application, and
A step of generating an updated version of the inspection-related algorithm by retraining the inspection-related algorithm with the modified labeled defect set.
Steps to apply the updated inspection-related algorithm to another unlabeled defect set,
A step of determining one or more differences between the application result of the updated version of the inspection-related algorithm and the application result of the first-edition or previous version of the inspection-related algorithm.
Modification of the labeled defect set, retraining of the inspection-related algorithm, application of the updated version of the inspection-related algorithm, and determination of the one or more differences are all performed by the one or more differences. Or a step that repeats until it meets multiple criteria,
When the one or more differences meet the one or more criteria, the step of outputting the latest version of the inspection-related algorithm as a trained inspection-related algorithm used for inspection of other samples, and
The execution of the initial training, the application of the first edition, the modification of the labeled set, the retraining of the inspection-related algorithm, the application of the updated version, the determination of one or more differences, the repetition, And a computer implementation method in which the output is performed by one or more computer systems.