TWI754764B - Generating high resolution images from low resolution images for semiconductor applications - Google Patents

Abstract

Methods and systems for generating a high resolution image for a specimen from a low resolution image of the specimen are provided. One system includes one or more computer subsystems configured for acquiring a low resolution image of a specimen. The system also includes one or more components executed by the one or more computer subsystems. The one or more components include a deep convolutional neural network that includes one or more first layers configured for generating a representation of the low resolution image. The deep convolutional neural network also includes one or more second layers configured for generating a high resolution image of the specimen from the representation of the low resolution image. The second layer(s) include a final layer configured to output the high resolution image and configured as a sub-pixel convolutional layer.

Description

Generating high-resolution images from low-resolution images for semiconductor applications

The present invention generally relates to methods and systems for generating high-resolution images from low-resolution images for semiconductor applications.

The following description and examples are not to be considered prior art by virtue of their inclusion in this section.

Fabricating semiconductor devices, such as logic and memory devices, typically involves processing a substrate, such as a semiconductor wafer, using a number of semiconductor fabrication processes to form various features and levels of the semiconductor device. For example, lithography is a semiconductor fabrication process that involves transferring a pattern from a reticle to a photoresist disposed on a semiconductor wafer. Additional examples of semiconductor fabrication processes include, but are not limited to, chemical mechanical polishing (CMP), etching, deposition, and ion implantation. Multiple semiconductor devices can be fabricated in an arrangement on a single semiconductor wafer and then separated into individual semiconductor devices.

Inspection processes are used at various steps during a semiconductor fabrication process to detect defects on samples to facilitate higher yields and thus higher profits in the fabrication process. Inspection has always been an important part of making semiconductor devices. However, as the size of semiconductor devices decreases, inspection becomes even more important to the successful manufacture of acceptable semiconductor devices, since smaller defects can lead to device failure.

Defect re-inspection typically involves re-detecting defects such as those detected by an inspection process and using a high magnification optical system or a scanning electron microscope (SEM) to generate additional information about the defect at a higher resolution. Thus, defect re-inspection is performed at discrete locations on the sample where defects have been detected by inspection. The higher-resolution data of defects generated by defect re-inspection are more suitable for determining defect attributes, such as contour, roughness, more accurate size information, and the like.

Metrology processes are also used at various steps during a semiconductor fabrication process to monitor and control the process. A metrology procedure differs from an inspection procedure in that, unlike an inspection procedure in which defects on a sample are detected, a metrology procedure is used to measure one or more characteristics of a sample that cannot be determined using the inspection tools currently in use. For example, metrology procedures are used to measure one or more properties of a sample, such as a dimension (eg, line width, thickness, etc.) of features formed on a sample during a procedure, such that the one or more properties can be measured according to to determine the effectiveness of the program. Additionally, if one or more properties of the sample are unacceptable (eg, outside a predetermined range of one of the properties), then measurement of one or more properties of the sample can be used to alter one or more parameters of the procedure, Additional samples made by the procedure have acceptable properties.

Metrology procedures differ from defect re-inspection procedures also in that, unlike defect re-inspection procedures in which defects detected by inspection are revisited in defect re-inspection, metrology procedures can be performed at locations where no defects were detected . In other words, unlike defect re-inspection, the location at which a metrology procedure is performed on a sample can be independent of the results of an inspection procedure performed on the sample. In particular, the location at which to perform a metrology procedure can be selected independently of the inspection results.

Therefore, as explained above, due to the limited resolution of performing inspection (optical inspection and sometimes e-beam inspection), samples are generally required to generate additional higher resolution images for defect detection of defects on the samples Re-inspection, which may include verifying the detected defects, classifying the detected defects, and characterizing the defects. In addition, higher resolution images are generally required to determine information such as patterned features formed on a sample in metrology, regardless of whether defects have been detected in the patterned features. Thus, defect re-detection and metrology can be a time-consuming procedure that requires the use of the physical sample itself and additional tools (besides the inspector) required to generate higher resolution images.

However, defect re-inspection and metrology is not a process that can be simply eliminated to save time and money. For example, due to the resolution at which inspection procedures are performed, inspection procedures do not generally generate image signals or data sufficient to classify the defect and/or to determine a root cause of the defect that can be used to determine a detected defect. In addition, due to the resolution at which the inspection process is performed, the inspection process does not generally generate image signals or data that can be used to determine with sufficient accuracy the information of the patterned features formed on the sample.

Accordingly, it would be advantageous to develop systems and methods for producing a high-resolution image of a sample that do not have one or more of the disadvantages set forth above.

The following description of various embodiments should not be considered in any way to limit the subject matter of the scope of the appended claims.

One embodiment relates to a system configured to generate a high-resolution image of a sample from a low-resolution image of the sample. The system includes one or more computer subsystems configured for acquiring a low-resolution image of a sample. The system also includes one or more components executed by the one or more computer subsystems. The one or more components include a deep convolutional neural network including one or more first layers configured to generate a representation of the low-resolution image. The deep convolutional neural network also includes one or more second layers configured for generating a high-resolution image of the sample from the representation of the low-resolution image. The one or more second layers include a final layer configured to output the high-resolution image. The final layer is configured as a subpixel convolutional layer. The system can be further configured as set forth herein.

An additional embodiment relates to another system configured to generate a high-resolution image of a sample from a low-resolution image of the sample. This system is configured as described above. The system also includes an imaging subsystem configured for generating low-resolution images of the sample. In this embodiment, the computer subsystem is configured for acquiring low-resolution images from the imaging subsystem. This embodiment of the system can be further configured as set forth herein.

Another embodiment relates to a computer-implemented method for generating a high-resolution image of a sample from a low-resolution image of the sample. The method includes acquiring a low-resolution image of a sample. The method also includes generating a representation of the low-resolution image by inputting the low-resolution image into one or more first layers of a deep convolutional neural network. Additionally, the method includes generating a high-resolution image of the sample based on the representation. Generating the high-resolution image is performed by one or more second layers of the deep convolutional neural network. The one or more second layers include a final layer configured to output the high-resolution image. The final layer is configured as a subpixel convolutional layer. The steps of acquiring, generating the representation, and generating the high-resolution image are performed by one or more computer systems. One or more components are executed by the one or more computer systems, and the one or more components include the deep convolutional neural network.

Each of the steps of the method set forth above may be further performed as set forth further herein. Additionally, embodiments of the method set forth above may include any other steps of any other method set forth herein. Furthermore, the methods set forth above may be performed by any of the systems set forth herein.

Another embodiment relates to a non-transitory computer-readable medium storing program instructions executable on one or more computer systems for performing a computer-implemented method for A low-resolution image of the sample produces a high-resolution image of the sample. The computer-implemented method comprises the steps of the method set forth above. The computer-readable medium can be further configured as set forth herein. The steps of the computer-implemented method can be performed as further described herein. In addition, the computer-implemented method for which the program instructions may be executed may include any other steps of any of the other methods set forth herein.

Turning now to the drawings, it should be noted that the figures are not drawn to scale. In particular, the scale of some of the elements of the figures has been greatly exaggerated to emphasize the characteristics of the elements. It should also be noted that the figures are not drawn to the same scale. Elements shown in more than one figure that may be configured similarly have been designated using the same reference numerals. Unless otherwise mentioned herein, any of the elements illustrated and shown may comprise any suitable commercially available elements.

One embodiment relates to a system configured to generate a high-resolution image of a sample from a low-resolution image of the sample. As further described herein, embodiments provide platform-independent data-driven methods and systems to produce stable and robust metrology-quality images. Embodiments can also be used to generate relatively high quality, noise-removed, and super-resolved images. Embodiments may further be used to increase imaging throughput. Additionally, embodiments may be used to generate re-inspection images from relatively low frame, relatively low electrons per pixel (e/p) inspection scans. ("Low frame" means a smaller number of image captures at the same location, eg, for better imaging and increased signal-to-noise ratio, multiple frames are captured and then combined to improve image quality ."e/p" is basically electrons/pixel, where a higher e/p means higher quality but lower flux. Use beam conditions to achieve higher e/p.)

The embodiments described herein are applicable to electron beam (ebeam), broadband plasma (BBP), laser scattering, limited resolution, and metrology platforms for use with a much higher flux from them Images produced by either of the platforms produce relatively high quality images. In other words, images can be generated by an imaging system at relatively high throughput and thus relatively low resolution and then transformed into relatively high resolution images by the embodiments described herein, which means that relatively high throughput can be achieved amount to efficiently produce high-resolution images. Embodiments described herein advantageously provide learned transformations between relatively low-resolution imaging manifolds and relatively high-resolution imaging manifolds, noise reduction, and quality transitions from higher quality scans to lower quality scans. An imaging "manifold" can generally be defined as a theoretical probability space of all possible images.

The term "low-resolution image" of a sample, as used herein, is generally defined as an image in which all patterned features formed in the region of the sample where the image was generated are not resolved. For example, if some of the patterned features in the sample area that produced the low-resolution image are large enough in size to become resolvable, then the low-resolution image can be resolved. The patterned features. However, low-resolution images are not generated at a resolution that makes all patterned features in the image resolvable. In this way, the term "low-resolution image" as used herein does not include information about the patterned features on the sample sufficient to enable the low-resolution image to be used, such as for defect re-inspection (which may include Defect classification and/or verification) and metrology applications. Additionally, the term a "low-resolution image" as used herein generally refers to images produced by inspection systems, which are typically of relatively lower resolution (eg, lower than defect re-inspection and/or metrology systems) in order to have a relatively fast flux. In this way, a "low-resolution image" may also be commonly referred to as a high-throughput or HT image. For example, to generate images at higher throughput, e/p and frame number may be reduced, thereby resulting in lower quality scanning electron microscope (SEM) images.

A "low-resolution image" may also be "low-resolution" because it has a lower resolution than one of the "high-resolution images" described herein. The term a "high-resolution image" as used herein may generally be defined as an image in which all patterned features of a sample are resolved with relatively high accuracy. In this way, all patterned features in the sample area where the high-resolution image was generated are resolved in a high-resolution image, regardless of the size of the patterned features. Thus, the term "high-resolution image" as used herein contains information about the patterned features of the sample sufficient to enable the high-resolution image to be used, for example, for defect re-inspection (which may include defect classification and/or verification) and metrology applications. Additionally, the term "high resolution image" as used herein generally refers to images that cannot be produced during normal operation by inspection systems that are configured to sacrifice resolution capability for increased throughput. In this manner, a "high-resolution image" may also be referred to herein and in the art as a "high-sensitivity image," which is another term for a "high-quality image." For example, to produce high quality images, e/p, frames, etc. can be increased, which produces good quality SEM images but significantly reduces throughput. These images are then "high-sensitivity" images because they can be used for high-sensitivity defect detection.

In contrast to the embodiments described further herein, most older methods use heuristics and preferentially selected parameters to produce relatively noise-free images. These methods are usually designed keeping in mind the statistical nature of the images they will run on and thus cannot be ported to other platforms without incorporating the heuristics of that platform. Some of the well-known methods for noise reduction in images are anisotropic diffusion, bilateral filters, Weiner filters, non-area averaging, and the like. Bilateral and Wiener filters remove pixel-level noise by using a filter designed from adjacent pixels. Anisotropic diffusion applies the law of diffusion to an image whereby the anisotropic diffusion smoothes the texture/intensity in an image according to the diffusion equation. A bounding function is used to prevent diffusion from occurring across edges and thus the bounding function largely preserves the edges in the image.

A disadvantage of older methods such as Wiener and bilateral filtering is that these are parametric methods that require fine tuning at an image level for optimal results. These methods are not data-driven, which limits their achievable performance on challenging imaging types. Another limitation is done within the tether for most of its processing, which limits the use cases in which it can be used due to throughput limitations.

An embodiment of a system configured to generate a high-resolution image of a sample from a low-resolution image of the sample is shown in FIG. 1 . The system includes one or more computer subsystems (eg, computer subsystem 36 and computer subsystem 102) and one or more components 100 executed by the one or more computer subsystems. In certain embodiments, the system includes an imaging system (or subsystem) 10 configured to generate a low-resolution image of a sample. In the embodiment of FIG. 1, the imaging system is configured to scan or direct light to a physical version of the sample while detecting light from the sample to thereby generate an image of the sample . Imaging systems can also be configured to perform scanning (or guiding) and detection in multiple modes.

In one embodiment, the sample is a wafer. Wafers can include any wafers known in the art. In another embodiment, the sample is a reticle. The reticle can include any reticle known in the art.

In one embodiment, the imaging system is an optical-based imaging system. In one such example, in the embodiment of the system shown in FIG. 1 , the optics-based imaging system 10 includes an illumination subsystem configured to direct light to the sample 14 . The illumination subsystem includes at least one light source. For example, as shown in FIG. 1 , the illumination subsystem includes a light source 16 . In one embodiment, the illumination subsystem is configured to direct light to the sample at one or more angles of incidence (which may include one or more oblique angles and/or one or more normal angles). For example, as shown in FIG. 1 , light from light source 16 is directed through optical element 18 at an oblique angle of incidence and then through lens 20 to sample 14 . The oblique incidence angle can include any suitable oblique incidence angle, which can vary depending on, for example, the characteristics of the sample.

The imaging system can be configured to direct light to the sample at different angles of incidence at different times. For example, the imaging system can be configured to alter one or more characteristics of one or more elements of the illumination subsystem such that light can be directed to the sample at an angle of incidence different from that shown in FIG. 1 . In one such example, the imaging system can be configured to move light source 16, optical element 18, and lens 20 such that light is directed to the sample at a different oblique angle of incidence or a normal (or near-normal) incidence angle.

In some instances, the imaging system can be configured to direct light to the sample at more than one angle of incidence at the same time. For example, an illumination subsystem may include more than one illumination channel, one of which may include light source 16, optical element 18, and lens 20 as shown in FIG. 1, and another of the illumination channels Those (not shown) may include similar elements that may be configured differently or identically, or may include at least one light source and possibly one or more other components such as those set forth further herein. If this light is directed to the sample at the same time as other light, then one or more properties (eg, wavelength, polarization, etc.) of the light directed to the sample at different angles of incidence may be different such that the The light generated by the irradiation can be distinguished from each other at the detector.

In another example, the illumination subsystem may include only one light source (eg, source 16 shown in FIG. 1 ), and the light from the light source may be illuminated by one or more optical elements (not shown) of the illumination subsystem are separated into different optical paths (eg, based on wavelength, polarization, etc.). The light in each of the different optical paths can then be directed to the sample. Multiple illumination channels can be configured to direct light to the sample simultaneously or at different times (eg, when different illumination channels are used to sequentially illuminate the sample). In another example, the same illumination channel can be configured to direct light with different characteristics to the sample at different times. For example, in some instances, optical element 18 can be configured as a spectral filter, and the properties of the spectral filter can be changed in a number of different ways (eg, by swapping out the spectral filter) , so that different wavelengths of light can be directed to the sample at different times. The illumination subsystem may have any other suitable configuration known in the art for directing light of different or identical properties to the sample sequentially or simultaneously at different or identical angles of incidence.

In one embodiment, the light source 16 may comprise a broadband plasma (BBP) light source. In this way, the light generated by the light source and directed to the sample may comprise broadband light. However, the light source may comprise any other suitable light source, such as a laser. The laser may comprise any suitable laser known in the art and may be configured to generate light at any suitable wavelength or wavelengths known in the art. In addition, lasers can be configured to produce light that is monochromatic or nearly monochromatic. In this way, the laser can be a narrowband laser. The light source may also include a polychromatic light source that produces light at a plurality of discrete wavelengths or wavelength bands.

Light from optical element 18 can be focused onto sample 14 by lens 20 . Although lens 20 is shown in FIG. 1 as a single refractive optical element, it should be understood that, in practice, lens 20 may include several refractive and/or reflective optical elements that in combination focus light from the optical element to the sample. The illumination subsystem shown in FIG. 1 and described herein may include any other suitable optical elements (not shown). Examples of such optical elements include, but are not limited to, polarizers, spectral filters, spatial filters, reflective optical elements, apodizers, beam splitters, apertures, and the like, which may include those known in the art any such suitable optical element. Additionally, the imaging system can be configured to alter one or more of the elements of the illumination subsystem based on the type of illumination to be used for imaging.

The imaging system may also include a scanning subsystem configured to cause the light to scan the sample. For example, the imaging system may include stage 22 on which sample 14 is placed during inspection. The scanning subsystem can include any suitable mechanical and/or robotic assembly (including stage 22) that can be configured to move the sample so that light can scan the sample. Additionally or alternatively, the imaging system may be configured such that one or more optical elements of the imaging system perform some scan of the sample with light. The light may scan the sample in any suitable manner, such as in a serpentine path or in a helical path.

The imaging system further includes one or more detection channels. At least one of the one or more detection channels includes a detector configured to detect, by the system, light from the sample due to illumination of the sample and responsive to the detected light to produce output. For example, the imaging system shown in FIG. 1 includes two detection channels, one detection channel formed by collector 24, element 26 and detector 28 and the other detection channel formed by collector 30, element 32 and A detector 34 is formed. As shown in Figure 1, the two detection channels are configured to collect and detect light according to different collection angles. In some instances, the two detection channels are configured to detect scattered light, and the detection channels are configured to detect light scattered from the sample at different angles. However, one or more of the detection channels can be configured to detect another type of light (eg, reflected light) from the sample.

As further shown in Figure 1, the two detection channels are shown positioned in the plane of the paper and the illumination subsystem is also shown positioned in the plane of the paper. Thus, in this embodiment, the two detection channels are positioned (eg, centered) in the plane of incidence. However, one or more of the detection channels may be positioned outside the plane of incidence. For example, the detection channel formed by collector 30, element 32, and detector 34 can be configured to collect and detect light scattered outside the plane of incidence. Therefore, such a detection channel may be commonly referred to as a "side" channel, and this side channel may be centered in a plane substantially perpendicular to the plane of incidence.

Although FIG. 1 shows one embodiment of an imaging system that includes two detection channels, the imaging system may include a different number of detection channels (eg, only one detection channel or two or more detection channels). In one such example, the detection channel formed by collector 30, element 32, and detector 34 may form a side channel as described above, and the imaging system may include an additional detection channel (not shown), The additional detection channel is formed as another side channel positioned on the opposite side of the incident plane. Thus, an imaging system can include a detection channel that includes a collector 24, an element 26, and a detector 28 and is centered in the plane of incidence and configured to collect and detect in the normal to the sample surface or in proximity Light at a scattering angle towards the sample surface. Thus, this detection channel may be commonly referred to as a "top" channel, and the imaging system may also include two or more side channels configured as described above. As such, an imaging system can include at least three channels (ie, one top channel and two side channels), and each of the at least three channels has its own collector, each of which is grouped state to collect light at a different scattering angle than the scattering angle of the light collected by each of the other collectors.

As further set forth above, each of the detection channels included in the imaging system can be configured to detect scattered light. Accordingly, the imaging system shown in Figure 1 can be configured for dark field (DF) imaging of samples. However, the imaging system may also or alternatively include a detection channel configured for bright field (BF) imaging of the sample. In other words, the imaging system can include at least one detection channel configured to detect specularly reflected light from the sample. Accordingly, the imaging systems described herein can be configured for DF-only imaging, BF-only imaging, or both DF imaging and BF imaging. Although each of the collectors is shown in FIG. 1 as a single refractive optical element, it should be understood that each of the collectors may include one or more refractive optical elements and/or one or more reflective optical elements.

The one or more detection channels may comprise any suitable detector known in the art. For example, detectors may include photomultiplier tubes (PMTs), charge coupled devices (CCDs), time delay integration (TDI) cameras, and any other suitable detectors known in the art. Detectors may also include non-imaging detectors or imaging detectors. In this way, if the detectors are non-imaging detectors, each of the detectors can be configured to detect certain characteristics of scattered light, such as intensity, but not These features are detected from locations within the imaging plane. As such, the output generated by each of the detectors included in each of the detection channels of the imaging system may be a signal or data, but not an image signal or image data. In these examples, a computer subsystem, such as computer subsystem 36, may be configured to generate an image of the sample from the non-imaging output of the detector. However, in other examples, the detectors may be configured as imaging detectors that are configured to generate image signals or image data. Accordingly, the imaging system can be configured to produce the images described herein in a number of ways.

It should be noted that FIG. 1 is provided herein to generally illustrate one configuration of an imaging system or subsystem that may be included in or that may result from the system embodiments set forth herein The image used by the system embodiment. Obviously, the imaging system configurations described herein can be modified to optimize the performance of the imaging system, as is commonly performed when designing a commercial imaging system. Additionally, one of the existing systems such as the 29xx/39xx and Puma 9xxx series of tools commercially available from KLA-Tencor, Inc. of Milpitas, Calif., can be used (eg, by adding the functionality described herein to to an existing system) to implement the system described herein. For some of these systems, the embodiments described herein may be provided as optional functionality of the system (eg, in addition to other functionality of the system). Alternatively, the imaging system described herein can be designed "from scratch" to provide an entirely new imaging system.

The imaging system's computer subsystem 36 may be coupled to the imaging system's detectors in any suitable manner (eg, via one or more transmission media, which may include "wired" and/or "wireless" transmission media), such that the computer subsystem Output generated by the detector during scanning of the sample can be received. Computer subsystem 36 may be configured to use the outputs of the detectors to perform several of the functions described further herein.

The computer subsystem shown in FIG. 1 (as well as other computer subsystems described herein) may also be referred to herein as a computer system. Each of the computer subsystems or systems described herein may take various forms, including a personal computer system, video computer, mainframe computer system, workstation, network appliance, Internet appliance, or other device. In general, the term "computer system" can be broadly defined to encompass any device having one or more processors that executes instructions from a memory medium. The computer subsystem or system may also include any suitable processor known in the art, such as a parallel processor. In addition, the computer subsystem or system may include a computer platform with high-speed processing and software as a stand-alone tool or as a network-connected tool.

If the system includes more than one computer subsystem, the different computer subsystems can be coupled to each other such that images, data, information, instructions, etc. can be sent between the computer subsystems, as further described herein. For example, computer subsystem 36 may be coupled to computer subsystem 102 (as shown by the dashed lines in FIG. 1 ) by any suitable transmission medium, which may include any suitable wired and/or wireless transmission medium known in the art . Two or more of these computer subsystems can also be effectively coupled by a common computer-readable storage medium (not shown).

Although the imaging system is described above as being an optical or light based imaging system, the imaging system may be an electron beam based imaging system. In one such embodiment shown in FIG. 1 a , the imaging system includes an electron column 122 coupled to a computer subsystem 124 . As also shown in FIG. 1 a , the electron column includes an electron beam source 126 that is configured to generate electrons that are focused to a sample 128 by one or more elements 130 . The electron beam source may include, for example, a cathode source or emitter tip, and the one or more elements 130 may include, for example, a gun lens, an anode, a beam-limiting aperture, a gate valve, a beam of current An aperture, an objective lens, and a scanning subsystem are selected, all of which may comprise any such suitable elements known in the art.

Electrons (eg, secondary electrons) returning from the sample may be focused to detector 134 by one or more elements 132 . One or more elements 132 may include, for example, a scanning subsystem, which may be the same scanning subsystem included in element 130 .

The electron column may comprise any other suitable element known in the art. Alternatively, the electron column can be as described in US Patent Nos. 8,664,594 issued to Jiang et al. on April 4, 2014; US Patent Nos. 8,692,204 issued to Kojima et al. on April 8, 2014; Further configurations are set forth in U.S. Patent No. 8,698,093 to Gubbens et al. and U.S. Patent No. 8,716,662 to MacDonald et al., issued on May 6, 2014, which are generally incorporated by reference as if fully set forth herein. Incorporated.

Although the electron column is shown in Figure 1a configured such that electrons are directed to the sample at one oblique angle of incidence and scattered from the sample at another oblique angle, it should be understood that the electron beam may be directed to the sample and from the sample at any suitable angle. The sample scatters. Additionally, electron beam-based imaging systems can be configured to use multiple modalities to generate images of samples (eg, at different illumination angles, collection angles, etc.), as further described herein. The various modes of electron beam based imaging systems may differ in any image generation parameter of the imaging system.

Computer subsystem 124 may be coupled to detector 134 as described above. The detector detects electrons returning from the surface of the sample, thereby forming an electron beam image of the sample. The electron beam images may comprise any suitable electron beam images. Computer subsystem 124 may be configured to use the output generated by detector 134 to perform one or more functions for the sample as further described herein. Computer subsystem 124 may be configured to perform any of the additional steps set forth herein. A system including one of the imaging systems shown in Figure 1a may be further configured as set forth herein.

It should be noted that FIG. 1 a is provided herein to generally illustrate one configuration of an electron beam-based imaging system that may be included in the embodiments set forth herein. As with the optics-based imaging system described above, the configuration of the electron beam-based imaging system described herein can be modified to optimize the performance of the imaging system, as is commonly performed when designing a commercial imaging system. Additionally, one of the existing systems, such as the eSxxx and eDR-xxxx series of tools commercially available from KLA-Tencor Corporation, can be used to implement this document (eg, by adding the functionality described herein to an existing system). the described system. For some of these systems, the embodiments described herein may be provided as optional functionality of the system (eg, in addition to other functionality of the system). Alternatively, the system described herein can be designed "from scratch" to provide an entirely new system.

Although the imaging system is described above as being an optical-based imaging system or an electron beam-based imaging system, the imaging system may be an ion beam-based imaging system. Such an imaging system can be configured as shown in Figure 2, except that the electron beam source can be replaced with any suitable ion beam source known in the art. Additionally, the imaging system may be any other suitable ion beam-based imaging system, such as those included in commercially available Focused Ion Beam (FIB) systems, Helium Ion Microscopy (HIM) systems, and Secondary Ion Mass Spectrometry (SIMS) systems The ion beam based imaging system.

As described above, the imaging system is configured for scanning a physical version of the sample with energy (eg, light or electrons), thereby producing a true image of the physical version of the sample. In this way, the imaging system can be configured as a "real" system rather than a "virtual" system. For example, a storage medium (not shown) and computer subsystem 102 shown in Figure 1 may be configured as a "virtual" system. In particular, storage media and computer subsystems are not part of imaging system 10 and do not have any capability for handling physical versions of samples. In other words, in a system configured as a virtual system, the output of one or more of its "detectors" may be outputs previously generated by one or more detectors of a real system and stored in the virtual system, And during a "scan", the virtual system can replay the stored output as if the sample was being scanned. In this way, scanning a sample with a virtual system may appear to be the same as scanning a physical sample with a real system, when in reality "scanning" involves simply replaying the output of the sample in the same manner as a scannable sample. Systems and methods configured as "virtual" inspection systems described in Common Assignment to Bhaskar et al. on Feb. 28, 2012 and Common Assignment of US Pat. Nos. 8,126,255 and Dec. 29, 2015 to Duffy et al. As in US Pat. No. 9,222,895, these US patents are incorporated by reference as if fully set forth herein. The embodiments described herein can be further configured as described in these patents. For example, one or more of the computer subsystems described herein may be further configured as described in these patents. Additionally, configuring one or more virtual systems as a central computing and storage (CCS) system may be performed as described in the above-referenced patents issued to Duffy. The persistent storage mechanism described herein may have a distributed computing and storage device, such as a CCS architecture, but the embodiments described herein are not limited to that architecture.

As further described above, the imaging system can be configured to generate images of the sample in multiple modes. In general, a "mode" can be defined by an imaging system used to generate an image of a sample or by parameter values of the output used to generate an image of a sample. Thus, the different modes may differ in the value of at least one of the imaging parameters of the imaging system. For example, in one embodiment of an optical-based imaging system, at least one of the plurality of modes uses at least one wavelength of light for illumination that is different from that for one of the plurality of modes At least one wavelength of light used for illumination by at least one other. The modes can be different in illumination wavelength (eg, by using different light sources, different spectral filters, etc.), as further described herein for different modes. In another embodiment, at least one of the modalities uses an illumination channel of an imaging system that is different from an illumination channel of the imaging system used for at least another of the modalities. For example, as described above, an imaging system may include more than one illumination channel. As such, different illumination channels can be used for different modes.

In one embodiment, the imaging system is an inspection system. For example, the optical and electron beam imaging systems described herein can be configured as inspection systems. In another embodiment, the imaging system is a defect re-inspection system. For example, the optical and electron beam imaging systems described herein can be configured as defect re-inspection systems. In another embodiment, the imaging system is a metrology system. For example, the optical and electron beam imaging systems described herein can be configured as metrology systems. In particular, the embodiments of the imaging systems described herein and shown in Figures 1 and 1a may be modified in one or more parameters to provide different imaging capabilities depending on the application in which they will be used. In one such example, if the imaging system shown in FIG. 1 is to be used for defect re-detection or metrology rather than inspection, the imaging system may be configured to have a higher resolution. In other words, the embodiments of the imaging system shown in FIGS. 1 and 1a illustrate certain general and various configurations of an imaging system that can be tailored to produce a number of ways that will be apparent to those skilled in the art Imaging systems with different imaging capabilities more or less suitable for different applications.

One or more computer subsystems are configured for acquiring a low-resolution image of a sample. Acquiring low-resolution images can be performed using one of the imaging systems described herein (eg, by directing light or an electron beam to the sample and detecting light or an electron beam, respectively, from the sample). In this way, acquiring low-resolution images can be performed using the solid sample itself and some kind of imaging hardware. However, acquiring low-resolution images does not necessarily involve imaging the sample using imaging hardware. For example, another system and/or method may generate a low-resolution image and may store the generated low-resolution image in one or more storage media as set forth herein (such as a virtual inspection system) or herein in another storage medium as described. Accordingly, acquiring the low-resolution image may include acquiring the low-resolution image from the storage medium in which the low-resolution image has been stored.

In some embodiments, the low-resolution images are generated by an inspection system. For example, as set forth herein, low-resolution images may be generated by an inspection system configured to have a lower resolution to thereby increase its throughput. The inspection system can be an optical inspection system or an electron beam inspection system. The inspection system may have any configuration set forth further herein.

In one embodiment, the low resolution image is produced by an electron beam based imaging system. In another embodiment, the low-resolution image is produced by an optical-based imaging system. For example, low-resolution images may be produced by any of the electron beam-based imaging systems or the optical-based imaging systems described herein.

In one embodiment, low-resolution images are generated in a single mode of an imaging system. In another embodiment, one or more low-resolution images of the sample are generated in multiple modes of an imaging system. For example, a low-resolution image input to a deep convolutional neural network (deep CNN) as set forth further herein may include generating a single low-resolution image in only a single mode of the imaging system. Alternatively, the low-resolution images input to a deep CNN as further described herein may include low-resolution images generated in multiple modes of the imaging system (eg, a first mode generated in a first mode). an image, generating a second image in a second mode, etc.). Single mode and multiple modes may include any of the modes set forth further herein.

Components (eg, component 100 shown in FIG. 1 ) executed by computer subsystems (eg, computer subsystem 36 and/or computer subsystem 102 ) include deep CNN 104 . The deep CNN includes one or more first layers configured for generating a representation of a low-resolution image and one or more first layers configured for generating a high-resolution image of a sample from the representation of the low-resolution image, or Multiple second layers. In this manner, embodiments described herein may use one of the deep CNNs (eg, one or more machine learning techniques) described herein for transforming a low-resolution image of a sample into the High-resolution image of one of the samples. For example, as shown in FIG. 2, a deep CNN is shown as an image transformation network 200. During generation and/or runtime (ie, after the image transformation network has been set up and/or trained (which may be performed as further described herein)), the input to the image transformation network may be an input The low-resolution (high-throughput) image 202, and the output of the image transformation network may be the output high-resolution (high-sensitivity) image 204.

The one or more second layers include a final layer configured to output a high-resolution layer, and the final layer is configured as a subpixel convolutional layer. 3 illustrates one embodiment of an image transformation network architecture that may be suitable for use in the embodiments described herein. In this embodiment, the image transformation network is a deep CNN with a sub-pixel layer as the final layer. In this architecture, the input may be a low-resolution image 300, which is shown in FIG. 3 as just a grid of pixels and does not represent any particular low-resolution that can be produced by the embodiments described herein degree image. The low-resolution images may be input to one or more first layers 302 and 304, which may be configured as convolutional layers configured for feature map extraction. These first layers may form the hidden layers of the image transformation network architecture.

The representation of the low-resolution image generated by the one or more first layers may thus be one or more features and/or feature maps. Features can be of any suitable feature type known in the art that can be inferred from the input and used to generate the output set forth further herein. For example, a feature may include a vector of intensity values per pixel. Features may also include any other type of feature set forth herein, such as a vector of scalar values, a vector of independent distributions, a vector of joint distributions, or any other suitable feature type known in the art. As set forth further herein, these features are learned by the network during training and may or may not be related to any real features known in the art.

The one or more second layers include the final layer 306, which is configured as a subpixel convolutional layer that aggregates feature maps from low-resolution space and builds high-resolution in a single step Image 308. The sub-pixel convolutional layer learns to upscaling an array of filters to upscale the final low-resolution feature map to a high-resolution output image. In this way, the image transformation network can take a high-throughput input image with poor resolution of noise, compute feature maps across many convolutional layers, and then use sub-pixel layers to transform the feature maps into relatively quiet super-resolution images. . The sub-pixel convolutional layer advantageously provides a relatively complex lifting filter specially trained for each feature map, while also reducing the computational complexity of the overall operation. The deep CNNs used in the embodiments set forth herein may be further as described in "Real-Time Single Image and Video Super-Real-Time Single Image and Video Super- Resolution Using an Efficient Sub-Pixel Convolutional Neural Network", configured as described in arXiv:1609.05158v2.

The deep CNNs described herein can be generally classified as deep learning models. In general, "deep learning" (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data. In a simple case, there may be two groups of neurons: one group of neurons receives an input signal and one group of neurons sends an output signal. When the input layer receives an input, the input layer passes a modified version of the input to the next layer. In a deep network, there are layers between the input and output (and the layers are not composed of neurons, but it may be helpful to think of the layers in this way), allowing the algorithm Use multiple processing layers consisting of multiple linear and nonlinear transformations.

Deep learning is part of a broader family of machine learning methods of data-based learning representations. An observation (eg, an image) can be represented in a number of ways (such as a vector of intensity values per pixel) or in a more abstract way as a set of edges, a region of a particular shape, etc. Certain representations outperform others when simplifying learning tasks (eg, facial recognition or facial expression recognition). One of the prospects for deep learning is to replace handcrafted features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction.

Research in this area attempts to make better representations and form models to learn these representations from large-scale unlabeled data. Some of these representations are inspired by advances in neuroscience and are loosely based on patterns of information processing and communication in a nervous system, such as neural coding, which attempt to define various stimuli and associated neuronal responses in the brain a relationship between).

The deep CNNs described in this paper can also be classified as machine learning models. Machine learning can generally be defined as a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that guide their own growth and change when exposed to new data. In other words, machine learning can be defined as the subfield of computer science that "endows computers the ability to learn without being explicitly programmed". Machine learning explores the study and construction of algorithms that can learn from data and make predictions about data, by building a model from sample input, and by making data-driven predictions or decisions that overcome the strictly static procedures that follow instruction.

The machine learning described in this article can be further performed as described in: "Introduction to Statistical Machine Learning" by Sugiyama, Morgan Kaufmann, 2016, p. 534; "Discriminative, Generative, and Imitative Learning" by Jebara , MIT Paper, 2002, p. 212; and Hand et al., "Principles of Data Mining (Adaptive Computation and Machine Learning)", MIT Press, 2001, p. 578; Generally incorporated by reference. The embodiments set forth herein may be further configured as set forth in such references.

Deep CNN is also a generative model. A "production" model can generally be defined as one that is probabilistic in nature. In other words, a "production" model is not a model that performs a forward simulation or rule-based approach and as such, a physical model of the process involved in generating a real image for which a simulated image is generated is not necessary . Alternatively, as set forth further herein, a production model (in that its parameters can be learned) can be learned based on a suitable training data set.

In one embodiment, the deep CNN is a deep generative model. For example, a deep CNN can be configured to have a deep learning architecture in that the model can include multiple layers that perform several algorithms or transforms. The number of layers on one or both sides of a deep CNN can vary from the number of layers shown in the figures set forth herein. For practical purposes, a suitable range of layers on either side is from 2 layers to dozens of layers.

A deep CNN can also be a deep neural network with a set of weights that models the world based on the data it is fed for training. A neural network can generally be defined as an algorithm based on a relatively large collection of neural units that loosely models the way a biological brain solves problems using relatively large clusters of biological neurons connected by axons . Each neuronal unit is connected to many other neuronal units, and the connection can enforce or inhibit its effect on the activation state of the connected neuronal unit. These systems are self-learning and trained rather than explicitly programmed, and excel in areas where solutions or feature detection are difficult to express with a traditional computer program.

Neural networks typically consist of multiple layers with signal paths traversing from front to back. The goal of neural networks is to solve problems in the same way as the human brain, although several are much more abstract. Modern neural network schemes typically work with thousands to millions of neural units and millions of connections. The neural network may have any suitable architecture and/or configuration known in the art.

Embodiments described herein may or may not be configured to train a deep CNN for generating a high-resolution image from a low-resolution image. For example, another method and/or system can be configured to generate a trained deep CNN that can then be accessed and used by the embodiments described herein. In general, training a deep CNN may include acquiring data (eg, both low- and high-resolution images, which may include any of the low- and high-resolution images described herein). A list of training, testing, and validation data sets can then be constructed using a list of input tuples and expected output tuples. The input tuple may be in the form of a low-resolution image, and the output tuple may be a high-resolution image corresponding to the low-resolution image. The training dataset can then be used to train the deep CNN.

In one embodiment, the one or more components include a context-aware loss module configured to train the deep CNN, and during the training of the deep CNN, the one or more computer subsystems will be run by one or more second One of the high-resolution images and samples generated by the layer corresponding to the known high-resolution images is input into the context-aware loss module, and the context-aware loss module determines that the high-resolution images generated by the one or more second layers correspond to the corresponding known high-resolution images. Contextual awareness loss compared to high-resolution imagery. For example, as shown in FIG. 4, a deep CNN network is shown as image transformation network 400. This figure shows a deep CNN during training or at setup time. The input to the image transformation network is a low-resolution (high-throughput) image 402, which can be generated as described further herein. The image transformation network may then output a high-resolution (high-sensitivity) image 404, as further described herein. The output high-resolution image and corresponding known high-resolution image (eg, a "ground truth" high-sensitivity image) 406 may be input to the context-aware loss module 408 . In this way, the complete network architecture of the embodiments described herein may include two blocks: the image transformation network and the context-aware loss. The context-aware loss module 408 can compare the two images it receives as input (ie, a high-resolution image generated by an image transformation network and a high-resolution image generated, for example, by an imaging system to identify the live image) to determine one or more differences between the two input images. The context-aware loss module can be further configured as set forth herein.

In this way, at setup time, embodiments take image pairs with noisy poor resolution and quiet super-resolution and then learn the transformation matrix between these images through a neural network using context-aware loss. The term "noisy" as used herein may generally be defined as an image having a relatively low signal-to-noise ratio (SNR), while the term "quiet" as used herein may generally be defined as having a relatively high One of the SNR images. These terms are therefore used interchangeably herein. These image pairs can be from any of the imaging platforms commercially available from KLA-Tencor Corporation (and others), such as electron beam (ebeam), BBP tools, a limited resolution imaging tool, and the like. Once trained, the network learns the transformation of its own noisy, poorly resolved images to quiet, super-resolved images, while maintaining spatial fidelity. In this manner, the embodiments described herein use a data-driven approach to exploit the data redundancy observed in semiconductor images by learning a transformation between noisy, poorly resolved images and quiet super-resolved images . The trained network can then be deployed in production, where the imaging system generates noisy high-throughput data, which is then transformed into data corresponding to low-noise super-resolution using a trained image transformation network . Once in production, the network executes like a typical post-processing algorithm.

In one such embodiment, the context-aware loss includes content loss, style loss, and total variation (TV) regularization. Figure 5 shows one such embodiment. In particular, the context-aware loss module 408 shown in FIG. 4 may include a content loss module 500, a style loss module 502, and a TV regularization module 504, as shown in FIG. For example, context-aware loss is a generic framework and is expressed through style and content loss. Deep neural networks tend to gradually learn image features starting from edges, shapes in lower layers, up to more complex features such as faces, or possibly entire objects in subsequent layers. This correlates well with biological vision. We assume that the learning of lower layers of a convolutional network is regarded as a perceptually important feature. Therefore, we design our context-aware loss based on the activation of the learned network. The context-aware loss mainly consists of style, content and regularization losses.

In one such embodiment, the content loss includes a loss in low-level features corresponding to known high-resolution images. For example, the content of an image is defined as lower-level features such as edges, shapes, and the like. The miniaturization of content loss helps preserve these low-level features that are important for producing metrology-quality, super-resolution imagery. For clarity, the content loss is included in the loss function to preserve edges and shapes in the image, which are therefore important for measurements on high-resolution images, etc. Conventional techniques such as bicubic interpolation, etc. or training with an L2 loss do not need to guarantee this preservation of edges and shapes.

The next major part of the loss is called the style shift loss. In one such embodiment, the style loss includes qualitatively defining a loss in one or more abstract entities corresponding to known high-resolution images. For example, we define style as an abstract entity that qualitatively defines an image, including properties such as sharpness, texture, color, and so on. One reason to use deep learning as described in this paper is that the difference between low-resolution images/high-resolution images described in this paper is not only resolution, but the images can have different noise characteristics, charging artifacts Shadows (charging artifacts), textures, etc. Therefore, it is not sufficient to super-resolve low-resolution images only, and deep learning is used to learn a mapping from low-resolution images to high-resolution images. The style of an image is represented by an upper layer activation of a trained network. The combination of style loss and content loss allows the image transformation network to learn transformations between noisy, poorly resolved images and quiet super-resolution images. Once an image transformation network is trained with a context-aware loss, the image transformation network can be deployed in production to produce quiet super-resolution images from noisy, poorly resolved images while maintaining spatial fidelity. In some embodiments, the style transfer loss is defined as the difference between the final layer features of the super-resolution high-resolution image (ie, the image generated by one or more second layers) and the identified live high-resolution image loss, especially when we want to classify super-resolved high-resolution images.

In another such embodiment, the context-aware loss module includes a pretrained VGG network. Figure 6 shows how activation from a predefined network is used to calculate style and content loss. For example, as shown in FIG. 6 , pretrained VGG network 600 may be coupled to content loss module 500 and style loss module 502 . VGG16 (also known as OxfordNet) is a convolutional neural network architecture named after the Visual Geometry Group developed by Oxford University. The VGG network may also be further incorporated by reference as in "Very Deep Convolutional Networks for Large-Scale Image Recognition" by Simonyan et al., April 2015, April 2015, as fully set forth herein, arXiv:1409.1556v6, p. configured as described on page 14. As shown in Figure 6, a pretrained VGG network can input an image to several layers, including convolutional layers (eg, conv-64, conv-128, conv-256, and conv-512), max pooling ( maxpool layer, fully connected layers (eg, FC-4096), and a softmax layer, all of which may have any suitable configuration known in the art.

The activation from the VGG network may be obtained by the content loss module 500 and the style loss module 502 to calculate the style and content loss therefrom. Embodiments described herein thus define a novel loss framework for training neural networks using pretrained networks. This helps optimize the neural network while preserving use-case critical characteristics in the resulting image.

Accordingly, the embodiments described herein use a pretrained deep learning network to introduce a use case dependent loss function. Traditional techniques include methods such as bicubic interpolation and L2 loss (when training deep networks), but we introduce different losses during training of our network. For example, bicubic interpolation reduces contrast loss on sharp edges, while L2 loss on full images focuses on preserving all aspects of an image, although preservation of most aspects is not necessarily the case for the embodiments described herein One of the use cases requires and we can build loss functions depending on which features in an image we want to preserve. In some such instances, content loss may be used to ensure that edges and shapes are preserved, and style loss may be used to ensure that texture, color, etc. are preserved.

Embodiments described herein may use the output from a pretrained network layer to define a use-case-dependent loss function for training a network. If the use case is critical size consistency or metrology measurements, embodiments may weight the content loss, and if the image is to be "beautified", the style loss may be used to preserve texture, color, etc. Additionally, for situations where classification is important, the last layer features can be matched between the generated high-resolution image and the identified live image, and the loss on the last layer features of the pretrained network can be defined, etc. A feature used for classification.

In signal processing, total variation denoising (also known as total variation regularization), one of the most commonly used procedures in image processing of coefficient bits, is applied to noise removal. The procedure is based on the principle that signals with excessive and possibly spurious details have high total variation, ie the integral of the absolute gradient of the signal is high. According to this principle, reducing the total variation of the signal in order to closely match the signal to the original signal removes unwanted details while preserving important details such as edges. This concept was pioneered by Rudin, Osher and Fatemi in 1992 and is therefore known today as the ROF model.

This noise removal technique has advantages over simpler techniques such as linear smoothing or median filtering, which reduce noise but at the same time eliminate edges to a greater or lesser extent. In contrast, total variation denoising is significantly effective at simultaneously preserving edges and removing noise in flat areas (even at relatively low signal-to-noise ratios).

In certain such embodiments, the one or more components include a tuning module configured to determine one or more parameters of the deep CNN based on the contextual awareness loss. For example, as shown in FIG. 4, one or more components may include a tuning module 410 configured for back-propagating errors and/or changes determined by the context-aware loss module network parameters. Each of the layers of the deep CNN described above may have one or more parameters (such as weights and biases B) that may be determined by training a model, which may be performed as further described herein the value of the one or more parameters. For example, the weights and biases of various layers included in a deep CNN can be determined during training by minimizing the context-aware loss.

In one embodiment, the deep CNN is configured such that high-resolution images produced by one or more second layers have less noise than low-resolution images. For example, the embodiments described herein provide a generalized framework for transforming noisy and under-resolved images into low-noise super-resolved images using learned representations.

In another embodiment, the deep CNN is configured such that high-resolution images generated by one or more second layers retain the structural and spatial characteristics of low-resolution images. For example, the embodiments described herein provide a generalized framework for using learned representations to transform noisy and under-resolved images into low-noise super-resolved images while preserving structural and spatial fidelity Spend.

In some embodiments, the deep convolutional neural network outputs high-resolution images at a higher throughput than is used to generate high-resolution images using a high-resolution imaging system. For example, the embodiments set forth herein can be used for deep learning based super-resolution to achieve higher throughput on e-beam tools. Thus, the embodiments described herein may be particularly useful when it may be advantageous to use a relatively low dose (electron beam, light, etc.) for image acquisition to prevent alterations (such as damage, contamination, etc.) to the sample . However, using a relatively low dose to avoid making changes to the sample generally produces low-resolution images. Therefore, the challenge is to generate high-resolution images without altering the sample. The embodiments described herein provide this capability. In particular, sample images can be acquired at a higher throughput and at a lower resolution (or lower quality), and the embodiments described herein can convert these higher throughput, lower quality images into ultra-high-quality images. Resolved or higher quality images without altering the sample (since the sample itself does not need to produce super-resolved or higher quality images).

Accordingly, the embodiments described herein are particularly advantageous for re-inspection use cases where a wafer may undergo inspection (eg, BBP inspection) and an e-beam re-inspection sequence. Additionally, in some instances, the user wants to put the wafer back on the inspection tool after inspection to try another inspection recipe condition (eg, to optimize for being detected during inspection and possibly re-inspection). inspection recipe conditions for a defect classified during inspection). However, if the e-beam (or other) re-detection damages or changes the locations that are re-detected, those sites are no longer valid for sensitivity analysis (ie, checking for recipe changes and/or optimization). Thus, preventing damage to samples or alterations by using low-frame-averaged e-beam image acquisition is one of the advantages of deep-learning-based classification of e-beam re-inspection images (eg, no need for original high-frame-averaged images) ). Thus, deep learning classification and deep learning image refinement can arguably be used in combination. Deep learning-based defect classification can be performed by the embodiments described herein, as set forth in commonly assigned US Patent Application Serial No. 15/697,426 filed by He et al on Sep. 6, 2017 , this US patent application is generally incorporated by reference as if fully set forth herein. The embodiments described herein can be further configured as described in this patent application.

Figure 7 illustrates an example of the results that can be produced using the embodiments set forth herein. The results show a horizontal profile 700 of a high-throughput image 702 with poorer resolution of noise, a horizontal profile 704 of a low-throughput image 706 with higher quality better resolution, and a low-resolution image by using the embodiments described herein. And a comparison between the horizontal contours 708 of the quiet super-resolved image 710 obtained. High-throughput image 702 and low-throughput image 706 are produced by a low-resolution imaging system and a high-resolution imaging system, respectively, as described herein. In this way, the results shown in FIG. 7 illustrate the horizontal variation between different images along the same line profile through the images. The results shown in Figure 7 demonstrate the ability of the embodiments described herein to generate substantially noise-free high-resolution images from lower quality images while maintaining structural and spatial fidelity in the images, as described herein The correlations in the contours (708 and 704) of the super-resolved images produced by the embodiments described in and of the high-resolution images produced by an imaging system were confirmed.

In one embodiment, the one or more computer subsystems are configured to perform one or more metrological measurements on the sample based on the high-resolution images generated by the one or more second layers. FIG. 8 demonstrates that the embodiments described herein work by closing loops with recognized live data. To further test the embodiments described herein in real-world metrology use cases, overlay measurements were performed on the three sets of images shown in FIG. 7 and the results compiled in FIG. 8 . Graphs 800 and 802 in FIG. 8 illustrate overlays along the x-axis and y-axis, respectively, between an image produced by a high-resolution imaging system and a high-resolution image produced by a deep CNN of embodiments described herein Correlation in measurements, and graphs 804 and 806 in FIG. 8 depict the overlay measurements along the x-axis and y-axis, respectively, between the image produced by the high-resolution imaging system and the lower-resolution image Correlation. The metric used to calculate the correlation is R2, r- ^squared . An r-squared value of 1 depicts a perfect fit. Near ^- perfect R2 values (>0.99) between the high-resolution images produced by the imaging system and those produced by deep CNNs demonstrate that higher-resolution imaging can be substituted in metrology measurements without compromising performance system-generated images using deep CNNs. Given the relatively high accuracy required in metrology use cases, an R value of ~0.8 in the case of images produced by low-resolution imaging systems and images produced by high ^- resolution imaging systems proved too low to obtain accurate measurements, And therefore measurements from higher resolution images that significantly reduce use case throughput (eg, from about 18K defects/hour to about 8K defects/hour in the experiments described herein) are required.

In another embodiment, the deep CNN functions independently of the imaging system that produces the low-resolution images. In certain embodiments, low-resolution images are generated by an imaging system having a first imaging platform, and one or more computer subsystems are configured for acquisition by a second imaging system having a different first imaging platform Another low-resolution image produced by another imaging system of the imaging platform for another sample, the one or more first layers configured for producing a representation of the other low-resolution image, and the one or more first layers The two layers are configured for generating a high-resolution image of another sample from a representation of another low-resolution image. For example, one important benefit of the embodiments described herein is that the same network architecture can be used to enhance images from different platforms (eg, BBP tools, tools specifically configured for low resolution imaging, etc.). Furthermore, the entire burden of optimizing and learning representations is taken offline, since training only happens during recipe setup time. Once training is complete, the runtime operations are drastically reduced. The learning procedure also helps to adaptively enhance the image without changing parameters each time as was required in the case of the old method.

In one such embodiment, the first imaging stage is an electron beam imaging stage and the second imaging stage is an optical imaging stage. For example, the embodiments described herein may use an electron beam imaging system and an optical imaging system to transform the resulting low-resolution image. Embodiments described herein are also capable of performing transformations for other different types of imaging platforms (eg, other charged particle type imaging systems).

In another such embodiment, the first imaging platform and the second imaging platform are different optical imaging platforms. In another such embodiment, the first imaging platform and the second imaging platform are different electron beam imaging platforms. For example, the first imaging platform and the second imaging platform may be the same type of imaging platform, but may differ significantly in their imaging capabilities. In one such example, the first optical imaging stage and the second optical imaging stage may be a laser scattering imaging stage and a BBP imaging stage. These imaging platforms obviously have substantially different capabilities and will produce substantially different low-resolution images. However, the embodiments described herein may use learned representations generated by training a deep CNN to generate high-resolution images of all such low-resolution images.

Another embodiment of a system configured to generate a high-resolution image of a sample from a low-resolution image of the sample includes an imager configured to generate a low-resolution image of a sample system. The imaging subsystem can have any of the configurations set forth herein. The system also includes one or more computer subsystems (eg, computer subsystem 102 shown in FIG. 1 ) that can be configured as further described herein and executed by the one or more computer subsystems A component (eg, component 100), the one or more components may include any of the components set forth herein. Components include a deep CNN (eg, deep CNN 104) that can be configured as set forth herein. For example, a deep CNN includes one or more first layers configured for generating a representation of a low-resolution image and a high-resolution one configured for generating a sample from the representation of the low-resolution image Image one or more second layers. The one or more second layers include a final layer configured to output high-resolution images. The final layer is also configured as a subpixel convolutional layer. The one or more first layers and the one or more second layers may be further configured as set forth further herein. This system embodiment may be further configured as set forth herein.

The embodiments set forth herein have several advantages as can be seen from the description provided above. For example, the embodiments described herein provide a generic, platform-independent, data-driven framework. Embodiments use training data during set-up time to learn the transformation between high-quality and low-quality images. Learning this transform enables an embodiment to use the learned transform to transform a noisy, poorly resolved input into a relatively quiet super-resolved output with metrological quality. Older methods are parametric methods that rely only on the current input image and do not utilize any other training data. The embodiments described herein are also generic and platform independent. Because the embodiments are generic and platform-independent, the same framework can be used to produce metrology-quality images on different platforms, such as electron beam, BBP, laser scattering, low-resolution imaging, and metrology platforms. Embodiments also achieve higher throughput by using only low quality (high throughput) images to produce the desired quality images in production. Embodiments also achieve noise reduction in the output image compared to the input image without affecting important features such as edges and contours in the image.

Each of the embodiments of each of the systems set forth above can be combined together into a single embodiment.

Another embodiment relates to a computer-implemented method for generating a high-resolution image of a sample from a low-resolution image of the sample. The method includes acquiring a low-resolution image of a sample. The method also includes generating a representation of the low-resolution image by inputting the low-resolution image into one or more first layers of a deep CNN. Additionally, the method includes generating a high-resolution image of the sample based on the representation. The high-resolution images are generated by performing one or more second layers of the deep CNN. The one or more second layers include a final layer configured to output high-resolution images, and the final layer is configured as a subpixel convolutional layer. The steps of acquiring, generating representations, and generating high-resolution images are performed by one or more computer systems. One or more components are executed by one or more computer systems, and the one or more components include a deep CNN.

Each of the steps of the method can be performed as set forth further herein. The method may also include any other steps that may be performed by the systems, computer systems or subsystems and/or imaging systems or subsystems set forth herein. One or more computer systems, one or more components, and a deep CNN may be configured according to any of the embodiments set forth herein (eg, computer subsystem 102, component 100, and deep CNN 104). Additionally, the methods set forth above may be performed by any of the system embodiments set forth herein.

All methods described herein can include storing the results of one or more steps of an embodiment of the method in a computer-readable storage medium. The results can include any of the results set forth herein and can be stored in any manner known in the art. Storage media may include any storage media set forth herein or any other suitable storage media known in the art. After the results have been stored, the results can be accessed in a storage medium and used by any of the method or system embodiments described herein, formatted for display to a user, by another Use of software modules, methods or systems, etc. For example, the resulting high-resolution images can be used to perform metrological measurements on the sample, classify one or more defects detected on the sample, verify that one or more defects were detected on the sample, and/or or based on one or more of the above to determine whether the procedure used to form patterned features on a sample should be altered in some way to thereby alter patterned features formed on other samples in the same procedure .

An additional embodiment pertains to a non-transitory computer-readable medium storing program instructions executable on one or more computer systems for performing a computer-implemented method for use by the same A low-resolution image of the sample produces a high-resolution image of the sample. One such embodiment is shown in FIG. 9 . In particular, as shown in FIG. 9 , non-transitory computer-readable medium 900 includes program instructions 902 executable on computer system 904 . A computer-implemented method may include any steps of any of the methods set forth above.

Program instructions 902 implementing methods such as those described herein may be stored on computer-readable medium 900 . The computer-readable medium may be a storage medium such as a magnetic or optical disk, a magnetic tape, or any other suitable non-transitory computer-readable medium known in the art.

Program instructions may be implemented in any of a variety of ways, including program-based techniques, component-based techniques, and/or object-oriented techniques, as well as other techniques. For example, ActiveX controls, C++ objects, JavaBeans, Microsoft Foundation Classes ("MFC"), SSE (Streaming SIMD Extensions), or other technologies or methods may be used to implement such program instructions as desired.

Computer system 904 may be configured according to any of the embodiments set forth herein.

In view of this description, other modifications and alternative embodiments of the various aspects of the invention will become apparent to those skilled in the art. For example, methods and systems are provided for generating a high-resolution image of a sample from a low-resolution image of the sample. Accordingly, this description should be regarded as illustrative only and for the purpose of teaching those skilled in the art the general manner of carrying out the invention. It is to be understood that the forms of the invention shown and described herein are to be considered presently preferred embodiments. Elements and materials may be substituted for those illustrated and described herein, parts and procedures may be reversed, and certain aspects of the invention may be substituted, as will be fully apparent to those skilled in the art having the benefit of this description of the invention Features can be utilized independently. Changes may be made in elements set forth herein without departing from the spirit and scope of the invention as set forth in the appended claims.

10‧‧‧Imaging System/Imaging Subsystem/Optical-Based Imaging System 14‧‧‧Sample 16‧‧‧Light Source/Source 18‧‧‧Optics 20‧‧‧Lens 22‧‧‧Stand 24‧‧‧Collecting Detector 26‧‧‧Component 28‧‧‧Detector 30‧‧‧Collector 32‧‧‧Component 34‧‧‧Detector 36‧‧‧Computer Subsystem 100‧‧‧Component 102‧‧‧Computer Subsystem 104‧‧‧Deep Convolutional Neural Network 122‧‧‧Electron Column 124‧‧‧Computer Subsystem 126‧‧‧Electron Beam Source 128‧‧‧Sample 130‧‧‧Component 132‧‧‧Component 134‧‧‧Detection 200‧‧‧Image Transformation Network 202‧‧‧Low Resolution Image/High Throughput Image 204‧‧‧High Resolution Image/High Sensitivity Image 300‧‧‧Low Resolution Image 302‧‧‧First Layer 304‧‧‧First layer 306‧‧‧Final layer 308‧‧‧High-resolution image 400‧‧‧Image transformation network 402‧‧‧Low-resolution image/high-throughput image 404‧‧‧High-resolution image /High Sensitivity Image 406‧‧‧High Resolution Image/Recognized Live High Sensitivity Image 408‧‧‧Context Awareness Loss Module 410‧‧‧Tuning Module 500‧‧‧Content Loss Module 502‧‧‧Style Loss Module 504‧‧‧Total Variation Regularization Module 600‧‧‧Pre-trained Visual Geometry Network 700‧‧‧Horizontal Contouring 702‧‧‧High-throughput Image/High-Throughput Image with Poor Resolution of Noise 704‧‧‧Horizontal contour 706‧‧‧Higher quality and better resolution low-throughput image/low-throughput image 708‧‧‧Horizontal contour 710‧‧‧Quiet super-resolution image 800‧‧‧Graph 802‧‧‧ Exhibit 804‧‧‧Exhibit 806‧‧‧Exhibit 900‧‧‧Non-transitory computer readable medium/computer readable medium 902‧‧‧Program instructions 904‧‧‧Computer system

Other advantages of the present invention will become apparent to those skilled in the art having the benefit of the following detailed description of the preferred embodiments and after reference to the accompanying drawings, in which: Figures 1 and 1a are illustrated as herein A schematic diagram of a side view of an embodiment of a system as described and configured; FIG. 2 is a block diagram illustrating an embodiment of a deep convolutional neural network that may be included in the embodiments described herein; Figure 3 illustrates a schematic diagram of one embodiment of a deep convolutional neural network that may be included in the embodiments described herein; Figures 4 and 5 illustrate embodiments that may be included in the embodiments described herein A block diagram of an embodiment of one or more components in; FIG. 6 is a block diagram illustrating an embodiment of a pretrained VGG network that may be included in a context-aware loss module embodiment; FIG. 7 Including corresponding high and low resolution images produced by an imaging system and a high resolution image produced from the low resolution image by the embodiments described herein and the contours produced for each of the images Examples; FIG. 8 includes between a high-resolution image produced by an imaging system and a high-resolution image produced by the embodiments described herein, and a low-resolution image and a high-resolution image produced by the imaging system Examples of correlations in results of overlay measurements taken along the x- and y-axes between degree images; and FIG. 9 illustrates one embodiment of a non-transitory computer-readable medium storing program instructions A block diagram of program instructions for causing one or more computer systems to perform a computer-implemented method described herein. While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail herein. The drawings may not be drawn to scale. It should be understood, however, that the drawings and detailed description thereof are not intended to limit the invention to the particular form disclosed, but on the contrary, this invention is to encompass the spirit attributable to the invention as defined by the appended claims and all modifications, equivalents and alternatives within the scope.

10‧‧‧Imaging system/imaging subsystem/optical-based imaging system

14‧‧‧Sample

16‧‧‧Light Source/Source

18‧‧‧Optics

20‧‧‧Lens

22‧‧‧Platform

24‧‧‧Collector

26‧‧‧Components

28‧‧‧Detector

30‧‧‧Collector

32‧‧‧Components

34‧‧‧Detector

36‧‧‧Computer Subsystem

100‧‧‧Components

102‧‧‧Computer subsystem

104‧‧‧Deep Convolutional Neural Networks

Claims (23)

A system configured to generate a high-resolution image of a sample from a low-resolution image of the sample, comprising: one or more computer subsystems configured for acquiring a low-resolution image of a sample high-resolution images; and one or more components executed by the one or more computer subsystems, wherein the one or more components include: a deep convolutional neural network, wherein the deep convolutional neural network includes : one or more first layers configured for generating a representation of the low-resolution image; one or more second layers configured for the representation from the low-resolution image generating a high-resolution image of the sample, wherein the one or more second layers include a final layer configured to output the high-resolution image, and wherein the final layer is further configured as a subpixel convolutional layer , and wherein the one or more components further include a context aware loss module configured to train the deep convolutional neural network, wherein during training of the deep convolutional neural network, the one or more The computer subsystem inputs the high-resolution image generated by the one or more second layers and a corresponding known high-resolution image of the sample into the context-awareness loss module, and the context-awareness loss module determines that the A loss of context awareness in the high-resolution image generated by one or more second layers compared to the corresponding known high-resolution image. The system of claim 1, wherein the deep convolutional neural network is configured such that the high resolution image generated by the one or more second layers has less noise than the low resolution image. The system of claim 1, wherein the deep convolutional neural network is configured such that the high-resolution images generated by the one or more second layers retain the structural and spatial characteristics of the low-resolution images. The system of claim 1, wherein the context-aware loss includes content loss, style loss, and total variation regularization. The system of claim 4, wherein the content loss includes a loss in the low-level features corresponding to known high-resolution images. The system of claim 4, wherein the stylistic loss includes qualitatively defining a loss in one or more abstract entities corresponding to the known high-resolution images. The system of claim 1, wherein the context-aware loss module includes a pretrained VGG network. The system of claim 1, wherein the one or more components further comprise a tuning module configured to determine one or more parameters of the deep convolutional neural network based on the situational awareness loss. The system of claim 1, wherein the one or more computer subsystems are further configured to be based on One or more metrological measurements are performed on the sample from the high-resolution images generated by the one or more second layers. The system of claim 1, wherein the deep convolutional neural network functions independently of an imaging system that produces the low-resolution image. The system of claim 1, wherein the low-resolution image is generated by an imaging system having a first imaging platform, wherein the one or more computer subsystems are further configured for acquiring images having a different image than the first imaging platform Another low-resolution image generated for another sample by another imaging system of a second imaging platform, one of the imaging platforms, wherein the one or more first layers are configured for use in generating the other low-resolution image A representation, and wherein the one or more second layers are further configured for generating a high-resolution image of the other sample from the representation of the other low-resolution image. The system of claim 11, wherein the first imaging stage is an electron beam imaging stage, and wherein the second imaging stage is an optical imaging stage. The system of claim 11, wherein the first imaging stage and the second imaging stage are different optical imaging stages. The system of claim 11, wherein the first imaging stage and the second imaging stage are different electron beam imaging stages. The system of claim 1, wherein the low-resolution image is produced by an electron beam-based imaging system produce. The system of claim 1, wherein the low-resolution image is produced by an optical-based imaging system. The system of claim 1, wherein the low-resolution image is generated by an inspection system. The system of claim 1, wherein the sample is a wafer. The system of claim 1, wherein the sample is a reticle. The system of claim 1, wherein the deep convolutional neural network outputs the high-resolution image at a higher throughput than using a high-resolution imaging system to generate the high-resolution image. A system configured to generate a high-resolution image of a sample from a low-resolution image of the sample, comprising: an imaging subsystem configured for generating a low-resolution image of a sample ; one or more computer subsystems configured for acquiring the low-resolution image of the sample; and one or more components executed by the one or more computer subsystems, wherein the one or more The components include: a deep convolutional neural network, wherein the deep convolutional neural network includes: one or more first layers configured for generating one of the low-resolution images a representation; and one or more second layers configured for generating a high-resolution image of the sample from the representation of the low-resolution image, wherein the one or more second layers include configured to output a final layer of the high-resolution image, and wherein the final layer is further configured as a sub-pixel convolutional layer, and wherein the one or more components further comprise a device configured to train the deep convolutional neural network a context aware loss module, wherein during training of the deep convolutional neural network, the one or more computer subsystems will be generated by the one or more second layers of the high-resolution image and the sample A corresponding known high-resolution image is input into the context-aware loss module and the context-aware loss module determines that the high-resolution image generated by the one or more second layers matches the corresponding known high-resolution Image vs. situational awareness loss. A non-transitory computer-readable medium storing program instructions executable on one or more computer systems for performing a computer-implemented method for recording a low-resolution sample from a sample The image generates a high-resolution image of the sample, wherein the computer-implemented method comprises: acquiring a low-resolution image of a sample; by inputting the low-resolution image into one or more first steps of a deep convolutional neural network generating a representation of the low-resolution image in one layer; and generating a high-resolution image of the sample based on the representation, wherein the high-resolution image is generated by one or more first steps of the deep convolutional neural network Two-layer implementation, wherein the one or more second layers include a final layer configured to output the high-resolution image, wherein The final layer is further configured as a sub-pixel convolution layer, wherein the acquiring, the generating the representation, and the generating the high-resolution image are performed by the one or more computer systems, wherein one or more components are performed by the one or more computer systems executing, and wherein the one or more components include the deep convolutional neural network, and wherein the one or more components further include a context awareness configured to train the deep convolutional neural network aware) loss module, wherein during training of the deep convolutional neural network, the one or more computer subsystems correspond to one of the high-resolution images and the samples generated by the one or more second layers corresponding to known high-resolution A high-resolution image is input into the context-aware loss module and the context-aware loss module determines a context in the high-resolution image generated by the one or more second layers compared to the corresponding known high-resolution image Perceived loss. A computer-implemented method for generating a high-resolution image of a sample from a low-resolution image of the sample, comprising: acquiring a low-resolution image of a sample; by inputting the low-resolution image to a generating a representation of the low-resolution image in one or more first layers of a deep convolutional neural network; and generating a high-resolution image of the sample based on the representation, wherein generating the high-resolution image is performed by the One or more second layers of the deep convolutional neural network execute, wherein the one or more second layers include a final layer configured to output the high-resolution image, wherein the final layer is further configured as a a subpixel convolutional layer, wherein the acquiring, the generating the representation, and the generating the high-resolution image are performed by one or more computer systems, wherein one or more components are performed by the one or more computer systems, and wherein the One or more components include the deep convolutional neural network, and wherein the one or more components further include configured to train a context aware loss module of the deep convolutional neural network, wherein during training of the deep convolutional neural network, the one or more computer subsystems will be generated by the one or more second layers The high-resolution image and one of the samples corresponding to a known high-resolution image are input into the context-aware loss module and the context-aware loss module determines the high-resolution image generated by the one or more second layers Context awareness loss compared to the corresponding known high-resolution imagery.

Images