US10169852B1 - Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging - Google Patents

Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging Download PDF

Info

Publication number
US10169852B1
US10169852B1 US16/027,056 US201816027056A US10169852B1 US 10169852 B1 US10169852 B1 US 10169852B1 US 201816027056 A US201816027056 A US 201816027056A US 10169852 B1 US10169852 B1 US 10169852B1
Authority
US
United States
Prior art keywords
super
resolution
image
resolution image
specimen
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/027,056
Other languages
English (en)
Inventor
Matthew C. Putman
John B. Putman
Vadim Pinskiy
Joseph R. Succar
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanotronics Imaging Inc
Original Assignee
Nanotronics Imaging Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US16/027,056 priority Critical patent/US10169852B1/en
Application filed by Nanotronics Imaging Inc filed Critical Nanotronics Imaging Inc
Assigned to NANOTRONICS IMAGING, INC. reassignment NANOTRONICS IMAGING, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PINSKIY, VADIM, SUCCAR, JOSEPH R., PUTMAN, JOHN B., PUTMAN, MATTHEW C.
Priority to US16/233,258 priority patent/US10467740B1/en
Publication of US10169852B1 publication Critical patent/US10169852B1/en
Application granted granted Critical
Priority to KR1020217003240A priority patent/KR102605314B1/ko
Priority to EP19830030.3A priority patent/EP3818407B1/en
Priority to CN201980040709.5A priority patent/CN112313554B/zh
Priority to PCT/US2019/033293 priority patent/WO2020009749A1/en
Priority to CN202310392846.3A priority patent/CN117350918A/zh
Priority to JP2020572647A priority patent/JP7011866B2/ja
Priority to TW108117905A priority patent/TWI829694B/zh
Priority to US16/576,732 priority patent/US10789695B2/en
Priority to US17/029,703 priority patent/US10970831B2/en
Priority to US17/222,425 priority patent/US11748846B2/en
Priority to JP2022001823A priority patent/JP7431458B2/ja
Priority to US18/240,910 priority patent/US11948270B2/en
Assigned to ORBIMED ROYALTY & CREDIT OPPORTUNITIES IV, LP reassignment ORBIMED ROYALTY & CREDIT OPPORTUNITIES IV, LP SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CUBEFABS INC., NANOTRONICS HEALTH LLC, NANOTRONICS IMAGING, INC.
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G06T3/4076—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution using the original low-resolution images to iteratively correct the high-resolution images
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/361—Optical details, e.g. image relay to the camera or image sensor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06K9/00134—
    • G06K9/627—
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
    • G06V10/993—Evaluation of the quality of the acquired pattern
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/693—Acquisition

Definitions

  • the present disclosure relates to providing feedback on and improving the accuracy of super-resolution imaging.
  • Specimens as used herein refer to an object of examination (e.g., wafer, substrate, etc.) and artifact refers to a specimen, portion of a specimen, features, abnormalities and/or defects in the specimen.
  • artifacts can be electronic devices such as transistors, resistors, capacitors, integrated circuits, microchips, etc., biological abnormalities, such as cancer cells, or defects in a bulk material such as cracks, scratches, chips, etc.
  • Microscopy inspection systems can be used to enhance what a naked eye can see. Specifically, microscopy inspection systems can magnify objects, e.g. features and abnormalities, by increasing the amount of detail that one can see (e.g., optical resolution).
  • Optical resolution refers to the smallest distance between two points on a specimen that can still be distinguished as two separate points that are still perceivable as separate points by a human.
  • Optical resolution can be influenced by the numerical aperture of an objective, among other parameters. Typically, the higher the numerical aperture of an objective, the better the resolution of a specimen which can be obtained with that objective.
  • a single microscopy inspection system can have more than one objective, with each objective having a different resolving power. Higher resolution objectives typically capture more detail than lower resolution objectives. However, higher resolution objectives, e.g. because of their smaller field of view, typically take much longer to scan a specimen than lower resolution objectives.
  • artificial intelligence models can be used to infer and simulate a super-resolution image from a low-resolution image.
  • Such methods can be achieved without actually scanning the specimen using a higher resolution objective but instead by using all or a portion of a low-resolution image of a specimen, e.g. detected artifacts in a low-resolution image.
  • super-resolution, super-resolution simulation, super-resolution generation, high-resolution simulation and the images produced by these methods will be referred to herein interchangeably as super-resolution images and high resolution images that are simulated, e.g.
  • Super-resolution images can include images created at resolutions greater than the resolution limits of a microscopy system. Specifically, super-resolution images can include images at resolutions beyond the diffraction limit of a given microscopy system or images created beyond the limits of digital image sensors of a given microscopy system. Super-resolution images, as used herein, can also include images simulated within resolution limits of a given microscopy system, but at a higher resolution than a low resolution image (e.g., a super-resolution image can be an image simulated at the highest resolution at which a microscopy system is capable of imaging).
  • an artifact detected using low resolution magnification can correspond to many artifacts detected by high resolution magnification and without additional information, which can be lacking in a low-resolution image of the artifact, it can be impossible to generate an accurate super-resolution image of the low resolution image, e.g. using high resolution simulation.
  • references to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure.
  • the appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
  • various features are described which can be exhibited by some embodiments and not by others.
  • a method can include obtaining a low resolution image of a specimen using a low resolution objective of a microscopy inspection system.
  • a super-resolution image of at least a portion of the specimen can be generated from the low resolution image using a super-resolution simulation. Further, an accuracy assessment of the generated super-resolution image can be identified based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier.
  • the method can also include determining whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image. Subsequently, the super-resolution image can be further processed if it is determined to further process the super-resolution image.
  • a system can include a microscopy inspection system for inspecting a specimen, one or more processors, and at least one computer-readable storage medium.
  • the microscopy inspection system can include a low resolution objective and a high resolution objective.
  • the computer-readable storage medium can store instructions which when executed by the one or more processors cause the one or more processors to obtain a low resolution image of a specimen using the low resolution objective of the microscopy inspection system.
  • the instructions can further cause the one or more processors to generate a super-resolution image of at least a portion of the specimen from the low resolution image using a super-resolution simulation.
  • the instructions can cause the one or more processors to generate an accuracy assessment of the generated super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier.
  • the one or more processors can also, according to execution of the instructions stored in the computer-readable storage medium, determine whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image. Subsequently, the super-resolution image can be further processed by the one or more processors if it is determined to further process the super-resolution image.
  • a non-transitory computer-readable storage medium can include instructions which, when executed by one or more processors, cause the one or more processors to perform operations for generating a super-resolution image for a specimen based on a low resolution image of the specimen.
  • the instructions can cause the one or more processors to receive the low resolution image of the specimen captured by a low resolution objective of a microscopy inspection system.
  • the instruction can also cause the one or more processors to generate the super-resolution image of at least a portion of the specimen from the low resolution image of the specimen using a super-resolution simulation.
  • the instructions can cause the one or more processors to identify an accuracy assessment of the super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier.
  • the instructions can also cause the one or more processors to determine whether to further process the super-resolution image based on the accuracy of the super-resolution image. Accordingly, the instructions can also cause the one or more processors to further process the super-resolution image if it is determined to further process the super-resolution image.
  • FIG. 1 illustrates an example super-resolution system for generating super-resolution images.
  • FIG. 2A is a side view of a general configuration of a microscopy inspection system, in accordance with some embodiments of the disclosed subject matter.
  • FIG. 2B is a front view of a general configuration of a microscopy inspection system, in accordance with some embodiments of the disclosed subject matter.
  • FIG. 3 is a flow of an example operation for using super-resolution image feedback control.
  • FIG. 4 illustrates an example computer system for controlling super-resolution image generation using super-resolution image feedback control.
  • FIG. 5 depicts a scheme of training a suitability classifier for use in providing super-resolution image feedback control.
  • FIG. 6 depicts a scheme of training a simulated image classifier for use in providing super-resolution image feedback control.
  • mechanisms (which can include systems, methods, devices, apparatuses, etc.) for providing feedback on which artifacts found at low resolution magnification are suitable or unsuitable for generating super-resolution images, which artifacts found in super-resolution images should be rescanned using a higher resolution objective, and improving the accuracy of generated super-resolution images are provided.
  • This type of feedback is useful, for example, to selectively employ super-resolution for suitable portions of a specimen, to identify problematic portions of a specimen, both at low resolution and at high resolution, and to train artificial intelligence models for those problematic areas to generate more accurate super-resolution images.
  • artificial intelligence can be used to generate super-resolution images from low resolution images, determine artifacts in a low resolution scan of a specimen that are unlikely to generate accurate super-resolution images, determine an image grade for super-resolution images and based on the image grade determine which artifacts need to be scanned using high resolution magnification.
  • the artificial intelligence algorithms can include one or more of the following, alone or in combination: machine learning, hidden Markov models; recurrent neural networks; convolutional neural networks; Bayesian symbolic methods; general adversarial networks; support vector machines; and/or any other suitable artificial intelligence algorithm.
  • FIG. 1 illustrates an example super-resolution system 100 that can implement super-resolution feedback control to microscopy inspection system 110 and/or computer system 150 , according to some embodiments of the disclosed subject matter.
  • Super-resolution feedback control can include: determining after a low resolution scan of a specimen, artifacts that are unlikely to produce accurate super-resolution images and should be scanned at higher resolution; determining an image grade for the super-resolution images and based on the image grade determining which artifacts should be scanned at higher resolution; comparing the total number of artifacts to a tolerance for a similar specimen or to a tolerance defined for super-resolution system 100 ; and/or using the higher resolution images captured for problematic areas of a specimen to train artificial intelligence models to generate more accurate super-resolution images for those problematic areas.
  • microscopy inspection system 110 can include an illumination source 115 to provide light to a specimen, an imaging device 120 , a stage 125 , a low-resolution objective 130 , a high resolution objective 132 , 135 , control module 140 comprising hardware, software and/or firmware.
  • Microscopy inspection system 110 can be implemented as part of any suitable type of microscope.
  • system 110 can be implemented as part of an optical microscope that uses transmitted light or reflected light.
  • system 100 can be implemented as part of the nSpec® optical microscope available from Nanotronics Imaging, Inc. of Cuyahoga Falls, Ohio
  • Microscopy inspection system can also be implemented as part of confocal or two-photon excitation microscopy.
  • FIGS. 2A (side view) and 2 B (front view), show the general configuration of an embodiment of microscopy inspection system 110 , in accordance with some embodiments of the disclosed subject matter.
  • microscopy inspection system 110 can include two or more objectives 130 , 132 and 135 .
  • Objectives 130 , 132 and 135 can have different resolving powers.
  • Objectives 130 , 132 and 135 can also have different magnification powers, and/or be configured to operate with brightfield/darkfield microscopy, differential interference contrast (DIC) microscopy and/or any other suitable form of microscopy including fluorescents.
  • DIC differential interference contrast
  • high resolution scanning of a specimen can be performed by using a high resolution microscope like a scanning electron microscope (SEM), a transmission electron microscope (TEM), and/or an atomic force microscope (AFM).
  • a high resolution microscope can be a microscope that has a magnifying power (e.g., 10 ⁇ ) two times greater than a low resolution microscopy (e.g., 5 ⁇ ).
  • the objective and/or microscopy technique used to inspect a specimen can be controlled by software, hardware, an/or firmware in some embodiments.
  • high resolution microscopy can be performed in a separate, stand-alone system from low resolution microscopy.
  • low resolution objective 130 and higher resolution objectives 132 and 135 can reside together in a microscopy inspection unit and be coupled to nosepiece 119 .
  • an XY translation stage can be used for stage 125 .
  • the XY translation stage can be driven by stepper motor, server motor, linear motor, piezo motor, and/or any other suitable mechanism.
  • the XY translation stage can be configured to move a specimen in the X axis and/or Y axis directions under the control of any suitable controller, in some embodiments.
  • An actuator can be used to make coarse focus adjustments of, for example, 0 to 5 mm, 0 to 10 mm, 0 to 30 mm, and/or any other suitable range(s) of distances.
  • microscopy inspection system 110 can include a focus mechanism that adjusts stage 125 in a Z direction towards and away from objectives 130 , 132 and 135 and/or adjusts objectives 130 , 132 and 135 towards and away from stage 125 .
  • Illumination source 115 can vary by intensity, number of light sources used, and/or the position and angle of illumination.
  • Light source 117 can transmit light through reflected light illuminator 118 and can be used to illuminate a portion of a specimen, so that light is reflected up through tube lens 123 to imaging device 120 (e.g., camera 122 ), and imaging device 120 can capture images and/or video of the specimen.
  • the lights source used can be a white light collimated light-emitting diode (LED), an ultraviolet collimated LED, lasers or fluorescent light.
  • imaging device 120 can be a camera that includes an image sensor.
  • the image sensor can be, for example, a CCD, a CMOS image sensor, and/or any other suitable electronic device that converts light into one or more electrical signals. Such electrical signals can be used to form images and/or video of a specimen.
  • topographical imaging techniques can be used (including but not limited to, shape-from-focus algorithms, shape-from-shading algorithms, photometric stereo algorithms, and Fourier ptychography modulation algorithms) with a predefined size, number, and position of illuminating light to generate one or more three-dimensional topography images of a specimen.
  • control module 140 comprising a controller and controller interface, can control any settings of super-resolution system 100 (e.g., illumination source 115 , objectives 130 , 132 and 135 , stage 125 , imaging device 120 ), as well as communications, operations (e.g., taking images, turning on and off an illumination source, moving stage 125 and/or objectives 130 , 132 and 135 ).
  • super-resolution system 100 e.g., illumination source 115 , objectives 130 , 132 and 135 , stage 125 , imaging device 120
  • communications e.g., taking images, turning on and off an illumination source, moving stage 125 and/or objectives 130 , 132 and 135 .
  • Control module 140 can include any suitable hardware (which can execute software in some embodiments), such as, for example, computers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), field-programmable gate arrays (FGPAs) and digital signal processors (DSPs) (any of which can be referred to as a hardware processor), encoders, circuitry to read encoders, memory devices (including one or more EPROMS, one or more EEPROMs, dynamic random access memory (“DRAM”), static random access memory (“SRAM”), and/or flash memory), and/or any other suitable hardware elements.
  • ASICs application specific integrated circuits
  • FGPAs field-programmable gate arrays
  • DSPs digital signal processors
  • encoders circuitry to read encoders
  • memory devices including one or more EPROMS, one or more EEPROMs, dynamic random access memory (“DRAM”), static random access memory (“SRAM”), and/or flash memory
  • DRAM dynamic random access memory
  • SRAM static random access memory
  • flash memory
  • communication between the control module (e.g., the controller and controller interface) and the components of super-resolution system 100 can use any suitable communication technologies, such as analog technologies (e.g., relay logic), digital technologies (e.g., RS232, ethernet, or wireless), network technologies (e.g., local area network (LAN), a wide area network (WAN), the Internet) Bluetooth technologies, Near-field communication technologies, Secure RF technologies, and/or any other suitable communication technologies.
  • analog technologies e.g., relay logic
  • digital technologies e.g., RS232, ethernet, or wireless
  • network technologies e.g., local area network (LAN), a wide area network (WAN), the Internet) Bluetooth technologies, Near-field communication technologies, Secure RF technologies, and/or any other suitable communication technologies.
  • operator inputs can be communicated to control module 140 using any suitable input device (e.g., a keyboard, mouse or joystick).
  • any suitable input device e.g., a keyboard, mouse or joystick.
  • Computer system 150 of super-resolution system 100 can be coupled to microscopy inspection system 110 in any suitable manner using any suitable communication technology, such as analog technologies (e.g., relay logic), digital technologies (e.g., RS232, ethernet, or wireless), network technologies (e.g., local area network (LAN), a wide area network (WAN), the Internet) Bluetooth technologies, Near-field communication technologies, Secure RF technologies, and/or any other suitable communication technologies.
  • Computer system 150 can include any suitable hardware (which can execute software in some embodiments), such as, for example, computers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), field-programmable gate arrays (FGPAs), and digital signal processors (DSPs) (any of which can be referred to as a hardware processor), encoders, circuitry to read encoders, memory devices (including one or more EPROMS, one or more EEPROMs, dynamic random access memory (“DRAM”), static random access memory (“SRAM”), and/or flash memory), and/or any other suitable hardware elements.
  • suitable hardware which can execute software in some embodiments
  • suitable hardware such as, for example, computers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), field-programmable gate arrays (FGPAs), and digital signal processors (DSPs) (any of which can be referred to as a hardware processor), encoders, circuitry to read encoders, memory devices (including one or more EPROMS, one or more
  • Computer-readable media can be any available media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media.
  • Computer readable media can comprise computer storage media and communication media.
  • Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.
  • computer system 150 can include an artifact suitability analysis module 160 , a super-resolution module 170 , a super-resolution analysis module 180 , an image assembly module 190 and an artifact comparison module 195 .
  • FIG. 3 shows at a high level, an example of a super-resolution operation 300 using super-resolution feedback control, in accordance with some embodiments of the disclosed subject matter.
  • super-resolution operation 300 can use super-resolution system 100 . Further details explaining how each module of computer system 150 can be configured, in accordance with some embodiments of the disclosed subject matter, will be described in connection with FIG. 4 .
  • microscopy inspection system 110 can scan a specimen using low resolution objective 130 .
  • the specimen can be scanned by moving imaging device 120 and/or stage 125 in an X/Y direction until the entire surface or a desired area of a specimen is scanned.
  • one or more areas of a specimen can be scanned by using different focus levels and moving stage 125 and/or low-resolution objective 130 in a Z direction.
  • Imaging device 120 can capture and generate low resolution images of the scanned specimen.
  • artifact suitability analysis module 160 can use artificial intelligence algorithms and/or other suitable computer programs (as explained further herein) to detect artifacts in the generated low resolution image and determine their suitability for super-resolution imaging.
  • suitability can be based on cross-correlation of an artifact to known artifacts that have been assessed as suitable or not suitable for super-resolution imaging.
  • Cross-correlation as referred to herein, can be a measure of similarity of two series (e.g., two images) as a function of the displacement of one relative to the other. More specifically, an image of an artifact being examined and an image of a known artifact, each represents a matrix of intensity values per pixel (0-255), and cross-correlation can specify the value associated with how different or similar the images are at each pixel.
  • suitable known artifacts can be artifacts where super-resolution images were generated and those images were determined to be high confidence super-resolution images, e.g. having a high image grade.
  • known unsuitable artifacts can be artifacts where super-resolution images were generated and those images were determined to be low confidence super-resolution images, e.g. having a low image grade. High confidence and low confidence super-resolution images and corresponding image grades are further described herein.
  • the techniques described herein are made with reference to identifying whether an artifact is suitable for use in a super-resolution simulation, in various embodiments, the techniques can be performed without determining suitability of artifacts for use in a super-resolution simulation.
  • a high resolution objective (e.g., high resolution objective 132 or 135 ) can scan the artifacts determined to be unsuitable by artifact suitability analysis module 160 , and imaging device 120 can capture and generate high resolution images of the scanned artifacts.
  • the generated high resolution images can be provided as feedback to: artifact suitability analysis module 160 to provide additional context data for determining the suitability of an artifact for super-resolution imaging; super-resolution module 170 to improve its accuracy; and/or super-resolution analysis module 180 to provide additional context data for determining the image grade of a super-resolution image.
  • the high resolution images can also be provided to image assembly module 190 for incorporation into a single coherent image, e.g. combining one or more super-resolution images and one or more high resolution images, of a scanned specimen.
  • super-resolution module 170 using one or more super-resolution algorithms, can generate super-resolution images for the entire specimen or just the artifacts determined to be suitable for super-resolution by artifact suitability analysis module 160 .
  • Super-resolution analysis module 180 can receive super-resolution images from super-resolution module 170 and using artificial intelligence algorithm and/or other suitable computer programs (as explained further herein) determine an image confidence of the super-resolution images.
  • an image confidence determination of a super-resolution image can include a specific image confidence determination of the super-resolution image, whether the super-resolution image is a high confidence super-resolution image, whether the super-resolution is a low confidence super-resolution image, and/or an image grade of the super-resolution image.
  • An image confidence determination of a super-resolution image, as determined by the super-resolution analysis module 180 can correspond to a predicted accuracy, e.g.
  • a predicted accuracy of a super-resolution image can be an estimate of how accurately a super-resolution image created from a low resolution image actually represents a specimen and artifacts in the specimen.
  • a predicted accuracy of a super-resolution image can be an estimate of how accurately a super-resolution image created from a low resolution image actually represents a specimen and artifacts in the specimen as if the super-resolution image was created by actually scanning the artifacts/specimen using a high resolution objective or an applicable mechanism for scanning the specimen at super-resolution. For example, if super-resolution analysis model 180 identifies that a simulated super-resolution image accurately represents 95% of an imaged specimen, then super-resolution analysis module 180 can identify that the super-resolution image is a high confidence super-resolution image.
  • An image confidence determination of a super-resolution image, as determined by the super-resolution analysis module can correspond to degrees of equivalence between a super-resolution image and one or more actually scanned high resolution images of a specimen.
  • super-resolution analysis module 180 can determine how closely a super-resolution image corresponds to an actual high resolution image of the same or similar type of specimen/artifact to determine a confidence in the super-resolution image and a degree of equivalence between the super-resolution image and the high resolution image. This can be based on cross-correlation methods.
  • a same or similar type of specimen/artifact is referred to as a related specimen/artifact.
  • a related specimen can include an imaged material that is the same or similar type of material as a currently analyzed specimen.
  • a related specimen to a current specimen can include the current specimen itself. If a super-resolution image closely correlates to an actual high resolution image of the same or similar type of specimen/artifact, then super-resolution analysis module 180 can indicate that the super-resolution image is a high confidence super-resolution image.
  • super-resolution analysis module 180 can indicate that the super-resolution image is a low confidence super-resolution image and indicate for the underlying artifact to be scanned using a high resolution objective (e.g., 132 and 135 ) and to generate high resolution images (as in step 330 ).
  • a high resolution objective e.g., 132 and 135
  • image assembly module 190 can assemble and stitch together (as described further herein), the received super-resolution images and the images scanned using a high resolution objective, into a single coherent image of a scanned specimen.
  • artifact comparison module 195 can receive a single coherent image of a specimen and determine a total number of artifacts for the specimen.
  • the artifact comparison module 195 can compare the total number with a tolerance that is typical for the type of specimen that was scanned, or based on a tolerance defined for super-resolution system 100 (e.g., by an operator, hardware/firmware/software constraints, industry guidelines, and/or any other suitable standard).
  • blocks of operation 300 can be performed at any suitable times. It should be understood that at least some of the portions of operation 300 described herein can be performed in any order or sequence not limited to the order and sequence shown in and described in connection with FIG. 3 , in some embodiments. Also, some portions of process 200 described herein can be performed substantially simultaneously where appropriate or in parallel in some embodiments. Additionally, or alternatively, some portions of process 200 can be omitted in some embodiments. Operation 300 can be implemented in any suitable hardware and/or software. For example, in some embodiments, operation 300 can be implemented in super-resolution system 100 .
  • FIG. 4 shows the general configuration of an embodiment of computer system 150 , in accordance with some embodiments of the disclosed subject matter
  • artifact suitability analysis module 160 can be configured to receive one or more low resolution images of a specimen from microscopy inspection system 110 and/or any suitable computer readable media.
  • the low resolution images can be images captured by imaging device 120 using low resolution objective 130 .
  • artifact suitability analysis module 160 can be configured to detect, using computer vision, one or more artifacts in the received image(s) and determine a suitability class for each detected artifact.
  • Detection of an artifact can be based on, e.g., information from a reference design (e.g., a computer aided design (CAD) file, physical layout of a specimen, etc.), deviations from a reference design, and/or data about known artifacts.
  • a reference design e.g., a computer aided design (CAD) file, physical layout of a specimen, etc.
  • one or more artificial intelligence algorithm(s) can be used to determine a suitability class for each identified artifact.
  • the class can be a binary class (e.g., “suitable” and “not suitable” for super-resolution imaging).
  • the class can provide greater or higher resolution distinctions of classes (e.g., a letter grade A-F, where A denotes the best grade and where F denotes the worst grade, or a number grade 1-100, where 1 denotes the worst grade and 100 denotes the best grade).
  • artifact suitability analyzer module 160 can apply a classification algorithm to determine whether a detected artifact in a low resolution image is or is not suitable for super-resolution generation.
  • the classification algorithm is first trained with training data to identify shared characteristics of artifacts that are suitable for super-resolution generation and those that are not.
  • training data can include examples of low resolution images of artifacts along with their assigned suitability classes.
  • training data can include examples of low resolution images of artifacts along with the image grades assigned to super-resolution images generated for those artifacts.
  • the classification algorithm can make inferences about suitability based on an artifact's type, size, shape, composition, location on the specimen and/or any other suitable characteristic.
  • training data can also include explicit suitability assignments based on a portion of a specimen that is being imaged, information from a reference design, an artifact location (i.e., location of an artifact on a specimen), type of artifact and/or its size, shape and/or composition.
  • artifact suitability analyzer module 160 can be applied by artifact suitability analyzer module 160 to determine whether a detected artifact in a low resolution image is suitable or not suitable for super-resolution generation.
  • attributes can be, for example, artifact's type, size, shape, composition, location on the specimen, reference design and/or any other suitable characteristic, to determine an artifact's suitability for super-resolution imaging.
  • a support vector machine is an example of a classifier that can be employed.
  • SVM operates by finding a hypersurface in the space of possible inputs that attempts to split the triggering criteria from the non-triggering events. This makes the classification correct for testing data that is near, but not identical to training data.
  • Directed and undirected model classification approaches can be used and include, e.g., na ⁇ ve Bayes, Bayesian networks, decision trees, and probabilistic classification models providing different patterns of independence can be employed. Classification as used herein is also inclusive of statistical regression that can be utilized to develop priority models.
  • the disclosed subject matter can employ classifiers that are trained via generic training data, extrinsic information (e.g., reference design, high resolution images of the same or similar type specimen (referred to herein as a ground truth high resolution image)), and/or feedback from super-resolution system 100 , as super-resolution operation 300 progresses.
  • SVM's can be configured via a learning or training phase within a classifier constructor and feature selection module.
  • the classifier(s) can be used to automatically perform a number of functions, including but not limited to the following: determining the context of an artifact (e.g., location of the artifact on a specimen, the type of specimen being inspected, similar artifacts on the same or similar type specimens, a reference design, a ground truth high resolution image), and analyzing the size, shape, composition of the artifact to better classify the artifact in order to correctly determine the suitability of the artifact for super-resolution imaging.
  • an artifact e.g., location of the artifact on a specimen, the type of specimen being inspected, similar artifacts on the same or similar type specimens, a reference design, a ground truth high resolution image
  • analyzing the size, shape, composition of the artifact to better classify the artifact in order to correctly determine the suitability of the artifact for super-resolution imaging.
  • the SVM is a parameterized function whose functional form is defined before training.
  • a SVM is a function defined by one or more separating hyperplanes in a dimensional space of multiple or infinite dimensions.
  • the SVM can be trained using an applicable method for training a supervised learning model, Training an SVM generally requires a labeled training set, since the SVM will fit the function from a set of examples.
  • the training set can consist of a set of N examples, Each example consists of an input, vector, xi, and a category yj, which describes whether the input vector is in a category. For each category there can be one or more parameters, e.g. N free parameters in an SVM trained with N examples, for training the SVM to form the separating hyperplanes.
  • Quadratic programming (QP) problem can be solved as is well understood.
  • sub-gradient descent and coordinate descent can be used to train the SVM using these parameters.
  • These techniques may include a Sequential Minimal Optimization technique as well as other techniques for finding/solving or otherwise training the SVM classifier using such techniques.
  • the disclosed subject matter can be implemented using unsupervised machine learning techniques. Specifically, confidence image determinations of super-resolution images can be identified using unsupervised learning techniques. Further, suitability of artifacts in low resolution images in being used to form a super-resolution image can be identified using unsupervised learning techniques. Unsupervised learning techniques include applicable methods for recognizing patterns in uncategorized/unlabeled data. For example, a neural network can be used to implement the disclosed subject matter through unsupervised learning techniques.
  • the diagram illustrates a scheme, in accordance with some embodiments of the disclosed subject matter, Wherein detected artifacts 510 are classified into two classes: suitable and not suitable for super-resolution imaging. This is just an example and a plurality of other training sets may be employed to provide greater or higher resolution distinctions of classes (e.g., the classes can represent different suitability grades A, B, C, D, E and F or suitability scores.). Suitability of an artifact can be a measure of a likelihood that the artifact can be used to produce all or a portion of an accurate super-resolution image, at 340 .
  • suitability of an artifact can be a measure of likelihood that a super-resolution image generated, at least in part, from a low resolution image will pass as a high confidence super-resolution image, e.g. at 350 .
  • suitability of an artifact can be a prediction of how closely a super-resolution image created from a low resolution image of the artifact will correspond to an actual high resolution image of the same or similar type of specimen/artifact. For example, if there is a 95% chance that a super-resolution image created from a low resolution image of an artifact will correspond greatly, e.g. 90% correlation, with an actual high resolution image of a related artifact, then the artifact can be identified as suitable for super-resolution imaging, e.g. have a high suitability grade of A.
  • the suitability classifier 520 can be trained by a group of known artifacts 515 that represent artifacts suitable for super-resolution imaging and a group of known artifacts 517 that represent artifacts not suitable for super-resolution imaging. In other embodiments, suitability classifier 520 can be trained by a group of known artifacts that represent different suitability grades. Artifacts 510 to be analyzed can be input into suitability classifier 520 , which can output a class 530 , which indicates the class that the detected artifact most likely falls into. Further classes (e.g., a grade) can also be added if desired. In some embodiments, suitability classifier 520 can also output a scalar number 525 , e.g. a suitability score, that can measure the likelihood that an artifact being analyzed falls into the class suitable for super-resolution imaging, if so desired, or the class not suitable for super-resolution imaging, for example.
  • a scalar number 525 e
  • linear regression modeling is a machine learning technique for modeling linear relationships between a dependent variable and one or more independent variables.
  • a simple linear regression model utilizing a single scalar prediction can be used to perform the scoring described herein.
  • a multiple linear regression model utilizing multiple predictors can be used to perform the scoring described herein.
  • Confidence level generally refers to the specified probability of containing the parameter of the sample data on which it is based is the only information available about the value of the parameter. For example, if a 95% confidence level is selected then it would mean that if the same population is sampled on numerous occasions and confidence interval estimates are made on each occasion, the resulting intervals would bracket the true population parameter in approximately 95% of the cases.
  • G An example of confidence level estimation that can be adapted for use by super-resolution system 100 is described by G.
  • artifact suitability analysis module 160 determines suitability in a non-binary manner (e.g., scoring an artifact by grade or by number)
  • artifact suitability analysis module 160 can be configured to compare the determined suitability score with an acceptable suitability tolerance for super-resolution system 100 , e.g. as defined by an operator, hardware/firmware/software constraints, industry guidelines, and/or any other suitable standard.
  • an acceptable suitability tolerance for super-resolution system 100 e.g. as defined by an operator, hardware/firmware/software constraints, industry guidelines, and/or any other suitable standard.
  • artifact suitability analysis module 160 can indicate for the identified artifacts to be scanned using a higher resolution objective.
  • artifact suitability analysis module 160 can indicate for super-resolution images to be generated for the detected artifacts.
  • the classifier can also be used to automatically adjust the acceptable suitability tolerance used for determining suitability of an artifact for super-resolution imaging.
  • a feedback mechanism can provide data to the classifier that automatically impacts the acceptable suitability tolerance based on historical performance data and/or improvement of one or more underlying artificial intelligence algorithms used by super-resolution system 100 .
  • an acceptable suitability tolerance can initially be set so that all detected artifacts receiving a letter grade of C and above, or a number grade of 50 and above, are deemed suitable for super-resolution imaging.
  • the classifier can raise the acceptable suitability tolerance making it more difficult for artifacts to be classified as suitable. Conversely, if feedback from super-resolution module 170 shows that its model has improved and is better able to generate super-resolution images for defects previously classified as unsuitable, then the classifier can lower the acceptable suitability tolerance making it easier for artifacts to be classified as suitable.
  • the acceptable suitability tolerance used by artifact suitability analyzer module 160 to determine suitability can also be automatically adjusted based on the importance of a specimen and/or an area of a specimen being examined. For example, artifact suitability analyzer module 160 can adjust the acceptable suitability tolerance upwards for specimens and/or areas of a specimen considered important and/or adjust the acceptable suitability tolerance downwards for specimens and/or areas of a specimen not considered important.
  • suitability analyzer module 160 is not restricted to employing artificial intelligence for determining suitability of an artifact for super-resolution imaging.
  • artifact suitability analyzer module 160 can be preprogrammed to recognize suitable and unsuitable artifacts. Based on the preprogrammed data, suitability analyzer module 160 can process one or more low resolution images to determine whether the low resolution images(s) include any artifacts similar to the preprogrammed artifacts and determine suitability based on the suitability of the preprogrammed artifacts.
  • the artificial intelligence algorithms used by artifact suitability analysis module 160 can be based on comparing characteristics of and/or context data for the detected artifact to characteristics of and/or context data of training data to generate a suitability score. For example, if a detected artifact closely resembles an artifact from the training data that received a score of A, then artifact suitability analysis module 160 can assign a similar score to the detected artifact.
  • the artificial intelligence algorithms used by artifact suitability analysis module 160 can be based on comparing characteristics of and/or context data for the detected artifact to characteristics of and/or context data of training data that yielded high confidence super-resolution images (e.g., as determined by super-resolution analysis module 180 ) to generate a suitability score.
  • artifact suitability analysis module 160 can assign a lower score to detected artifacts resembling training data that yielded low confidence super-resolution images and a higher score to detected artifacts resembling training data that yielded high confidence super-resolution images.
  • the artificial intelligence algorithms used by artifact suitability analysis module 160 can be based on comparing detected artifacts on a specimen to artifacts in a high resolution image of the same or similar type specimen (also referred to as the ground truth high resolution image). If the detected artifact corresponds to two or more artifacts in the ground truth high resolution scan, and the context data for the detected artifact does not provide additional information, then artifact suitability analysis module 160 can assign a low suitability score. Conversely, if the detected artifact corresponds to only one artifact in the ground truth high resolution image, then artifact suitability analysis module 160 can assign a high suitability score to the detected artifact.
  • Artifact suitability analysis module 160 can also be configured, in some embodiments, to record the identified artifacts, their suitability scores and the acceptable suitability tolerance at which the analysis was performed.
  • super-resolution module 170 can be configured to receive one or more low resolution images of a specimen that are determined to be suitable for super-resolution generation, and to generate one or more super-resolution image(s) from the received image(s).
  • super-resolution module 170 can be configured to receive one or more low resolution images of a specimen irrespective of whether the low resolution images are deemed actually suitable for super-resolution generation, and to generate one or more super-resolution image(s) from the received images.
  • one or more artificial intelligence algorithm(s) can be used to generate one or more super-resolution images from one or more low resolution images.
  • the algorithms used by super-resolution module 170 can consider context date like location of the artifact on the specimen, the type of specimen being inspected, a comparison of the artifact to other artifacts detected on the same or similar specimens, a reference design, low resolution images taken at different focus levels and/or using different lighting techniques, high resolution images taken at different focus levels and/or using different lighting techniques, etc.
  • the algorithms used by super-resolution module 170 can include classifying an artifact, as well as identifying its size, shape, composition, location on the specimen and/or any other suitable characteristic to infer an accurate high resolution image.
  • Super-resolution methods employed by super-resolution module 170 can include, but are not limited to: interpolation, super-resolution from low resolution depth image frames, super-resolution through fusing depth image and high resolution color image, example-based super-resolution, and depth image super-resolution based on edge-guided method.
  • interpolation Some examples of interpolation that can be adapted for use by super-resolution module 170 are described by: Xie, J. et al., “Edge-guided Single Depth Image Super-resolution,” IEEE Trans. Image Process. 2016, 25, 428-438; Prajapati, A. et al., “Evaluation of Different Image Interpolation Algorithms,” Int. J. Comput. Appl. 2012, 58, 466-476; Pang, Z. et al, “An Improved Low-cost Adaptive Bilinear Image Interpolation Algorithm,” In Proceedings of the 2nd International Conference on Green Communications and Networks, Chongqing, China, 14-16 Dec.
  • super-resolution module 170 Some examples of super-resolution from low resolution depth image frames that can be adapted for use by super-resolution module 170 are described by: Schuon, S. et al., “LidarBoost: Depth Superresolution for ToF 3D Shape Scanning,” In Proceedings of the 2009 the 22nd International Conference on Computer Vision and Pattern Recognition, Miami, Fla., USA, 20-25 Jun. 2009; pp. 343-350; Rajagopalan, A. N. et al., “Resolution Enhancement of PMD Range Maps,” In Proceedings of the Joint Pattern Recognition Symposium, Kunststoff, Germany, 10-13 Jun. 2008; Springer: Berlin/Heidelberg, Germany, 2008; pp. 304-313; A1 Ismaeil, K.
  • super-resolution module 170 Some examples of super-resolution through fusing depth image and high resolution color image, that can be adapted for use by super-resolution module 170 are described by: Ferstl, D. et al., “Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation,” In Proceedings of the IEEE International Conference on ComputernVision, Sydney, NSW, Australia, 1-8 Dec. 2013; pp. 993-1000; Yang, Q. et al., “Spatial-Depth Super-resolution for Range Images, In Proceedings of the 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Minneapolis, Minn., USA, 17-22 Jun. 2007; pp. 1-8; Lo, K. H.
  • example-based super-resolution that can be adapted for use by super-resolution module 170 are described by: Timofte, R. et al., “A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution,” In Proceedings of the Asian Conference on Computer Vision, Singapore, 1-5 Nov. 2014; Springer: Cham, Switzerland, 2014; pp. 111-126; Yang, J. et al., “Image Super-resolution via Sparse Representation,” IEEE Trans. Image Process. 2010, 19, 2861-2873; Xie, J.
  • an artificial intelligence algorithm used by super-resolution module 170 can be trained using low resolution images only.
  • super-resolution analysis module 180 can be configured to receive one or more super-resolution images from super-resolution module 170 and/or from any computer readable media, and determine an image class (or grade) for each super-resolution image of an artifact.
  • one or more artificial intelligence algorithm(s) can be used to determine an image class for super-resolution images of an artifact.
  • the image class can be a binary class (e.g., “high confidence super-resolution image” and “low confidence super-resolution image”).
  • the class can provide greater or higher resolution distinctions of classes (e.g., a letter grade A-F, where A denotes the best grade and where F denotes the worst grade, or a number grade 1-100, where 1 denotes the worst grade and 100 denotes the best grade).
  • super-resolution analysis module 180 can apply a classification algorithm to determine an image grade for a super-resolution image.
  • the classification algorithm is first trained with training data to identify shared characteristics of super-resolution images of artifacts that are high confidence super-resolution images and those that are low confidence super-resolution images.
  • training data can include examples of super-resolution images for the types of artifacts/specimens that are being examined by super-resolution system 100 and their corresponding image scores/grades and/or cross correspondence to actual high resolution images of the same or similar type of specimen/artifact.
  • the classification algorithm can make inferences about an image class based on a reference design, a ground truth high resolution image of the same or similar specimen type, a ground truth high resolution image of the same or similar artifact type, an artifact's type, size, shape, composition, location on the specimen and/or any other suitable characteristic.
  • training data can also include explicit image class assignments based on a portion of a specimen that is being imaged, an artifact location (i.e., location of an artifact on a specimen), a reference design, a ground truth high resolution image, type of artifact and/or its size, shape and/or composition.
  • the classification algorithm can be applied by super-resolution analysis module 180 to determine an image class for an image of an artifact generated by super-resolution.
  • a support vector machine is an example of a classifier that can be employed.
  • Directed and undirected model classification approaches can also be used and include, e.g., na ⁇ ve Bayes, Bayesian networks, decision trees, and probabilistic classification models providing different patterns of independence can be employed.
  • Classification as used herein is also inclusive of statistical regression that can be utilized to develop priority models.
  • the disclosed subject matter can employ classifiers that are trained via generic training data, extrinsic information (e.g., reference design, ground truth high resolution image of the same or similar type specimen/artifact), and/or feedback from super-resolution system 100 , as super-resolution operation 300 progresses.
  • SVM's can be configured via a learning or training phase within a classifier constructor and feature selection module.
  • the classifier(s) can be used to automatically perform a number of functions, including but not limited to the following: determining context data for a super-resolution image (e.g., location of artifact on the specimen, the type of specimen being inspected, similar artifacts on similar specimens, a reference design, a ground truth high resolution image of the same or similar type specimen, a ground truth high resolution image of the same or similar type artifact) and analyzing the size, shape, composition of the artifact to better classify the artifact in order to correctly determine the image grade of a super-resolution image for an artifact.
  • context data for a super-resolution image e.g., location of artifact on the specimen, the type of specimen being inspected, similar artifacts on similar specimens, a reference design, a ground truth high resolution image of the same or similar type specimen, a ground truth high resolution image of the same or similar type artifact
  • analyzing the size, shape, composition of the artifact to better classify the art
  • the diagram illustrates a scheme, in accordance with some embodiments of the disclosed subject matter, wherein super-resolution images of artifacts 610 are classified into two classes: low confidence super-resolution images and high confidence super-resolution images.
  • This is just an example and a plurality of other training sets can be employed to provide greater or higher resolution distinctions of classes (e.g., the classes can represent different image grades A, B, C, D, E and F or image scores 1-100).
  • the simulated image classifier 620 can be trained by a group of known super-resolution images 615 that are high confidence super-resolution images of artifacts and a group of known super-resolution images 617 that represent low confidence super-resolution images of artifacts.
  • simulated image classifier 620 can be trained by a group of known super-resolution images that represent different image grades.
  • Super-resolution images of artifacts 610 to be analyzed can be input into simulated image classifier 620 , which can output a confidence interval 625 that can measure the likelihood that the super-resolution image being analyzed falls into a particular class (e.g., high confidence super-resolution image and low confidence super-resolution image).
  • simulated image classifier 620 can also output a class 630 , which indicates the class that the super-resolution image most likely falls into. Further classes (e.g., a lettered or numbered grade) can also be added if desired
  • super-resolution analysis module 180 determines image classification in a non-binary manner (e.g., scoring a super-resolution image by grade or by number)
  • super-resolution analysis module 180 can be configured to compare the determined image grade with an acceptable image tolerance for super-resolution system 100 , as defined by an operator, hardware/firmware/software constraints, industry guidelines, and/or any other suitable standard.
  • an acceptable image tolerance for super-resolution system 100 as defined by an operator, hardware/firmware/software constraints, industry guidelines, and/or any other suitable standard.
  • super-resolution analysis module 180 can indicate for the artifacts in the super-resolution images to be scanned using a higher resolution objective.
  • Image tolerances and corresponding image scores assigned by the super-resolution analysis module 180 can indicate whether a super-resolution image is a high confidence super-resolution image or a low confidence super-resolution image. For example, super-resolution images having image scores at or above an acceptable image tolerance can be identified as high confidence super-resolution images. Conversely, super-resolution images having image scores below an acceptable image tolerance can be identified as low confidence super-resolution images. For super-resolution images receiving image scores at or above the acceptable image tolerance for super-resolution system 100 , the super-resolution images can be provided to image assembly module 190 .
  • the classifier can also be used to automatically adjust the acceptable image tolerance used for determining whether an artifact rendered by super-resolution passes or fails the tolerance.
  • a feedback mechanism can provide data to the classifier that automatically impacts the tolerance based on historical performance data and/or improvement of one or more underlying artificial intelligence algorithms used by super-resolution system 100 .
  • the classifier can adjust the tolerance based on feedback about super-resolution images correctly and/or incorrectly classified as high or low confidence super-resolution images. For example, if feedback from artifact comparison module 195 shows that a large number of super-resolution images had to be rescanned using a higher resolution objective, then the classifier can raise the acceptable image tolerance making it more difficult for super-resolution images to qualify.
  • the classifier can lower the acceptable image tolerance making it easier for super-resolution images to qualify.
  • the image tolerance used by super-resolution analysis module 180 to determine an acceptable image tolerance can also be automatically adjusted based on the importance of a specimen and/or an area of a specimen being examined. For example, super-resolution analysis module 180 can adjust the acceptable image tolerance upwards for specimens and/or areas of a specimen considered important and/or adjust the acceptable image tolerance downwards for specimens and/or areas of a specimen not considered important.
  • super-resolution analysis module 180 is not restricted to employing artificial intelligence for determining an image grade for super-resolution images.
  • super-resolution analysis module 180 can be preprogrammed to recognize super-resolution images of artifacts that have acceptable and non-acceptable image grades. Based on the preprogrammed data, super-resolution analysis module 180 can process one or more super-resolution image to determine whether the super-resolution images(s) include any images similar to the preprogrammed images and determine acceptable image grades based on the image grades of the preprogrammed super-resolution and/or high resolution images.
  • the artificial intelligence algorithms used by super-resolution analysis module 180 can be based on comparing characteristics of and/or context data for the super-resolution image to characteristics of and/or context data of training data to generate an image score. For example, if a super-resolution image of an artifact closely resembles a super-resolution image of an artifact from the training data set that received an image score of A, then super-resolution analysis module 180 can assign a similar score to the super-resolution image.
  • the artificial intelligence algorithms used by super-resolution analysis module 180 can be based on comparing super-resolution images of an artifact found on a specimen to a high resolution image of the same or similar type artifact or specimen. If the super-resolution analysis module 180 finds a close correspondence, then it can assign a high image score to the super-resolution image. Conversely, if super-resolution analysis module 180 finds a poor correspondence, then it can assign a low image score to the super-resolution image.
  • Super-resolution analysis module 180 can also be configured, in some embodiments, to record the received super-resolution images and their image grades, as well as the acceptable image tolerance at which the analysis was performed.
  • image assembly module 190 can be configured to assemble and stitch together the super-resolution images, and the actual high resolution images into a single coherent image of a specimen.
  • each image of a specimen is referred to as a tile, wherein each tile can be located by its XY coordinate position in a specimen space.
  • the high resolution objective can then scan the area on the specimen representing the tile or tiles that contain the identified artifacts.
  • super-resolution module 170 can simulate the entire tile or tiles that contain the artifacts determined to be suitable for super-resolution imaging.
  • the high resolution images of the tiles and the super-resolution images of the tiles can be stitched together based on their XY coordinate positions and/or feature-based registration methods. This is just one example of how a single coherent image can be assembled, and other suitable methods for accomplishing this can be performed.
  • super-resolution module 170 can simulate the entire specimen (even portions of the specimen that were indicated unsuitable for super-resolution imaging) and image assembly module 190 can replace the unsuitable portions with high resolution images of those portions.
  • Image assembly module 190 can use a high resolution image tile's XY location, as well as identify similar features between the high resolution image tile and the super-resolution image tile to determine where to place the high resolution image tiles.
  • the super-resolution image tile can be replaced with the high resolution image tile. While the above method assumes no more than a single artifact per tile, the method can be adapted to accommodate multiple artifacts per tile.
  • artifact comparison module 195 can be configured to receive a single coherent image of a specimen (e.g., from image assembly module 190 and/or any suitable computer readable media) and determine a total number of artifacts for the specimen.
  • the artifact comparison module 195 can compare the total number with a tolerance that is typical for the type of specimen that was scanned, or based on a tolerance defined for super-resolution system 100 , by an operator, hardware/firmware/software constraints, industry guidelines, and/or any other suitable standard.
  • super-resolution analysis module 180 if the total number of artifacts exceed or fall below the tolerance for the type of specimen that was scanned, and/or the defined tolerance for super-resolution system 100 , then super-resolution analysis module 180 , based on feedback from artifact comparison module 195 , can select a second set of super-resolution images falling below a higher acceptable image tolerance to be rescanned using high resolution objective 132 or 135 . Specifically, super-resolution analysis module 180 can select a set of super-resolution images as part of further controlling operation of super-resolution system 100 to generate one or more high resolution images for a specimen.
  • super-resolution analysis module 180 can raise the acceptable image tolerance to 60%, and the super-resolution images that were assigned an image grade between 50-59% will be rescanned using a high resolution objective. Feedback to super-resolution analysis module 180 and adjustment to the acceptable image tolerance can occur as many times as necessary.
  • artifact suitability analysis module 160 can select a second set of artifacts falling below a higher acceptable suitability tolerance to be rescanned using high resolution objective 135 .
  • the artifact suitability analysis module 160 can select a set of second artifacts as part of further controlling operation of super-resolution system 100 to generate one or more high resolution images for a specimen.
  • artifact suitability analysis module 160 can raise the suitability threshold to 60% and the artifacts that were assigned a suitability score between 50-59% will be rescanned using a high resolution objective. Feedback to artifact suitability analysis module 160 and adjustment to the acceptable suitability tolerance can occur as many times as necessary.
  • artifact comparison module 195 can determine that super-resolution module 170 is using an unsuitable artificial intelligence model to generate super-resolution images and instruct super-resolution module 170 to use a different artificial intelligence model to generate super-resolution images for a particular specimen.
  • the super-resolution module 170 can use different artificial intelligence models as part of further controlling operation of super-resolution system 100 to generate one or more high resolution images for a specimen.
  • artifact suitability analysis module 160 can determine suitability based on analyzing distinct areas of a specimen.
  • super-resolution analysis module 180 can determine image grades based on analyzing distinct areas of a specimen rendered using super-resolution.
  • the functionality of the components for super-resolution system 100 can be combined into a single component or spread across several components.
  • the functionality of some of the components e.g., high resolution scanning by high resolution objective 132 or 135 and computer processing by computer system 150 ) can be performed remotely from microscopy inspection system 110 .
  • super-resolution system 100 can include other suitable components not shown. Additionally or alternatively, some of the components included in super-resolution system 100 can be omitted.
  • any suitable computer readable media can be used for storing instructions for performing the functions and/or processes described herein.
  • computer readable media can be transitory or non-transitory.
  • non-transitory computer readable media can include media such as non-transitory magnetic media (such as hard disks, floppy disks, etc.), non-transitory optical media (such as compact discs, digital video discs, Blu-ray discs, etc.), non-transitory semiconductor media (such as flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), etc.), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and/or any suitable tangible media.
  • EPROM electrically programmable read only memory
  • EEPROM electrically erasable programmable read only memory
  • transitory computer readable media can include signals on networks, in wires, conductors, optical fibers, circuits, and any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and/or any suitable intangible media.
  • a cloud-based computing system is a system that provides virtualized computing resources, software and/or information to client devices.
  • the computing resources, software and/or information can be virtualized by maintaining centralized services and resources that the edge devices can access over a communication interface, such as a network.
  • the cloud can provide various cloud computing services via cloud elements, such as software as a service (SaaS) (e.g., collaboration services, email services, enterprise resource planning services, content services, communication services, etc.), infrastructure as a service (IaaS) (e.g., security services, networking services, systems management services, etc.), platform as a service (PaaS) (e.g., web services, streaming services, application development services, etc.), and other types of services such as desktop as a service (DaaS), information technology management as a service (ITaaS), managed software as a service (MSaaS), mobile backend as a service (MBaaS), etc.
  • SaaS software as a service
  • IaaS infrastructure as a service
  • PaaS platform as a service
  • DaaS desktop as a service
  • ITaaS information technology management as a service
  • MSaaS managed software as a service
  • MaaS mobile backend as a service
  • Certain aspects of the present disclosure include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present disclosure could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.
  • the present disclosure also relates to an apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer.
  • a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of non-transient computer-readable storage medium suitable for storing electronic instructions.
  • the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Optics & Photonics (AREA)
  • Microscoopes, Condenser (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
US16/027,056 2018-07-03 2018-07-03 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging Active US10169852B1 (en)

Priority Applications (14)

Application Number Priority Date Filing Date Title
US16/027,056 US10169852B1 (en) 2018-07-03 2018-07-03 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US16/233,258 US10467740B1 (en) 2018-07-03 2018-12-27 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
KR1020217003240A KR102605314B1 (ko) 2018-07-03 2019-05-21 초해상도 이미징에 대한 피드백을 제공하고 이의 정확도를 개선하기 위한 시스템, 장치 및 방법
EP19830030.3A EP3818407B1 (en) 2018-07-03 2019-05-21 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN201980040709.5A CN112313554B (zh) 2018-07-03 2019-05-21 用于提供超分辨率成像的准确度的反馈并改进超分辨率成像的准确度的系统、装置和方法
PCT/US2019/033293 WO2020009749A1 (en) 2018-07-03 2019-05-21 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN202310392846.3A CN117350918A (zh) 2018-07-03 2019-05-21 用于提供超分辨率成像的准确度的反馈并改进超分辨率成像的准确度的系统、装置和方法
JP2020572647A JP7011866B2 (ja) 2018-07-03 2019-05-21 超解像イメージングの精度に関するフィードバックを行い、その精度を改善するためのシステム、デバイス、および方法
TW108117905A TWI829694B (zh) 2018-07-03 2019-05-23 用於提供超解析度成像之精準度回饋及改良超解析度成像之精準度之系統、裝置及方法
US16/576,732 US10789695B2 (en) 2018-07-03 2019-09-19 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US17/029,703 US10970831B2 (en) 2018-07-03 2020-09-23 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US17/222,425 US11748846B2 (en) 2018-07-03 2021-04-05 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
JP2022001823A JP7431458B2 (ja) 2018-07-03 2022-01-07 超解像イメージングの精度に関するフィードバックを行い、その精度を改善するためのシステム、デバイス、および方法
US18/240,910 US11948270B2 (en) 2018-07-03 2023-08-31 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US16/027,056 US10169852B1 (en) 2018-07-03 2018-07-03 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/233,258 Continuation US10467740B1 (en) 2018-07-03 2018-12-27 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Publications (1)

Publication Number Publication Date
US10169852B1 true US10169852B1 (en) 2019-01-01

Family

ID=64736339

Family Applications (6)

Application Number Title Priority Date Filing Date
US16/027,056 Active US10169852B1 (en) 2018-07-03 2018-07-03 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US16/233,258 Active US10467740B1 (en) 2018-07-03 2018-12-27 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US16/576,732 Active US10789695B2 (en) 2018-07-03 2019-09-19 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US17/029,703 Active US10970831B2 (en) 2018-07-03 2020-09-23 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US17/222,425 Active 2038-09-20 US11748846B2 (en) 2018-07-03 2021-04-05 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US18/240,910 Active US11948270B2 (en) 2018-07-03 2023-08-31 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Family Applications After (5)

Application Number Title Priority Date Filing Date
US16/233,258 Active US10467740B1 (en) 2018-07-03 2018-12-27 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US16/576,732 Active US10789695B2 (en) 2018-07-03 2019-09-19 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US17/029,703 Active US10970831B2 (en) 2018-07-03 2020-09-23 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US17/222,425 Active 2038-09-20 US11748846B2 (en) 2018-07-03 2021-04-05 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US18/240,910 Active US11948270B2 (en) 2018-07-03 2023-08-31 Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Country Status (7)

Country Link
US (6) US10169852B1 (https=)
EP (1) EP3818407B1 (https=)
JP (2) JP7011866B2 (https=)
KR (1) KR102605314B1 (https=)
CN (2) CN112313554B (https=)
TW (1) TWI829694B (https=)
WO (1) WO2020009749A1 (https=)

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10467740B1 (en) * 2018-07-03 2019-11-05 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US10481379B1 (en) * 2018-10-19 2019-11-19 Nanotronics Imaging, Inc. Method and system for automatically mapping fluid objects on a substrate
US10599951B2 (en) * 2018-03-28 2020-03-24 Kla-Tencor Corp. Training a neural network for defect detection in low resolution images
CN112653834A (zh) * 2020-12-01 2021-04-13 广东鼎诚电子科技有限公司 超分辨率扫描成像方法、系统和存储介质
CN113112405A (zh) * 2021-04-12 2021-07-13 广州超视计生物科技有限公司 超分辨率显微镜图像的自适应校正方法及sim-odt双模态系统
US11067786B2 (en) * 2019-06-07 2021-07-20 Leica Microsystems Inc. Artifact regulation methods in deep model training for image transformation
US20210407042A1 (en) * 2018-11-16 2021-12-30 Google Llc Generating super-resolution images using neural networks
CN114223016A (zh) * 2019-06-28 2022-03-22 威思制药公司 载玻片的扫描/预扫描质量控制
US11335443B1 (en) 2020-09-07 2022-05-17 OpenNano Pte. Ltd. Phenotypic patient data derivation from economic data
US20220261973A1 (en) * 2019-04-18 2022-08-18 Hitachi High-Tech Corporation Charged Particle Beam Apparatus
CN114943642A (zh) * 2021-02-09 2022-08-26 华为技术有限公司 图像处理方法、装置、系统及电子设备
US20220350129A1 (en) * 2017-11-20 2022-11-03 Scopio Labs Ltd. Accelerating digital microscopy scans using empty/dirty area detection
CN115482509A (zh) * 2022-10-13 2022-12-16 中国铁塔股份有限公司 烟火识别方法、装置、电子设备及存储介质
US20230186502A1 (en) * 2020-02-03 2023-06-15 Nanotronics Imaging, Inc. Deep Photometric Learning (DPL) Systems, Apparatus and Methods
US12230013B2 (en) 2018-12-31 2025-02-18 Asml Netherlands B.V. Fully automated SEM sampling system for e-beam image enhancement
CN119540623A (zh) * 2024-11-12 2025-02-28 河海大学 一种融合历史高分土地覆盖数据的深度时空超分辨率制图方法

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102712777B1 (ko) * 2018-10-29 2024-10-04 삼성전자주식회사 전자 장치 및 전자 장치의 제어 방법
WO2021206687A1 (en) 2020-04-07 2021-10-14 Purdue Research Foundation Image upsampling
CN111553840B (zh) * 2020-04-10 2023-06-27 北京百度网讯科技有限公司 图像超分辨的模型训练和处理方法、装置、设备和介质
US20230345115A1 (en) * 2020-09-21 2023-10-26 Molecular Devices, Llc Method and system of developing an imaging configuration to optimize performance of a microscopy system
TW202238110A (zh) * 2021-02-23 2022-10-01 以色列商奧寶科技有限公司 使用混合成像系統之自動光學檢測
US20220270212A1 (en) * 2021-02-25 2022-08-25 Kla Corporation Methods for improving optical inspection and metrology image quality using chip design data
WO2022197001A1 (en) 2021-03-16 2022-09-22 +Samsung Electronics Co., Ltd. Method and electronic device for removing artifact in high resolution image
CN113191495A (zh) * 2021-03-26 2021-07-30 网易（杭州）网络有限公司 超分模型的训练及人脸识别方法、装置、介质及电子设备
KR102730875B1 (ko) * 2021-07-27 2024-11-15 한국과학기술원 서로 다른 공간 해상도를 갖는 방사선 영상을 위한 영상 업샘플링 방법 및 물질 분별 방법
US20240013365A9 (en) * 2021-10-04 2024-01-11 Kla Corporation Unsupervised or self-supervised deep learning for semiconductor-based applications
CN116152690B (zh) * 2021-11-17 2025-11-28 瑞昱半导体股份有限公司 视频分类系统与方法以及神经网络训练系统与方法
US12423772B2 (en) * 2022-02-17 2025-09-23 Fei Company Systems and methods for hybrid enhancement of scanning electron microscope images
TWI788251B (zh) * 2022-04-01 2022-12-21 偉詮電子股份有限公司 超解析度影像的重建方法以及超解析度影像的重建系統
US12536613B2 (en) * 2022-12-06 2026-01-27 Samsung Electronics Co., Ltd. Method and system with image super-resolution
JPWO2024203900A1 (https=) * 2023-03-24 2024-10-03
JP2024140962A (ja) * 2023-03-28 2024-10-10 シスメックス株式会社 分析方法、検体分析装置およびプログラム
US12511731B2 (en) * 2023-05-09 2025-12-30 Orbotech Ltd. System and method for three-dimensional imaging of samples using a machine learning algorithm
US12293010B1 (en) 2024-07-08 2025-05-06 AYL Tech, Inc. Context-sensitive portable messaging based on artificial intelligence

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7570796B2 (en) 2005-11-18 2009-08-04 Kla-Tencor Technologies Corp. Methods and systems for utilizing design data in combination with inspection data
US20090257684A1 (en) * 2008-04-15 2009-10-15 Sony Corporation Method and apparatus for suppressing ringing artifacts within super-resolution image processing
US7676077B2 (en) 2005-11-18 2010-03-09 Kla-Tencor Technologies Corp. Methods and systems for utilizing design data in combination with inspection data
US20110037894A1 (en) * 2009-08-11 2011-02-17 Google Inc. Enhanced image and video super-resolution processing
US20120201436A1 (en) * 2011-02-03 2012-08-09 Jonathan Oakley Method and system for image analysis and interpretation
US20120213452A1 (en) * 2011-02-17 2012-08-23 Panasonic Corporation Image processing apparatus, image processing method, computer program and imaging apparatus
US20120223214A1 (en) * 2011-03-03 2012-09-06 California Institute Of Technology Light Guided Pixel
US20130070060A1 (en) * 2011-09-19 2013-03-21 Pelican Imaging Corporation Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion
US20140177706A1 (en) * 2012-12-21 2014-06-26 Samsung Electronics Co., Ltd Method and system for providing super-resolution of quantized images and video
US20140267890A1 (en) * 2013-03-13 2014-09-18 Pelican Imaging Corporation Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing
US20150071567A1 (en) * 2013-09-06 2015-03-12 Kabushiki Kaisha Toshiba Image processing device, image processing method and non-transitory computer readable medium
US20150172726A1 (en) * 2013-12-16 2015-06-18 Samsung Electronics Co., Ltd. Method for real-time implementation of super resolution
US20170148226A1 (en) 2015-11-19 2017-05-25 Kla-Tencor Corporation Generating simulated images from design information
US20170193680A1 (en) 2016-01-04 2017-07-06 Kla-Tencor Corporation Generating high resolution images from low resolution images for semiconductor applications
US20170193400A1 (en) 2015-12-31 2017-07-06 Kla-Tencor Corporation Accelerated training of a machine learning based model for semiconductor applications
US20170220000A1 (en) * 2014-08-01 2017-08-03 The Regents Of The University Of California Device and method for iterative phase recovery based on pixel super-resolved on-chip holography
US20170323223A1 (en) * 2014-11-13 2017-11-09 Mizuho Information & Research Institute, Inc. System, method, and program for predicting information

Family Cites Families (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7447382B2 (en) * 2004-06-30 2008-11-04 Intel Corporation Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation
US8031940B2 (en) * 2006-06-29 2011-10-04 Google Inc. Recognizing text in images using ranging data
JP4699406B2 (ja) 2007-02-28 2011-06-08 シャープ株式会社 画像処理装置、画像処理方法、画像処理装置制御プログラム、及び該プログラムを記録したコンピュータ読み取り可能な記録媒体
JP2009246935A (ja) * 2008-03-14 2009-10-22 Sanyo Electric Co Ltd 画像処理装置およびそれを搭載した撮像装置
US20090297017A1 (en) * 2008-03-25 2009-12-03 Hudgings Janice A High resolution multimodal imaging for non-destructive evaluation of polysilicon solar cells
US8866920B2 (en) * 2008-05-20 2014-10-21 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
US8385422B2 (en) * 2008-08-04 2013-02-26 Kabushiki Kaisha Toshiba Image processing apparatus and image processing method
JP2011059897A (ja) * 2009-09-08 2011-03-24 Fujifilm Corp 画像解析装置、画像解析方法およびプログラム
US8237786B2 (en) * 2009-12-23 2012-08-07 Applied Precision, Inc. System and method for dense-stochastic-sampling imaging
JP5645052B2 (ja) * 2010-02-12 2014-12-24 国立大学法人東京工業大学 画像処理装置
JP5645051B2 (ja) * 2010-02-12 2014-12-24 国立大学法人東京工業大学 画像処理装置
US9569664B2 (en) 2010-10-26 2017-02-14 California Institute Of Technology Methods for rapid distinction between debris and growing cells
WO2013069564A1 (ja) * 2011-11-08 2013-05-16 富士フイルム株式会社 撮像装置およびその制御方法
US9075013B2 (en) * 2012-04-29 2015-07-07 Periodic Structures, Inc. Apparatus and methods for microscopy having resolution beyond the Abbe limit
US9743891B2 (en) * 2012-07-09 2017-08-29 The Trustees Of The University Of Pennsylvania Super-resolution tomosynthesis imaging systems and methods
US9494785B2 (en) * 2012-12-07 2016-11-15 Purdue Research Foundation Single image super-resolution microscopy and telescope systems
CN105074818B (zh) * 2013-02-21 2019-08-13 杜比国际公司 音频编码系统、用于产生比特流的方法以及音频解码器
JP2015025758A (ja) * 2013-07-26 2015-02-05 Hoya株式会社 基板検査方法、基板製造方法および基板検査装置
JP2015052663A (ja) 2013-09-06 2015-03-19 キヤノン株式会社 画像処理方法、画像処理装置、撮像装置およびプログラム
JP2015197818A (ja) 2014-04-01 2015-11-09 キヤノン株式会社 画像処理装置およびその方法
US9384386B2 (en) * 2014-08-29 2016-07-05 Motorola Solutions, Inc. Methods and systems for increasing facial recognition working rang through adaptive super-resolution
US10437034B2 (en) * 2014-10-14 2019-10-08 Nanotronics Imaging, Inc. Unique oblique lighting technique using a brightfield darkfield objective and imaging method relating thereto
WO2016118884A1 (en) * 2015-01-22 2016-07-28 The Regents Of The University Of California Device and method for nanoparticle sizing based on time-resolved on-chip microscopy
CA2914892C (en) 2015-04-30 2023-09-19 Farnoud Kazemzadeh A system, method and apparatus for ultra-resolved ultra-wide field-of-view multispectral and hyperspectral holographic microscopy
US10088663B2 (en) * 2015-05-13 2018-10-02 The Regents Of The University Of California Device and method for tunable vapor condensed nanolenses
JP2017028024A (ja) * 2015-07-17 2017-02-02 富士通株式会社 部品搭載基板、部品内蔵基板、部品搭載基板の製造方法および部品内蔵基板の製造方法
WO2017055609A1 (en) * 2015-09-30 2017-04-06 Piksel, Inc Improved video stream delivery via adaptive quality enhancement using error correction models
US20170168285A1 (en) * 2015-12-14 2017-06-15 The Regents Of The University Of California Systems and methods for image reconstruction
WO2017196885A1 (en) * 2016-05-10 2017-11-16 The Regents Of The University Of California Method and device for high-resolution color imaging using merged images from holographic and lens-based devices
US10346740B2 (en) * 2016-06-01 2019-07-09 Kla-Tencor Corp. Systems and methods incorporating a neural network and a forward physical model for semiconductor applications
US11024009B2 (en) 2016-09-15 2021-06-01 Twitter, Inc. Super resolution using a generative adversarial network
JP7249326B2 (ja) * 2017-07-31 2023-03-30 アンスティテュ パストゥール 単一分子局在化顕微鏡法によって取得された回折限界画像からの高密度超解像度画像の再構築を改善する方法、装置、及びコンピュータプログラム
US11222415B2 (en) * 2018-04-26 2022-01-11 The Regents Of The University Of California Systems and methods for deep learning microscopy
US10169852B1 (en) 2018-07-03 2019-01-01 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US10481379B1 (en) * 2018-10-19 2019-11-19 Nanotronics Imaging, Inc. Method and system for automatically mapping fluid objects on a substrate
US11137719B2 (en) * 2018-12-11 2021-10-05 University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for improved digital holography and display incorporating same
US11776108B2 (en) * 2020-08-05 2023-10-03 KLA Corp. Deep learning based defect detection
US11448603B1 (en) * 2021-09-02 2022-09-20 Axiomatique Technologies, Inc. Methods and apparatuses for microscopy and spectroscopy in semiconductor systems

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7570796B2 (en) 2005-11-18 2009-08-04 Kla-Tencor Technologies Corp. Methods and systems for utilizing design data in combination with inspection data
US7676077B2 (en) 2005-11-18 2010-03-09 Kla-Tencor Technologies Corp. Methods and systems for utilizing design data in combination with inspection data
US20090257684A1 (en) * 2008-04-15 2009-10-15 Sony Corporation Method and apparatus for suppressing ringing artifacts within super-resolution image processing
US20110037894A1 (en) * 2009-08-11 2011-02-17 Google Inc. Enhanced image and video super-resolution processing
US20120201436A1 (en) * 2011-02-03 2012-08-09 Jonathan Oakley Method and system for image analysis and interpretation
US20120213452A1 (en) * 2011-02-17 2012-08-23 Panasonic Corporation Image processing apparatus, image processing method, computer program and imaging apparatus
US20120223214A1 (en) * 2011-03-03 2012-09-06 California Institute Of Technology Light Guided Pixel
US20130070060A1 (en) * 2011-09-19 2013-03-21 Pelican Imaging Corporation Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion
US20140177706A1 (en) * 2012-12-21 2014-06-26 Samsung Electronics Co., Ltd Method and system for providing super-resolution of quantized images and video
US20140267890A1 (en) * 2013-03-13 2014-09-18 Pelican Imaging Corporation Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing
US20150071567A1 (en) * 2013-09-06 2015-03-12 Kabushiki Kaisha Toshiba Image processing device, image processing method and non-transitory computer readable medium
US20150172726A1 (en) * 2013-12-16 2015-06-18 Samsung Electronics Co., Ltd. Method for real-time implementation of super resolution
US20170220000A1 (en) * 2014-08-01 2017-08-03 The Regents Of The University Of California Device and method for iterative phase recovery based on pixel super-resolved on-chip holography
US20170323223A1 (en) * 2014-11-13 2017-11-09 Mizuho Information & Research Institute, Inc. System, method, and program for predicting information
US20170148226A1 (en) 2015-11-19 2017-05-25 Kla-Tencor Corporation Generating simulated images from design information
US20170193400A1 (en) 2015-12-31 2017-07-06 Kla-Tencor Corporation Accelerated training of a machine learning based model for semiconductor applications
US20170193680A1 (en) 2016-01-04 2017-07-06 Kla-Tencor Corporation Generating high resolution images from low resolution images for semiconductor applications

Non-Patent Citations (11)

* Cited by examiner, † Cited by third party
Title
A mixed-scale dense convolutional neural network for image analysis; Daniël M. Pelt and James A. Sethian; PNAS Jan. 9, 2018 115 (2) 254-259; published ahead of print Dec. 26, 2017.
Deconvolution and Checkerboard Artifacts; Augustus Odena, Vince Dumoulin; Chris Olah; Oct. 17, 2016.
EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis; Medhi S.M. Sajjadi, Bernhard Scholkopf, Michael Hirsch; arXiv:1612.07919v2 [cs.CV] Jul. 30, 2017.
Generative Adversarial Nets; Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio; arXiv:1406.2661v1 [stat.ML] Jun. 10, 2014.
Image-to-Image Translation with Conditional Adversarial Networks; Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros; arXiv:1611.07004v2 [cs.CV] Nov. 22, 2017.
Learning from Simulated and Unsupervised Images through Adversarial Training; Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb; arXiv:1612.07828v2 [cs.CV] Jul. 19, 2017.
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network; Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi; arXiv:1609.04802v5 [cs.CV] May 25, 2017.
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network; Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, Zehan Wang; arXiv:1609.05158v2 [cs.CV] Sep. 23, 2016.
Single-Image Super-Resolution: A Benchmark; Chih-Yuan Yang, Chao Ma, and Ming-Hsuan Yang; University of California at Merced, USA; D. Springer International Publishing Switzerland 2014.
Super-Resolution with Deep Convolutional Sufficient Statistics; Joan Bruna, Pablo Sprechmann, Yann LeCun; arXiv:1511.05666v4 [cs.CV] Mar. 1, 2016.
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks; Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros; arXiv:1703.10593v5 [cs.CV] Aug. 30, 2018.

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11828927B2 (en) * 2017-11-20 2023-11-28 Scopio Labs Ltd. Accelerating digital microscopy scans using empty/dirty area detection
US20220350129A1 (en) * 2017-11-20 2022-11-03 Scopio Labs Ltd. Accelerating digital microscopy scans using empty/dirty area detection
US10599951B2 (en) * 2018-03-28 2020-03-24 Kla-Tencor Corp. Training a neural network for defect detection in low resolution images
US11948270B2 (en) 2018-07-03 2024-04-02 Nanotronics Imaging , Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US10467740B1 (en) * 2018-07-03 2019-11-05 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US11748846B2 (en) 2018-07-03 2023-09-05 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US10970831B2 (en) 2018-07-03 2021-04-06 Nanotronics Imaging, Inc. Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
US12174361B2 (en) 2018-10-19 2024-12-24 Nanotronics Imaging, Inc. Method and system for mapping objects on unknown specimens
US11815673B2 (en) 2018-10-19 2023-11-14 Nanotronics Imaging, Inc. Method and system for mapping objects on unknown specimens
US11333876B2 (en) 2018-10-19 2022-05-17 Nanotronics Imaging, Inc. Method and system for mapping objects on unknown specimens
US10809516B2 (en) 2018-10-19 2020-10-20 Nanotronics Imaging, Inc. Method and system for automatically mapping fluid objects on a substrate
US10481379B1 (en) * 2018-10-19 2019-11-19 Nanotronics Imaging, Inc. Method and system for automatically mapping fluid objects on a substrate
US20210407042A1 (en) * 2018-11-16 2021-12-30 Google Llc Generating super-resolution images using neural networks
US11869170B2 (en) * 2018-11-16 2024-01-09 Google Llc Generating super-resolution images using neural networks
US12230013B2 (en) 2018-12-31 2025-02-18 Asml Netherlands B.V. Fully automated SEM sampling system for e-beam image enhancement
US20220261973A1 (en) * 2019-04-18 2022-08-18 Hitachi High-Tech Corporation Charged Particle Beam Apparatus
US11928801B2 (en) * 2019-04-18 2024-03-12 Hitachi High-Tech Corporation Charged particle beam apparatus
US11067786B2 (en) * 2019-06-07 2021-07-20 Leica Microsystems Inc. Artifact regulation methods in deep model training for image transformation
CN114223016A (zh) * 2019-06-28 2022-03-22 威思制药公司 载玻片的扫描/预扫描质量控制
US12462412B2 (en) * 2020-02-03 2025-11-04 Nanotronics Imaging, Inc. Deep photometric learning (DPL) systems, apparatus and methods
US20230186502A1 (en) * 2020-02-03 2023-06-15 Nanotronics Imaging, Inc. Deep Photometric Learning (DPL) Systems, Apparatus and Methods
US11335443B1 (en) 2020-09-07 2022-05-17 OpenNano Pte. Ltd. Phenotypic patient data derivation from economic data
CN112653834B (zh) * 2020-12-01 2022-04-08 广东鼎诚电子科技有限公司 超分辨率扫描成像方法、系统和存储介质
CN112653834A (zh) * 2020-12-01 2021-04-13 广东鼎诚电子科技有限公司 超分辨率扫描成像方法、系统和存储介质
CN114943642A (zh) * 2021-02-09 2022-08-26 华为技术有限公司 图像处理方法、装置、系统及电子设备
CN113112405B (zh) * 2021-04-12 2022-04-12 广州超视计生物科技有限公司 超分辨率显微镜图像的自适应校正方法及sim-odt双模态系统
CN113112405A (zh) * 2021-04-12 2021-07-13 广州超视计生物科技有限公司 超分辨率显微镜图像的自适应校正方法及sim-odt双模态系统
CN115482509A (zh) * 2022-10-13 2022-12-16 中国铁塔股份有限公司 烟火识别方法、装置、电子设备及存储介质
CN119540623A (zh) * 2024-11-12 2025-02-28 河海大学 一种融合历史高分土地覆盖数据的深度时空超分辨率制图方法

Also Published As

Publication number Publication date
TW202018366A (zh) 2020-05-16
US20200013155A1 (en) 2020-01-09
CN112313554A (zh) 2021-02-02
JP2022050565A (ja) 2022-03-30
KR20210025104A (ko) 2021-03-08
CN117350918A (zh) 2024-01-05
EP3818407A4 (en) 2022-04-13
CN112313554B (zh) 2023-04-28
TWI829694B (zh) 2024-01-21
WO2020009749A1 (en) 2020-01-09
US10467740B1 (en) 2019-11-05
US20210224966A1 (en) 2021-07-22
US11748846B2 (en) 2023-09-05
JP7011866B2 (ja) 2022-01-27
EP3818407B1 (en) 2025-01-08
US20230419444A1 (en) 2023-12-28
JP7431458B2 (ja) 2024-02-15
KR102605314B1 (ko) 2023-11-22
EP3818407A1 (en) 2021-05-12
JP2021528775A (ja) 2021-10-21
US20210012473A1 (en) 2021-01-14
US10789695B2 (en) 2020-09-29
US11948270B2 (en) 2024-04-02
US10970831B2 (en) 2021-04-06

Similar Documents

Publication Publication Date Title
US11948270B2 (en) Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging
CN110189288B (zh) 生成可用于半导体样本的检查的训练集的方法和其系统
US10818000B2 (en) Iterative defect filtering process
US12174361B2 (en) Method and system for mapping objects on unknown specimens
CN113439276A (zh) 基于机器学习的半导体样本中的缺陷分类
JP2022013667A (ja) 半導体試料の画像のセグメンテーション
KR102956982B1 (ko) 반도체 시편에서의 결함들의 기계 학습 기반 분류

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

Year of fee payment: 4