JP2023115913A - Method and system for visual inspection of sensor chip - Google Patents

Abstract

To provide a method, a system and a computer program product for detection based on an image of an abnormality event (e.g., defects) in an environment of chip manufacture.SOLUTION: A method for visually detecting a defect includes: identifying an area of concern by processing an image of a chip (e.g., pattern matching/template matching); dividing the identified area of concern into a first image and a second image (e.g., image segmentation); analyzing the first image and/or the second image using a trained model to determine whether or not a defect is detected in the first image and/or the second image; and issuing an alert if a defect is detected within the first image and/or the second image.SELECTED DRAWING: Figure 4

Description

COPYRIGHT NOTICE A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by any person of this patent document or this patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. reserves any right to

FIELD OF THE DISCLOSURE The present disclosure relates generally to visual detection of anomalies in chips, and more particularly to detection of anomalies based on image processing and analysis.

To ensure quality, chip manufacturers perform inspections for defects. One method of identifying defects is manual inspection. However, manually inspecting chips for defects can be inefficient (eg, time consuming, labor intensive, and/or error prone). One solution to achieving more consistent and cost-effective defect detection is to use vision-based systems (e.g., camera-based solutions for detecting anomalies). be. This solution involves receiving an image and running an image processing algorithm on the received image to identify anomalous events (eg, defects). When an abnormal event is detected, an alert or alarm can be triggered soliciting corrective action.

Embodiments of the present disclosure seek to visually detect defects/anomalies based on image processing and analysis using models trained using datasets containing defect-free images.

Accordingly, it is an aspect of the present disclosure to provide a method of visually inspecting chips for defects using image segmentation. More specifically, processing an image of the chip to identify an area of interest, splitting the identified area of interest into a first image and a second image, and using the trained model analyzing the first image and/or the second image to determine whether defects are detected in the first image and/or the second image; and issuing an alert if a defect is detected in the image of 2.

Existing solutions require relatively large datasets to train models to detect defects. However, the size of the dataset can be reduced by using only datasets containing only defect-free samples. In addition, unconfirmed/unknown defects can also be detected by using only data sets containing only defect-free samples. That is, rather than the model having to be trained to detect specific defects, the model is trained to detect deviations from the expected state of the components.

For example, a captured image may have a scale of 4000*4000 pixels. Raw image data in RGB or grayscale format may contain image features that are not of interest, may only increase training time rather than contribute to model training, and may sometimes be a source of noise. There are a lot of things that cause it. Therefore, processing such images and training an artificial intelligence (AI) model using such images requires significant computational resources, and the training results in accuracy in defect/anomaly identification. may be lower. Using image segmentation, an image is divided into two or more smaller parts, each part representing an area of interest (eg, a unique repeating pattern). A set of templates (eg, tile segments) is created using a technique called template matching. Each tile portion can represent an area of interest. In another example, the methods and systems search for matching patterns in the image and, when found, crop that portion from the entire image.

By pre-processing the images to extract only the key features (e.g., areas of interest) that contribute to anomaly classification, the method and system provide more efficient defect/anomaly identification in both AI model implementation and training. enable In some embodiments, computer vision libraries can be used to pre-process the images. In addition, image segmentation enables methods and systems to target different regions of the chip and/or target different types of defects/anomalies, whether known or unknown. .

An autoencoder is an artificial neural network used to recreate a given input for the purpose of scale reduction, the process of reducing the number of features that represent the input data. A set of inputs is encoded and then useful information is extracted. For example, you want to extract information (eg identify defects) using a 256x256 pixel grayscale image as an example. An image size of 256×256 pixels corresponds to an input vector of magnitude 65,536. Using a 4000x3000 pixel image, there are 12 million scales to analyze. As this example shows, the scale increases exponentially with the size of the image. However, we generally do not need to use all 65,536 magnitudes to identify defects. Defects/abnormalities can be identified from deviations of a particular area of interest, and exemplary defects are shown in FIGS. 2A-2B and 3A-3B.

An autoencoder can be divided into two parts, an encoder and a decoder. The encoder compresses the representation of the input (eg, reduces image size by selecting/extracting useful information). The decoder performs the reverse process (eg, recreates the input as closely as possible). For example, the decoder can produce radially symmetric images. During this process, the aim is to preserve the maximum amount of information when encoding, thus minimizing reconstruction errors when decoding. Variational autoencoders are autoencoders aimed at regenerating data. The autoencoder is trained with samples consisting only of defect-free images in the range of 500 to 1000 sample images. A model is generated and used for predictive analysis. A test image is then predicted by this model. Any image with large reconstruction loss (eg, reconstruction loss above a predefined threshold) is then classified as defective. In other words, the tile images are used to train an autoencoder to classify defects based on the reconstruction loss for each image.

In one embodiment, a method for detecting defects in a chip using images comprises receiving a reference image of the chip, the reference image including at least one area of interest; receiving an image, wherein the input image substantially includes the at least one area of interest; and reconstructing a reference image using the input image and determining a reconstruction loss. the reconstruction loss is indicative of information lost during reconstruction; and determining whether the reconstruction loss exceeds a predefined threshold. be.

In another embodiment, a system for performing a method of using images to detect defects in a chip, comprising: at least one memory configured to store computer program code; and at least one processor configured to access memory and operate according to the computer program code, the computer program code receiving a reference image of the chip, the reference image representing at least one area of interest. receiving an input image, wherein the input image substantially includes the at least one area of interest; reconstructing a reference image using the input image; determining a loss, wherein the reconstruction loss indicates information lost during reconstruction; and determining whether the reconstruction loss exceeds a predefined threshold. system, including

In another embodiment, a computer-implemented method of training an autoencoder to train a neural network for visually detecting defects in a chip, comprising collecting a set of entire images; , using this set of whole images to generate a first training set, training a neural network using the first training set in the first model, and generating a second training set said generating a second training set comprising at least one segment of the whole image; and training a neural network using the second training set in the second model. and training a second model to analyze first and second symmetrical halves.

In another embodiment, a method of visually inspecting a chip for defects comprises processing an image of the chip to identify an area of interest; Splitting into images and analyzing the first image and/or the second image using the trained model to determine whether defects are detected in the first image and/or the second image and issuing an alert if a defect is detected in the first image and/or the second image.

In another embodiment, a system for performing a method of detecting defects in a chip, comprising: at least one memory configured to store computer program code; accessing the at least one memory; and at least one processor configured to operate according to the computer program code, the computer program code processing the image of the chip to identify an area of interest, and the identified area of interest to the first and analyzing the first image and/or the second image using the trained model to detect defects in the first image and/or the second image is detected, and issuing an alert if a defect is detected in the first image and/or the second image.

In another embodiment, a computer-implemented method of training an autoencoder to train a neural network for visually detecting defects in a chip, comprising collecting a set of entire images; , using this set of whole images to generate a first training set, training a neural network using the first training set in the first model, and generating a second training set said generating a second training set comprising at least one segment of the whole image; and training a neural network using the second training set in the second model. and training a second model to analyze first and second symmetric halves.

An aspect of any one or more of the foregoing embodiments includes detecting a defect in the input image if the reconstruction loss exceeds a predefined threshold.

An aspect of any one or more of the foregoing embodiments includes generating a reference image from a set of defect-free images of a plurality of chips.

An aspect of any one or more of the foregoing embodiments includes dividing at least one area of interest into first and second segments.

An aspect of any one or more of the foregoing embodiments includes the first and second segments being radially symmetrical.

An aspect of any one or more of the foregoing embodiments is identifying at least one area of interest using a template, wherein the template identifies at least one known area associated with the chip. The identifying includes providing a predetermined input pattern of areas of interest.

An aspect of any one or more of the foregoing embodiments includes the chip comprising a sensor chip.

An aspect of any one or more of the foregoing embodiments processes the first image and the second image if the first image and the second image are defect-free, and trains the Including incorporation into datasets.

An aspect of any one or more of the foregoing embodiments is processing the first image and the second image if the first image or the second image is defect-free, and referencing embedding in an image; and receiving an input image, wherein the input image substantially includes an area of interest; reconstructing a reference image using the input image; and determining a reconstruction loss. determining whether the reconstruction loss is indicative of information lost during reconstruction; and determining whether the reconstruction loss exceeds a predefined threshold.

An aspect of any one or more of the foregoing embodiments includes the first image and the second image being radially symmetrical to each other.

An aspect of any one or more of the foregoing embodiments includes generating a trained model using a dataset containing defect-free images.

An aspect of any one or more of the foregoing embodiments includes creating a template for a type associated with the chip, wherein the template comprises at least one known template for that type associated with the chip. Providing a predetermined input pattern of the area of interest of the chip and processing the image of the chip to identify the area of interest includes performing template matching.

A system-on-chip (SoC) comprising any one or more of the above embodiments or aspects of embodiments described herein.

One or more means for implementing any one or more of the above embodiments or aspects of the embodiments described herein.

Any aspect in combination with any one or more other aspects.

Any one or more of the features disclosed herein.

Any one or more of the features substantially disclosed herein.

Any one or more of the features substantially disclosed herein in combination with any one or more other features substantially disclosed herein.

Any one of the aspects/features/embodiments in combination with any one or more of the other aspects/features/embodiments.

Use of any one or more of the aspects or features disclosed herein.

The data storage comprises a non-temporary storage device, the non-temporary storage device being on-chip memory within the processor, registers of the processor, on-board memory co-located with the processor on a processing board, accessible by the processor via a bus memory, magnetic media, optical media, solid-state media, input-output buffers, memory of input-output components that communicate with the processor, network communication buffers, and network-connected components that communicate with the processor via network interfaces Any of the above embodiments, or aspects of those embodiments, comprising at least one of:

Note that any feature described in this specification may be claimed in combination with any other feature described in this specification, whether or not those features originate from the same embodiment described. be understood.

The terms "defect", "abnormality", or "abnormal event", and variations thereof, are used herein to deviate from or change the expected or desired process state or component state in the process environment. Refers to an event or condition that fails to meet. Exemplary cases of anomalies that may be identified within a chip (eg, sensor chip) are shown in FIGS. 2A-2B and 3A-3B.

The term "image acquisition device" and variations thereof, as used herein, is any device capable of acquiring images in any region of the electromagnetic spectrum, including the ultraviolet, visible, near-infrared, or infrared spectrum. and includes digital cameras, video cameras, infrared cameras, surveillance cameras, and image sensors capable of producing digital and thermal images.

The term "automatic" and variations thereof, as used herein, refer to processes or actions that occur without tangible human input when any process or action is performed. However, even if the performance of the process or action uses tangible or immaterial human input, the process or action can be automatic if the input is received prior to the performance of the process or action. . Human input is considered tangible if such input affects how a process or action is performed. Human input that agrees to perform a process or action is not considered "substantial."

The term "computer-readable medium" as used herein refers to any tangible storage that participates in providing instructions to a processor for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, NVRAM or magnetic or optical disks. Volatile media includes dynamic memory, such as main memory. Common forms of computer readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tape or any other magnetic medium, magneto-optical media, CD-ROMs, any other optical medium, punch cards, paper tapes. , any other physical medium having a pattern of holes, RAM, PROM and EPROM, flash EPROM, solid state media such as memory cards, any other memory chip or memory cartridge, or from which a computer reads Any other medium capable of It should be appreciated that when the computer-readable medium is configured as a database, the database may be a graph database as described herein. Accordingly, the present disclosure is considered to include tangible storage media, and art-recognized equivalents and successor media, in which the software implementations of the present disclosure are stored.

The terms "determine," "calculate," and "compute," and variations thereof, are used interchangeably herein and include any type of method, process, mathematical operation, or technique. .

The term "module" is used herein to refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or hardware capable of performing the functions associated with that element. Refers to a combination of software. Also, while the disclosure is described in terms of exemplary embodiments, it should be understood that individual aspects of the disclosure may be separately claimed.

The following description and related drawings teach the best mode of the invention. For the purpose of teaching inventive principles, some conventional aspects of the best mode may be simplified or omitted. The appended claims define the scope of the invention. Note that some aspects of the best mode may not fall within the scope of the invention as defined by the claims. Accordingly, those skilled in the art will recognize variations from the best mode that fall within the scope of this invention. Those skilled in the art will appreciate that the features described below can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific examples described below, but only by the claims and their equivalents.

The present disclosure is described with respect to the accompanying figures.

FIG. 12 shows an exemplary image of a sensor chip according to embodiments of the present disclosure; [0014] Figure 4 illustrates an example of a template and/or tile portion according to embodiments of the present disclosure; [0014] Figure 4 illustrates an example of a template and/or tile portion according to embodiments of the present disclosure; 4A-4D illustrate examples of various defects according to embodiments of the present disclosure; 4A-4D illustrate examples of various defects according to embodiments of the present disclosure; 4A-4D illustrate further examples of various defects according to embodiments of the present disclosure; 4A-4D illustrate further examples of various defects according to embodiments of the present disclosure; 4 is a flowchart describing a method for visual detection of chip defects according to embodiments of the present disclosure; 5 is a flow chart describing another method for visual detection of chip defects according to an embodiment of the present disclosure; FIG. 10 illustrates an exemplary dataset used to train a model, in accordance with embodiments of the present disclosure; 1 depicts an exemplary process environment configured to perform a method, in accordance with an embodiment of the present disclosure; FIG. FIG. 1 illustrates an exemplary computing system configured to perform methods, in accordance with embodiments of the present disclosure;

The ensuing description provides embodiments only and is not intended to limit the scope, applicability, or configuration of the claims. Rather, the ensuing description provides those skilled in the art with an enabling description for implementing the embodiments. It will be appreciated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

In descriptions that include numerical reference numbers, when any reference without an alphabetic sub-reference identifier is used in the plural when a sub-reference identifier is present in a figure, it refers to any reference with a similar reference number. A reference to two or more elements. When such a reference is made in the singular but without identification of a sub-reference identifier, it is a reference to one of like numbered elements but to a particular one of those elements. is not limited. Any express use herein to the contrary or to provide further modification or identification shall prevail.

The exemplary systems and methods of this disclosure are also described in terms of analysis software, analysis modules, and associated analysis hardware. However, to avoid unnecessarily obscuring the present disclosure, the following description omits well-known structures, components and devices that may or may not be omitted from the drawings. or shown in simplified form in the figures or otherwise summarized.

For purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the present disclosure. However, it should be understood that the present disclosure can be practiced in many ways beyond the specific details set forth herein.

It should be appreciated that embodiments of the present disclosure can be utilized in many computing environments, such as WIFI networks and multi-link subnet networks.

Further, although the exemplary embodiments herein show various components of the system collocated, the various components of the system may be remote portions of a distributed network, such as a communication network and/or the Internet, or It should be appreciated that it can reside within a dedicated secure, non-secure and/or encrypted system. It is therefore understood that the components of the system may be integrated into one or more devices, such as enterprise servers, or may be collocated on specific nodes of distributed networks such as analog and/or digital communications networks. want to be As will be appreciated from the discussion below, and for computational efficiency reasons, the components of the system can be placed anywhere within the distributed network without affecting the operation of the system. For example, various components may be located within a local server, at one or more of the user's premises, or some combination thereof.

The following description and related drawings teach the best mode of the invention. For the purpose of teaching inventive principles, some conventional aspects of the best mode may be simplified or omitted. The appended claims define the scope of the invention. Note that some aspects of the best mode may not fall within the scope of the invention as defined by the claims. Accordingly, those skilled in the art will recognize variations from the best mode that fall within the scope of this invention. Those skilled in the art will appreciate that the features described below can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific examples described below, but only by the claims and their equivalents.

Anomaly detection can be implemented at various stages of manufacturing. For example, images can be captured and anomaly detection can be performed after each step of the manufacturing process. Anomaly detection performed after each step allows defects that occur during a particular stage of manufacturing to be identified and corrected.

In addition to known/identified defects, there may be unknown/unidentified defects. There is no image data for training as the defects are unknown/unconfirmed. Advantageously, by using defect-free images to observe deviations from a defect-free condition, visual defect detection is improved to identify unknown/unconfirmed defects that may be present.

Image analysis can be performed based on any one or more image analysis algorithms, and includes, in one embodiment, one or more neural networks configured to detect or identify anomalies. Processing image information using a network may be involved. Various embodiments may implement one or more techniques for anomaly detection, including trajectory analysis, object-based image recognition, and pixel-based image analysis.

Any suitable image pre-processing step may be performed prior to image analysis for the purposes of this disclosure and/or the image analysis may include any suitable image pre-processing step, which suitable image pre-processing step may include: includes, but is not limited to, noise filtering, cropping, resizing, de-sizing, subsampling, etc. of the received image.

In some embodiments, the image analysis step generates a score that indicates or identifies the likelihood that the anomaly detection is accurate (e.g., the likelihood that image information interpreted to represent an anomaly actually represents a true anomaly). Or it may involve determining/calculating other indicators (eg, confidence scores). In some embodiments, verification actions (eg, manual inspection) can be triggered when the confidence score is below a threshold. In addition, the defect-free image (and/or portions of the defect-free image) can be mirrored along the central axis to generate an image, which is used to train the model used to detect anomalies. can be used for

FIG. 1A shows an exemplary image 100 of a sensor chip according to embodiments of the disclosure. In some embodiments, the sensor chip image 100 can be bisected along a central axis (e.g., along dotted line A) and mirroring each half of the image 100 produces two additional images. can be generated. In other words, one image is generated by mirroring the left side and the second image is generated by mirroring the right side, thereby generating a data set of three images from one image. be. Although not shown, the image 100 can be horizontally divided into two halves, and each half (eg top and bottom) mirrored to produce a further defect-free image.

1B-1C illustrate examples of templates according to embodiments of the present disclosure. In some embodiments, the areas of interest (eg, tile portions) 102, 104, 106, and 108 can be bisected and mirrored to generate a further defect-free image.

Using image segmentation, the sensor chip image 100 is divided into two or more smaller portions (eg, portions 102, 104, 106, and 108). Each tile portion 102, 104, 106, and 108 represents an area of interest. In addition, image segmentation allows methods and systems to target different regions of chip 100 and/or target different types of defects/anomalies. An enlarged view of areas of interest (eg, tile portions) 102, 104, 106, and 108 is shown in FIG. 1C.

2A-2B and 3A-3B illustrate examples of various chip defects according to embodiments of the present disclosure. Defects are indicated by dotted lines 202, 204, 206 and 208 respectively. Although various defects are shown on area of interest 106 for purposes of illustration, it is understood that defects may exist within any portion of chip 100 . Further, as shown, the defects may be within the same area (eg, area of interest 106), but the defects may be different (eg, different sizes, shapes, etc.). In addition, images without defects/abnormalities can be used to generate defect-free images for model training. For example, in FIG. 2A, a defect is detected on the left side, but if the image in FIG. 2A is bisected and mirrored on the right side, a defect-free image can be produced. Similarly, in FIG. 2B the defect is detected on the right side, but if the image in FIG. 2B is bisected and mirrored on the left side, a defect-free image can be produced.

By pre-processing the images to extract only the key features (e.g., areas of interest) that contribute to anomaly classification, the method and system provide more efficient defect/anomaly identification in both AI model implementation and training. enable In some embodiments, computer vision libraries can be used to pre-process the images. In other words, the image preprocessing produces input vectors that have a smaller magnitude than the original image.

FIG. 4 shows a flowchart describing a process 400 for visual detection of chip defects according to an embodiment of the present disclosure.

Process 400 may be embodied as machine-readable instructions retained in non-transitory memory that, when read by a processor, such as processor 804 of device 802, cause the processor to perform the steps of process 400.

Method 400 includes analyzing a received image (eg, received from acquisition device 702) and determining whether the received image contains image information indicative of the presence of an anomaly/defect. Process 400 begins at step 402 .

At step 404, an image is received. For example, images may be captured by image acquisition device 702 . Anomaly detection can be performed repeatedly after each stage of manufacturing.

At step 406, the received image is processed. The images received at step 404 may undergo pre-processing (eg, color images may be changed to black and white, etc.). At step 408, the image is segmented into one or more tile portions. For example, templates can be created that identify areas of interest (eg, tile portions) as shown in FIGS. 1B-1C. These areas/segments/parts of interest are analyzed in step 410 . For example, these portions can be compared to a reference image (eg, defect-free image) and/or a model trained to detect deviations in the received image.

If a defect is detected at step 412 (yes/step 414), an alert is triggered at step 416. FIG. Additionally, a confidence score associated with the defect determination may be shown along with the alert. Additionally or alternatively, verification actions may be triggered based on the confidence score. For example, if the confidence score is below a threshold, a verification action (eg, manual inspection) is triggered.

If no defects are detected in step 412 (no/step 418), the images can be added to the training data set. At step 422, process 400 ends.

These steps can be performed continuously while other steps of process 400 are performed until the process is terminated.

FIG. 5 shows a flowchart describing another process 500 for visually detecting chip defects using an autoencoder, according to an embodiment of the present disclosure.

Process 500 may be embodied as machine-readable instructions retained in non-transitory memory that, when read by a processor, such as processor 804 of device 802 , cause the processor to perform the steps of process 500 . Process 500 begins at step 502 .

At step 504, the encoder compresses the representation of the input (eg, reduces image size by selecting/extracting useful information). At step 506, the decoder performs the reverse process (eg, recreates the input as closely as possible). For example, the decoder can produce radially symmetric images. Images with large reconstruction loss (eg, reconstruction loss above a predefined threshold) are then classified as defects. At step 508, the reconstruction loss is determined. If the reconstruction loss is above a predefined threshold (yes/step 512), a defect is detected (step 514). If the reconstruction loss does not exceed the predefined threshold (no/step 516), no defect is detected. At step 518, process 500 ends.

These steps can be performed continuously while other steps of process 500 are performed until the process is terminated.

FIG. 6 shows an exemplary data set 600. FIG. The dataset 600 consists only of defect-free images, which reduces the relative size of the dataset required to properly train the model. In some embodiments, dataset 600 is used to train a model for visually identifying defects in chip images. Additionally, some or all of the images in dataset 600 may be generated by radial symmetry (eg mirroring a defect-free image or even a partially defect-free image about a central axis). Using radial symmetry to generate defect-free images makes it possible to create a larger training set than would be created from using only defect-free images captured when scanning for anomalies. Become.

FIG. 7 illustrates an exemplary process environment 700 configured to perform a method of visually detecting defects according to embodiments of the present disclosure.

The process environment 700 comprises a plurality of image acquisition devices 702a-n, each communicatively coupled to an image processing node 704. As shown in FIG. In some embodiments, each image acquisition device 702a-n can be communicatively coupled to a separate image processing node 704a-n. Image processing node 704 is communicatively coupled to image analyzer 706 . Image analyzer 706 is configured to perform image-based anomaly detection according to any of the methods described herein.

Image analyzer 706 can also be communicatively coupled to database 710, which can comprise a repository of images. For example, a dataset consisting of defect-free images can be stored in database 710 and used to train the image analyzer. Image analyzer 706 can include anomaly detector 708 . Anomaly detector 708 may be configured to detect the presence of anomalies/defects according to any of the methods described herein. Anomaly detector 708 can be further configured to trigger or implement any one or more predefined responses (eg, notifications, flags, and/or alerts) to detection of anomalies in images.

FIG. 8 shows device 802 in system 800 according to an embodiment of the disclosure. A component can be variously embodied and comprise a processor 804 . The term "processor" is used herein exclusively to an electronic hardware component comprising electrical circuitry and connections (e.g., pinouts) for communicating encoded electrical signals to and from the electrical circuitry. Point. Processor 804 further includes electrical circuitry therein, which may further include control units, input/output units, arithmetic logic units, registers, main memory, and/or those received via bus 814 . A single electronic microprocessor or multiprocessor device (e.g., data, instructions, etc.) that can have other components that access information (e.g., data, instructions, etc.), execute instructions, and output data, also via bus 814. For example, it can be embodied as a multi-core).

In other embodiments, processor 804 may be used by other processes and/or process owners, such as within a processing array (e.g., blade, multiprocessor board, etc.) within a system or within a distributed processing system (e.g., "cloud," farm, etc.). A shared processing device can be provided that can be utilized. It should be understood that processor 804 is a non-transitory computing device (eg, an electronic machine with circuits and connections for communicating with other components and devices). The processor 804 handles machine instructions that are not native to the processor (e.g., the VAX operating system and the VAX machine instruction code set by Intel(R) to allow VAX-specific applications to run on virtual VAX processors). 9xx chipset code), etc., but as those skilled in the art will appreciate, such virtual processors are implemented by hardware, more specifically , the applications executed by the underlying electrical circuitry of a processor (eg, processor 804) and other hardware. Processor 804 can be run by a virtual processor, such as when an application (or Pod) is orchestrated by Kubernetes. Virtual processors allow an application to be given what appears to be a static and/or dedicated processor that executes the application's instructions, while the underlying non-virtual processors execute the instructions. and this non-virtual processor may be dynamic and/or split among several processors.

In addition to the components of processor 804, device 802 can utilize memory 806 and/or data storage 808 for storing accessible data such as instructions, values, and the like. A communication interface 810 facilitates communication of components, such as processor 804 , with components not accessible via bus 814 . Communication interface 810 may be embodied as a network port, network card, network cable, or other configured hardware device. Additionally or alternatively, a human input/output interface 812 receives and/or presents information (eg, instructions, data, values, etc.) from and/or to humans and/or electronic devices. connect to one or more interface components for Examples of input/output devices 830 that can be connected to the input/output interface include, but are not limited to, keyboards, mice, trackballs, printers, displays, sensors, switches, relays, speakers, microphones, still cameras and and/or include video cameras and the like. In another embodiment, the communication interface 810 may include a human input/output interface 812 and may be included in the human input/output interface 812 . Communication interface 810 can be configured to communicate directly with networked components or to utilize one or more networks, such as network 820 and/or network 824 .

Network 820 can be a wired network (eg, Ethernet), a wireless (eg, Wi-Fi, Bluetooth, cellular, etc.) network, or a combination thereof, allowing networked components 822 and devices 802 to communicate. can be In other embodiments, network 820 may be embodied in whole or in part as a telephone network (eg, public switched telephone network (PSTN), private branch exchange (PBX), cellular telephone network, etc.).

Additionally or alternatively, one or more other networks may be utilized. For example, network 824 can represent a second network, which can facilitate communication with components utilized by device 802 . For example, network 824 may be an internal network leading to a business entity or other organization, whereby components connect to network 820 comprising a public network (eg, the Internet) that may not be as trusted as the internal network. more trusted (or at least more trusted) than the networked component 822 that can.

Components connected to network 824 may include memory 826 , data storage 828 , input/output devices 830 , and/or other components that may be accessible to processor 804 . For example, memory 826 and/or data storage 828 may supplement memory 806 and/or data storage 808 or replace memory 806 and/or data storage 808 generally or for a particular task or purpose. be able to. For example, memory 826 and/or data storage 828 can be an external data repository (eg, server farm, array, “cloud,” etc.) that device 802 and/or other devices can access data on. can make it possible. Similarly, input/output devices 830 can be accessed by processor 804 via human input/output interface 812 and/or via communication interface 810 directly, via network 824, via network It can be accessed via 820 alone (not shown) or via networks 824 and 820 . Memory 806, data storage 808, memory 826, and data storage 828 each comprise non-transitory data storage comprising data storage devices.

It should be appreciated that computer readable data can be sent, received, stored, processed, and presented by a variety of components. It should also be understood that the illustrated components can control other components, whether they are illustrated herein or not. For example, an input/output device 830 may be a router, switch, port, or other communication component by which a particular output of processor 804 is associated with network 820 and/or network 824. Enabling (or disabling) an input/output device 830 may enable (or disable) communication between two or more nodes on network 820 and/or network 824 . Those skilled in the art will appreciate that other communication devices may be utilized in addition to or in place of those described herein without departing from the scope of the embodiments.

In the foregoing description, for purposes of illustration, the methods were described in a particular order. It should be appreciated that in alternate embodiments, the methods may be performed in a different order than the order described without departing from the scope of the embodiments. It is also understood that the methods described above can be implemented as algorithms executed by hardware components (eg, circuits) constructed for the purpose of executing one or more of the algorithms or portions thereof described herein. want to be In another embodiment, the hardware component may comprise a general purpose microprocessor (eg CPU, GPU), which is first converted to a dedicated microprocessor. In that case, the dedicated microprocessor has been loaded with encoded signals therein such that the microprocessor, which is now dedicated, holds machine-readable instructions to implement the algorithms described herein. and/or enable a microprocessor to read and execute a machine-readable instruction set derived from other instructions. The machine-readable instructions utilized to implement the algorithm, or portions thereof, utilize a finite instruction set known to microprocessors, rather than being unlimited. The machine-readable instructions can be encoded in the microprocessor as signals or values in the signal generating components, and in one or more embodiments, voltages in memory circuits, configuration of switching circuits, and/or selective use of certain logic gate circuits. Additionally or alternatively, the machine-readable instructions are accessible to the microprocessor to store magnetic fields, voltage values, charge values, reflective/non-reflective portions, and/or physical indicia in the medium or device. indicium).

In another embodiment, the microprocessor is one of a single microprocessor, a multicore processor, multiple microprocessors, distributed processing systems (e.g., arrays, blades, server farms, "clouds," multipurpose processor arrays, clusters, etc.). One or more may be further provided and/or co-located with a microprocessor to perform other processing operations. Any one or more microprocessors may be integrated into a single processing unit (eg, computer, server, blade, etc.) or a communication link (eg, bus, network, backplane, etc., or multiple thereof). It can also be located wholly or partly within separate components connected via the .

An example of a general-purpose microprocessor can be a central processing unit (CPU), which has data values encoded in instruction registers (or other circuitry that holds instructions) or contains memory locations. contains values that are used as instructions. The memory locations can further include memory locations external to the CPU. Such CPU external components include field programmable gate arrays (FPGA), dedicated memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), random access memory (RAM). ), bus-accessible storage, network-accessible storage, and the like.

These machine-executable instructions may be on a CD-ROM or other type of optical disk, floppy diskette, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other suitable medium for storing electronic instructions. can be stored on one or more machine-readable media, such as any type of machine-readable media. Alternatively, the method can be implemented by a combination of hardware and software.

In another embodiment, the microprocessor is a microprocessor on a client device and a microprocessor on a server, a collection of devices and their respective microprocessors, or a shared or remote processing service (e.g., a "cloud"). It can be a system or collection of processing hardware components, such as a microprocessor based processor. A system of microprocessors may include a task-based allocation of processing tasks and/or shared or distributed processing tasks. In yet another embodiment, the microprocessor may execute software that provides services that emulate one or more different microprocessors. As a result, a first microprocessor comprising a first set of hardware components can substantially provide the services of a second microprocessor, thereby providing hardware associated with the first microprocessor. ware can operate using an instruction set associated with the second microprocessor.

Although machine-executable instructions may be stored and executed locally on a particular machine (e.g., personal computer, mobile computing device, laptop, etc.), it should be understood that data and/or instructions storage and/or execution of at least a portion of the instructions may be via connectivity to remote data storage and/or processing devices or to a collection of devices commonly known as the "cloud"; provided, however, that it may include public, private, dedicated, shared, and/or other service bureaus, computing services, and/or "server farms."

Examples of microprocessors described herein include, but are not limited to, Qualcomm® Snapdragon® 800 and 801, Qualcomm® Snapdragon® with 4G LTE integration and 64-bit computing ( 610 and 615, 64-bit architecture Apple® A7 microprocessor, Apple® M7 motion coprocessor, Samsung® Exynos® series, Intel® Core ( trademark) family of microprocessors, Intel® Xeon® family of microprocessors, Intel® Atom™ family of microprocessors, Intel Itanium® family of microprocessors, Intel® Trademarked Core® i5-4670K and i7-4770K 22nm Haswell, Intel® Core® i5-3570K 22nm Ivy Bridge, AMD® FX™ family of microprocessors, AMD ( FX-4300, FX-6300, and FX-8350 32nm Vishera, AMD Kaveri Microprocessor, Texas Instruments JacintoC6000 Automotive Infotainment Microprocessor, Texas Instruments OMAP ™ Automotive Grade Mobile Microprocessors, ARM ® Cortex ™-M Microprocessors, ARM ® Cortex-A and ARM926EJ-S ™ Microprocessors, industry-equivalent ) other microprocessors, examples of which perform computational functions using any known or future-developed standards, instruction sets, libraries, and/or architectures be able to.

Any of the steps, functions and actions discussed herein can be executed continuously and automatically.

The exemplary systems and methods of the present invention have been described in terms of communication systems and components and methods for monitoring, enhancing, and embellishing communications and messages. However, in order to avoid unnecessarily obscuring the present invention, the above description omits some well-known structures and devices. This omission should not be construed as limiting the scope of the claimed invention. Specific details are set forth to enable an understanding of the invention. However, it should be understood that the present invention may be practiced in many different ways beyond the specific details set forth herein.

Further, although the exemplary embodiments presented herein show various components of the system collocated, some components of the system may be remotely located in distributed networks such as LANs and/or the Internet. It can be located within a partial or dedicated system. As such, components of the system or portions thereof (e.g., microprocessors, memory/storage, interfaces, etc.) can It will be appreciated that it may be integrated into one or more devices or collocated on a particular node of a distributed network such as an analog and/or digital telecommunications network, packet switched network, or circuit switched network. sea bream. In another embodiment, the components may be physically or logically distributed across multiple components (e.g., microprocessors each perform portions of shared and/or assigned tasks, a first microprocessor on one component and a second microprocessor on another component). From the foregoing discussion, and for computational efficiency reasons, it will be appreciated that the components of the system can be placed anywhere within the distributed network of components without affecting the operation of the system. For example, various components may be located within switches such as PBXs and media servers, within gateways, within one or more communication devices, at one or more user premises, or some combination thereof. be able to. Similarly, one or more functional portions of the system may be distributed between telecommunications devices and associated computing devices.

Moreover, the various links connecting the elements may be wired or wireless links, or any combination thereof, or any other known or It should be understood that it can be a later developed element. These wired or wireless links may also be secure links and may be capable of communicating encrypted information. The transmission medium used as a link can be any carrier suitable for electrical signals, including, for example, coaxial cable, copper wire, and fiber optics; It may also take the form of acoustic or light waves, such as those generated during communications and infrared data communications.

Also, while the flowchart has been discussed and illustrated with respect to a particular sequence of events, it should be understood that modifications, additions, and omissions can be made to this sequence without materially affecting the operation of the present invention.

Several variations and modifications of the invention can be used. It is possible to provide some features of the invention and not others.

In yet another embodiment, the system and method of the present invention can be implemented using special purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit elements, ASICs or other integrated circuits, digital signal microprocessors, discrete component circuits, and the like. may be implemented in combination with hard-wired electronic or hard-wired logic circuits, programmable logic devices or programmable gate arrays such as PLDs, PLA, FPGAs, PALs, dedicated computers, any equivalent means, or the like; can. In general, any device or means capable of performing the methods presented herein can be used to implement the various aspects of the present invention. Exemplary hardware that can be used with the present invention include computers, handheld devices, telephones (eg, cellular, internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. is included. Some of these devices include microprocessors (eg, single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. Further, alternative software implementations including, but not limited to, distributed or component/object distributed processing, parallel processing, or virtual machine processing are also performed by one or more processing components described herein. It can also be constructed to perform the method described above.

In yet another embodiment, the methods disclosed herein use an object or object-oriented software development environment that provides portable source code that can be used on a variety of computer or workstation platforms. can be easily implemented in combination with software. Alternatively, the system disclosed herein can be implemented partially or fully in hardware using standard logic circuits or VLSI design. Whether software or hardware is used to implement a system according to the invention depends on the speed and/or efficiency requirements of the system, the particular functionality and the particular software system or systems being utilized. It depends on the hardware system or microprocessor system or microcomputer system.

In yet another embodiment, the methods disclosed herein can be partially implemented in the form of software, which software is stored on a storage medium and which operates in cooperation with a controller and memory on a programmed general purpose computer. It can be run on a dedicated computer, on a microprocessor, or the like. In these cases, the system and method of the present invention can be implemented as a program, such as an applet, JAVA, or CGI script, embedded on a personal computer, as a resource residing on a server or computer workstation. It can be implemented as a routine embedded in a measurement system, system component, or the like. Systems may also be implemented by physically incorporating systems and/or methods into software and/or hardware systems.

Embodiments herein that include software are executed as executable code, stored for execution by one or more microprocessors, or stored for later execution. Executable code is selected to execute instructions comprising a particular embodiment. The instructions to be executed are a constrained set of instructions selected from a separate native instruction set understood by the microprocessor and, prior to execution, are injected into memory accessible to the microprocessor. In another embodiment, the human-readable "source code" software is selected from a platform's (e.g., computer, microprocessor, database, etc.) native instruction set prior to execution by one or more microprocessors. It is first translated into system software to include the instruction set.

Although the invention describes the components and functionality implemented in embodiments with reference to particular standards and protocols, the invention is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in the present invention. Moreover, the standards and protocols mentioned herein, and other similar standards and protocols not Periodically superseded. Such replacement standards and protocols having the same functionality are considered equivalents included in this invention.

The present invention, in its various embodiments, configurations, and aspects, complies with the components, methods, processes, systems, and components substantially illustrated and described herein, including various embodiments, subcombinations, and subsets thereof. / or includes a device. Those of skill in the art will understand how to make and use the present invention after understanding the present disclosure. The present invention, in various embodiments, configurations, and aspects, has been used in conventional devices or processes, e.g., to improve performance, obtain ease, and/or reduce implementation costs. To provide devices and processes in the absence of items not shown and/or described herein or in various embodiments, configurations or aspects herein, including possible items Including.

The foregoing discussion of the invention has been presented for purposes of illustration and description. The foregoing is not intended to limit the invention to the form or forms disclosed herein. For example, in the foregoing Detailed Description, various features of the invention are summarized in one or more embodiments, configurations, or aspects for the purpose of streamlining the disclosure. Features of embodiments, configurations, or aspects of the invention may be combined in alternative embodiments, configurations, or aspects other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment, configuration, or aspect. be. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the present invention.

Further, while the description of the present invention includes a description of one or more embodiments, configurations, or aspects, and some variations and modifications, other variations, combinations, and modifications may include: It is included within the scope of the present invention, for example, because it is within the skill and knowledge of one of ordinary skill in the art after understanding the present disclosure. Any subject matter patentable, whether or not alternative, interchangeable, and/or equivalent structures, functions, ranges, or steps to those recited in a claim are disclosed herein Alternate embodiments, configurations, or aspects are permitted, including such alternative, interchangeable, and/or equivalent structures, functions, ranges, or steps, without intending to make public the It is intended to obtain inclusive rights to the extent possible.

It should also be understood that the methods described above may be implemented by means of hardware components or embodied in a sequence of machine-executable instructions, which machine-executable instructions can be used to cause a machine, such as a general-purpose or special-purpose processor (GPU or CPU) or a logic circuit (FPGA) programmed with this instruction, to perform the method. These machine-executable instructions may be on a CD-ROM or other type of optical disk, floppy diskette, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other suitable medium for storing electronic instructions. can be stored on one or more machine-readable media, such as any type of machine-readable media. Alternatively, the method can be performed by a combination of hardware and software.

Specific details are given in the description to enable a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, circuits may be shown in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail in order to avoid obscuring the embodiments.

Also, note that embodiments have been described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although a flowchart may represent the actions as a sequential process, many of the actions can be performed in parallel or concurrently. Additionally, the order of operations can be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the figure. A process can correspond to a method, function, procedure, subroutine, subprogram, or the like. When a process corresponds to a function, its termination corresponds to that function's return value to the calling or main function.

Furthermore, embodiments can be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the required tasks can be stored in a machine-readable medium such as a storage medium. A processor can perform the necessary tasks. A code segment can represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, transferred, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, and the like.

Although exemplary embodiments of the present disclosure have been described in detail above, the concepts of the present invention can be variously embodied and used in other ways, and the scope of the claims appended hereto. is intended to be construed as including such variations except as limited by the prior art.

The foregoing description and related drawings teach the best mode of the invention. The appended claims define the scope of the invention. Note that some aspects of the best mode may not fall within the scope of the invention as defined by the claims. Those skilled in the art will appreciate that the features described above can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific embodiments described above, but only by the appended claims and their equivalents.

100 images, chips
102 Area of Interest, Tile Section
104 Area of Interest, Tile Section
106 Area of Interest, Tile Section
108 Area of Interest, Tiles
202 dotted line
204 dotted line
206 dotted line
208 dotted line
400 processes, methods
500 processes
600 datasets
700 process environment
702 image acquisition device
702a-n Image Acquisition Devices
704 Image Processing Node
704a-n Image processing nodes
706 Image Analyzer
708 Anomaly Detector
710 databases
800 system
802 devices
804 processor
806 memory
808 data storage
810 communication interface
812 Human Input/Output Interface
814 Bus
820 network
822 networked components
824 network
826 memory
828 data storage
830 input/output devices

Claims (20)

A method of visually inspecting a chip for defects, comprising:
processing an image of the chip to identify areas of interest;
dividing the identified area of interest into a first image and a second image;
Analyzing the first image and/or the second image using a trained model to determine if defects are detected in the first image and/or the second image a step;
and issuing an alert if said defect is detected in said first image and/or said second image. 2. The method of claim 1, further comprising processing and incorporating the first image and the second image into a training data set if the first image and the second image are defect-free. . if the first image or the second image is defect-free, processing the first image and the second image and incorporating them into a reference image;
receiving an input image, said input image substantially comprising said area of interest;
reconstructing the reference image using the input image and determining a reconstruction loss, the reconstruction loss indicating information lost during the reconstruction;
and determining whether the reconstruction loss exceeds a predefined threshold. 2. The method of claim 1, wherein said first image and said second image are radially symmetrical to each other. 2. The method of claim 1, wherein the trained model is generated using a dataset containing defect-free images. creating a template for a type associated with said chip, said template providing a predetermined input pattern of at least one known area of interest for said type associated with said chip; 2. The method of claim 1, wherein processing an image to identify the area of interest comprises performing template matching. 2. The method of claim 1, wherein said chip comprises a sensor chip. A system for performing a method of detecting defects in a chip, comprising:
at least one memory configured to store computer program code;
at least one processor configured to access said at least one memory and operate according to said computer program code, said computer program code comprising:
processing an image of the chip to identify areas of interest;
splitting the identified area of interest into a first image and a second image;
Analyzing the first image and/or the second image using a trained model to determine if defects are detected in the first image and/or the second image and
and issuing an alert if the defect is detected in the first image and/or the second image. The program code is
9. The system of claim 8, further comprising processing the first image and the second image if the first image and the second image are defect-free and incorporating them into a training data set. . The program code is
if the first image or the second image is defect-free, processing the first image and the second image and incorporating them into a reference image;
receiving an input image, said input image substantially including said area of interest;
reconstructing the reference image using the input image and determining a reconstruction loss, wherein the reconstruction loss indicates information lost during the reconstruction;
9. The system of claim 8, further comprising determining whether the reconstruction loss exceeds a predefined threshold. 9. The system of claim 8, wherein said first image and said second image are radially symmetrical to each other. 9. The system of claim 8, wherein the trained model is generated using a dataset containing defect-free images. The program code is
further comprising creating a template for the type associated with the chip, the template providing a predetermined input pattern of at least one known area of interest for the type associated with the chip; 9. The system of Claim 8, wherein processing to identify the area of interest comprises performing template matching. 9. The system of claim 8, wherein said chip comprises a sensor chip. A computer-implemented method of training an autoencoder to train a neural network to visually detect defects in a chip, the method comprising:
collecting a set of whole images;
generating a first training set using the set of full images;
training the neural network using the first training set in a first model;
generating a second training set, said second training set comprising at least one segment of a full image;
training the neural network using the second training set in a second model, wherein the second model analyzes first and second symmetric halves; A computer-implemented method. 16. The computer-implemented method of Claim 15, wherein the first and second symmetrical halves are radially symmetrical to each other. 16. The computer-implemented method of Claim 15, wherein the chip comprises a sensor chip. generating a third training set;
training the neural network using the third training set in a third model, wherein the third model is the smaller component in the first and second symmetric halves; 16. The computer-implemented method of claim 15, further comprising the step of parsing the . 19. The computer-implemented method of claim 18, wherein a reconstruction loss threshold for the third model is greater than a reconstruction loss threshold for the second model. 16. The computer-implemented method of Claim 15, wherein the neural network comprises at least one of an encoder neural network and a decoder neural network.