Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0004—Industrial image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
  • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/243—Classification techniques relating to the number of classes
  • G06F18/2431—Multiple classes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
  • G06N20/20—Ensemble learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/0464—Convolutional networks [CNN, ConvNet]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/09—Supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N5/00—Computing arrangements using knowledge-based models
  • G06N5/04—Inference or reasoning models
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
  • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
  • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
  • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
  • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
  • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20076—Probabilistic image processing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30108—Industrial image inspection
  • G06T2207/30148—Semiconductor; IC; Wafer

Definitions

• This application relates to semiconductor manufacturing processes, and more particularly, to systems and methods for classifying semiconductor wafer issues/problems using image analysis.
• Image classification in general has become highly developed, for example, by web- based efforts such as Google, Facebook, etc., which by their nature have access to literally millions of pictures that can be utilized as a training set for machine-based image classification schemes.
• the typical semiconductor company however, has a much smaller dataset of perhaps only a few hundred images to utilize for training sets. Thus, it would be desirable to adapt and develop techniques for machine-based analysis and classification of semiconductor wafers using these smaller datasets.
• Fig. 1 is a flow chart illustrating a process for classifying wafer issues using image processing techniques.
• Figs. 2A-2D illustrate four different types or classes of original wafer images, including normal, blemish, unspecified defect and stripes.
• Figs. 3A-3D are the four different wafer images of Figs. 2A-2D, respectively, each modified by a Gaussian blur function.
• Fig. 4 is an example 7x7 kernel for implementing the Gaussian blur function illustrated in Figs. 3A-3D.
• Figs. 5A-5D are the four different wafer images of Figs. 2A-2D, respectively, modified by an edge detection function.
• Figs. 6A-6C are example 3x3 kernels for implementing an edge detection function, with the edge detection function of Fig. 6C implementing the edge detection function illustrated in Figs. 5A-5D.
• Figs. 7A-7D are the four different wafer images of Figs. 2A-2D, respectively, modified by a sharpen function.
• Fig. 8 is an example 3x3 kernel for implementing the sharpen function illustrated in Figs. 7A-7D.
• Fig. 9 is a flow chart illustrating a hierarchical modeling process useful for feature selection and pairwise classification.
• Fig. 10 is a flow chart illustrating in more detail the pairwise classification step of Fig. 9.
• Fig. 11 is a flow chart illustrating in more detail the variable selection step of Fig. 10.
• Figs. 12A and 12B are graphical illustrations of the accuracy of example pairwise classifiers.
• Figs. 13 A and 13B are tables illustrating the accuracy of prediction results aggregated from the individual pairwise classifiers.
• This disclosure is directed to methods and systems for classifying images of semiconductor wafers based on the type of abnormalities.
• the core of a machine learning solution for a semiconductor processing application is developed through feature engineering and feature selection.
• Feature engineering is the process of generating features from raw data to better represent a problem for solution to predictive machine learning models.
• Feature selection also called variable selection
• Feature engineering is the process of selecting the features that contribute most significantly to understanding and predicting the particular problem for modeling and discarding those features that do not contribute significantly.
• variable selection is the manner in which feature engineering and feature selection are implemented for any specific application continues to evolve, and each application is of course dependent on the client-based context including the details of the semiconductor processing environment.
• Feature engineering is often considered to include the feature selection step.
• features can be identified and generated by first applying one or more convolution functions to an original image of a wafer image to try and better identify a specific wafer problem through image observation and analysis.
• the result of each convolution is a modified image that (hopefully) shows the specific wafer problem in better detail.
• different problems may be revealed by different convolution functions, or by a combination of convolution functions, and because each fabrication facility typically has operating and environmental differences, there may be different functions or combinations of functions that work as solution sets for the same type of problem (such as a blemish, defect, or stripe) at different locations.
• pooling functions are applied to condense the modified image into a single-dimensional vector result.
• the vector result is easier to deal with in many respects: simpler representation; less storage and processing requirements; numerical metric can be used in logical circuits; etc.
• the pooling step is thus used to define features that may be of interest by applying various pooling functions to the image.
• statistical functions may be applied at various points across the image, although other types of functions could also be employed.
• fuzzy integrals such as the Choquet Integral have been used to condense data sets based on fuzzy math, a fast Fourier Transform has been used for frequency analysis, simple linear aggregation applied, etc..
• each predictive classifier model performs its own variable selection steps in order to reduce and limit the number of variables as input to the model to significant variables for that predictive model.
• the result of each predictive classifier model thus indicates whether the wafer problem is more like the first problem class or the second problem class, rather than the approach of conventional classifiers, which attempt to assign a class to the vector rather than, as here, a series of pairwise comparisons.
• FIG. 1 is a flow chart illustrating a simple process 100 for classifying wafers based on image processing.
• step 102 sample images are obtained.
• step 104 One or more convolution functions are then applied in step 104 to generate modified images in step 106.
• step 106 a number of different convolution functions should be tried for a new or updated client application, both individually and in combination, to look for the most effective image processing techniques that will work for the images from that application.
• Pooling functions are then applied in step 108 which act to reduce the two- dimensional images into one-dimensional representations having a plurality of defined features in step 110.
• Each one-dimensional representation of the sample images i.e., the vectors are processed through a series of pairwise classifiers in step 112 including iteratively running each classifier as part of a variable selection process 114 that reduces the number of features down to two to four features for each classifier model.
• the result in step 116 is a probability or likelihood: is the vector more like the first of the unique pair or the second of the unique pair?
• the probabilities from all the pairwise classifiers are collected at step 118 to make a final prediction for each pairwise comparison.
• Figs. 2A-2D are four different original images of wafers (on substrates) that were taken using conventional optical imaging equipment in a semiconductor process. Each image may be stored as a 200kB JPG file or similar.
• Fig. 2A is an example of an image 220 of substrate 222 having a circular wafer 224 formed on the substrate that appears “normal,” i.e., no apparent problems from the image.
• Fig. 2B is an example of an image 240 of substrate 242 having a circular wafer 244 formed on the substrate that includes a prominent dark “blemish” appearing as a large dark circle 246 in the top right comer of the image as well as several other blemishes on the substrate adjacent the wafer.
• Fig. 2C is an example of an image 260 of substrate 262 having a circular wafer 264 with an unspecified “defect” appearing as several small dark circles on the edge of the wafer, such as defect 246 on the lower right corner of this image.
• Fig. 2D is an image 280 of substrate 282 having a circular wafer 284 on the substrate with a series of vertical edges or “stripes” 286 appearing across the entirety of the substrate and wafer image.
• Each of these types of problems has a different root cause and requires different analysis and attention. There may of course be other problems or defects that may be the subject of the same kind of analysis described here.
• one or more convolution functions are applied to the samples.
• the techniques can be applied to black and white or grayscale images as well as color images.
• a different convolution function or set of functions may be necessary or important to the particular problem of interest.
• some trial and error is likely required to determine the right combination of convolution and aggregation (pooling) functions work to capture a signal for a particular type of problem at a particular fab location.
• Gaussian blur is an image softening technique that blurs an image using a low-pass filter to smooth out uneven pixel values by removing extreme outliers.
• This technique appears useful for detecting circular blemishes such as shown in Fig. 2B.
• FIGS. 3A- 3D show the result of processing each of the images 220, 240, 260 and 280 after a Gaussian blur filter has been applied, resulting in images 320, 340, 360 and 380, respectively.
• the convolution matrix or kernel selected to apply to the images as the Gaussian function should be related to the size of the defect or blemish, e.g., the kernel size should relate to the radius of the defect.
• a small kernel such as 2x2 or 3x3, will be adequate. If the defect is large, the kernel should be larger; e.g., if the defect a size 10, then apply a 10x10 kernel.
• the size of the defect is typically measured by an x,y pixel count (e.g., 2x2 or 3x3) rather than actual dimensional length and width of the defect. Thus, if the same defect has an image taken with different zoom ratio, the relevant kernel will be sized accordingly.
• convolution matrices that have been studied and shown to be effective for image analytics, and thus, it is possible to select a convolution function that should be effective to capture the defect of interest.
• a 7x7 kernel as shown in Fig. 4 was applied to the original images.
• the expectation is that lighter images get filtered out while darker images remain, thus providing a basis for distinguishing different images.
• the blemish 246 in modified image 340 (transformed from image 240 in Fig. 2B) still appears as a prominent dark circle in Fig. 3B.
• the most prominent of the small defects 266 still appears in Fig. 3C.
• the small items like defects 266 could be filtered out when looking for larger items like blemish 246.
• the convolution function will be applied multiple times as this step may cause some features to display better. By having multiple convolutions of the Gaussian blur function, for example, the darker regions remain dark while other regions get lighter thus enabling a detectable signal difference that the predictive machine learning model can use as input for predicting/detecting this type of blemish.
• FIGs. 5A-5D the result of processing each of the original images 220, 240, 260, 280 is shown after an edge detection function has been applied to generate images 520, 540, 560 and 580, respectively.
• Edge detection algorithms would seem to make sense for detecting the stripe pattern 286 shown in Fig. 2D. However, when the edges are visible but not really distinct, then it may still be difficult to detect the stripe pattern, such as in the example of Fig. 5D.
• Figs. 6A-6C various 3x3 kernels (shown in Figs. 6A-6C) were applied to original images 220, 240, 260, 280.
• the kernel of Fig. 6C seemed to work best in this particular example, resulting in the images of Figs. 5A-5D; but other variations of kernel values may produce good results depending on the number and quality of the images that are sampled and the problem to be detected. Also, different results can be obtained by selecting a kernel to be applied in a single direction (x or y) or in multiple directions (x and y).
• a sharpen function has been applied to the original images, and it turns out that the stripes 286 are quite a bit more distinct after the sharpen function, as shown in Fig, 7D.
• Hu Hu
• Other metric techniques include Hu’s set of moments, which can be useful to show a strong moment body if the defect is symmetrical in the image. See, e.g., Huang, Zhihu, and Jinsong Leng, Analysis of Hu's Moment Invariants on Image Scaling and Rotation, 20102nd International Conference on Computer Engineering and Technology, Vol. 7, pp. V7-476 (2010).
• Another technique is orthonormal rectangular polynomials decomposition. See, e.g., Ye, et al., Comparative Assessment of Orthogonal Polynomials for Wavefront Reconstruction over the Square Aperture , JOSA A 31, No. 10, pp. 2304-11 (2014).
• a lower dimensional representation of the image could be used as a metric input to the model, such as Discrete Cosine Transformation, Singular Value Decomposition, etc., including the metric representing compression errors from the image processing.
• Metrics can be provided as inputs directly to machine learning models but still require feature selection. Convolution results must be condensed into a one-dimensional representation, such as a vector, in order to be input to machine learning models and undergo feature selection. [0039] Any of the convolutional techniques and metric techniques can be used either individually or in combination as necessary to be effective for a particular customer application in order to make a decision for distinguishing between different classes or categories of images.
• a vector for a modified image is a collection of standard deviation values in the A-dircction while another vector may for the same modified image is a collection of standard deviation values in the y-dircction.
• Each of the individual statistical values of the vector is thus an “engineered” feature or variable that may be selected if determined to be significant, as discussed below.
• the present technique uses a classifier for each pair of image types to determine whether the vector is more like the first image type or the second image type. The result is taken as a single number that represents the likelihood that the subject image is one type or the other. Then all the probability results from all the pairwise classifiers are fed into a final model to make a final predictive determination.
• FIG. 9 illustrates one embodiment of a process 900 for hierarchical modeling consisting simply of step 902 for pairwise classification of wafer images and step 904 for predicting the most probable class for each pairwise image comparison.
• Fig. 10 illustrates the pairwise classification step 902 in more detail.
• step 1002 a separate predictive model is built for each pair of classes.
• six probability scores are computed from the pairwise classifier.
• step 1004 for each pairwise classifier model, image samples are selected from limited subsets of images, e.g., only two subsets are selected for training sets. Note that a defect image should not be used for training a classifier to distinguish normal vs. blemish.
• Variable (feature) selection is performed in step 1006, and a final prediction for each pairwise classifier in performed in step 1008.
• variable selection step 1006 is illustrated in more detail in Fig. 11.
• step 1102 the features generated in the convolution and pooling steps (or the metrics) for each image are ranked ordered, for example, using a Random Forest algorithm to determine variable importance.
• This variable importance is obtained by building each model using A- fold cross-validation (e.g., 20-fold) and taking the sum of variable importance.
• Forward variable selection is performed in step 1104. Based on the rank ordering from step 1102, a 20-fold cross validation can be used to run each model adding one feature at time, keeping the feature if it improves the accuracy of the model. Other metrics like an FI score or average AUC (area under curve) might also be used.
• Backward variable selection is performed in step 1106 for pruning back the generated features.
• Each model is run again with dropping one feature at a time that was added from last to first; and returning the feature if it reduces the accuracy of the model by removing it.
• this variable selection step 1006 reduces the number of features down to two to four.
• Other known variable selection techniques could also be employed or tried, including but not limited to a “greedy” optimization algorithm that computes variable importance and drops all variables lower than a threshold value, or a dimensional reduction method designed to remove redundant variables.
• the accuracy of the pairwise classifier has been demonstrated as illustrated in Figs. 12A-12B.
• the graph 1210 in Fig. 12A includes six ROC curves that are the result of the six pairwise classifiers of four different image types, A-D.
• Plot 1211 is the result of the A-B classifier;
• plot 1212 is the result of the A-C classifier;
• plot 1213 is the result of the A-D classifier;
• plot 1214 is the result of the B-C classifier;
• plot 1215 is the result of the B-D classifier;
• plot 1216 is the result of the C-D classifier.
• the B-D pair is the most difficult to classify in this example having the smallest area under the curve and a large number of possible false positives while the plot 1211 of the A-B pair has a very large area under the curve but also spikes quickly upward indicating very few false positive and therefore is an effective and reliable classifier.
• the graph 1220 of Fig. 12B includes six ROC curves that are the result of the six pairwise classifiers of four different image types, blemish, defect, normal and stripe.
• Plot 1221 is the result of the blemish-defect classifier
• plot 1222 is the result of the blemish-normal classifier
• plot 1223 is the result of the blemish- stripes classifier
• plot 1224 is the result of the defect-normal classifier
• plot 1225 is the result of the defect- stripes classifier
• plot 1226 is the result of the normal-stripes classifier.
• the final prediction for each pairwise modeling step is determined (Fig. 10, step 1008) by running each model using only the significant contributing features as input variables to the model using 20-fold cross-validation and Random Forest techniques to reduce overfitting.
• a prediction is made for all wafers for all models.
• the model for predicting a defect image uses the “normal vs. blemish” classifier to help exclude those as likely possibilities when looking for a defect.
• a secondary model is built (Fig. 9, step 904) to predict the most probable class for each model based on the results from the pairwise classifiers.
• the predicted probability for each pairwise model is the input to this secondary model, and should consist of N*(N-l)/2 different probability values.
• the final classifier can be any classification algorithm including Logistic Regression, Support Vector Machine (SVM), Random Forest, Gradient Boosting Machine, XGBoost, Artificial Neural Network, etc.
• SVM Support Vector Machine
• Random Forest Random Forest
• Gradient Boosting Machine Gradient Boosting Machine
• XGBoost Artificial Neural Network
• parameter tuning can be achieved by using A- Fold stratified cross-validation.
• Figs. 13A and 13B show the results of the final image classification corresponding to the examples shown for the pairwise classification in Figs. 12A and 12B. For both examples of the techniques described herein, the accuracy rate of the predictive model was approximately 95%. These results also show that pairwise classification is not perfect. However, many of the images may contain multiple defects as well thereby increasing the complexity of classification. More confidence in the prediction could be provided by including a confidence score.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Software Systems (AREA)
• Data Mining & Analysis (AREA)
• Computing Systems (AREA)
• General Engineering & Computer Science (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Mathematical Physics (AREA)
• Medical Informatics (AREA)
• Multimedia (AREA)
• Biomedical Technology (AREA)
• Molecular Biology (AREA)
• Computational Linguistics (AREA)
• Databases & Information Systems (AREA)
• Evolutionary Biology (AREA)
• Bioinformatics & Computational Biology (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Biophysics (AREA)
• Quality & Reliability (AREA)
• Biodiversity & Conservation Biology (AREA)
• Probability & Statistics with Applications (AREA)
• Image Analysis (AREA)
• Testing Or Measuring Of Semiconductors Or The Like (AREA)
• Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=78222360

Family Applications (1)

Country Status (6)

Families Citing this family (7)

Citations (4)

Family Cites Families (2)

• 2021
  • 2021-04-22 TW TW110114543A patent/TW202147250A/zh unknown
  • 2021-04-22 WO PCT/US2021/028563 patent/WO2021216822A1/en not_active Ceased
  • 2021-04-22 JP JP2022564488A patent/JP2023523426A/ja not_active Ceased
  • 2021-04-22 CN CN202180029759.0A patent/CN115485740A/zh active Pending
  • 2021-04-22 US US17/237,516 patent/US11972552B2/en active Active
  • 2021-04-22 KR KR1020227038349A patent/KR20230006822A/ko active Pending

Patent Citations (4)

Also Published As

Similar Documents

Legal Events