CN110992315A - Chip surface defect classification device and method based on generative countermeasure network - Google Patents
Chip surface defect classification device and method based on generative countermeasure network Download PDFInfo
- Publication number
- CN110992315A CN110992315A CN201911126986.6A CN201911126986A CN110992315A CN 110992315 A CN110992315 A CN 110992315A CN 201911126986 A CN201911126986 A CN 201911126986A CN 110992315 A CN110992315 A CN 110992315A
- Authority
- CN
- China
- Prior art keywords
- training
- image
- image set
- model
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
- G01N21/956—Inspecting patterns on the surface of objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8854—Grading and classifying of flaws
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
- G01N21/956—Inspecting patterns on the surface of objects
- G01N2021/95638—Inspecting patterns on the surface of objects for PCB's
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30148—Semiconductor; IC; Wafer
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Chemical & Material Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Pathology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Immunology (AREA)
- Biochemistry (AREA)
- Analytical Chemistry (AREA)
- Bioinformatics & Computational Biology (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a chip surface defect classification method based on a generative countermeasure network, which comprises the following steps: s01: inputting the random vector into an image processing module to obtain a synthetic image, and combining a shot image and the synthetic image into an image set, wherein the image set comprises a training image set; s02: inputting the training image set into a training unit for model training to obtain an identification model; s03: and inputting the image to be recognized into the recognition model, and obtaining the corresponding defect type through recognition of the recognition model. The invention provides a chip surface defect classification device and method based on a generation type countermeasure network.
Description
Technical Field
The invention relates to the field of machine vision, in particular to a chip surface defect classification device and method based on a generative countermeasure network.
Background
The integrated circuit chip is widely applied to various fields and is the key of national economic development and information safety. But defects generated on the surface of the chip during the manufacturing process of the package directly affect the service life and reliability. The traditional manual detection and classification method has the defects of dependence on subjective experience, time and labor consumption, high false detection rate and the like, and cannot meet the requirements of high precision and high speed on a production line.
The traditional chip defect classification technology based on machine vision has a mature framework system and sequentially comprises the following processes: the method comprises the steps of image acquisition, defect image segmentation, defect region extraction, defect feature extraction and dimension reduction, and defect identification and classification. However, the flow depends on the defect image preprocessing and feature extraction stages, and the selection and use of the image processing algorithm are also important. In view of this, the deep convolutional neural network has proven the effectiveness of feature extraction and the high accuracy of computer vision tasks such as classification processing, recognition and detection in various application scenes related to images.
However, in the process of chip defect identification of the convolutional neural network, a convolutional neural network model needs to be trained first to obtain a model capable of performing defect identification; during the training of the model, a large number of defect images and corresponding defect types are required. In actual production, the characteristic of large demand of deep convolutional neural network training data is difficult to meet due to the fact that the number of defect pictures generated on a line is limited; if the number of defect images used for training is insufficient, the accuracy of the model is affected. The traditional image enhancement method based on shot image processing and change only generates a new image based on limited invariance, has weak generalization capability and cannot effectively improve the classification performance of an identification model.
Disclosure of Invention
The invention aims to provide a chip surface defect classification device and method based on a generative countermeasure network.
In order to achieve the purpose, the invention adopts the following technical scheme: a chip surface defect classification method based on a generative countermeasure network comprises the following steps:
s01: inputting the random vector into an image processing module to obtain a synthetic image, and combining a shot image and the synthetic image into an image set, wherein the image set comprises a training image set; the random vector is a vector meeting multivariate normal distribution, the image processing module comprises a generating type confrontation network, and the training image set comprises a plurality of training images and corresponding defect types thereof;
s02: inputting the training image set into a training unit for model training to obtain an identification model;
s03: and inputting the image to be recognized into the recognition model, and obtaining the corresponding defect type through recognition of the recognition model.
Further, the image processing module includes a generator into which a random vector is input, and a determiner into which a synthesized image generated by the generator is input, the determiner removing a synthesized image that cannot be a training image; the remaining composite images and the captured images are formed into a training image set.
Further, the generator and the judger are both convolutional neural network based units.
Further, the composite image is input to a determiner together with the captured image, and if the output of the determiner is equal to or less than a set threshold, the composite image is removed.
Further, in step S01, the captured image and the synthesized image are merged into an image set, and the image set is divided into a training image set and a test image set, where the training image set and the test image set have no intersection, and the test image set includes a plurality of test images and their corresponding defect types.
Further, the step S02 of obtaining the identification model specifically includes the following steps:
s021: inputting the training image set into a training unit for model training to obtain a recognition model in training;
s022: inputting the test image set into a recognition model in training for model testing to obtain the classification accuracy of the recognition model in the training;
s023: repeating the steps S021-S022M times, and repeating the steps S021-S022 for the Mth time to obtain an identification model; wherein, the classification accuracy obtained by the Mth repeated step S021-S022 is more than or equal to the accuracy threshold, and M is an integer more than 0.
Further, the step S021-S022N times is repeated in the step S023, and if the classification accuracy obtained in the nth time is greater than or equal to the accuracy threshold and the classification accuracy is not improved any more in the N iterations, the step S021-S022 is repeated the nth time to obtain the identification model; wherein N is an integer less than or equal to M.
Further, in the step S021, all training images in the training image set are sequentially input into the training unit for model training; in the step S022, all the test images in the test image set are sequentially input to a recognition module in training for model testing.
Further, before the training image set and the test image set are transmitted to the training unit in step S02, the training image set and the test image set are input to the image preprocessing unit for noise determination, and if the training image or the test image is a noise image, the training image or the test image is deleted.
A chip surface defect classification device based on a generative countermeasure network comprises an image processing module and a training module, wherein the training module comprises a training unit;
inputting the random vector into the image processing module to obtain a synthetic image; the synthetic image and the shot image form a training image set, the training image set is input into the training unit for model training to obtain a recognition model, the image to be recognized is input into the recognition model, and the corresponding defect type is obtained through recognition of the recognition model; the random vector is a vector meeting multivariate normal distribution, the training image set comprises a plurality of training images and corresponding defect types thereof, and the image processing module comprises a generative confrontation network.
The invention has the beneficial effects that: the image processing module generates the synthetic image by using the random vector, and because the image processing module comprises a generating type countermeasure network, the formed synthetic image is adopted to train or test the recognition model in the training process, so that the training accuracy of the recognition model can be improved, and the generalization capability of the recognition model can be enhanced; the invention can simplify the chip surface defect classification process and greatly improve the classification accuracy; the deep convolutional neural network model is adopted, so that the model has strong generalization capability and is beneficial to the extension of classification functions; the device can be deployed at a server side, and the model can process input images in batch, so that the online image classification efficiency is improved.
Drawings
FIG. 1 is a flow chart of a method for classifying chip surface defects based on a generative countermeasure network according to the present invention;
FIG. 2 is a diagram of an image processing module according to the present invention.
Fig. 3 is a schematic diagram of the input and output of the generator based on the deep convolutional neural network in embodiment 1.
FIG. 4 is a schematic diagram of the input and output of the deep convolutional neural network-based determiner in embodiment 1.
FIG. 5 is a schematic diagram of deep convolutional neural network model training in example 1.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for classifying chip surface defects based on a generative countermeasure network provided by the present invention includes the following steps:
s01: inputting the random vector into an image processing module to obtain a synthetic image, and forming a training image set by the shot image and the synthetic image; the random vector is a vector satisfying a multivariate normal distribution, and the image processing module includes a Generative Adaptive Network (GAN).
As shown in fig. 2, the image processing module includes a generator and a judger, the random vector is input into the generator to obtain a composite image, the composite image and the photographed image are input into the judger, and the judger compares the composite image with the generated image and removes the composite image with a large difference from the photographed image; the remaining composite images and the captured images are formed into a training image set. The generator and the decider may be elements based on a convolutional neural network.
The specific method for judging the composite image by the judger may be: inputting one of the synthesized images and one of the shot images into a judger, wherein the judger outputs a comparison value of the synthesized image and the shot image, and if the comparison value output by the judger is less than or equal to a comparison threshold value, the synthesized image is removed; if the comparison value output by the judger is larger than the comparison threshold, the composite image is retained. And (3) screening all the synthetic images, merging the screened synthetic images and the shot images into an image set, and dividing the image set into a training image set and a test image set. The image processing module of the present invention includes a generative confrontation network, and the synthesizing and determining process of the generative confrontation network on the random vector belongs to the content of the prior art, and will not be described in detail here. The specific value of the comparison threshold can be specifically set according to factors such as the accuracy requirement of the recognition model.
In order to ensure the accuracy of a recognition model obtained by subsequent training, an image set is divided into a training image set and a test image set; and the training image set and the test image set have no intersection, the training image set comprises a plurality of training images and corresponding defect types, and the test image set comprises a plurality of test images and corresponding defect types. The specific method for dividing the image set into the training image set and the test image set in the invention can be random classification, for example, the training image number in the training image set and the test image number in the test image set are randomly distributed according to the proportion of 5:5, or the training image number in the training image set and the test image number in the test image set are randomly distributed according to the proportion of 6:4, or the training image number in the training image set and the test image number in the test image set are randomly distributed according to the proportion of 8:2, and the like. Preferably, the number of training images in the training image set and the number of test images in the test image set are randomly distributed according to a ratio of 8:2, that is, the number of training images accounts for 80% of the total number of images in the image set, and the number of test images accounts for 20% of the total number of images in the image set. Because the training images in the training image set are used for model training, the test images in the test image set are used for model testing, if the same image is used for model training and testing at the same time, the accuracy of model identification testing can be improved with high probability, and the judgment on the accuracy of the identification model is not facilitated, therefore, when the test image set and the training image set are formed, preferably, no intersection between the training image set and the test image set is ensured.
S02: and inputting the training image set into a training unit for model training to obtain a recognition model. The training module comprises an image preprocessing unit and a training unit, wherein the image preprocessing unit is used for carrying out image preprocessing on a test image set and a training image set training image, specifically noise filtering and the like. Taking the filtered noise image as an example:
before the training image set and the test image set are transmitted to the training unit, the training image set and the test image set are input to the image preprocessing unit for noise judgment, and if the training image or the test image shooting image is a noise image, the training image or the test image is deleted. Specifically, the method for judging the noise image by the image preprocessing unit is as follows:
for an mxn training image or test image I, the image gradient G (x, y) is calculated:
dx(i,j)＝I(i+1,j)-I(i,j)；
dy(i,j)＝I(i,j+1)-I(i,j)；
G(x,y)＝dx(i,j)+dy(i,j)；
and from this the variance of the gradient G (x, y)
Wherein, the training image or the test image I is an image of m rows and n columns, and I (I, j) represents the pixel value of the ith row and the jth column; m and n are integers greater than 1, i is an integer less than or equal to m, and j is an integer less than or equal to n. When the variance V of the gradient G (x, y) is smaller than a given threshold t, a noise image is determined and deleted from the training image set or the test image set.
Preferably, in order to ensure the accuracy of the recognition model, after each training, the recognition model is tested, and if the test result meets certain accuracy, the training is stopped to obtain the recognition model; if the accuracy of the test result is not enough, continuing training, wherein the specific training and test process is as follows:
s021: inputting the training image set into a training unit for model training to obtain a recognition model in training; specifically, all training images in the training image set are sequentially input into the training unit for model training.
The recognition model can be a deep convolutional neural network model, parameters of the recognition model are unknown before training, the parameters of the recognition model can be determined only through training of a training image set, and once the parameters are determined, the model can be determined.
Wherein, the recognition model is based on a deep convolution neural network model. The deep convolutional neural network model comprises an input layer, Y convolutional layers, a full-link layer and an output layer, a training image is input through the input layer, passes through the Y convolutional layers and the full-link layer and is output through the output layer, and the size of the output layer is the same as the type of concentrated defects of the training image. The training process of the deep convolutional neural network model is described in detail in example 1 below.
S022: inputting the test image set into a recognition model in training for model testing to obtain the classification accuracy of the recognition model in the training; specifically, all the test images in the test image set are sequentially input into a recognition module in training for model testing.
The defect type of the test image is known, in the step, only the test image is input, the defect type of the test image is identified by using the identification model in the training, the identified defect type is compared with the known defect type, and the classification accuracy in each identification process is calculated.
S023: repeating S021-S022M times, and repeating the steps S021-S022 for the Mth time to obtain an identification model; wherein, the classification accuracy obtained by the Mth repeated step S021-S022 is more than or equal to the accuracy threshold, and M is an integer more than 0. Under the condition that external factors such as data labeling errors do not exist, the iteration times are generally set to be 100 times, and after 100 iterations, the classification accuracy is stably improved to be more than 85%. In practical application, in order to finish iteration in advance, save time and prevent overfitting of the recognition model, the iteration can be finished in advance by adopting the following modes: repeating the steps S021-S022N times, if the classification accuracy obtained in the Nth time is larger than or equal to the accuracy threshold and the classification accuracy is not improved in the N iterations, repeating the steps S021-S022 in the Nth time to obtain an identification model; wherein N is an integer less than or equal to M. Therefore, the iteration number can be controlled between fifty and sixty times, and the generalization of the recognition model caused by overfitting of the training image is prevented from being deteriorated. The specific calculation mode of the classification accuracy is as follows: assuming that the test image set comprises 50 test images, inputting the 50 test images into the identification model in sequence during each test, and if the defect types of 48 test images in the final identification result are correctly identified, the classification accuracy of the test is 96%.
S03: and inputting the image to be recognized into the recognition model, and obtaining the corresponding defect type through recognition of the recognition model. After multiple training and testing in step S02, the accuracy of the recognition model can be ensured, and the recognition model can automatically recognize the defect type by directly inputting the image to be recognized in this step. Wherein, each parameter of the recognition model is obtained in the training unit, and the determined recognition model can be obtained.
The invention provides a chip surface defect classification device based on a generative countermeasure network, which comprises an image processing module and a training module, wherein the image processing module is used for processing a plurality of images; the image processing module comprises a generator and a judger, and the training module comprises an image preprocessing unit and a training unit;
inputting the random vector into a generator to form a synthetic image, and inputting the synthetic image and the shot image into a judger together for judging whether the synthetic image meets the standard or not and removing the synthetic image which cannot be used as a training image; combining the rest synthetic images and the shot images into an image set, dividing the image set into a training image set and a test image set, inputting the training image set into a training unit for model training to obtain a recognition model in training, inputting the test image set into the recognition model in training for model testing to obtain the classification accuracy of the recognition model in training, obtaining the recognition model meeting the set classification accuracy through repeated iterative training and testing, inputting the images to be recognized into the recognition model, and obtaining the corresponding defect types through recognition of the recognition model; the training image set comprises a plurality of training images and corresponding defect types;
the random vector is a vector meeting multivariate normal distribution, the training image set comprises a plurality of training images and corresponding defect types thereof, the testing image set comprises a plurality of testing images and corresponding defect types thereof, and the image processing module comprises a generating type countermeasure network.
Example 1
The image processing module comprises a generation type confrontation network, when the generator and the judger are both units based on a convolutional neural network, the input and output schematic diagram of the generator is shown as the attached figure 3, the input and output schematic diagram of the judger is shown as the attached figure 4, and the steps of the generator and the judger generating the synthetic image in the image processing module are as follows:
t01: inputting the random vector into a generator input layer, wherein the size of the generator input layer is 100 multiplied by 1; in this embodiment, the size of the input layer of the generator is set according to the defect type of the synthesized image, and may be other values;
t02: the synthesized image output from the generator input layer is input to the generator first convolution layer GC1, the size of the generator first convolution layer GC1 is 4 × 4 × 1024, and ReLU is used as an activation function;
t03: the composite image output by the first convolution layer GC1 of the generator is input to the up-sampling layer GU1 of the generator; a first convolution layer GC1 of the generator is connected with an up-sampling layer GU1 of the generator, and the size of a sliding matrix is 2 multiplied by 2; in this embodiment, the size of the sliding matrix may also be 3 × 3 or other values;
t04: the synthesized image output by the generator upsampling layer GU1 is input to the generator second convolution layer GC2, the size of the generator second convolution layer GC2 is 8 × 8 × 512, and ReLU is used as an activation function;
t05: the synthesized image output by the generator second convolution layer GC2 is input to the generator upsampling layer GU 2; the second convolution layer GC2 of the generator is connected with the upper sampling layer GU2 of the generator, and the size of the sliding matrix is 2 multiplied by 2;
t06: the synthesized image output by the generator upsampling layer GU2 is input to the generator third convolution layer GC3, the size of the generator third convolution layer GC3 is 16 × 16 × 256, and ReLU is used as an activation function;
t07: the synthesized image output by the third convolutional layer GC3 of the generator is input to the generator upsampling layer GU 3; the third convolution layer GC3 of the generator is connected with an up-sampling layer GU3 of the generator, and the size of a sliding matrix is 2 multiplied by 2;
t08: the synthesized image output by the generator upsampling layer GU3 is input to the generator fourth convolution layer GC4, the size of the generator fourth convolution layer GC4 is 32 × 32 × 128, and ReLU is used as an activation function;
t09: the synthesized image output from the fourth convolution layer GC4 of the generator is input to the generator output layer, the size of which is 224 × 224 × 3, and tanh is used as an activation function. At this time, the generator outputs a 3-channel synthesized image having a size of 224 × 224 in the number of layers, and then the synthesized image enters the determiner. In this embodiment, the generator deep convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, and the specific number of convolutional layers can be freely set.
T010: the synthetic image and the shot image output by the generator are simultaneously input to an input layer of a judger; the determiner input layer size is 224 × 224 × 3; in the present invention, the judger input layer can accept inputs from two different sources, namely, the captured image and the composite image.
T011: the composite image and the photographed image output from the determiner input layer are input to the determiner first convolution layer DC1, the determiner first convolution layer DC1 has a size of 112 × 112 × 32, and the ReLU is used as an activation function;
t012: the composite image and the captured image output from the determiner first convolution layer DC1 are input to the determiner down-sampling layer DP 1; the first convolution layer DC1 of the judger is followed by the downsampling layer DP1 of the judger, the size of the sliding matrix is 2 multiplied by 2, and the maximum pooling is used as a downsampling function;
t013: the composite image and the photographed image output by the sample layer DP1 are input to the determiner second convolutional layer DC2, the size of the determiner second convolutional layer DC2 is 56 × 56 × 64, and ReLU is used as an activation function;
t014: the synthesized image and the captured image output from the determiner second convolutional layer DC2 are input to the determiner downsampling layer DP 2; the determiner second convolutional layer DC2 is followed by the determiner downsampled layer DP2, with a sliding matrix size of 2 × 2, using maximum pooling as the downsampling function;
t015: the composite image and the photographed image output by the sample layer DP2 are input to the determiner third convolutional layer DC3, the size of the determiner third convolutional layer DC3 is 28 × 28 × 128, and ReLU is used as an activation function;
t016: the synthesized image and the captured image output from the determiner third convolutional layer DC3 are input to the determiner down-sampling layer DP 3; the determiner, the third convolutional layer DC3, is followed by the determiner, the downsampling layer DP3, the sliding matrix size is 2 x 2, and the maximum pooling is used as a downsampling function; in this embodiment, the generator deep convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, and in this embodiment, the determiner deep convolutional neural network includes the first convolutional layer, the second convolutional layer, and the third convolutional layer, and the specific number of convolutional layers can be freely set.
T017: the composite image and the captured image output from the sample layer DP3 are input to the determiner first-layer fully-connected layer by the determiner; the decider first layer fully connected layer size is 1 × 1 × 128, using ReLU as the activation function.
T018: inputting a synthetic image and a shot image output by a first full-connection layer of a judger into an output layer of the judger, outputting a comparison value of the two images by the output layer of the judger, and removing the synthetic image if the comparison value output by the judger is less than or equal to a comparison threshold; and if the comparison value output by the judger is greater than the comparison threshold, putting the composite image into a test image set or a training image set. The comparison value output by the judger is the comparison value of the synthetic image and the shot image and is used for representing the similarity of the two images. The comparison threshold is a value set in advance as a criterion for the determiner to determine whether the composite image is usable.
In the embodiment, the deep convolutional neural network uses the cross entropy as a loss function, and Adam is used as an optimizer for training.
As shown in fig. 5, the recognition model in this embodiment may be a deep convolutional neural network model, and the specific training steps are as follows:
step 1: setting the training image size in the training image set to 3 channels 224 × 224; it can also be set to 1 channel image and the channel size can be any value between 0-256;
step 2: inputting a training image into a recognition model input layer, wherein the size of the input layer is 224 multiplied by 3;
and step 3: inputting the training image output by the recognition model input layer into the recognition model first layer convolution layer; identifying the size of the first layer convolutional layer C1 of the model as 224 multiplied by 64, and using the ReLU as an activation function;
and 4, step 4: inputting the training image output by the first layer convolution layer of the recognition model into a down-sampling layer P1 of the recognition model; a first convolution layer C1 of the identification model is connected with a downsampling layer P1 of the identification model, the size of a sliding matrix is 2 multiplied by 2, and the maximum pooling is used as a downsampling function; in this embodiment, the size of the sliding matrix may also be 3 × 3 or other values;
and 5: inputting the training image output by the identification model downsampling layer P1 into the identification model second layer convolution layer; identifying the size of the model second layer convolutional layer C2 as 112 × 112 × 128, using ReLU as the activation function;
step 6: inputting the training image output by the identification model second layer convolution layer into an identification model down-sampling layer P2; the identification model second layer convolution layer C2 is connected with the identification model down-sampling layer P2, the size of the sliding matrix is 2 multiplied by 2, and the maximum pooling is used as a down-sampling function;
and 7: the training image output by the recognition model downsampling layer P2 is input into a recognition model third layer convolutional layer C3; identifying the size of the model third layer convolutional layer C3 as 56 × 56 × 256, and using ReLU as an activation function;
and 8: the training image output by the recognition model third layer convolutional layer C3 is input to a recognition model down-sampling layer P3; the third convolution layer C3 of the recognition model is connected with a down-sampling layer P3 of the recognition model, the size of a sliding matrix is 2 multiplied by 2, and the maximum pooling is used as a down-sampling function;
and step 9: the training image output by the recognition model downsampling layer P3 is input into a recognition model fourth convolution layer C4; identifying the size of the model fourth layer convolutional layer C4 as 28 × 28 × 512, and using ReLU as an activation function;
step 10: the training image output by the recognition model fourth layer convolutional layer C4 is input into a recognition model down-sampling layer P4; the identification model fourth layer convolution layer C4 is connected with the identification model down-sampling layer P4, the size of the sliding matrix is 2 multiplied by 2, and the maximum pooling is used as a down-sampling function;
step 11: the training image output by the identification model downsampling layer P4 is input into an identification model first layer full-connection layer F1; identifying the size of the first layer fully-connected layer F1 of the model as 1 × 1 × 4096, and using ReLU as an activation function;
step 12: the training image output by the first layer full connection layer F1 of the recognition model is input to the recognition model output layer; and connecting the first full connecting layer F1 of the recognition model with a recognition model output layer, wherein the size of the recognition model output layer is the same as the total number of the defect types, calculating the probability of the training image belonging to each type by using a softmax function in the first weight connecting layer of the recognition model, and taking the defect type corresponding to the maximum probability as the defect type of the training image.
The deep convolutional neural network in this embodiment uses cross entropy as a loss function and Adam as an optimizer for training. In this embodiment, the deep convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, and the specific number of convolutional layers can be freely set.
The above description is only a preferred embodiment of the present invention, and the embodiment is not intended to limit the scope of the present invention, so that all equivalent structural changes made by using the contents of the specification and the drawings of the present invention should be included in the scope of the appended claims.
Claims (10)
1. A chip surface defect classification method based on a generative countermeasure network is characterized by comprising the following steps:
s01: inputting the random vector into an image processing module to obtain a synthetic image, and combining a shot image and the synthetic image into an image set, wherein the image set comprises a training image set; the random vector is a vector meeting multivariate normal distribution, the image processing module comprises a generating type confrontation network, and the training image set comprises a plurality of training images and corresponding defect types thereof;
s02: inputting the training image set into a training unit for model training to obtain an identification model;
s03: and inputting the image to be recognized into the recognition model, and obtaining the corresponding defect type through recognition of the recognition model.
2. The method as claimed in claim 1, wherein the image processing module comprises a generator and a determiner, wherein the generator inputs the random vector into the generator, the composite image generated by the generator is input into the determiner, and the determiner removes the composite image that cannot be used as the training image; the remaining composite images and the captured images are formed into a training image set.
3. The method as claimed in claim 2, wherein the generator and the determiner are all units based on convolutional neural network.
4. The method as claimed in claim 2, wherein the composite image and the captured image are inputted into the determiner, and the composite image is removed if the comparison value outputted from the determiner is less than or equal to the comparison threshold.
5. The method as claimed in claim 1, wherein the image set is divided into a training image set and a test image set in step S01, and the training image set and the test image set have no intersection, and the test image set includes a plurality of test images and their corresponding defect types.
6. The method as claimed in claim 5, wherein the step S02 of obtaining the recognition model specifically includes the following steps:
s021: inputting the training image set into a training unit for model training to obtain a recognition model in training;
s022: inputting the test image set into a recognition model in training for model testing to obtain the classification accuracy of the recognition model in the training;
s023: repeating the steps S021-S022M times, and repeating the steps S021-S022 for the Mth time to obtain an identification model; wherein, the classification accuracy obtained by the Mth repeated step S021-S022 is more than or equal to the accuracy threshold, and M is an integer more than 0.
7. The method as claimed in claim 6, wherein the step S023 is repeated with steps S021-S022N times, and if the classification accuracy obtained at the nth time is greater than or equal to the accuracy threshold and the classification accuracy is not further improved within the N iterations, the nth time is repeated with steps S021-S022 to obtain the recognition model; wherein N is an integer less than or equal to M.
8. The method as claimed in claim 6, wherein in step S021, all training images in the training image set are sequentially input into a training unit for model training; in the step S022, all the test images in the test image set are sequentially input to a recognition module in training for model testing.
9. The method as claimed in claim 6, wherein before the training image set and the test image set are transmitted to the training unit in step S02, the training image set and the test image set are input to an image preprocessing unit for noise determination, and if the training image or the test image is a noise image, the training image or the test image is deleted.
10. The chip surface defect classification device based on the generative countermeasure network is characterized by comprising an image processing module and a training module, wherein the training module comprises a training unit;
inputting the random vector into the image processing module to obtain a synthetic image; the synthetic image and the shot image are combined into an image set, the image set comprises a training image set, the training image set is input into the training unit for model training to obtain a recognition model, the image to be recognized is input into the recognition model, and the corresponding defect type is obtained through recognition of the recognition model; the random vector is a vector meeting multivariate normal distribution, the training image set comprises a plurality of training images and corresponding defect types thereof, and the image processing module comprises a generative confrontation network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911126986.6A CN110992315A (en) | 2019-11-18 | 2019-11-18 | Chip surface defect classification device and method based on generative countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911126986.6A CN110992315A (en) | 2019-11-18 | 2019-11-18 | Chip surface defect classification device and method based on generative countermeasure network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110992315A true CN110992315A (en) | 2020-04-10 |
Family
ID=70084715
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911126986.6A Withdrawn CN110992315A (en) | 2019-11-18 | 2019-11-18 | Chip surface defect classification device and method based on generative countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110992315A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111291769A (en) * | 2020-05-12 | 2020-06-16 | 南京智莲森信息技术有限公司 | High-speed rail contact net foreign matter detection method and system |
| CN111693534A (en) * | 2020-06-12 | 2020-09-22 | 北京百度网讯科技有限公司 | Surface defect detection method, model training method, device, equipment and medium |
| CN112446429A (en) * | 2020-11-27 | 2021-03-05 | 广东电网有限责任公司肇庆供电局 | CGAN (Carrier grade Access network) -based routing inspection image data small sample expansion method |
| CN113052865A (en) * | 2021-04-16 | 2021-06-29 | 南通大学 | Power transmission line small sample temperature image amplification method based on image similarity |
| WO2021217852A1 (en) * | 2020-04-30 | 2021-11-04 | 平安科技（深圳）有限公司 | Damage detection method and apparatus, and electronic device and medium |
| CN113920096A (en) * | 2021-10-14 | 2022-01-11 | 广东工业大学 | Method for detecting metal packaging defects of integrated circuit |
| CN114119460A (en) * | 2021-09-18 | 2022-03-01 | 上海联麓半导体技术有限公司 | Semiconductor image defect identification method, semiconductor image defect identification device, computer equipment and storage medium |
| CN113920096B (en) * | 2021-10-14 | 2024-07-02 | 广东工业大学 | Integrated circuit metal packaging defect detection method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107563999A (en) * | 2017-09-05 | 2018-01-09 | 华中科技大学 | A kind of chip defect recognition methods based on convolutional neural networks |
| CN107945118A (en) * | 2017-10-30 | 2018-04-20 | 南京邮电大学 | A kind of facial image restorative procedure based on production confrontation network |
| CN109543720A (en) * | 2018-10-30 | 2019-03-29 | 东华大学 | A kind of wafer figure defect mode recognition methods generating network based on confrontation |
| US20190197358A1 (en) * | 2017-12-21 | 2019-06-27 | International Business Machines Corporation | Generative Adversarial Network Medical Image Generation for Training of a Classifier |
| CN110070124A (en) * | 2019-04-15 | 2019-07-30 | 广州小鹏汽车科技有限公司 | A kind of image amplification method and system based on production confrontation network |
-
2019
- 2019-11-18 CN CN201911126986.6A patent/CN110992315A/en not_active Withdrawn
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107563999A (en) * | 2017-09-05 | 2018-01-09 | 华中科技大学 | A kind of chip defect recognition methods based on convolutional neural networks |
| CN107945118A (en) * | 2017-10-30 | 2018-04-20 | 南京邮电大学 | A kind of facial image restorative procedure based on production confrontation network |
| US20190197358A1 (en) * | 2017-12-21 | 2019-06-27 | International Business Machines Corporation | Generative Adversarial Network Medical Image Generation for Training of a Classifier |
| CN109543720A (en) * | 2018-10-30 | 2019-03-29 | 东华大学 | A kind of wafer figure defect mode recognition methods generating network based on confrontation |
| CN110070124A (en) * | 2019-04-15 | 2019-07-30 | 广州小鹏汽车科技有限公司 | A kind of image amplification method and system based on production confrontation network |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021217852A1 (en) * | 2020-04-30 | 2021-11-04 | 平安科技（深圳）有限公司 | Damage detection method and apparatus, and electronic device and medium |
| CN111291769A (en) * | 2020-05-12 | 2020-06-16 | 南京智莲森信息技术有限公司 | High-speed rail contact net foreign matter detection method and system |
| CN111693534A (en) * | 2020-06-12 | 2020-09-22 | 北京百度网讯科技有限公司 | Surface defect detection method, model training method, device, equipment and medium |
| CN111693534B (en) * | 2020-06-12 | 2023-09-15 | 北京百度网讯科技有限公司 | Surface defect detection method, model training method, device, equipment and medium |
| US11763552B2 (en) | 2020-06-12 | 2023-09-19 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method for detecting surface defect, method for training model, apparatus, device, and media |
| CN112446429A (en) * | 2020-11-27 | 2021-03-05 | 广东电网有限责任公司肇庆供电局 | CGAN (Carrier grade Access network) -based routing inspection image data small sample expansion method |
| CN112446429B (en) * | 2020-11-27 | 2022-06-21 | 广东电网有限责任公司肇庆供电局 | CGAN (Carrier grade Access network) -based routing inspection image data small sample expansion method |
| CN113052865A (en) * | 2021-04-16 | 2021-06-29 | 南通大学 | Power transmission line small sample temperature image amplification method based on image similarity |
| CN113052865B (en) * | 2021-04-16 | 2023-12-19 | 南通大学 | Power transmission line small sample temperature image amplification method based on image similarity |
| CN114119460A (en) * | 2021-09-18 | 2022-03-01 | 上海联麓半导体技术有限公司 | Semiconductor image defect identification method, semiconductor image defect identification device, computer equipment and storage medium |
| CN113920096A (en) * | 2021-10-14 | 2022-01-11 | 广东工业大学 | Method for detecting metal packaging defects of integrated circuit |
| CN113920096B (en) * | 2021-10-14 | 2024-07-02 | 广东工业大学 | Integrated circuit metal packaging defect detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110992315A (en) | Chip surface defect classification device and method based on generative countermeasure network | |
| CN109272500B (en) | Fabric classification method based on adaptive convolutional neural network | |
| CN105825235B (en) | A kind of image-recognizing method based on multi-characteristic deep learning | |
| CN108509978A (en) | The multi-class targets detection method and model of multi-stage characteristics fusion based on CNN | |
| CN111369563A (en) | Semantic segmentation method based on pyramid void convolutional network | |
| CN107016413B (en) | A kind of online stage division of tobacco leaf based on deep learning algorithm | |
| CN111242185B (en) | Defect rapid preliminary screening method and system based on deep learning | |
| CN109784386A (en) | A method of it is detected with semantic segmentation helpers | |
| CN107644221A (en) | Convolutional neural networks traffic sign recognition method based on compression of parameters | |
| CN108932523B (en) | Image classification and classified data creation system and method, and storage medium | |
| CN111798409A (en) | Deep learning-based PCB defect data generation method | |
| Roy et al. | Script identification from handwritten document | |
| CN111914797A (en) | Traffic sign identification method based on multi-scale lightweight convolutional neural network | |
| US20210096530A1 (en) | System and method for identifying manufacturing defects | |
| CN112036454B (en) | Image classification method based on multi-core dense connection network | |
| CN116843650A (en) | SMT welding defect detection method and system integrating AOI detection and deep learning | |
| CN110348448A (en) | A kind of license plate character recognition method based on convolutional neural networks | |
| CN108416795B (en) | Video action identification method based on sorting pooling fusion space characteristics | |
| CN111145145A (en) | Image surface defect detection method based on MobileNet | |
| CN114332942A (en) | Night infrared pedestrian detection method and system based on improved YOLOv3 | |
| CN116129291A (en) | Unmanned aerial vehicle animal husbandry-oriented image target recognition method and device | |
| CN117422696A (en) | Belt wear state detection method based on improved YOLOv8-Efficient Net | |
| CN117788402A (en) | Industrial product defect detection method based on LIDD-Net high real-time light-weight network | |
| CN117237683B (en) | Chip defect intelligent detection system based on improved neural network | |
| CN113379685A (en) | PCB defect detection method and device based on dual-channel feature comparison model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200410 |
|
| WW01 | Invention patent application withdrawn after publication |