CN114463297A

CN114463297A - Improved chip defect detection method based on FPN and DETR fusion

Info

Publication number: CN114463297A
Application number: CN202210081334.0A
Authority: CN
Inventors: 任获荣; 于泽洋; 李向宁; 赵伟; 焦小强; 骆虎林
Original assignee: Xidian University
Current assignee: Xidian University
Priority date: 2022-01-24
Filing date: 2022-01-24
Publication date: 2022-05-10

Abstract

The invention discloses a chip defect detection method based on FPN and DETR fusion improvement, which solves the problem that the detection effect of the micro chip defect in the prior art needs to be improved. The invention comprises the following steps of step 1: acquiring image information of a defective chip at the front end of industrial production, marking the defect position and defect type, and expanding a data set by using an image data enhancement method; and 2, step: constructing an improved FPN characteristic pyramid network to extract the hierarchical characteristics of the image and fusing; and 3, step 3: constructing an improved DETR network, combining the improved FPN network in the step 2 with the DETR network to form an integral classification network, firstly training by using a public data set, and continuing training by using a chip defect data set after fixing partial parameters to obtain a trained network model; and 4, step 4: and (4) using the trained model to detect and classify the image to be detected and output a result. The technology effectively solves the problems of difficult deep learning training and less data sets in the field of chip detection.

Description

Improved chip defect detection method based on FPN and DETR fusion

Technical Field

The invention relates to the technical field of artificial intelligence deep learning, in particular to an improved chip defect detection method based on FPN and DETR fusion.

Background

With the rapid development of the semiconductor industry, the automatic inspection technology of integrated circuits becomes one of the research hotspots. The appearance detection method comprises 3 methods: (1) traditional manual detection methods; (2) detecting the appearance of the chip by using a laser measurement technology; (3) machine vision based detection method. The machine vision method has the advantages of low cost, high speed, high efficiency, non-contact and the like, and is a research hotspot.

With the rapid development of deep learning and artificial intelligence in recent years, machine vision technology is also updated. In the field of detection, more and more deep learning methods are applied. Compared with the traditional machine vision method, the deep learning method has the characteristics of high accuracy, high running speed and strong generalization capability.

In the chip defect detection task, multiple defects may exist on one chip sample, and the complexity of the multiple detection task is high. The traditional detection algorithm can only detect a certain defect on a chip, and when a plurality of defects are needed, the detection can only be carried out by utilizing a plurality of different algorithms or for a plurality of times, and the complex task can not be solved well. The more kinds of defects cause the complex detection task. The target detection based on deep learning can better solve the complex problems.

In the field of target detection, the commonly used methods such as the R-CNN method and the Faster R-CNN method need to select the content which is possibly an object from the frame in the image and then carry out detection. The two-step method has the problems of low operation speed and algorithm load detection, and meanwhile, the detection result has strong correlation with the selection of the anchor frame. The two-step approach also makes training of the neural network difficult. Because the characteristics obtained after multilayer convolution is carried out by the convolution network are complex, when the characteristics are used for detection, the neural network cannot sufficiently pay attention to effective information, and the detection effect is poor. This problem is known as the distraction problem.

The Transformer network is a network with stronger performance in natural language processing. Since the self-attention mechanism in the Transformer network can effectively solve the problem of distraction, the self-attention mechanism can be used in target task detection. Such networks are known as DEtection trees (deta). Meanwhile, the DETR network does not need to select frames and identify the frames, so that the correlation between the detection result and the anchor frame is eliminated, the algorithm complexity is reduced, and the detection efficiency is improved. In the invention, the DETR network is utilized to learn the characteristics and the position characteristics of the chip defect image, so that the detection and classification are completed in one step.

When a Transformer network is used for target detection, the problems of poor detection effect and low accuracy of a tiny target can occur. This is because the Transformer often relies on features extracted from the backbone network when detecting a target. Since in deep learning, the feature extraction level is higher as the number of convolution layers is increased, which results in discarding information of low levels, detection of a tiny target depends on features of the low levels. Particularly in the field of chip defect detection, various tiny defects depending on low-level information detection often exist. To solve this problem, the present invention uses Feature Pyramid Networks (FPN) Networks. The FPN feature pyramid network is an important means in feature extraction. And after obtaining the high-level features through convolution, the FPN obtains the multi-scale fusion features through upsampling and low-level feature fusion. By means of fusion of different multi-scale features, the low-level features can be reserved under the condition that the high-level features are correctly extracted, and the detection accuracy of the system on the micro target is improved.

The task of defect detection of chips also has the problem of sample imbalance. Sample imbalance refers to a greater number of classes and a lesser number of classes among the classes classified. For example, in the defect detection of a chip, scratches and scratch defects of the chip are common, and pin loss of the chip is rare. The training set collected in the actual production has more samples of scratches and few samples with missing pins. This results in a network that is trained heavily and that performs less well on such few classes. In order to solve the problem, the invention improves the network structure of the DETR, changes the FFN at the topmost end in the DETR network into a plurality of SVM and classifies by the classifier, and can effectively solve the problem of poor small sample class detection effect.

Disclosure of Invention

The invention solves the problem that the detection effect of the micro chip defect in the prior art needs to be improved, and provides the chip defect detection method which is high in speed, high in accuracy and capable of realizing multi-target simultaneous detection and is improved based on the fusion of the FPN and the DETR.

The technical scheme of the invention is to provide a chip defect detection method based on FPN and DETR fusion improvement, which comprises the following steps: comprises the following steps of (a) carrying out,

step 1: acquiring image information of a defective chip at the front end of industrial production, marking the defect position and defect type, and expanding a data set by using an image data enhancement method;

step 2: constructing an improved FPN characteristic pyramid network to extract the hierarchical characteristics of the image and fusing;

and step 3: constructing an improved DETR network, combining the improved FPN network in the step 2 with the DETR network to form an integral classification network, firstly training by using a public data set, and continuing training by using a chip defect data set after fixing partial parameters to obtain a trained network model;

and 4, step 4: and (4) using the trained model to detect and classify the image to be detected and output a result.

Preferably, in the step 1, the rectangular frame is used to mark the defect position and record the parameters of the rectangular frame, the defect type is marked to set the defect type to be 10 types, and the operations of translation, inversion, mirroring, projection, scaling and noise adding are performed on the image to complete the expansion of the data set.

Preferably, the FPN feature pyramid structure established in step 2 has 4 layers in total, where C1 represents an original image, C2-C4 represent different scale features after convolution respectively, the result obtained by C4 is subjected to convolution operation of 1 × 1 to obtain M4, M4 is upsampled by nearest neighbor interpolation method and subjected to convolution with C3 by 1 × 1 and added to obtain M3, and M2 is obtained in the same way, at this time, the M2 has fused the feature information of C2-C4, and M2 is flattened into a vector and stored, where the first layer convolution and the second layer convolution both use convolution kernels with expansion rate of 1, that is, ordinary convolution, the third layer convolution uses convolution kernels with expansion rate of 2 to expand the receptive field, and the fourth layer convolution uses convolution kernels with expansion rate of 4 to enhance extraction of global features.

Preferably, the improved DETR network in step 3 includes three parts, namely an encoder, a decoder and a classification network, the feature vectors extracted in step 2 and the spatial position codes thereof are added and sent to the encoder to obtain the encoder output, the input of the decoder is composed of two parts, the first part is the output of the encoder, the second part is an object query vector, the dimension of the object query is the maximum target number of the predicted defect, the output of the decoder is continuously input into the classification network, the classification network is composed of a feedforward neural network and N SVM two-classification networks, the feedforward neural network is used for predicting the position and size of the frame, all classes of the objects to be detected are classified by using an SVM classification network, and finally, the class with the maximum probability is selected as the class of the object.

Preferably, in the step 4, the image to be detected is input into the model, so as to obtain all defects and types thereof in the image.

Compared with the prior art, the improved chip defect detection method based on the fusion of the FPN and the DETR has the following advantages: the invention uses the improved DEtection trunk (DETR) network as the main structure of DEtection, can effectively solve the dependence of the DEtection result of the existing method on the selection of the anchor frame, can realize the simultaneous DEtection of a plurality of defect targets of various types, and can realize the one-step completion of the selection and identification of the DEtection frame. The method has the advantages that the DETR network is improved, the feedforward neural network part used for classification in the original network is changed into the two classifiers of the N SVM, the problem of unbalanced samples in chip defects is effectively solved, and the accuracy of rare defect detection is effectively improved. The problem of complex multi-defect simultaneous detection in the field of chip detection is effectively solved.

According to the method, on the basis of DETR, a characteristic pyramid structure of an improved FPN is added, multi-scale high-low order characteristics can be extracted for fusion, and the problem that the DETR network has a poor detection effect on a tiny target is effectively solved. The improved FPN network is realized by using the hole convolution, the number of layers and network parameters of the network are effectively reduced, and the complexity of the network is reduced. Meanwhile, the detection of the micro defect target is obviously improved by adding the improved FPN network. The problem that a tiny defect target is difficult to detect in the field of chip detection is effectively solved.

The invention uses an image data enhancement method and a transfer learning training method. The problems of difficulty in deep learning and training and few data sets in the field of chip detection are effectively solved.

Drawings

FIG. 1 is a schematic flow diagram of the present invention;

FIG. 2 is a schematic diagram of an improved FPN feature pyramid network structure according to the present invention;

fig. 3 is a schematic diagram of an improved DETR network structure of the present invention;

FIG. 4 is a schematic diagram of the overall architecture of the converged network of the present invention;

Detailed Description

The chip defect detection method based on FPN and DETR fusion improvement of the invention is further explained with reference to the accompanying drawings and the specific implementation mode:

as shown in FIG. 1, the overall flow of the present embodiment includes four steps S1-S4.

The operation of step S1 includes operations such as image translation, image rotation, image mirroring, image projection, image scaling, and the like. And collecting the chip image with the appearance defect to ensure the clearness and integrity of the chip image. And marking the defect position by using a rectangular frame, recording the parameters of the rectangular frame, marking the defect type, and setting the defect type to be 10. And the image is subjected to operations of translation, turning, mirroring and noise addition to complete the expansion of the data set.

S2: building an FPN feature pyramid network as shown in fig. 2, firstly setting a step length to 2 by using a convolution kernel of 3 × 3 for an original image, performing convolution to obtain a C1 layer, and obtaining a C2 layer on an output feature map of C1 in the same manner. The convolution was performed using a 3 × 3 convolution kernel with an expansion rate of 2 with a step size of 2 to obtain C3 layers, and the convolution was performed using a 3 × 3 convolution kernel with an expansion rate of 4 with a step size of 2 to obtain C4 layers on the output feature map of C3.

The resulting C4 was convolved with a 1 × 1 convolution kernel to yield M4, upsampled by nearest neighbor interpolation on M4 to make the feature map size the same as C4, and convolved with a 1 × 1 convolution kernel on C4 to make the two channels the same, which were added to yield M4. The formula for calculating the upsampling multiplying power n is as follows:

where i is the input feature map size, p is the fill mode, k is the convolution kernel size, s is the step size, and d is the expansion ratio.

Wherein the nearest neighbor interpolation algorithm formula is as follows

M3, M2 were calculated using the same calculation. M1 is discarded because the feature information level is too low. And flattening the obtained fused feature information into a vector of (H × W) × C and storing the vector.

Wherein the pyramid structure of the FPN features created in S2 has 4 layers. Wherein C1 represents the original image, and C2-C4 represent the different scale features after convolution, respectively. The result obtained from C4 was subjected to a convolution operation with 1 × 1 to obtain M4. M3 is obtained by adding up M4 upsampled by nearest neighbor interpolation and C3 after 1 × 1 convolution, and M2 is obtained by the same method. At this time, M2 has fused the characteristic information of C2-C4, and M2 is flattened into a vector and is stored. The first layer of convolution and the second layer of convolution both use convolution kernels with the expansion rate of 1, namely common convolution. The third layer of convolution uses a convolution kernel with the expansion rate of 2 to enlarge the receptive field, and the fourth layer of convolution uses a convolution kernel with the expansion rate of 4 to enhance the extraction of the global features.

The improved DETR network in S3 includes three parts, an encoder, a decoder, and a classification network; adding the feature vectors extracted in the S2 and the space position codes thereof, sending the sum into an encoder to obtain the output of the encoder, wherein the input of a decoder consists of two parts, the first part is the output of the encoder, the second part is an object query vector, the dimension of the object query is the maximum target number of the predicted defect, the input is continuously input into a classification network after the output of the decoder, the classification network consists of a feedforward neural network and N SVM two-classification networks, the feedforward neural network is used for predicting the position and the size of a selection frame, all classes of the target to be detected are classified by using an SVM classification network, and finally, the class with the maximum probability is selected as the class of the target.

S3: a modified DETR network is constructed as shown in FIG. 3, wherein the Detection transform network comprises six encoders and six decoders, a feed-forward neural network and 11 SVM classifiers. The encoder is composed of a multi-head self-attention mechanism unit and a feedforward neural network unit, the encoders are sequentially connected according to the sequence, the encoders are identical in structure but parameters are not shared, and the parameters need to be continuously learned from training.

Using the formula

Position coding is performed, even positions are coded using a sin function, and odd positions are coded using a cos function. The encoding result and the eigenvector obtained in S2 are added and input to the encoder, and the output of the previous encoder is the input of the next encoder, and the output of the sixth encoder is stored.

Each decoder comprises two multi-head self-attention mechanism units and a feedforward neural network. The input of the first multi-headed attention mechanism unit of the decoder is the object query vector, where the maximum defect target is set to 5, i.e. the dimension of the input query vector is 5. The input query vector is initialized randomly and needs to be improved continuously through learning to reach the best. The input of the second multi-headed attention mechanism unit in each decoder is the output of the encoder. And finally, the output decoded by the six decoders is saved.

And respectively transmitting the output of the last decoder into a feedforward neural network and an SVM classifier, wherein the feedforward neural network is used for predicting a rectangular frame, 11 SVM is used as the classifier of each defect class, the classifier comprises ten defect classes and a null class, and one class with the highest probability in all the predictors of the SVM is taken as the defect class.

When training a neural network, the loss function can be calculated and expressed as

Firstly, a network is pre-trained by using a network public data set VOC data set, the parameters of the first three layers in the FPN are fixed after training, the network is trained by using a chip defect data set continuously, and a network model is saved after the expected accuracy is reached.

S4: and inputting the image to be detected into the model to obtain all defects and types thereof in the image.

The invention is based on the overall network structure improved by FPN and DETR as shown in figure 4, effectively solves the technical problems that the existing chip detection technology cannot simultaneously detect a plurality of defects, the defect detection excessively depends on the selection of an anchor frame, the two-step detection diffusion efficiency is too low, the complexity is too high, and the detection effect on tiny chip defects is poor by an image data expansion mode and a multi-scale fusion idea, solves the problem of sample imbalance of the chip defects, and simultaneously relieves the problems of more deep learning parameters and large required training data amount. The design aims of high speed, high accuracy and multi-target simultaneous detection are fulfilled.

The existing deep learning scheme can better solve the complex detection problems, but the deep learning method also has some difficulties, such as more deep learning parameters, higher network complexity, more difficult training and the like. In order to alleviate the problems, the invention optimizes the FPN network, realizes the FPN network by using a method of combining hole convolution and common convolution, reduces the number of layers of the FPN network and reduces network parameters.

Training for deep learning is also one of the current problems, and more training data is required due to the complex network structure. In order to effectively solve the problem of less data of a training set, the invention adopts an image data enhancement technology and a transfer learning technology. The image data enhancement technology carries out operations such as translation, rotation, noise addition, mirror surface turning and the like on the acquired defect image to obtain more defect image data. And (3) learning by using a transfer learning method, firstly training by using a large data volume data set disclosed by a network, then fixing the lower-level network parameters obtained by training, and continuously retraining the model by using the chip defect data set.

Based on the summary of the existing method, an improved chip detection method based on FPN and DETR fusion is provided. Firstly, multi-scale fusion feature information is obtained through the FPN network, then the information is input into the DETR network, and finally the position and the type of the target defect are output. The invention finally achieves the results of high speed, high accuracy and simultaneous detection of multiple targets.

Claims

1. A chip defect detection method based on FPN and DETR fusion improvement is characterized in that: comprises the following steps of (a) carrying out,

2. The improved chip defect detection method based on FPN and DETR fusion of claim 1, wherein: in the step 1, the rectangular frame is used for marking the defect position and recording the parameters of the rectangular frame, marking the defect type and setting the defect type to be 10 types, and the image is subjected to operations of translation, turnover, mirror image, projection, scaling and noise addition to complete the expansion of the data set.

3. The improved chip defect detection method based on FPN and DETR fusion as claimed in claim 1, wherein: the FPN feature pyramid structure established in step 2 has 4 layers, where C1 represents an original image, C2-C4 represent different scale features after convolution respectively, the result obtained by C4 is subjected to convolution operation of 1 × 1 to obtain M4, M4 is subjected to up-sampling by nearest neighbor interpolation and convolution of C3 by 1 × 1 and then added to obtain M3, similarly, M2 can be obtained, at this time, M2 has fused the feature information of C2-C4, M2 is flattened as a vector and stored, where the first layer convolution and the second layer convolution both use a convolution kernel with an expansion rate of 1, that is, a common convolution, the third layer convolution uses a convolution kernel with an expansion rate of 2 to expand a receptive field, and the fourth layer convolution uses a convolution kernel with an expansion rate of 4 to enhance extraction of global features.

4. The improved chip defect detection method based on FPN and DETR fusion of claim 1, wherein: the improved DETR network in the step 3 comprises an encoder, a decoder and a classification network, the feature vectors extracted in the step 2 and the space position codes thereof are added and sent to the encoder to obtain the output of the encoder, the input of the decoder consists of two parts, the first part is the output of the encoder, the second part is an object query vector, the dimension of the object query is the maximum target number of the predicted defects, the input is continuously input into the classification network after the output of the decoder, the classification network consists of a feed-forward neural network and N SVM two classification networks, the feed-forward neural network is used for predicting the position and the size of a selection frame, all classes of the targets to be detected are classified by using an SVM classification network, and finally, the class with the maximum probability is selected as the class of the targets.

5. The improved chip defect detection method based on FPN and DETR fusion of claim 1, wherein: and 4, inputting the image to be detected into the model to obtain all defects and types thereof in the image.