CN116309429A - A Chip Defect Detection Method Based on Deep Learning - Google Patents

Abstract

The invention discloses a chip defect detection method based on deep learning, which includes the following steps: collecting different types of chip surface defect images through a high-speed industrial camera; using an image labeling tool to mark the position and type of chip surface image defects, and constructing defects Data set, the defect data set is divided into training data set D and test data set T; based on the improved FasterR-CNN framework, a deep learning network model for chip defect detection is constructed; this chip defect detection method based on deep learning Improve the Faster R-CNN algorithm by adopting deformable convolution reconstruction feature extraction network, multi-scale feature fusion, region of interest alignment (ROI alignment), soft non-maximum suppression and other measures to solve the problem of small size and complex shape of chip defects and other issues, improving the accuracy of defect detection.

Description

A Chip Defect Detection Method Based on Deep Learning

technical field

The invention relates to the technical field of artificial intelligence, in particular to a chip defect detection method based on deep learning.

Background technique

After the chip is powered on, it will generate a start command to transmit signal 0 and transmit data, and it can also make home appliances intelligently connected, which is the core foundation of high-end manufacturing. However, the chip manufacturing process is very complicated, and it is very easy to produce some unexpected structures during the production process, resulting in target defects. In recent years, deep learning technology has developed rapidly. Based on the powerful learning ability and feature extraction ability of deep learning in a large amount of data, many researchers try to apply deep learning technology to product defect detection, which greatly improves the efficiency of defect detection. improve. However, the problem of small target detection is still one of the difficulties in the field of surface defect detection of industrial products, so improving the defect detection rate of small targets such as chips has become an urgent problem to be solved.

Contents of the invention

The purpose of the present invention is to provide a chip defect detection method based on deep learning to solve the problems raised in the background technology, aiming at the deficiencies of the prior art.

To achieve the above object, the present invention provides the following technical solutions: comprising the steps of:

Step 1: Collect different types of chip surface defect images through high-speed industrial cameras;

Step 2: Use the image annotation tool to mark the position and type of the chip surface image defect, construct a defect data set, and divide the defect data set into a training data set D and a test data set T;

Step 3: Based on the improved FasterR-CNN framework, build a deep learning network model for chip defect detection:

1) Select the feature extraction network. Classical feature extraction networks include VggNet, ResNet, GoogleNet, etc. Among them, the advantage of the ResNet network lies in the residual unit of its structure, which solves the problem of gradient disappearance and increases the accuracy of the network. Therefore, ResNet- 50 is used as a feature extraction network. The ResNet-50 network contains 49 convolutional layers and a fully connected layer. The ResNet-50 network structure can be divided into seven parts. The first part is mainly responsible for the calculation of input data convolution, regularization, activation function, and maximization pool. The first part does not contain a residual block, and the second, third, fourth, and fifth parts of the structure all contain a residual block. In the ResNet-50 network structure, the residual block has three layers of convolution, so there are a total of 49 convolution layers in the network, plus the last fully connected layer is a total of 50 layers;

2) Deformable convolution reconstruction feature extraction network, deformable convolution means that the convolution kernel adds an additional parameter direction parameter to each element, so that the convolution kernel can be expanded to a large range during the training process . Due to the different shapes of defects on the chip surface, there is no fixed geometry. Therefore, for ResNet-50, which has poor ability of unknown change and weak generalization ability, the idea of deformable convolution is introduced to improve the recognition ability of neural network for irregular targets. The implementation process of deformable convolution is as follows:

Sub-step 1: adjust the input size and perform preprocessing operations on the collected chip image.

Sub-step 2: According to the input image, use the traditional convolution kernel to extract the feature map.

Sub-step 3: Get the deformed offset of the deformable convolution: take the obtained feature map as input, and add another convolution layer to the feature map.

Sub-step 4: The offset layer is 2N (where 2 refers to the two directions of x and y, and N refers to the size of the convolution kernel). Since translation is required on the two-dimensional plane, only the x and y values need to be changed. Wherein the x value and the y value represent the offset of the pixel in the image data in the x and y directions, respectively.

During the training process, the convolution kernel used to generate output features and the convolution kernel used to generate offsets are learned synchronously.

3) Multi-scale feature fusion. Since the chip defect detection itself is characterized by small target detection, the feature pyramid FPN is introduced for feature fusion. Feature fusion is to fuse deep semantic information into shallow feature maps, use deep features to enrich semantic information, and use shallow features at the same time. Introduce the feature pyramid FPN into the faster R-CNN, integrate the features of all layers into the feature pyramid, and at the same time use the feature activation output of the second, third, fourth, and fifth layers of the residual block in the ResNet-50 network as input . The number of channels of C2-C5 is reduced to 256 by 1x1 convolution, and M2-M5 is obtained (take C2 and M2 as examples, C2 represents the feature matrix of a series of residual structures corresponding to Conv2, and M2 represents C2 through 1×1 volume feature matrix of the residual structure after product). The shallow feature map and deep feature map of the same size are added by upsampling to obtain the output P2-P5 of the FPN, and then 3x3 convolution is used. The feature map P6 is obtained by downsampling the two largest pools on the output P5 of the FPN, outputting a multi-scale fusion feature combination. Use the fusion features in RPN to generate target candidate frames, and obtain detection results through classification;

4) Region of interest alignment (ROI alignment). ROIAlign differs from ROI pooling not only by simply quantizing and then pooling, but also by using a region feature aggregation method to convert it into a continuous operation. The specific implementation steps of the alignment of the region of interest are as follows:

Sub-step 1: Loop through all candidate regions and keep the floating-point coordinates of the mapped candidate regions unquantized.

Sub-step 2: The candidate area is divided into z×z cells, and each cell is not quantized.

Sub-step 3: Determine the floating-point coordinates of the sampling points in each unit, calculate the floating-point coordinates of the sampling points using a bilinear interpolation method, and then obtain a fixed-dimensional ROI output.

5) Soft non-maximum suppression (SoftNMS). Soft non-maximum suppression is to re-evaluate all candidate frames, retain the poorer frames but suppress the scores, and the final realization result is that SoftNMS has made a modification to the frame scores, retaining a higher recall Rate. SoftNMS can be expressed by the following formula:

As a preferred technical solution of the present invention, Si is the score of the detection frame i in the SoftNMS formula, M is the highest score frame, the bi is the collection of detection frames, and the Nt is the threshold set, and the NMS will set the IOU greater than The score of the detection frame of the threshold is set to 0, while the soft NMS attenuates the score of the detection frame whose IOU is greater than the threshold, which can alleviate the problem of target missed detection and false detection. The IOU threshold of the soft NMS is 0.5, and the minimum score is 0.05.

Compared with the prior art, the present invention provides a chip defect detection method based on deep learning, which has the following beneficial effects:

The purpose of the present invention is to aim at the characteristics and requirements of chip defect detection, take Faster R-CNN as the basic framework, and consist of a feature extraction network, a region recommendation network, and a detection network. Therefore, a chip defect detection algorithm based on deep learning is proposed. The algorithm adopts measures such as deformable convolution reconstruction feature extraction network, multi-scale feature fusion, alignment of regions of interest (ROI alignment), soft non-maximum suppression, etc. -The CNN algorithm is improved to solve the problems of small size and complex shape of chip defects, and improve the accuracy of defect detection.

Description of drawings

Fig. 1 is the network structural diagram of ResNet50 of the present invention;

Fig. 2 is a schematic diagram of deformable convolution in the present invention;

Fig. 3 is a structure diagram of the multi-scale feature fusion network of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to Fig. 1-3, in this embodiment: comprise the following steps:

Step 1: Collect different types of chip surface defect images through high-speed industrial cameras;

Step 2: Use the image annotation tool to mark the position and type of the chip surface image defect, construct a defect data set, and divide the defect data set into a training data set D and a test data set T;

Step 3: Based on the improved FasterR-CNN framework, build a deep learning network model for chip defect detection:

1) Select the feature extraction network. Classical feature extraction networks include VggNet, ResNet, GoogleNet, etc. Among them, the advantage of the ResNet network lies in the residual unit of its structure, which solves the problem of gradient disappearance and increases the accuracy of the network. Therefore, ResNet- 50 is used as a feature extraction network. The ResNet-50 network contains 49 convolutional layers and a fully connected layer. The ResNet-50 network structure can be divided into seven parts. The first part is mainly responsible for the calculation of input data convolution, regularization, activation function, and maximization pool. The first part does not contain a residual block, and the second, third, fourth, and fifth parts of the structure all contain a residual block. In the ResNet-50 network structure, the residual block has three layers of convolution, so there are a total of 49 convolution layers in the network, plus the last fully connected layer is a total of 50 layers;

2) Deformable convolution reconstruction feature extraction network, deformable convolution means that the convolution kernel adds an additional parameter direction parameter to each element, so that the convolution kernel can be expanded to a large range during the training process . Due to the different shapes of defects on the chip surface, there is no fixed geometry. Therefore, for ResNet-50, which has poor ability of unknown change and weak generalization ability, the idea of deformable convolution is introduced to improve the recognition ability of neural network for irregular targets. The implementation process of deformable convolution is as follows:

Sub-step 1: adjust the input size and perform preprocessing operations on the collected chip image.

Sub-step 2: According to the input image, use the traditional convolution kernel to extract the feature map.

Sub-step 3: Get the deformed offset of the deformable convolution: take the obtained feature map as input, and add another convolution layer to the feature map.

Sub-step 4: The offset layer is 2N (where 2 refers to the two directions of x and y, and N refers to the size of the convolution kernel). Since translation is required on the two-dimensional plane, only the x and y values need to be changed. Wherein the x value and the y value represent the offset of the pixel in the image data in the x and y directions, respectively.

During the training process, the convolution kernel used to generate output features and the convolution kernel used to generate offsets are learned synchronously.

3) Multi-scale feature fusion. Since the chip defect detection itself is characterized by small target detection, the feature pyramid FPN is introduced for feature fusion. Feature fusion is to fuse deep semantic information into shallow feature maps, use deep features to enrich semantic information, and use shallow features at the same time. Introduce the feature pyramid FPN into the faster R-CNN, integrate the features of all layers into the feature pyramid, and at the same time use the feature activation output of the second, third, fourth, and fifth layers of the residual block in the ResNet-50 network as input . The number of channels of C2-C5 is reduced to 256 by 1x1 convolution, and M2-M5 is obtained (take C2 and M2 as examples, C2 represents the feature matrix of a series of residual structures corresponding to Conv2, and M2 represents C2 through 1×1 volume feature matrix of the residual structure after product). The shallow feature map and deep feature map of the same size are added by upsampling to obtain the output P2-P5 of the FPN, and then 3x3 convolution is used. The feature map P6 is obtained by downsampling the two largest pools on the output P5 of the FPN, outputting a multi-scale fusion feature combination. Use the fusion features in RPN to generate target candidate frames, and obtain detection results through classification;

4) Region of interest alignment (ROI alignment). ROIAlign differs from ROI pooling not only by simply quantizing and then pooling, but also by using a region feature aggregation method to convert it into a continuous operation. The specific implementation steps of the alignment of the region of interest are as follows:

Sub-step 1: Loop through all candidate regions and keep the floating-point coordinates of the mapped candidate regions unquantized.

Sub-step 2: The candidate area is divided into z×z cells, and each cell is not quantized.

Sub-step 3: Determine the floating-point coordinates of the sampling points in each unit, calculate the floating-point coordinates of the sampling points using a bilinear interpolation method, and then obtain a fixed-dimensional ROI output.

5) Soft non-maximum suppression (SoftNMS). Soft non-maximum suppression is to re-evaluate all candidate frames, retain the poorer frames but suppress the scores, and the final realization result is that SoftNMS has made a modification to the frame scores, retaining a higher recall Rate. SoftNMS can be expressed by the following formula:

It should be noted that in the SoftNMS formula, Si is the score of the detection frame i, M is the highest score frame, bi is the set of detection frames, Nt is the set threshold, NMS sets the score of the detection frame with IOU greater than the threshold to 0, and Soft NMS attenuates the scores of detection frames whose IOU is greater than the threshold, which can alleviate the problem of missed and false detection of targets. The IOU threshold of soft NMS is 0.5, and the minimum score is 0.05.

Finally, it should be noted that: the above is only a preferred embodiment of the present invention, and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it still The technical solutions recorded in the foregoing embodiments may be modified, or some technical features thereof may be equivalently replaced. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (2)

1. A chip defect detection method based on deep learning is characterized in that: the method comprises the following steps:

step one: collecting different types of chip surface defect images through a high-speed industrial camera;

step two: marking positions and types of image defects on the surface of the chip through an image marking tool, constructing a defect data set, and dividing the defect data set into a training data set D and a test data set T;

step three: based on the improved Faster R-CNN framework, a deep learning network model for chip defect detection is constructed:

1) The characteristic extraction network is selected, the classical characteristic extraction network comprises VggNet, resNet, googleNet and the like, wherein the ResNet network has the advantages of residual units of the structure, the gradient disappearance problem is solved, the accuracy of the network is improved, and the ResNet-50 is selected as the characteristic extraction network. The ResNet-50 network comprises 49 convolution layers and one full connection layer, and the ResNet-50 network structure can be divided into seven parts, wherein the first part is mainly responsible for calculating input data convolution, regularization, activation functions and maximization pools. Wherein the first part does not contain residual blocks, and the second, third, fourth and fifth part structures all contain residual blocks. In the ResNet-50 network structure, the residual blocks all have three convolutions, and the total of 49 convolution layers in the network and the total of the last full connection layer is 50 layers;

2) The deformable convolution reconstructing the feature extraction network means that a convolution kernel is additionally added with a parameter direction parameter on each element, so that the convolution kernel can be expanded to a large range in the training process. There is no fixed geometry due to the different shape of the chip surface defects. Therefore, the concept of deformable convolution is introduced aiming at ResNet-50 with poor unknown change capability and weak generalization capability, and the recognition capability of the neural network on irregular targets is improved. The realization process of the deformable convolution is as follows:

sub-step 1: and adjusting the input size of the acquired chip image and performing preprocessing operation.

Sub-step 2: from the input image, a feature map is extracted using a conventional convolution kernel.

Sub-step 3: the offset of the deformation resulting in the deformable convolution: and taking the obtained characteristic diagram as input, and adding a convolution layer to the characteristic diagram.

Sub-step 4: the offset layer is 2N (where 2 refers to both x and y directions and N refers to the convolution kernel size) and only the x and y values, representing the offsets in the x and y directions of the image data pixels, respectively, need to be changed since translation is required in the two-dimensional plane.

Wherein during training, the convolution kernels used to generate the output features and the convolution kernels used to generate the offsets are synchronously learned.

3) The multi-scale feature fusion is carried out by introducing a feature pyramid FPN because the chip defect detection is the characteristic of small target detection. Feature fusion is to fuse deep semantic information into a shallow feature map, enrich semantic information by using deep features and simultaneously use features of shallow features. Feature pyramid FPN is introduced into faster R-CNN, features of all layers are integrated into feature pyramid, and feature activation output of second, third, fourth and fifth residual blocks in ResNet-50 network is taken as input. The number of channels of C2-C5 is reduced to 256 by 1x1 convolution, resulting in M2-M5 (taking C2 and M2 as examples, C2 represents the feature matrix of a series of residual structures corresponding to Conv2, and M2 represents the feature matrix of the residual structure of C2 after 1x1 convolution). The shallow layer feature map and the depth feature map with the same size are added through upsampling to obtain the outputs P2-P5 of the FPN, and then 3x3 convolution is adopted. The feature map P6 is obtained by downsampling the two largest pools on the output P5 of the FPN, outputting a multi-scale fusion feature combination. Generating a target candidate frame by utilizing fusion characteristics in the RPN, and obtaining a detection result through classification;

4) Region of interest alignment (ROI alignment). ROI alignment differs from ROI pooling not only in simple quantization and pooling, but also in converting it to continuous operation using a region feature aggregation method. The specific implementation steps of the region of interest alignment are as follows:

sub-step 1: all candidate regions are accessed in a loop and the mapped candidate region floating point coordinates are kept unquantized.

Sub-step 2: the candidate region is divided into zxz cells and each cell is also not quantized.

Sub-step 3: floating point coordinates of the sampling points in each unit are determined, floating point coordinates of the sampling points are calculated using a bilinear interpolation method, and then ROI output of a fixed dimension can be obtained.

5) Soft non-maximum suppression (Soft NMS). Soft non-maximum suppression is a re-estimation of all candidate boxes, but the score is preserved for the worse boxes, with the end result that Soft NMS makes a modification to the score of the boxes, preserving higher recall. The Soft NMS may be represented by the following formula:

2. the method for detecting the chip defects based on deep learning as claimed in claim 1, wherein: in the third step, si is the score of the detection frame i, M is the highest score frame, bi is the set of detection frames, nt is the set threshold, the NMS sets the score of the detection frame with the IOU greater than the threshold to 0, and the Soft NMS attenuates the score of the detection frame with the IOU greater than the threshold, so as to alleviate the problems of target missing detection and false detection, the IOU threshold of the Soft NMS is 0.5, and the lowest score is 0.05.

Images