CN116228740A - Small sample chip appearance defect detection method and detection system based on improved YOLOv5 - Google Patents

Small sample chip appearance defect detection method and detection system based on improved YOLOv5 Download PDF

Info

Publication number
CN116228740A
CN116228740A CN202310363269.5A CN202310363269A CN116228740A CN 116228740 A CN116228740 A CN 116228740A CN 202310363269 A CN202310363269 A CN 202310363269A CN 116228740 A CN116228740 A CN 116228740A
Authority
CN
China
Prior art keywords
picture
training
model
neural network
defect detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310363269.5A
Other languages
Chinese (zh)
Inventor
陈俊风
成佳康
谢迎娟
王海滨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hohai University HHU
Original Assignee
Hohai University HHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hohai University HHU filed Critical Hohai University HHU
Priority to CN202310363269.5A priority Critical patent/CN116228740A/en
Publication of CN116228740A publication Critical patent/CN116228740A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a small sample chip appearance defect detection method and a detection system based on improved YOLOv5, wherein the method comprises the steps of obtaining a defective image as an original data set; training the original data set to generate an countermeasure network, thereby obtaining an amplified data set; introducing a CBAM attention mechanism into a Yolov5 neural network model to obtain improved Yolov5: training and improving the YOLOv5 neural network model by using the amplification data set to obtain a defect detection model; acquiring pictures of chips to be detected, preprocessing the pictures, inputting the preprocessed pictures into a defect detection model for detection, inputting the pictures with defects to a display for display, and finding out corresponding chips according to the displayed pictures; the system comprises a model training module, a picture acquisition module and a display and marking module. The method solves the problem that the network model is easy to be over-fitted due to the fact that the deep learning network model used for defect detection and constructed based on the YOLOv4 network in the prior art has too small defect data quantity.

Description

Small sample chip appearance defect detection method and detection system based on improved YOLOv5
Technical Field
The invention relates to the technical field of chip defect detection, in particular to a small sample chip appearance defect detection method and system based on improved YOLOv 5.
Background
After the chip is packaged, appearance defects such as bubbles and the like are required to be detected, and the method is also an important link in quality detection, wherein the main method for detecting the defects is manual detection and chip appearance defect detection based on traditional machine vision. The manual detection efficiency is low, the real-time performance is poor, the detection cost is high, the detection precision is low, and the detection precision is easily influenced by subjective factors of people; the chip defect detection method based on traditional machine vision mainly utilizes an image algorithm to extract the defect characteristics of a chip, and then distinguishes and identifies defects through the numerical values of various characteristics.
The prior art provides a deep learning-based method, which is to input a large amount of image data marked manually, learn the surface defects of various chips by utilizing the very strong image feature extraction capability of a deep convolutional neural network, enable the deep convolutional neural network to memorize the surface defect features of different types of chips, and can identify the defect pictures under the complex condition.
Chinese patent CN112967243B proposes a YOLO-based deep learning chip package crack defect detection method, which utilizes a YOLOv4 network constructed by training a data set, inputs a picture to be detected after standardized treatment to predict the network, and further filters the result by using a confidence threshold and a crack boundary threshold to obtain a final result.
Disclosure of Invention
The invention aims to: the invention aims to provide a small sample chip appearance defect detection method and a detection system based on improved YOLOv5, which are suitable for small sample training and have high accuracy.
The technical scheme is as follows: in order to achieve the above object, the method for detecting the appearance defect of the small sample chip based on the improved YOLOv5 comprises the following steps:
step S1: acquiring a plurality of groups of images, wherein each image comprises chips arranged in a two-dimensional array, and laser marking is carried out on the chips with defects to obtain an original data set;
step S2: training the original data set to generate an countermeasure network, and obtaining an amplified data set from the generated countermeasure network;
step S3: introducing a CBAM attention mechanism into a YOLOv5 neural network model to obtain improved YOLOv5;
step S4: training an improved YOLOv5 neural network model introduced into a CBAM attention mechanism module by using an amplification data set to obtain a defect detection model;
step S5: acquiring a picture of a chip to be tested, and preprocessing;
step S6: inputting the preprocessed picture into a defect detection model for detection, inputting the picture with the detected defect into a display for display, and carrying out laser marking on a corresponding chip;
step S7: and finding out the corresponding chip according to the displayed picture.
Wherein, training the generated countermeasure network by using the original data set in the step 2, obtaining the amplified data set from the generated countermeasure network, including the following substeps:
step S201: generating an countermeasure network comprises generating a model and a judging model, wherein the generating model adopts a generator to generate a picture capable of being in false spurious according to input random noise; judging whether the input image is real data or a picture generated by a generator by adopting a judging model;
generating a picture by a generator G according to random noise z, and marking the picture as G (z); g (z) and a real picture in the original data set are input into a discriminator D as samples; the discriminator D marks any picture x in the sample as D (x);
step S202: taking the generated picture and the real picture which can be correctly distinguished as losses of the discriminator, taking the picture which can be generated approximately and truly, judging the generated picture as the losses of the generator by the discriminator, and generating a target loss function of the countermeasure network as follows:
V(D,G)＝E x～μ [logD(x)]+E z～γ [log(1-D(G(z)))]；
wherein, E represents the expectation of corresponding distribution of subscripts, z is random noise, x is a real picture, G (z) is a picture generated by a generator according to the random noise, mu is the distribution of the real picture x, gamma is the distribution of the generated picture G (z), and D (x) is the probability of judging whether the picture is the real picture by a discriminator;
keeping the parameters of the generator G unchanged, optimizing the discriminator D according to the loss of the discriminator D, wherein the loss function of the discriminator D is as follows:
Figure BDA0004165758870000021
when D (x) is close to 1 and D (G (x)) is close to 0, the discriminator D can discriminate the generated picture;
keeping parameters of the discriminator D unchanged, optimizing the generator G according to the loss of the generator G, wherein the loss function of the generator G is as follows:
Figure BDA0004165758870000022
when D (G (z)) is close to 1, the discriminator D cannot discriminate the generated picture;
when mu is equal to gamma or is very close to gamma, the discriminator judges the generated picture G (z) as a real picture;
step S203: setting iteration times, inputting random noise into the generator G in each iteration process, executing steps S201 and S202 until the distribution gamma of the generated pictures approaches to the distribution mu of the real pictures x, namely, finishing the training of the generated countermeasure network, otherwise, continuing iteration;
step S204: when the training of the countermeasure network is finished, the generator G generates a plurality of pictures which can be in false and spurious, and the pictures are input into the original data set to obtain an amplified data set.
In step S3, a CBAM attention mechanism module is introduced into the YOLOv5 neural network model to obtain an improved YOLOv5, which specifically includes: the CBAM attention mechanism is a double attention mechanism of channel attention and space attention, an intermediate feature map of a Yolov5 neural network model introduced into a CBAM attention mechanism module is taken as an input feature map, and the CBAM attention mechanism module sequentially deduces a channel attention feature map M C (F) And a spatial attention profile M S (F) M is set to C (F) Performing element-by-element multiplication operation on the input feature map F to generate a transition feature map F' required by the spatial attention module, and performing M S Element-by-element multiplication is carried out on the (F ') and the transitional characteristic diagram F ', so that a final characteristic diagram F ' of a CBAM attention mechanism is obtained:
Figure BDA0004165758870000031
Figure BDA0004165758870000032
/>
wherein F is a characteristic diagram with input height H, width W and channel number C,
Figure BDA0004165758870000033
for multiplication by element, M C For channel attention operation, M S Is a spatial attention operation.
Wherein the CBAM attention mechanism module deduces a channel attention feature map M C (F) The method specifically comprises the following steps: will inputThe feature map F is subjected to global average pooling and global maximum pooling based on the height H and the width W respectively to obtain two feature maps of 1 multiplied by C, the two feature maps of 1 multiplied by C are respectively sent to the same two layers of neural networks, the number of neurons of the first layer is C/r, r is a reduction rate, an activation function is a ReLu function, the number of neurons of the second layer is C, and convolution operation of 1 multiplied by 1 is carried out to realize cross-channel information interaction;
the characteristics output by the double-layer neural network are subjected to element-by-element addition operation and then subjected to Sigmoid activation operation to generate a channel attention characteristic diagram M C (F)：
Figure BDA0004165758870000034
Where σ is a sigmoid function, avgPool is average pooling, maxPool is maximum pooling, MLP is a double-layer neural network, MLP (AvgPool (F)) is the result of inputting AvgPool (F) into the double-layer neural network,
Figure BDA0004165758870000035
for the result of the pooling of the characteristic map F with the number of channels C,/for the characteristic map F with the number of channels C>
Figure BDA0004165758870000036
Results of maximizing pooling of feature map F with channel number C, W 0 Is the first layer of the double-layer neural network, W 1 Is the second layer of the two-layer neural network.
Wherein the CBAM attention mechanism module deduces a spatial attention profile M S (F) The method specifically comprises the following steps: taking the intermediate feature diagram F 'as an input feature diagram of the module, carrying out global maximum pooling and global average pooling on the basis of a channel C on the F' to obtain two H multiplied by W multiplied by 1 feature diagrams, carrying out channel splicing operation on the two H multiplied by W multiplied by 1 feature diagrams on the basis of the channel C, and then carrying out 7 multiplied by 7 convolution operation to reduce the dimension to 1 channel, namely H multiplied by W multiplied by 1;
generating a spatial attention feature map M through BN normalization and Sigmoid activation S (F`)：
Figure BDA0004165758870000041
Where σ is the sigmoid function, avgPool is the average pooling, maxPool is the maximum pooling, f is the convolution operation,
Figure BDA0004165758870000042
for the result of the mean pooling of the feature map F with the number of channels S,/for the feature map F>
Figure BDA0004165758870000043
The result is obtained by maximizing the pooling of the characteristic diagram F with the number of channels being S.
The training of the improved YOLOv5 neural network model introduced into the CBAM attention mechanism module by using the amplified data set in step S4 refers to training the improved YOLOv5 neural network model introduced into the CBAM attention mechanism module by using an early-stop method based on the amplified data set, specifically comprising:
dividing the amplification data set into a training set and a verification set, and inputting the training set into an improved YOLOv5 neural network model for training; setting the iteration times t, verifying the training effect by using a verification set every n iterations, finishing training when the generalization loss GL of the verification set is larger than a set value alpha, taking the parameter of the last iteration as the final parameter of the model, and obtaining a defect detection model, wherein the generalization loss is as follows:
Figure BDA0004165758870000044
wherein t is the iteration number, E va (t) is the verification set error at the t-th iteration, E opt (t) is the smallest validation set error in the first t iterations,
Figure BDA0004165758870000045
the step S5 of acquiring the images of the chips to be tested refers to using a camera to acquire pictures of all the chips, inputting the pictures into a computer, marking the chips on the pictures with position information one by the computer, and preprocessing the pictures.
Wherein the pretreatment comprises the following steps: graying the picture, reducing the noise by median filtering, detecting the edge and correcting the inclined image.
The step S6 of inputting the detected defective picture to the display for displaying and laser marking the corresponding chip means that when the defect detection model detects that the chip on the picture is defective, the computer inputs the picture to the display for displaying and controls the marking machine to perform laser marking on the defective chip according to the position information of the picture.
A small sample chip appearance defect detection system based on improved YOLOv5 comprises a model training module, a picture acquisition module and a display and marking module;
the model training module comprises a generating countermeasure network training module and a neural network model training module, wherein the generating countermeasure network training module is used for obtaining an amplification data set, and the neural network model training module is used for inputting the amplification data set into an improved YOLOv5 neural network model which is introduced into the CBAM attention mechanism module for training so as to obtain a defect detection model;
the image acquisition module is used for acquiring a chip image and inputting the chip image into the defect detection model for detection; the display and marking module is used for displaying the detection result and marking the defective chip.
The beneficial effects are that: the invention has the following advantages: 1. according to the invention, the training data set is greatly expanded by using the method for generating the training data of the countermeasure network, so that the training of the small sample is realized;
2. the YOLOv5 model structure used by the invention has the characteristics of high detection precision, high reasoning speed, good target detection effect and the like, can identify the types of defects in pictures and output the coordinates of a target boundary box, and realizes the improved YOLOv5 neural network model which introduces CBAM by training a small sample by using a GAN extended data set, thereby realizing the identification of the appearance defects of the small sample chip;
3. the YOLOv5 neural network which introduces the CBAM attention mechanism is trained by the early-stop method, so that the network can pay more attention to the target to be detected, the aim of further improving the detection effect is fulfilled, and the problem that the network model is easy to be fitted excessively due to too small defect data quantity is solved.
Drawings
FIG. 1 is a schematic flow chart of a defect detection method according to the present invention;
FIG. 2 is a schematic diagram of a flow of generating an countermeasure network according to the present invention;
FIG. 3 is a schematic diagram of the attention mechanism flow of the CBAM of the present invention;
FIG. 4 is a modified YOLOv5 network architecture of the present invention;
FIG. 5 is a comparison of SPP and SPPF structures of the present invention;
FIG. 6 is an original diagram of a chip under test according to the present invention;
FIG. 7 is a graph of the result of edge detection of the Canny operator of the present invention;
FIG. 8 (a) is a picture before tilt correction in an embodiment of the present invention, (b) is a picture after tilt correction;
fig. 9 is an output image of the original image of the chip to be tested after detection.
Detailed Description
The technical scheme of the present invention will be described in detail with reference to the following examples and the accompanying drawings.
As shown in fig. 1, the method for detecting the appearance defects of the small sample chip based on the improved YOLOv5 comprises the following steps:
step S1: acquiring a plurality of groups of images, wherein each image comprises chips arranged in a two-dimensional array, and laser marking is carried out on the chips with defects to obtain an original data set;
step S2: generating an countermeasure network using raw data set training, obtaining an augmentation data set from the generated countermeasure network, as shown in fig. 2, comprising the sub-steps of:
step S201: generating an countermeasure network comprises generating a model and a judging model, wherein the generating model adopts a generator to generate a picture capable of being in false spurious according to input random noise; judging whether the input image is real data or a picture generated by a generator by adopting a judging model;
generating a picture by a generator G according to random noise z, and marking the picture as G (z); g (z) and a real picture in the original data set are input into a discriminator D as samples; the discriminator D marks any picture x in the sample as D (x);
step S202: taking the generated picture and the real picture which can be correctly distinguished as losses of the discriminator, taking the picture which can be generated approximately and truly, judging the generated picture as the losses of the generator by the discriminator, and generating a target loss function of the countermeasure network as follows:
V(D,G)＝E x～μ [logD(x)]+E z～γ [log(1-D(G(z)))]；
wherein, E represents the expectation of corresponding distribution of subscripts, z is random noise, x is a real picture, G (z) is a picture generated by a generator according to the random noise, mu is the distribution of the real picture x, gamma is the distribution of the generated picture G (z), and D (x) is the probability of judging whether the picture is the real picture by a discriminator;
keeping the parameters of the generator G unchanged, optimizing the discriminator D according to the loss of the discriminator D, wherein the loss function of the discriminator D is as follows:
Figure BDA0004165758870000061
when D (x) is close to 1 and D (G (x)) is close to 0, the discriminator D can discriminate the generated picture;
keeping parameters of the discriminator D unchanged, optimizing the generator G according to the loss of the generator G, wherein the loss function of the generator G is as follows:
Figure BDA0004165758870000062
when D (G (z)) is close to 1, the discriminator D cannot discriminate the generated picture;
when mu is equal to gamma or is very close to gamma, the discriminator judges the generated picture G (z) as a real picture;
step S203: setting iteration times, inputting random noise into the generator G in each iteration process, executing steps S201 and S202 until the distribution gamma of the generated pictures approaches to the distribution mu of the real pictures x, namely, finishing the training of the generated countermeasure network, otherwise, continuing iteration;
step S204: when the training of the countermeasure network is finished, the generator G generates a plurality of pictures which can be in false and spurious, and the pictures are input into the original data set to obtain an amplified data set.
Step S3: introducing a CBAM attention mechanism into a YOLOv5 neural network model to obtain improved YOLOv5, wherein the improved YOLOv5 is specifically: as shown in FIG. 3, the CBAM attention mechanism is a dual attention mechanism of channel attention and spatial attention, an intermediate feature map of a YOLOv5 neural network model introduced into a CBAM attention mechanism module is taken as an input feature map, and the CBAM attention mechanism module sequentially deduces a channel attention feature map M C (F) And a spatial attention profile M S (F) M is set to C (F) Performing element-by-element multiplication operation on the input feature map F to generate a transition feature map F' required by the spatial attention module, and performing M S Element-by-element multiplication is carried out on the (F ') and the transitional characteristic diagram F ', so that a final characteristic diagram F ' of a CBAM attention mechanism is obtained:
Figure BDA0004165758870000071
Figure BDA0004165758870000076
wherein F is a characteristic diagram with input height H, width W and channel number C,
Figure BDA0004165758870000072
for multiplication by element, M C For channel attention operation, M S Is a spatial attention operation.
Wherein the CBAM attention mechanism module deduces a channel attention feature map M C (F) The method specifically comprises the following steps: the input characteristic diagram F is respectively processed based on the height H and the width WThe global average pooling and global maximum pooling of the system are carried out to obtain two characteristic diagrams of 1 multiplied by C, the two characteristic diagrams of 1 multiplied by C are respectively sent to the same two layers of neural networks, the number of neurons of the first layer is C/r, r is the reduction rate, the activation function is a ReLu function, the number of neurons of the second layer is C, and the convolution operation of 1 multiplied by 1 is carried out to realize the cross-channel information interaction;
the characteristics output by the double-layer neural network are subjected to element-by-element addition operation and then subjected to Sigmoid activation operation to generate a channel attention characteristic diagram M C (F)：
Figure BDA0004165758870000073
Where σ is a sigmoid function, avgPool is average pooling, maxPool is maximum pooling, MLP is a double-layer neural network, MLP (AvgPool (F)) is the result of inputting AvgPool (F) into the double-layer neural network,
Figure BDA0004165758870000074
for the result of the pooling of the characteristic map F with the number of channels C,/for the characteristic map F with the number of channels C>
Figure BDA0004165758870000075
Results of maximizing pooling of feature map F with channel number C, W 0 Is the first layer of the double-layer neural network, W 1 Is the second layer of the two-layer neural network.
Wherein the CBAM attention mechanism module deduces a spatial attention profile M S (F) The method specifically comprises the following steps: taking the intermediate feature diagram F 'as an input feature diagram of the module, carrying out global maximum pooling and global average pooling on the basis of a channel C on the F' to obtain two H multiplied by W multiplied by 1 feature diagrams, carrying out channel splicing operation on the two H multiplied by W multiplied by 1 feature diagrams on the basis of the channel C, and then carrying out 7 multiplied by 7 convolution operation to reduce the dimension to 1 channel, namely H multiplied by W multiplied by 1;
generating a spatial attention feature map M through BN normalization and Sigmoid activation S (F`)：
Figure BDA0004165758870000081
Where σ is the sigmoid function, avgPool is the average pooling, maxPool is the maximum pooling, f is the convolution operation,
Figure BDA0004165758870000082
for the result of the mean pooling of the feature map F with the number of channels S,/for the feature map F>
Figure BDA0004165758870000083
The result is obtained by maximizing the pooling of the characteristic diagram F with the number of channels being S.
According to the invention, a CBAM attention mechanism is inserted after the Conv module is activated in the YOLOv5 neural network model, the Conv module consists of convolution operation, BN normalization and activation function, and the Conv module is improved to be a Conv-CBAM module, so that the network can pay more attention to the target to be detected, and the aim of improving the detection effect is fulfilled.
The improved YOLOv5 neural network model structure is shown in fig. 4, and compared with YOLOv4, YOLOv5 can simultaneously consider accuracy and speed, specifically: compared with the data enhancement method of the YOLOv4, the data enhancement method of the YOLOv5 is simpler, and only three methods of scaling, color space adjustment, and Mosaic data enhancement are used; the anchor block of YOLOv4 is fixed, while the anchor block of YOLOv5 is adaptively learned based on the training set; YOLOv5 employs leakyReLu and Sigmoid as activation functions; in the neck of network detection, unlike the SPP of YOLOv4, the SPPF is used in YOLOv5, so that the operation efficiency can be effectively improved, and the SPP and SPPF structure pair is shown in fig. 5.
The backbone network of YOLOv5 is CSP-Draknet-53, the Darknet-53 feature extraction network comprises 53 convolution layers, and integrates the thought of a residual network, so that the problems of gradient disappearance, gradient explosion and the like can be effectively avoided, CSP-Darknet-53 adds a CSP structure on the basis of Darknet-53, residual blocks originally stacked together are split into left and right parts, one part carries out convolution operation, and the other part is fused with a feature map after the convolution operation is completed with the last part, so that the detection speed can be improved on the premise of ensuring the detection precision.
Step S4: training an improved YOLOv5 neural network model introduced into a CBAM attention mechanism module by using an amplification data set to obtain a defect detection model, namely training the improved YOLOv5 neural network model introduced into the CBAM attention mechanism module by adopting an early-stop method based on the amplification data set, wherein the method specifically comprises the following steps of:
dividing the amplification data set into a training set and a verification set, and inputting the training set into an improved YOLOv5 neural network model for training; setting the iteration times t, verifying the training effect by using a verification set every 15 iterations, finishing training when the generalization loss GL of the verification set is larger than a set value alpha, taking the parameter of the last iteration as the final parameter of the model, and obtaining a defect detection model, wherein the generalization loss is as follows:
Figure BDA0004165758870000084
wherein t is the iteration number, E va (t) is the verification set error at the t-th iteration, E opt (t) is the smallest validation set error in the first t iterations,
Figure BDA0004165758870000091
alpha is required to be selected according to the training purpose, if alpha is selected to be larger, the difficulty of training stopping is higher, the possibility of errors is higher, and the global optimal solution is more likely to be found; if alpha is smaller, the training stop requirement is lower, the possibility of errors is smaller, but the best solution possibly found is only the local best solution.
The early-stopping method is a skill when training the deep neural network, the training is stopped once the data test error rises, and the weight after stopping is used as the final parameter of the network so as to obtain the best generalization performance; when training a deep neural network, a better generalization performance is often required, a model is expected to be capable of fitting data better, but if the iteration times are too many, the deep neural network may have an overfitting phenomenon, namely the network works better and better when detecting a training set, but the effect of detecting a testing set becomes worse and the overfitting phenomenon can be better solved by an early stop method.
Step S5: acquiring a picture of a chip to be tested, and preprocessing; the method comprises the steps of acquiring images of chips to be detected, namely acquiring pictures of all the chips by using a camera, inputting the pictures into a computer, marking the position information of the chips on the pictures one by the computer, and preprocessing the pictures; the pretreatment comprises the following steps: graying the picture, reducing the noise by median filtering, detecting the edge and correcting the inclined image.
As shown in fig. 6, for the original image of the chip to be tested, the image is subjected to graying treatment to avoid stripe distortion and increase the operation speed, and then median filtering is used to reduce noise, so that the noise of the image is suppressed or eliminated to a certain extent, and the quality of the image is improved, and the accuracy of detecting and identifying the appearance defects of the chip is improved indirectly.
As shown in fig. 7, a set of pixels with rapidly changing brightness in an image is found based on a canny edge detection operator, so that edge detection of a chip is realized, the data volume can be greatly reduced, and the processing speed is improved; the tilted chip image is corrected based on Hough transform, and the pre-correction and post-correction pictures are shown in fig. 8 (a) and (b), respectively.
Step S6: the method comprises the steps of inputting a preprocessed picture into a defect detection model for detection, inputting the detected picture with the defect into a display for display, and carrying out laser marking on a corresponding chip, wherein when the defect detection model detects that the chip on the picture is defective, a computer inputs the picture into the display for display, and a marking machine is controlled to carry out laser marking on the defective chip according to the position information of the picture. And finally, outputting a result after detection, wherein the situation that pins are broken at the upper left part, the middle part and the lower middle part of the picture is shown in fig. 9.
Chinese patent CN107755879a provides a laser marking machine, a method for adjusting the distance between a scanning head and a marking object, and an automatic focusing method for the marking machine, by which the present invention uses the method to perform laser marking on a defective chip.
Step S7: and finding out the corresponding chip according to the displayed picture.
The invention also provides a small sample chip appearance defect detection system based on the improved YOLOv5, which comprises a model training module, a picture acquisition module and a display and marking module; the model training module comprises a generating countermeasure network training module and a neural network model training module, wherein the generating countermeasure network training module is used for obtaining an amplification data set, and the neural network model training module is used for inputting the amplification data set into an improved YOLOv5 neural network model which is introduced into the CBAM attention mechanism module for training so as to obtain a defect detection model; the image acquisition module is used for acquiring a chip image and inputting the chip image into the defect detection model for detection; the display and marking module is used for displaying the detection result and marking the defective chip.
The deep convolutional neural network used in the invention is a target detection model YOLO (You Only Look Once), can identify the types of defects in pictures and output the coordinates of a target boundary box, utilizes the existing few chip defect samples, and realizes the improved YOLOv5 neural network model which introduces CBAM by using small sample training through GAN expansion data set, thereby realizing the identification of the appearance defects of the small sample chip, and introducing CBAM can improve the attention of the YOLOv5 neural network to the target to be detected, thereby achieving the purpose of improving the detection effect, and effectively solving the problems of low efficiency, poor instantaneity, high detection cost, low detection precision and the like in the manual detection and the chip appearance defect detection based on the traditional machine vision.
According to the invention, the training data set is greatly expanded by using the method for generating the training data of the countermeasure network, so that the training of the small sample is realized; compared with the YOLOv4, the used YOLOv5 model structure has improved detection precision, reasoning speed, target detection effect and the like;
the YOLOv5 neural network which introduces the CBAM attention mechanism is trained by the early-stop method, so that the network can pay more attention to the target to be detected, the aim of further improving the detection effect is fulfilled, and the problem that the network model is easy to be fitted excessively due to too small defect data quantity is solved.

Claims (10)

1. The small sample chip appearance defect detection method based on improved YOLOv5 is characterized by comprising the following steps of:
step S1: acquiring a plurality of groups of images, wherein each image comprises chips arranged in a two-dimensional array, and laser marking is carried out on the chips with defects to obtain an original data set;
step S2: training the original data set to generate an countermeasure network, and obtaining an amplified data set from the generated countermeasure network;
step S3: introducing a CBAM attention mechanism into a YOLOv5 neural network model to obtain improved YOLOv5;
step S4: training an improved YOLOv5 neural network model introduced into a CBAM attention mechanism module by using an amplification data set to obtain a defect detection model;
step S5: acquiring a picture of a chip to be tested, and preprocessing;
step S6: inputting the preprocessed picture into a defect detection model for detection, inputting the picture with the detected defect into a display for display, and carrying out laser marking on a corresponding chip;
step S7: and finding out the corresponding chip according to the displayed picture.
2. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 1, wherein: training the generated countermeasure network by using the original data set in the step 2, and obtaining the amplified data set from the generated countermeasure network, wherein the method comprises the following substeps:
step S201: generating an countermeasure network comprises generating a model and a judging model, wherein the generating model adopts a generator to generate a picture capable of being in false spurious according to input random noise; judging whether the input image is real data or a picture generated by a generator by adopting a judging model;
generating a picture by a generator G according to random noise z, and marking the picture as G (z); g (z) and a real picture in the original data set are input into a discriminator D as samples; the discriminator D marks any picture x in the sample as D (x);
step S202: taking the generated picture and the real picture which can be correctly distinguished as losses of the discriminator, taking the picture which can be generated approximately and truly, judging the generated picture as the losses of the generator by the discriminator, and generating a target loss function of the countermeasure network as follows:
V(D,G)＝E x～μ [log D(x)]+E z～γ [log(1-D(g(z)))]；
wherein, E represents the expectation of corresponding distribution of subscripts, z is random noise, x is a real picture, G (z) is a picture generated by a generator according to the random noise, mu is the distribution of the real picture x, gamma is the distribution of the generated picture G (z), and D (x) is the probability of judging whether the picture is the real picture by a discriminator;
keeping the parameters of the generator G unchanged, optimizing the discriminator D according to the loss of the discriminator D, wherein the loss function of the discriminator D is as follows:
Figure FDA0004165758820000021
when D (x) is close to 1 and D (G (x)) is close to 0, the discriminator D can discriminate the generated picture;
keeping parameters of the discriminator D unchanged, optimizing the generator G according to the loss of the generator G, wherein the loss function of the generator G is as follows:
Figure FDA0004165758820000022
when D (G (z)) is close to 1, the discriminator D cannot discriminate the generated picture;
when mu is equal to gamma or is very close to gamma, the discriminator judges the generated picture G (z) as a real picture;
step S203: setting iteration times, inputting random noise into the generator G in each iteration process, executing steps S201 and S202 until the distribution gamma of the generated pictures approaches to the distribution mu of the real pictures x, namely, finishing the training of the generated countermeasure network, otherwise, continuing iteration;
step S204: when the training of the countermeasure network is finished, the generator G generates a plurality of pictures which can be in false and spurious, and the pictures are input into the original data set to obtain an amplified data set.
3. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 1, wherein: step S3, introducing a CBAM attention mechanism module into the Yolov5 neural network model to obtain improved Yolov5, wherein the improved Yolov5 is specifically as follows: the CBAM attention mechanism is a double attention mechanism of channel attention and space attention, an intermediate feature map of a Yolov5 neural network model introduced into a CBAM attention mechanism module is taken as an input feature map, and the CBAM attention mechanism module sequentially deduces a channel attention feature map M C (F) And a spatial attention profile M S (F) M is set to C (F) Performing element-by-element multiplication operation on the input feature map F to generate a transition feature map F' required by the spatial attention module, and performing M S Element-by-element multiplication is carried out on the (F ') and the transitional characteristic diagram F ', so that a final characteristic diagram F ' of a CBAM attention mechanism is obtained:
Figure FDA0004165758820000023
Figure FDA0004165758820000024
wherein F is a characteristic diagram with input height H, width W and channel number C,
Figure FDA0004165758820000025
for multiplication by element, M C For channel attention operation, M S Is a spatial attention operation.
4. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 3, wherein: the CBAM attention mechanism module deduces a channel attention feature map M C (F) The method specifically comprises the following steps: respectively carrying out global average pooling and global maximum pooling on the input feature map F based on the height H and the width W to obtain two feature maps of 1 multiplied by C, respectively sending the two feature maps of 1 multiplied by C to the same two-layer neural network, wherein the number of neurons of the first layer is C/r, r is a reduction rate, an activation function is a ReLu function, the number of neurons of the second layer is C, and carrying out convolution operation of 1 multiplied by 1 to realize cross-channel information interaction;
the characteristics output by the double-layer neural network are subjected to element-by-element addition operation and then subjected to Sigmoid activation operation to generate a channel attention characteristic diagram M C (F)：
Figure FDA0004165758820000031
Where σ is a sigmoid function, avgPool is average pooling, maxPool is maximum pooling, MLP is a double-layer neural network, MLP (AvgPool (F)) is the result of inputting AvgPool (F) into the double-layer neural network,
Figure FDA0004165758820000032
for the result of the pooling of the characteristic map F with the number of channels C,/for the characteristic map F with the number of channels C>
Figure FDA0004165758820000033
Results of maximizing pooling of feature map F with channel number C, W 0 Is the first layer of the double-layer neural network, W 1 Is the second layer of the two-layer neural network.
5. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 4, wherein: the CBAM attention mechanism module deduces a spatial attention feature map M S (F) The method specifically comprises the following steps: taking the intermediate feature diagram F 'as an input feature diagram of the module, carrying out global maximum pooling and global average pooling on the basis of a channel C on the F', obtaining two H multiplied by W multiplied by 1 feature diagrams, carrying out channel splicing operation on the two H multiplied by W multiplied by 1 feature diagrams on the basis of the channel C,then through a 7×7 convolution operation, the dimension is reduced to 1 channel, namely H×W×1;
generating a spatial attention feature map M through BN normalization and Sigmoid activation S (F`)：
Figure FDA0004165758820000034
Where σ is the sigmoid function, avgPool is the average pooling, maxPool is the maximum pooling, f is the convolution operation,
Figure FDA0004165758820000035
for the result of the mean pooling of the feature map F with the number of channels S,/for the feature map F>
Figure FDA0004165758820000036
The result is obtained by maximizing the pooling of the characteristic diagram F with the number of channels being S.
6. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 1, wherein: the training of the improved YOLOv5 neural network model introduced into the CBAM attention mechanism module by using the amplification data set in step S4 refers to training the improved YOLOv5 neural network model introduced into the CBAM attention mechanism module by using an early-stop method based on the amplification data set, specifically comprising the following steps:
dividing the amplification data set into a training set and a verification set, and inputting the training set into an improved YOLOv5 neural network model for training; setting the iteration times t, verifying the training effect by using a verification set every n iterations, finishing training when the generalization loss GL of the verification set is larger than a set value alpha, taking the parameter of the last iteration as the final parameter of the model, and obtaining a defect detection model, wherein the generalization loss is as follows:
Figure FDA0004165758820000041
wherein t is stackNumber of generations, E va (t) is the verification set error at the t-th iteration, E opt (t) is the smallest validation set error in the first t iterations,
Figure FDA0004165758820000042
7. the improved YOLOv 5-based small sample chip appearance defect detection method of claim 1, wherein: and S5, acquiring images of the chips to be tested, namely acquiring pictures of all the chips by using a camera, inputting the pictures into a computer, marking the position information of the chips on the pictures one by the computer, and preprocessing the pictures.
8. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 7, wherein: the pretreatment comprises the following steps: graying the picture, reducing the noise by median filtering, detecting the edge and correcting the inclined image.
9. The improved YOLOv 5-based small sample chip appearance defect detection method of claim 1, wherein: and step S6, inputting the detected defective picture to a display for display, and carrying out laser marking on the corresponding chip, namely, when the defect detection model detects that the chip on the picture is defective, inputting the picture to the display for display by a computer, and controlling a marking machine to carry out laser marking on the defective chip according to the position information of the picture.
10. A small sample chip appearance defect detection system based on improved YOLOv5 is characterized in that: the system comprises a model training module, a picture acquisition module and a display and marking module;
the model training module comprises a generating countermeasure network training module and a neural network model training module, wherein the generating countermeasure network training module is used for obtaining an amplification data set, and the neural network model training module is used for inputting the amplification data set into an improved YOLOv5 neural network model which is introduced into the CBAM attention mechanism module for training so as to obtain a defect detection model;
the image acquisition module is used for acquiring a chip image and inputting the chip image into the defect detection model for detection; the display and marking module is used for displaying the detection result and marking the defective chip.
CN202310363269.5A 2023-04-07 2023-04-07 Small sample chip appearance defect detection method and detection system based on improved YOLOv5 Pending CN116228740A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310363269.5A CN116228740A (en) 2023-04-07 2023-04-07 Small sample chip appearance defect detection method and detection system based on improved YOLOv5

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310363269.5A CN116228740A (en) 2023-04-07 2023-04-07 Small sample chip appearance defect detection method and detection system based on improved YOLOv5

Publications (1)

Publication Number Publication Date
CN116228740A true CN116228740A (en) 2023-06-06

Family

ID=86591287

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310363269.5A Pending CN116228740A (en) 2023-04-07 2023-04-07 Small sample chip appearance defect detection method and detection system based on improved YOLOv5

Country Status (1)

Country Link
CN (1) CN116228740A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116824271A (en) * 2023-08-02 2023-09-29 上海互觉科技有限公司 SMT chip defect detection system and method based on tri-modal vector space alignment
CN116934733A (en) * 2023-08-04 2023-10-24 湖南恩智测控技术有限公司 Reliability test method and system for chip
CN116993679A (en) * 2023-06-30 2023-11-03 芜湖合德传动科技有限公司 Method for detecting belt abrasion of telescopic machine based on target detection
CN117115162A (en) * 2023-10-24 2023-11-24 中安芯界控股集团有限公司 Chip production control system for detecting battery based on visual analysis

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116993679A (en) * 2023-06-30 2023-11-03 芜湖合德传动科技有限公司 Method for detecting belt abrasion of telescopic machine based on target detection
CN116993679B (en) * 2023-06-30 2024-04-30 芜湖合德传动科技有限公司 Method for detecting belt abrasion of telescopic machine based on target detection
CN116824271A (en) * 2023-08-02 2023-09-29 上海互觉科技有限公司 SMT chip defect detection system and method based on tri-modal vector space alignment
CN116824271B (en) * 2023-08-02 2024-02-09 上海互觉科技有限公司 SMT chip defect detection system and method based on tri-modal vector space alignment
CN116934733A (en) * 2023-08-04 2023-10-24 湖南恩智测控技术有限公司 Reliability test method and system for chip
CN116934733B (en) * 2023-08-04 2024-04-09 湖南恩智测控技术有限公司 Reliability test method and system for chip
CN117115162A (en) * 2023-10-24 2023-11-24 中安芯界控股集团有限公司 Chip production control system for detecting battery based on visual analysis

Similar Documents

Publication Publication Date Title
CN116228740A (en) Small sample chip appearance defect detection method and detection system based on improved YOLOv5
CN108961235B (en) Defective insulator identification method based on YOLOv3 network and particle filter algorithm
CN111814867A (en) Defect detection model training method, defect detection method and related device
US7215829B2 (en) Method and system for object recognition using fractal map
CN111223093A (en) AOI defect detection method
US7257267B2 (en) Method and system for image segmentation
WO2023168972A1 (en) Linear array camera-based copper surface defect detection method and apparatus
CN107328776A (en) A kind of quick determination method of immune chromatography test card
CN111415339B (en) Image defect detection method for complex texture industrial product
CN112990392A (en) New material floor defect target detection system based on improved YOLOv5 algorithm
CN110533036B (en) Rapid inclination correction method and system for bill scanned image
CN110517213A (en) A kind of real time field depth continuation method based on laplacian pyramid of microscope
CN116433669B (en) Machine vision-based quality detection method for weld joints of steel frame of anti-seismic structure
CN110689003A (en) Low-illumination imaging license plate recognition method and system, computer equipment and storage medium
CN114463314A (en) Wafer defect detection method and system based on color difference shadow model
CN115690073A (en) Local characterization method, device and medium for ceramic microstructure manufactured by laser additive manufacturing
CN115752969A (en) Method, system and equipment for detecting sealing performance of aluminum foil seal
CN114549414A (en) Abnormal change detection method and system for track data
CN106600615A (en) Image edge detection algorithm evaluation system and method
CN109102500A (en) A kind of automatic recognition system of chip package defect
CN110298347A (en) A kind of recognition methods of the automobile exhaust analyzer screen based on GrayWorld and PCA-CNN
CN114511851B (en) Hairspring algae cell statistical method based on microscope image
CN115546141A (en) Small sample Mini LED defect detection method and system based on multi-dimensional measurement
CN114943729A (en) Cell counting method and system for high-resolution cell image
CN114943738A (en) Sensor packaging curing adhesive defect identification method based on visual identification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination