CN114092441A

CN114092441A - Product surface defect detection method and system based on dual neural network

Info

Publication number: CN114092441A
Application number: CN202111386718.5A
Authority: CN
Inventors: 杨玮林; 董越; 许德智; 潘庭龙; 张永巍
Original assignee: Jiangnan University
Current assignee: Jiangnan University
Priority date: 2021-11-22
Filing date: 2021-11-22
Publication date: 2022-02-25

Abstract

The invention discloses a product surface defect detection method based on a double neural network, which comprises the following steps: s1: collecting a sample image with defects; s2: processing the sample image to form a data set; s3: respectively training a first neural network and a second neural network which are built in advance by adopting a data set to obtain a corresponding first defect prediction model and a corresponding second defect prediction model; s4: acquiring an image to be detected in real time by using a CCD detection array; s5: and preprocessing the image to be detected, and then sequentially inputting the preprocessed image into the first defect prediction model and the second defect prediction model to obtain the type, the size and the position of the defect on the image. The invention solves the problems of large data quantity, high omission factor, high false detection rate, low efficiency and the like required by model training.

Description

Product surface defect detection method and system based on dual neural network

Technical Field

The invention relates to the technical field of machine vision detection, in particular to a product surface defect detection method and system based on a dual neural network.

Background

With the rapid development of national economy in China, the quality requirements of people on industrial products are higher and higher, and as one of important factors influencing the quality of the products, the defect detection of the surfaces of the products is the most visual and important detection link. The existence of the surface defects of the products is often caused by the damage of a process system in production or errors in a transportation process, and the defect detection of the products in time is beneficial to improving the product quality, saving the production cost and avoiding the process waste.

At present, most of surface defect detection of industrial products adopts a manual visual inspection mode, manual detection needs to consume a large amount of time and energy, and long-time detection accuracy is difficult to guarantee. Particularly, in the aspect of fine defect detection, very fine defects with a width less than 50 μm are difficult to be successfully detected manually.

The existing deep learning network has high detection accuracy on obvious scratches and other defects, but has serious false detection and missing detection conditions on slight defects with the width less than 50 mu m, so that the actual requirements are difficult to meet, and the learning network has many learning model parameters, high training difficulty and low efficiency.

Disclosure of Invention

The invention aims to provide a product surface defect detection method and system based on a double neural network, and solves the problems of large data volume, high omission factor, high false detection rate, low efficiency and the like required by model training.

In order to solve the technical problem, the invention provides a product surface defect detection method based on a dual neural network, which comprises the following steps:

s1: collecting a sample image with defects;

s2: processing the sample image to form a data set;

s3: respectively training a first neural network and a second neural network which are built in advance by adopting a data set to obtain a corresponding first defect prediction model and a corresponding second defect prediction model;

s4: acquiring an image to be detected in real time by using a CCD detection array;

s5: and preprocessing the image to be detected and then sequentially inputting the preprocessed image into the first defect prediction model and the second defect prediction model to obtain the type, size and position of the defect on the image.

As a further improvement of the present invention, the processing of the sample image in step S2 includes the following steps:

s21: unifying the size of the sample image;

s22: reducing noise of a sample image by adopting a median filtering algorithm, and enhancing the image by calculating a local histogram of the image and redistributing brightness to change the contrast of the image by using a self-adaptive contrast-limited self-adaptive histogram equalization algorithm so as to make defects in the sample image more prominent;

s23: translating, overturning or rotating the sample image to expand the sample;

s24: and marking a defect area on the sample image and the type, size and position of the defect by using labelimg, and taking the marked sample image as a training data set.

As a further improvement of the present invention, in step S3, the first neural network is trained using a yolo-v3 neural network to obtain a first defect prediction model, and the yolo-v3 neural network includes a Darknet-53 trunk feature extraction network, 25 convolutional layers, and two fully-connected layers, where Darknet-53 uses a Residual error network Residual.

As a further improvement of the present invention, before the yolo-v3 neural network is trained, each sample image in the training dataset is divided into S × S grids, each grid generates a plurality of rectangular frames that may contain defects, and the position of the label on each sample image is composed of six parameters, which are w, h, x, y, conf, respectively, where w, h are the width and height of the rectangular frame, x, y are the adjustment parameters of the center position of the rectangular frame, and conf is the confidence that whether the rectangular frame contains defects or not.

As a further improvement of the invention, each convolution part in the Darknet-53 trunk feature extraction network adopts a Draknet Conv2D structure, L2 regularization is carried out when each convolution is carried out by the Draknet Conv2D structure, Batchnormalization standardization and LeakyReLU are carried out after the convolution is completed, and LeakyReLU is a slope which adds a nonzero slope to all negative values.

As a further improvement of the present invention, in the step S3, the second neural network is trained by using the fast RCNN neural network to obtain a second defect prediction model, which includes the following steps:

s31: adopting ResNet50 with residual structure as main feature extraction network to extract defect image feature in data set;

s32: generating a detection frame by using RPN, taking the obtained characteristic image as input, and obtaining a preselected frame by performing one convolution of 3 x 3 and two convolutions of 1 x 1;

s33: the pre-selected box is input to the pooling layer of ResNet50 along with the original feature images, and the category to which each object specifically belongs is calculated by the fully connected layer and the softmax function.

As a further improvement of the present invention, the ResNet50 includes a Conv2d convolutional layer, a batch normalization layer, a ReLU function layer, a max pooling layer, 4 convolutional blocks, 12 identification blocks, an average pooling layer, and a full connection layer, which are sequentially arranged.

As a further improvement of the present invention, the fast RCNN neural network is pre-trained by using the Imagenet data set, and the pre-trained fast RCNN neural network is used for the training in step S3.

A product surface defect detection system based on a dual neural network comprises:

the acquisition device is used for collecting sample images with defects and acquiring images to be detected in real time;

the data processing module is used for processing the sample image to form a data set;

the neural network module is used for training a first neural network and a second neural network which are built in advance by adopting a data set to obtain a corresponding first defect prediction model and a corresponding second defect prediction model;

and the detection module is used for preprocessing the image to be detected and then sequentially inputting the preprocessed image into the first defect prediction model and the second defect prediction model to obtain the type, the size and the position of the defect on the image.

As a further improvement of the invention, the acquisition device comprises a 1200 ten thousand pixel CCD detection array, the 1200 ten thousand pixel CCD detection array is configured with a 25mm focal length lens, and the data processing module, the neural network module and the detection module are all configured in the FPGA.

The invention has the beneficial effects that: according to the invention, deep learning is used, through extraction and learning of a large number of sample characteristics, label characteristic information obtained by a machine is continuously deepened, the process of extracting artificial characteristics of the vision of the traditional machine is greatly reduced, autonomous learning of defects with strong randomness can be realized, and along with the increase of the number of samples, through detection of a double neural network, the classification efficiency and precision of the defects can be greatly improved; the detection method provided by the invention is different from the traditional semantic segmentation neural network, the required data volume is small, labelimg is adopted to label the sample, the labeling task is greatly reduced, the learning model parameters are few, the training time is short, the precision is high, and the false detection rate and the missing detection rate are low; the established defect detection system can detect scratches with the width less than 20 micrometers, so that the precision and the detection efficiency are ensured.

Drawings

FIG. 1 is a schematic flow chart of the detection method of the present invention;

FIG. 2 is a flow chart of the neural network training of yolo-v3 in an embodiment of the present invention;

FIG. 3 is a flow chart of the neural network detection of yolo-v3 in the embodiment of the present invention;

FIG. 4 is a schematic diagram of the detection result of the yolo-v3 neural network on defect detection in the embodiment of the invention;

FIG. 5 is a diagram illustrating the second detection result of the yolo-v3 neural network in defect detection according to the embodiment of the present invention;

FIG. 6 is a flow chart of fast RCNN neural network training in accordance with an embodiment of the present invention;

FIG. 7 is a flow chart of the fast RCNN neural network detection in an embodiment of the present invention;

FIG. 8 is a schematic diagram of specific dimension marks of the microscratches measured by an optical microscope in an embodiment of the invention;

FIG. 9 is a schematic diagram of the detection result of the Faster RCNN neural network on the micro scratch according to the embodiment of the present invention;

FIG. 10 is a diagram illustrating a system platform according to an embodiment of the present invention.

Detailed Description

The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.

Referring to fig. 1, an embodiment of the present invention provides a product surface defect detection method based on a dual neural network, including the following steps:

s1: collecting a sample image with defects;

s2: processing the sample image to form a data set;

s5: and preprocessing the image to be detected, and then sequentially inputting the preprocessed image into the first defect prediction model and the second defect prediction model to obtain the type, the size and the position of the defect on the image.

Specifically, a proper amount of sample images with various defects (the defects comprise spots, fine scratches and the like) are collected, the sizes of the collected images are unified into a standard size, and the images are processed in a processing mode that the images are subjected to noise reduction by using a median filtering algorithm, and then the image contrast of the images is enhanced, so that the defects are clearer and more visible; expanding the sample, wherein the expansion mode comprises up-down overturning, left-right reversing, translation and rotation; if the fine scratches are the scratches, the specific size of the scratches on the sample is obtained by using an optical microscope, the defect area of the sample picture is marked to form a complete data set, and the neural network is trained by using the manufactured data set; the dual neural networks are adopted, the detection precision is improved, samples needing to be detected are detected through two layers of network models obtained through training, feature extraction is carried out on the samples, whether defects exist or not is determined, and the names and position areas of the defects are marked. The invention combines deep learning with machine vision, has exceptional performance in the aspect of fine defect detection, and has the advantages that through the extraction and learning of a large number of sample characteristics, the label characteristic information obtained by a machine is continuously deepened, the deep learning greatly shortens the process of artificial characteristic extraction of the traditional machine vision, the autonomous learning can be realized for fine scratches with strong randomness, along with the increase of the number of samples, the deep learning can greatly improve the detection efficiency and precision of the fine scratches, and after the detection of the double neural networks, the fine detection result with high confidence coefficient is obtained.

Further, for the processing of the image: the invention uses a median filtering and self-adaptive contrast-limited self-adaptive histogram equalization (CLAHE) algorithm to enhance the image, firstly, a 3X 3 median filtering is adopted to filter the salt-pepper noise in the sample label, then the self-adaptive contrast-limited self-adaptive histogram equalization (CLAHE) is carried out on the filtered image, the CLAHE is different from the common histogram equalization algorithm, the CLAHE algorithm changes the image contrast by calculating the local histogram of the image and then redistributing the brightness, a better enhancement effect can be obtained, the contrast between the defect and the background is improved, and the detection is facilitated.

Further, labeling of the image: the labellimg is adopted to label the sample, so that the labeling task is greatly reduced, the learning model parameters are few, the training time is short, the precision is high, and the false detection rate and the omission factor are much lower than those of the traditional U-net neural network.

Example one

As shown in fig. 2-9, in the present embodiment, based on the above-mentioned detection method, the first neural network uses the yolo-v3 neural network to detect the overall defect on the surface of the industrial product, and the second neural network uses the fast RCNN neural network to further detect the fine scratches on the surface of the industrial product;

as shown in fig. 2 and 3, the first neural network adopts the yolo-v3 neural network to detect the general defects on the surface of the industrial product, and comprises the following steps:

step 1, collecting a proper amount of sample images with various defects (the defects mainly comprise spots and scratches);

step 2, unifying the sizes of the collected images into a standard size of 416 × 416;

step 3, processing the image, wherein the processing mode comprises the steps of firstly using a median filtering algorithm to reduce noise of the image, and then carrying out image contrast enhancement on the image so as to enable the defect to be more clear and visible;

step 4, expanding the sample, wherein the expansion mode comprises up-down turning, left-right turning, translation and rotation;

step 5, making a data set;

step 6, training a yolo-v3 neural network by using the manufactured data set;

and 7, detecting a sample needing to be detected through the trained yolo-v3 network model, extracting the characteristics of the sample, determining whether the sample has a defect, and marking the name and the position area of the defect.

Specifically, as shown in fig. 2, the model training process: collecting a large number of training samples with different defects; standardizing the collected sample icons to be 416 x 416 in size; preprocessing the image, and denoising the image by using median filtering, wherein the step can effectively filter the salt-pepper noise of the image, and then the contrast-limiting self-adaptive histogram equalization is used for enhancing the image, so that the defects are more prominent; expanding a sample, and translating, overturning and rotating a sample image, wherein the step is to increase the scale of a data set and simultaneously prevent overfitting of a model during training so that the training effect is better; making a data set; inputting the marked data set into a yolo-v3 neural network for training; and obtaining a first prediction weight of defect classification detection.

As shown in fig. 3, during the inspection process, images are collected in real time using a 1200 ten thousand pixel CCD detection array; standardizing the collected images to be 416 × 416 size; preprocessing the acquired image, performing median filtering and then enhancing the contrast, and improving the accuracy of prediction; loading a first prediction weight obtained by training, namely a trained neural network, detecting, and marking out the type, position and confidence coefficient of the detected defect; and detecting the next picture until the detection is finished.

The detection results are shown in fig. 4 and 5 (scratch, stand, and last are label names, and numbers such as 0.85 represent confidence). It can be seen that the confidence result of the detection is low at this time, the scratch result in the defect is not obvious, and further detection of the defect result is required.

As shown in fig. 6 and 7, the second neural network further detects the fine scratch process by using the fast RCNN neural network, and comprises the following steps:

step 1, collecting a proper amount of sample images with fine scratch defects;

step 3, processing the image, wherein the processing mode comprises the steps of firstly using a median filtering algorithm to reduce noise of the image, and then carrying out image contrast enhancement on the image so as to enable scratches to be clearer and more visible;

step 5, acquiring the specific size of the scratch on the sample by using an optical microscope, referring to fig. 8, marking the defect area of the sample picture, and forming a complete data set;

step 6, training a Faster RCNN neural network by adopting the scale data set;

and 7, detecting a detection result picture of the yolo-v3 network model needing to be detected through the fast RCNN network model obtained through training, extracting the characteristics of the picture, and determining the position of the final scratch.

Specifically, as shown in fig. 6, the model training process: collecting a large number of training samples with different ultra-fine scratches; standardizing the collected sample icons to be 416 x 416 in size; preprocessing the image, and denoising the image by using median filtering, wherein the step can effectively filter the salt-pepper noise of the image, and then the contrast-limiting self-adaptive histogram equalization is used for enhancing the image, so that scratches are more prominent; expanding the sample, and translating, overturning and rotating the sample image, wherein the step is to increase the scale of the data set and ensure that the training effect is better; acquiring the specific size of a scratch on a sample by using an optical microscope, manually marking the defect area and size of a sample picture, and taking the marked image as a data set for next training; inputting the marked data set into a fast RCNN neural network for training; a predictive weight for the detection of the micro scratch defects is obtained.

As shown in fig. 7, in the detection process, a first recurrent neural network detection result image is input for detection, and the specific position of the detected scratch is marked; and detecting the next picture until the detection is finished.

When the method for detecting the ultra-fine scratches described in this embodiment is used for detecting scratches on the surface of a glass screen, the detection result is shown in fig. 9 (scratch is a label name, and numbers such as 0.95 represent confidence), and the confidence is high.

Further, in the yolo-v3 neural network used in the training of the present embodiment, each training sample is divided into S × S grids, each grid generates a plurality of rectangular frames that may contain defects, the position of the label on each training sample is composed of six parameters, which are w, h, x, y, conf, respectively, where w, h are the width and height of the prior frame, x, y are the adjustment parameters of the center position of the prior frame, and conf is the confidence that whether the prior frame contains defects. The model comprises a trunk feature extraction network of Darknet-53, 25 convolutional layers and two full-connection layers, the Darknet-53 uses a Residual error network Residual, the Residual error convolution in the Darknet53 is to firstly carry out convolution with convolution kernel size of 3 × 3 and step length of 2, the feature layers are compressed through the convolution, and the compressed feature layers are named layer. And then, performing one convolution of 1 × 1 and one convolution of 3 × 3 on the layer feature layer, and adding the obtained new feature layer and the layer to form a residual error structure. The residual network is characterized by easy optimization and can improve accuracy by adding considerable depth. In the yolo-v3 neural network, the idea of transfer learning is introduced, and the model is pre-trained, so that the convergence speed of the model can be increased, and the training efficiency and accuracy can be improved; the Darknet-53 backbone extracts a special Draknet Conv2D structure adopted by each convolution part in the network, L2 regularization is carried out during each convolution, and Batchnormalization standardization and LeakyReLU are carried out after the convolution is completed. The normal ReLu is to set all negative values to 0, the leakyreu is to add a non-zero slope to all negative values, and the mathematical expression is:

in the embodiment, images are processed by adopting median filtering and adaptive contrast limiting adaptive histogram equalization, so that the contrast of defects is improved; the yolo-v3 neural network is adopted to classify and detect the defects, and compared with the traditional semantic segmentation network, the method has higher precision and higher speed; and the idea of transfer learning is introduced to pre-train the model, so that the training efficiency is improved.

Furthermore, a fast RCNN neural network can be trained by utilizing manually marked data to obtain a superfine scratch target detection model; initializing training parameter setting, taking RGB (Red, Green and Blue) three-channel images with the input image size unified as 800 × 800 in the step 2 as the input of a Faster RCNN neural network, and meanwhile, setting the initial learning rate of the Faster RCNN neural network to be 0.0001, the initial iteration times to be 50 and the batch processing data volume to be 8; during training, 90% of all input data is randomly selected for training and 10% is selected for verification.

Furthermore, by utilizing the idea of transfer learning, the fast RCNN neural network is pre-trained on the known large data set Imagenet to obtain pre-training weights, and the pre-training weights are loaded into the fast RCNN neural network used by the invention, so that the training time is greatly reduced, and the training precision is improved.

Furthermore, the Faster RCNN neural network adopts ResNet50 with a residual error structure as a main feature extraction network to extract the image features of fine scratches in a data set through the network, the residual error structure can directly skip a certain layer of data output of a plurality of layers in front and introduce the data output to an input part of a later data layer, and the problems that the learning efficiency is low and the accuracy cannot be effectively improved due to the deepening of the network depth can be effectively solved; the specific structure of ResNet50 is as follows: 1 Conv2d convolutional layer, 1 batch normalization layer (BN),1 ReLU function layer, 1 max pooling layer, 4 convolutional blocks (Conv Block),12 Identity blocks (Identity Block),1 average pooling layer and 1 fully connected layer; conv Block functions to change the dimensionality of the network, and Identity Block functions to deepen the network. The RPN is directly used for generating the detection frame, the generation speed of the detection frame can be greatly improved, the RPN takes a common feature map obtained by a series of convolution operations as input, then the common feature map is firstly subjected to one convolution by 3 times and then subjected to two convolutions by 1 times, a preselected frame is obtained by post-processing, then the preselected frame and the original feature map are input into an ROI pooling layer together, and finally the category to which each target belongs is calculated through a full-connection layer and a softmax function.

In the embodiment, the image is equalized by adopting median filtering and self-adaptive contrast-limiting self-adaptive histogram, so that the contrast of scratches is improved; the extremely fine scratches are detected by adopting a fast RCNN neural network, so that the accuracy is high; the labellimg is adopted to label the sample, so that the labeling task is greatly reduced, the learning model parameters are few, the training time is short, the precision is high, and the false detection rate and the omission factor are much lower than those of the traditional U-net neural network.

Example two

On the basis of the defect detection method, a collecting device is set up and used for collecting sample images with defects and collecting images to be detected in real time;

As shown in fig. 10, in particular, the system includes an image acquisition platform:

the acquisition device comprises: the detector comprises a 1200 ten thousand pixel CCD detection array, wherein the 1200 ten thousand pixel CCD detection array is used for acquiring images;

a square LED illumination source for providing high intensity and uniform light to the sample;

the light source regulator is used for regulating the brightness of the light source to obtain the most appropriate light intensity;

the lens with the focal length of 25mm is matched with the detection array and can be used for detecting the defects with the width less than twenty microns;

the FPGA-based image acquisition system and the detection algorithm implementation platform are used for connecting the detection array to receive image data and process a detection image;

when the device is used, a sample to be detected is placed at a fixed position above a light source, the camera has 1200 ten thousand pixels, the focal length of the lens is 25mm, the distance from the lens to the sample is 16cm, the defect that the width in a rectangular area of 40 cm-50 cm is smaller than twenty micrometers can be detected, the precision is guaranteed, and the detection efficiency is also guaranteed.

The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitution or change made by the technical personnel in the technical field on the basis of the invention is all within the protection scope of the invention. The protection scope of the invention is subject to the claims.

Claims

1. A product surface defect detection method based on a dual neural network is characterized in that: the method comprises the following steps:

s1: collecting a sample image with defects;

s2: processing the sample image to form a data set;

2. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 1, wherein: the processing of the sample image in the step S2 includes the following steps:

s21: unifying the size of the sample image;

s22: reducing noise of the sample image by adopting a median filtering algorithm, and enhancing the image by calculating a local histogram of the image and redistributing brightness to change the contrast of the image by using a self-adaptive contrast-limiting self-adaptive histogram equalization algorithm so as to make defects in the sample image more prominent;

3. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 1, wherein: in the step S3, the first neural network is trained by using a yolo-v3 neural network to obtain a first defect prediction model, the yolo-v3 neural network includes a Darknet-53 trunk feature extraction network, 25 convolutional layers and two full-link layers, wherein Darknet-53 uses a Residual error network Residual.

4. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 3, wherein: before the yolo-v3 neural network is trained, dividing each sample picture in a trained data set into S × S grids, generating a plurality of rectangular frames which may contain defects by each grid, wherein the position of a label on each sample picture is composed of six parameters which are respectively w, h, x, y and conf, wherein w and h are the width and height of each rectangular frame, x and y are adjusting parameters of the center position of each rectangular frame, and conf is the confidence coefficient of whether each rectangular frame contains defects.

5. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 4, wherein: each convolution part in the Darknet-53 trunk feature extraction network adopts a DraknetConv2D structure, L2 regularization is carried out when the DraknetConv2D structure is convoluted each time, Batchnormalization standardization and LeakyReLU are carried out after the convolution is completed, and LeakyReLU is a slope which adds a nonzero slope to all negative values.

6. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 1, wherein: in the step S3, the second neural network is trained by using the fast RCNN neural network to obtain a second defect prediction model, which includes the following steps:

7. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 6, wherein: the ResNet50 comprises a Conv2d convolutional layer, a batch normalization layer, a ReLU function layer, a maximum pooling layer, 4 convolutional blocks, 12 identification blocks, an average pooling layer and a full connection layer which are arranged in sequence.

8. The method for detecting the surface defects of the product based on the dual neural network as claimed in claim 6, wherein: pre-training the fast RCNN neural network by using the Imagenet data set, and using the pre-trained fast RCNN neural network for the training in the step S3.

9. A product surface defect detection system based on a dual neural network is characterized in that: the method comprises the following steps:

10. The system of claim 9, wherein the dual neural network based product surface defect detection system comprises: the acquisition device comprises a 1200 ten thousand pixel CCD detection array, the 1200 ten thousand pixel CCD detection array is provided with a 25mm focal length lens, and the data processing module, the neural network module and the detection module are all arranged in the FPGA.