CN113066075B - A denim defect detection method and device based on multi-image fusion - Google Patents

Abstract

The invention relates to a method and a device for detecting denim flaws with multi-image fusion. The method includes collecting denim image data through a cloth inspection mechanism, constructing a data set of denim image data, and manually marking the denim cloth through a marking mechanism. Image data with flaws in the image data, the data set of the denim image data includes a front light source image and a back light source image, and then multiple sets of the denim images are preprocessed, and MinPooling enhancement and difference combination three channels are performed Processing, finally using the output data to train the neural network, processing the ROI selected by the neural network training with the OHEM algorithm, and finally using the neural network that has completed the training to detect the denim, and detect the flaws. Denim is automatically marked. The invention integrates the template image, the front light source image and the back light source image, so that the features of defects are more abundant and prominent, and the recognition accuracy is improved based on the deep learning network.

Description

A multi-image fusion denim defect detection method and device

technical field

The invention relates to the field of intelligent detection, in particular to a multi-image fusion denim defect detection method and device.

Background technique

The detection of denim defects is generally carried out manually. Manual detection has the problems of slow detection speed, high rate of missed detection and false detection, and difficult to unify the detection standards. It cannot meet the mass production requirements in the industry.

The existing denim defect detection algorithm based on deep learning, such as "A method for intelligent recognition of cloth defects based on deep learning" in the publication number CN111062925A, is based on the improved SSD model, which reduces the amount of model calculation and improves the detection speed. However, the denim images collected in reality have the problem of indistinct features. A single defect image is difficult to accurately identify defects;

Another example is "A Method, System and Equipment for Cloth Defect Detection Based on Gabor Filtering and CNN" of Publication No. CN111398292A, which uses the Gabor filter to extract the texture features of the image after preprocessing the collected cloth images, and then sends the texture features to the Enter the CNN network for classification, which can quickly identify whether it is a normal cloth texture or a defect. This method is still insufficient in the feature extraction of denim, resulting in inaccurate screening;

Therefore, there is an urgent need for a fast and accurate automatic denim defect detection method for mass production of denim.

Contents of the invention

In order to effectively solve the problems of low accuracy and slow speed of existing denim defect detection, the present invention provides a multi-image fusion denim defect detection method and device, which integrates template images, front light source images and back light source images, The feature of the defect is richer and more prominent, and the accuracy of recognition is improved based on the deep learning network, so as to realize the rapid and accurate detection of denim defects.

In order to achieve the above object, the first aspect of the present invention proposes a multi-image fusion denim defect detection method, the method includes the following steps:

Step 1: collect denim image data through the cloth inspection mechanism, construct a data set of denim image data, and manually mark the defective image data in the denim image data through the marking mechanism;

Step 2: Define the image obtained by the shooting mechanism under the action of the light source mechanism on the front of the body as the front light source image;

The image obtained by the shooting mechanism under the action of the light source mechanism on the back of the body is the image of the back light source;

Forming a group of denim images in one-to-one correspondence with the front light source image and the back light source image;

Step 3: preprocessing multiple groups of denim images to expand the data set to obtain a balanced data set;

Step 4: The balanced data set is subjected to three-channel processing of MinPooling strengthening and difference combination, and the neural network is trained using the balanced data set processed through MinPooling strengthening and difference combination three-channel;

Step 5: process the ROI selected by the neural network training with the OHEM algorithm;

Step 6: Use the trained neural network to detect the denim, and automatically mark the denim with defects detected.

Further, the preprocessing described in step 3 includes:

Step 3.1: Carry out weighted averaging on the denim image by Gaussian filtering to obtain a data set that eliminates Gaussian noise;

Step 3.2: Equalize the denim image to enhance image contrast;

Step 3.3: expand the highlighted area or white part of the denim image through expansion;

Step 3.4: Perform edge padding on the denim image, set the pixel value to 0 at the edge of the image, and fill it with a black background;

Random cropping is performed on the denim picture, and image frame removal processing is performed on the randomly cropped denim picture.

Further, the MinPooling enhancement described in step 4 includes:

Set the pixel value of the denim image to a negative value, and perform maximum pooling on the denim image, and then set the pixel value of the denim image negative again to strengthen the denim detail texture in the denim image.

Further, the difference combination three-channel processing described in step 4 includes:

Step 4.2.1: Set the denim template image;

Step 4.2.2: setting a fused image, the fused image has three channels;

The first channel is the denim image to be tested;

The second channel is the denim template image;

The third channel is a difference map obtained by using a weighted difference operation on the pixel matrix of the first channel and the second channel;

Step 4.2.3: Input the fused image into the neural network.

Further, the neural network described in step 4 includes two parallel feature extraction network models based on Faster-RCNN;

The collected front light source image and back light source image are used as input respectively, and ResNeXt50 and ResNet50 are respectively used as the backbone respectively, and deformable convolution is added, and then PositionEncoding is used to encode the position information of the feature;

Two groups of features are extracted, and the two groups of features are correspondingly connected to form an output feature vector of the feature extraction network.

The second aspect of the present invention proposes a multi-image fusion denim flaw detection device, which includes a body, a support mechanism, a rotation mechanism, a motor and a transmission device, and is characterized in that the support mechanism is arranged on the body, and the support mechanism includes Support board for placing denim;

The motor is arranged inside the body, the motor is connected to the rotating mechanism through a transmission device, and drives the rotating mechanism to rotate, and the rotating mechanism is fixed on the upper part of the supporting mechanism;

The upper part of the body is also provided with a cloth inspection mechanism, which includes a camera, a light source, and a marking mechanism. The shooting mechanism provides sufficient light, and the marking mechanism is used to mark the image data with defects.

Through the above technical scheme, the beneficial effects of the present invention are:

The present invention first collects denim image data through a cloth inspection mechanism, constructs a data set of denim image data, and manually marks defective image data in the denim image data through a marking mechanism, and the data set of the denim image data Including a front light source image and a back light source image, forming a set of denim images in one-to-one correspondence between the front light source images and the back light source images, and then performing preprocessing on multiple sets of the denim images to expand the data set to obtain a balance data set;

In order to strengthen the detailed texture of denim, the balanced data set is subjected to three-channel processing of MinPooling enhancement and difference combination, and the neural network is trained by using the balanced data set processed by MinPooling enhancement and difference combination three-channel, and the neural network is trained. The ROI selected by the network training is processed with the OHEM algorithm, and finally the trained neural network is used to detect the denim and automatically mark the denim with defects detected.

The present invention fuses the front light source image and the back light source image based on the deep learning method, so that the detection model can obtain more dimensional information input, which is of great significance for the multi-dimensional recognition of defect features, and the fusion of template information and defect information is helpful for improving defects The detection accuracy is of great significance. The Loss function has been optimized, which has greatly improved the recognition of small defects.

Description of drawings

Fig. 1 is the flow chart of a kind of multi-image fusion denim flaw detection method of the present invention;

Fig. 2 is a network structure diagram of a denim defect detection method of multi-image fusion in the present invention;

Fig. 3 is a schematic structural diagram of a multi-image fusion denim defect detection device according to the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the present invention Examples, not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Example 1

As shown in Figure 1, a kind of denim defect detection method of multi-image fusion, described method comprises:

Step 1: collect denim image data through the cloth inspection mechanism, construct a data set of denim image data, and manually mark the defective image data in the denim image data through the marking mechanism;

Step 2: Define the image obtained by the shooting mechanism under the action of the light source mechanism on the front of the body as the front light source image;

The image obtained by the shooting mechanism under the action of the light source mechanism on the back of the body is the image of the back light source;

Forming a group of denim images in one-to-one correspondence with the front light source image and the back light source image;

Step 3: preprocessing multiple groups of denim images to expand the data set to obtain a balanced data set;

Step 4: The balanced data set is subjected to three-channel processing of MinPooling strengthening and difference combination, and the neural network is trained using the balanced data set processed through MinPooling strengthening and difference combination three-channel;

Step 5: process the ROI selected by the neural network training with the OHEM algorithm;

Step 6: Use the trained neural network to detect the denim, and automatically mark the denim with defects detected.

In the present invention, the front light source image and the back light source image are introduced to increase the multi-dimensional information of image recognition, and at the same time, the balanced data set of MinPooling strengthening and difference combination three-channel processing is used to train the neural network, which improves the feature extraction ability of the model and makes the cowboy The detection results of cloth defects are more accurate, especially in the display of small defect features.

Example 2

Because the image acquisition cost of denim defects is relatively high and difficult to obtain, so based on the above-mentioned embodiment 1, the data set is expanded by preprocessing in this embodiment, specifically:

Step 3.1: Weighted average the denim image through Gaussian filtering to obtain a data set that eliminates Gaussian noise; Gaussian filtering can weight the entire image, that is, the value of each pixel is determined by itself and other pixels in the neighborhood The value is obtained by weighted average;

Step 3.2: Equalize the denim image to enhance the image contrast, making the denim image clearer than the original image, and increasing the number of data sets to train the program.

Step 3.3: Expand the highlighted area or white part of the denim image by dilation; make the running result image larger than the highlighted area of the original image.

Step 3.4: Perform edge padding on the denim image, set the pixel value to 0 at the edge of the image, and fill it with a black background;

Random cropping is performed on the denim picture, and image frame removal processing is performed on the randomly cropped denim picture.

Through the above steps, without affecting the characteristics of the image itself, not only the accuracy of the model can be improved, but also the robustness of the model can be enhanced, and the image can be flipped at random angles to enhance the scale invariance and direction invariance of the model. The ability of the model for image recognition.

Example 3

Based on the above-mentioned embodiment 1, in order to strengthen the denim small defect feature display ability and realize the denim small defect detection, step 4 is optimized in this embodiment, specifically:

The MinPooling enhancement described in step 4 includes:

Set the pixel value of the denim image to a negative value, and perform maximum pooling on the denim image, and then set the pixel value of the denim image negative again to strengthen the denim detail texture in the denim image.

As an implementable manner, the three-channel processing of the difference combination includes:

Step 4.2.1: Set the denim template image;

Step 4.2.2: setting a fused image, the fused image has three channels;

The first channel is the denim image to be tested;

The second channel is the denim template image;

The third channel is a difference map obtained by using a weighted difference operation on the pixel matrix of the first channel and the second channel;

Step 4.2.3: Input the fused image into the neural network.

Example 4

Based on the above-mentioned multiple embodiments, step 5 is optimized for constructing a neural network, specifically:

Neural network described in step 4 comprises two parallel feature extraction network models based on Faster-RCNN;

The collected front light source image and back light source image are used as input respectively, and ResNeXt50 and ResNet50 are respectively used as the backbone respectively, and deformable convolution is added, and then PositionEncoding is used to encode the position information of the feature;

Two groups of features are extracted, and the two groups of features are correspondingly connected to form an output feature vector of the feature extraction network.

In this embodiment, as shown in Figure 2, the candidate frame and the picture in the training are used as input, and two identical RoI networks are established for the ROI network layer therein, one of which is only readable, and the other is readable and writable. The readable network performs forward calculation for all RoIs, and the readable and writable network performs not only forward calculation but also backpropagation calculation for the selected hardRoIs;

The network parameters are then updated using stochastic gradient descent.

The Faster R-CNN mainly includes the following steps:

Step S01: Input the picture into the convolutional neural network (CNN) to obtain a feature map;

Step S02: Input the extracted convolution features into the region generation network (RPN) to obtain the feature information of the candidate frame;

Step S03: Perform pooling (RoI pooling) on the feature information extracted from the candidate frame to obtain a fixed-size image;

Step S04: For the candidate frame of a specific category, use a classifier to determine whether it belongs to a certain category, and use a regressor to further adjust the position information;

Wherein, the step S01 is specifically:

Step S01.1: Scale the image to a fixed size of M×N, and then send the M×N image to the convolutional layer (Conv Layers);

Step S01.2: Get the details of the feature map (Feature map):

Use ResNet50 and ResNeXt50 to extract features respectively.

ResNet50 includes 5 levels corresponding to Conv1 to Conv5, which first passes through a Conv1 layer (kernel_size (length of the convolution kernel) = 7, pad (filling pixels) = 3, stride (step size) = 2), and then passes through a pooling layer , and finally output the feature map through the Bottleneck (bottleneck layer) from Conv2 to Conv5.

The Bottleneck (bottleneck layer) introduced above is a network structure. The 4 layers of ResNet50 (Conv2 to Conv5) all contain this structure. Each Bottleneck consists of 1×1, 3×3, 1×1 sequential volumes The number of bottlenecks in different layers is different, and the total number of bottlenecks is also different. Conv2 to Conv5 contain 3, 4, 6, and 3 bottlenecks respectively.

The network structure of ResNeXt50 is similar to that of ResNet50, the difference is that the second convolution of each Bottleneck is replaced by group convolution;

The step S02 is specifically:

Step S02.1: First, the feature map undergoes a (3×3) convolution to obtain a (256×M×M) feature vector

Step S02.2: Convolve the obtained (256×M×M) feature vectors twice (1×1) to obtain a (18×M×M) feature map and a (36×M ×M) feature map, corresponding to (M×M×9) results, and each result contains 2 scores (foreground and background scores) and 4 coordinates (offset for the original image coordinates), and 9 anchor boxes (anchors) corresponding to each point of the input feature map;

Anchor boxes (anchors) are introduced in step S02.2, which is a set of generated rectangular boxes, where each anchor box (anchors) has 4 values (x ₁ , y ₁ , x ₂ , y ₂ ) respectively Represents the coordinates of the upper left and lower right corners of the rectangular frame, and the rectangular frame can be divided into three types with a ratio of 1:1, 1:2, and 2:1. By setting anchor frames (anchors), determine which anchor frames (anchors) are positive anchor boxes (positive anchor), which are negative anchor boxes (negative anchor).

Step S02.3: Input the obtained (18×M×M) feature map to the reshape layer (reshape layer), and then input it to the classification layer (softmax layer) to obtain the classified image, and finally input the reshape layer The matrix layer (reshape layer) gets the candidate area (proposal)

Given anchor S=(S _x , S _y , S _w , S _h ), through translation and scaling operations (d _x (S), d _y (S), d _w (S), d _h (S)) make The regression window (G′) is closer to the real window (G)

panning:

G′ _X ＝S _W ·d _x (S)+S _X

G′ _y =S _h d _y (S)+S _y

Zoom:

G′ _w ＝ S _W exp(d _w (S))

G′ _y =S _h exp(d _h (S))

d _x (S), d _y (S), d _w (S), d _h (S) are obtained through linear regression, and the objective function is shown in formula (1):

是anthors的特征图(Feature map)所组成的特征向量；Among them, d _* (S) is the predicted value, W _* is the parameter of learning,

Is the feature vector composed of the feature map (Feature map) of anthors;

The L1 loss function is shown in formula (2):

Step S02.6: generate anchor boxes (anchors), and perform bounding box regression on all anchor boxes (anchors);

Step S02.7: Sort the anchor boxes (anchors) from large to small for the input positive sample softmax regression scores (positive softmax scores), and extract the positive anchor boxes (positive anchors) after the corrected position;

Step S02.8: Take the positive anchors beyond the boundary as the image boundary, and remove the very small positive anchors;

Step S02.9: Perform Non-Maximum Suppression on the remaining positive anchors;

Step S02.10: output the result of the corresponding classifier as a proposal;

The step S03 is specifically:

Step S03.1: Map the candidate area (proposals) of size M×N back to size (M/16)×(N/16);

Step S03.2: Divide the feature map region corresponding to each candidate region (proposal) into a network of pooling width (pooled_w)×pooling height (pooled_h) horizontally;

Step S03.3: Perform max pooling processing on each part of the grid, and output pooled_w×pooled_h size;

The step S04 is specifically embodied as:

Step S04.1: Input the candidate region feature maps (proposal Feature maps) into the network, and classify the selected regions (proposals) through full connection and softmax;

Step S04.2: Perform frame regression/bounding box regression (bounding box regression) on candidate regions (proposals) to obtain high-precision target detection frames.

Example 5

The embodiment of the present invention provides a multi-image fusion denim defect detection device, as shown in Figure 3, including a body, a support mechanism, a rotation mechanism, a motor, and a transmission device. It is characterized in that the support mechanism is arranged on the body to support Mechanism includes a support plate for placing denim;

The motor is arranged inside the body, the motor is connected to the rotating mechanism through a transmission device, and drives the rotating mechanism to rotate, and the rotating mechanism is fixed on the upper part of the supporting mechanism;

The upper part of the body is also provided with a cloth inspection mechanism, which includes a camera, a light source, and a marking mechanism. The shooting mechanism provides sufficient light, and the marking mechanism is used to mark the image data with defects.

The above-described embodiments are only preferred embodiments of the present invention, and do not limit the scope of the present invention. Therefore, all equivalent changes or modifications made according to the structure, features and principles described in the patent scope of the present invention are valid. Should be included in the patent scope of the present invention.

Claims (3)

1. A method for detecting defects of multi-image fused denim is characterized by comprising the following steps:

step 1: collecting denim layout image data through a cloth inspecting mechanism, constructing a data set of the denim layout image data, and manually marking defective image data in the denim layout image data through a marking mechanism;

and 2, step: defining an image obtained by a shooting mechanism under the action of a light source mechanism on the front side of the machine body as a front light source image;

the image obtained by the shooting mechanism under the action of the light source mechanism on the back of the machine body is a back light source image;

the front light source image and the back light source image correspond to each other one by one to form a group of denim images;

and step 3: preprocessing a plurality of groups of the denim layout images to expand the data set to obtain a balanced data set;

and 4, step 4: subjecting the equalized data set toMinPoolingThree-channel processing of strengthening and difference combination is adoptedMinPoolingTraining a neural network by using a balanced data set processed by combining the strengthening channel and the difference value with three channels;

and 5: selected for training the neural networkROIBy usingOHEMProcessing an algorithm;

step 6: detecting the denim by using the trained neural network, and automatically marking the denim with the detected flaws;

step 3, the pretreatment comprises the following steps:

step 3.1: carrying out weighted average on the denim image through Gaussian filtering to obtain a data set for eliminating Gaussian noise;

step 3.2: equalizing the denim layout image to enhance the image contrast;

step 3.3: expanding the highlight area or the white part in the jean layout image through expansion;

step 3.4: performing edging on the jean picturepaddingProcessing, setting the pixel value to 0 at the edge of the image, and filling the pixel value as a black background;

randomly cutting the jean picture, and removing an image frame of the jean picture after random cutting;

step 4 is as described aboveMinPoolingThe strengthening comprises the following steps:

setting the pixel value of the denim picture as a negative value, performing maximum pooling on the denim picture, and further taking the pixel value of the denim picture as a negative value again to strengthen the denim detail texture in the denim picture;

the processing of the difference combined three channels in the step 4 comprises the following steps:

step 4.2.1: setting a denim template image;

step 4.2.2: setting a fused image, wherein the fused image has three channels;

the first channel is a denim picture to be detected;

the second channel is a denim template image;

the third channel is a difference image obtained by performing weighted difference operation on the pixel matrix of the first channel and the pixel matrix of the second channel;

step 4.2.3: and inputting the fused image into a neural network.

2. The method for detecting the defects of the denim fabric with the multi-image fusion function according to claim 1, wherein the neural network of the step 4 comprises two parallel denim fabric defects based on the multi-image fusion functionFaster-RCNNExtracting a network model;

respectively taking the collected front light source image and back light source image as input, and respectively correspondingly takingResNeXt50、ResNet50AsbackboneAnd adding a deformable convolution for reusePositionEncoding, carrying out position information coding on the features;

and extracting two groups of features, and correspondingly connecting the two groups of features to form an output feature vector of the feature extraction network.

3. The denim flaw detection device based on the multi-image fusion denim flaw detection method of any one of claims 1 to 2, characterized by comprising a machine body, a supporting mechanism, a rotating mechanism, a motor and a transmission device, wherein the supporting mechanism is arranged on the machine body and comprises a supporting plate for placing denim;

the motor is arranged in the machine body, is connected with the rotating mechanism through a transmission device and drives the rotating mechanism to rotate, and the rotating mechanism is fixed at the upper part of the supporting mechanism;

the cloth inspecting mechanism is further arranged on the upper portion of the machine body and comprises a shooting mechanism, a light source mechanism and a marking mechanism, the shooting mechanism is used for establishing image data, the light source mechanism is respectively arranged on the front side and the back side of the machine body, the light source mechanism is used for providing sufficient illumination for the shooting mechanism, and the marking mechanism is used for marking the defective image data.

Images