CN111161243A - Surface defect detection method for industrial products based on sample enhancement - Google Patents

Surface defect detection method for industrial products based on sample enhancement Download PDF

Info

Publication number
CN111161243A
CN111161243A CN201911390407.9A CN201911390407A CN111161243A CN 111161243 A CN111161243 A CN 111161243A CN 201911390407 A CN201911390407 A CN 201911390407A CN 111161243 A CN111161243 A CN 111161243A
Authority
CN
China
Prior art keywords
defect
picture
image
new
rectangular
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911390407.9A
Other languages
Chinese (zh)
Other versions
CN111161243B (en
Inventor
许玉格
郭子兴
戴诗陆
吴宗泽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN201911390407.9A priority Critical patent/CN111161243B/en
Publication of CN111161243A publication Critical patent/CN111161243A/en
Application granted granted Critical
Publication of CN111161243B publication Critical patent/CN111161243B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

本发明公开了一种基于样本增强的工业产品表面缺陷检测方法，包括步骤：1)对工业产品表面图像进行尺寸标准化、归一化，切割并分类；2)带缺陷的图片进行随机翻转的数据增强；3)将带缺陷图片和正常图片进行随机拼接增强；4)使用Cascade‑RCNN算法进行迭代训练；5)得到Cascade‑RCNN检测模型；6)通过Cascade‑RCNN检测模型对需要检测的工业产品表面图片和确定无缺陷的纹理模板图片进行滑窗检测，对滑窗检测到的结果进行拼接，并对二者得到的结果进行比较，最终得到待检测图片的缺陷类别及区域标注。本发明可有效降低光照、曝光和位移等条件对缺陷检测的影响，提高了检测稳定性，同时提高了二阶段目标检测器对花纹和背景的分辨能力，降低了误检率。

Figure 201911390407

The invention discloses a method for detecting surface defects of industrial products based on sample enhancement. Enhancement; 3) Randomly splicing and enhancing images with defects and normal images; 4) Iterative training using Cascade-RCNN algorithm; 5) Obtaining Cascade-RCNN detection model; 6) Using Cascade-RCNN detection model to detect industrial products to be detected The surface image and the texture template image that is determined to be defect-free are subjected to sliding window detection, the results detected by the sliding window are spliced, and the results obtained by the two are compared, and finally the defect category and area annotation of the image to be detected are obtained. The invention can effectively reduce the influence of illumination, exposure, displacement and other conditions on defect detection, improve the detection stability, and at the same time improve the distinguishing ability of the two-stage target detector for patterns and backgrounds, and reduce the false detection rate.

Figure 201911390407

Description

Industrial product surface defect detection method based on sample enhancement
Technical Field
The invention relates to the technical field of industrial product surface defect detection, in particular to a sample enhancement-based industrial product surface defect detection method.
Background
Defect detection is an important part of the production process, and ensures the reliability of industrial products. Surface defect detection of industrial products requires precise positioning of defect positions on a surface and classification of the positioned defects, which is a typical target detection problem. In the past, the surface defect detection technology of industrial products generally uses the traditional machine vision technology to perform operations such as picture gray level binarization, edge contour extraction, template matching and the like, and the defects of the operations are that the operations are very sensitive to changes such as illumination, displacement and the like of pictures and the robustness is poor. In addition, previous surface defect detection studies were based on solid color product surfaces, and since the surface texture features of specially textured products and the defect texture features are similar, it is difficult to distinguish between the two in previous methods.
The target detection in deep learning is realized by taking a convolutional neural network as a feature extractor, and the extracted feature graph is insensitive to changes such as illumination, displacement and the like and has better robustness. A two-stage target detector is composed of a Region Proposal Network (Region Proposal Network) and a classification regression Network, wherein the Region Proposal Network is responsible for generating suggestions of regions where targets may be located, and the classification regression Network classifies the suggested regions and finely adjusts a labeling frame. The function of the network consists of classification loss and regression loss weighting, and a random gradient descent method is adopted for back propagation iteration.
The existing two-stage deep learning target detector has high precision and good universality, but the problems that defects and background textures are difficult to distinguish, normal pictures without the defects cannot participate in model training, pictures of industrial products have high video memory requirements and the like still exist in the surface defect detection with the textures.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a sample enhancement-based industrial product surface defect detection method, can effectively reduce the influence of conditions such as illumination, exposure and displacement on defect detection, improves the detection stability, improves the resolution of a two-stage target detector on patterns and backgrounds, and reduces the false detection rate.
In order to achieve the purpose, the technical scheme provided by the invention is as follows: the industrial product surface defect detection method based on sample enhancement comprises the following steps:
1) carrying out size standardization operation on a picture set on the surface of an industrial product, wherein the picture containing the defects has a corresponding defect marking file, carrying out cutting operation on the defect picture and the defect marking file corresponding to each picture, and dividing the defect picture and the defect marking file into a normal picture set and a defect picture set according to the cut marks;
2) normalizing and enhancing online random data of the defect picture set obtained in the step 1), including randomly turning the defect picture set up and down, left and right, and dividing the defect picture set into batches;
3) for each defective picture in the batch in the step 2), randomly searching a normal picture which corresponds to the defective picture and has the same cutting position and the same texture template pattern in the normal picture set, performing left or right splicing operation on the normal picture and the defective picture, and correspondingly modifying the label file;
4) performing iterative training on the pictures and labels of each batch obtained in the step 3) by using a Cascade-RCNN algorithm, and finishing a round of training after finishing training all batches;
5) after finishing one round of training, repeating the steps 2) to 4) until reaching the set iteration round, outputting and storing parameters in the network to obtain a Cascade-RCNN detection model;
6) and 5) performing sliding window detection on the surface picture of the industrial product to be detected and the determined non-defective texture template picture by using the Cascade-RCNN detection model obtained in the step 5), splicing the results detected by the sliding window, and comparing the results obtained by the sliding window and the determined non-defective texture template picture to obtain the defect type and the region label of the picture to be detected finally.
In step 1), the image set on the surface of the industrial product comprises a template image set Z consisting of a defect-containing image set X, a defect-free normal image set Y and a pattern example image of each texture template, wherein the defect image set X contains labels, and each defect label is a rectangular label frame with a format of (name, category, X)min,ymin,xmax,ymax) Where name represents the picture name, category represents the type of defect, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) The horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, and neither the picture set Y nor the picture set Z has labeling information; carrying out size standardization on the three picture sets to enable all pictures to be RGB pictures of H x W, wherein H and W are the height and width of the pictures;
the same average cutting is carried out on the three picture sets, and the defect marks on the picture set X are cut according to the following rules: cutting the defective rectangular marking frame in the same way as the picture, mapping the defective rectangular marking frame into the range of the cut small picture, if the marking is cut off, calculating the proportion of the area of the cut rectangular marking frame to the area of the original rectangular marking frame, if the area is larger than a set threshold epsilon, keeping the marking, and otherwise, discarding;
dividing the picture into a defect picture set X according to the cut marking informationnewSet of normal pictures YnewAnd template picture set ZnewAnd stored according to the position where they were cut.
The step 3) comprises the following steps:
3.1) for the defect picture x which is randomly flipped in step 2)i∈XnewFinding out corresponding normal picture set M e (Y) according to template texture and cutting positionnew,Znew) So that x isiAnd the texture template and cut position are the same for all samples in M, where XnewFor a set of cut defect pictures, Ynew,ZnewRespectively a normal picture set and a template picture set after cutting;
3.2) randomly selecting a normal picture yiE.g. M, according to xiSize of (a) to yiIs filled with a value of 0, such that yiSize and x ofiAre all the same in size and for yiCarrying out normalization processing;
3.3) to yiCarrying out data enhancement operation, namely randomly turning up, down, left and right;
3.4) generating a random number between (0,1) at 50%Probability is such that xnew＝(xi,yi) Another 50% probability is such that xnew＝(yi,xi) I.e. randomly splicing left or right, where xnewRepresenting the generated new sample;
3.5) processing the marking information according to the splicing mode, if the picture x has defectsiOn the left, there is no need to change the defect label, if the defect picture xiAnd on the right, the rectangular marking box needs to be corrected correspondingly.
In step 4), the Cascade-RCNN algorithm comprises a trunk network, a region proposing network and a classification regression network, which are respectively used for extracting features, generating region suggestions and classifying and fine-tuning candidate frames; the convolutional neural network ResNeXt-101 and the feature pyramid FPN are used as a backbone network, the area proposal network uses an area proposal network part in a two-stage target detector, namely, fast-RCNN, and the classification regression network uses a multilayer cascade network.
In step 6), the following detection process is performed:
6.1) for an industrial product surface picture to be detected, using a preset sliding window size, using the Cascade-RCNN detection model obtained in the step 5) to perform sliding window detection on the picture to be detected, and mapping the result back to the area of the original image to obtain the label format (category, x) of each defectmin,ymin,xmax,ymaxScore), category denotes the defect class, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) The horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, score represents the confidence coefficient of defect judgment, and the value of the confidence coefficient is (0, 1);
6.2) judging the defect labels close to the edges of the sliding windows, if the adjacent sliding windows have labels with the same category and the similar position size, carrying out label merging operation according to the confidence degree sequence, wherein the merged rectangular label frame is the minimum circumscribed rectangle of a plurality of rectangular label frames, and calculating the new confidence degree to obtain the average value of the minimum circumscribed rectangle, which is as follows:
scorenew＝(score1+score2+...+scoren)/n
wherein, scorenewIndicates the new confidence, scoreiRepresenting the confidence of the ith rectangular labeling frame participating in synthesis, and n representing the total number of the rectangular labeling frames participating in synthesis;
6.3) adopting the steps 6.1) and 6.2) to carry out defect detection on the template picture set Z in advance under the offline condition, and storing the obtained result;
6.4) detecting the surface picture of the industrial product to be detected online by adopting the steps 6.1) and 6.2), comparing the obtained result with the detection result of the corresponding template stored in the step 6.3), adopting IoU as a comparison standard, and obtaining the following calculation formula:
Figure BDA0002344783280000051
wherein DR represents a defect rectangular marking frame detected on the to-be-detected image, and GT represents a real defect rectangular marking frame; the specific method for comparison is as follows: and comparing the defect labels belonging to the same category, and if IoU is greater than a set threshold tau and the defect confidence coefficient on the picture to be detected is less than a set threshold gamma, considering the defect labels on the picture to be detected as texture false detection and removing the texture false detection, thereby obtaining the final defect position labels and the corresponding categories of the picture to be detected.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the method adopts deep learning target detection as an integral detection frame, reduces the problem of algorithm quality reduction caused by ambient illumination, camera exposure and displacement, and improves the stability of the algorithm for detecting the surface defects of industrial products.
2. The method provided by the invention cuts data with larger picture size, increases the training data amount, reduces the requirement on video memory in the training process, enables the picture to keep the original size input during training, and is not easy to lose the characteristics of tiny flaws. Meanwhile, the method for detecting the combined defect frame by using the sliding window during detection ensures the uniformity and the integrity of the output of the defect marking frame.
3. In order to improve the difference of the detector on the background patterns and the defects of the texture textiles, in addition to the traditional deep learning data enhancement method, before each iteration of each picture, a mixed splicing mode of the defect pictures and the corresponding normal pictures of the texture patterns is used as an online data enhancement method, so that in the training process, under the condition that the data volume is not increased in a large scale, the diversity of negative samples is enhanced, and the false detection rate of the detector is reduced.
4. The texture template picture is used for pre-detection in the detection process, and the false detection position generated by the texture template is relatively fixed, so that the false detection of small defects generated by the texture template at certain fixed positions on the surface of an industrial product can be eliminated by comparing the defect positions and types of the picture result to be detected and the template result thereof, and the overall identification accuracy is improved.
Drawings
FIG. 1 is a training flow diagram of a sample-based enhanced industrial product surface defect detection method.
FIG. 2 is a defect detection flow chart of a sample enhancement based industrial product surface defect detection method.
Detailed Description
The present invention will be further described with reference to the following specific examples.
The example uses the real collected data of textile pattern with patterns, which includes 15 defects such as stain, stitch mark, hole, etc., the pattern template has 68 kinds, including each template picture, several normal pictures and defect pictures with marks, and the picture size is 4096 x 1810 to 4096 x 1696.
As shown in fig. 1 and fig. 2, the method for detecting surface defects of an industrial product based on sample enhancement provided by the present embodiment includes the following steps:
1) and carrying out size standardization operation on the textile picture set with the patterns, wherein the pictures containing the defects have corresponding defect marking files, cutting the defect pictures and the corresponding defect marks of each picture, and dividing the pictures into a normal picture set and a defect picture set according to the cut marks.
The textile data set with patterns comprises a defect picture set X, a normal picture set Y and a pattern template picture set Z, wherein the defect picture set X contains marks, and each mark of each defect is a rectangular mark frame with a format of (name, category, X)min,ymin,xmax,ymax) Where name represents the picture name, category represents the type of defect, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) And the horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, and the image sets Y and Z have no labeling information. The three sets of pictures were normalized in size so that all pictures were RGB pictures of 4096 x 1810.
The three picture sets are equally cut, and the defect marks on the picture set X are cut according to the following rules: and cutting the defective rectangular marking frame in the same way as the picture, mapping the defective rectangular marking frame into the range of the cut small picture, if the marking is cut off, calculating the proportion of the area of the cut rectangular marking frame to the area of the original rectangular marking frame, if the area is greater than a set threshold value epsilon, namely 0.25, keeping the marking, and otherwise, discarding the marking.
2) Normalizing and enhancing online random data of the defect picture set obtained in the step 1), wherein the normalization and online random data enhancement comprise random turning up and down, left and right and dividing the defect picture set into batches. The method specifically comprises the following steps: the value 0 is filled in the multiple of the length and the width of 32, the pictures are normalized, a series of operations of random inversion is carried out with the probability of 50%, and then batch processing is carried out, so that the training of the network is facilitated, the training batch number used in the example is 1, namely, one picture is obtained in each batch, and the size of the picture is 1024 x 928.
3) And 2) for each defect picture in the batch in the step 2), randomly searching a normal picture which corresponds to the same normal picture and has the same cutting position and the same texture template pattern in the normal picture set, performing left or right splicing operation on the normal picture and the defect picture, and correspondingly modifying the label file.
3.1) for the defect picture x which is randomly flipped in step 2)i∈XnewFinding out the pattern and cutting position according to the templateCorresponding normal picture set M e (Y)new,Znew) So that x isiAnd the texture template and cut position are the same for all samples in M, where XnewFor a set of cut defect pictures, Ynew,ZnewRespectively a normal picture set and a template picture set after cutting.
3.2) randomly selecting a normal picture yiE is M, for yiIs filled with a value of 0, such that yiIs also filled to 1024 x 928 for yiAnd (6) carrying out normalization processing.
3.3) to yiAnd performing data enhancement operation, namely randomly turning up, down, left and right.
3.4) generating a random number between (0,1) such that x is given a probability of 50%new＝(xi,yi) Another 50% probability is such that xnew＝(yi,xi) I.e. randomly splicing left or right, where xnewThe new samples generated are shown, and the new sample size after splicing is 2048 x 928.
3.5) processing the marking information according to the splicing mode, if the picture x has defectsiOn the left, the defect label does not need to be changed, if the defect picture xiOn the right, 1024 is added to both the minimum and maximum of the broadside coordinates in the rectangular box label.
4) And (3) carrying out iterative training on the pictures and labels of each batch obtained in the step 3) by using a Cascade-RCNN algorithm, and finishing a round of training after all batches are trained.
The Cascade-RCNN algorithm comprises a trunk network, a region proposing network and a classification regression network, which are respectively used for extracting features, generating region propositions and classifying and fine-tuning candidate frames. In the invention, a convolutional neural network ResNeXt-101 and a characteristic pyramid FPN are used as a backbone network, a region proposing network in a two-stage target detector, namely, fast-RCNN, is used as a region proposing network, and a multi-layer cascade network is used as a classification regression network. And training all the enhanced pictures for one round, and finishing one round of training.
5) And after finishing one round of training, repeating the steps 2) to 4) until reaching the set iteration round, wherein the set iteration round is 12 in the example, and outputting and storing the parameters in the network to obtain the Cascade-RCNN detection model.
6) And 5) carrying out sliding window detection on the textile picture to be detected and the pattern template picture determined to be free of defects by using the Cascade-RCNN detection model obtained in the step 5), splicing the results detected by the sliding window, and comparing the results obtained by the sliding window and the pattern template picture to obtain the defect type and the area label of the picture to be detected.
6.1) setting the size of a sliding window to be 1024 x 905 for a picture of the surface of an industrial product to be detected, performing sliding window detection on the picture to be detected by using the Cascade-RCNN detection model obtained in the step 5), and mapping the result back to the region of an original image to obtain the label format (category, x) of each defectmin,ymin,xmax,ymaxScore), category, indicates the defect class, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) And the horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, the score represents the confidence coefficient of defect judgment, and the value of the confidence coefficient is (0, 1).
6.2) judging the defect labels close to the edges of the sliding windows, wherein if the adjacent sliding windows have labels with the same category and similar position and size, the adjacent standard of the judgment label is as follows: the distance between the rectangular marking frame and the boundary position of the picture is less than 20 pixels, and the shortest distance between the rectangular marking frame and the other marking frame is less than 30 pixels. And then, carrying out annotation merging operation on the holding annotation frames meeting the conditions according to the confidence degree sequence of the holding annotation frames, and merging the defective rectangular annotation frames to allow at most one annotation in each cutting area to participate. The merged rectangle frame is the minimum bounding rectangle of the plurality of rectangle frames, and the new confidence calculation takes the mean value of the rectangle frames as follows:
scorenew＝(score1+score2+...+scoren)/n
wherein, scoreiAnd representing the confidence of the ith rectangular labeling box participating in synthesis, and n represents the total number of the rectangular boxes participating in synthesis.
6.3) adopting the steps 6.1) and 6.2) to carry out defect detection on the pattern template picture Z of the textile with the patterns in advance under the offline condition, and storing the obtained result.
6.4) adopting the steps 6.1) and 6.2) to carry out defect detection on the textile picture with the patterns needing to be detected on line, comparing the obtained result with the result of the corresponding template stored in the step 6.3), and adopting an interaction over Union (IoU) as a comparison standard, wherein the calculation formula is as follows:
Figure BDA0002344783280000091
wherein DR represents a defect rectangular marking frame detected on the to-be-detected image, and GT represents a real defect rectangular marking frame. The specific method for comparison is to compare the defect labels belonging to the same category, and if IoU is greater than a set threshold τ of 0.5 and the confidence of the defect on the picture to be detected is less than a set threshold γ of 0.3, the defect label on the picture to be detected is regarded as a texture false detection and is removed, so as to obtain the final defect position label and the corresponding category of the picture to be detected.
The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be regarded as equivalent substitutions, and are included in the scope of the present invention.

Claims (5)

1.基于样本增强的工业产品表面缺陷检测方法，其特征在于，包括以下步骤：1. A method for detecting surface defects of industrial products based on sample enhancement is characterized in that, comprising the following steps: 1)对工业产品表面图片集进行尺寸标准化操作，其中含有缺陷的图片有对应的缺陷标注文件，对缺陷图片以及每张图片对应缺陷标注文件进行切割操作，并根据切割后的标注划分为正常图片集和缺陷图片集；1) Carry out the size standardization operation on the surface picture set of industrial products, the pictures containing defects have corresponding defect annotation files, and cut the defect pictures and the corresponding defect annotation files of each picture, and divide them into normal pictures according to the cut annotations collections and defect picture collections; 2)将步骤1)得到的缺陷图片集进行归一化和在线随机数据增强，包括上下左右随机翻转，并划分批次；2) Normalize the defect picture set obtained in step 1) and enhance online random data, including random flipping up and down, left and right, and dividing into batches; 3)对于步骤2)一个批次中的每一张缺陷图片，在正常图片集中随机寻找一张与之对应切割位置相同，且纹理模板图案相同的正常图片，与缺陷图片进行左或右的拼接操作，并相应地修改标注文件；3) For each defective picture in a batch in step 2), randomly find a normal picture with the same cutting position and the same texture template pattern in the normal picture set, and perform left or right splicing with the defective picture operation, and modify the annotation file accordingly; 4)将步骤3)得到的每个批次的图片和标注，使用Cascade-RCNN算法进行迭代训练，训练完所有批次后完成一个轮次的训练；4) Use the Cascade-RCNN algorithm to iteratively train the pictures and labels of each batch obtained in step 3), and complete one round of training after training all batches; 5)完成一个轮次的训练后，重复进行步骤2)至步骤4)，直至达到设定的迭代轮次，把网络中的参数输出并保存，得到Cascade-RCNN检测模型；5) After completing one round of training, repeat steps 2) to 4) until the set iteration round is reached, output and save the parameters in the network to obtain the Cascade-RCNN detection model; 6)利用步骤5)得到的Cascade-RCNN检测模型对需要检测的工业产品表面图片和确定无缺陷的纹理模板图片进行滑窗检测，对滑窗检测到的结果进行拼接，并对二者得到的结果进行比较，最终得到待检测图片的缺陷类别及区域标注。6) Use the Cascade-RCNN detection model obtained in step 5) to perform sliding window detection on the surface image of the industrial product to be detected and the texture template image that is determined to be defect-free, splicing the results detected by the sliding window, and compare the results obtained by the two. The results are compared, and finally the defect category and area annotation of the image to be detected are obtained. 2.根据权利要求1所述的基于样本增强的工业产品表面缺陷检测方法，其特征在于：在步骤1)中，工业产品表面图片集包括含缺陷图片集X、不含缺陷的正常图片集Y和每种纹理模板的样式示例图片组成的模板图片集Z，其中，缺陷图片集X含有标注，每一条缺陷的标注为一个矩形标注框，其格式为(name,category,xmin,ymin,xmax,ymax)，其中name表示图片名称，category表示缺陷的种类，(xmin,ymin)表示矩形标注框左上角的横纵坐标，(xmax,ymax)表示矩形标注框右下角的横纵坐标，图片集Y和Z都没有标注信息；对三个图片集进行尺寸标准化，使所有图片均为H*W的RGB图片，其中H和W为图片的高和宽；2. The method for detecting surface defects of industrial products based on sample enhancement according to claim 1, characterized in that: in step 1), the set of images on the surface of industrial products comprises a set of pictures containing defects X and a set of normal pictures Y without defects Template image set Z composed of style example images of each texture template, wherein the defect image set X contains annotations, and the annotation of each defect is a rectangular annotation box, and its format is (name, category, x min , y min , x max , y max ), where name represents the image name, category represents the type of defect, (x min , y min ) represents the horizontal and vertical coordinates of the upper left corner of the rectangular labeling box, (x max , y max ) represents the lower right corner of the rectangular labeling box The horizontal and vertical coordinates of the image set, Y and Z, have no label information; the size of the three image sets is standardized, so that all the images are H*W RGB images, where H and W are the height and width of the image; 对三个图片集做相同的平均切割，并对图片集X上的缺陷标注进行切割，其规则为：将缺陷的矩形标注框做与图片同样的切割，并映射到切割后的小图范围内，若标注被切断，则计算被切断后的矩形标注框面积占原有矩形标注框面积的比例，若大于设定阈值ε，则保留该标注，否则丢弃；Make the same average cut for the three image sets, and cut the defect annotations on the image set X. The rule is: make the same cut as the picture and map the rectangular annotation frame of the defect to the small image after the cut. , if the label is cut off, calculate the ratio of the area of the cut rectangular label frame to the area of the original rectangular label frame, if it is greater than the set threshold ε, keep the label, otherwise discard it; 根据切割后的标注信息将图片分为缺陷图片集Xnew，正常图片集Ynew和模板图片集Znew，并按照它们被切割的位置进行保存。According to the marked information after cutting, the pictures are divided into a defect picture set X new , a normal picture set Y new and a template picture set Z new , and they are saved according to their cut positions. 3.根据权利要求1所述的基于样本增强的工业产品表面缺陷检测方法，其特征在于：所述步骤3)包括以下步骤：3. The method for detecting surface defects of industrial products based on sample enhancement according to claim 1, wherein the step 3) comprises the following steps: 3.1)对于步骤2)中经过随机翻转的缺陷图片xi∈Xnew，根据其模板纹理和切割位置找到对应的正常图片集M∈(Ynew,Znew)，使得xi和M中的所有样本的纹理模板和切割位置都相同，其中Xnew为经过切割后的缺陷图片集，Ynew,Znew分别为切割后的正常图片集和模板图片集；3.1) For the randomly flipped defect image x i ∈ X new in step 2), find the corresponding normal image set M∈ (Y new ,Z new ) according to its template texture and cutting position, so that all the The texture template and cutting position of the sample are the same, wherein X new is the defect picture set after cutting, and Y new and Z new are the normal picture set and template picture set after cutting respectively; 3.2)随机选取正常图片yi∈M，根据xi的尺寸，对yi的边缘用0值进行填充，使得yi的尺寸与xi的尺寸完全相同，并对yi进行归一化处理；3.2) Randomly select a normal picture y i ∈ M, and fill the edge of yi with 0 value according to the size of xi , so that the size of yi is exactly the same as that of xi , and normalize yi ; 3.3)对yi进行数据增强操作，即随机的上下左右翻转；3.3) Perform a data enhancement operation on yi , that is, flip up, down, left and right at random; 3.4)生成一个(0,1)之间的随机数，以50％的概率使得xnew＝(xi,yi)，另外50％的概率使得xnew＝(yi,xi)，即随机进行左或右的拼接，其中xnew表示生成的新样本；3.4) Generate a random number between (0, 1), with 50% probability to make x new = (x i , y i ), and another 50% probability to make x new = (y i , x i ), that is Randomly perform left or right splicing, where x new represents the new sample generated; 3.5)根据拼接的方式处理标注信息，若缺陷图片xi在左，则无需改变缺陷标注，若缺陷图片xi在右，则矩形标注框需要进行相应的更正。3.5) The labeling information is processed according to the splicing method. If the defect picture xi is on the left, there is no need to change the defect label. If the defect picture xi is on the right, the rectangular labeling box needs to be corrected accordingly. 4.根据权利要求1所述的基于样本增强的工业产品表面缺陷检测方法，其特征在于：在步骤4)中，所述Cascade-RCNN算法包括主干网络、区域提议网络、分类回归网络三部分，分别用来提取特征，生成区域建议和候选框的分类及微调；其中，使用卷积神经网络ResNeXt-101和特征金字塔FPN作为主干网络，区域提议网络使用二阶段目标检测器Faster-RCNN中的区域提议网络部分，分类回归网络使用多层级联网络。4. The industrial product surface defect detection method based on sample enhancement according to claim 1, is characterized in that: in step 4), described Cascade-RCNN algorithm comprises three parts of backbone network, regional proposal network, classification and regression network, They are used to extract features, generate region proposals and classification and fine-tuning of candidate boxes; among them, the convolutional neural network ResNeXt-101 and feature pyramid FPN are used as the backbone network, and the region proposal network uses the two-stage target detector Faster-RCNN in the region In the proposed network part, the classification and regression network uses a multi-layer cascaded network. 5.根据权利要求1所述的基于样本增强的工业产品表面缺陷检测方法，其特征在于：在步骤6)中，进行如下检测过程：5. the industrial product surface defect detection method based on sample enhancement according to claim 1, is characterized in that: in step 6) in, carry out following detection process: 6.1)对于一张待检测的工业产品表面图片，使用预先设定好的滑窗尺寸，使用步骤5)得到的Cascade-RCNN检测模型在待检测图片上进行滑窗检测，再将结果映射回原图的区域上，得到每一条缺陷的标注格式为(category,xmin,ymin,xmax,ymax,score)，category表示缺陷类别，(xmin,ymin)表示矩形标注框左上角的横纵坐标，(xmax,ymax)表示矩形标注框右下角的横纵坐标，score表示缺陷判断的置信度，置信度取值为(0,1)之间；6.1) For a surface image of an industrial product to be detected, use the pre-set sliding window size, use the Cascade-RCNN detection model obtained in step 5) to perform sliding window detection on the image to be detected, and then map the result back to the original On the area of the figure, the label format of each defect is (category, x min , y min , x max , y max , score), category represents the defect category, and (x min , y min ) represents the upper left corner of the rectangular labeling box. Horizontal and vertical coordinates, (x max , y max ) represents the horizontal and vertical coordinates of the lower right corner of the rectangular labeling box, score represents the confidence level of defect judgment, and the confidence level is between (0, 1); 6.2)对靠近滑窗边缘的缺陷标注进行判断，若相邻滑窗有相同类别且位置尺寸相近的标注，则按其置信度排序进行标注合并操作，合并后的矩形标注框为多个矩形标注框的最小外接矩形，新的置信度计算取它们的均值，如下：6.2) Judging the defect annotations near the edge of the sliding window, if the adjacent sliding windows have annotations of the same category and similar position and size, the annotation merging operation is performed according to their confidence order, and the merged rectangular annotation frame is a plurality of rectangular annotations The minimum enclosing rectangle of the box, and the new confidence calculation takes their mean, as follows: scorenew＝(score1+score2+...+scoren)/nscore new =(score 1 +score 2 +...+score n )/n 其中，scorenew表示新的置信度，scorei表示参与合成的第i个矩形标注框的置信度，n表示参与合成的矩形标注框总个数；Among them, score new represents the new confidence, score i represents the confidence of the i-th rectangular annotation frame participating in the synthesis, and n represents the total number of rectangular annotation frames participating in the synthesis; 6.3)采用步骤6.1)和6.2)在离线的情况下预先对模板图片集Z进行缺陷检测，将得到的结果保存；6.3) Use steps 6.1) and 6.2) to perform defect detection on the template image set Z in advance under the offline situation, and save the obtained results; 6.4)采用步骤6.1)和6.2)在线对需要检测的工业产品表面图片进行检测，将得到的结果与步骤6.3)保存的相应模板的检测结果做对比，采用IoU作为对比标准，其计算公式如下：6.4) Use steps 6.1) and 6.2) to detect the surface image of the industrial product that needs to be detected online, and compare the obtained result with the detection result of the corresponding template saved in step 6.3), using IoU as the comparison standard, and its calculation formula is as follows:
Figure FDA0002344783270000041
Figure FDA0002344783270000041
 其中，DR表示在待检测图片上检测到的缺陷矩形标注框，GT表示真实的缺陷矩形标注框；对比的具体方法是：将属于同一种类别的缺陷标注进行对比，若其IoU大于设定阈值τ，且待检测图片上的缺陷置信度小于设定阈值γ，则将待检测图片上的缺陷标注认为是纹理误检，予以剔除，由此得到待检测图片最终的缺陷位置标注和对应类别。Among them, DR represents the defect rectangle annotation frame detected on the image to be inspected, GT represents the real defect rectangle annotation frame; the specific method of comparison is to compare the defect annotations belonging to the same category, if the IoU is greater than the set threshold value τ, and the defect confidence degree on the image to be detected is less than the set threshold γ, the defect annotation on the image to be detected is considered to be a texture misdetection, and it is eliminated, thereby obtaining the final defect location annotation and corresponding category of the image to be detected.
CN201911390407.9A 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement Active CN111161243B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911390407.9A CN111161243B (en) 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911390407.9A CN111161243B (en) 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement

Publications (2)

Publication Number Publication Date
CN111161243A true CN111161243A (en) 2020-05-15
CN111161243B CN111161243B (en) 2023-04-07

Family

ID=70559383

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911390407.9A Active CN111161243B (en) 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement

Country Status (1)

Country Link
CN (1) CN111161243B (en)

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709452A (en) * 2020-05-21 2020-09-25 五邑大学 Evaluation method, electronic device and storage medium of wine bottle surface defect model
CN111862089A (en) * 2020-08-03 2020-10-30 上海闻泰信息技术有限公司 A method, device and electronic device for identifying defects in a dust-proof net
CN111860508A (en) * 2020-07-28 2020-10-30 平安科技（深圳）有限公司 Image sample selection method and related equipment
CN111882520A (en) * 2020-06-16 2020-11-03 歌尔股份有限公司 Screen defect detection method and device and head-mounted display equipment
CN112508836A (en) * 2020-12-14 2021-03-16 磐基技术有限公司 Deep learning image enhancement method based on label frame splicing
CN112508857A (en) * 2020-11-17 2021-03-16 华南理工大学 Aluminum product surface defect detection method based on improved Cascade R-CNN
CN112700435A (en) * 2021-01-12 2021-04-23 华南理工大学 Wall defect detection method based on deep learning
CN113192061A (en) * 2021-05-25 2021-07-30 安徽睿微信息科技有限公司 LED package appearance detection image extraction method and device, electronic equipment and storage medium
CN113204868A (en) * 2021-04-25 2021-08-03 中车青岛四方机车车辆股份有限公司 Defect detection parameter optimization method and optimization system based on POD quantitative analysis
CN113222990A (en) * 2021-06-11 2021-08-06 青岛高重信息科技有限公司 Chip counting method based on image data enhancement
CN113643235A (en) * 2021-07-07 2021-11-12 青岛高重信息科技有限公司 Chip counting method based on deep learning
CN114120209A (en) * 2022-01-27 2022-03-01 深圳市博铭维技术股份有限公司 Pipeline defect detection method, system, equipment and storage medium
CN114332754A (en) * 2021-12-14 2022-04-12 长讯通信服务有限公司 Cascade R-CNN pedestrian detection method based on multi-metric detector
CN114332073A (en) * 2022-01-14 2022-04-12 湖南视比特机器人有限公司 Detection method and device for target workpiece, intelligent sorting system and storage medium
CN114612471A (en) * 2022-05-10 2022-06-10 佛山市阿瑞斯数字设备有限公司 Method for detecting surface defects of ceramic tiles with complex textures
CN114998236A (en) * 2022-05-25 2022-09-02 征图新视(江苏)科技股份有限公司 Defect extraction algorithm based on target detection
CN115205626A (en) * 2022-06-24 2022-10-18 合肥工业大学 Data enhancement method applied to field of coating defect detection
CN115797314A (en) * 2022-12-16 2023-03-14 哈尔滨耐是智能科技有限公司 Part surface defect detection method, system, equipment and storage medium
CN115965872A (en) * 2022-07-22 2023-04-14 中科三清科技有限公司 A tea picking method, device, electronic equipment and storage medium
CN115965582A (en) * 2022-11-22 2023-04-14 哈尔滨岛田大鹏工业股份有限公司 Ultrahigh-resolution-based engine cylinder body and cylinder cover surface defect detection method
WO2024044915A1 (en) * 2022-08-29 2024-03-07 西门子股份公司 Image comparison method and apparatus for error detection, and computer device

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108663431A (en) * 2014-05-18 2018-10-16 查尔斯斯塔克德雷珀实验室有限公司 The method for detecting the defects of ferromagnetic material
CN109829893A (en) * 2019-01-03 2019-05-31 武汉精测电子集团股份有限公司 A kind of defect object detection method based on attention mechanism

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108663431A (en) * 2014-05-18 2018-10-16 查尔斯斯塔克德雷珀实验室有限公司 The method for detecting the defects of ferromagnetic material
CN109829893A (en) * 2019-01-03 2019-05-31 武汉精测电子集团股份有限公司 A kind of defect object detection method based on attention mechanism

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
徐秀 等: "基于Faster R-CNN的圆柱形金属工件表面缺陷检测", 《软件导刊》 *

Cited By (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709452A (en) * 2020-05-21 2020-09-25 五邑大学 Evaluation method, electronic device and storage medium of wine bottle surface defect model
CN111709452B (en) * 2020-05-21 2023-06-16 五邑大学 Method for evaluating surface defect model of wine bottle, electronic device and storage medium
CN111882520A (en) * 2020-06-16 2020-11-03 歌尔股份有限公司 Screen defect detection method and device and head-mounted display equipment
CN111882520B (en) * 2020-06-16 2023-10-17 歌尔光学科技有限公司 Screen defect detection method and device and head-mounted display equipment
CN111860508A (en) * 2020-07-28 2020-10-30 平安科技（深圳）有限公司 Image sample selection method and related equipment
CN111862089A (en) * 2020-08-03 2020-10-30 上海闻泰信息技术有限公司 A method, device and electronic device for identifying defects in a dust-proof net
CN112508857B (en) * 2020-11-17 2022-04-22 华南理工大学 Surface defect detection method of aluminum material based on improved Cascade R-CNN
CN112508857A (en) * 2020-11-17 2021-03-16 华南理工大学 Aluminum product surface defect detection method based on improved Cascade R-CNN
CN112508836A (en) * 2020-12-14 2021-03-16 磐基技术有限公司 Deep learning image enhancement method based on label frame splicing
CN112700435A (en) * 2021-01-12 2021-04-23 华南理工大学 Wall defect detection method based on deep learning
CN113204868A (en) * 2021-04-25 2021-08-03 中车青岛四方机车车辆股份有限公司 Defect detection parameter optimization method and optimization system based on POD quantitative analysis
CN113204868B (en) * 2021-04-25 2023-02-28 中车青岛四方机车车辆股份有限公司 Defect detection parameter optimization method and optimization system based on POD quantitative analysis
CN113192061A (en) * 2021-05-25 2021-07-30 安徽睿微信息科技有限公司 LED package appearance detection image extraction method and device, electronic equipment and storage medium
CN113192061B (en) * 2021-05-25 2023-08-29 安徽睿微信息科技有限公司 Extraction method and device of LED package appearance detection image, electronic equipment and storage medium
CN113222990B (en) * 2021-06-11 2023-03-14 青岛高重信息科技有限公司 Chip counting method based on image data enhancement
CN113222990A (en) * 2021-06-11 2021-08-06 青岛高重信息科技有限公司 Chip counting method based on image data enhancement
CN113643235A (en) * 2021-07-07 2021-11-12 青岛高重信息科技有限公司 Chip counting method based on deep learning
CN113643235B (en) * 2021-07-07 2023-12-29 青岛高重信息科技有限公司 Chip counting method based on deep learning
CN114332754A (en) * 2021-12-14 2022-04-12 长讯通信服务有限公司 Cascade R-CNN pedestrian detection method based on multi-metric detector
CN114332073A (en) * 2022-01-14 2022-04-12 湖南视比特机器人有限公司 Detection method and device for target workpiece, intelligent sorting system and storage medium
CN114120209A (en) * 2022-01-27 2022-03-01 深圳市博铭维技术股份有限公司 Pipeline defect detection method, system, equipment and storage medium
CN114120209B (en) * 2022-01-27 2022-05-20 深圳市博铭维技术股份有限公司 Pipeline defect detection method, system, equipment and storage medium
CN114612471A (en) * 2022-05-10 2022-06-10 佛山市阿瑞斯数字设备有限公司 Method for detecting surface defects of ceramic tiles with complex textures
CN114998236A (en) * 2022-05-25 2022-09-02 征图新视(江苏)科技股份有限公司 Defect extraction algorithm based on target detection
CN115205626A (en) * 2022-06-24 2022-10-18 合肥工业大学 Data enhancement method applied to field of coating defect detection
CN115965872A (en) * 2022-07-22 2023-04-14 中科三清科技有限公司 A tea picking method, device, electronic equipment and storage medium
CN115965872B (en) * 2022-07-22 2023-08-15 中科三清科技有限公司 Tea picking method and device, electronic equipment and storage medium
WO2024044915A1 (en) * 2022-08-29 2024-03-07 西门子股份公司 Image comparison method and apparatus for error detection, and computer device
CN115965582A (en) * 2022-11-22 2023-04-14 哈尔滨岛田大鹏工业股份有限公司 Ultrahigh-resolution-based engine cylinder body and cylinder cover surface defect detection method
CN115965582B (en) * 2022-11-22 2024-03-08 哈尔滨岛田大鹏工业股份有限公司 Ultrahigh-resolution-based method for detecting surface defects of cylinder body and cylinder cover of engine
CN115797314A (en) * 2022-12-16 2023-03-14 哈尔滨耐是智能科技有限公司 Part surface defect detection method, system, equipment and storage medium
CN115797314B (en) * 2022-12-16 2024-04-12 哈尔滨耐是智能科技有限公司 Method, system, equipment and storage medium for detecting surface defects of parts

Also Published As

Publication number Publication date
CN111161243B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
CN111161243B (en) Industrial product surface defect detection method based on sample enhancement
CN115082419B (en) Blow-molded luggage production defect detection method
CN110276754B (en) A surface defect detection method, terminal equipment and storage medium
CN109829914B (en) Method and device for detecting product defects
CN108562589B (en) Method for detecting surface defects of magnetic circuit material
CN104866862B (en) A kind of method of belt steel surface area-type defect recognition classification
CN114140679B (en) Defect fusion method, device, recognition system and storage medium
CN111915704A (en) Apple hierarchical identification method based on deep learning
CN109961049A (en) A method for cigarette brand recognition in complex scenes
CN109308700A (en) A kind of visual identity defect inspection method based on printed matter character
CN113706490B (en) Wafer defect detection method
CN111696077A (en) Wafer defect detection method based on wafer Det network
CN116559183B (en) Method and system for improving defect judging efficiency
CN111582294A (en) Method for constructing convolutional neural network model for surface defect detection and application thereof
CN112508857B (en) Surface defect detection method of aluminum material based on improved Cascade R-CNN
CN111461133B (en) Express delivery surface single item name identification method, device, equipment and storage medium
CN114972356B (en) Plastic product surface defect detection and identification method and system
CN110599463B (en) Tongue image detection and positioning algorithm based on lightweight cascade neural network
CN114627089B (en) Defect identification method, defect identification device, computer equipment and computer readable storage medium
CN111260645B (en) Tampered image detection method and system based on block classification deep learning
CN116012291A (en) Industrial part image defect detection method and system, electronic equipment and storage medium
CN112926694A (en) Method for automatically identifying pigs in image based on improved neural network
CN114359235A (en) Wood surface defect detection method based on improved YOLOv5l network
CN115147363A (en) Image defect detection and classification method and system based on deep learning algorithm
CN116342525A (en) SOP chip pin defect detection method and system based on Lenet-5 model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant