CN116258703A

CN116258703A - Defect detection method, defect detection device, electronic equipment and computer readable storage medium

Info

Publication number: CN116258703A
Application number: CN202310202673.4A
Authority: CN
Inventors: 肖寒琼; 胡流彬; 程乔; 刘豪豪; 马晚辉; 刘衍忠; 张修龙; 范振亮
Original assignee: Futaihua Industry Shenzhen Co Ltd
Current assignee: Futaihua Industry Shenzhen Co Ltd
Priority date: 2023-02-23
Filing date: 2023-02-23
Publication date: 2023-06-13

Abstract

A defect detection method, apparatus, electronic device and computer readable storage medium, the method includes: obtaining a product defect picture to be subjected to defect detection; identifying the detection category to which the product defect picture belongs; determining a defect detection model matched with the detection category; the defect detection model is a pre-trained defect detection model based on deep learning; and inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product. The method and the device can reduce the labor cost of defect detection, improve the detection efficiency and the accuracy of defect detection, and reduce the false detection rate.

Description

Defect detection method, defect detection device, electronic equipment and computer readable storage medium

Technical Field

The present invention relates to the field of image processing, and in particular, to a defect detection method, device, electronic apparatus, and computer readable storage medium.

Background

With the popularization of electronic products, the requirements on the manufacturing process of the parts of the electronic products are higher and higher, and the aesthetic appearance based on the product appearance is also higher. Most electronic products, such as housings for cell phones, tablets, notebook computers, etc., are manufactured from a multi-purpose alloy, such as an aluminum alloy. Alloy housings are often anodized from raw materials to manufacturing processes that can be used in assembled housings as many as ten steps in order to be more resistant to wear and interference from external factors such as perspiration.

After anodizing, the product may have appearance defects such as scraping pressure, chromatic aberration, spots and the like, and the appearance is extremely influenced, and the product can only be used as a defective product for secondary processing. At present, the detection of the appearance defects depends on the detection of the defects of the products by naked eyes so as to distinguish whether the products are good products or not.

However, manual visual inspection is high in labor cost and low in detection efficiency on one hand, and the false detection rate is high on the other hand.

Disclosure of Invention

In view of the foregoing, the present application provides a defect detection method, apparatus, electronic device, and computer readable storage medium, which, on one hand, reduces the labor cost of defect detection, improves the detection efficiency, and on the other hand, improves the accuracy of defect detection, and reduces the false detection rate.

An embodiment of the present application provides a defect detection method, including: obtaining a product defect picture to be subjected to defect detection; identifying the detection category to which the product defect picture belongs;

determining a defect detection model matched with the detection category; the defect detection model is a pre-trained defect detection model based on deep learning; and inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product.

By adopting the technical scheme, corresponding defect detection models can be determined according to different product defect pictures, so that automatic defect detection can be realized, defect detection accuracy and efficiency are improved, and labor cost is saved.

In some embodiments, the detection categories to which the product defect pictures belong are distinguished based on morphological features and positional features of defects in the product defect pictures; identifying the detection category to which the product defect picture belongs, including: if the defects in the product defect picture are located at the preset positions, determining the detection category as a first category; if the defects in the product defect picture meet the preset small-size condition, determining the detection category as a second category; and if the defects in the product defect picture meet the preset large-size condition, determining the detection category as a third category.

By adopting the technical scheme, the defect detection model is matched based on the morphological characteristics and the position characteristics of the defects, so that the defect detection model can process the defect picture of the product based on the morphology and the position of the defects in a targeted manner, and the accuracy of the defect detection result is improved.

In some embodiments, inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product, including: detecting a defect position in the product defect picture through the detection module; inputting the defect position into the segmentation module to obtain a segmented defect region; obtaining the size of the defect based on the defect area; and if the size of the defect is in the preset good product range, outputting a defect detection result representing that the product is good.

Because the type of defects are generally positioned at the edges or the characteristic structure positions of the products, the type of product defect images are processed by the detection module and the segmentation module, so that the accurate positioning of the defects is convenient to realize, and the detection result of good products is improved.

In some embodiments, the second class of matched defect detection models includes a segmentation module and a classification module, and the inputting the product defect picture into the matched defect detection models to obtain defect detection results of the product includes: processing the product defect picture through the segmentation module to obtain segmented defect areas; obtaining a defect size based on the defect area; and if the defect size is larger than a preset filtering parameter, inputting the divided defect areas into the classification module to obtain a defect detection result of the product.

By adopting the technical scheme, the general size of the type of defects is larger, the edge positions are more fuzzy, and accurate segmentation is difficult to carry out through a segmentation model, so that the defect detection result of the type of product defect pictures is conveniently obtained through a classification module based on the position of the detected defects by the detection module.

In some embodiments, the training step of the defect detection model comprises: obtaining a defect image sample belonging to a target detection category; inputting the defect image sample belonging to the target detection category into a defect detection model matched with the target detection category to obtain a detection result of the defect image sample; determining whether the detection result is abnormal or not based on the detection result of the defect detection model and the label of the defect image sample; the abnormality includes one or more of detection of overdischarge, false detection of defect type, detection of omission; outputting a model optimization strategy based on the anomaly; and adjusting the defect detection model based on the model optimization strategy.

By adopting the technical scheme, the defect detection model can be adjusted based on the abnormal result, the accuracy of the defect detection model is improved, and the over-killing rate, the omission rate and the defect false detection are reduced.

In some embodiments, obtaining a defect image sample belonging to a target detection class includes: acquiring a product image sample belonging to a target detection category; shielding the background of the product image sample to obtain a product image sample with the shielded background; cutting the product image sample subjected to the background shielding according to a preset cutting method to obtain a cut image; and adjusting the contrast of the clipping image to obtain the defect image sample.

By adopting the technical scheme, the image is preprocessed, interference information in the image can be shielded, defects in the image are highlighted, a defect detection model is convenient to process a defect detection picture, and the success rate and the accuracy of detection are improved.

An embodiment of the present application provides a defect detection apparatus, including: the device comprises an acquisition module, a detection module and a display module, wherein the acquisition module is used for acquiring a product defect picture to be subjected to defect detection and identifying a detection category to which the product defect picture belongs; the matching module is used for determining a defect detection model matched with the detection category; the defect detection model is a pre-trained defect detection model based on deep learning; and the detection module is used for inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product.

An embodiment of the present application provides an electronic device, including a processor and a memory, where the memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device executes the defect detection method described above.

An embodiment of the present application provides a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform the defect detection method described above.

Drawings

FIG. 1 is a flow chart illustrating steps of a defect detection method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a defect detection method according to an embodiment of the present disclosure;

FIG. 3 is a flowchart illustrating steps of a method for training a defect detection model according to an embodiment of the present application;

FIG. 4 is a flowchart of sub-steps of step 301 provided by an embodiment of the present application;

FIG. 5 is a schematic view of image cropping according to an embodiment of the present application;

FIG. 6 is a schematic diagram of labeling defect areas by a segmentation module according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a training process curve of a defect detection model according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a classification module according to an embodiment of the present application obtaining a defect detection result;

FIG. 9 is a schematic diagram of a third defect detection model for performing defect detection according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a defect detecting device according to an embodiment of the present disclosure;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order that the above-recited objects, features and advantages of the present application will be more clearly understood, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. In addition, embodiments of the present application and features of the embodiments may be combined with each other without conflict.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, and the described embodiments are merely some, rather than all, of the embodiments of the present application.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

It is further intended that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The term "at least one" in this application means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and the representation may have three relationships, for example, a and/or B may represent: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The terms "first," "second," "third," "fourth" and the like in the description and in the claims and drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

In the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations, or descriptions. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

After anodizing, the product may have appearance defects such as scraping pressure, chromatic aberration, spots and the like, and the appearance is extremely influenced, and the product can only be used as a defective product for secondary processing. Currently, detection of such appearance defects depends on manual visual inspection.

However, the manual visual inspection has the following problems:

1. the visual fatigue is easy to detect manually, and the false detection exists;

2. the judgment standards of different human defect detection are different, and the sensitivity of human eye colors is inconsistent, so that different people can have differences on the detection results of the same defect, and the detection results are inaccurate;

3. the manpower cost of manual detection is high, and manual detection efficiency is low.

In view of this, an embodiment of the present application provides a defect detection method, including: obtaining a product defect picture to be subjected to defect detection; identifying the detection category to which the product defect picture belongs; determining a defect detection model matched with the detection category; the defect detection model is a pre-trained deep learning model; and inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product.

According to the defect detection method and device, corresponding defect detection models can be determined according to different product defect pictures, so that automatic defect detection can be achieved, defect detection accuracy and efficiency are improved, and labor cost is saved.

In addition, when the defect detection model is trained, the product defect image samples matched with the detection types can be input into the corresponding defect detection model, and the product defect image samples of all types are not required to be input into the defect detection model for training, so that convenience is brought to training of the defect detection model, and the defect detection model can be converged rapidly.

In addition, the defect detection model of the embodiment of the application is a pre-trained deep learning model, the deep learning model adopts an end-to-end learning mode, the end-to-end learning mode can eliminate data preprocessing problems and some dependence on human experts, the method has good universality and strong learning capacity, the characteristics of various defects can be automatically extracted and learned, the coverage range of the deep learning model is wide, the adaptability is good, the function can be mapped to any function theoretically, the method can be applied to a complex defect detection scene, in addition, the deep learning model is driven by data, the larger the data quantity is, the better the deep learning model is expressed, and the efficiency and the accuracy can exceed that of artificial naked eye detection when the defect detection is carried out.

Therefore, the embodiment is used for detecting the defects of the product in appearance through the defect detection model taking deep learning as a core, so that the labor can be replaced, the production cost is reduced, and the generation efficiency is improved.

The defect detection method of the present embodiment may be applied to one or more electronic devices. The electronic device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and its hardware includes, but is not limited to, a processor, a micro-program controller (Microprogrammed Control Unit, MCU), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, and the like. The electronic device may be a portable electronic device (e.g., a cell phone, tablet computer), a personal computer, a server, etc.

FIG. 1 is a flowchart illustrating steps of an embodiment of a defect detection method according to the present application. FIG. 2 is a schematic diagram of a defect detection method according to an embodiment of the present application. The defect detection method according to the embodiment of the present application is described below with reference to fig. 1 and 2.

It will be appreciated that the order of the steps in the flow chart shown in fig. 1 may be changed, some steps may be omitted, and the defect detection method may include the following steps according to different requirements.

And step 101, obtaining a product defect picture to be subjected to defect detection.

The product defect picture may be an appearance defect picture of the product. For example, the product defect picture can be an aluminum alloy shell defect picture of electronic equipment such as a tablet, a mobile phone, a notebook computer and the like. Product defects include, but are not limited to, crush, scratch, bright spot, corrosion spot, bruise, gas mark, uneven dyeing, and off-color.

In some embodiments, step 101 may comprise: and acquiring an initial product image, and taking the initial product image as a product defect picture.

For example, the defect of the product is shot by a camera to obtain an initial product image.

Because the defects are too small, the light rays are poor and the like, the image shot by the camera is likely to be difficult to show the product defects, so that the linear array time-sharing stroboscopic technology can be used for taking images of multiple scenes of bright, dark and bright fields under the condition that the product defects cannot be shot at a single angle; under the condition that the product defect is a chromatic aberration defect, a 3CCD beam splitting prism camera can be tried, and the color gradation is highlighted by a sub-band exposure or a 10bit color depth camera; under the condition that the product defect is a concave-convex defect, a 'line laser' can be tried to take a depth map, so that the contrast ratio is enhanced; under the condition that the size of the defect is undersized, a microscope can be used for drawing, and the detail is highlighted; under the oblique shooting scene, the 'Kara lens' can be used to solve the lens defocus problem.

In other embodiments, step 101 may include: and acquiring an initial product image, performing preprocessing operation on the initial product image, and taking the preprocessed initial product image as a product defect picture.

Wherein the preprocessing operation may include: image background masking, image cropping, adjusting image contrast, etc., but is not limited thereto. In the actual preprocessing process, the preprocessing operation on the image can be set according to the requirement.

Image background masking refers to masking the background in the image except for the product area, for example, using solid black or solid white to cover the background so that features and light and shadow variations in the background do not interfere with the detection of product defects.

Under the condition that the product size is larger and the defect size is smaller, the part of the image which is not defective in the product can be reduced in an image cutting mode, the duty ratio of the product defect in the image is improved, and therefore the condition that image information is lost in the defect detection process due to the fact that the product defect is too small is reduced.

The image contrast is adjusted. For example, due to the influence of the product material, some product defects are not obvious in the original image, so that the image contrast can be enhanced, so that the product defects are obviously presented in the image.

After obtaining a product defect picture to be defect detected, step 102 may be performed.

Step 102, identifying the detection category to which the product defect picture belongs.

The detection category to which the product defect picture belongs can be distinguished based on morphological features and position features of defects in the product defect picture.

The shape and the position of the product defects have the characteristic of diversification, and the product defects possibly exist at any position on the surface of the product, so that the product defect pictures are classified based on the shape characteristics of the defects and the position characteristics of the defects, thereby detecting the defects, enabling the detection of the defects to be more targeted, and improving the accuracy of the defect detection.

In some embodiments, the detection categories may be classified into a first category, a second category, and a third category, and the classification basis of the detection categories in step 102 may include:

1. if the defects in the product defect picture are located at the preset positions, determining the detection category as a first category.

The defects of the preset positions can be defects such as bruise, material extrusion, collapse and the like of the edges of the product and some special structure positions, and the preset positions can be set according to requirements. For example, if the defect in the product defect picture is a bruise, a squeeze, or a collapse, it may be determined that the defect is located at a predetermined position, and the detection class of the product defect picture is the first class.

2. If the defects in the product defect picture meet the preset small-size condition, determining the detection category as a second category.

The defects meeting the preset small-size conditions can be fine defects such as scratches, crush injuries, bright spots, corrosion spots and the like, which have unfixed positions and different morphological characteristics and are in planar dot shapes, long strips and the like.

For example, a small-size threshold may be set, the defect size of the product defect picture is compared with the small-size threshold, and if the defect size is smaller than the small-size threshold, the defect is determined to be a defect meeting a preset small-size condition.

For another example, a defect type meeting the preset small-size condition may be set, and if the defect in the product defect picture is a scratch, a crush injury, a bright spot, or a corrosion point, it may be determined that the defect meets the preset small-size condition, and the detection type of the product defect picture corresponding to the defect is a second type.

3. If the defects in the product defect picture meet the preset large-size condition, determining the detection category as a third category.

Defects meeting the preset large-size conditions can be defects of unclear boundaries such as dirt, heterochromatic and the like and large area occupation of a defect picture of the whole product.

For example, a large-size threshold may be set, the defect size of the product defect picture is compared with the large-size threshold, and if the defect size is greater than the large-size threshold, the defect is determined to be a defect meeting a preset large-size condition.

For another example, a defect type satisfying a preset large-size condition may be set, and if the defect in the defect picture of the product is dirty or abnormal, it may be determined that the defect satisfies the preset large-size condition, and the detection type of the defect picture of the product is a third type.

Table 1 shows the minimum dimension parameters of various types of defects, the data in the table are values in pixel units, namely, the minimum bounding rectangle short side and the minimum bounding rectangle long side are each in pixel (px), and the minimum area is in pixel square (px ² ). In the actual application process, specific data is based on actual measurement.

TABLE 1

Defect type	Minimum external rectangle short side	Minimum circumscribed rectangle long side	Minimum area
				Bruise with accident	8	15	120
Scratch and scratch	8	30	260
				Bright spot	10	20	200
Corrosion pitting/crushing	5	15	50
				Dirt/colour change	120	200	24000

After the product defect picture is collected, the product defect picture may be classified and stored according to the classification basis of the detection category, for example, if a defect in the product defect picture is a bruise, a squeeze, a collapse, or the like, the product defect picture is stored in a first folder, if a defect in the product defect picture is a scratch, a crush, a bright spot, or a corrosion point, the product defect picture is stored in a second folder, if a defect in the product defect picture is a stain or a different color, the product defect picture may be stored in a third folder, and the electronic device may determine the detection category of the product defect picture based on the folder in which the product defect picture is stored, for example, the product defect picture stored in the first folder corresponds to the first category.

Step 103, determining a defect detection model matched with the detection category.

The defect detection model is a pre-trained defect detection model based on deep learning.

Deep learning is derived from artificial neural networks. Deep learning forms a more abstract high-level representation attribute category or feature by combining underlying features to discover a distributed feature representation of the data.

Deep learning is a method for characterizing and learning data in machine learning. The observations can be represented in a number of ways, such as a vector of intensity values for each pixel, or more abstract as a series of edges, a region of a particular shape, etc. Learning from instances may be easier using some specific representation. The advantage of deep learning is the use of unsupervised or semi-supervised feature learning and hierarchical feature extraction efficient algorithms instead of manually acquired features. Deep learning is a new field of machine learning, which aims to build and simulate mechanisms of the human brain to interpret data.

Deep learning network structures include convolutional neural networks (Convolutional Nueral Network, CNN), recurrent neural networks (Recurent Neural Network, RNN), deep neural networks (Deep Neural Networks, DNN), and the like.

In some embodiments, if the detection class is the first class, the first class matches the defect detection model, i.e. the first defect detection model 201 includes a detection module and a segmentation module. The detection module is used for detecting the defect position in the product defect picture, the segmentation module is used for segmenting out the region of the defect module, and the pixel-level defect position can be accurately determined based on the segmentation module, so that the outline and the size of the defect are obtained.

If the detection class is the second class, the second class matches the defect detection model, that is, the second defect detection model 202 includes a segmentation module and a classification module. The segmentation module is used for segmenting out a defect area; the classification module is used for obtaining a defect detection result representing whether the product is good or not based on the defect area.

The classification module may also be used to output specific types of defects, such as scratches, crush, bright spots, or corrosion spots.

If the detection class is a third class, the third class matches the defect detection model, that is, the third defect detection model 203 includes a detection module and a classification module. The detection module is used for detecting the defect position in the defect picture of the product, and the classification module is used for obtaining a defect detection result representing whether the product is good or not based on the defect position.

The classification module may also be used to output specific types of defects, such as dirt or discoloration.

The detection module can adopt target detection models such as R-CNN, fast-R-CNN, YOLO, SSD and the like.

The classification module may employ a classification model such as logistic regression, decision tree, support vector machine, or naive Bayes.

The segmentation module can adopt VGGNet or ResNet image segmentation models.

And 104, inputting the defect picture of the product into a matched defect detection model to obtain a defect detection result of the product.

1. In the case where the detection class is the first class, step 104 may include: detecting a defect position, such as a defect edge position, in the product defect picture by a detection module of a first defect detection model; inputting the defect position into a segmentation module of a first defect detection model to obtain a segmented defect region; obtaining the size of the defect based on the defect area; and if the size of the defect is in the preset good product range, outputting a defect detection result representing that the product is good.

For example, in the case that the detection class is the first class, the defects in the product defect picture may be bruise, extrusion, collapse, or the like, and the size ranges of the bruise, extrusion, and collapse may be as shown in table 2 below, the long-side range and the short-side range in table 2 are both in units of pixels (px), and the area range is in units of squares of pixels (px ² )。

TABLE 2

Defect type	Area range	Long side range	Short side range
				Bruise with accident	120—10000	15—10000	8—10000
Extrusion material	120—10000	15—10000	8—10000
				Collapse edge	300—10000	30—10000	10—10000

The size ranges in table 2 are merely examples, and in practical applications, the actually measured size ranges are subject to. The user can set the good product range based on the measured size range, and the setting of the good product range can be changed according to the defect judgment standard, so that the flexibility of model detection can be improved, and the over-killing rate can be reduced. For example, a filter parameter may be set, and a smaller filter parameter is determined to be within the good range, and a larger filter parameter is determined to be within the bad range.

2. In the case where the detection class is the second class, step 104 may include: processing the product defect picture through the segmentation module to obtain segmented defect areas; obtaining a defect size based on the defect area; and if the defect size is larger than a preset filtering parameter, inputting the divided defect areas into the classification module to obtain a defect detection result of the product. That is, the classification module may output whether the product is good.

For example, in the case that the detection class is the second class, the defects in the product defect picture may be scratches, bright spots, corroded spots, or crush injuries, and the size ranges of the scratches, bright spots, corroded spots, and crush injuries may be as shown in the following table 3, each value in table 3 is a value of a pixel level, that is, the long side range and the short side range in table 3 are each in units of a pixel (px), and the area range is in units of a square of the pixel (px ² )。

TABLE 3 Table 3

Defect type	Area range	Long side range	Short side range
				Scratch and scratch	260—10000	30—10000	8—10000
Bright spot	200—10000	20—10000	10—10000
				Corrosion site/pressure damage	50—10000	15—10000	5—10000

The preset filtering parameters may include any one of a preset area threshold, a preset long-side threshold, a preset short-side threshold, or a combination thereof, and the filtering parameter settings of different kinds of defects may be different. The preset area threshold, the preset long side threshold, and the preset short side threshold may be set based on the size range of the defect in table 3 described above.

3. In the case where the detection class is the third class, step 104 may include: detecting the defect position in the product defect picture through a detection module of a third defect detection model; and inputting the defect position into a classification module of a third defect detection model to obtain a defect detection result of the product.

According to the method and the device for detecting the defects, corresponding defect detection models can be determined aiming at different product defect pictures, so that automatic detection of defects is achieved, defect detection accuracy and efficiency are improved, labor cost is saved, in addition, different types of defect detection models are used for different product defect pictures, and therefore when the models are trained, different detection types of defect detection models can be trained through different types of product defect image samples, training of the defect detection models is facilitated, and convergence speed of the defect detection models is accelerated.

The embodiment of the application also provides a training method of the defect detection model, and the defect detection model trained by the training method can be applied to the defect detection method.

The training method of the defect detection model can be applied to an initial training stage of the defect detection model and a subsequent optimizing stage for optimizing the defect detection model, for example, after the defect detection model is pre-trained initially, the pre-trained defect detection model can detect defects, aiming at defect detection results, defect detection quality inspection can be performed, if the quality inspection is unqualified, the pre-trained defect detection model can be trained again based on the quality inspection results, so that the defect detection model can be optimized.

Referring to fig. 3, the training method of the defect detection model may include:

step 301, obtaining a defect image sample belonging to the target detection category.

The detection category to which the defect image sample belongs can be distinguished based on morphological features and position features of defects in the product defect picture.

The target detection category may be the first category, the second category, and the third category of the defect detection method, which are not described herein.

In some embodiments, referring to fig. 4, step 301 may include:

Step 3011, obtain a product image sample belonging to the target detection category.

Step 3012, shielding the background of the product image sample, and obtaining the product image sample after shielding the background.

For example, the background except the product area in the product image sample is covered by pure black or pure white so as to prevent the feature and the shadow change in the background from interfering with the learning effect of the model.

And 3013, clipping the product image sample with the background shielded according to a preset clipping method to obtain a clipping image.

If the product is larger, the defect is smaller, the proportion of the defect target in the product is smaller, the resolution of the image is influenced after the image is processed by the deep learning model, and the image information is easy to lose, so that the influence of the deep learning model on the resolution of the image can be reduced by cutting the image, and the deep learning precision is improved.

The preset clipping method may be set according to the requirement, for example, the preset clipping method is to clip the product image into a rectangle, a circle, or the like.

In some embodiments, in the case of cropping the product image to be rectangular, step 3013 may include: cutting is carried out according to the ratio of the length and width of the defect circumscribed rectangle to the length and width of the image.

Specifically, referring to fig. 5, the length of the product image sample is denoted as length, the width is denoted as width, the length of the defect circumscribed rectangle is denoted as X, the width of the defect circumscribed rectangle is denoted as Y, the length of the product image sample is cut into M equal parts, and the width of the product image sample is cut into N equal parts, where m=length/(X j); n=width/(y×j).

The pixel value of the clipping image is j times of the pixel value of the defect image, the value range of j can be set according to requirements, and through a large amount of data verification, when the value range of j is between 50 and 120 times, the tiny defect is clearer.

And 3014, adjusting the contrast of the cut image to obtain a product defect picture.

Some defects are not particularly apparent in the presentation of the original image, and image contrast may be enhanced by some image processing tools, as affected by the product material.

In some embodiments, after the electronic device obtains the product image sample, the product image sample may be stored according to the workstation number, the defect type, and the hierarchy of the picture effect. The station number is the product station identification of the product. The defect types can be pits, multiple grinding marks, multiple materials, corrosion points, scratches and the like, and the picture effects can be dark fields, bright and dark fields and the like.

The product image sample can be named according to defect types, severity, shooting modes, product numbers, picture effects and the like, and is beneficial to later searching and importing. For example, a product image sample may be named "pit-mid-longitudinal-55-dark".

The defects can be clearly presented in the image, the product occupies the whole visual field of the camera as much as possible, and if the product is large, the product needs to be photographed in a partitioned mode, the position and the size of the product in the image are ensured to be consistent as much as possible, so that the uniformity of the picture is ensured. The quality of image acquisition has a crucial influence on the detection effect of the deep learning model.

Step 302, inputting the defect image sample belonging to the target detection category into a defect detection model matched with the target detection category to obtain a detection result of the defect image sample.

The first type of the matched defect detection model is a first defect detection model, and the first defect detection model comprises a detection module and a segmentation module. The first category aims at the defect image samples with defects at preset positions, such as at the edges of products and at some special structure positions, and adopts a modeling mode that detection modules are connected in series with segmentation modules, firstly, the defect positions can be positioned through the detection modules, then, defect areas are obtained through the segmentation modules, and therefore detection of specific positions (pixel levels), sizes and outlines of the defects is obtained.

Based on the defect area of the segmentation model, the size information such as the length, width and area information of the defect can be obtained, therefore, the good product range can be set according to the judging standard of the defect, if a certain filtering parameter is set, when the detected defect length, width and area are smaller than the set filtering parameter, the defect is judged to be in the good product range, and the product is judged to be good product.

The detection module can be positioned to the defect edge position, the number of the defect image samples of the same kind required by the detection module in the training process can be set according to requirements, for example, 60-1000, and the iteration times can be set according to requirements, for example, 600-1000.

The segmentation module may segment the defect area, i.e. label the defect outline, for example, referring to fig. 6, the defect image sample is labeled with the defect outline, training and learning the defect feature, and may accurately detect the defect position, in the training process, the number of the defect image samples may be set at about 50, and the iteration number is set at 600-1200. The number of training iterations may be set according to a training process curve, fig. 7 is a schematic diagram of a training process curve, where the training process curve includes a loss value curve, a positive precision curve, and a negative precision curve, the abscissa of the training process curve is the number of model training iterations, and the ordinate of the training process curve is the loss value (loss), the positive precision value (nacc), and the negative precision value (pacc). The closer the loss is to 0, the closer the nacc and pacc functions are to 1, the better the learning effect of the model, and the set training iteration times need to make the nacc and pacc smooth.

The second class of matched defect detection models are second defect detection models, and the second defect detection models comprise a segmentation module and a classification module. The second category aims at the defect image samples with unfixed defect positions and different morphological characteristics, firstly, a segmentation module is adopted to detect defects, a defect area with preset pixel size, such as 100-200, is cut out, and then, classification module processing is carried out on the defect area pictures so as to reduce false detection and over-killing. After the model is divided, the defect size of the defect area can be compared with preset filtering parameters, and after the defect size is judged to be larger than the preset filtering parameters, the divided defect area is input into a classification module so as to reduce the over-killing rate.

The number of the defect image samples trained by the segmentation model in the second defect detection model can be set between 300 and 500, the number is influenced by the number of defect types, the training data samples of the same defect are at least more than 30 pictures, and the training iteration number can be set between 600 and 1000.

The training effect can be obtained from a training process curve, the abscissa of which is the number of model training iterations, and the ordinate of which is the values of loss, nacc and pacc.

And obtaining a defect contour through a segmentation model in the second defect detection model, cutting the defect contour as a center into a graph with preset resolution, and inputting the graph into a classification module for detection. The preset resolution may be set according to the area of the minimum defect, for example, between 100 and 200.

The classification module may set two labels, namely good (OK) and bad (NG), and reference u indicates that the classification module obtains a defect detection result, as shown in fig. 8. The training iteration number of the classification module can be set between 1000 and 1200, and the number of training data can be set between 3000 and 10000.

The defect detection model matched with the third category is a third defect detection model, and the third defect detection model comprises a detection module and a classification module. The third category aims at a defect image sample with unclear large-area defect boundary but high pixel proportion in the whole picture, and the third defect detection model adopts a modeling mode of a detection module serial classification module. For example, a detection module is used to detect the defect position of the product defect image sample, and then a classification module classifies the detected area to reduce the over-killing rate.

For example, the defect types in fig. 9 are dirt, the detection module may position the dirt position first, the dirt range is shown by a rectangular frame, and the classification module determines the result of the detection module to obtain the defect detection result.

The processing manner of the first defect detection model, the second defect detection model, and the defect image sample by u is substantially the same as the defect detection manner in the embodiment of the defect detection method, and will not be described herein.

Step 303, determining whether the detection result is abnormal or not based on the detection result of the defect detection model and the label of the defect image sample.

The anomaly includes one or more of detection of overdischarge, false detection of defect type, detection of missed detection. Wherein, the product that the sign is actually good is killed to the overkill is judged as defective product. The product with the missing detection characterization being actually defective is judged to be good. The defect type misdetection characterizes a defect specific type inspection error.

Step 304, outputting a model optimization strategy based on the anomaly.

In some embodiments, if the anomaly includes a detection miss, various scenarios of the detection miss and their corresponding model optimization strategies may include:

1. the segmentation module of the defect detection model is not detected. For example, a defective region is identified in a label in the defective image sample, and a defective region is not labeled in the segmentation module. The model optimization strategy may be to add images that have failed to detect in the defect image sample.

For example, during quality inspection of image defect detection, it may be found that the segmentation module fails to detect, and an image with the failure to detect may be added to the defect image sample, so as to optimize the segmentation module in the defect detection model.

2. The classification module of the defect detection model is not used for detection. For example, the label of the defective image sample is defective, and the classification model detects that the defective image sample is defective. The model optimization strategy may be to add images that have failed to detect in the defect image sample.

For example, during quality inspection of image defect detection, the classification module may fail to detect, and an image with fail to detect may be added to the defect image sample, so as to optimize the classification module in the defect detection model.

3. And the defects are too small and too light to cause missed detection. The model optimization strategy may be to reduce the length threshold in the management and control parameters and to add a defect picture with defects smaller than a preset size to the defect image sample.

The control parameters are evaluation standards for marking defects based on the defect image samples, and the control parameters can comprise size thresholds of the defects, such as length, width, area and the like, the number of the defects on the same surface of the product and the distances between different defects on the same surface of the product.

Wherein, the length threshold T in the management and control parameter may be set as: the value range of T=kD, k is 0.8 to 0.9, D=d×a/A, wherein the pixel length of the defect is D, the field size is a, the camera resolution of shooting the defect image sample is A, and D is the actual length of the defect.

4. The defect is not shot in the image, resulting in missed detection. The model optimization strategy is to adjust shooting modes. For example, the defect shooting is clearer by adjusting the light source, shooting angle, camera mode and the like used for shooting.

In some embodiments, if the anomaly includes defect detection overducing, each type of situation of defect detection overducing and corresponding model optimization strategies include:

1. the segmentation module is overdcidal. For example, the defective region is detected by detecting a position in the defective image sample where the defective region is not identified by the segmentation model. The model optimization strategy may be to adjust the marked defect region in the defect image sample. For example, if there is a difference of 20% or more between the gray value of a certain region and the gray value of a normal portion in the defect image sample, the length of the region is 50% or more of the filtering parameter, and the defect image sample can be marked as a defect region.

The model optimization strategy may also be to add defective image samples for which the label is good. The segmentation module is usually not added with a defect image sample with a good label in the training process, but when the over-killing rate is serious, for example, exceeds a preset threshold value, the defect image sample with the good label can be added for training, and when the segmentation module is trained, the data ratio of the defect image sample with the good label to the defect image sample with the bad label can be set below 0.8.

2. The classification model is overdcidal. For example, the product corresponding to the defective image sample is good, and the output of the classification model is bad.

The model optimization strategy is to adjust the labels of the defective image samples. For example, in the defective image sample, if the difference between the gray value of the defective area and the gray value of the normal portion is a preset gray value, for example, more than 20%, the label of the defective image sample may be set as a defective product, and the condition may cause the classification model to be overdue, so that the preset gray value may be changed, and the preset gray value may be adjusted upwards.

The model optimization strategy may also be to add a picture of the kill to the defect image sample. For example, the same image of the same defect is augmented by at least 50 images to enhance the detection logic.

3. Negligible defect overdischarge. For example, the defect is very small and can be ignored, but still determined as a defective product. The model optimization strategy may include relaxing the control parameters, for example, 3 defects appear in the same original product, and the determination is defective, and may be changed to 5 defects appear in the same product, and the determination is defective.

In some embodiments, the defective condition of a product is that a single defect is that the length X of the defect is greater than the preset length X0, the width is greater than the preset width Y0, the area S is greater than the preset area S0", or that N0 or more defects are on the same surface, and the distance Δe is smaller than Δe0.

In some embodiments, if the imaging precision is B, in the case that the number of defects n=1 on the same surface of the same product, the following condition is satisfied to set the label of the defective image sample as a defective product:

x＝X/B＞X0/B，y＝Y/B＞Y0/B，s＝S/B2＞S0/B2；

if Δe=Δe/B > Δe0/B, the label of the defective image sample is set as defective if the number of defects N > N0 on the same surface of the same product.

4. Dust, wiping dirt and other external interference to cause false killing. The model optimization strategy can comprise the steps of setting control parameters based on the size, the number and the gray level of dust, and effectively filtering the dust.

If the good products have clean dirt, the over-killing rate of the model is improved, and the influence of the clean dirt on the model can be reduced by adding the static removing brush at the product feed inlet; the defective products are also misjudged as defective products by the defect detection model due to dust on the defective products, and the over-killing rate is improved, so that the influence of dust can be effectively reduced by adding the negative ion dust removal system into the product feed inlet.

In some embodiments, if the anomaly includes a defect type false detection, each type of situation of the defect type false detection and the corresponding model optimization strategy include:

1. optimizing the label, and updating the label of the defect image sample which is easy to misdetect the defect type. For example, the crush injury false detection is a corrosion point, and whether the crush injury in the label is wrongly marked as the corrosion point is checked.

2. And adding a data set corresponding to the confusing defects, and strengthening learning. For example, if the false detection of crush injury is corrosion point, the original data can be increased to 100 pieces each for 50 pieces each.

3. Switching to a higher-level neural network to realize classification, and selecting a higher training precision.

4. The classification model of the easily mixed defects is independently added, the recognition accuracy of the model on the two types of defects is enhanced, and the recognition accuracy of the two types of defects can be improved by 5 to 12 percent.

Step 305, adjusting the defect detection model based on the model optimization strategy.

According to the model optimization strategy, the electronic equipment can execute the model optimization strategy or output the model optimization strategy based on a human-computer interaction interface so as to adjust the defect detection model.

In addition, in the defect detection process, a new defect type may be found, so that a defect image sample corresponding to the new defect type may be collected, and then the defect image sample corresponding to the new defect type is input into the defect detection model for training, so as to optimize the defect detection model and improve the accuracy of the defect detection model in detecting the new defect.

Based on the same ideas of the defect detection method in the above-described embodiments, the present application also provides a defect detection apparatus that can be used to perform the above-described defect detection method. For ease of illustration, only those portions of the defect detection apparatus embodiments are shown in the schematic structural drawings that relate to the embodiments of the present application, and those skilled in the art will appreciate that the illustrated structures are not limiting of the apparatus and may include more or fewer components than illustrated, or may combine certain components, or a different arrangement of components.

As shown in fig. 10, the defect detecting apparatus includes an acquisition module 1001, a matching module 1002, and a detection module 1003. In some embodiments, the modules described above may be programmable software instructions stored in memory and executable by a processor call. It will be appreciated that in other embodiments, the modules may be program instructions or firmware (firmware) that are resident in the processor.

An obtaining module 1001, configured to obtain a product defect picture to be subjected to defect detection, and identify a detection category to which the product defect picture belongs;

a matching module 1002, configured to determine a defect detection model that matches the detection category; the defect detection model is a pre-trained defect detection model based on deep learning;

and the detection module 1003 is configured to input the product defect picture into the matched defect detection model, so as to obtain a defect detection result of the product.

Fig. 11 is a schematic diagram of an embodiment of an electronic device of the present application.

The electronic device 100 comprises a memory 20, a processor 30 and a computer program 40 stored in the memory 20 and executable on the processor 30. The steps of the above-described defect detection method embodiment, such as steps 101 through 104 shown in fig. 1, are implemented when the processor 30 executes the computer program 40.

By way of example, the computer program 40 may likewise be partitioned into one or more modules/units that are stored in the memory 20 and executed by the processor 30. The one or more modules/units may be a series of computer program instruction segments capable of performing particular functions for describing the execution of the computer program 40 in the electronic device 100. For example, it may be divided into an acquisition module 1001, a matching module 1002, and a detection module 1003 shown in fig. 10.

It will be appreciated by those skilled in the art that the schematic diagram is merely an example of the electronic device 100 and is not meant to be limiting of the electronic device 100, and may include more or fewer components than shown, or may combine certain components, or different components, e.g., the electronic device 100 may also include input-output devices, network access devices, buses, etc.

The processor 30 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor, a single-chip microcomputer or the processor 30 may be any conventional processor or the like.

The memory 20 may be used to store computer programs 40 and/or modules/units, and the processor 30 implements various functions of the electronic device 100 by running or executing the computer programs and/or modules/units stored in the memory 20, as well as invoking data stored in the memory 20. The memory 20 may mainly include a storage program area that may store an operating system, application programs required for at least one function (such as a sound playing function, an image playing function, etc.), and a storage data area; the storage data area may store data (such as audio data) created according to the use of the electronic device 100, and the like. In addition, the memory 20 may include high-speed random access memory, and may also include nonvolatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other nonvolatile solid state storage device.

The modules/units integrated with the electronic device 100 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as a stand alone product. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each method embodiment described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

In several embodiments provided in the present application, it should be understood that the disclosed electronic device and method may be implemented in other manners. For example, the above-described embodiments of the electronic device are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be other manners of division when actually implemented.

In addition, each functional unit in each embodiment of the present application may be integrated in the same processing unit, or each unit may exist alone physically, or two or more units may be integrated in the same unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.

It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the embodiments are to be considered in all respects as illustrative and not restrictive. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. Multiple units or electronic devices recited in the electronic device claims may also be implemented in software or hardware by means of one and the same unit or electronic device. The terms first, second, etc. are used to denote a name, but not any particular order.

Finally, it should be noted that the above embodiments are merely for illustrating the technical solutions of the present application and not for limiting, and although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made to the technical solutions of the present application without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method of defect detection, the method comprising:

obtaining a product defect picture to be subjected to defect detection;

identifying the detection category to which the product defect picture belongs;

determining a defect detection model matched with the detection category; the defect detection model is a pre-trained defect detection model based on deep learning;

and inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product.

2. The defect detection method of claim 1, wherein the detection category to which the product defect picture belongs is distinguished based on morphological features and positional features of defects in the product defect picture;

the identifying the detection category to which the product defect picture belongs comprises:

if the defects in the product defect picture are located at the preset positions, determining the detection category as a first category;

If the defects in the product defect picture meet the preset small-size condition, determining the detection category as a second category;

and if the defects in the product defect picture meet the preset large-size condition, determining the detection category as a third category.

3. The defect detection method of claim 2, wherein the first type of matched defect detection model includes a detection module and a segmentation module, and the inputting the product defect picture into the matched defect detection model, to obtain a defect detection result of the product, includes:

detecting a defect position in the product defect picture through the detection module;

inputting the defect position into the segmentation module to obtain a segmented defect region;

obtaining the size of the defect based on the defect area;

and if the size of the defect is in the preset good product range, outputting a defect detection result representing that the product is good.

4. The defect detection method of claim 2, wherein the second class of matched defect detection models includes a segmentation module and a classification module, and the inputting the product defect picture into the matched defect detection models to obtain the defect detection result of the product includes:

Processing the product defect picture through the segmentation module to obtain segmented defect areas;

obtaining a defect size based on the defect area;

and if the defect size is larger than a preset filtering parameter, inputting the divided defect areas into the classification module to obtain a defect detection result of the product.

5. The defect detection method of claim 2, wherein the third category-matched defect detection model includes a detection module and a classification module, and the inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product includes:

and inputting the defect position into the classification module to obtain a defect detection result of the product.

6. The defect detection method of any one of claims 1 to 5, wherein the training step of the defect detection model comprises:

obtaining a defect image sample belonging to a target detection category;

inputting the defect image sample belonging to the target detection category into a defect detection model matched with the target detection category to obtain a detection result of the defect image sample;

Determining whether the detection result is abnormal or not based on the detection result of the defect detection model and the label of the defect image sample; the abnormality includes one or more of detection of overdischarge, false detection of defect type, detection of omission;

outputting a model optimization strategy based on the anomaly;

and adjusting the defect detection model based on the model optimization strategy.

7. The defect detection method of claim 6, wherein the acquiring a defect image sample belonging to a target detection class comprises:

acquiring a product image sample belonging to a target detection category;

shielding the background of the product image sample to obtain a product image sample with the shielded background;

cutting the product image sample subjected to the background shielding according to a preset cutting method to obtain a cut image;

and adjusting the contrast of the clipping image to obtain the defect image sample.

8. A defect detection apparatus, the apparatus comprising:

the device comprises an acquisition module, a detection module and a display module, wherein the acquisition module is used for acquiring a product defect picture to be subjected to defect detection and identifying a detection category to which the product defect picture belongs;

the matching module is used for determining a defect detection model matched with the detection category; the defect detection model is a pre-trained defect detection model based on deep learning;

And the detection module is used for inputting the product defect picture into the matched defect detection model to obtain a defect detection result of the product.

9. An electronic device comprising a processor and a memory, wherein the memory is configured to store instructions, the processor configured to invoke the instructions in the memory, to cause the electronic device to perform the defect detection method of any of claims 1-7.

10. A computer readable storage medium storing computer instructions which, when run on an electronic device, cause the electronic device to perform the defect detection method of any of claims 1 to 7.