CN111507357A - Defect detection semantic segmentation model modeling method, device, medium and equipment - Google Patents
Defect detection semantic segmentation model modeling method, device, medium and equipment Download PDFInfo
- Publication number
- CN111507357A CN111507357A CN202010554557.5A CN202010554557A CN111507357A CN 111507357 A CN111507357 A CN 111507357A CN 202010554557 A CN202010554557 A CN 202010554557A CN 111507357 A CN111507357 A CN 111507357A
- Authority
- CN
- China
- Prior art keywords
- defect
- workpiece
- point cloud
- image
- cloud data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000007547 defect Effects 0.000 title claims abstract description 328
- 230000011218 segmentation Effects 0.000 title claims abstract description 104
- 238000001514 detection method Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 47
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims description 57
- 239000013598 vector Substances 0.000 claims description 23
- 239000002131 composite material Substances 0.000 claims description 20
- 238000013135 deep learning Methods 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 18
- 238000011176 pooling Methods 0.000 claims description 16
- 230000006835 compression Effects 0.000 claims description 14
- 238000007906 compression Methods 0.000 claims description 14
- 238000003860 storage Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 7
- 238000002372 labelling Methods 0.000 claims description 6
- 238000000354 decomposition reaction Methods 0.000 claims description 5
- YSIITVVESCNIPR-UHFFFAOYSA-N Troxipide Chemical compound COC1=C(OC)C(OC)=CC(C(=O)NC2CNCCC2)=C1 YSIITVVESCNIPR-UHFFFAOYSA-N 0.000 claims description 4
- 229960001341 troxipide Drugs 0.000 claims description 4
- 238000009795 derivation Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 description 14
- 238000012545 processing Methods 0.000 description 14
- 238000003786 synthesis reaction Methods 0.000 description 11
- 230000015572 biosynthetic process Effects 0.000 description 9
- 230000000007 visual effect Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000011002 quantification Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000012821 model calculation Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
Abstract
The invention discloses a semantic segmentation model modeling method for workpiece defect detection, which comprises the steps of obtaining multiple groups of original workpiece point cloud data for training, determining a defect synthetic image of each workpiece based on the original workpiece point cloud data, converting the original workpiece point cloud data into a two-dimensional depth map, respectively obtaining a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map, synthesizing the two-dimensional depth map, Scale1 and Scale2 based on an RGB three-channel color image to obtain a defect synthetic image, obtaining a defect label L abel image corresponding to the workpiece, constructing a semantic segmentation model, training and semantically iterating segmentation model parameters based on the multiple groups of workpiece defect synthetic images and corresponding defect labels L abel images to obtain a converged semantic segmentation model, solidifying the model layer and corresponding weight into a pb file after the semantic segmentation training is finished, and exporting the pb file as a workpiece defect image corresponding model file, so that the precision and detection rate of 3D type defects can be effectively improved.
Description
Technical Field
The invention relates to the technical field of industrial field defect detection, in particular to a defect detection semantic segmentation model modeling method, device, medium and equipment.
Background
In an industrial scene, the factory indexes of product parts are very strict, and the requirements provide guarantee for the product parts to play functional roles after being shipped. In industrial settings, the complex mechanical, acoustic, optical, electrical environments and numerous processes can damage the appearance of a product part, making it a defective product part.
Currently, there are two main ways of detecting product defects commonly adopted in the industry: (1) manual detection; (2) machine vision based detection method. The manual detection has the defects of low efficiency, high cost, easy fatigue caused by manual work and the like; in mass industrial production, the defect detection method based on machine vision is superior to the manual method in the aspects of accuracy, speed, cost and the like, so that the adoption of an intelligent detection means based on machine vision to replace the manual method is an inevitable trend of industrial quality detection.
In recent years, with the wide application of deep learning represented by a convolutional neural network in the fields of target detection and the like, researchers gradually begin to apply the detection method to the field of industrial defect detection, so that a computer can automatically learn the characteristics of an industrial defect mode from image data of an industrial product and establish a model, and intelligent industrial product detection is realized. Compared with the image extraction features which cannot be quantified by the traditional vision technology, the deep learning neural network can effectively perform optimization processing, and the accuracy and stability of visual detection of the workpiece defects can be improved to a certain extent.
The existing semantic segmentation model generally adopts training data, after a defect sample is scanned to obtain corresponding three-dimensional point cloud data and the three-dimensional point cloud data is converted into a gray image, the gray image and an artificially labeled label L abel image are directly merged and input into the semantic segmentation model for training.
Disclosure of Invention
The invention aims to provide a defect detection semantic segmentation model modeling method to solve the problems in the prior art.
In order to achieve the above object, the present invention provides a semantic segmentation model modeling method for workpiece defect detection, which is characterized by comprising the following steps:
acquiring multiple groups of original workpiece point cloud data for training, wherein the workpieces comprise specified defect types;
determining a defect composite image for each workpiece based on the original workpiece point cloud data: converting the original workpiece point cloud data into a two-dimensional depth map; respectively acquiring a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map; synthesizing the two-dimensional depth map, the Scale1 and the Scale2 based on an RGB three-channel color image to obtain the defect synthesized image;
acquiring a defect label L abel image corresponding to the workpiece:
constructing a semantic segmentation model, training based on a plurality of groups of workpiece defect synthetic images and corresponding defect labeling L abel images, and obtaining a converged semantic segmentation model after iterating parameters of the semantic segmentation model;
and after the semantic segmentation training is finished, solidifying the model layer and corresponding weight as a pb file, and exporting the pb file as a deep learning pb model file corresponding to the workpiece defect image.
As a preferred scheme, acquiring a first gradient map Scale1 corresponding to the two-dimensional depth map based on a Sobel operator;
and/or acquiring a second gradient map Scale2 corresponding to the two-dimensional depth map based on an L aplace operator.
Further, converting the original workpiece point cloud data into a two-dimensional depth map via a three-dimensional point cloud projection function;
as a preferred scheme, the defect marking L abel image is determined based on the original workpiece point cloud data, the defect marking L abel image comprises deleting a defect area point cloud in the original workpiece point cloud data and storing the defect area point cloud as new workpiece point cloud data, performing difference on the new workpiece point cloud data and the original workpiece point cloud data to obtain workpiece defect area point cloud data, combining the workpiece defect area point cloud data and the new workpiece point cloud data to obtain workpiece defect image data, and marking the workpiece defect image data to obtain a defect marking L abel image.
As a preferred scheme, obtaining point cloud data of the defect area of the workpiece after performing difference on point cloud data of a new workpiece and point cloud data of an original workpiece through Matlab;
as a preferred scheme, the semantic segmentation model comprises a feature extraction network and a feature prediction network which are connected in sequence; the feature extraction network is used for predicting the occurrence position of the defect area and comprises a feature extraction layer, a feature compression layer, a feature flattening layer and a feature classification layer;
extracting the features of the image data according to the feature extraction layer to obtain a feature map of the image data;
compressing the feature map according to the feature compression layer and outputting a feature vector of the feature map;
performing convolution kernel decomposition on the feature vector according to the feature flattening layer and outputting an enumeration vector of the feature map;
inputting the enumeration vectors into the feature classification layer to independently predict different types of defects and acquire the positions of defect regions;
the feature prediction network is used for classifying each pixel point in the defect region determined by the feature extraction network, and outputting a defect prediction L abel graph of the workpiece after positioning and segmenting the defect region.
Further, in the feature extraction network, the feature extraction layer includes a plurality of convolution layers connected in sequence;
the characteristic compression layer comprises a plurality of convolution layers and pooling layers which are alternately arranged;
the feature flattening layer comprises a plurality of one-dimensional convolution layers;
the feature classification layer comprises a plurality of fully connected layers;
the feature prediction network comprises a plurality of pooling layers and a full connection layer which are connected in sequence.
The invention also provides a workpiece defect detection method based on the semantic segmentation model, which comprises the following steps: acquiring image data of a work to be detected, wherein the image data of the work to be detected is a defect composite image of the work to be detected, and the acquiring of the defect composite image of the work to be detected comprises the following steps: acquiring original workpiece point cloud data of a workpiece to be detected; converting the original workpiece point cloud data into a two-dimensional depth map; respectively acquiring a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map; synthesizing the two-dimensional depth map, the Scale1 and the Scale2 based on an RGB three-channel color image to obtain the defect synthesized image;
inputting the defect synthetic image of the workpiece to be detected into a semantic segmentation model generated by the method shown in the previous item, and acquiring a 3D defect prediction image of the workpiece to be detected;
and carrying out online defect detection and quantification on the workpiece to be detected based on the 3D defect prediction graph.
The invention also provides a workpiece defect detection device based on the semantic segmentation model, which comprises the following steps:
the system comprises an original workpiece point cloud data acquisition module, a defect detection module and a defect classification module, wherein the original workpiece point cloud data acquisition module is used for acquiring a plurality of groups of original workpiece point cloud data for training, and the workpieces comprise specified defect types;
a defect composite image acquisition module for determining a defect composite image for each workpiece based on the original workpiece point cloud data, comprising: the two-dimensional depth map acquisition unit is used for converting the original workpiece point cloud data into a two-dimensional depth map; a gradient map acquiring unit, configured to acquire a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map; a synthesizing unit, configured to synthesize the two-dimensional depth map, the Scale1, and the Scale2 based on an RGB three-channel color image to obtain the defect synthesized image;
a defect label L abel image obtaining module, configured to obtain a defect label L abel image corresponding to the workpiece:
the semantic segmentation model building module is used for training and iterating parameters of the semantic segmentation model based on the plurality of groups of workpiece defect synthetic images and the corresponding defect label L abel images to obtain a converged semantic segmentation model;
and the deep learning pb model file derivation module is used for solidifying the model layer and deriving the pb file as a deep learning pb model file corresponding to the workpiece defect image.
In order to achieve the above object, the present invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the aforementioned method when executing the computer program.
In order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the aforementioned method.
Through the technical scheme, the invention has the following beneficial effects:
first, the semantic segmentation model modeling method, the semantic segmentation model modeling device, and the storage medium for workpiece defect detection disclosed in the present application use a three-channel fusion technology to synthesize a workpiece defect synthetic image, and use a corresponding defect label L abel image as training data of a semantic segmentation model, compared with using a defect gray image and a corresponding defect label as training data, the defect image extracted based on three-channel synthesis has more abundant features, the corresponding defect region is displayed more visually, the feature vector obtained by a model feature extraction network has higher dimensionality and more obvious relative difference, the deep learning neural network analyzes the image features more sufficiently, which is beneficial to improving the detection rate and segmentation accuracy of 3D defects, and improving the defect prediction accuracy of the semantic segmentation model.
Secondly, according to the semantic segmentation model modeling method, device and storage medium for workpiece defect detection, the defect label L abel image is obtained by processing point cloud data, compared with the defect label of the pixel level obtained by acquiring the workpiece image by a direct camera, the method can identify the defect area with the size of 0.015mm, and when the method is matched with the workpiece defect synthetic image to be used for semantic segmentation model training, the model prediction precision is better.
Thirdly, the semantic segmentation model modeling method, the semantic segmentation model modeling device and the storage medium for workpiece defect detection disclosed by the application are used for designing a semantic segmentation model structure, a feature extraction network is used for classifying and roughly positioning existing defects (namely roughly positioning the defects to the defect positions), a feature prediction network is used for positioning and segmenting a defect region determined on the feature extraction network based on pixel levels (namely segmenting the defect region at high precision), and the semantic segmentation model modeling method, the semantic segmentation model modeling device and the storage medium are combined with the training data to be used for semantic segmentation model modeling, so that the model calculation amount can be effectively reduced, and the visual detection of workpiece defects can be quickly and accurately realized.
In addition, the defect detection method based on the semantic segmentation model comprises the steps of preprocessing an input workpiece image to be detected, namely inputting a semantic segmentation model into a defect synthetic image of the workpiece to be detected, calling a defect prediction model pb file after preprocessing the defect synthetic image of the workpiece to be detected, so that the online prediction and quantification of the 3D defects of the workpiece are realized, the defect form representation is more visual, and a better detection effect is obtained.
Drawings
FIG. 1 is a flowchart illustrating a semantic segmentation model modeling method for workpiece defect detection according to an embodiment;
FIG. 2 is a graph of the model test effect in the embodiment of FIG. 1;
FIG. 3 is a frame diagram of a semantic segmentation defect detection model;
FIG. 4 is a flowchart illustrating a defect detection method based on a semantic segmentation model according to a third embodiment;
FIG. 5(a) is a defect synthesized image of a workpiece to be detected, and FIG. 5(b) is a defect prediction image of the workpiece to be detected, which is obtained by a defect detection method based on a semantic segmentation model;
FIG. 6 is a block diagram of an embodiment of a semantic segmentation model modeling apparatus for defect detection according to the present application;
FIG. 7 is a hardware architecture of an embodiment of a computer device of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention in any way.
Like reference numerals refer to like elements throughout the specification. The expression "and/or" includes any and all combinations of one or more of the associated listed items. In the drawings, the thickness, size, and shape of an object have been slightly exaggerated for convenience of explanation. The figures are purely diagrammatic and not drawn to scale.
It will be further understood that the terms "comprises," "comprising," "includes," "including," "has," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, integers, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, integers, operations, elements, components, and/or groups thereof.
The terms "substantially", "about" and the like as used in the specification are used as terms of approximation and not as terms of degree, and are intended to account for inherent deviations in measured or calculated values that would be recognized by one of ordinary skill in the art.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Example one
As shown in fig. 1, the present application provides a semantic segmentation model modeling method for workpiece defect detection, which includes the following steps:
s1, acquiring multiple groups of original workpiece point cloud data for training, wherein the workpieces comprise specified defect types;
s2 determining a defect composite image for each workpiece based on the original workpiece point cloud data;
s21, converting the original workpiece point cloud data into a two-dimensional depth map;
s22, respectively acquiring a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map;
s23 synthesizing a two-dimensional depth map, Scale1 and Scale2 based on the RGB three-channel color image to obtain a defect synthesized image;
s3, acquiring a defect label L abel image corresponding to the workpiece:
s4, constructing a semantic segmentation model, namely setting initialization parameters, a loss function and an optimizer in the model, inputting a plurality of groups of workpiece defect synthetic images and corresponding defect labeling L abel images into a semantic segmentation network, and iterating model parameters to obtain a converged semantic segmentation model;
s5, after model training is finished, the solidified model layer and the corresponding weight are pb files, and the pb files are exported to be used as deep learning pb model files corresponding to the workpiece defect images.
According to the semantic segmentation model modeling method for workpiece defect detection, a workpiece defect synthetic image synthesized by a three-channel fusion technology is adopted, corresponding defect labels L abel images are used as training data of a semantic segmentation model, compared with a defect gray image and corresponding defect labels which are used as training data, the defect image extracted based on three-channel synthesis has rich characteristics, corresponding defect regions are displayed visually, a feature vector obtained by a model feature extraction network has high dimensionality and obvious relative difference, the deep learning neural network is sufficient in image feature analysis, the detection rate and segmentation precision of 3D defects are improved, and the defect prediction precision of the semantic segmentation model is improved.
For data acquisition of a workpiece for training, in step S1, a line laser may be used to scan and acquire a point cloud data pcd file, in this embodiment, a L MI gobcator 2350D line laser is used to perform original workpiece point cloud data acquisition, and more than 200 training samples are acquired, so that a visual detection platform required for data acquisition is conveniently built, and simply includes only one line laser and is not interfered by an external light source, so that complexity and uncertainty of a 2D camera and backlight visual detection method are reduced to a certain extent, and in addition, a 2D camera may be used to cooperate with multi-light source replacement line laser scanning to acquire workpiece point cloud data point cloud processing, for example, four strip light sources are distributed at four positions, i.e., four bar light sources are distributed at four directions, the light sources are periodically controlled to be turned on and off, workpiece images at four directions, i.e., workpiece images at four directions, and then 3D defect forms of the workpiece are highlighted.
In the step S21, the workpiece point cloud data pcd file may be converted into a two-dimensional depth map I according to a three-dimensional point cloud projection function, where the two-dimensional depth map has a corresponding pixel value span of 0-255, and the larger the pixel value is, the larger the corresponding three-dimensional data Z value is; in this embodiment, a normalization processing mode is adopted, and a specific conversion function formula is as follows:
wherein D is the corresponding data of the point cloud, N is the D data dimension, and fix () is the rounding operation
In step S22, let the convolution of the two-dimensional depth map I in the x direction be Gx, the convolution in the y direction be Gy, the gradient map under the corresponding algorithm be G, and the Scale1 gradient map calculation method is shown in the following formula;
in step S23, the two-dimensional depth map is recorded as I having a partial derivative of ∂ x in the x direction and a partial derivative of ∂ y in the y direction, the gradient map under the corresponding algorithm is Ʌ, and the calculation method of the Scale2 gradient map is shown in the following formula;
in step S24, synthesizing a two-dimensional depth map, a Scale1 gradient map, and a Scale2 gradient map according to the RGB three-channel color image, which are recorded as: a defect composite image; in this embodiment, a three-channel synthesis technique is used to synthesize a defect synthesis image, and the calculation formula is: i = I1+ I2+ I3, where I1, I2, and I3 correspond to two-dimensional data of the two-dimensional depth map, Scale1, and Scale2, respectively, and I corresponds to the post-synthesis defect synthesis image.
When the step S3 obtains the defect label L abel image corresponding to the workpiece, it may be selected to obtain a workpiece image based on a camera and obtain a corresponding defect label L abel image based on direct labeling of the synthesized defect image, as a preferred scheme, in this embodiment, the defect label L abel image is determined based on the original workpiece point cloud data, and compared with the defect label of the pixel level obtained by directly obtaining the workpiece image or the synthesized defect image by a camera, it is able to identify a defect area of 0.015mm in size, and when the synthesized defect image is used for semantic segmentation model training in cooperation with the workpiece defect, the model prediction accuracy is also better.
Determining the defect annotation L abel image based on the original workpiece point cloud data specifically includes the steps of:
s31, deleting the point cloud of the defect area in the original workpiece point cloud data and storing the point cloud of the defect area as new workpiece point cloud data;
s32, performing difference on the new workpiece point cloud data and the original workpiece point cloud data to obtain workpiece defect area point cloud data;
s33 merging the point cloud data of the workpiece defect area and the point cloud data of the new workpiece to obtain the image data of the workpiece defect;
s34, acquiring a defect label L abel image corresponding to the workpiece defect image data;
in the step S31, the software 3DReshaper Application is used for opening the workpiece point cloud data, at the moment, the coordinate system corresponding to the three-dimensional point cloud data changes, the original three-dimensional point cloud XYZ data are proportionally scaled to the values corresponding to the coordinate system adapted to the software 3DReshaper Application, and then the point cloud data at the workpiece defect position are manually selected and deleted to be stored as a new workpiece point cloud pcd file.
In step S32, performing difference calculation on the point cloud pcd files before and after the defect area is eliminated through Matlab software, that is, performing difference calculation on the original workpiece point cloud and the new workpiece point cloud to obtain XYZ values of pixels in the defect area, and then obtaining pixel row and column coordinates corresponding to the X and Y directions through scanning precision conversion (Z-axis direction data is not used in the process); in one embodiment, according to a line laser scanning rule, the scanning interval of two points of a three-dimensional point cloud in the X direction is 0.01mm, and the scanning interval corresponding to the Y direction is 0.02mm, so that row-column pixel values corresponding to a workpiece defect area are obtained;
s33 merging the point cloud data of the workpiece defect area and the point cloud data of the new workpiece to obtain the image data of the workpiece defect;
in one embodiment, the original workpiece point cloud data and the new workpiece point cloud data acquired based on L MI godator have a scanning precision of 0.015mm/step in the XYZ direction, and for the point cloud data at the workpiece defect, the corresponding image pixel coordinates (x, y) are converted by knowing the corresponding three-dimensional point cloud coordinates (XYZ) and using the following formula:
wherein, X0And Y0Corresponding to the initial point cloud coordinates in the X and Y directions, adding the point cloud data of the workpiece defect position to the corresponding position in the new workpiece point cloud data according to the converted coordinates to obtain the workpiece defect image data
S34, setting the pixel value of a defect-free area in the workpiece defect image data to be 0, setting the pixel value of the point cloud data of the defect area in the workpiece defect image to be different values according to different defect types, and acquiring a defect label L abel image corresponding to the workpiece defect image data;
presetting two types of defects including a concave defect and a tilted defect, setting pixel values corresponding to a workpiece defect area as 1 and 2 (corresponding to the two types of defects of the concave defect and the tilted defect) and setting the pixel value of an area without the defect as 0 for workpiece defect image data, and thus obtaining a defect label L abel image corresponding to the workpiece defect image data.
In step S4, a common semantic segmentation model may be selected for training, and after a deep learning semantic segmentation network framework is built, a plurality of defect synthetic images and defect L abel images corresponding thereto are input with set model parameters and training steps to obtain a deep learning pb model file corresponding to a workpiece defect image.
In order to verify the effect of the semantic segmentation model modeling method for workpiece defect detection, 6 popular semantic segmentation models are selected for summarizing, three-channel synthesized defect synthetic images and defect label images are adopted for training and predicting for each model one by one, the model test effect is shown in FIG. 2, mIoU and fps are introduced as model test performance indexes, wherein the time consumed by a single image model is represented by the fps on the abscissa, the precision is represented by the mIoU on the ordinate, and the smaller the numerical value on the abscissa is, the shorter the model response time is represented; the larger the ordinate value is, the higher the model precision is, and as shown in fig. 2, the semantic segmentation model modeling method for detecting the workpiece defects provided by the application has relatively good effect.
Example two
The defect types of the workpieces for training can be preset to be one or more types according to requirements, and the model design can set the types and the number of the models in a user-defined mode, namely a user-defined defect segmentation network is designed. On the basis of the first embodiment, the semantic segmentation model is continuously further designed, and the deep learning semantic segmentation defect detection model built by the method comprises a feature extraction network and a feature prediction network which are sequentially connected;
the characteristic extraction network is used for predicting the occurrence position of the defect area and comprises a characteristic extraction layer, a characteristic compression layer, a characteristic flattening layer and a characteristic classification layer;
extracting the features of the image data according to the feature extraction layer to obtain a feature map of the image data;
compressing the feature map according to the feature compression layer and outputting a feature vector of the feature map;
performing convolution kernel decomposition on the feature vector according to the feature flattening layer and outputting an enumeration vector of the feature map;
inputting the enumeration vectors into a feature classification layer to independently predict different types of defects and obtain the positions of defect areas;
the characteristic prediction network is used for positioning and segmenting the defect region determined by the characteristic extraction network, classifies each pixel point in the defect region determined by the characteristic extraction network and outputs a defect prediction L abel graph of the workpiece;
and detecting the defects of the workpiece based on the defect prediction L abel graph.
The semantic segmentation model is designed on the basis of the training data, the feature extraction network is used for classifying and roughly positioning the existing defects (namely roughly positioning the defects to the positions), and the feature prediction network is used for positioning and segmenting the defect regions determined on the feature extraction network based on the pixel level (namely segmenting the defect regions at high precision), so that the model calculation amount can be effectively reduced, and the visual detection of the defects of the workpiece can be quickly and accurately realized.
As a preferred scheme, in the feature extraction network, the feature extraction layer comprises a plurality of convolution layers which are connected in sequence;
the characteristic compression layer comprises a plurality of convolution layers and pooling layers which are alternately arranged; the characteristic flattening layer comprises a plurality of one-dimensional convolution layers; the feature classification layer includes a plurality of fully connected layers. The convolutional layer functions in the depth extraction of image features, including color, texture, spatial information, and the like; the pooling layer is used for compressing the input feature map, so that on one hand, the dimension of the feature map is reduced, the network calculation complexity is simplified, and on the other hand, the feature compression is carried out, and the main features of the image are extracted; the full-link layer is used for linking all image features and sending output values to the classifier.
Furthermore, in the feature extraction network, the number of fully connected layers is set corresponding to the defect type. Each full connection layer is used for connecting image characteristics of one type of defects, and a plurality of full connection layers are arranged in series or in parallel in the characteristic extraction network and used for classifying different types of defects.
Preferably, the feature prediction network comprises a plurality of pooling layers and a fully-connected layer which are connected in sequence. The feature extraction network disclosed by the application realizes deep extraction and classification of different types of defect features, but only predicts the defect occurrence positions, and further classifies each pixel point in a defect area through the feature prediction network, so that subsequent defect quantification processing is facilitated.
As a preferable scheme, the training of the semantic segmentation model can be carried out based on a gradient descent algorithm, wherein L oss calculation function is softmax cross entropy.
During the network model training, the maximum iteration step number is taken as large as possible, the loss value is taken as small as possible, the model parameters are trained to the maximum degree, and the lower limit of the model convergence is obtained at the same time. In the embodiment, the model loss function is calculated by adopting softmax cross entropy (softmax _ cross _ entropy _ with _ logits), wherein the model loss function comprises the steps of converting logits into probabilities, calculating cross entropy loss, firstly converting the logits into the probabilities by using softmax, then calculating the value of each element in the logits, and the corresponding calculation function relationship is as follows:
prob is the probability of corresponding logits elements, and each element is not less than 0 and the sum is 1, so a logits probability distribution is constructed; then, the formula is calculated according to the cross entropy, namely:
where y' is the true label value and y is the logits probability distribution.
The defect detection method based on the semantic segmentation model shown in the present application is further described below by using the preset defect types as two defects of dishing and tilting, and the training data as a workpiece defect synthetic image and a defect label L abel image corresponding to the workpiece defect synthetic image.
Fig. 3 is a deep learning semantic segmentation defect detection model framework diagram constructed by the present invention, wherein a trained output model network framework includes a feature extraction network and a feature prediction network, the feature extraction network includes 14 convolutional layers, 5 pooling layers and 2 full-link layers, the feature prediction network includes 5 pooling layers and 1 full-link layer, in the illustrated embodiment, the front end of the feature extraction network is a five convolutional layers, and simultaneously performs deep feature extraction on an input defect synthetic image and a corresponding L abel label image, each layer corresponds to different sizes and numbers of convolutional cores, taking an RGB image (corresponding to a size of 1600 × 1200 × 3) as an example, simultaneously performs scaling processing on a defect synthetic image for training and a label L abel image, so as to reduce the time consumption of model training, and the processing results corresponding to each convolutional layer after scaling to 801 × 801 and 801 are shown in table 1:
table 1 feature extraction front-end network image data changes
The model convolutional layer output image size corresponds to the length L, width W and depth H of the image, where the length L and width W are calculated as L' = (L-F +2 × P)/S +1 as shown in the following formula
W’=(W-F+2×P)/S + 1
Wherein L, W corresponds to the convolutional layer input image size, L ', W' corresponds to the convolutional layer output image size, F corresponds to the convolutional kernel size, P is the step size, the depth of the output image is equal to the number of convolutional kernels, and the convolutional layer processing image size is the rounding-down operation.
The middle end of the feature extraction network in the application is of a convolution layer and pooling layer composite structure, the feature vector dimension is reduced while the image feature extraction capability is enhanced, the training capability of the model is improved, the output is a feature vector (corresponding to an array of a feature map), and the corresponding change of a defect synthetic image and a label L abel image is shown in table 2:
table 2 feature extraction middle end network image data changes
The model convolutional layer output image size corresponds to the length L, width W and depth H of the image, where the length L and width W are calculated as shown in L' = (L-F)/S + 1
W’=(W-F)/S + 1
Wherein L, W corresponds to the input image size of the convolutional layer, L ', W' corresponds to the output image size of the convolutional layer, F corresponds to the size of the convolutional kernel, the depth of the output image is equal to the number of the convolutional kernels, and the processing image size of the pooling layer is the rounding-up operation.
The feature extraction network back end in the design of the present invention includes 4 consecutive small convolution layers to replace the large convolution layer, and aims to reduce the number of network parameters by convolution kernel decomposition, and output as an enumeration vector (corresponding to a one-dimensional array of feature map vectors), for example, 1 convolution kernel of 5 × 5 has 25 parameters, while 2 convolution kernels of 3 × 3 have only 18 parameters, 1 convolution kernel of 1 × 5 and 5 × 1 has only 10 parameters, and the corresponding changes of defect synthesized image and label L abel image are shown in table 3:
table 3 feature extraction backend network image data changes
Finally, the feature extraction network is provided with 2 fully-connected layers corresponding to the two types of defects, the fully-connected layers play a role of a classifier in the whole design network model, two defect types are considered, the invention relates to the independent prediction of the 2 fully-connected layers, and each layer comprises 48 × 48 feature maps with the length of 48 × 24=55926 feature maps and is used for classifying the two types of defects.
The feature extraction network has implemented depth extraction and classification of two defect features, but only predicts the defect occurrence positions, and before this basis, the present application further adds a feature prediction network, which in this embodiment includes 5 pooling layers and 1 full-link layer, and aims to classify each pixel point in a defect region, so as to facilitate subsequent defect quantization processing, where the output of the feature prediction network is a label L abel image corresponding to an input image, and the size change of the corresponding input image is shown in table 4:
table 4 feature prediction network image data changes
The feature prediction network output comprises 1 × 1 × 128=128 feature maps, and comprises classification attributes of each pixel of the defect region, so that the positioning and segmentation effects of the defect region can be realized.
EXAMPLE III
On the basis of the first embodiment and/or the second embodiment, corresponding to the training data, the input of the semantic segmentation model is further limited, and a defect detection method based on the semantic segmentation model is further provided.
As shown in fig. 4, in the defect detection method based on the semantic segmentation model according to this embodiment, point cloud data of workpiece defects is obtained through line laser scanning, then a point cloud depth map and a corresponding gradient map are obtained, a three-channel fusion technique is adopted to synthesize a 3D defect sample image of a workpiece, and then a deep learning semantic segmentation model generated in the embodiment is invoked to perform feature extraction on the synthesized image, so as to further improve the detection rate of such defects.
The method for acquiring the defect composite image of the workpiece to be detected comprises the following steps:
c1, acquiring original workpiece point cloud data of the workpiece to be detected;
in the step, a line laser is adopted to scan and obtain a point cloud data pcd file of the point cloud data pcd file, in one embodiment, a line laser of L MIG scanner 2350D type is used for collecting original workpiece point cloud data of a workpiece to be detected, and therefore, a visual detection platform required by data collection is convenient to set up, the visual detection platform only comprises one line laser, and is not interfered by an external light source, complexity and uncertainty of visual detection of a 2D camera and a backlight source are reduced to a certain extent, in addition, the point cloud processing of the workpiece point cloud data can be obtained by matching the 2D camera with multi-light source replacement line laser scanning, for example, four strip light sources are distributed at four positions of the upper part, the lower part, the left part and the right part of the workpiece, the light sources are periodically controlled to be turned on and off, workpiece images in the upper part, the left part and the right part.
C2 determines a composite image of the defect of the workpiece to be inspected based on the raw workpiece point cloud data.
C21 converting the original workpiece point cloud data into a two-dimensional depth map;
in the step, a workpiece point cloud data pcd file can be converted into a two-dimensional depth map I according to a three-dimensional point cloud projection function, the span of pixel values corresponding to the two-dimensional depth map is 0-255, and the larger the pixel value is, the larger the Z value of the corresponding three-dimensional data is represented; in this embodiment, a normalization processing mode is adopted, and a specific conversion function formula is as follows:
wherein D is the data corresponding to the point cloud, N is the D data dimension, and fix () is the rounding operation.
C22 respectively acquiring a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map;
as a preferable scheme, a first gradient map Scale1 corresponding to the two-dimensional depth map can be obtained based on a Sobel operator, and a second gradient map Scale2 corresponding to the two-dimensional depth map can be obtained based on a L aplace operator.
Recording the convolution of the two-dimensional depth image I in the x direction as Gx, the convolution of the two-dimensional depth image I in the y direction as Gy, wherein the gradient image under the corresponding algorithm is G, and the calculation mode of the Scale1 gradient image is shown in the following formula;
recording a two-dimensional depth map as that a partial derivative of I in the x direction is ∂ x, a partial derivative of I in the y direction is ∂ y, a gradient map under a corresponding algorithm is Ʌ, and a calculation mode of a Scale2 gradient map is shown in the following formula;
c23 synthesizes the two-dimensional depth map, Scale1 and Scale2 based on the RGB three-channel color image to obtain a defect synthesized image.
Synthesizing a two-dimensional depth map, a Scale1 gradient map and a Scale2 gradient map according to the RGB three-channel color image, and recording as follows: a defect composite image; in this embodiment, a three-channel synthesis technique is used to synthesize a defect synthesis image, and the calculation formula is: i = I1+ I2+ I3, where I1, I2, and I3 correspond to two-dimensional data of the two-dimensional depth map, Scale1, and Scale2, respectively, and I corresponds to the post-synthesis defect synthesis image.
Acquiring a defect composite image of a workpiece to be detected, calling a defect prediction model pb file, taking the defect composite image of the workpiece to be detected as input of the defect prediction model pb file, obtaining a 3D defect prediction image of the workpiece to be detected, then acquiring a ratio of pixel length to real length by adopting a camera calibration technology, learning the area of pixels in a defect L abel area by statistics depth, converting the ratio into a real unit area (mm ^2), and further completing online defect detection and quantification, wherein the defect composite image obtained based on 3D point cloud data of the workpiece to be detected is more intuitive in defect forms such as 'depression' and 'tilting', and the like, and the defect composite image obtained based on the 3D point cloud data of the workpiece to be detected is more intuitive in defect form, and the implementation effect image (namely the 3D defect prediction image of the workpiece to be detected) corresponding to the application is shown in figure 5(b), and the corresponding defect can be predicted and2)。
according to the defect detection method based on the semantic segmentation model, the three-dimensional point cloud obtained by scanning of the line laser is fused through three RGB channels to obtain a defect synthetic image, the defect area of the workpiece is visually represented in the defect synthetic image, then the defect area is predicted and quantized by utilizing the deep learning semantic segmentation model network, three-dimensional data is effectively converted into two-dimensional image data, and dimensionality and redundancy of data calculation are reduced to a certain extent. Due to the fact that the defect area can be rapidly and precisely segmented, single-piece prediction time can be shorter than 100ms by matching with the high-performance display card, and the on-line real-time detection requirement can be effectively met.
Example four
As shown in fig. 6, the present application further provides a semantic segmentation model modeling apparatus 10 for workpiece defect detection, which includes:
an original workpiece point cloud data acquisition module 11, configured to acquire multiple sets of original workpiece point cloud data for training, where the workpieces include specified defect types;
a defect composite image acquisition module 12 for determining a defect composite image for each workpiece based on the original workpiece point cloud data, comprising: the two-dimensional depth map acquisition unit is used for converting the original workpiece point cloud data into a two-dimensional depth map; a gradient map acquiring unit, configured to acquire a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map; a synthesizing unit, configured to synthesize the two-dimensional depth map, the Scale1, and the Scale2 based on an RGB three-channel color image to obtain the defect synthesized image;
a defect label L abel image obtaining module 13, configured to obtain a defect label L abel image corresponding to the workpiece:
a semantic segmentation model building module 14, which trains and iterates the semantic segmentation model parameters based on the plurality of groups of workpiece defect synthetic images and the corresponding defect label L abel images to obtain a converged semantic segmentation model;
and the deep learning pb model file export module 15 is used for solidifying the model layer and exporting the pb file as a deep learning pb model file corresponding to the workpiece defect image, wherein the corresponding weight of the pb file is a pb file.
The semantic segmentation model modeling device 10 for workpiece defect detection disclosed by the application adopts a workpiece defect synthetic image synthesized by a three-channel fusion technology, and uses corresponding defect label L abel images as training data of a semantic segmentation model, compared with a defect gray image and corresponding defect label training data, the defect image extracted based on three-channel synthesis has rich characteristics, corresponding defect regions are displayed visually, a feature vector obtained by a model feature extraction network has high dimensionality and obvious relative difference, and the deep learning neural network analysis image characteristics are sufficient, so that the detection rate and the segmentation precision of 3D defects are improved, and the defect prediction precision of the semantic segmentation model is improved.
As a preferred scheme, the defect label L abel image acquisition module is used for determining a defect label L abel image of each workpiece based on original workpiece point cloud data for training, and comprises a new workpiece point cloud data acquisition unit, a workpiece defect area point cloud data acquisition unit and a defect label L abel image acquisition unit, wherein the new workpiece point cloud data acquisition unit is used for deleting and storing defect area point clouds in the original workpiece point cloud data into new workpiece point cloud data, the workpiece defect area point cloud data acquisition unit is used for carrying out difference on the new workpiece point cloud data and the original workpiece point cloud data to acquire workpiece defect area point cloud data, the workpiece defect image data acquisition unit is used for combining the workpiece defect area point cloud data and the new workpiece point cloud data to acquire workpiece defect image data, and the defect label L abel image corresponding to the workpiece defect image data is acquired by setting a pixel value of a defect area which does not generate defects in the workpiece defect image data to be 0 and setting a pixel value to be different values according to different defect types in the defect area point cloud data.
As a preferable scheme, in the gradient map obtaining subunit, a first gradient map Scale1 corresponding to the two-dimensional depth map is obtained based on a Sobel operator, and a second gradient map Scale2 corresponding to the two-dimensional depth map is obtained based on a L aplace operator.
In a preferred embodiment, in the two-dimensional depth map acquisition subunit, the original workpiece point cloud data is converted into a two-dimensional depth map through a three-dimensional point cloud projection function.
As a preferred scheme, in the workpiece defect area point cloud data acquisition subunit, the new workpiece point cloud data and the original workpiece point cloud data are subjected to difference by Matlab to acquire the workpiece defect area point cloud data.
As a preferred scheme, the semantic segmentation model comprises a feature extraction network and a feature prediction network which are connected in sequence;
the feature extraction network is used for predicting the occurrence position of the defect area and comprises a feature extraction layer, a feature compression layer, a feature flattening layer and a feature classification layer: the characteristic extraction layer comprises a plurality of convolution layers which are sequentially connected and is used for extracting the characteristics of the image data according to the characteristic extraction layer to obtain a characteristic diagram of the image data; the characteristic compression layer comprises a plurality of convolution layers and pooling layers which are alternately arranged and is used for compressing the characteristic diagram according to the characteristic compression layer and outputting a characteristic vector of the characteristic diagram; the feature flattening layer comprises a plurality of one-dimensional convolution layers and is used for performing convolution kernel decomposition on the feature vectors according to the feature flattening layer and outputting enumeration vectors of the feature map; the characteristic classification layer comprises a plurality of full-connection layers and is used for inputting enumeration vectors into the full-connection layers so as to independently predict different types of defects and obtain the positions of defect areas;
the feature prediction network is used for positioning and segmenting the defect region and comprises a plurality of pooling layers and a full-connection layer which are sequentially connected, and the feature prediction network is used for classifying each pixel point in the defect region determined by the feature extraction network and outputting a corresponding workpiece defect prediction L abel graph of the workpiece to be detected.
Furthermore, in the feature extraction network, the feature extraction layer comprises a plurality of convolution layers which are connected in sequence; the characteristic compression layer comprises a plurality of convolution layers and pooling layers which are alternately arranged; the characteristic flattening layer comprises a plurality of one-dimensional convolution layers; the characteristic classification layer comprises a plurality of full connection layers;
further, the feature prediction network comprises a plurality of pooling layers and a full connection layer which are connected in sequence.
As a preferred scheme, the semantic segmentation model building module 13 further includes a training submodule, configured to perform semantic segmentation network training based on a plurality of groups of workpiece defect synthetic images for training and corresponding defect labeling L abel images, and obtain a converged semantic segmentation model after model parameters are iterated;
further, in the training submodule, training of the semantic segmentation model is performed based on a gradient descent algorithm, wherein L oss calculation function is softmax cross entropy.
During the network model training, the maximum iteration step number is taken as large as possible, the loss value is taken as small as possible, the model parameters are trained to the maximum degree, and the lower limit of the model convergence is obtained at the same time. In the embodiment, the model loss function is calculated by adopting softmax cross entropy (softmax _ cross _ entropy _ with _ logits), wherein the model loss function comprises the steps of converting logits into probabilities, calculating cross entropy loss, firstly converting the logits into the probabilities by using softmax, then calculating the value of each element in the logits, and the corresponding calculation function relationship is as follows:
prob is the probability of corresponding logits elements, and each element is not less than 0 and the sum is 1, so a logits probability distribution is constructed; then, the formula is calculated according to the cross entropy, namely:
where y' is the true label value and y is the logits probability distribution.
EXAMPLE five
Fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present invention, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack-mounted server (including an independent server or a server cluster formed by multiple servers) that can execute programs. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in FIG. 7. It is noted that fig. 7 only shows a computer device 20 with components 21-22, but it is to be understood that not all shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 21 (i.e., the readable storage medium) includes a Flash memory, a hard disk, a multimedia Card, a Card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), and a Programmable Read Only Memory (PROM), and the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), and the like provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various types of application software installed in the computer device 20, such as program codes of a semantic segmentation model modeling method for workpiece defect detection of the method embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In the present embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute a semantic segmentation model modeling device for workpiece defect detection, so as to implement the semantic segmentation model modeling method for workpiece defect detection in the method embodiment.
EXAMPLE six
The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of the present embodiment is for storing program code of a semantic segmentation model modeling apparatus for workpiece defect detection, which when executed by a processor implements the semantic segmentation model modeling method for workpiece defect detection in the method embodiments.
It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A semantic segmentation model modeling method for workpiece defect detection is characterized by comprising the following steps:
acquiring multiple groups of original workpiece point cloud data for training, wherein the workpieces comprise specified defect types;
determining a defect composite image for each workpiece based on the original workpiece point cloud data: converting the original workpiece point cloud data into a two-dimensional depth map; respectively acquiring a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map; synthesizing the two-dimensional depth map, the Scale1 and the Scale2 based on an RGB three-channel color image to obtain the defect synthesized image;
acquiring a defect label L abel image corresponding to the workpiece:
constructing a semantic segmentation model, training based on a plurality of groups of workpiece defect synthetic images and corresponding defect labeling L abel images, and obtaining a converged semantic segmentation model after iterating parameters of the semantic segmentation model;
and solidifying the model layer and the corresponding weight as a pb file, and exporting the pb file as a deep learning pb model file corresponding to the workpiece defect image.
2. The modeling method of the semantic segmentation model according to claim 1, characterized in that a first gradient map Scale1 corresponding to the two-dimensional depth map is obtained based on a Sobel operator;
and/or acquiring a second gradient map Scale2 corresponding to the two-dimensional depth map based on an L aplace operator.
3. The method of modeling a semantic segmentation model of claim 1 wherein the raw workpiece point cloud data is converted to a two-dimensional depth map via a three-dimensional point cloud projection function.
4. The semantic segmentation model modeling method according to claim 1, wherein the determining of the defect label L abel image based on the original workpiece point cloud data comprises deleting a defect region point cloud in the original workpiece point cloud data and then storing the deleted defect region point cloud as new workpiece point cloud data, performing difference on the new workpiece point cloud data and the original workpiece point cloud data to obtain workpiece defect region point cloud data, combining the workpiece defect region point cloud data and the new workpiece point cloud data to obtain workpiece defect image data, and labeling the workpiece defect image data to obtain a defect label L abel image.
5. The semantic segmentation model modeling method according to claim 4, characterized in that the point cloud data of the workpiece defect area is obtained by performing a difference on the point cloud data of the new workpiece and the point cloud data of the original workpiece through Matlab.
6. The semantic segmentation model modeling method according to claim 1, characterized in that the semantic segmentation model comprises a feature extraction network and a feature prediction network connected in sequence;
the feature extraction network is used for predicting the occurrence position of the defect area and comprises a feature extraction layer, a feature compression layer, a feature flattening layer and a feature classification layer; extracting the features of the image data according to the feature extraction layer to obtain a feature map of the image data; compressing the feature map according to the feature compression layer and outputting a feature vector of the feature map;
performing convolution kernel decomposition on the feature vector according to the feature flattening layer and outputting an enumeration vector of the feature map; inputting the enumeration vectors into the feature classification layer to independently predict different types of defects and acquire the positions of defect regions;
the feature prediction network is used for classifying each pixel point in the defect region determined by the feature extraction network, and outputting a defect prediction L abel graph of the workpiece after positioning and segmenting the defect region.
7. The semantic segmentation model modeling method according to claim 6, wherein in the feature extraction network, the feature extraction layer comprises a plurality of convolutional layers connected in sequence; the characteristic compression layer comprises a plurality of convolution layers and pooling layers which are alternately arranged; the feature flattening layer comprises a plurality of one-dimensional convolution layers; the feature classification layer comprises a plurality of fully connected layers;
and/or the characteristic prediction network comprises a plurality of pooling layers and a full connection layer which are connected in sequence.
8. A workpiece defect detection device based on a semantic segmentation model is characterized by comprising:
the system comprises an original workpiece point cloud data acquisition module, a defect detection module and a defect classification module, wherein the original workpiece point cloud data acquisition module is used for acquiring a plurality of groups of original workpiece point cloud data for training, and the workpieces comprise specified defect types;
a defect composite image acquisition module for determining a defect composite image for each workpiece based on the original workpiece point cloud data, comprising: the two-dimensional depth map acquisition unit is used for converting the original workpiece point cloud data into a two-dimensional depth map; a gradient map acquiring unit, configured to acquire a first gradient map Scale1 and a second gradient map Scale2 based on the two-dimensional depth map; a synthesizing unit, configured to synthesize the two-dimensional depth map, the Scale1, and the Scale2 based on an RGB three-channel color image to obtain the defect synthesized image;
a defect label L abel image obtaining module, configured to obtain a defect label L abel image corresponding to the workpiece:
the semantic segmentation model building module is used for training and iterating parameters of the semantic segmentation model based on the plurality of groups of workpiece defect synthetic images and the corresponding defect label L abel images to obtain a converged semantic segmentation model;
and the deep learning pb model file derivation module is used for solidifying the model layer and deriving the pb file as a deep learning pb model file corresponding to the workpiece defect image.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented by the processor when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010554557.5A CN111507357B (en) | 2020-06-17 | 2020-06-17 | Defect detection semantic segmentation model modeling method, device, medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010554557.5A CN111507357B (en) | 2020-06-17 | 2020-06-17 | Defect detection semantic segmentation model modeling method, device, medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111507357A true CN111507357A (en) | 2020-08-07 |
| CN111507357B CN111507357B (en) | 2020-09-25 |
Family
ID=71875381
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010554557.5A Active CN111507357B (en) | 2020-06-17 | 2020-06-17 | Defect detection semantic segmentation model modeling method, device, medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111507357B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112907528A (en) * | 2021-02-09 | 2021-06-04 | 南京航空航天大学 | Point cloud-to-image-based composite material laying wire surface defect detection and identification method |
| CN113516660A (en) * | 2021-09-15 | 2021-10-19 | 江苏中车数字科技有限公司 | Visual positioning and defect detection method and device suitable for train |
| CN113781585A (en) * | 2021-08-27 | 2021-12-10 | 华中科技大学 | Additive manufacturing part surface defect online detection method and system |
| CN114998331A (en) * | 2022-07-28 | 2022-09-02 | 深圳市信润富联数字科技有限公司 | LED lampshade defect detection method and device, electronic equipment and storage medium |
| CN115375692A (en) * | 2022-10-27 | 2022-11-22 | 北京矩视智能科技有限公司 | Workpiece surface defect segmentation method, device and equipment based on boundary guidance |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102135417A (en) * | 2010-12-26 | 2011-07-27 | 北京航空航天大学 | Full-automatic three-dimension characteristic extracting method |
| CN109087274A (en) * | 2018-08-10 | 2018-12-25 | 哈尔滨工业大学 | Electronic device defect inspection method and device based on multidimensional fusion and semantic segmentation |
| CN109344825A (en) * | 2018-09-14 | 2019-02-15 | 广州麦仑信息科技有限公司 | A kind of licence plate recognition method based on convolutional neural networks |
| CN111080622A (en) * | 2019-12-13 | 2020-04-28 | 熵智科技(深圳)有限公司 | Neural network training method, workpiece surface defect classification and detection method and device |
| CN111126391A (en) * | 2019-12-24 | 2020-05-08 | 熵智科技(深圳)有限公司 | Method for positioning defects of printed characters |
-
2020
- 2020-06-17 CN CN202010554557.5A patent/CN111507357B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102135417A (en) * | 2010-12-26 | 2011-07-27 | 北京航空航天大学 | Full-automatic three-dimension characteristic extracting method |
| CN109087274A (en) * | 2018-08-10 | 2018-12-25 | 哈尔滨工业大学 | Electronic device defect inspection method and device based on multidimensional fusion and semantic segmentation |
| CN109344825A (en) * | 2018-09-14 | 2019-02-15 | 广州麦仑信息科技有限公司 | A kind of licence plate recognition method based on convolutional neural networks |
| CN111080622A (en) * | 2019-12-13 | 2020-04-28 | 熵智科技(深圳)有限公司 | Neural network training method, workpiece surface defect classification and detection method and device |
| CN111126391A (en) * | 2019-12-24 | 2020-05-08 | 熵智科技(深圳)有限公司 | Method for positioning defects of printed characters |
Non-Patent Citations (2)
| Title |
|---|
| LONGLONG JING ET AL: "Coarse-to-fine Semantic Segmentation from Image-level Labels", 《IEEE TRANSACTIONS ON IMAGE PROCESSING》 * |
| 刘希: "基于深度卷积神经网络的图像语义分割", 《信息技术》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112907528A (en) * | 2021-02-09 | 2021-06-04 | 南京航空航天大学 | Point cloud-to-image-based composite material laying wire surface defect detection and identification method |
| CN112907528B (en) * | 2021-02-09 | 2021-11-09 | 南京航空航天大学 | Point cloud-to-image-based composite material laying wire surface defect detection and identification method |
| US11557029B2 (en) | 2021-02-09 | 2023-01-17 | Nanjing University Of Aeronautics And Astronautics | Method for detecting and recognizing surface defects of automated fiber placement composite based on image converted from point cloud |
| CN113781585A (en) * | 2021-08-27 | 2021-12-10 | 华中科技大学 | Additive manufacturing part surface defect online detection method and system |
| CN113781585B (en) * | 2021-08-27 | 2024-02-06 | 华中科技大学 | Online detection method and system for surface defects of additive manufactured parts |
| CN113516660A (en) * | 2021-09-15 | 2021-10-19 | 江苏中车数字科技有限公司 | Visual positioning and defect detection method and device suitable for train |
| CN114998331A (en) * | 2022-07-28 | 2022-09-02 | 深圳市信润富联数字科技有限公司 | LED lampshade defect detection method and device, electronic equipment and storage medium |
| CN115375692A (en) * | 2022-10-27 | 2022-11-22 | 北京矩视智能科技有限公司 | Workpiece surface defect segmentation method, device and equipment based on boundary guidance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111507357B (en) | 2020-09-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462120B (en) | Defect detection method, device, medium and equipment based on semantic segmentation model | |
| CN111507357B (en) | Defect detection semantic segmentation model modeling method, device, medium and equipment | |
| CN107229757B (en) | Video retrieval method based on deep learning and Hash coding | |
| CN112070727B (en) | Metal surface defect detection method based on machine learning | |
| CN108171133B (en) | Dynamic gesture recognition method based on characteristic covariance matrix | |
| CN109376787B (en) | Manifold learning network and computer vision image set classification method based on manifold learning network | |
| CN111738344A (en) | Rapid target detection method based on multi-scale fusion | |
| CN111652273B (en) | Deep learning-based RGB-D image classification method | |
| CN106844620B (en) | View-based feature matching three-dimensional model retrieval method | |
| CN112926441B (en) | Home identification method, system, device and medium based on machine vision | |
| CN115984662B (en) | Multi-mode data pre-training and identifying method, device, equipment and medium | |
| CN115439694A (en) | High-precision point cloud completion method and device based on deep learning | |
| CN113269224A (en) | Scene image classification method, system and storage medium | |
| CN115147380A (en) | Small transparent plastic product defect detection method based on YOLOv5 | |
| Wei et al. | GeoDualCNN: Geometry-supporting dual convolutional neural network for noisy point clouds | |
| CN114841992A (en) | Defect detection method based on cyclic generation countermeasure network and structural similarity | |
| CN110019901A (en) | Three-dimensional model search device, searching system, search method and computer readable storage medium | |
| CN116580156A (en) | Text generation 3D printing model method based on big data deep learning | |
| CN116823885A (en) | End-to-end single target tracking method based on pyramid pooling attention mechanism | |
| CN114677468B (en) | Model correction method, device, equipment and storage medium based on reverse modeling | |
| CN114548250A (en) | Mobile phone appearance detection method and device based on data analysis | |
| CN113011506A (en) | Texture image classification method based on depth re-fractal spectrum network | |
| CN112434710A (en) | Intelligent goods shelf goods identification method based on target detection technology | |
| GB2585722A (en) | Image manipulation | |
| CN110827238A (en) | Improved side-scan sonar image feature extraction method of full convolution neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231208 Address after: 518000, Building 5, Building C, Building C, Huaqiang Creative Park, Biyan Community, Guangming Street, Guangming District, Shenzhen, Guangdong Province, China 1301 Patentee after: SHENZHEN ROBOT VISION TECHNOLOGY Co.,Ltd. Address before: 518034 703, 7th floor, Zhongdian Difu building, Zhenhua Road, Fuqiang community, Huaqiang North Street, Futian District, Shenzhen City, Guangdong Province Patentee before: SHANGZHI TECHNOLOGY (SHENZHEN) Co.,Ltd. |
|
| TR01 | Transfer of patent right |