CN115631127B - Image segmentation method for industrial defect detection - Google Patents
Image segmentation method for industrial defect detection Download PDFInfo
- Publication number
- CN115631127B CN115631127B CN202210981514.4A CN202210981514A CN115631127B CN 115631127 B CN115631127 B CN 115631127B CN 202210981514 A CN202210981514 A CN 202210981514A CN 115631127 B CN115631127 B CN 115631127B
- Authority
- CN
- China
- Prior art keywords
- feature
- image
- industrial
- industrial defect
- defect detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image segmentation method for industrial defect detection, which comprises the steps of firstly, model learning on a heterogeneous data set, and transmitting parameters related to a classifier in an information stream to a fusion migration module by using a recursion weighting model through the quasi-optimal distribution of parameters in a pre-training model for deducing image data in a process; and then, constructing a loss function of the fusion migration module by calculating the generalized maximum average difference of the fusion migration model, realizing fusion learning of heterogeneous data, recursively updating optimal parameters, primarily transmitting to the right branch of the double-branch information flow, and realizing parameter optimization of three-level feature extraction through knowledge migration in a global network after model training supervision learning and knowledge extraction of the heterogeneous image dataset are completed. The invention adopts heterogeneous data fusion learning to perform feature extraction, and solves the problems of boundary blurring and multi-scene fusion segmentation in the image semantic segmentation process of industrial defect detection based on knowledge extraction and reasoning.
Description
Technical Field
The invention relates to a semantic segmentation method for an industrial defect detection heterogeneous data image, which relates to the field of intelligent manufacturing and machine vision.
Background
In the industrial production process, a large amount of detection requirements exist, the industrial defect detection application scene is wide, the industrial defect detection application scene comprises a plurality of industries such as mechanical manufacturing, energy chemical industry, steel production and the like, a manual detection method is often adopted in the traditional production mode, the manual industrial defect detection workload is very large, huge manpower and material resource cost is required to be input, even if the manual detection effect is often unsatisfactory, on one hand, the efficiency is generally low, and the subjective judgment standards of the testers are difficult to unify, so that the estimation deviation is large and the detection quality fluctuation is large, and as a result, the industrial defect detection accuracy cannot be ensured. Therefore, the machine vision industrial defect detection realized by adopting the artificial intelligence algorithm can remarkably improve the intelligent level of the industrial production process, the intelligent manufacturing level and the production efficiency and the economic benefit.
One of the core technologies of machine vision industrial defect detection is an image semantic segmentation method. At present, the image++ image segmentation technology in natural images has reached a satisfactory performance level, but the algorithm models are often poor in effect when facing industrial defect detection application scenes. By training a network in the sliding window setting, the class label of each pixel is predicted by providing a local area around the pixel as input, so that the network can be localized, and the quantity of training data in the patch form can be far greater than that of the original training image, thereby improving the algorithm performance to a certain extent. However, there are two disadvantages: first, it is very slow because the network must run separately for each local area and there is a lot of redundancy due to overlapping local areas. Second, there is a tradeoff between positioning accuracy and context usage, larger local areas require more pooling layer computations, significantly reducing positioning accuracy, while selecting small local areas allows the network to receive only few context inputs. The construction of industrial defect detection data sets applied to machine vision is time-consuming and labor-consuming, the label types and sets of the data sets constructed in different industrial defect detection scenes are different, and no detection method with excellent performance is available for the heterogeneous data set problem at present.
Disclosure of Invention
The technical problem to be solved by the invention is that an image segmentation method aiming at industrial defect detection is slow in algorithm operation, redundant calculation exists in a data local area, and heterogeneous data sets of different industrial defect detection scenes cannot be effectively fused. The invention provides a heterogeneous data image semantic segmentation method, which adopts heterogeneous data fusion learning to perform feature extraction and realizes high-performance up-sampling expansion compound convolution based on knowledge extraction and reasoning.
The invention adopts the following technical scheme for solving the technical problems:
the invention provides an image segmentation method for industrial defect detection, which comprises the following steps:
s1, heterogeneous data fusion learning, specifically:
s101, constructing a first image data set A and a second image data set B from different industrial defect detection fields to form an industrial defect detection heterogeneous image data set;
s102, heterogeneous data preprocessing: preprocessing a first image data set A and a second image data set B respectively, extracting image space dimension features, and obtaining two space feature information matrixes respectively;
s103, respectively inputting the two spatial feature information matrixes obtained in the step S102 into two independent recursion weighting models, executing recursion weighting operation, and outputting industrial defect image spatial feature matrixes respectively corresponding to the two input spatial feature information matrixes;
s104, constructing a loss function of the fusion migration module by calculating the generalized maximum average difference of the fusion migration model, regularizing a weighted model vector in the recursion weighted model to optimize the structure of the loss function, realizing fusion learning of heterogeneous data, taking a parameter result obtained by fusion learning training of the heterogeneous data as a pre-training model parameter, and realizing recursion weighted fusion learning;
step S2, real-time detection of different types of industrial defects, specifically:
s201, inputting the acquired industrial defect image to a feature extraction module, after knowledge extraction of a heterogeneous image data set is completed, realizing parameter optimization of three-level feature extraction in the feature extraction module through knowledge migration, completing feature extraction of the industrial defect image, and respectively outputting three-level feature imagesAnd->Wherein the third layer is output as a feature map +.>Through knowledge extraction and reasoning module, it is used to predict whether the input image contains industrial defect, and finally output characteristic diagram +.>
S202, inputting a feature mapImage segmentation up-sampling expansion module, and simultaneously combining feature graphs +.>And feature map->Further through channel compression operation, the signals are respectively and correspondingly output +.>Three types of feature graphs;
s203, each result of the complex convolution operation unit of the up-sampling expansion module, namelyGenerating a side output, and implementing deep supervised learning on the generated full-resolution feature map by correct annotation data;
s204, realizing pixel-level classification of the original input image by adopting multi-channel industrial defect segmentation, namely semantic segmentation of industrial defect detection, and realizing a feature map by adopting 1X 1 convolution for N different types of industrial defectsAnd converting the number of the wide channels into the number N of the channels, and outputting an image segmentation result to be real-time detection of at most N industrial defects.
Compared with the prior art, the technical scheme provided by the invention has the following technical effects:
the method solves the problem that heterogeneous data sets of different industrial defect detection scenes cannot be effectively fused, and the performance of the industrial defect detection multi-scene fusion segmentation algorithm is remarkably improved. Using knowledge from industrial defect detection without application domain heterogeneous image datasets, performance of models on inferred datasets is improved through transfer learning. For example, in the task of detecting the surface defects of the sheet material in the cold-rolled galvanization production line of the steel plant, some defects belong to various hard scratch forms, and some defect types are various greasy dirt forms. The presented image semantic context feature types belong to heterogeneous data sets. To this end, we first learn the model over heterogeneous data set a and data set B, after the pre-training is completed, the parameters in the model will learn a quasi-optimal distribution that can be used to infer the image data in the process. In this section, using a recursive weighting model, parameters associated with the classifier in the information stream are transmitted to a fusion migration module; and then, constructing a loss function of the fusion migration module by calculating the generalized maximum average difference of the fusion migration model, realizing fusion learning of heterogeneous data, recursively updating optimal parameters, primarily transmitting to the right branch of a double-branch information flow, after model training supervision learning and knowledge extraction of a heterogeneous image dataset are completed, further realizing parameter optimization of three-level feature extraction through knowledge migration in a global network, and further simplifying parameters in an inference stage in model production deployment, thereby remarkably improving model instantaneity and operation efficiency. The method solves the problem that the conventional image segmentation method for industrial defect detection is slow in algorithm operation, and in order to reduce the cost to the maximum extent and utilize the GPU memory to the maximum extent, large input collages are used instead of large batches, so that the batches are reduced to single images, and the problem that redundant calculation exists in a data local area is solved.
The invention solves the problem of boundary blurring in the process of image semantic segmentation of industrial defect detection, not only can detect industrial defects and flaws, but also can analyze the specific shapes and positions of the defects and flaws, and provides basis for further analyzing the causes of the defects and flaws, so that the detection is realized, and the semantic segmentation can be realized. The algorithm utilizes fewer parameters based on heterogeneous data fusion migration, but can generate more accurate position perception and technical innovation of the boundary enhancement segmentation map. Compared with the existing advanced algorithm, the method has fewer parameters, obviously reduces the computational complexity and has higher algorithm reasoning precision. Both segmentation methods with common connections and segmentation methods with nested and dense connections lack sufficient information to explore from a full scale, failing to explicitly learn the location and boundaries of organs. The invention adopts full-scale bridging and full-scale deep supervised learning, reduces excessive semantic segmentation to prevent false positives in semantic segmentation, namely avoids the following errors: samples without industrial defects were misjudged to be defective.
In conclusion, the method solves the problem of boundary blurring in the process of image semantic segmentation of industrial defect detection, effectively solves the problem of multi-scene fusion segmentation of industrial defect detection, reduces excessive semantic segmentation, avoids error occurrence caused by error judgment as defect due to no industrial defect, and effectively prevents false positives in semantic segmentation.
Drawings
FIG. 1 is a schematic diagram of an industrial defect detection heterogeneous image dataset construction.
FIG. 2 is a schematic diagram of heterogeneous data preprocessing, recursively weighted modeling, and fusion migration modeling.
Fig. 3 is a schematic diagram of the overall structure of the algorithm of the present invention.
Detailed Description
The technical scheme of the invention is further described in detail below with reference to the accompanying drawings:
it will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Step 1, constructing an industrial defect detection heterogeneous image data set.
For heterogeneous data sets, such as data set a and data set B (e.g., fig. 1) from different industrial defect detection domains. Data set a contains U (u=8 shown in the figure) different types of industrial defect features and data set B contains another V (v=9 shown in the figure) different types of industrial defect features, where set a and set B have W (intersection w=5 of set a and set B shown in the figure) different types of industrial defect features in common.
As shown in fig. 1, the industrial defect feature types of the data set a and the data set B are not completely the same, are basically different data sets, and are heterogeneous data sets, so that the data sets cannot be simply mixed together and combined, and cannot be directly used for model training and parameter learning. Therefore, the optimal algorithm model is obtained by training by adopting a double-information flow method and fully utilizing the data set A and the data set B in an information fusion migration mode.
Step 2 heterogeneous data preprocessing (see dotted box on top of fig. 2, "heterogeneous data set preprocessing").
First, the original image data set a is preprocessed to extract the image space dimension features. In particular RGB information in the spatial dimension is extracted. The spatial feature extraction task is directly performed by a deep neural network model using a pre-trained dense network (DenseNet) for the original image with RGB three channel information, respectively. The output is the corresponding s-dimensional vector. Each group of given image data contains G images, outputs G s-dimensional vectors, and forms an sxg matrix in turn. Represented asWe name it as spatial feature extractor +.>And similarly, preprocessing the set B to extract the image space dimension characteristics. And obtaining an output t-dimensional vector. Each group of given image data contains G images, outputs G t-dimensional vectors, and forms a t×g matrix in turn. Denoted as->Named spatial feature extractor +.>
Step 3 recursive weighting model (see dashed box in the middle "recursive weighting model" of fig. 2).
The result of the data preprocessing comprises two information streams, the first information stream is obtained by extracting from the spatial feature extractorSpatial feature information matrix obtained by extracting group A images>The second information stream is from the spatial feature extractor +.>Spatial feature information matrix obtained by extracting group B images>Then, two independent recursive weighted models are respectively input, and the output result is a spatial feature information matrix +.>And->A corresponding information matrix. These two output information matrices are denoted +.>And->
The recursive weighted model structure is shown in fig. 2, and comprises a left information stream and a right information stream which respectively correspond to the data sets A and B. The recursive weighted model structure of each is the same, using the same activation function tanh, i.e., hyperbolic tangent function, expressed as tanh (x) = (e) x -e -x )/(e x +e -x )。
Left branch for dataset ADuring the process of the information flow, the data input is processed through the expansion convolution layer DC11, the function operation unit, the expansion convolution layer DC12 and the exponential linear unit (ELU, exponential linear units), wherein the expanded convolution layer DC11 corresponds to the parameter +.>a, c is a configurable hyper-parameter; the activation function is that; expansion convolution layer DC12, corresponding parameters +.>The exponential linear unit output produces a weighted model vector V S Wherein->The operational relationship is as follows.
And then will beMultiplying by the spatial feature matrix>To obtain a recursively weighted spatial feature matrix +.>Wherein-> Representing a set of real numbers.
For the right branch information flow processing procedure of the data set B, the structure is the same as that of the left branch, and the data input passes through the expansion convolution layer DC21, the activation function operation unit, the expansion convolution layer of DC22 and the exponential linear unit. Wherein the expansion convolution layer DC21, the corresponding parametersb is configurable super parameter, the activation function is tanh, DC22 of the expansion convolution layer, corresponding parameter The exponential linear unit output produces a weighted model vector V T Wherein->The operational relationship is as follows.
And then will beMultiplying by the time feature matrix>To obtain a recursively weighted spatial feature matrix +.>Wherein the method comprises the steps of
Industrial defect image space feature matrix obtained by the above operationAnd feature matrix->Performing a recursive weighting operation for calculating a loss function of the training phase based on the following, expressed as +.>The method is applied to industrial defect feature identification.
Wherein T represents real matrix transposition, P i ,Q i I=1, 2, m represent the calculated vector and matrix parameters in the inflated convolutional layer, respectively, optimized during model training,the symbol operation is to sequentially combine a plurality of column vectors side by side to form a matrix, and combine the spatial feature column vectors side by side to obtain a k×2 matrix, wherein k is the classification number of the industrial defect feature recognition, and the detailed structure is shown in a dotted line frame part of a "recursion weighting model" in the middle part of fig. 2.
Step 4 fusion migration model (see dotted box at bottom of fig. 2 "fusion migration model").
Weighting model vector V in a recursive weighting model S And V T Regularization is required to optimize the loss function loss ψ Is a structure of (a). Given a givenLet->Wherein->Representing transpose operations, loss ψ The form is as follows:
wherein the method comprises the steps ofRepresenting the hadamard product of the two, I 2 Representing 2 norms, matrix->Defined as follows:
in summary, the loss function for training the industrial detection heterogeneous data set is constructed as follows:
and taking a parameter result obtained by heterogeneous data fusion learning training as a pre-training model parameter, realizing recursion weighted fusion learning, and providing model parameter simplifying support for a subsequent algorithm through knowledge migration.
Step 5 feature extraction module (see fig. 3 "algorithm overall structure" upper dashed box "feature extraction module").
The feature extraction module for the industrial defect image comprises a heterogeneous data fusion learning functional unit and a 3-layer feature extraction functional unit. The whole structure of the algorithm of the invention is shown in fig. 3, and comprises three components of input and output and a core: 1. a feature extraction module; 2. knowledge extraction and reasoning module; 3. and an image segmentation up-sampling expansion module. The feature extraction module (see the upper dashed line frame part of fig. 3) includes a left heterogeneous data fusion learning module and a right feature extraction module (F1, F2, F3), and the detailed working principle of the left heterogeneous data fusion learning module is described. The right feature extraction module comprises a (F1, F2, F3) three-layer serial 2D convolution sub-module, wherein the feature extraction sub-module F1 has the specific structure that the input is adoptedThe industrial defect image data of the collection is cut into pixelsIn the context of our practical industrial defect detection application scenario, q×q=1024×1024; the image with the same size is input into a feature extraction submodule F1, and is subjected to 5X 5 convolution with the same width and the same height by zero filling to obtain the image with the same size +.>Is subjected to ELU activation function, zero padding, 5X 5 convolution with equal width and equal height, 2X 2 averaging pooling, and F1 output is halved to be +.>Is expressed as the characteristic diagram ofThe internal structure of the feature extraction sub-modules F2 and F3 is the same as that of the right feature extraction module of the double information flow in the heterogeneous data fusion learning module, and the network parameters of the feature extraction sub-modules are shared by the knowledge migration functional units of the knowledge extraction and reasoning module, so that the model training complexity is greatly reduced, and the model training time is shortened. The outputs of the feature extraction submodules F2, F3 are respectively feature graphs +.>And feature map->
Step 6 knowledge extraction and reasoning module (see fig. 3, "algorithm overall structure" middle dashed box "knowledge extraction and reasoning module").
The module receives the output of the heterogeneous data fusion learning function unit of the feature extraction module, combines the realization of supervised learning auxiliary training, completes the judgment of whether the image has industrial defects or not, and judges which type or types if the image has the industrial defectsThe parameter sharing to the feature extraction sub-modules F2 and F3 is realized through a knowledge migration functional unit; based on the knowledge base of industrial defect common sense, the knowledge representation and extraction functional unit receives the output from the feature extraction sub-module F3, so as to realize the knowledge extraction and reasoning function, and to predict whether the input image contains industrial defects, and provide guidance for the subsequent semantic segmentation task so as to prevent false positives in the semantic segmentation. That is, the industrial defect is avoided being judged to be defective, and finally, the characteristic diagram is output(see the dashed box "knowledge extraction and reasoning module" in the middle of fig. 3).
Step 7 image segmentation upsampling extension module (see fig. 3 "algorithm overall structure" bottom dashed box "image segmentation upsampling extension module").
Receiving an input profileLength and width of +.>Is mapped by extending the characteristic diagram (with E xpand (. Cndot.) the length-width expansion of the obtained feature map is +.>Simultaneously merging output feature maps from feature extraction submodule F2And the output feature map of feature extraction submodule F3->Wherein the feature map->The feature map compression mapping A needs to be passed first bstract Compressing the length and width of the characteristic diagram to be +.>Output characteristic map->Is->The three feature maps with the same length and width are cascaded into a whole. Through composite convolution F CNA [·]Calculating to obtain an output characteristic diagram->Length and width of +.>The channel number conversion is further implemented by channel compression operation, i.e. 1×1 convolution, and then the next processing unit is entered. Three composite convolution processing units are cascaded in total on the image segmentation up-sampling expansion module, and the three composite convolution processing units respectively output +.>Three types of feature graphs.
And 8, information flow operation construction of an up-sampling expansion module, wherein a knowledge extraction and reasoning module is taken as a bridge, and the feature extraction module and the up-sampling expansion module are fused to realize global context feature aggregation and industrial defect target local positioning
Wherein the operator isRepresenting bitwise cascade operation on left and right items thereof; f (F) CNA [·]Representing a complex convolution operation, representing a set of sequential operations on elements therein, i.e. a 5 x 5 convolution operation with adaptive zero padding is performed first, realizing that the length and width of the input feature diagram are kept unchanged after output, and then carrying out normalization and S-shaped function activation operation (wherein S-shaped function Sigmod (t) =S (t) =1/(1+e) -t ))；A bstract (. Cndot.) represents the feature map compression mapping, and the input feature map length and width are compressed to half of the original length and width by a non-overlapping 2×2 average pooling algorithm, so that the number of channels is kept unchanged; e (E) xpand (. Cndot.) represents feature map expansion mapping, and inverse convolution operation is adopted to realize that the length and width of the input feature map are expanded to be twice as much as the original length and width, and the number of channels is kept unchanged; c (C) hannel (. Cndot.) represents channel compression operation, and 1×1 convolution is adopted to realize conversion from more channels to fewer channels, so that the length, width, input and output of the feature map are unchanged.
And 9, constructing a loss function of deep supervision learning, and solving the problem of boundary blurring in the process of image semantic segmentation of industrial defect detection. For this purpose, each result of the complex convolution operation unit of the up-sampling expansion module (i.e) And generating a side output, and implementing deep supervised learning on the generated full-resolution feature map by the correct annotation data. Firstly, the last layer of each complex convolution operation unit is fed into a 5×5 convolution with zero padding and equal width and equal height, then the channel number is adjusted by 1×1 convolution, then a 2D deconvolution expansion operation is performed, and the result is bitwise operated by a hyperbolic tangent function tah (), wherein tanh (x) =2/(1+e) -2x ) -1. To enhance the boundary segmentation capability of the present algorithm for processing industrial defect detection targets, we construct a supervised learning loss function +.>The following are provided:
the first term of the loss function is the soft cross-ratio loss, meets continuous and micro-requirements, and can effectively implement a gradient descent back propagation algorithm in the model training process. In particular, it outputs the result by using a hyperbolic tangent function tanh (). Wherein the method comprises the steps ofIs the position->Predictive score for U class (industrial defect detection target class number),>is correctly labeling tag distribution, ++>Belongs to the pixel set->Correspond to->Square of pixel size.
The second term of the loss function is an objective function of the edge adaptive enhancement index of the panoramic objective structure constructed by us, and is used forIt means that a high weight can be adapted for boundary adaptation of industrial defect detection target blurring. The larger the distribution difference of the industrial defect detection target area is, the higher the adaptation weight value is. Semantic segmentation of industrial defect detection targets from algorithms (with +.>Representation) and correctly labeling the data segmentation map (with +.>Representation) are respectively cut out of two corresponding spatial positionsSquare of pixel size, each pixel number of which is represented as a set of pixels, respectivelyWherein->And pixel set +.>Wherein the method comprises the steps ofThen pixel set +.>And->Is defined as +.about.the above formula>Form of the invention. Wherein->Representing the pixel adaptation intensity, set as a super parameter +.>Constant->Psi=0.009 is used as a super parameter setting to prevent the denominator value in the calculation formula from being too small to be classified as zero-value pathological state; />And->Respectively representing pixel sets +.>And->Mean and standard deviation of>Representing pixel set +.>And->Covariance; />And->Two components per pixel are defined separately (pixel set +.>And pixel set +.>) Is adaptive to the fuzzy boundary of the industrial defect detection target, and is obtained by iterative optimization in the process of monitoring and learning the model training.
Step 10 Multi-channel Industrial Defect segmentation Module
The multi-channel (the number of output channels is equal to the number of industrial defect categories) industrial defect segmentation realizes pixel-level classification of the original input image, namely semantic segmentation of industrial defect detection, and is set as NFor different types of industrial defects, the multi-channel industrial defect segmentation module adopts 1X 1 convolution to realize the characteristic diagramAnd converting the number of the wide channels into the number N of the channels, and outputting an image segmentation result to be real-time detection of at most N industrial defects.
And step 11, outputting an industrial defect image segmentation result.
The foregoing is only a partial embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.
Claims (10)
1. An image segmentation method for industrial defect detection, which is characterized by comprising the following steps:
s1, heterogeneous data fusion learning, specifically:
s101, constructing a first image data set A and a second image data set B from different industrial defect detection fields to form an industrial defect detection heterogeneous image data set;
s102, heterogeneous data preprocessing: preprocessing a first image data set A and a second image data set B respectively, extracting image space dimension features, and obtaining two space feature information matrixes respectively;
s103, respectively inputting the two spatial feature information matrixes obtained in the step S102 into two independent recursion weighting models, executing recursion weighting operation, and outputting industrial defect image spatial feature matrixes respectively corresponding to the two input spatial feature information matrixes;
s104, constructing a loss function of the fusion migration model by calculating the generalized maximum average difference of the fusion migration model, regularizing a weighted model vector in the recursion weighted model to optimize the structure of the loss function, realizing fusion learning of heterogeneous data, taking a parameter result obtained by fusion learning training of the heterogeneous data as a pre-training model parameter, and realizing recursion weighted fusion learning;
step S2, real-time detection of different types of industrial defects, specifically:
s201, inputting the acquired industrial defect image to a feature extraction module, after the heterogeneous data fusion learning of the heterogeneous image data set is completed, realizing the parameter optimization of three-level feature extraction in the feature extraction module through knowledge migration, completing the feature extraction of the industrial defect image, and respectively outputting three-level feature imagesAnd->Wherein the third layer is outputThrough knowledge extraction and reasoning module, it is used to predict whether the input image contains industrial defect, and finally output characteristic diagram +.>
S202, mapping the characteristic diagramInput to the image segmentation up-sampling expansion module while combining feature map +.>And feature map->Further through channel compression operation, the signals are respectively and correspondingly output +.>Three types of feature graphs;
s203, composite convolution operation of sampling expansion module on image segmentationEach result of the arithmetic unit, i.eGenerating a side output, and implementing deep supervised learning on the generated full-resolution feature map by correct annotation data;
s204, realizing pixel-level classification of the original input image by adopting multi-channel industrial defect segmentation, namely semantic segmentation of industrial defect detection, and realizing a feature map by adopting 1X 1 convolution for N different types of industrial defectsAnd converting the number of the wide channels into the number N of the channels, and outputting an image segmentation result to be real-time detection of at most N industrial defects.
2. The method for image segmentation for industrial defect detection according to claim 1, wherein: in step S101, for heterogeneous data sets, i.e. a first image data set a and a second image data set B from different industrial defect detection fields, wherein the first image data set a contains U different types of industrial defect features and the second image data set B contains another V different types of industrial defect features, the first image data set a and the second image data set B have W different types of industrial defect features in common.
3. An image segmentation method for industrial defect detection according to claim 2, wherein: step S102 is to use a spatial feature extractorSpatial feature extractor->The treatment is carried out as follows:
(1) Spatial feature extractorThe treatment process of (2) is as follows: preprocessing the first image data set A to extract image space dimension features: extracting RGB information in space dimension, directly executing space feature extraction task by using a pre-trained dense network DenseNet for original images with RGB three-channel information through a deep neural network model, outputting corresponding s-dimension vectors, giving image data each group containing G images, outputting G s-dimension vectors, and sequentially forming an sXG matrix, which is expressed as->Is a matrix of (a);
(2) Spatial feature extractorThe treatment process of (2) is as follows: preprocessing a second image data set B by adopting the same method as the step (1) to extract image space dimension features to obtain output t-dimension vectors, wherein each group of given image data comprises G images, G t-dimension vectors are output, and a t multiplied by G matrix is formed in sequence and is expressed as +.>Is a matrix of (a) in the matrix.
4. An image segmentation method for industrial defect detection according to claim 3, wherein: in step S103, the two spatial feature information matrices obtained in step S102 are combinedTwo independent recursive weighted models are respectively input, and the output result is an information matrix corresponding to the input spatial feature information matrix respectively +.>
The recursive weighted model structure comprises a left information flow and a right information flow, the left information flow and the right information flow correspond to the first image data set A and the second image data set B respectively, the recursive weighted model structure of each information flow is the same, the same activation function tanh, namely the hyperbolic tangent function is used, and the expression is as follows:
tanh(x)＝(e x -e -x )/(e x +e -x )。
5. the method for image segmentation for industrial defect detection of claim 4, wherein: the left branch information flow processing procedure for the first image data set A is as follows:
spatial feature information matrix to be inputSequentially passing through an expansion convolution layer DC11, an activation function operation unit, an expansion convolution layer DC12 and an exponential linear unit ELU,>
wherein the inflated convolution layer DC11 corresponds to the parametera, c is a configurable hyper-parameter; the activation function is tanh; expansion convolution layer DC12, corresponding parameters +.>The exponential linear unit output produces a weighted model vector V S Wherein The operational relationship is as follows:
and then will beMultiplying by the spatial characteristic information matrix respectively>Obtaining a recursively weighted spatial feature matrix>Wherein-> Representing a real set;
the right branch information flow processing procedure for the second image data set B is the same as the left branch information flow processing procedure: the data input is performed through an expansion convolution layer DC21, an activation function operation unit, an expansion convolution layer DC22 and an exponential linear unit; wherein the expansion convolution layer DC21, the corresponding parametersb is configurable super parameter, the activation function is tanh, DC22 of the expansion convolution layer, corresponding parameter +.>The exponential linear unit output produces a weighted model vector V T Wherein->The operational relationship is as follows:
and then will beMultiplying by the spatial feature matrix>To obtain a recursively weighted spatial feature matrix +.>Wherein the method comprises the steps ofThe industrial defect image recursively weighted spatial feature matrix obtained by the above operation>And->Performing a recursive weighting operation for calculating a loss function of the training phase based on the following, expressed as +.>The method is applied to industrial defect feature identification:
wherein T represents real matrix transposition, P i ,Q i Respectively representing the calculated vector and matrix parameters in the expansion convolution layer, i=1, 2; the calculation vectors and matrix parameters are optimized during model training, the symbol operation × is to sequentially combine a plurality of column vectors side by side into a matrix, and combine the spatial feature column vectors side by side to obtain a kx2 matrix, where k is the classification number of the industrial defect feature recognition.
6. The method for image segmentation for industrial defect detection of claim 5, wherein: the step S104 specifically includes:
weighting model vector V in a recursive weighting model S And V T Regularization is required to optimize the loss function loss ψ Given the structure of (a)Let->Wherein->Representing transpose operations, loss ψ The form is as follows:
wherein the method comprises the steps ofRepresenting the hadamard product of the two, I 2 Representing 2 norms, matrix->Defined as follows:
in summary, the loss function for training the industrial detection heterogeneous data set is constructed as follows:
7. the method for image segmentation for industrial defect detection of claim 5, wherein: in step S201 of the process of the present invention,
the feature extraction module is used for: comprises a heterogeneous data fusion learning unit and a characteristic extraction unit, wherein the characteristic extraction unit comprises three layers of 2D convolution sub-modules (F1, F2 and F3) which are sequentially connected in series,
wherein: input the collected industrial defect image data to be cut into pixelsInputting the cut image into a feature extraction submodule F1, and performing zero padding 5×5 convolution with equal width and equal height to obtain the same sizeIs subjected to ELU activation function, zero padding, 5×5 convolution of equal width and equal height, 2×2 averaging pooling, and finally output with size +.>Is expressed as +.>The processing procedure of the feature extraction sub-modules F2 and F3 is the same as that of the right branch information flow, the network parameters of the feature extraction sub-modules F2 and F3 realize parameter sharing through a knowledge migration unit of the knowledge extraction and reasoning module, and the outputs of the feature extraction sub-modules F2 and F3 are feature graphs respectively>And feature map->
The knowledge extraction and reasoning module comprises a knowledge migration unit and a knowledge representation and extraction unit:
the knowledge transfer unit is used for receiving the output of the heterogeneous data fusion learning function unit of the feature extraction module, combining with the realization of supervised learning auxiliary training, completing the judgment of whether industrial defects exist in the image and the judgment of the defect types, and realizing parameter sharing to the feature extraction sub-modules F2 and F3 through the knowledge transfer unit; based on the knowledge base of industrial defect knowledge, the knowledge representation and extraction unit receives the output from the feature extraction sub-module F3 to realize knowledge extraction and reasoning function for predicting whether the input image contains industrial defect, and finally outputs the feature image
8. The method for image segmentation for industrial defect detection of claim 7, wherein: in step S202, the processing of the image segmentation up-sampling expansion module specifically includes:
image segmentation up-sampling expansion module receives input feature imageFeature map->Is +.>Feature map +.>Through feature map expansion mapping, the length and width are expanded to +.>Wherein the feature map is expanded and mapped with E xpand (-) representation; simultaneously combining the output feature map ++>And the output feature map of feature extraction submodule F3->Wherein the feature map->The feature map compression mapping A needs to be passed first bstract Compressing the length and width of the characteristic diagram into Output characteristic map->Is->The three feature maps with the same length and width are cascaded into a whole;
through the process ofComposite convolution F CNA [·]Calculating to obtain an output characteristic diagramLength and width of +.>The channel number is converted into the next compound convolution operation unit by further channel compression operation, namely 1X 1 convolution is adopted, three compound convolution operation units are cascaded in total, and the three compound convolution operation units respectively output +.>Three types of feature graphs.
9. The method for image segmentation for industrial defect detection of claim 7, wherein: the system also comprises an information flow operation construction of an image segmentation up-sampling expansion module, and takes a knowledge extraction and reasoning module as a bridge, and the fusion feature extraction module and the image segmentation up-sampling expansion module realize global context feature convergence and industrial defect target local positioning:
wherein the operator isRepresenting bitwise cascade operation on left and right items thereof; f (F) CNA [·]Representing a complex convolution operation, representing the sameA group of sequential operations of elements in the matrix, namely, firstly performing 5×5 convolution operation of adaptive zero filling to realize that the length and width of an input feature map are kept unchanged after output, and then performing normalization and S-type function activation operation, wherein S-type function Sigmod (t) =S (t) =1/(1+e) -t )；A bstract (. Cndot.) represents the feature map compression mapping, and the input feature map length and width are compressed to half of the original length and width by a non-overlapping 2×2 average pooling algorithm, so that the number of channels is kept unchanged; e (E) xpand (. Cndot.) represents feature map expansion mapping, and inverse convolution operation is adopted to realize that the length and width of the input feature map are expanded to be twice as much as the original length and width, and the number of channels is kept unchanged; c (C) hannel (. Cndot.) represents channel compression operation, and 1×1 convolution is adopted to realize conversion from more channels to fewer channels, so that the length, width, input and output of the feature map are unchanged.
10. The method for image segmentation for industrial defect detection of claim 9, wherein: in step S2, a supervised learning loss function is constructed for enhancing the boundary segmentation capability of the process industrial defect detection targetThe following are provided:
the first term of the loss function is the soft cross-ratio loss, which meets the continuous micro-requirement, whereinIs the position->Predictive score for U-type industrial defect detection target class number,/-for U-type industrial defect detection target class number>Is correctly labeling tag distribution, ++>Belongs to the pixel set-> Correspond to->Square of pixel size;
the second term of the loss function is an objective function of an edge adaptive enhancement index of the panoramic objective structure, which is constructed byThe method is characterized in that the fuzzy boundary of the industrial defect detection target is adaptively adaptive to high weight, the larger the distribution difference of the industrial defect detection target area is, the higher the adaptive weight value is, and the semantic segmentation result of the industrial defect detection target is obtained from an algorithm->And correctly labeling the data segmentation map->Respectively cutting out the +.>Square of pixel size, each pixel number of which is denoted as pixel set +.>Sum pixel setWherein->Then pixel set +.>And->Is defined as +.about.the above formula> Form of (c); wherein T represents the pixel adaptation intensity, set as a super parameter to t=128; constant->Psi=0.009 is used as a super parameter setting to prevent the denominator value in the calculation formula from being too small to be classified as zero-value pathological state; />And->Respectively representing pixel sets +.>And->Mean and standard deviation of>Representing pixel set +.>And->Covariance; />And->Two components per pixel are defined, namely the pixel set +.>And pixel set +.>The relative reinforcement index of the target is adaptive to the fuzzy boundary of the industrial defect detection target, and the boundary is obtained through iterative optimization in the process of supervised learning model training.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210981514.4A CN115631127B (en) | 2022-08-15 | 2022-08-15 | Image segmentation method for industrial defect detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210981514.4A CN115631127B (en) | 2022-08-15 | 2022-08-15 | Image segmentation method for industrial defect detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115631127A CN115631127A (en) | 2023-01-20 |
| CN115631127B true CN115631127B (en) | 2023-09-19 |
Family
ID=84903437
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210981514.4A Active CN115631127B (en) | 2022-08-15 | 2022-08-15 | Image segmentation method for industrial defect detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115631127B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115984235A (en) * | 2023-01-31 | 2023-04-18 | 苏州大学 | Wafer map mixed defect mode identification method and system based on image segmentation |
| CN116429782B (en) * | 2023-03-29 | 2024-01-09 | 南通大学 | Saw chain defect detection method based on residual error network and knowledge coding |
| CN116645369B (en) * | 2023-07-27 | 2023-11-07 | 山东锋士信息技术有限公司 | Anomaly detection method based on twin self-encoder and two-way information depth supervision |
| CN117952983B (en) * | 2024-03-27 | 2024-07-26 | 中电科大数据研究院有限公司 | Intelligent manufacturing production process monitoring method and system based on artificial intelligence |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109711413A (en) * | 2018-12-30 | 2019-05-03 | 陕西师范大学 | Image, semantic dividing method based on deep learning |
| CN110111297A (en) * | 2019-03-15 | 2019-08-09 | 浙江大学 | A kind of injection-molded item surface image defect identification method based on transfer learning |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
| CN114742799A (en) * | 2022-04-18 | 2022-07-12 | 华中科技大学 | Industrial scene unknown type defect segmentation method based on self-supervision heterogeneous network |
| CN114820579A (en) * | 2022-05-27 | 2022-07-29 | 广东工业大学 | Semantic segmentation based image composite defect detection method and system |
-
2022
- 2022-08-15 CN CN202210981514.4A patent/CN115631127B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109711413A (en) * | 2018-12-30 | 2019-05-03 | 陕西师范大学 | Image, semantic dividing method based on deep learning |
| CN110111297A (en) * | 2019-03-15 | 2019-08-09 | 浙江大学 | A kind of injection-molded item surface image defect identification method based on transfer learning |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
| CN114742799A (en) * | 2022-04-18 | 2022-07-12 | 华中科技大学 | Industrial scene unknown type defect segmentation method based on self-supervision heterogeneous network |
| CN114820579A (en) * | 2022-05-27 | 2022-07-29 | 广东工业大学 | Semantic segmentation based image composite defect detection method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115631127A (en) | 2023-01-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115631127B (en) | Image segmentation method for industrial defect detection | |
| CN111882002B (en) | MSF-AM-based low-illumination target detection method | |
| CN109685776B (en) | Pulmonary nodule detection method and system based on CT image | |
| US7724256B2 (en) | Fast graph cuts: a weak shape assumption provides a fast exact method for graph cuts segmentation | |
| CN111523521A (en) | Remote sensing image classification method for double-branch fusion multi-scale attention neural network | |
| CN107169535A (en) | The deep learning sorting technique and device of biological multispectral image | |
| CN111310598B (en) | Hyperspectral remote sensing image classification method based on 3-dimensional and 2-dimensional mixed convolution | |
| CN117253154B (en) | Container weak and small serial number target detection and identification method based on deep learning | |
| CN114283120B (en) | Domain-adaptive-based end-to-end multisource heterogeneous remote sensing image change detection method | |
| CN111524140B (en) | Medical image semantic segmentation method based on CNN and random forest method | |
| CN116757988B (en) | Infrared and visible light image fusion method based on semantic enrichment and segmentation tasks | |
| CN111768415A (en) | Image instance segmentation method without quantization pooling | |
| CN116342516B (en) | Model integration-based method and system for assessing bone age of X-ray images of hand bones of children | |
| CN109190511A (en) | Hyperspectral classification method based on part Yu structural constraint low-rank representation | |
| CN112132878A (en) | End-to-end brain nuclear magnetic resonance image registration method based on convolutional neural network | |
| CN115526829A (en) | Honeycomb lung focus segmentation method and network based on ViT and context feature fusion | |
| CN113450313A (en) | Image significance visualization method based on regional contrast learning | |
| CN111899203A (en) | Real image generation method based on label graph under unsupervised training and storage medium | |
| CN117557779A (en) | YOLO-based multi-scale target detection method | |
| CN112215079A (en) | Global multistage target tracking method | |
| CN117392125B (en) | Mammary gland ultrasonic image analysis method and system based on deep convolutional neural network | |
| CN112766340B (en) | Depth capsule network image classification method and system based on self-adaptive spatial mode | |
| CN112990359A (en) | Image data processing method and device, computer and storage medium | |
| CN115170564A (en) | Colorectal cancer chemoradiotherapy reaction automatic prediction system | |
| CN115330759A (en) | Method and device for calculating distance loss based on Hausdorff distance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |