CN116664839B - Weak supervision semi-supervision defect perception segmentation method and device and electronic equipment - Google Patents
Weak supervision semi-supervision defect perception segmentation method and device and electronic equipment Download PDFInfo
- Publication number
- CN116664839B CN116664839B CN202310641253.6A CN202310641253A CN116664839B CN 116664839 B CN116664839 B CN 116664839B CN 202310641253 A CN202310641253 A CN 202310641253A CN 116664839 B CN116664839 B CN 116664839B
- Authority
- CN
- China
- Prior art keywords
- defect
- defect image
- image
- target
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims abstract description 441
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000011218 segmentation Effects 0.000 title claims abstract description 17
- 230000008447 perception Effects 0.000 title claims abstract description 13
- 230000004913 activation Effects 0.000 claims abstract description 30
- 230000006870 function Effects 0.000 claims description 46
- 238000012549 training Methods 0.000 claims description 41
- 238000001994 activation Methods 0.000 claims description 28
- 230000008569 process Effects 0.000 claims description 27
- 230000008439 repair process Effects 0.000 claims description 21
- 238000010586 diagram Methods 0.000 claims description 17
- 238000011176 pooling Methods 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 9
- 238000010606 normalization Methods 0.000 claims description 7
- 125000004122 cyclic group Chemical group 0.000 claims description 6
- 230000003213 activating effect Effects 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 238000007725 thermal activation Methods 0.000 claims description 3
- 230000002950 deficient Effects 0.000 description 11
- 238000004364 calculation method Methods 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 7
- 230000004044 response Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0475—Generative networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/094—Adversarial learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a weak supervision semi-supervision defect perception segmentation method, a device and electronic equipment, wherein the method comprises the steps of obtaining original characteristics of a target defect image through a downsampling module and a first residual convolution block of a defect repairing device; the method comprises the steps that an original binary mask corresponding to a defect position in a target defect image is obtained in advance according to original characteristics through a defect activation module of a defect repairing device; and inputting an original binary mask obtained by the defect activation module according to the original characteristics into a residual convolution block normalized and constructed by a defect example to obtain a target non-defect image. And carrying out binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask for predicting the defect position. The invention can reduce the workload of defect data marking and improve the defect perception performance, and can be widely applied to the field of computer vision.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a weak supervision semi-supervision defect perception segmentation method and device and electronic equipment.
Background
Defect detection has been the focus of research community attention due to the practical importance of quality assurance and maintenance of industrial products. However, while the development of computer vision has prompted effective resolution of defect detection tasks, the high cost of defect data labeling, particularly in task scenarios of pixel-level defect segmentation, limits the ability of deep learning-based approaches.
Therefore, the above-described problems are to be solved.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a method, an apparatus, and an electronic device for performing weak supervision and semi-supervision of defect sensing segmentation, which are used for reducing the workload of defect data labeling and improving the performance of defect sensing.
An aspect of an embodiment of the present invention provides a weakly-supervised semi-supervised defect-aware segmentation method, including:
inputting a target defect image into a pre-trained defect repair device to obtain a target non-defect image corresponding to the target defect image;
performing binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask for predicting the defect position;
the training process of the defect repair device is to perform counterdamage training with a defect generator, and the process of the counterdamage training comprises the following steps:
inputting a first defect image into a defect repairing device to obtain a first non-defect image corresponding to the first defect image; inputting the first non-defect image into a defect generator to obtain a defect image corresponding to the first non-defect image;
taking a new non-defect image as a second non-defect image, and inputting the second non-defect image into the defect generator to obtain a defect image corresponding to the second non-defect image; inputting a defect image corresponding to the second non-defect image into the defect repairing device to obtain a non-defect image corresponding to the second non-defect image;
and performing alternating countermeasure training on the non-defect images corresponding to the second non-defect images according to the defect images corresponding to the first non-defect images to obtain a pre-trained defect repair device.
Optionally, the inputting the target defect image to a pre-trained defect healer to obtain a target non-defect image corresponding to the target defect image includes:
acquiring original characteristics of a target defect image through a downsampling module of the defect healer and a first residual convolution block;
acquiring an original binary mask corresponding to a defect position in the target defect image according to the original characteristic by a defect activation module of the defect repair device;
and inputting the original binary mask obtained by the defect activation module according to the original characteristics into a residual convolution block constructed by normalizing the defect example to obtain a target non-defect image.
Optionally, the obtaining, by the defect activation module of the defect healer, an original binary mask corresponding to a defect position in the target defect image according to the original feature includes:
respectively carrying out global average pooling and global maximum pooling on the original features through a defect activation module to obtain a first classification weight corresponding to the defect repairing device and a second classification weight corresponding to the defect generator;
and acquiring an original binary mask corresponding to the defect position in the target defect image according to the first classification weight and the second classification weight and combining a thermodynamic diagram of the characteristic activation response.
Optionally, the obtaining an original binary mask corresponding to the defect position in the target defect image according to the first classification weight and the second classification weight and combining the thermodynamic diagram of the feature activation response includes:
calculating a thermodynamic activation graph of the original feature according to the first classification weight and the second classification weight;
and acquiring an original binary mask corresponding to the defect position in the target defect image according to the thermal activation diagram.
Optionally, the construction process of the second residual convolution block consisting of defect instance normalization includes:
extracting foreground features and background features through the original binary mask;
calculating the mean and variance of the foreground features and calculating the mean and variance of the background features;
normalizing the mean value and the variance of the foreground features and the mean value and the variance of the background features to obtain normalized features;
and merging the normalized features, and embedding the merged normalized features into a residual convolution block to obtain the second residual convolution block.
Optionally, the defect generator and the defect healer include identifying a data distribution of defect data and non-defect data during the loss fight training process.
Optionally, the penalty functions of the contrast penalty training process include training a contrast penalty function, a cyclic consistency penalty function, an identity mapping penalty function, a binary cross entropy penalty function, and a background consistency penalty function;
wherein the training fight loss function is used to transform the input image from the source domain to the target domain;
the cyclic consistency loss function is used for reconstructing the input image converted to the target domain back to the source domain image;
the identity mapping loss function is used for determining identity mapping output of a defect perception model, and the defect perception model is obtained according to the defect healer and the defect generator;
the binary cross entropy loss function is used for training a preset classifier so that the classifier can identify distinguishing features of an input defect image and a non-defect image;
the background consistency loss function is used for keeping the input defect image consistent with the background of the non-defect image.
Another aspect of the embodiments of the present invention further provides a weakly-supervised semi-supervised defect-aware segmentation apparatus, including:
the original characteristic acquisition unit is used for acquiring the original characteristic of the target defect image through the downsampling module of the defect healer and the first residual convolution block;
an original binary mask obtaining unit, configured to obtain, by using a defect activation module of the defect repair device, an original binary mask corresponding to a defect position in the target defect image according to the original feature;
the defect repairing unit is used for inputting the original binary mask obtained by the defect activating module according to the original characteristics into a residual convolution block constructed by normalizing defect examples to obtain a target non-defect image;
the training process of the defect repair device is to perform counterdamage training with a defect generator, and the process of the counterdamage training comprises the following steps:
inputting a first defect image into a defect repairing device to obtain a first non-defect image corresponding to the first defect image; inputting the first non-defect image into a defect generator to obtain a defect image corresponding to the first non-defect image;
taking a new non-defect image as a second non-defect image, and inputting the second non-defect image into the defect generator to obtain a defect image corresponding to the second non-defect image; inputting a defect image corresponding to the second non-defect image into the defect repairing device to obtain a non-defect image corresponding to the second non-defect image;
and performing alternating countermeasure training on the non-defect images corresponding to the second non-defect images according to the defect images corresponding to the first non-defect images to obtain a pre-trained defect repair device.
And the predicted binary mask obtaining unit is used for carrying out binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask of the predicted defect position.
Another aspect of the embodiment of the invention also provides an electronic device, which includes a processor and a memory;
the memory is used for storing programs;
the processor executing the program implements the method of any of the above.
Another aspect of the embodiments of the present invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the method of any one of the above.
Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the foregoing method.
The training process of the defect repairing device needs to be participated with a defect generator, and the defect repairing device can construct a defect image and non-defect image conversion neural network, namely the conversion from the defect image to the non-defect image and the conversion from the non-defect image to the defect image. After the defect repairing device and the defect generator are trained, the defect image can be directly predicted through the defect repairing device, and a corresponding non-defect image can be obtained. The invention can perform weak supervision and semi-supervision training based on the defect image only, does not need to use a large number of marked images, and meanwhile, the trained defect repairing device has higher prediction performance, can accurately output the non-defect image corresponding to the defect image, and realizes efficient segmentation of defect perception.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a diagram illustrating an exemplary implementation of a defect repair apparatus and a defect generator for performing loss-of-countermeasure training according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a weak supervision semi-supervised defect perception segmentation method according to an embodiment of the present invention;
FIG. 3 is an exemplary flowchart of a weakly-supervised semi-supervised defect aware segmentation method according to an embodiment of the present invention;
fig. 4 is a block diagram of a weak supervision semi-supervised defect sensing segmentation apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment of the invention provides a weak supervision semi-supervision defect perception segmentation method, which specifically comprises the following steps:
inputting the target defect image into a pre-trained defect repair device to obtain a target non-defect image corresponding to the target defect image.
And carrying out binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask for predicting the defect position.
First, describing the above-mentioned training process of the defect repair device, referring to fig. 1, an exemplary diagram of the defect repair device and the defect generator performing counterdamage training is provided in an embodiment of the present invention, which may specifically include the following steps:
s1, inputting a first defect image into a defect repair device to obtain a first non-defect image corresponding to the first defect image; and inputting the first non-defect image into a defect generator to obtain a defect image corresponding to the first non-defect image.
S2, taking a new non-defect image as a second non-defect image, and inputting the second non-defect image into the defect generator to obtain a defect image corresponding to the second non-defect image; and inputting the defect image corresponding to the second non-defect image into the defect repairing device to obtain the non-defect image corresponding to the second non-defect image.
S3, performing alternate countermeasure training on the non-defect image corresponding to the second non-defect image according to the defect image corresponding to the first non-defect image, and obtaining the pre-trained defect repair device.
In particular, the present invention may include a defect healer and a defect generator that construct a two-set image conversion neural network, i.e., a conversion of a defective image Y to a non-defective image X, and a conversion of the non-defective image X to the defective image Y. The defect healer and the defect generator of the invention can identify the data distribution of the defect data and the non-defect data so as to realize the countermeasure training process. After the defect repairing device and the defect generator are trained, the defect repairing network R can be used for directly predicting the defect image and obtaining the corresponding non-defect image X, namely, absolute value difference calculation abs (Y-X) can be directly carried out on the two images to obtain a predicted binary mask so as to extract the position of the defect.
Further, the above-mentioned step of inputting the target defect image into the pre-trained defect repair device to obtain the target non-defect image corresponding to the target defect image will be described in detail with reference to fig. 2.
S100: and acquiring original characteristics of the target defect image through a downsampling module of the defect healer and a first residual convolution block.
Specifically, the target defect image may be first input to a value downsampling module for downsampling, and then residual convolution is performed to obtain the original feature.
S110: and acquiring an original binary mask corresponding to the defect position in the target defect image according to the original characteristic by a defect activation module of the defect repairing device.
Specifically, given the original feature obtained by downsampling, the embodiment of the invention additionally trains a classifier to distinguish whether the original feature corresponds to a non-defective image or a defective image, namely, a defective image and a non-defective image based on the original feature are classified. Specifically, global average pooling and global maximum pooling can be performed on the original features, then corresponding classification weights are obtained through a defect image classifier and a non-defect image classifier respectively, logits are obtained, and the calculation expression of the classification weights is as follows:
F R-classifier (y)＝[P GMP ，P GAP ]，
wherein Feat (y) represents the thermodynamic diagram characteristics after passing through the residual block. sigma represents the sigmoid activation function. Wgmp and Wgap are the classification weights of the two sets of classifiers, respectively. GMP and GAP represent global max pooling and global average pooling, respectively. [, ] represents a con-figuration splice operation.
Through the obtained classification weight, the embodiment of the invention can calculate the thermodynamic activation diagram about the original feature, and further obtain the original binary mask corresponding to the defect position in the target defect image according to the thermodynamic activation diagram, and the specific calculation process is as follows:
A GMP ＝w GMP ⊙Feat(y)，
A GAP ＝w GAP ⊙ Feat(y)，
A＝[A GMP ，A GAP ]，
where a represents features obtained by two sets of classifier weights and thermodynamic diagram features. As indicated by the channel level dot product.
Where Ac represents the characteristic of the c-th channel of characteristic A, and summing is performed according to the channels to obtain the thermodynamic activation diagram.
S120: and inputting the original binary mask obtained by the defect activation module according to the original characteristics into a residual convolution block constructed by normalizing the defect example to obtain a target non-defect image.
Specifically, a second residual convolution block composed of defect instance normalization is first described.
The embodiment of the invention provides a feature normalization method based on a response thermodynamic diagram of a defect. Specifically, according to the binary mask obtained by thermodynamic diagram, extracting the binary mask to obtain the foreground feature and the background feature, wherein the extraction and calculation process is as follows:
where BKGD represents foreground features and FGD represents background features.
Then, calculating the mean and variance of the foreground features, calculating the mean and variance of the background features, normalizing, and calculating the mean and variance of the foreground features as follows:
where H and W represent feature height and width, respectively.
The mean and variance calculation process of the background features is as follows, and the normalization calculation process is as follows:
the normalized features are combined, and the specific calculation process is as follows:
and finally embedding the combined normalized features into a residual convolution block to obtain a second residual convolution block.
Next, the defect healer and defect generator are presented with loss functions during the combat loss training process, including loss functions including training combat loss functions, cyclic consistency loss functions, identity mapping loss functions, binary cross entropy loss functions, and background consistency loss functions.
Specifically, training the contrast loss function is used to transform the input image from the source domain to the target domain, expressed as follows:
where Dx represents a non-defect discriminator and R represents a defect healer. x and y represent a non-defective image and a defective image, respectively.
The cyclic consistency loss function is used to reconstruct the input image converted to the target domain back to the source domain image, expressed as follows:
as above, where R represents the defect healer and y represents the defect image. G represents a defect generator.
According to the embodiment of the invention, a two-class classifier can be additionally added and trained for identifying the distinguishing characteristics of the defect image and the non-defect image, so that the response to the specific defect area by using the activation thermodynamic diagram can be well performed. The binary cross entropy loss function is used to train the classifier, and the expression is as follows:
where F represents the classifier probability output.
Because, in the embodiment of the invention, the background of the defect image and the background of the corresponding non-defect image can be extracted through the binary mask, the background consistency loss function is used for keeping the input defect image consistent with the background of the non-defect image, and only the response of the defect point is concerned, and the background consistency loss function expression is as follows:
where M represents the operation of background extraction.
The identity mapping loss function is used to determine an identity mapping output of a defect perception model, which is derived from the defect healer and the defect generator.
In an alternative embodiment, the above loss functions are weighted and summed to obtain a total loss function of the anti-loss training process, where the expression is as follows:
where λ, β, γ, η represent the weight parameters of the total loss function, respectively.
And after the defect repairing device is trained, giving a target defect image, inputting the target defect image into a residual convolution block which is normalized and constructed by a defect example through the original binary mask obtained by the defect activating module according to the original characteristics, and obtaining a target non-defect image. And carrying out binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask for predicting the defect position.
The application of the present invention will be described in the following with specific examples.
Referring to fig. 3, an example flow chart of a weakly-supervised semi-supervised defect aware segmentation method is provided by an embodiment of the present invention.
Specifically, the defect healer may convert the defect image into a corresponding healed non-defect image. The architecture of the defect healer may include a defect activation module and a residual convolution block of defect instance normalization construction. First, the original features for the defective image are obtained by convolving the downsampled and residual convolution block. The original features pass through a defect activation module, the defect activation module performs unsupervised response on a defect region of the defect image, and outputs a binary mask of the defect position for a subsequent process of converting the defect image into a non-defect image. Then, the original features are sent to a residual convolution block normalized by the defect instance, further feature extraction and image conversion are carried out, and finally, a non-defect image corresponding to the defect image is obtained through convolution up-sampling, and the non-defect image is output. And carrying out binary difference on the defect image and the output non-defect image to obtain a predicted binary mask for predicting the defect position.
Referring to fig. 4, an embodiment of the present invention provides a weak supervision semi-supervised defect aware segmentation apparatus, including:
the original characteristic acquisition unit is used for acquiring the original characteristic of the target defect image through the downsampling module of the defect healer and the first residual convolution block;
an original binary mask obtaining unit, configured to obtain, in advance, an original binary mask corresponding to a defect position in the target defect image according to the original feature by using a defect activation module of the defect repair device;
the defect repairing unit is used for inputting the original binary mask obtained by the defect activating module according to the original characteristics into a residual convolution block constructed by normalizing defect examples to obtain a target non-defect image;
the training process of the defect repair device is to perform counterdamage training with a defect generator, and the process of the counterdamage training comprises the following steps:
inputting a first defect image into a defect repairing device to obtain a first non-defect image corresponding to the first defect image; inputting the first non-defect image into a defect generator to obtain a defect image corresponding to the first non-defect image;
taking a new non-defect image as a second non-defect image, and inputting the second non-defect image into the defect generator to obtain a defect image corresponding to the second non-defect image; inputting a defect image corresponding to the second non-defect image into the defect repairing device to obtain a non-defect image corresponding to the second non-defect image;
and performing alternating countermeasure training on the non-defect images corresponding to the second non-defect images according to the defect images corresponding to the first non-defect images to obtain a pre-trained defect repair device.
And the predicted binary mask obtaining unit is used for carrying out binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask of the predicted defect position.
Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the method shown in fig. 2.
In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.
Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the described functions and/or features may be integrated in a single physical device and/or software module or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments described above, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are included in the scope of the present invention as defined in the appended claims.
Claims (6)
1. A weakly-supervised semi-supervised defect-aware segmentation method, comprising:
inputting a target defect image into a pre-trained defect repair device to obtain a target non-defect image corresponding to the target defect image;
performing binary difference on the target defect image and the obtained corresponding target non-defect image to obtain a predicted binary mask for predicting the defect position;
the training process of the defect repair device is to perform counterdamage training with a defect generator, and the process of the counterdamage training comprises the following steps:
inputting a first defect image into a defect repairing device to obtain a first non-defect image corresponding to the first defect image; inputting the first non-defect image into a defect generator to obtain a defect image corresponding to the first non-defect image;
taking a new non-defect image as a second non-defect image, and inputting the second non-defect image into the defect generator to obtain a defect image corresponding to the second non-defect image; inputting a defect image corresponding to the second non-defect image into the defect repairing device to obtain a non-defect image corresponding to the second non-defect image;
according to the defect image corresponding to the first non-defect image, alternating countermeasure training is carried out on the non-defect image corresponding to the second non-defect image, and a pre-trained defect repairing device is obtained;
inputting the target defect image into a pre-trained defect healer to obtain a target non-defect image corresponding to the target defect image, wherein the method comprises the following steps of: acquiring original characteristics of a target defect image through a downsampling module of the defect healer and a first residual convolution block; acquiring an original binary mask corresponding to a defect position in the target defect image in advance according to the original characteristics by a defect activation module of the defect repair device; inputting the original binary mask obtained by the defect activation module according to the original characteristics into a residual convolution block constructed by normalizing defect examples to obtain a target non-defect image;
the defect activation module of the defect repairing device pre-obtains an original binary mask corresponding to a defect position in the target defect image according to the original characteristic, including: respectively carrying out global average pooling and global maximum pooling on the original features through a defect activation module to obtain a first classification weight corresponding to the defect repairing device and a second classification weight corresponding to the defect generator; acquiring an original binary mask corresponding to a defect position in the target defect image according to the first classification weight and the second classification weight and combining original features;
the obtaining an original binary mask corresponding to a defect position in the target defect image according to the first classification weight and the second classification weight and combining original features includes: calculating a thermodynamic activation graph of the original feature according to the first classification weight and the second classification weight; acquiring an original binary mask corresponding to a defect position in the target defect image according to the thermal activation diagram;
the construction process of the second residual convolution block consisting of defect instance normalization comprises the following steps: extracting foreground features and background features through the original binary mask; calculating the mean and variance of the foreground features and calculating the mean and variance of the background features; normalizing the mean value and the variance of the foreground features and the mean value and the variance of the background features to obtain normalized features; and merging the normalized features, and embedding the merged normalized features into a residual convolution block to obtain the second residual convolution block.
2. The method of claim 1, wherein the defect generator and the defect healer include identifying a data distribution of defect data and non-defect data during the loss-of-immunity training.
3. The method of claim 1, wherein the loss functions of the counterloss training process include training counterloss functions, cyclic consistency loss functions, identity mapping loss functions, binary cross entropy loss functions, and background consistency loss functions;
wherein the training fight loss function is used to transform the input image from the source domain to the target domain;
the cyclic consistency loss function is used for reconstructing the input image converted to the target domain back to the source domain image;
the identity mapping loss function is used for determining identity mapping output of a defect perception model, and the defect perception model is obtained according to the defect healer and the defect generator;
the binary cross entropy loss function is used for training a preset classifier so that the classifier can identify distinguishing features of an input defect image and a non-defect image;
the background consistency loss function is used for keeping the input defect image consistent with the background of the non-defect image.
4. A weakly-supervised semi-supervised defect aware segmentation apparatus, comprising:
the original characteristic acquisition unit is used for acquiring the original characteristic of the target defect image through the downsampling module of the defect healer and the first residual convolution block;
an original binary mask obtaining unit, configured to obtain, by using a defect activation module of the defect healer, a binary mask corresponding to a defect position in the target defect image according to the original feature;
the defect repairing unit is used for inputting the original binary mask obtained by the defect activating module according to the original characteristics into a residual convolution block constructed by normalizing defect examples to obtain a target non-defect image;
the training process of the defect repair device is to perform counterdamage training with a defect generator, and the process of the counterdamage training comprises the following steps:
inputting a first defect image into a defect repairing device to obtain a first non-defect image corresponding to the first defect image; inputting the first non-defect image into a defect generator to obtain a defect image corresponding to the first non-defect image;
taking a new non-defect image as a second non-defect image, and inputting the second non-defect image into the defect generator to obtain a defect image corresponding to the second non-defect image; inputting a defect image corresponding to the second non-defect image into the defect repairing device to obtain a non-defect image corresponding to the second non-defect image;
according to the defect image corresponding to the first non-defect image, alternating countermeasure training is carried out on the non-defect image corresponding to the second non-defect image, and a pre-trained defect repairing device is obtained;
a predicted binary mask obtaining unit, configured to perform binary difference on the target defect image and the obtained corresponding target non-defect image, to obtain a predicted binary mask of a predicted defect position;
the defect activation module of the defect repairing device pre-obtains an original binary mask corresponding to a defect position in the target defect image according to the original characteristic, including: respectively carrying out global average pooling and global maximum pooling on the original features through a defect activation module to obtain a first classification weight corresponding to the defect repairing device and a second classification weight corresponding to the defect generator; acquiring an original binary mask corresponding to a defect position in the target defect image according to the first classification weight and the second classification weight and combining original features;
the obtaining an original binary mask corresponding to a defect position in the target defect image according to the first classification weight and the second classification weight and combining original features includes: calculating a thermodynamic activation graph of the original feature according to the first classification weight and the second classification weight; acquiring an original binary mask corresponding to a defect position in the target defect image according to the thermal activation diagram;
the construction process of the second residual convolution block consisting of defect instance normalization comprises the following steps: extracting foreground features and background features through the original binary mask; calculating the mean and variance of the foreground features and calculating the mean and variance of the background features; normalizing the mean value and the variance of the foreground features and the mean value and the variance of the background features to obtain normalized features; and merging the normalized features, and embedding the merged normalized features into a residual convolution block to obtain the second residual convolution block.
5. An electronic device comprising a processor and a memory;
the memory is used for storing programs;
the processor executing the program implements the method of any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that the storage medium stores a program that is executed by a processor to implement the method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310641253.6A CN116664839B (en) | 2023-05-31 | 2023-05-31 | Weak supervision semi-supervision defect perception segmentation method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310641253.6A CN116664839B (en) | 2023-05-31 | 2023-05-31 | Weak supervision semi-supervision defect perception segmentation method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116664839A CN116664839A (en) | 2023-08-29 |
| CN116664839B true CN116664839B (en) | 2024-02-13 |
Family
ID=87709234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310641253.6A Active CN116664839B (en) | 2023-05-31 | 2023-05-31 | Weak supervision semi-supervision defect perception segmentation method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116664839B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117671431B (en) * | 2024-01-29 | 2024-05-07 | 杭州安脉盛智能技术有限公司 | Industrial defect image generation method, device, equipment and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111798409A (en) * | 2020-05-19 | 2020-10-20 | 佛山市南海区广工大数控装备协同创新研究院 | Deep learning-based PCB defect data generation method |
| CN112070733A (en) * | 2020-08-28 | 2020-12-11 | 深兰人工智能芯片研究院（江苏）有限公司 | Defect rough positioning method and device based on weak supervision mode |
| CN112581463A (en) * | 2020-12-25 | 2021-03-30 | 北京百度网讯科技有限公司 | Image defect detection method and device, electronic equipment, storage medium and product |
| CN112884758A (en) * | 2021-03-12 | 2021-06-01 | 国网四川省电力公司电力科学研究院 | Defective insulator sample generation method and system based on style migration method |
| CN113240673A (en) * | 2021-07-09 | 2021-08-10 | 武汉Tcl集团工业研究院有限公司 | Defect detection method, defect detection device, electronic equipment and storage medium |
| CN214895702U (en) * | 2021-05-12 | 2021-11-26 | 合肥锐威博衍科技有限公司 | On-line detection device for electric quantity of Tadilan lithium thionyl chloride battery |
| CN114049309A (en) * | 2021-10-26 | 2022-02-15 | 广州大学 | Image defect detection method and device based on semi-supervised network and storage medium |
| CN114841992A (en) * | 2022-05-31 | 2022-08-02 | 厦门大学嘉庚学院 | Defect detection method based on cyclic generation countermeasure network and structural similarity |
| CN115239719A (en) * | 2022-09-22 | 2022-10-25 | 南昌昂坤半导体设备有限公司 | Defect detection method, system, electronic device and storage medium |
| CN115240195A (en) * | 2022-08-04 | 2022-10-25 | 成都泓睿科技有限责任公司 | Medicine bottle defect automatic detection method and device and storage medium |
| CN115311504A (en) * | 2022-10-10 | 2022-11-08 | 之江实验室 | Weak supervision positioning method and device based on attention repositioning |
| CN115410024A (en) * | 2022-05-27 | 2022-11-29 | 福建亿榕信息技术有限公司 | Power image defect detection method based on dynamic activation thermodynamic diagram |
| CN115471463A (en) * | 2022-08-26 | 2022-12-13 | 江苏大学 | Solar cell defect detection method based on countermeasure type generation network |
| CN116188516A (en) * | 2022-12-30 | 2023-05-30 | 深存科技(无锡)有限公司 | Training method of defect data generation model |
-
2023
- 2023-05-31 CN CN202310641253.6A patent/CN116664839B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111798409A (en) * | 2020-05-19 | 2020-10-20 | 佛山市南海区广工大数控装备协同创新研究院 | Deep learning-based PCB defect data generation method |
| CN112070733A (en) * | 2020-08-28 | 2020-12-11 | 深兰人工智能芯片研究院（江苏）有限公司 | Defect rough positioning method and device based on weak supervision mode |
| CN112581463A (en) * | 2020-12-25 | 2021-03-30 | 北京百度网讯科技有限公司 | Image defect detection method and device, electronic equipment, storage medium and product |
| CN112884758A (en) * | 2021-03-12 | 2021-06-01 | 国网四川省电力公司电力科学研究院 | Defective insulator sample generation method and system based on style migration method |
| CN214895702U (en) * | 2021-05-12 | 2021-11-26 | 合肥锐威博衍科技有限公司 | On-line detection device for electric quantity of Tadilan lithium thionyl chloride battery |
| CN113240673A (en) * | 2021-07-09 | 2021-08-10 | 武汉Tcl集团工业研究院有限公司 | Defect detection method, defect detection device, electronic equipment and storage medium |
| CN114049309A (en) * | 2021-10-26 | 2022-02-15 | 广州大学 | Image defect detection method and device based on semi-supervised network and storage medium |
| CN115410024A (en) * | 2022-05-27 | 2022-11-29 | 福建亿榕信息技术有限公司 | Power image defect detection method based on dynamic activation thermodynamic diagram |
| CN114841992A (en) * | 2022-05-31 | 2022-08-02 | 厦门大学嘉庚学院 | Defect detection method based on cyclic generation countermeasure network and structural similarity |
| CN115240195A (en) * | 2022-08-04 | 2022-10-25 | 成都泓睿科技有限责任公司 | Medicine bottle defect automatic detection method and device and storage medium |
| CN115471463A (en) * | 2022-08-26 | 2022-12-13 | 江苏大学 | Solar cell defect detection method based on countermeasure type generation network |
| CN115239719A (en) * | 2022-09-22 | 2022-10-25 | 南昌昂坤半导体设备有限公司 | Defect detection method, system, electronic device and storage medium |
| CN115311504A (en) * | 2022-10-10 | 2022-11-08 | 之江实验室 | Weak supervision positioning method and device based on attention repositioning |
| CN116188516A (en) * | 2022-12-30 | 2023-05-30 | 深存科技(无锡)有限公司 | Training method of defect data generation model |
Non-Patent Citations (2)
| Title |
|---|
| Defect attention template generation cycleGAN for weakly supervised surface defect segmentation;Shuanlong Niu ET AL;《Pattern Recognition》;第123卷;1-13 * |
| 基于正样本的深度学习缺陷检测方法;邓勇;《中国优秀硕士学位论文全文数据库》(第03期);I138-1117 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116664839A (en) | 2023-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11645744B2 (en) | Inspection device and inspection method | |
| CN112329702B (en) | Method and device for rapid face density prediction and face detection, electronic equipment and storage medium | |
| CN116664839B (en) | Weak supervision semi-supervision defect perception segmentation method and device and electronic equipment | |
| Marques et al. | Automatic road pavement crack detection using SVM | |
| CN114170184A (en) | Product image anomaly detection method and device based on embedded feature vector | |
| CN112949344B (en) | Characteristic autoregression method for anomaly detection | |
| Chen et al. | Deep-learning-based road crack detection frameworks for dashcam-captured images under different illumination conditions | |
| CN112348011B (en) | Vehicle damage assessment method and device and storage medium | |
| Neagoe et al. | A concurrent neural network approach to pedestrian detection in thermal imagery | |
| CN116630286B (en) | Method, device, equipment and storage medium for detecting and positioning image abnormality | |
| KR20190125569A (en) | Method and Apparatus for Generating Scene Situation Information of Video Using Differentiation of Image Feature and Supervised Learning | |
| CN112613341A (en) | Training method and device, fingerprint identification method and device, and electronic device | |
| CN116580232A (en) | Automatic image labeling method and system and electronic equipment | |
| Guo et al. | Udtiri: An open-source road pothole detection benchmark suite | |
| Mousavi et al. | Crowd estimation using histogram model classificationbased on improved uniform local binary pattern | |
| Pandey et al. | CoreDeep: Improving crack detection algorithms using width stochasticity | |
| Mathiyalagan et al. | Image fusion using convolutional neural network with bilateral filtering | |
| Zhang et al. | Recognition of bridge over water in remote sensing image using discrete hopfield neural network | |
| Balakrishnan et al. | Image denoising and contrast via intensity histogram equalization method | |
| CN117934979B (en) | Target identification method based on fractal coder-decoder | |
| García-Gozález et al. | Foreground Segmentation Improvement by Image Denoising Preprocessing Applied to Noisy Video Sequences | |
| Niaz et al. | Increment-CAM: Incrementally-weighted Class Activation Maps for Better Visual Explanations | |
| Bukała et al. | The impact of distortions on the image recognition with histograms of oriented gradients | |
| Vankayalapati et al. | Interfacing Kera’s deep learning technique for Real-Time Age and Gender Prediction | |
| Sahli et al. | Automatic Recognition System of Tunisian License Plates for Car Park Access Control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |