CN113033656A - Interactive hole exploration data expansion method based on generation countermeasure network - Google Patents
Interactive hole exploration data expansion method based on generation countermeasure network Download PDFInfo
- Publication number
- CN113033656A CN113033656A CN202110311132.6A CN202110311132A CN113033656A CN 113033656 A CN113033656 A CN 113033656A CN 202110311132 A CN202110311132 A CN 202110311132A CN 113033656 A CN113033656 A CN 113033656A
- Authority
- CN
- China
- Prior art keywords
- network
- defect
- image
- interactive
- shape
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 40
- 230000007547 defect Effects 0.000 claims abstract description 133
- 238000012549 training Methods 0.000 claims abstract description 56
- 239000013598 vector Substances 0.000 claims abstract description 48
- 238000001514 detection method Methods 0.000 claims abstract description 42
- 230000004927 fusion Effects 0.000 claims abstract description 25
- 238000012360 testing method Methods 0.000 claims abstract description 8
- 230000003044 adaptive effect Effects 0.000 claims abstract description 6
- 230000011218 segmentation Effects 0.000 claims description 21
- 239000011248 coating agent Substances 0.000 claims description 5
- 238000000576 coating method Methods 0.000 claims description 5
- 238000002679 ablation Methods 0.000 claims description 4
- 238000005299 abrasion Methods 0.000 claims description 4
- 230000002950 deficient Effects 0.000 claims description 4
- 238000011068 loading method Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims 1
- 239000011148 porous material Substances 0.000 claims 1
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000012952 Resampling Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000013434 data augmentation Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- BATBOVZTQBLKIL-UHFFFAOYSA-N [1-(5,8-dihydroxy-1,4-dioxonaphthalen-2-yl)-4-methylpent-3-enyl] 3-methylbut-2-enoate Chemical compound C1=CC(O)=C2C(=O)C(C(OC(=O)C=C(C)C)CC=C(C)C)=CC(=O)C2=C1O BATBOVZTQBLKIL-UHFFFAOYSA-N 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 238000011550 data transformation method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an interactive hole exploration data expansion method based on a generated countermeasure network, which comprises the following steps: classifying the defect images according to defect types, storing the defect images in corresponding folders, constructing an aeroengine hole detection image data set, and constructing and generating a confrontation network model structure based on depth convolution; training a generated confrontation network model, and acquiring one or more engine hole detection defect image generators for generating different defects; constructing a P network, inputting shape information in a training sample, coding the shape information into an implicit vector identified by an engine hole detection defect image generator, and training the P network by using the characteristics of a fourth convolution layer of an AlexNet model; acquiring a generated image with a specified shape based on the hidden vector and the engine hole detection defect image generator, and testing through the trained P network; and constructing an interactive basic framework, segmenting defects by using an adaptive threshold, and correcting the edge traces of the fusion region by using a Poisson fusion algorithm.
Description
Technical Field
The invention relates to the field of data expansion, in particular to an interactive hole exploration data expansion method based on a generation countermeasure network.
Background
Data Augmentation (Data Augmentation) is one of the most practical ways to enhance model performance without increasing computational cost during the deep neural network training phase. Data expansion can be divided into two categories: one is that a traditional data transformation method is used, for example, a random clipping algorithm introduced in document [1] can make the network less sensitive to the scale information of the target to be detected to some extent, thereby improving the identification effect of the network on small objects; the random erasure algorithm introduced in the document [2] can achieve the purpose similar to random clipping; the geometric transformation algorithm comprises classical data expansion algorithms such as translation, scaling, affine transformation, perspective transformation and the like, and is one of the most popular data expansion methods; the color space transformation algorithm introduced in the document [3] can simulate different illumination and color temperature environments in the forms of color dithering and the like, and improve the robustness of the model to image colors; the neural style conversion algorithm described in document [4] can change the style, texture, and other features of an image. The reasonable use of the methods can achieve the aims of rapidly expanding the data set and enhancing the robustness of the model to a certain extent.
However, for the task of target detection, these methods only change the representation form of the image, do not increase the proportion of the target to be detected in the sample space, and cannot solve the problem of uneven distribution of the target to be detected in the data set. Therefore, another data expansion method needs to be adopted: a data resampling method. The data resampling method is that before or during model training, the sampling frequency of the classes with more samples is reduced, and the sampling frequency of the classes with less samples is increased, so that the number of the samples of various classes is maintained at a more balanced level. Document [5] improves the accuracy of the face detection model by a bidirectional resampling technique. The current data resampling method focuses mainly on the synthesis of new examples. For example, the image mixing method proposed in document [6] improves the smoothness in the neighborhood by serializing the discrete sample space by interpolation; the feature space expansion method proposed in document [7] obtains more reasonable synthetic data by applying data transformation to the feature level; documents [8] and [9] generate some more realistic samples for deep learning task by using GAN (generation countermeasure network) to generate images. These methods can generate new extended images, but the newly generated extended images have no pixel-level labels, can only be manually re-labeled, or are only used for the classification task of the images.
The label information at the pixel level is important in the object detection task. Document [10] proposes a method for randomly synthesizing a scene image and a labeled target instance, but experiments show that if context information of a scene is ignored, only the target instance and the image are randomly combined without considering whether the target instance can appear at a specific position in the scene, the purpose of improving detection accuracy cannot be achieved, and even the performance of a model may be reduced.
Reference to the literature
[1]Liu W,Anguelov D,Erhan D,et al.SSD:Single shot multibox detector[C]//European conference on computer vision.Springer,Cham,2016:21-37.
[2]Zhong Z,Zheng L,Kang G,et al.Random Erasing Data Augmentation[C]//AAAI.2020:13001-13008.
[3]Shorten C,Khoshgoftaar T M.A survey on image data augmentation for deep learning[J].Journal of Big Data,2019,6(1):60.
[4]Jing Y,Yang Y,Feng Z,et al.Neural style transfer:A review[J].IEEE transactions on visualization and computer graphics,2019.
[5] Liwenhui, Yuan, Wangying, etc., a face recognition algorithm [ J ] based on resampling two-way 2DLDA fusion, 2011,39(11):2526 + 2533.
[6]Zhang H,Cisse M,Dauphin Y N,et al.mixup:Beyond Empirical Risk Minimization[C]//International Conference on Learning Representations.2018.
[7]DeVries T,Taylor G W.Dataset augmentation in feature space[J].arXiv preprint arXiv:1702.05538,2017.
[8]Antoniou A,Storkey A,Edwards H.Data augmentation generative adversarial networks[J].arXiv preprint arXiv:1711.04340,2017.
[9]Wang Y X,Girshick R,Hebert M,et al.Low-shot learning from imaginary data[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2018:7278-7286.
[10]Dvornik N,Mairal J,Schmid C.Modeling visual context is key to augmenting object detection datasets[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:364-380.
[11]Steiner B,Devito Z,Chintala S,et al.PyTorch:An Imperative Style,High-Performance Deep Learning Library[C].neural information processing systems,2019:8026-8037.
[12]Deng J,Dong W,Socher R,et al.ImageNet:A large-scale hierarchical image database[C].computer vision and pattern recognition,2009:248-255.
[13]Krizhevsky A,Sutskever I,Hinton G E,et al.ImageNet Classification with Deep Convolutional Neural Networks[C].neural information processing systems,2012:1097-1105.
[14]Radford A,Metz L,Chintala S,et al.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[C].international conference on learning representations,2016.
[15]Canny J.A Computational Approach to Edge Detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1986,8(6):679-698.
[16]Perez P,Gangnet M,Blake A,et al.Poisson image editing[C].international conference on computer graphics and interactive techniques,2003,22(3):313-318.
[17]Land E H,Mccann J J.Lightness and Retinex Theory[J].Journal of the Optical Society of America,1971,61(1):1-11.
Disclosure of Invention
The invention provides an interactive hole exploration data expansion method based on a generated countermeasure network, which provides the construction and training of the generated countermeasure network and a P network, and adopts an interactive interface to select the type, shape and position of a generated defect by a user; encoding the shape information into a hidden vector using the trained P-network model; generating defects of a specified shape by generating a countermeasure network generator decoding; dividing defects in the generated image by adopting a mode of manually adjusting a threshold value; the fusion edge is corrected using a poisson fusion algorithm, as described in detail below:
an interactive hole-exploring data expansion method based on generation of a countermeasure network, the method comprising:
classifying the defect images according to defect types, storing the defect images in corresponding folders, constructing an aeroengine hole detection image data set, and constructing and generating a confrontation network model structure based on depth convolution;
training a generated confrontation network model, and acquiring one or more engine hole detection defect image generators for generating different defects;
constructing a P network, inputting shape information in a training sample, coding the shape information into an implicit vector identified by an engine hole detection defect image generator, and training the P network by using the characteristics of a fourth convolution layer of an AlexNet model;
based on the hidden vector and the engine hole detection defect image generator, obtaining a generated image with a specified shape, and testing through a trained P network;
and constructing an interactive basic framework, segmenting defects by using an adaptive threshold, and correcting the edge traces of the fusion region by using a Poisson fusion algorithm.
The defect types are divided into four types of cracks, ablation, abrasion and coating loss, and a generation model is trained independently for each type of defect.
Further, the P network includes 5 convolutional layers, the first convolutional layer for convolving 64 × 3 pictures into a 32 × 128 tensor; a second convolution layer for convolving the 32 x 128 tensor into a 16 x 256 tensor, a third convolution layer for convolving the 16 x 256 tensor into an 8 x 512 tensor, and a fourth convolution layer for convolving the 8 x 512 tensor into a 4 x 1024 tensor; the last convolution layer is used to convolve the 4 x 1024 tensors into 100 dimensional vectors.
The shape information in the input training sample is encoded into an implicit vector recognized by an engine hole detection defect image generator, and the training of the P network by using the characteristics of the fourth convolution layer of the AlexNet model specifically comprises the following steps:
extracting shape information of the training samples, and constructing an image pair according to the training samples and the corresponding shape information; inputting shape information into PiA network encoded into hidden vectors;
the hidden vector is decoded by an engine hole detection defect image generator to generate a defect image; loading an AlexNet model trained on the Imagenet data set, and extracting the characteristics of a conv4 layer from the generated defect image; extracting conv4 layer features from the training sample corresponding to the shape information;
calculating the minimum mean square error of the characteristics of the two conv4 layers as PiA loss function of the network model; shape information as PiAnd the output implicit vector of the network is used as the input of the generator to obtain a generated image with a specified shape and obtain the target function.
Further, the objective function is specifically:
wherein C is a conv4 layer characteristic of AlexNet, GiTo a generator, ziThe vector is a hidden vector, and the vector is a hidden vector,
to train the samples, θPiIs PiParameters to be updated in the network.
The testing by using the trained P network specifically comprises the following steps:
input shape I to be specifiedshapeInputting trained PiGenerating hidden vectors z in a networkiFor hidden vector ziAdding a fine noise disturbance N (z),
Dfake＝Gi(Pi(Ishape)+N(z))
wherein D isfakeFor the defect image generated, Pi(Ishape) Represents the utilization of PiNetwork prediction input shape IshapeN (z) represents a fine noise with gaussian distribution.
Further, the building of the interactive basic framework and the segmentation of the defects by using the adaptive threshold are specifically as follows:
selecting a corresponding defect generator, acquiring strokes input by a user in real time, and extracting the strokes as input shapes; by PiNetwork encodes input strokes into ziReuse of GiNetwork pair ziDecoding to generate a defective image Dfake；
Then D is putfakeFusing the stroke drawn by the user to generate a specified type defect which is similar to the shape of the stroke and has the same position; by making a pair of ziUsing different noise disturbances to generate a plurality of different defect images with similar shapes and slight differences;
and changing the low threshold value tau in Canny in real time, and adjusting the segmentation results of different defect images by setting a high threshold value.
Wherein the method further comprises: an interactive interface is constructed based on the Pyqt5 framework.
The technical scheme provided by the invention has the beneficial effects that:
1. the method constructs and generates a confrontation network, uses a hole detection image data set as a training sample, learns the distribution of aeroengine defect images, and obtains an engine hole detection defect image generator which can generate specified type defects and has a vivid effect;
2. the method constructs a P network model for encoding the shape information of an input image into a hidden vector; decoding the hidden vector by using an engine hole detection defect image generator to generate a defect with a shape similar to that of the input image;
3. the invention carries out tiny noise disturbance on the hidden vector obtained by P network model coding, and can obtain a plurality of defect images with similar shapes but different details;
4. the method adopts a self-adaptive threshold value mode to segment the generated defects, and adopts a Poisson fusion algorithm to correct the edges of the fusion region, the generated new engine image has no fusion trace, the defect generation position accords with a defect generation mechanism, the marking information can be automatically generated, and the generated engine data set has excellent performance after being detected by professional personnel;
5. the interactive data expansion module is designed, and the advantages of the generation countermeasure network, the P network and the Poisson fusion algorithm are combined, so that the generated image is more in line with the requirement of project development.
Drawings
FIG. 1 is a schematic diagram of a training process of a P network model according to the present invention;
FIG. 2 is a schematic diagram of a generative countermeasure network architecture for use with the present invention;
FIG. 3 is a diagram of a P network model architecture according to the present invention;
FIG. 4 is a flow chart of an interactive data expansion module according to the present invention;
FIG. 5 is a schematic diagram of an interactive interface designed according to the present invention;
FIG. 6 is a schematic diagram of a segmentation result and its extended image under different thresholds obtained by adjusting the segmentation threshold in real time according to the present invention;
FIG. 7 is an aeroengine hole-detecting image contrast map generated by the method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
The method provides an interactive hole detection data expansion method based on a generation countermeasure network for the defect detection problem in the hole detection image of the aeroengine, and can generate a target to be detected on the background of the hole detection image of the aeroengine and enable the target to be detected to be closer to a real image.
Example 1
An interactive hole exploration data expansion method based on generation of a countermeasure network, referring to fig. 1 and 4, comprises the following steps:
firstly, constructing a training set of hole detection images of an aircraft engine, and training to generate a confrontation network
The method comprises the following steps: reading an engine hole detection image with marking information, intercepting defect information from the engine hole detection image, respectively storing the defect information as a defect image only containing the defect information and a defect image (namely 2 defect types) containing background information, classifying the two defect images according to the defect types again, and storing the two defect images under different folders so as to construct an aeroengine hole detection image training set.
In the embodiment of the invention, the defects of the aero-engine are divided into four types, namely cracks, ablation, abrasion and coating loss, and the model is generated for each type of defects through independent training. When the crack generation model is trained, the characteristic that the length-width ratio of crack defects is large is considered, the longer side is cut by taking the shorter side as a reference, the long and narrow crack images are cut into a plurality of square images, and 550 crack defect images are reserved as a training set after manual screening. And if the length and the width of other defects are relatively close, the length and the width are ensured to be consistent by directly adopting a scaling mode, 902 ablated images, 206 abraded images and 1712 lost images of the coating are reserved after screening. The embodiments of the present invention are described by taking the above numerical values as examples, and when the embodiments of the present invention are implemented specifically, the embodiments of the present invention are not limited to this.
The training process of the generated confrontation network in the embodiment of the invention refers to the training GAN part in fig. 1, the generated confrontation network model structure adopts the deep convolution proposed in the reference [14] to generate the confrontation network model DCGAN, and the model structure comprises two parts of a generation network and a discrimination network. The generated network includes 5 deconvolution layers, the first deconvolution layer for deconvolving a 100-dimensional implicit vector to a 4 x 1024 tensor, the second deconvolution layer for deconvolution of a 4 x 1024 tensor to a 8 x 512 tensor, the third deconvolution layer for deconvolution of a 8 x 512 tensor to a 16 x 256 tensor, the fourth deconvolution layer for deconvolution of a 16 x 256 tensor to a 32 x 128 tensor, and the last deconvolution layer for deconvolution of a 32 x 128 tensor to a 64 x 3 picture. The generating network may generate 100 dimensional hidden vectors into 64 x 3 pictures through a deconvolution layer.
The judging network structure is opposite to the generating network structure, the model structure comprises 5 convolution layers, and the first convolution layer is used for convolving the pictures of 64 x 3 into a tensor of 32 x 128; the second convolution layer is used to convolve the 32 x 128 tensor into a 16 x 256 tensor, the third convolution layer is used to convolve the 16 x 256 tensor into an 8 x 512 tensor, and the fourth convolution layer is used to convolve the 8 x 512 tensor into a 4 x 1024 tensor; the last convolution layer is used to convolve the 4 x 1024 tensors into a 2-dimensional vector. The decision network may compress an input picture with a size of 64 × 3 into a 2-dimensional vector through a convolution layer, and use a first dimension of the 2-dimensional vector to represent a probability that the input picture is a real picture, and a second dimension to represent a probability that the input picture is a false picture (a generated network generated picture). The network structure of DCGAN is shown in fig. 2.
Embodiments of the invention use a Pythrch framework[11]And adjusting the training parameters, and learning the distribution of the aeroengine defect images by using an engine hole detection image training set as a training sample. For four types of defect numbersThe data set is trained for 1000 epochs respectively (each epoch means that all training data is completely trained once), and 4 engine hole inspection defect image generators which can generate specified types of defects (namely cracks, ablation, abrasion and coating loss) and have vivid effects are obtained.
Second, constructing and training P network
The training process of the P network in the embodiment of the present invention is shown in fig. 1. Wherein the structure of the P network portion comprises 5 convolutional layers, the first convolutional layer for convolving 64 x 3 pictures into a 32 x 128 tensor; the second convolution layer is used to convolve the 32 x 128 tensor into a 16 x 256 tensor, the third convolution layer is used to convolve the 16 x 256 tensor into an 8 x 512 tensor, and the fourth convolution layer is used to convolve the 8 x 512 tensor into a 4 x 1024 tensor; the last convolution layer is used to convolve the 4 x 1024 tensors into a 100-dimensional vector. The P-network structure is shown in fig. 3. Before training the P network, the training of generating the confrontation network is completed to obtain one or more generator G capable of generating different defects and having vivid effecti。
Engine hole detection defect image generator GiAfter training is finished, extracting shape information of the training sample in the first step
And press the training sample
And shape information corresponding to the training sample
Constructing an image pair
Because the generator G is needed in the P network training processiIn participation, different producers will train out different P networks, so PiNetwork and generator GiOne-to-one correspondence is realized; form information
Input PiNetwork encoded as a hidden vector zi。
Formalization is as follows:
in the formula (I), the compound is shown in the specification,
is PiThe parameters to be updated in the network are,
is the shape information of the training sample.
Wherein the hidden vector ziDefect image generator G for engine hole detectioniGenerating a defective image G after decodingi(z); loading Imagenet-based datasets[12]Trained AlexNet[13]The model is used, and the AlexNet model is used for extracting the characteristics of the conv4 layer from the generated defect image; then the shape information is processed
Corresponding training sample
Extracting conv4 layer features; calculating L2 loss (minimum mean square error) of the characteristics of two conv4 layers as PiLoss function of the network model. Shape information
As PiInput to the network, which outputs a hidden vector ziAs a generator GiThe generated image of the specified shape can be obtained.
The objective function is as follows:
wherein C is the conv4 layer characteristic of AlexNet. Using 4 different defect generators respectivelyTrain 4 different PiNetwork of each PiAfter 1000 epochs are trained by the network, the shape information can be encoded into a hidden vector on the training set corresponding to the defect more accurately.
Thirdly, testing by using the trained P network
Referring to fig. 4, a P-network test flow section in the embodiment of the present invention specifies an input shape IshapeInputting trained PiGenerating hidden vectors z in a networkiFor hidden vector ziAnd adding fine noise disturbance N (z) to achieve the aim that each input can generate a defect image with similar shape but different details.
The process can be formalized as: dfake＝Gi(Pi(Ishape)+N(z)) (3)
Wherein D isfakeFor the defect image generated, Pi(Ishape) Represents the utilization of PiNetwork prediction input shape IshapeIs represented by a hidden vector. N (z) represents a fine noise conforming to a Gaussian distribution.
Fourthly, constructing an interactive basic framework and segmenting defects by using adaptive threshold
In the embodiment of the invention, the interactive flow is shown in figure 4, the interactive interface is shown in figure 5, and before the interactive program runs, the defect image generator G is ensurediAnd shape information coding model PiAnd finishing the training.
Adding event function in interactive interface, real-time obtaining user wanted defect type, selecting corresponding defect generator Gi(ii) a The interactive interface allows the user to draw strokes, and the strokes input by the user can be acquired in real time and extracted as an input shape Ishape(i.e., a specified shape entered by the user through the interactive interface); clicking the update button after completing the rendering can pass through PiNetwork encodes input strokes into ziReuse of GiNetwork pair ziDecoding to generate a defective image Dfake(ii) a Then D is putfakeFusing the stroke drawn by the user to generate a specified type defect which is similar to the shape of the stroke and has the same position; simultaneous connectionZ is passing pairiA plurality of different defect images with similar shapes but slight differences are generated by using different noise disturbances and displayed on the right side of the main interface, and the defect images can be moved to the main interface to be viewed by clicking a small window corresponding to the right side.
The Canny edge detection algorithm proposed in Canny 1986 can be used for the generated defect image[15]Performing edge extraction on the generated defect image; then, connecting relatively close edges through morphological operation, and searching a maximum connected domain of edge information as a true value of a corresponding defect contour in marking information of a subsequent pixel level; the Canny edge detection algorithm has two thresholds of high and low, the set threshold is too high, important information can be missed, the threshold is too low, branch information can be regarded as important, and a general threshold suitable for all images is difficult to give, so that the low threshold tau in the Canny algorithm is changed in real time by a user, the segmentation results of different defect images are adjusted by setting the high threshold to 3 tau, and a suitable segmentation profile D is obtainedmask。Dmask＝Canny(Dfake,τ) (4)
And finally, clicking a storage button to store the current main interface image and automatically generating a matched annotation information file for the new sample.
Fifthly, correcting the edge trace of the fusion area by using Poisson fusion algorithm
The Poisson fusion algorithm is Perez et al (2003)[16]The method for naturally interpolating the boundary of the region to be fused by utilizing the gradient information is provided. The method utilizes psychologist Land (1971) in article [17]The principle proposed in (1) is to smooth the gradient of gradual change in the image through the limitation of a Laplace operator, so as to reduce the trace left by fusing two images.
Poisson fusion algorithm enables background image InewAccording to the defect image D at the edge of the region to be fusedfakeGeneration of gradient information with background image InewSimilar pixels to achieve a smoothing effect; dfakeEdge region of
Can pass throughContour information D in the fourth stepmaskThus obtaining the product.
The process can be formalized as:
wherein, InewIn order to have a new image after the fusion,
representing edge regions for fused images
The poisson fusion algorithm is executed and the (x, y) coordinates of the center of the stroke are input by the user.
Network training and testing
Based on a Pythrch deep learning network framework, training by using the confrontation network model and the data set generated in the first step to obtain an engine hole detection defect image generator with vivid generation effect; and training by using the P network model and the data set introduced in the second step to obtain the P network model. The P-network model, in combination with the engine hole inspection defect image generator, may generate a defect image of a specified shape.
Constructing an interactive interface based on a Pyqt5 framework (which is an open source library in a python development environment and is used for building a visual interface and is well known in the art), wherein the interactive interface is shown in fig. 5 and comprises: the system comprises a main interface interaction area, a small window area, a defect type selection area, a partition threshold value adjusting area and a functional area;
displaying a background image through a main interface interaction area, and acquiring stroke information input by a user; the small window area is used for displaying other generated defect expansion results except the main interface; the defect type selection area is used for selecting the defect type which the user wants to generate; the segmentation threshold adjusting area is used for manually adjusting the segmentation threshold and segmenting the generated defects to obtain pixel-level marking information; the functional area selects different background images by clicking a previous button and a next button, converts strokes input by a user in the main interface area into corresponding defect information by clicking an update button, stores the main interface expansion image under a designated folder by clicking a save button, and simultaneously generates the label information of the expansion image.
Selecting defects to be generated by a user through an interactive interface; using a mouse to outline the shape of the defect to be generated at the position of the defect to be generated; inputting the stroke shape into a P network and coding the stroke shape into a hidden vector; inputting the hidden vector into an engine hole detection defect image generator for decoding to generate a defect with a specified shape; the segmentation threshold is adjusted through the segmentation threshold adjusting area to obtain a more accurate defect segmentation result, and the segmentation result pairs of different thresholds are shown in FIG. 6; and fusing the generated defect image with the background image, and correcting the segmented defect boundary by using a Poisson fusion algorithm to reduce fusion traces to obtain an engine hole detection image with new defects. The resulting image obtained by the embodiment of the present invention is shown in fig. 7.
In summary, the embodiment of the present invention uses an interactive data extension method based on a generated countermeasure network to encode the input shape information by constructing and training a P network, so as to obtain a hidden vector; decoding the image by an engine hole detection defect image generator to generate a defect image with a specified shape; and the method for adjusting the segmentation threshold is used for segmenting the defect image, the Poisson fusion technology is used for solving the problem of obvious image fusion edge traces, and various requirements in practical application are met.
Example 2
The feasibility verification of the scheme in example 1 is performed with reference to fig. 6 and 7, which are described in detail below:
establishing and training a generation countermeasure network model and a P network model in the embodiment of the invention according to the network structure and the training process shown in the figures 1, 2 and 3; according to the flow and the interface shown in fig. 4 and 5, an interactive framework is built, and the defect type, the defect position and the defect shape information selected by a user are acquired in real time; the shape information is encoded through a P network model, and a defect image with a specified shape is generated through decoding of a generation model; adjusting the segmentation result of the defect image by manually adjusting the segmentation threshold; and fusing the generated defect image into a background image, and performing Poisson correction on the edge area of the fused defect image to eliminate the fusion trace.
In the interactive interface of fig. 5, the main interface is a background image, and the user can draw strokes on the main interface; the first small window in the five small windows on the right side represents an unexpanded original image, and the remaining four small windows represent a plurality of expanded defect images which are similar to the main interface stroke in shape and different in details; the lower part is sequentially provided with a defect type selection area used for selecting the defect type to be generated; a segmentation threshold adjusting area, which takes a value of 0-255 and is used for adjusting the segmentation threshold of the generated defect; the storage button is used for storing the expanded image of the main interface and automatically generating the annotation information of the expanded image; an update button for updating the strokes drawn by the user to the specified type of defects; and selecting other hole detection images under the current folder by the previous button and the next button.
As shown in fig. 6, it can be seen that in the embodiment of the present invention, different segmentation results can be obtained on the generated defect image by manually adjusting the segmentation threshold, and a more accurate segmentation result can be obtained by manually fine-tuning, so as to provide reference for automatically generating the annotation information.
From fig. 7, it can be seen that the data expansion result obtained by the embodiment of the present invention has no obvious fusion trace and high sample availability. In fig. 7, the first column is the original image, the second column and the fourth column are the images generated by the embodiment of the present invention, and the third column and the fifth column are the labeling information automatically generated by the embodiment. As can be seen from the experimental comparison result of fig. 7, compared with the existing method for performing data expansion on the whole image, the method provided in the embodiment of the present invention can add detection information on a fixed background to generate a new training sample, and the boundary of the fusion region of the new training sample is smoother and more natural.
In the embodiment of the present invention, except for the specific description of the model of each device, the model of other devices is not limited, as long as the device can perform the above functions.
Those skilled in the art will appreciate that the drawings are only schematic illustrations of preferred embodiments, and the above-described embodiments of the present invention are merely provided for description and do not represent the merits of the embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (8)
1. An interactive hole-exploring data expansion method based on a generation countermeasure network, the method comprising:
classifying the defect images according to defect types, storing the defect images in corresponding folders, constructing an aeroengine hole detection image data set, and constructing and generating a confrontation network model structure based on depth convolution;
training a generated confrontation network model, and acquiring one or more engine hole detection defect image generators for generating different defects;
constructing a P network, inputting shape information in a training sample, coding the shape information into an implicit vector identified by an engine hole detection defect image generator, and training the P network by using the characteristics of a fourth convolution layer of an AlexNet model;
acquiring a generated image with a specified shape based on the hidden vector and the engine hole detection defect image generator, and testing through the trained P network;
and constructing an interactive basic framework, segmenting defects by using an adaptive threshold, and correcting the edge traces of the fusion region by using a Poisson fusion algorithm.
2. The interactive hole detection data expansion method based on the generation countermeasure network, according to claim 1, characterized in that the defect types are divided into four types, namely crack, ablation, abrasion and coating loss, and a generation model is trained separately for each type of defect.
3. The interactive pore exploration data expansion method based on generation of countermeasure networks according to claim 1, characterized in that the P network comprises 5 convolution layers, the first convolution layer is used to convolve 64 x 3 pictures into 32 x 128 tensors; a second convolution layer for convolving the 32 x 128 tensor into a 16 x 256 tensor, a third convolution layer for convolving the 16 x 256 tensor into an 8 x 512 tensor, and a fourth convolution layer for convolving the 8 x 512 tensor into a 4 x 1024 tensor; the last convolution layer is used to convolve the 4 x 1024 tensors into 100 dimensional vectors.
4. The interactive hole detection data expansion method based on generation of the countermeasure network according to claim 1, wherein the shape information in the input training sample is encoded into a hidden vector recognized by an engine hole detection defect image generator, and the training of the P network by using the features of the fourth convolution layer of the AlexNet model is specifically as follows:
extracting shape information of the training samples, and constructing an image pair according to the training samples and the corresponding shape information; inputting shape information into PiA network encoded into hidden vectors;
the hidden vector is decoded by an engine hole detection defect image generator to generate a defect image; loading an AlexNet model trained on the Imagenet data set, and extracting the characteristics of a conv4 layer from the generated defect image; extracting conv4 layer features from the training sample corresponding to the shape information;
calculating the minimum mean square error of the characteristics of the two conv4 layers as PiA loss function of the network model; shape information as PiAnd the output implicit vector of the network is used as the input of the generator to obtain a generated image with a specified shape and obtain the target function.
5. The interactive hole exploration data expansion method based on generation of the countermeasure network according to claim 4, wherein the objective function is specifically:
wherein C is a conv4 layer characteristic of AlexNet, GiTo a generator, ziThe vector is a hidden vector, and the vector is a hidden vector,
in order to train the sample to be trained,
is PiParameters to be updated in the network.
6. The interactive hole exploration data expansion method based on generation of the countermeasure network as claimed in claim 1, wherein the testing through the trained P network specifically comprises:
input shape I to be specifiedshapeInputting trained PiGenerating hidden vectors z in a networkiFor hidden vector ziAdding a fine noise disturbance N (z),
Dfake＝Gi(Pi(Ishape)+N(z))
wherein D isfakeFor the defect image generated, Pi(Ishape) Represents the utilization of PiNetwork prediction input shape IshapeN (z) represents a fine noise with gaussian distribution.
7. The interactive hole exploration data expansion method based on generation of the countermeasure network according to claim 1, wherein the interactive basic framework is constructed and the adaptive threshold is used for segmenting the defects, and specifically the defects are:
selecting corresponding PiNetwork and defect generator GiThe network acquires the strokes input by the user in real time and extracts the strokes as input shapes; by PiNetwork encodes input strokes into ziReuse of GiNetwork pair ziDecoding to generate a defective image Dfake；
Then D is putfakeFusing the stroke drawn by the user to generate a specified type defect which is similar to the shape of the stroke and has the same position; by making a pair of ziUsing different noise disturbances to generate a plurality of different defect images with similar shapes and slight differences;
and changing the low threshold value tau in Canny in real time, and adjusting the segmentation results of different defect images by setting a high threshold value.
8. The interactive hole exploration data expansion method based on generation of the countermeasure network, according to claim 1, wherein the method further comprises: an interactive interface is constructed based on the Pyqt5 framework.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110311132.6A CN113033656B (en) | 2021-03-24 | 2021-03-24 | Interactive hole detection data expansion method based on generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110311132.6A CN113033656B (en) | 2021-03-24 | 2021-03-24 | Interactive hole detection data expansion method based on generation countermeasure network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113033656A true CN113033656A (en) | 2021-06-25 |
| CN113033656B CN113033656B (en) | 2023-12-26 |
Family
ID=76473089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110311132.6A Active CN113033656B (en) | 2021-03-24 | 2021-03-24 | Interactive hole detection data expansion method based on generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113033656B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679465A (en) * | 2017-09-20 | 2018-02-09 | 上海交通大学 | A kind of pedestrian's weight identification data generation and extending method based on generation network |
| CN108769993A (en) * | 2018-05-15 | 2018-11-06 | 南京邮电大学 | Based on the communication network abnormal user detection method for generating confrontation network |
| CN110533606A (en) * | 2019-07-30 | 2019-12-03 | 中国民航大学 | Safety check X-ray contraband image data Enhancement Method based on production confrontation network |
| EP3576020A1 (en) * | 2018-05-30 | 2019-12-04 | Siemens Healthcare GmbH | Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium |
| CN111145116A (en) * | 2019-12-23 | 2020-05-12 | 哈尔滨工程大学 | Sea surface rainy day image sample augmentation method based on generation of countermeasure network |
| CN111199531A (en) * | 2019-12-27 | 2020-05-26 | 中国民航大学 | Interactive data expansion method based on Poisson image fusion and image stylization |
| CN111415316A (en) * | 2020-03-18 | 2020-07-14 | 山西安数智能科技有限公司 | Defect data synthesis algorithm based on generation of countermeasure network |
-
2021
- 2021-03-24 CN CN202110311132.6A patent/CN113033656B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679465A (en) * | 2017-09-20 | 2018-02-09 | 上海交通大学 | A kind of pedestrian's weight identification data generation and extending method based on generation network |
| CN108769993A (en) * | 2018-05-15 | 2018-11-06 | 南京邮电大学 | Based on the communication network abnormal user detection method for generating confrontation network |
| EP3576020A1 (en) * | 2018-05-30 | 2019-12-04 | Siemens Healthcare GmbH | Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium |
| CN110533606A (en) * | 2019-07-30 | 2019-12-03 | 中国民航大学 | Safety check X-ray contraband image data Enhancement Method based on production confrontation network |
| CN111145116A (en) * | 2019-12-23 | 2020-05-12 | 哈尔滨工程大学 | Sea surface rainy day image sample augmentation method based on generation of countermeasure network |
| CN111199531A (en) * | 2019-12-27 | 2020-05-26 | 中国民航大学 | Interactive data expansion method based on Poisson image fusion and image stylization |
| CN111415316A (en) * | 2020-03-18 | 2020-07-14 | 山西安数智能科技有限公司 | Defect data synthesis algorithm based on generation of countermeasure network |
Non-Patent Citations (1)
| Title |
|---|
| 樊玮等: "基于风格迁移的交互式航空发动机孔探图像扩展方法", 基于风格迁移的交互式航空发动机孔探图像扩展方法, vol. 40, no. 12, pages 3631 - 3636 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113033656B (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jiang et al. | Scfont: Structure-guided chinese font generation via deep stacked networks | |
| CN108376244B (en) | Method for identifying text font in natural scene picture | |
| CN106547880B (en) | Multi-dimensional geographic scene identification method fusing geographic area knowledge | |
| CN111462120B (en) | Defect detection method, device, medium and equipment based on semantic segmentation model | |
| Li et al. | Deep extraction of manga structural lines | |
| CN112784736B (en) | Character interaction behavior recognition method based on multi-modal feature fusion | |
| CN108595558B (en) | Image annotation method based on data equalization strategy and multi-feature fusion | |
| CN112434745A (en) | Occlusion target detection and identification method based on multi-source cognitive fusion | |
| CN111967533B (en) | Sketch image translation method based on scene recognition | |
| CN112529768A (en) | Garment editing and generating method based on generation countermeasure network | |
| CN113033432A (en) | Remote sensing image residential area extraction method based on progressive supervision | |
| Rios et al. | Feature visualization for 3D point cloud autoencoders | |
| CN112541884A (en) | Defect detection method and apparatus, and computer-readable storage medium | |
| CN111612802A (en) | Re-optimization training method based on existing image semantic segmentation model and application | |
| CN114841992A (en) | Defect detection method based on cyclic generation countermeasure network and structural similarity | |
| CN113393546B (en) | Fashion clothing image generation method based on clothing type and texture pattern control | |
| Fu et al. | Fast accurate and automatic brushstroke extraction | |
| Shit et al. | An encoder‐decoder based CNN architecture using end to end dehaze and detection network for proper image visualization and detection | |
| CN113421210A (en) | Surface point cloud reconstruction method based on binocular stereo vision | |
| Jertec et al. | On using pointnet architecture for human body segmentation | |
| CN110929632A (en) | Complex scene-oriented vehicle target detection method and device | |
| Li et al. | A review of advances in image inpainting research | |
| CN113033656A (en) | Interactive hole exploration data expansion method based on generation countermeasure network | |
| Chu et al. | Hole-filling framework by combining structural and textural information for the 3D Terracotta Warriors | |
| Johnston et al. | Single View 3D Point Cloud Reconstruction using Novel View Synthesis and Self-Supervised Depth Estimation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |