CN112508817A - Image motion blind deblurring method based on loop generation countermeasure network - Google Patents
Image motion blind deblurring method based on loop generation countermeasure network Download PDFInfo
- Publication number
- CN112508817A CN112508817A CN202011484067.9A CN202011484067A CN112508817A CN 112508817 A CN112508817 A CN 112508817A CN 202011484067 A CN202011484067 A CN 202011484067A CN 112508817 A CN112508817 A CN 112508817A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- generator
- discriminator
- blur
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000012549 training Methods 0.000 claims abstract description 57
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 14
- 230000004913 activation Effects 0.000 claims description 18
- 238000010606 normalization Methods 0.000 claims description 18
- 238000009826 distribution Methods 0.000 claims description 17
- 239000002131 composite material Substances 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 15
- 238000005457 optimization Methods 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 8
- 239000004576 sand Substances 0.000 claims description 8
- 238000012360 testing method Methods 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 6
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 5
- 238000011156 evaluation Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 230000003042 antagnostic effect Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 8
- 238000011160 research Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012630 chemometric algorithm Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image motion blind deblurring method based on a cyclic generation countermeasure network, which comprises the steps of firstly constructing a data set, and acquiring an image motion blind deblurring method with the quantity ratio of 1: 1 as input data; then, a generator network and a discriminator network are constructed, and then a loss function is defined, wherein the loss function consists of the countermeasure loss and the inter-domain circulation invariant loss; and finally, training the discriminator network and the generator network in sequence to finish the training after the discriminator network and the generator network reach the Nash equilibrium state. The invention utilizes the principle of circularly generating the countermeasure network, effectively utilizes the unpaired data to train in the deblurring task lacking the support of the paired data and produces better effect.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a blind deblurring method for image motion.
Background
With the continuous development of shooting equipment such as mobile phones and cameras, shooting becomes an indispensable part of people's daily life. In the photographing process, due to the shake of the camera or the relative movement of the photographed object, the occurrence of image blur is often occurred. The blurred image not only seriously influences the acquisition of information by people, but also has negative influence on the subsequent computer vision analysis, so that the image deblurring has extremely high practical significance and research value nowadays.
The deblurring task is divided into two broad categories depending on whether the blur kernel is known or not: non-blind image deblurring and blind image deblurring. Most of early researches are based on non-blind image deblurring and expansion, and most of the early researches are based on algorithms such as classical wiener filtering and Gihonov filtering, and deconvolution operation is carried out to obtain clear image estimation. The basic principle of the algorithms is to adopt a mathematical optimization method to estimate the estimation problem of the image from the degraded image under a certain criterion. But in general the blur kernel of a blurred image is unknown, and most algorithms in the first place rely on heuristics, image statistics and assumptions about the source of the blur. With the development of deep learning, students are also beginning to use a deep network to deal with the image deblurring problem, and the deblurring algorithm based on the deep convolutional neural network achieves better performance and higher speed compared with the traditional algorithm. However, the deep learning method needs a large amount of paired data to train and learn, and cannot shoot motion blur and corresponding sharp images at the same time, so it is very difficult to obtain paired motion blur data sets. Some of the disclosed data sets use multiple sharp frames to synthesize a blurred image to obtain paired data, but such synthesized data does not simulate the blur in the real world very well.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides an image motion blind deblurring method based on a loop generation countermeasure network, which comprises the steps of firstly constructing a data set, and acquiring an image motion blind deblurring method with the quantity ratio of 1: 1 as input data; then, a generator network and a discriminator network are constructed, and then a loss function is defined, wherein the loss function consists of the countermeasure loss and the inter-domain circulation invariant loss; and finally, training the discriminator network and the generator network in sequence to finish the training after the discriminator network and the generator network reach the Nash equilibrium state. The invention utilizes the principle of circularly generating the countermeasure network, effectively utilizes the unpaired data to train in the deblurring task lacking the support of the paired data and produces better effect.
The technical scheme adopted by the invention for solving the technical problem comprises the following steps:
step 1: constructing and preprocessing a data set;
taking the fuzzy images in the first 11 scenes of the training set of the GOPRO data set as a fuzzy data set, taking the clear images in the last 11 scenes as a clear data set, and forming a new training set by the clear data set and the fuzzy data set; randomly cutting all images in the new training set into a plurality of images with the size of 256 multiplied by 256, and performing standardization processing to obtain an input data set;
taking the test set of the GOPRO data set as a new test set;
step 2: constructing a generator network;
the generator network comprises two generators, respectively a blur-sharpness generator Gb2sAnd a sharpness-blur generator Gs2b(ii) a The roles of the two generators are respectively: blur-sharpness generator Gb2sOutputting the input blurred image as a corresponding sharp image, a sharp-blur generator Gs2bOutputting the input clear image as a corresponding blurred image; the two generators have the same structure but do not share parameters, and both use InstanceNorm as a normalization layer and LeakyReLU as an activation layer; the concrete structure is as follows:
(1) an input module: the convolution block comprises a convolution layer with a channel of 64 and a convolution kernel size of 7 multiplied by 7, an example normalization layer and a ReLu activation layer;
(2) a feature extraction module: the convolution block comprises two identical convolution blocks, wherein each convolution block comprises a convolution layer with a channel of 128 and a convolution kernel size of 3 x 3, an example normalization layer and a ReLu activation layer;
(3) residual error intensive learning module: contains 5 residual dense blocks RDB, each of which contains four volume blocks: the first three convolution blocks have the same structure and are all composed of convolution layers with the size of 3 multiplied by 3 convolution kernels, an InstanceNorm normalization layer and a Relu activation layer, the first three convolution blocks are densely connected, and then the input of a residual error dense block and the output channel of each convolution block in the first three convolution blocks jointly form 4 characteristic diagrams; reducing the number of channels of the 4 characteristic graphs to the size of the input of the residual dense block through a 1 multiplied by 1 convolution kernel, and finally adding the residual dense block input with the residual dense block input to learn the residual;
(4) an image reconstruction module: the convolution block comprises two identical convolution blocks, wherein each convolution block comprises a convolution layer with a channel of 128 and a convolution kernel size of 3 x 3, an example normalization layer and a ReLu activation layer;
(5) an output module: the convolution block comprises a convolution layer with a channel of 64 and a convolution kernel size of 7 multiplied by 7, an example normalization layer and a ReLu activation layer;
step three: constructing a discriminator network;
the discriminator network comprises two discriminators, respectively a sharp discriminator D for discriminating sharp imagessAnd a blur discriminator D for discriminating a blurred imagebThe two discriminators have the same structure but do not share parameters, and the specific structure is shown in table 1:
table 1: discriminator structure
And 4, step 4: defining a loss function;
the loss function of the network consists of two parts: resistance loss and inter-domain cyclic invariant loss;
the resistance loss:
wherein G denotes a generator, D denotes a discriminator, b and s are respectively a blurred image and a sharp image, and p (b) and p(s) are respectively a data distribution of the blurred image and a data distribution of the sharp image; the generator G aims to make the generated G (b) consistent with the distribution of s, and the discriminator aims to distinguish G (b) from s;
the inter-domain circulation has constant loss:
wherein b and s are respectively a blurred image and a sharp image, and p (b) and p(s) are respectively data distribution of the blurred image and data distribution of the sharp image;
the overall loss function is:
L(Gb2s,Gs2b,Ds,Db,s,b)＝Ladv(Gb2s,Ds,b,s)+Ladv(Gs2b,Db,s,b)+λLCycle(Gb2s,Gs2b,s,b)
the overall loss function includes three parts: the first part is a blur-sharpness generator Gb2sAgainst loss, the second part being the sharpness-blur generator Gs2bThe third part is the inter-domain circulation invariant loss; wherein λ acts to control the degree of importance of the antagonistic losses and the inter-domain cyclic invariant losses;
and 5: inputting the input data set in the step 1 into a network, and performing optimization training on the network by adopting an Adam optimization algorithm;
step 5-1: training a discriminator network;
training the real data: a blurred image and a sharp image are respectively taken from the input data set and the two images are respectively input to a sharpness discriminator DsAnd a fuzzy discriminator DbObtaining the judgment of the two images, solving the loss value according to the judgment result, and performing back propagation to adjust the network parameters to optimize the network;
training the synthetic data: a blurred image and a sharp image are respectively taken out from an input data set, and the two images are respectively input into a blur-sharp generator Gb2sAnd a sharpness-blur generator Gs2bIn the method, a corresponding synthesized sharp image and a synthesized blurred image are obtained, and then the two synthesized images are divided into twoCloth input to the sharpness discriminator DsAnd a fuzzy discriminator DbObtaining the judgment of the two images, solving the loss value according to the judgment result, and performing back propagation to adjust the network parameters to optimize the network;
the result obtained by the discriminator network is a two-dimensional matrix, all values in the two-dimensional matrix are averaged, and the obtained average value is used as the evaluation of the discriminator network on the whole image;
step 5-2: training a generator network;
fuzzy-clear-fuzzy training cycle: a blurred image is selected from the input data set and input to a blur-sharpness generator Gb2sTo obtain a corresponding synthesized sharp image, and inputting the corresponding synthesized sharp image to a sharp discriminator DsObtaining a clear discriminator DsEvaluating the resultant corresponding sharp image to obtain a loss value against loss, the blur-sharpness process being aimed at making the blur-sharpness generator Gb2sThe generated image can be clearly identified by a discriminator DsIdentifying the image as a real clear image; simultaneously inputting the synthesized sharp image to a sharpness-blur generator Gs2bObtaining a composite blurred image, i.e. G, generated from a composite sharp imageb2s(Gs2b(s)), and obtaining a loss value of the inter-domain cyclic invariant loss; the goal of the blur-sharpness-blur training cycle is to pass the input blurred image through a blur-sharpness generator Gb2sThen passes through a clear-fuzzy generator Gs2bThe obtained fuzzy image is consistent with the originally input fuzzy image; finally, the obtained countermeasure loss and the inter-domain circulation invariant loss are subjected to back propagation to adjust the parameter optimization network of the network;
clear-fuzzy-clear training cycle: selecting a sharp image from the input data set and inputting the sharp image to a sharpness-blur generator Gs2bTo obtain a corresponding blurred image, and inputting the blurred image to a blur discriminator DbObtaining a fuzzy discriminator DbEvaluating the composite image to obtain a loss value for resisting loss; the goal of the sharpness-blur process is to makeSharpness-blur generator Gs2bThe generated image can be blurred discriminated by a discriminator DbIdentifying as a true blurred image; simultaneously inputting the combined blurred image to a blur-sharpness generator Gb2sObtaining a composite sharp image, i.e. G, generated from the composite blurred images2b(Gb2s(s)), and obtaining a loss value of the inter-domain cyclic invariant loss; the goal of the sharpness-blur-sharpness training cycle is to pass the input sharp image through a sharpness-blur generator Gs2bThen passes through a fuzzy-clear generator Gb2sThe obtained clear image is consistent with the originally input clear image; finally, the obtained countermeasure loss and the inter-domain circulation invariant loss are subjected to back propagation to adjust the parameter optimization network of the network;
step 5-3: and training the discriminator network and the generator network in sequence, and finishing the training when the discriminator network and the generator network reach a Nash equilibrium state.
Preferably, λ is 10.0.
Preferably, the hyper-parameters for performing optimization training on the network by using the Adam optimization algorithm in the step 5 are set as: learning rate of 1 × 10-4The epochs training times for all training samples are 300, the learning rates of the first 150 epochs are unchanged, the learning rates of the last 150 epochs linearly decay to zero, and the amount of data batchsize in each batch of training is set to 1.
The invention has the following beneficial effects:
1. the invention adopts an end-to-end deep neural network to realize the motion blind deblurring task without a fuzzy core.
2. The invention utilizes the principle of circularly generating the countermeasure network, effectively utilizes the unpaired data to train in the deblurring task lacking the support of the paired data and produces better effect.
3. The generator of the invention adopts the residual error intensive module, so that the network has stronger adaptability and the learning efficiency of the network is improved, thereby obtaining better deblurring effect.
4. The inter-domain cyclic invariant loss defined in the loss function effectively limits the mapping space of the two generators, thereby ensuring the effectiveness of the generators.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention.
Fig. 2 is a diagram of a generator network architecture of the present invention.
Fig. 3 is a detailed block diagram of the residual dense block in the generator network of the present invention.
FIG. 4 is a diagram of the motion blur removal effect of the present invention, with the left column being the input blurred image, the middle column being the output after deblurring by the method of the present invention, and the right column being the true sharp image.
Detailed Description
The invention is further illustrated with reference to the following figures and examples.
The invention provides an image motion blind deblurring method based on a loop generation countermeasure network to realize an end-to-end motion deblurring task without paired data.
As shown in fig. 1, the present invention provides an image motion blind deblurring method based on a loop-generated countermeasure network, which includes the following steps:
step 1: constructing and preprocessing a data set;
taking the fuzzy images in the first 11 scenes of the training set of the GOPRO data set as a fuzzy data set, taking the clear images in the last 11 scenes as a clear data set, and forming a new training set by the clear data set and the fuzzy data set; randomly cutting all images in the new training set into a plurality of images with the size of 256 multiplied by 256, and performing standardization processing to obtain an input data set;
taking the test set of the GOPRO data set as a new test set;
step 2: building a generator network, as shown in FIG. 2;
the generator network comprises two generators, respectively a blur-sharpness generator Gb2sAnd a sharpness-blur generator Gs2b(ii) a The roles of the two generators are respectively: blur-sharpness generator Gb2sOutputting the input blurred image as a corresponding sharp image, a sharp-blur generator Gs2bOutputting the input clear image as a corresponding blurred image; two are providedThe generators have consistent structures, but parameters are not shared, and both use InstanceNorm as a normalization layer and LeakyReLU as an activation layer; the concrete structure is as follows:
(1) an input module: the convolution block comprises a convolution layer with a channel of 64 and a convolution kernel size of 7 multiplied by 7, an example normalization layer and a ReLu activation layer;
(2) a feature extraction module: the convolution block comprises two identical convolution blocks, wherein each convolution block comprises a convolution layer with a channel of 128 and a convolution kernel size of 3 x 3, an example normalization layer and a ReLu activation layer;
(3) residual error intensive learning module: as shown in fig. 3, 5 residual dense blocks RDB are included, and each residual dense block RDB includes four volume blocks: the first three convolution blocks have the same structure and are all composed of convolution layers with the size of 3 multiplied by 3 convolution kernels, an InstanceNorm normalization layer and a Relu activation layer, the first three convolution blocks are densely connected, and then the input of a residual error dense block and the output channel of each convolution block in the first three convolution blocks jointly form 4 characteristic diagrams; reducing the number of channels of the 4 characteristic graphs to the size of the input of the residual dense block through a 1 multiplied by 1 convolution kernel, and finally adding the residual dense block input with the residual dense block input to learn the residual;
(4) an image reconstruction module: the convolution block comprises two identical convolution blocks, wherein each convolution block comprises a convolution layer with a channel of 128 and a convolution kernel size of 3 x 3, an example normalization layer and a ReLu activation layer;
(5) an output module: the convolution block comprises a convolution layer with a channel of 64 and a convolution kernel size of 7 multiplied by 7, an example normalization layer and a ReLu activation layer;
step three: constructing a discriminator network;
the discriminator network comprises two discriminators, respectively a sharp discriminator D for discriminating sharp imagessAnd a blur discriminator D for discriminating a blurred imagebThe two discriminators have the same structure but do not share parameters, and the specific structure is shown in table 1:
table 1: discriminator structure
And 4, step 4: defining a loss function;
the loss function of the network consists of two parts: resistance loss and inter-domain cyclic invariant loss;
the resistance loss:
wherein G denotes a generator, D denotes a discriminator, b and s are respectively a blurred image and a sharp image, and p (b) and p(s) are respectively a data distribution of the blurred image and a data distribution of the sharp image; the generator G aims to make the generated G (b) consistent with the distribution of s, and the discriminator aims to distinguish G (b) from s;
the inter-domain circulation has constant loss:
wherein b and s are respectively a blurred image and a sharp image, and p (b) and p(s) are respectively data distribution of the blurred image and data distribution of the sharp image;
the overall loss function is:
L(Gb2s,Gs2b,Ds,Db,s,b)＝Ladv(Gb2s,Ds,b,s)+Ladv(Gs2b,Db,s,b)+λLCycle(Gb2s,Gs2b,s,b)
the overall loss function includes three parts: the first part is a blur-sharpness generator Gb2sAgainst loss, the second part being the sharpness-blur generator Gs2bThe third part is the inter-domain circulation invariant loss; wherein λ is used to control the degree of importance of the antagonistic loss and the inter-domain cyclic invariant loss, λ is 10.0;
and 5: inputting the input data set in the step 1 into a network, and adopting Adam optimalOptimizing and training the network by using a chemometric algorithm, wherein the hyper-parameters are set as: learning rate of 1 × 10-4The epochs training times for all training samples are 300, the learning rates of the first 150 epochs are unchanged, the learning rates of the last 150 epochs linearly decay to zero, and the amount of data batchsize in each batch of training is set to 1. (ii) a
Step 5-1: training a discriminator network;
training the real data: a blurred image and a sharp image are respectively taken from the input data set and the two images are respectively input to a sharpness discriminator DsAnd a fuzzy discriminator DbObtaining the judgment of the two images, solving the loss value according to the judgment result, and performing back propagation to adjust the network parameters to optimize the network;
training the synthetic data: a blurred image and a sharp image are respectively taken out from an input data set, and the two images are respectively input into a blur-sharp generator Gb2sAnd a sharpness-blur generator Gs2bThen the two synthesized image distributions are input to the sharpness discriminator DsAnd a fuzzy discriminator DbObtaining the judgment of the two images, solving the loss value according to the judgment result, and performing back propagation to adjust the network parameters to optimize the network;
the result obtained by the discriminator network is a two-dimensional matrix, all values in the two-dimensional matrix are averaged, and the obtained average value is used as the evaluation of the discriminator network on the whole image;
step 5-2: training a generator network;
fuzzy-clear-fuzzy training cycle: a blurred image is selected from the input data set and input to a blur-sharpness generator Gb2sTo obtain a corresponding synthesized sharp image, and inputting the corresponding synthesized sharp image to a sharp discriminator DsObtaining a clear discriminator DsEvaluating the resultant corresponding sharp image to obtain a loss value against loss, the blur-sharpness process being aimed at making the blur-sharpness generator Gb2sGenerated imageCan be clearly identified by discriminator DsIdentifying the image as a real clear image; simultaneously inputting the synthesized sharp image to a sharpness-blur generator Gs2bObtaining a composite blurred image, i.e. G, generated from a composite sharp imageb2s(Gs2b(s)), and obtaining a loss value of the inter-domain cyclic invariant loss; the goal of the blur-sharpness-blur training cycle is to pass the input blurred image through a blur-sharpness generator Gb2sThen passes through a clear-fuzzy generator Gs2bThe obtained fuzzy image is consistent with the originally input fuzzy image; finally, the obtained countermeasure loss and the inter-domain circulation invariant loss are subjected to back propagation to adjust the parameter optimization network of the network;
clear-fuzzy-clear training cycle: selecting a sharp image from the input data set and inputting the sharp image to a sharpness-blur generator Gs2bTo obtain a corresponding blurred image, and inputting the blurred image to a blur discriminator DbObtaining a fuzzy discriminator DbEvaluating the composite image to obtain a loss value for resisting loss; the goal of the sharpness-blur process is to make the sharpness-blur generator Gs2bThe generated image can be blurred discriminated by a discriminator DbIdentifying as a true blurred image; simultaneously inputting the combined blurred image to a blur-sharpness generator Gb2sObtaining a composite sharp image, i.e. G, generated from the composite blurred images2b(Gb2s(s)), and obtaining a loss value of the inter-domain cyclic invariant loss; the goal of the sharpness-blur-sharpness training cycle is to pass the input sharp image through a sharpness-blur generator Gs2bThen passes through a fuzzy-clear generator Gb2sThe obtained clear image is consistent with the originally input clear image; finally, the obtained countermeasure loss and the inter-domain circulation invariant loss are subjected to back propagation to adjust the parameter optimization network of the network;
step 5-3: and training the discriminator network and the generator network in sequence, and finishing the training when the discriminator network and the generator network reach a Nash equilibrium state.
The specific embodiment is as follows:
the method of the present invention is tested by using the test set defined in the step 1, the deblurred image is input into the deblurring network of the present invention to obtain a deblurred clear image, and the test result of the clear image is shown in table 2:
TABLE 2 evaluation of the results
As shown in table 2, the experimental results of this example are measured by PSNR and SSIM indexes, and compared with the current three well-known advanced algorithms Kim et al, Sun et al, and DeblurGAN, all of which are obtained from the GOPRO dataset. By comparison, the method of the invention achieves the optimal effect in the PSNR index, has better result in the SSIM index, and fully shows the effectiveness of the method of the invention.
As shown in fig. 4, which is a visualization result diagram of the present embodiment, the leftmost column in the diagram is an input original image, that is, a blurred image, the middle column is an image output through a deblurring network, and the rightmost column is a corresponding sharp image. Three areas are marked in each figure with three rectangular boxes, corresponding to the enlargement of these three areas below the image, to better observe the deblurring effect in detail. From the visual angle, the method has better deblurring effect obviously, thereby proving the effectiveness of the method.
Claims (3)
1. An image motion blind deblurring method based on a loop generation countermeasure network is characterized by comprising the following steps:
step 1: constructing and preprocessing a data set;
taking the fuzzy images in the first 11 scenes of the training set of the GOPRO data set as a fuzzy data set, taking the clear images in the last 11 scenes as a clear data set, and forming a new training set by the clear data set and the fuzzy data set; randomly cutting all images in the new training set into a plurality of images with the size of 256 multiplied by 256, and performing standardization processing to obtain an input data set;
taking the test set of the GOPRO data set as a new test set;
step 2: constructing a generator network;
the generator network comprises two generators, respectively a blur-sharpness generator Gb2sAnd a sharpness-blur generator Gs2b(ii) a The roles of the two generators are respectively: blur-sharpness generator Gb2sOutputting the input blurred image as a corresponding sharp image, a sharp-blur generator Gs2bOutputting the input clear image as a corresponding blurred image; the two generators have the same structure but do not share parameters, and both use InstanceNorm as a normalization layer and LeakyReLU as an activation layer; the concrete structure is as follows:
(1) an input module: the convolution block comprises a convolution layer with a channel of 64 and a convolution kernel size of 7 multiplied by 7, an example normalization layer and a ReLu activation layer;
(2) a feature extraction module: the convolution block comprises two identical convolution blocks, wherein each convolution block comprises a convolution layer with a channel of 128 and a convolution kernel size of 3 x 3, an example normalization layer and a ReLu activation layer;
(3) residual error intensive learning module: contains 5 residual dense blocks RDB, each of which contains four volume blocks: the first three convolution blocks have the same structure and are all composed of convolution layers with the size of 3 multiplied by 3 convolution kernels, an InstanceNorm normalization layer and a Relu activation layer, the first three convolution blocks are densely connected, and then the input of a residual error dense block and the output channel of each convolution block in the first three convolution blocks jointly form 4 characteristic diagrams; reducing the number of channels of the 4 characteristic graphs to the size of the input of the residual dense block through a 1 multiplied by 1 convolution kernel, and finally adding the residual dense block input with the residual dense block input to learn the residual;
(4) an image reconstruction module: the convolution block comprises two identical convolution blocks, wherein each convolution block comprises a convolution layer with a channel of 128 and a convolution kernel size of 3 x 3, an example normalization layer and a ReLu activation layer;
(5) an output module: the convolution block comprises a convolution layer with a channel of 64 and a convolution kernel size of 7 multiplied by 7, an example normalization layer and a ReLu activation layer;
step three: constructing a discriminator network;
the discriminator network comprises two discriminators, respectively a sharp discriminator Ds for discriminating sharp images and a blurred discriminator D for discriminating blurred imagesbThe two discriminators have the same structure but do not share the parameters, and the specific structure is as shown in table 1:
table 1: discriminator structure
And 4, step 4: defining a loss function;
the loss function of the network consists of two parts: resistance loss and inter-domain cyclic invariant loss;
the resistance loss:
wherein G denotes a generator, D denotes a discriminator, b and s are respectively a blurred image and a sharp image, and p (b) and p(s) are respectively a data distribution of the blurred image and a data distribution of the sharp image; the generator G aims to make the generated G (b) consistent with the distribution of s, and the discriminator aims to distinguish G (b) from s;
the inter-domain circulation has constant loss:
wherein b and s are respectively a blurred image and a sharp image, and p (b) and p(s) are respectively data distribution of the blurred image and data distribution of the sharp image;
the overall loss function is:
L(Gb2s，Gs2b，Ds，Db，s，b)＝Ladv(Gb2s，Ds，b，s)+Ladv(Gs2b，Db，s，b)+λLCycle(Gb2s，Gs2b，s，b)
the overall loss function includes three parts: the first part is a blur-sharpness generator Gb2sAgainst loss, the second part being the sharpness-blur generator Gs2bThe third part is the inter-domain circulation invariant loss; wherein λ acts to control the degree of importance of the antagonistic losses and the inter-domain cyclic invariant losses;
and 5: inputting the input data set in the step 1 into a network, and performing optimization training on the network by adopting an Adam optimization algorithm;
step 5-1: training a discriminator network;
training the real data: a blurred image and a sharp image are respectively taken from the input data set and the two images are respectively input to a sharpness discriminator DsAnd a fuzzy discriminator DbObtaining the judgment of the two images, solving the loss value according to the judgment result, and performing back propagation to adjust the network parameters to optimize the network;
training the synthetic data: a blurred image and a sharp image are respectively taken out from an input data set, and the two images are respectively input into a blur-sharp generator Gb2sAnd a sharpness-blur generator Gs2bThen the two synthesized image distributions are input to the sharpness discriminator DsAnd a fuzzy discriminator DbObtaining the judgment of the two images, solving the loss value according to the judgment result, and performing back propagation to adjust the network parameters to optimize the network;
the result obtained by the discriminator network is a two-dimensional matrix, all values in the two-dimensional matrix are averaged, and the obtained average value is used as the evaluation of the discriminator network on the whole image;
step 5-2: training a generator network;
fuzzy-clear-fuzzy training cycle: a blurred image is selected from the input data set and input to a blur-sharpness generator Gb2sTo obtain a corresponding synthesized sharp image, and then combining the corresponding sharp imageImage input to sharpness discriminator DsObtaining a clear discriminator DsEvaluating the resultant corresponding sharp image to obtain a loss value against loss, the blur-sharpness process being aimed at making the blur-sharpness generator Gb2sThe generated image can be clearly identified by a discriminator DsIdentifying the image as a real clear image; simultaneously inputting the synthesized sharp image to a sharpness-blur generator Gs2bObtaining a composite blurred image, i.e. G, generated from a composite sharp imageb2s(Gs2b(s)), and obtaining a loss value of the inter-domain cyclic invariant loss; the goal of the blur-sharpness-blur training cycle is to pass the input blurred image through a blur-sharpness generator Gb2sThen passes through a clear-fuzzy generator Gs2bThe obtained fuzzy image is consistent with the originally input fuzzy image; finally, the obtained countermeasure loss and the inter-domain circulation invariant loss are subjected to back propagation to adjust the parameter optimization network of the network;
clear-fuzzy-clear training cycle: selecting a sharp image from the input data set and inputting the sharp image to a sharpness-blur generator Gs2bTo obtain a corresponding blurred image, and inputting the blurred image to a blur discriminator DbObtaining a fuzzy discriminator DbEvaluating the composite image to obtain a loss value for resisting loss; the goal of the sharpness-blur process is to make the sharpness-blur generator Gs2bThe generated image can be blurred discriminated by a discriminator DbIdentifying as a true blurred image; simultaneously inputting the combined blurred image to a blur-sharpness generator Gb2sObtaining a composite sharp image, i.e. G, generated from the composite blurred images2b(Gb2s(s)), and obtaining a loss value of the inter-domain cyclic invariant loss; the goal of the sharpness-blur-sharpness training cycle is to pass the input sharp image through a sharpness-blur generator Gs2bThen passes through a fuzzy-clear generator Gb2sThe obtained clear image is consistent with the originally input clear image; finally, the obtained countermeasure loss and the inter-domain circulation invariant loss are subjected to back propagation to adjust the parameter optimization network of the network;
step 5-3: and training the discriminator network and the generator network in sequence, and finishing the training when the discriminator network and the generator network reach a Nash equilibrium state.
2. The blind deblurring method for image motion based on loop-generated countermeasure network of claim 1, wherein λ is 10.0.
3. The blind image motion deblurring method based on the loop-generated countermeasure network of claim 1, wherein the hyper-parameters for the optimization training of the network by using the Adam optimization algorithm in the step 5 are set as: learning rate of 1 × 10-4The epochs training times for all training samples are 300, the learning rates of the first 150 epochs are unchanged, the learning rates of the last 150 epochs linearly decay to zero, and the amount of data batchsize in each batch of training is set to 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011484067.9A CN112508817B (en) | 2020-12-16 | 2020-12-16 | Image motion blind deblurring method based on cyclic generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011484067.9A CN112508817B (en) | 2020-12-16 | 2020-12-16 | Image motion blind deblurring method based on cyclic generation countermeasure network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112508817A true CN112508817A (en) | 2021-03-16 |
| CN112508817B CN112508817B (en) | 2024-05-14 |
Family
ID=74972447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011484067.9A Active CN112508817B (en) | 2020-12-16 | 2020-12-16 | Image motion blind deblurring method based on cyclic generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112508817B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538263A (en) * | 2021-06-28 | 2021-10-22 | 江苏威尔曼科技有限公司 | Motion blur removing method, medium, and device based on improved DeblurgAN model |
| CN113689348A (en) * | 2021-08-18 | 2021-11-23 | 中国科学院自动化研究所 | Multitask image restoration method, multitask image restoration system, electronic device and storage medium |
| CN113763282A (en) * | 2021-09-22 | 2021-12-07 | 北京中电兴发科技有限公司 | Fuzzy image generation method for license plate image |
| CN114820389A (en) * | 2022-06-23 | 2022-07-29 | 北京科技大学 | Face image deblurring method based on unsupervised decoupling representation |
| CN114913095A (en) * | 2022-06-08 | 2022-08-16 | 西北工业大学 | Depth deblurring method based on domain adaptation |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108376387A (en) * | 2018-01-04 | 2018-08-07 | 复旦大学 | Image deblurring method based on polymerization expansion convolutional network |
| CN110378844A (en) * | 2019-06-14 | 2019-10-25 | 杭州电子科技大学 | Motion blur method is gone based on the multiple dimensioned Image Blind for generating confrontation network is recycled |
| CN111199522A (en) * | 2019-12-24 | 2020-05-26 | 重庆邮电大学 | Single-image blind motion blur removing method for generating countermeasure network based on multi-scale residual errors |
| CN111223062A (en) * | 2020-01-08 | 2020-06-02 | 西安电子科技大学 | Image deblurring method based on generation countermeasure network |
| CN111612703A (en) * | 2020-04-22 | 2020-09-01 | 杭州电子科技大学 | Image blind deblurring method based on generation countermeasure network |
-
2020
- 2020-12-16 CN CN202011484067.9A patent/CN112508817B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108376387A (en) * | 2018-01-04 | 2018-08-07 | 复旦大学 | Image deblurring method based on polymerization expansion convolutional network |
| CN110378844A (en) * | 2019-06-14 | 2019-10-25 | 杭州电子科技大学 | Motion blur method is gone based on the multiple dimensioned Image Blind for generating confrontation network is recycled |
| CN111199522A (en) * | 2019-12-24 | 2020-05-26 | 重庆邮电大学 | Single-image blind motion blur removing method for generating countermeasure network based on multi-scale residual errors |
| CN111223062A (en) * | 2020-01-08 | 2020-06-02 | 西安电子科技大学 | Image deblurring method based on generation countermeasure network |
| CN111612703A (en) * | 2020-04-22 | 2020-09-01 | 杭州电子科技大学 | Image blind deblurring method based on generation countermeasure network |
Non-Patent Citations (2)
| Title |
|---|
| 罗琪彬;蔡强;: "采用双框架生成对抗网络的图像运动模糊盲去除", 图学学报, no. 06, 15 December 2019 (2019-12-15) * |
| 谷静;王琦雯;张敏;王金金;: "基于DenseNet网络的焊缝缺陷检测识别", 传感器与微系统, no. 09, 26 August 2020 (2020-08-26) * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538263A (en) * | 2021-06-28 | 2021-10-22 | 江苏威尔曼科技有限公司 | Motion blur removing method, medium, and device based on improved DeblurgAN model |
| CN113689348A (en) * | 2021-08-18 | 2021-11-23 | 中国科学院自动化研究所 | Multitask image restoration method, multitask image restoration system, electronic device and storage medium |
| CN113689348B (en) * | 2021-08-18 | 2023-12-26 | 中国科学院自动化研究所 | Method, system, electronic device and storage medium for restoring multi-task image |
| CN113763282A (en) * | 2021-09-22 | 2021-12-07 | 北京中电兴发科技有限公司 | Fuzzy image generation method for license plate image |
| CN113763282B (en) * | 2021-09-22 | 2023-07-14 | 北京中电兴发科技有限公司 | Fuzzy image generation method of license plate image |
| CN114913095A (en) * | 2022-06-08 | 2022-08-16 | 西北工业大学 | Depth deblurring method based on domain adaptation |
| CN114913095B (en) * | 2022-06-08 | 2024-03-12 | 西北工业大学 | Depth deblurring method based on domain adaptation |
| CN114820389A (en) * | 2022-06-23 | 2022-07-29 | 北京科技大学 | Face image deblurring method based on unsupervised decoupling representation |
| CN114820389B (en) * | 2022-06-23 | 2022-09-23 | 北京科技大学 | Face image deblurring method based on unsupervised decoupling representation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112508817B (en) | 2024-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112508817B (en) | Image motion blind deblurring method based on cyclic generation countermeasure network | |
| CN110378844B (en) | Image blind motion blur removing method based on cyclic multi-scale generation countermeasure network | |
| CN111861894B (en) | Image motion blur removing method based on generation type countermeasure network | |
| CN112884671B (en) | Fuzzy image restoration method based on unsupervised generation countermeasure network | |
| CN110807757B (en) | Image quality evaluation method and device based on artificial intelligence and computer equipment | |
| CN112288627B (en) | Recognition-oriented low-resolution face image super-resolution method | |
| CN111738953A (en) | Atmospheric turbulence degraded image restoration method based on boundary perception counterstudy | |
| CN111462206A (en) | Monocular structure light depth imaging method based on convolutional neural network | |
| CN109087255A (en) | A kind of lightweight depth image denoising method based on losses by mixture | |
| CN112435187A (en) | Single-image blind motion blur removing method for generating countermeasure network based on aggregation residual | |
| CN111833267A (en) | Dual generation countermeasure network for motion blur restoration and operation method thereof | |
| CN114463218B (en) | Video deblurring method based on event data driving | |
| CN113822830A (en) | Multi-exposure image fusion method based on depth perception enhancement | |
| Zhao et al. | A simple and robust deep convolutional approach to blind image denoising | |
| Lai et al. | Mixed attention network for hyperspectral image denoising | |
| CN114283058A (en) | Image super-resolution reconstruction method based on countermeasure network and maximum mutual information optimization | |
| Yue et al. | High iso jpeg image denoising by deep fusion of collaborative and convolutional filtering | |
| CN116777745A (en) | Image super-resolution reconstruction method based on sparse self-adaptive clustering | |
| CN114820389B (en) | Face image deblurring method based on unsupervised decoupling representation | |
| CN113129237B (en) | Depth image deblurring method based on multi-scale fusion coding network | |
| Xu et al. | Blind image quality assessment by pairwise ranking image series | |
| CN113645419A (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| Qian et al. | Dense connected residual generative adversarial network for single image deblurring | |
| CN111723864A (en) | Method and device for performing countermeasure training by using internet pictures based on active learning | |
| CN111524060A (en) | System, method, storage medium and device for blurring portrait background in real time |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |