CN110827213B - Super-resolution image restoration method based on generation type countermeasure network - Google Patents
Super-resolution image restoration method based on generation type countermeasure network Download PDFInfo
- Publication number
- CN110827213B CN110827213B CN201910963790.6A CN201910963790A CN110827213B CN 110827213 B CN110827213 B CN 110827213B CN 201910963790 A CN201910963790 A CN 201910963790A CN 110827213 B CN110827213 B CN 110827213B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution image
- network
- repaired
- image data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000012549 training Methods 0.000 claims abstract description 47
- 230000000007 visual effect Effects 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 9
- 230000008439 repair process Effects 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 39
- 238000009826 distribution Methods 0.000 claims description 31
- 238000013507 mapping Methods 0.000 claims description 17
- 230000008602 contraction Effects 0.000 claims description 13
- 238000005070 sampling Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 230000008447 perception Effects 0.000 claims description 5
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 210000002569 neuron Anatomy 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 abstract description 2
- 230000037430 deletion Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 4
- 230000004438 eyesight Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000009792 diffusion process Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000002715 modification method Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000004153 renaturation Methods 0.000 description 1
- 239000004753 textile Substances 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a super-resolution image restoration method based on a generative countermeasure network, which specifically comprises the following steps: step 1, collecting and sorting real image data, processing the collected images into images with the same size, and forming a training data set by the processed images with the same size; step 2, constructing a generating type confrontation network model which comprises a generator and a discriminator; step 3, printing masks with the same size on each image in the training data set to obtain an image to be repaired, importing the image to be repaired into the generative confrontation network model, and training the generative confrontation network model by adopting a weak supervision learning mode to obtain a trained image; and 4, inputting the training image into the U-Net generation network to repair the trained image to be repaired, and outputting a repaired high-resolution image. The problems of low resolution and inconsistent visual effect after large-area image deletion repair in the prior art are solved.
Description
Technical Field
The invention belongs to the technical field of computer vision and image processing, and particularly relates to a super-resolution image restoration method based on a generative confrontation network.
Background
In recent years, computer graphics and computer vision have been rapidly developed, and image restoration methods are diversified, and image restoration techniques have appeared early in the period of renaturation of art, which is a long-history technique, and artworks damaged by improper preservation or other different reasons often appear early, and craftsmen have done some filling work to maintain the integrity of artworks through the skill they master. With the rapid development of computer vision technology, digital image processing technology gradually replaces manual image restoration technology.
At present, image restoration is an important content in the field of computer vision, and aims to automatically restore lost content according to known content in an image, and the method has wide application value in the fields of image editing, movie and television special effect production, virtual reality, digital cultural heritage protection and the like. The traditional image restoration method is based on a diffusion method, for example, smooth prior is introduced through a parameter model or a partial differential equation, the local structure to be restored is spread from outside to inside or diffused, the texture is simple, the image restoration effect with small damaged area is good, but the complex image restoration is easy to generate the phenomena of blurring and artifacts, and the resolution is low after the image restoration; the image restoration method based on the variational partial differential equation algorithm is mainly started from pixel points, effective information is found out according to the damaged edge structure of an image, so that the expansion content and the direction are determined, the diffusion of the edge information of a to-be-restored area to the inside is ensured, and the purpose of image restoration is achieved; when selecting a pixel block in a to-be-repaired area, the texture structure-based image repairing method mainly takes a central point of the pixel block as an object, searches and compares the pixel block in a good area, and fills the compared optimal pixel block into the to-be-repaired area, so that the detailed repairing effect on image textures is good.
Disclosure of Invention
The invention aims to provide a super-resolution image restoration method based on a generative confrontation network, which solves the problems of low resolution and inconsistent visual effect after large-area image deletion restoration in the prior art.
The technical scheme adopted by the invention is that the super-resolution image restoration method based on the generative countermeasure network is implemented according to the following steps:
step 1, collecting and sorting real image data to form a real image data sample set, processing the collected images into images with the same size, and forming a training data set by the processed images with the same size;
step 2, constructing a generative confrontation network model, wherein the generative confrontation network model comprises a generator and a discriminator;
step 3, printing masks with the same size on each low-resolution image in the training data set formed in the step 1 to obtain an image to be repaired, importing the image to be repaired into the generative confrontation network model constructed in the step 2, training the generative confrontation network model by adopting a weak supervision learning mode to obtain a trained image, updating the weights of the generator and the VDB discriminator in the training process, and calculating system loss by utilizing confrontation loss, content loss, perception loss and reconstruction loss;
and 4, inputting the training image obtained in the step 3 into a U-Net generation network to repair the trained image to be repaired, and outputting a repaired high-resolution image.
The invention is also characterized in that:
in step 2, the objective function expression of the generative confrontation network model is:
wherein D represents a discriminator, G represents a generator, and E represents a mathematical expectation; x to p (x) represent the image distribution p (x) in the x obeying real image data sample set; z to G (z) represent a generation distribution G (z) of a z-obeyed image, x is real image data, z is random noise, D (x) represents a discriminant function, and D (z | x) represents a discriminant function of the random noise z under the condition of the real image data x.
In step 2, the pair of the variational arbiter bottleneck VDB to the arbiter objective function is:
J(D,E)＝min D,E E x～p(x) [E z～E(z|x) [-log(D(x))]]+E z～G(z) [E z～E(z|x) [-log(1-D(z))]] (2)
wherein D represents a discriminator, E represents an expectation of encoded image information, and x to p * (x) Representing x obedience generating image data distribution p * (x) (ii) a z-E (z | x) represents z obeying Gaussian distribution E (z | x), KL is a method of divergence, also called relative entropy information probability distribution, r (z) is the characteristic distribution of random noise z, and I c To generate mutual information of the image data by the generator, D (z) represents a discriminant function of the random noise z, and equation (3) is a constraint condition of equation (2).
By
Introducing a discriminator Lagrange optimization objective function:
wherein, beta is Lagrange multiplier, formula (5) is constraint condition of formula (4), and mutual information I c The correlation between variables is expressed for limiting the correlation of the real image data x and the random noise z.
In step 2, the VDB discriminator is a binary classifier, and judges whether the image information is true or false by the correlation of the image information generated by the generator
Limiting mutual information of image data samples for limiting conditions, obfuscating true and false image information, and updating parameters of a VDB discriminator as follows:
wherein, I r And (d) representing mutual characteristic information of the real image data, wherein r (x) is the characteristic distribution of the real image data.
In step 2, the generator adopts a U-Net generation network, the U-Net generation network is composed of a contraction path and an expansion path, the contraction path and the expansion path are symmetrical to each other, the U-Net generation network totally comprises 23 convolution layers, a maximum pooling layer of 2*2 is formed after every two convolution layers of 3*3 on the contraction path, a relu activation function is used for performing down-sampling operation on images in the training data set after each convolution layer, the number of channels is increased for each down-sampling operation to obtain context information of a real image, two convolution layers of 3*3 are formed after each convolution layer of 2*2 on the expansion path, a relu activation function is used for performing up-sampling operation on the images in the training data set after each convolution layer, a characteristic diagram from the corresponding contraction path is added in each up-sampling operation, and the shapes are kept the same after being cut.
In step 2, the discriminator adopts a VDB discrimination network which maps the real image data transmitted from the generator to the encoder onto gaussian distribution, classifies the images generated by the generator, and acts information bottleneck on random noise z.
In step 3, updating parameters of the generative confrontation network model by using a random gradient descent method, specifically:
for a set of real image data samples { x i ,y j Its loss function is:
wherein, y j Is x for e {0,1} i The label of (1);
let the training data set be
Image sample->
The input is obtained corresponding weak supervision output
The structured risk function of the image samples on the training dataset M is therefore:
wherein W and b represent all weights and bias vectors in the network, respectively;
the regularization term is used for preventing overfitting during training of the image restoration model; λ is a hyper-parameter which is a positive number and is used for controlling the weight, the larger λ is, the closer W is to '0', N represents the nth image sample, N represents the total number of samples of the training data set, l represents the number of layers, and m represents the number of neurons;
then the weight W of the l-th layer in the iterative operation of the gradient descent algorithm (l) And an offset vector b (l) The updating method comprises the following steps:
where α is the learning rate, where the optimum learning rate α is 0.001.
In step 3, the generator countermeasure loss of the generator countermeasure network model is:
L G ＝-E x～p(x) [log(1-D(z|x))]-E x～G(x) [logD(z|y)] (13)
wherein D (z | y) represents a discriminant function distribution of random noise under a condition of real image data output;
the 1-norm distance is used to measure the difference between the undamaged area of the generator generated image and the undamaged area in the real image data:
the content loss is:
wherein, I represents the image loss area and the mutual information of the pixel points;
the perceptual loss is:
wherein, C j H j W j For normalization processing on the image path, [ phi ] j Pixel points in the real image data;
the reconstruction loss is:
L pixel ＝||x i -G(x i )|| 2 (16)
the system loss function is:
L＝L G +L ij +L percep +L pixel (17)。
the specific steps of step 4 are as follows:
step 4.1, inputting the trained image obtained in the step 3 into an encoder, and inputting random noise z into a generator;
step 4.2, a generator G (z) generates an image according to the input random noise z and the related information of the real image data in the step 1, and a U-Net generation network generates a repaired high-resolution image according to a system loss function in the image generation process;
step 4.3, the real image data input to the encoder is mapped to Gaussian distribution to obtain a real image mapping relation, meanwhile, the generator generates the mapping relation of the high-resolution image repaired in the step 4.2, then the real image mapping relation and the mapping relation of the repaired high-resolution image are input into a VDB (visual desktop bus) discrimination network, and the VDB network applies the repaired high-resolution image information bottleneck to a discriminator to judge;
and 4.4, outputting the repaired high-resolution image by the discriminator, returning to the step 4.2 if the repaired high-resolution image does not meet the mapping relation of the repaired high-resolution image, otherwise, outputting the repaired high-resolution image, and setting the iteration number of repairing the high-resolution image according to the visual effect of the repaired high-resolution image.
The beneficial effects of the invention are:
(1) The super-resolution image restoration method based on the generative confrontation network trains the generative confrontation network model in a weak supervision learning mode, does not need excessive labels from the mapping of a real image data sample to a feature space, can sequence the mapping relation between the real image data and the generated image, has low training cost, and is beneficial to training the required trained image;
(2) The invention relates to a super-resolution image restoration method based on a generating type countermeasure network.A generator adopts a U-Net network, the network is a full convolution neural network, the input and the output of the network are images, a full connection layer structure is not provided, the problem of pixel positioning is solved by a shallower layer, the restoration of damaged pixels of the images is realized by a deeper layer, and the high-resolution restoration and reconstruction of the images are realized;
(3) The invention relates to a super-resolution image restoration method based on a generative confrontation network.A discriminator utilizes a variable score discriminator bottleneck (VDB) to weaken the capability of the discriminator by limiting mutual information, thereby balancing the training of a generative confrontation network model, meeting the thought of zero sum game of the generative confrontation network, improving the difficulty of the discriminator in distinguishing true and false, preventing the generative confrontation network model from being over-learned in the training, simultaneously playing a role in reducing dimension on image characteristic information, optimizing the restoration performance of the generative confrontation network model, and ensuring that a shallow network realizes high-resolution image restoration;
(4) The super-resolution image restoration method based on the generative confrontation network adopts a random gradient algorithm to update a generative confrontation network model, introduces a hyper-parameter control weight, accelerates the training speed of the generative confrontation network model, and enables the generative confrontation network model to be converged;
(5) The super-resolution image restoration method based on the generative confrontation network adopts the confrontation loss, the content loss, the perception loss and the reconstruction loss to calculate the system loss, so that the restored image can reach high resolution, and introduces a regular term to prevent the generative confrontation network model from training and trying to fit, thereby improving the stability of the generative confrontation network model.
Drawings
FIG. 1 is a schematic block diagram of a super-resolution image modification method based on a generative confrontation network according to the present invention;
FIG. 2 is a diagram of a generator network structure of a super-resolution image restoration method based on a generator countermeasure network according to the present invention;
FIG. 3 is a diagram of an example of a super-resolution image restoration method based on a generative countermeasure network according to FIG. 1;
fig. 4 is a diagram 2 of an example of high-resolution image restoration based on the super-resolution image restoration method of the generative countermeasure network according to the present invention.
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
The invention discloses a super-resolution image restoration method based on a generative countermeasure network, which is implemented according to the following steps as shown in figure 1:
step 1, collecting and sorting real image data to form a real image data sample set, processing the collected real images into images with the same size, and forming a training data set by the processed images with the same size; wherein, the real image data comprises an ancient fabric image damaged naturally;
step 2, constructing a generative confrontation network (GAN) model, wherein the generative confrontation network model comprises a generator and a discriminator;
the method for constructing the generative confrontation network model specifically comprises the following steps:
suppose the true image data sample set is { x } i ,y j At this time x i For image data samples, y j And if the label is a corresponding weak label, the objective function expression of the generated countermeasure network model is as follows:
wherein D represents a discriminator, G represents a generator, and E represents a mathematical expectation; x to p (x) represent the image distribution p (x) in the x obeying real image data sample set; z to G (z) represent a generation distribution G (z) of z-compliant images, x is real image data, z is random noise, D (x) represents a discriminant function, and D (z | x) represents a discriminant function of random noise z under the condition of the real image data x;
the Variational Discriminator Bottleneck (VDB) to the discriminator objective function in the generative confrontation network model is:
J(D,E)＝min D,E E x～p(x) [E z～E(z|x) [-log(D(x))]]+E z～G(z) [E z～E(z|x) [-log(1-D(z))]] (2)
wherein D represents the discriminator, E represents the expectation of the encoded image information, i.e. the mathematical expectation, x-p * (x) Representing x obedience generating image data distribution p * (x) (ii) a z-E (z | x) represents z obeying Gaussian distribution E (z | x), KL is a method of divergence, also called relative entropy information probability distribution, r (z) is the characteristic distribution of random noise z, and I c Generating mutual information of the image data for the generator, wherein D (z) represents a discriminant function of random noise z, and formula (3) is a constraint condition of formula (2);
by
Introducing a discriminator Lagrange optimization objective function:
wherein, beta is Lagrange multiplier, formula (5) is constraint condition of formula (4), and mutual information I c Representing the correlation between variables for limiting the correlation of real image data x and random noise z;
the VDB discriminator is a binary discriminator for discriminating between true and false by generating the correlation of image information by the generator
Limiting mutual information of image data samples as a limiting condition, mixing true and false image information, improving the distinguishing difficulty of a discriminator, balancing a system model, and updating parameters of a VDB discriminator as follows:
wherein, I r Representing the characteristic mutual information of the real image data, wherein r (x) is the characteristic distribution of the real image data;
as shown in fig. 2, the generator adopts U-Net to generate a network, the U-Net generating network is composed of a contraction path and an expansion path, and the contraction path and the expansion path are symmetrical to each other, the U-Net generating network totally includes 23 layers of convolution layers, after every two convolution layers of 3*3 on the contraction path, a maximum pooling layer of 2*2 is formed, and after each convolution layer, a relu activation function is used to perform down-sampling operation on an image original image in a training data set, wherein each down-sampling operation increases one cup of channel number for obtaining context information of a real image, after each convolution layer of 2*2 on the expansion path, two convolution layers of 3*3 are formed, and after each convolution layer, a relu activation function is used to perform up-sampling operation on an image in the training data set, wherein each up-sampling operation adds a feature map from the corresponding contraction path, performs clipping on a boundary portion of a lost pixel, and after that the image in the training data set and the contraction path keep the same shape as the feature map;
the discriminator adopts a VDB discrimination network which maps real image data transmitted to the encoder by the generator to Gaussian distribution, classifies images generated by the generator and acts information bottleneck on random noise z.
Step 3, printing masks with the same size on each low-resolution (LR) image in the training data set formed in the step 1 to obtain an image to be repaired, importing the image to be repaired into the generative confrontation network model constructed in the step 2, training the generative confrontation network model by adopting a weak supervision learning mode to obtain a trained image, updating the weights of the generator and the VDB discriminator in the training process, and calculating system loss by utilizing confrontation loss, content loss, perception loss and reconstruction loss;
the method comprises the following steps of updating the weights of a generative confrontation network model generator and a VDB (visual data base) discriminator by adopting a random gradient descent method, and specifically comprises the following steps:
for a set of real image data samples { x i ,y j Its loss function is:
wherein, y j E {0,1} is x i The label of (1);
let the training data set be
Image sample->
The input is obtained as corresponding weak supervision output
So that the image sample->
The structured risk function on the training dataset M is:
wherein, W and b respectively represent all weights and offset vectors in the network;
the regularization term is used for preventing overfitting during training of the image restoration model; λ is a hyper-parameter which is a positive number and is used for controlling the weight, the larger λ is, the closer W is to '0', N represents the nth image sample, N represents the total number of samples of the training data set, l represents the number of layers, and m represents the number of neurons;
then the weight W of the l-th layer in the iterative operation of the gradient descent algorithm (l) And an offset vector b (l) The updating method comprises the following steps:
wherein α is a learning rate, wherein the optimum learning rate α is 0.001;
the process of calculating the system loss by using the confrontation loss, the content loss, the perception loss and the reconstruction loss specifically comprises the following steps:
the generator confrontation loss of the generator confrontation network model is as follows:
L G ＝-E x～p(x) [log(1-D(z|x))]-E x～G(x) [logD(z|y)] (13)
wherein D (z | y) represents discriminant function distribution of random noise under a condition of real image data output;
in the image restoration problem, in order to match the restored damaged area of the image with the real image as much as possible, make the undamaged area clearer, and keep the high resolution of the restored image, the 1-norm distance is used to measure the difference between the undamaged area of the generator generated image and the undamaged area in the real image data:
the content loss is:
wherein, I represents the image loss area and the mutual information of the pixel points;
the perceptual loss is:
wherein, C j H j W j For normalization processing on the image path, [ phi ] j Pixel points in the real image data;
the reconstruction loss is:
L pixel ＝||x i -G(x i )|| 2 (16)
the system loss function is:
L＝L G +L ij +L percep +L pixel (17)；
step 4, inputting the training image obtained in the step 3 into a U-Net generation network to repair the image to be repaired after training, and outputting a repaired high-resolution image, wherein the specific process is as follows:
step 4.1, inputting the trained image obtained in the step 3 into an encoder, and inputting random noise z into a generator;
step 4.2, a generator G (z) generates an image according to the input random noise z and the related information (the upper, lower, left and right pixel information of the image missing region and the characteristic distribution information of the pixels) of the real image data in the step 1, and a U-Net generation network generates a repaired high-resolution (HR) image according to a system loss function in the image generation process;
step 4.3, the real image data input to the encoder is mapped to Gaussian distribution to obtain a real image mapping relation, meanwhile, the generator generates the mapping relation of the high-resolution image repaired in the step 4.2, then the real image mapping relation and the mapping relation of the repaired high-resolution image are input into a VDB (visual desktop bus) discrimination network, and the VDB network applies the repaired high-resolution image information bottleneck to a discriminator to judge;
and 4.4, outputting the repaired high-resolution image by the discriminator, returning to the step 4.2 if the repaired high-resolution image does not meet the mapping relation of the repaired high-resolution image, otherwise, outputting the repaired high-resolution image, and setting the iteration number of repairing the high-resolution image according to the visual effect of the repaired high-resolution image.
Fig. 3 is a diagram of an example of high-resolution image restoration by using the image restoration method of the present invention, where fig. 3 is a first column, a second column, and a third column from left to right, respectively, where the first column of pictures is a low-resolution (LR) image in a training data set, the second column is a to-be-restored image obtained by masking the first column of images with the same size, and the third column is a restored high-resolution image.
Fig. 4 is a diagram 2 of an example of high-resolution image restoration by using the image restoration method of the present invention, where fig. 4 is a first column, a second column and a third column from left to right, respectively, where the first column is an ancient textile image damaged naturally, the second column is a mask with the same size as the first column image, and an image to be restored is obtained, and the third column is a restored high-resolution image.
By adopting the mode, the generative confrontation network model is trained in a weak supervision learning mode, so that the training cost is reduced, and the convergence speed of the generative confrontation network model is improved; the high-resolution (HR) image is generated through the U-Net generation network, high-resolution restoration of a damaged image (a masked image, namely an image to be restored) is achieved, the network weight and the random gradient descent algorithm are balanced by the VDB discriminator through the whole image generation type confrontation network model to update parameters, complex iterative operation is avoided, and a good visual perception effect of high-resolution restoration of the damaged image is achieved.
Claims (10)
1. A super-resolution image restoration method based on a generative confrontation network is characterized by comprising the following steps:
step 1, collecting and sorting real image data to form a real image data sample set, processing the collected images into images with the same size, and forming a training data set by the processed images with the same size;
step 2, constructing a generative confrontation network model, wherein the generative confrontation network model comprises a generator and a discriminator;
step 3, printing a mask with the same size on each low-resolution image in the training data set formed in the step 1 to obtain an image to be repaired, importing the image to be repaired into the generative confrontation network model constructed in the step 2, training the generative confrontation network model by adopting a weak supervision learning mode to obtain a trained image, updating the weights of a generator and a VDB (visual desktop manager) discriminator in the training process, and calculating system loss by utilizing confrontation loss, content loss, perception loss and reconstruction loss;
and 4, inputting the training image obtained in the step 3 into a U-Net generation network to repair the trained image to be repaired, and outputting a repaired high-resolution image.
2. The super-resolution image restoration method based on the generative confrontation network as claimed in claim 1, wherein in step 2, the objective function expression of the generative confrontation network model is:
wherein D represents a discriminator, G represents a generator, and E represents a mathematical expectation; x to p (x) represent the image distribution p (x) in the x obeying real image data sample set; z to G (z) denote a generation distribution G (z) of z-compliant images, x is real image data, z is random noise, D (x) denotes a discriminant function, and D (z | x) denotes a discriminant function of the random noise z under the condition of the real image data x.
3. The super-resolution image restoration method based on the generative countermeasure network as claimed in claim 1, wherein in step 2, the pair of classifier objective functions of the variational classifier bottleneck VDB is:
J(D,E)＝min D,E E x～p(x) [E z～E(z|x) [-log(D(x))]]+E z～G(z) [E z～E(z|x) [-log(1-D(z))]] (2)
wherein D represents a discriminator, E represents an expectation of encoded image information, and x to p * (x) Representing x obedience generating image data distribution p * (x) (ii) a z-E (z | x) represents z obeying Gaussian distribution E (z | x), KL is a method of divergence, also called relative entropy information probability distribution, r (z) is the characteristic distribution of random noise z, and I c For the life of a living beingThe generator generates mutual information of the image data, D (z) represents a discriminant function of random noise z, and formula (3) is a constraint condition of formula (2).
4. The super-resolution image restoration method based on the generative countermeasure network as claimed in claim 3, wherein the super-resolution image restoration method is implemented by
Introducing a discriminator Lagrange optimization objective function:
wherein beta is Lagrange multiplier, formula (5) is constraint condition of formula (4), and mutual information I c The correlation between variables is expressed for limiting the correlation of the real image data x and the random noise z.
5. The super-resolution image restoration method based on the generative countermeasure network as claimed in claim 1, wherein in step 2, the VDB discriminator is a binary discriminator, and the correlation between the image information generated by the generator is used to discriminate between true and false
Limiting mutual information of image data samples for limiting conditions, obfuscating true and false image information, and updating parameters of a VDB discriminator as follows:
wherein, I r And (d) representing mutual characteristic information of the real image data, wherein r (x) is the characteristic distribution of the real image data.
6. The super-resolution image restoration method based on the generative countermeasure network as claimed in claim 1, wherein in step 2, the generator employs U-Net to generate the network, the U-Net generates the network, the network is composed of a contraction path and an expansion path, and the contraction path and the expansion path are symmetrical to each other, the U-Net generates the network totally includes 23 layers of convolution layers, each two convolution layers of 3*3 on the contraction path are followed by a maximum pooling layer of 2*2, and each convolution layer is followed by a relu activation function to perform down-sampling operation on the images in the training data set, wherein each down-sampling operation increases one cup channel number for obtaining context information of a real image, each convolution layer of 2*2 on the expansion path is followed by two convolution layers of 3*3, and each convolution layer is followed by a relu activation function to perform up-sampling operation on the images in the training data set, wherein each up-sampling operation adds a feature map from the corresponding contraction path, and keeps the same shape after each up-sampling operation.
7. The super-resolution image restoration method based on the generative countermeasure network as claimed in claim 1, wherein in step 2, the discriminator employs a VDB discrimination network, which maps the real image data transmitted from the generator to the encoder onto gaussian distribution, classifies the images generated by the generator, and applies information bottleneck to random noise z.
8. The super-resolution image restoration method based on the generative confrontation network according to claim 1, wherein in step 3, the parameters of the generative confrontation network model are updated by a stochastic gradient descent method, specifically:
for real image dataSample set { x i ,y j Its loss function is:
wherein, y j E {0,1} is x i The label of (1);
let the training data set be
Image sample->
The input gets a corresponding weakly supervised output->
The structured risk function of the image samples on the training dataset M is therefore:
wherein W and b represent all weights and bias vectors in the network, respectively;
the regularization term is used for preventing overfitting during training of the image restoration model; λ is a hyper-parameter which is a positive number and is used for controlling the weight, the larger λ is, the closer W is to '0', N represents the nth image sample, N represents the total number of samples of the training data set, l represents the number of layers, and m represents the number of neurons;
then the weight W of the l-th layer in the iterative operation of the gradient descent algorithm (l) And an offset vector b (l) The updating method comprises the following steps:
where α is the learning rate, where the optimum learning rate α is 0.001.
9. The super-resolution image restoration method based on the generative confrontation network as claimed in claim 1, wherein in step 3, the generative confrontation loss of the generative confrontation network model is:
L G ＝-E x～p(x) [log(1-D(z|x))]-E x～G(x) [logD(z|y)] (13)
wherein D (z | y) represents discriminant function distribution of random noise under a condition of real image data output;
the 1-norm distance is used to measure the difference between the undamaged area of the generator generated image and the undamaged area in the real image data:
the content loss is:
wherein, I represents the image loss area and the mutual information of the pixel points;
the perceptual loss is:
wherein, C j H j W j For normalization processing on the image path, [ phi ] j Pixel points in the real image data;
the reconstruction loss is:
L pixel ＝||x i -G(x i )|| 2 (16)
the system loss function is:
L＝L G +L ij +L percep +L pixel (17)。
10. the super-resolution image restoration method based on the generative countermeasure network according to claim 1, wherein the specific steps of step 4 are as follows:
step 4.1, inputting the trained image obtained in the step 3 into an encoder, and inputting random noise z into a generator;
step 4.2, a generator G (z) generates an image according to the input random noise z and the related information of the real image data in the step 1, and a U-Net generation network generates a repaired high-resolution image according to a system loss function in the image generation process;
step 4.3, the real image data input to the encoder is mapped to Gaussian distribution to obtain a real image mapping relation, meanwhile, the generator generates the mapping relation of the high-resolution image repaired in the step 4.2, then the real image mapping relation and the mapping relation of the repaired high-resolution image are input into a VDB (visual desktop bus) discrimination network, and the VDB network applies the repaired high-resolution image information bottleneck to a discriminator to judge;
and 4.4, outputting the repaired high-resolution image by the discriminator, returning to the step 4.2 if the repaired high-resolution image does not meet the mapping relation of the repaired high-resolution image, otherwise, outputting the repaired high-resolution image, and setting the iteration number of repairing the high-resolution image according to the visual effect of the repaired high-resolution image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910963790.6A CN110827213B (en) | 2019-10-11 | 2019-10-11 | Super-resolution image restoration method based on generation type countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910963790.6A CN110827213B (en) | 2019-10-11 | 2019-10-11 | Super-resolution image restoration method based on generation type countermeasure network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110827213A CN110827213A (en) | 2020-02-21 |
| CN110827213B true CN110827213B (en) | 2023-03-31 |
Family
ID=69549240
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910963790.6A Active CN110827213B (en) | 2019-10-11 | 2019-10-11 | Super-resolution image restoration method based on generation type countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110827213B (en) |
Families Citing this family (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111445395B (en) * | 2020-03-03 | 2023-03-21 | 哈尔滨工程大学 | Method for repairing middle area of side-scan sonar waterfall image based on deep learning |
| CN111199550B (en) * | 2020-04-09 | 2020-08-11 | 腾讯科技（深圳）有限公司 | Training method, segmentation method, device and storage medium of image segmentation network |
| CN111553854A (en) * | 2020-04-21 | 2020-08-18 | 维沃移动通信有限公司 | Image processing method and electronic equipment |
| CN111553858B (en) * | 2020-04-28 | 2022-04-08 | 四川大学青岛研究院 | Image restoration method and system based on generation countermeasure network and application thereof |
| CN111612721B (en) * | 2020-05-22 | 2023-09-22 | 哈尔滨工业大学（深圳） | Image restoration model training method and device and satellite image restoration method and device |
| CN111681178B (en) * | 2020-05-22 | 2022-04-26 | 厦门大学 | Knowledge distillation-based image defogging method |
| CN111968193B (en) * | 2020-07-28 | 2023-11-21 | 西安工程大学 | Text image generation method based on StackGAN (secure gas network) |
| CN112102186B (en) * | 2020-09-07 | 2024-04-05 | 河海大学 | Real-time enhancement method for underwater video image |
| CN112634135B (en) * | 2020-12-23 | 2022-09-13 | 中国地质大学（武汉） | Remote sensing image super-resolution reconstruction method based on super-resolution style migration network |
| CN113012049B (en) * | 2021-04-15 | 2022-08-02 | 山东新一代信息产业技术研究院有限公司 | Remote sensing data privacy protection method based on GAN network |
| CN113378980B (en) * | 2021-07-02 | 2023-05-09 | 西安电子科技大学 | Mask face shielding recovery method based on self-adaptive context attention mechanism |
| CN113487565A (en) * | 2021-07-03 | 2021-10-08 | 肇庆学院 | Super-resolution analysis method for Sentiel-2 satellite remote sensing image |
| CN113469913B (en) * | 2021-07-06 | 2022-06-03 | 中南大学 | Hot-rolled strip steel surface water drop removing method based on gradual cycle generation countermeasure network |
| CN113538238B (en) * | 2021-07-09 | 2024-06-28 | 深圳市深光粟科技有限公司 | High-resolution photoacoustic image imaging method and device and electronic equipment |
| CN113781316B (en) * | 2021-07-28 | 2024-05-17 | 杭州火烧云科技有限公司 | High-resolution image restoration method and restoration system based on countermeasure generation network |
| CN113592715B (en) * | 2021-08-05 | 2024-05-24 | 昆明理工大学 | Super-resolution image reconstruction method for small sample image set |
| CN113627538B (en) * | 2021-08-12 | 2024-03-01 | 群联电子股份有限公司 | Method for training asymmetric generation of image generated by countermeasure network and electronic device |
| CN113744166B (en) * | 2021-08-18 | 2023-08-08 | 中国科学院计算技术研究所 | Method for enhancing image of turbid underwater fish |
| CN113516656B (en) * | 2021-09-14 | 2021-12-14 | 浙江双元科技股份有限公司 | Defect image data processing simulation method based on ACGAN and Cameralink cameras |
| CN113918716B (en) * | 2021-10-14 | 2023-06-02 | 中山大学 | Method and device for constructing generated countermeasure topic model based on spectrum norm normalization |
| CN114066729B (en) * | 2021-10-19 | 2024-09-13 | 华南理工大学 | Face super-resolution reconstruction method capable of recovering identity information |
| CN113793286B (en) * | 2021-11-18 | 2022-05-10 | 成都索贝数码科技股份有限公司 | Media image watermark removing method based on multi-order attention neural network |
| CN114549328B (en) * | 2022-04-24 | 2022-07-22 | 西南财经大学 | JPG image super-resolution restoration method, computer readable storage medium and terminal |
| CN115880158B (en) * | 2023-01-30 | 2023-10-27 | 西安邮电大学 | Blind image super-resolution reconstruction method and system based on variation self-coding |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108305239B (en) * | 2018-03-06 | 2020-08-11 | 陕西师范大学 | Bridge crack image repairing method based on generation type countermeasure network |
| CN109377448B (en) * | 2018-05-20 | 2021-05-07 | 北京工业大学 | Face image restoration method based on generation countermeasure network |
| GB201809604D0 (en) * | 2018-06-12 | 2018-07-25 | Tom Tom Global Content B V | Generative adversarial networks for image segmentation |
-
2019
- 2019-10-11 CN CN201910963790.6A patent/CN110827213B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110827213A (en) | 2020-02-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110827213B (en) | Super-resolution image restoration method based on generation type countermeasure network | |
| CN109859190B (en) | Target area detection method based on deep learning | |
| US10891511B1 (en) | Human hairstyle generation method based on multi-feature retrieval and deformation | |
| CN109919013A (en) | Method for detecting human face and device in video image based on deep learning | |
| WO2020165557A1 (en) | 3d face reconstruction system and method | |
| CN109934826A (en) | A kind of characteristics of image dividing method based on figure convolutional network | |
| CN108932536A (en) | Human face posture method for reconstructing based on deep neural network | |
| CN110310277B (en) | Fabric defect detection method based on depth feature and NTV-RPCA | |
| CN113724354B (en) | Gray image coloring method based on reference picture color style | |
| CN108765540B (en) | Relighting method based on image and ensemble learning | |
| CN110660020A (en) | Image super-resolution method of countermeasure generation network based on fusion mutual information | |
| CN113870157A (en) | SAR image synthesis method based on cycleGAN | |
| CN112233129A (en) | Deep learning-based parallel multi-scale attention mechanism semantic segmentation method and device | |
| CN109003287A (en) | Image partition method based on improved adaptive GA-IAGA | |
| CN116051683B (en) | Remote sensing image generation method, storage medium and device based on style self-organization | |
| CN112580502A (en) | SICNN-based low-quality video face recognition method | |
| CN109829857A (en) | A kind of antidote and device based on the tilted image for generating confrontation network | |
| CN116580121B (en) | Method and system for generating 2D model by single drawing based on deep learning | |
| CN111882495A (en) | Image highlight processing method based on user-defined fuzzy logic and GAN | |
| CN115018729B (en) | Content-oriented white box image enhancement method | |
| CN114445726B (en) | Sample library establishing method and device based on deep learning | |
| CN109948541A (en) | A kind of facial emotion recognition methods and system | |
| CN115294424A (en) | Sample data enhancement method based on generation countermeasure network | |
| CN114820381A (en) | Digital image restoration method based on structure information embedding and attention mechanism | |
| Geng et al. | A novel color image segmentation algorithm based on JSEG and Normalized Cuts |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |