CN113689360A - Image restoration method based on generation countermeasure network - Google Patents

Abstract

The invention discloses an image restoration method based on a generation countermeasure network, which comprises the following steps: 1. preprocessing an original image to obtain an image to be restored; 2. coding an image data set Z to be repaired to obtain a coded image data set; 3. constructing a generation countermeasure network consisting of a generation network and a discrimination network; 4. and training the pre-constructed confrontation network model by utilizing the encoded image data set to be repaired. According to the method, the key characteristics of the image to be restored are collected, the generation model and the discrimination model are trained, the efficiency of model optimization can be effectively improved, and the restoration effect of the image can be improved.

Description

Image restoration method based on generation countermeasure network

Technical Field

The invention relates to the field of image restoration, in particular to an image restoration method based on a generation countermeasure network.

Background

The image restoration technology is a hot problem in the field of image processing, and belongs to the cross problem of multiple disciplines such as pattern recognition, computer vision and the like. Image restoration refers to processing image damage or loss of a local area according to a specific requirement to restore the integrity of the image. The principles of similarity, structure, texture consistency, structure priority, etc. must be followed during the repair process. At present, a restoration method based on deep learning is a new method proposed in recent years, a deep neural network is utilized to obtain mapping of nonlinear complex relations among training samples through training and learning of a large amount of data, researchers propose various image restoration methods on the basis of the mapping, and the method is widely applied to the fields of ancient painting cultural relic protection, medicine, movie industry and the like.

The current image restoration technology restores an image to be restored by using a preset algorithm, but the restoration result has a large difference with an original image, so that a boundary artifact and inconsistent fuzzy textures in a surrounding area are often generated, the training is slow or unstable, the image to be restored has a large damage degree and is easy to blur, the restoration result is unsatisfactory, and the image restoration work is inconvenient.

Disclosure of Invention

The invention aims to overcome the defects in the prior art, and provides an image restoration method based on a generation countermeasure network, so that a generation model and a discrimination model are trained by collecting key features of an image to be restored, the efficiency of model optimization can be effectively improved, and the restoration effect of the image can be improved.

In order to achieve the purpose, the invention adopts the following technical scheme:

the invention discloses an image restoration method based on a generation countermeasure network, which is characterized by comprising the following steps:

step 1, preprocessing an original image and obtaining an image to be restored;

step 1.1, extracting M images from the original image set data to form an image training set M ═ x₁,x₂,…,x_mIn which x_mRepresenting the m-th image, for the m-th image x_mAfter selecting the key characteristic area, the mth image x_mThe non-key characteristic region is randomly divided into regions with fixed sizes and used as different damaged regions;

step 1.2, comparing each damaged areaFinding the domain block with the highest similarity to fill the corresponding damaged area so as to obtain the mth image x_mTo-be-repaired image, and then obtaining a to-be-repaired image data set Z ═ Z₁,Z₂,…,Z_m}；

Step 2, coding the image data set to be repaired Z to obtain a coded image data set Z '═ Z'₁,Z′₂,…,Z′_mWherein, Z'_mCoded image data representing an mth image to be restored;

step 3, constructing a generation countermeasure network consisting of a generation network and a judgment network;

step 3.1, reconstructing a generation network of the styleGAN network and using the reconstructed generation network as a generation network in a generation countermeasure network;

setting a generation network in the generation countermeasure network to be composed of a mapping network f and a synthesis network g;

the mapping network f is composed of a fully-connected layers, and each fully-connected layer comprises s neurons;

encoding image data set Z '{ Z'₁,Z′₂,…,Z′_mNormalizing, inputting into the mapping network f, and performing spatial mapping, thereby outputting an intermediate mapping vector W ═ W₁,W₂,…,W_mIn which W_mRepresents the mth intermediate mapping vector;

the synthetic network consists of two composite networks; the first composite network is formed by sequentially connecting a first AdaIN module, a first convolution module and a second AdaIN module in series, and the output of each module is used as the input of the next module;

the second composite network is formed by sequentially connecting a second convolution module, a third AdaIN module, a third convolution module and a fourth AdaIN module, wherein the first convolution module, the second convolution module and the third convolution module are all composed of an NxN convolution layer, a batch normalization layer and a LeakyReLu activation function, and random noise is added behind each convolution module;

using a constant tensor with dimension of c × c × e asAn input to a first AdaIN module in a first composite network, an intermediate mapping vector W ═ W₁,W₂,…,W_m-as another input to a first AdaIN module in a first composite network, and outputting a set of images of dimensions c × c × e via said first composite network;

inputting the image set with the dimension of c × c × e into a second convolution module in a second composite network, obtaining convolution results, inputting the convolution results into a third AdaIN module and a third convolution module respectively, and outputting a repaired image data set with the dimension of f × f × e by a fourth AdaIN module, wherein G (Z) ═ G (Z) (Z is the same as G (Z) × e)₁),G(Z₂),…,G(Z_m)}；

Step 3.2, constructing and generating a discrimination network in the countermeasure network;

the method comprises the steps that a discrimination network is arranged and composed of b convolution modules, d full-connection layers and a Sigmoid activation function which are sequentially connected in series, each convolution module comprises a convolution layer with a convolution kernel of NxN and a LeakyReLu activation function, and the other b-1 convolution modules are respectively connected with a batch normalization layer except a first convolution module;

the inpainted image dataset G (Z) ═ G (Z)₁),G(Z₂),…,G(Z_m) X and image training set M ═ x₁,x₂,…,x_mInputting the result into the discrimination network for processing, outputting discrimination true and false values by the Sigmoid activation function, feeding the values back to the generation network to accelerate network training, and obtaining a repair result D (G (Z)) which passes through the discrimination network after training is finished₁)),D(G(Z₂)),…,D(G(Z_m) In which D (G (Z)) is_m) Represents the mth image to be restored Z passing through the discrimination network_mThe repair result of (2);

step 4, establishing a confrontation target function L shown in the formula (1)_D：

In the formula (1), E represents a desired, D (x)_m) Representing the m-th original image x_mInput deviceThe repair result output after the network is judged,

representing the distribution of a training set M from images

Taking out any image x;

representing a distribution from a data set Z to be repaired

Taking out any image to be repaired;

step 5, establishing a generation target function L shown in the formula (2)_G：

In the formula (2), L_CRepresents the content loss and is obtained by the formula (3), and lambda is a regular parameter;

L_C＝||D(G(Z_m))-x_m||₂ (3)

step 6, based on the confrontation objective function L_DAnd generating an objective function L_GUsing the encoded image data set Z '{ Z'₁,Z′₂,…,Z′_mAnd training the generated countermeasure network until the output values of the network are judged to be true, so as to obtain a trained image restoration model for restoring the damaged image.

The image restoration method based on the generation countermeasure network is characterized in that the input of other AdaIN modules except the first AdaIN module is an intermediate mapping vector W ═ W₁,W₂,…,W_mAnd the output of the previous module, wherein any AdaIN module obtains a corresponding output result AdaIN (x) by using the formula (4)_m)：

In the formula (4), y_s,mRepresenting the m-th image x_mScaling factor of, y_b,mRepresenting the m-th image x_mDeviation factor of, mu (x)_m) Is the m-th image x_mMean value of (a), σ (x)_m) Is the m-th image x_mIf the variance of (a) is the first AdaIN block, then x_mIs a constant tensor.

Compared with the prior art, the invention has the beneficial effects that:

1. according to the method, the key features of the image to be restored are collected, and the image preprocessing is performed by using similar block filling to form a training set, so that the details and image semantics of the original image can be better retained, and the training of the model is facilitated;

2. according to the method, the restored image obtained by the generation model is used as the input of the discrimination model, the output of the discrimination model is used as the input of the generation model, so that the generation model and the discrimination model in the confrontation network model are continuously optimized, the optimized image restoration model based on the confrontation network model can be obtained and used for restoring the image, the efficiency of model optimization is effectively improved, and the restoration effect of the image is improved.

Drawings

FIG. 1 is a flow chart of an image inpainting method of the present invention;

FIG. 2 is a schematic diagram of the generation of a countermeasure network model of the present invention;

fig. 3 is a schematic diagram of the generation network of the countermeasure network of the present invention.

Detailed Description

In this embodiment, as shown in fig. 1, an image restoration method based on a generative countermeasure network includes the following steps:

step 1, preprocessing an original image and obtaining an image to be restored;

step 1.1, 6000 images are extracted from the original image set data to form an image training set M ═ x₁,x₂,…,x_mIn which x_mTo representM-th image, for m-th image x_mSelecting a key characteristic area, keeping the detail points of the image as much as possible, and taking the m-th image x_mThe non-key characteristic region is randomly divided into regions with fixed sizes and used as different damaged regions;

step 1.2, comparing the similarity degree of the neighborhood blocks around each damaged area, finding the domain block with the highest similarity degree to fill the corresponding damaged area, utilizing the surrounding similar blocks to enable the structure and texture of the damaged area to be similar to the key feature area, reducing loss, and obtaining the mth image x_mTo-be-repaired image, and then obtaining a to-be-repaired image data set Z ═ Z₁,Z₂,…,Z_m}；

Step 2, coding the image data set to be repaired Z to obtain a coded image data set Z '═ Z'₁,Z′₂,…,Z′_mWherein, Z'_mCoded image data representing an mth image to be restored;

step 3, as shown in fig. 2, constructing a generation countermeasure network composed of a generation network and a discrimination network;

step 3.1, reconstructing a generation network of the styleGAN network and using the reconstructed generation network as a generation network in a generation countermeasure network;

as shown in fig. 3, the generation network in the setup generation countermeasure network is composed of a mapping network f and a synthesis network g, the mapping network f controls the style of the generated image, and the generation network g is used to generate the image like a conventional generator;

the mapping network f is composed of 8 fully-connected layers, and each fully-connected layer comprises 128 neurons;

encoding image data set Z '{ Z'₁,Z′₂,…,Z′_mNormalizing the vector, inputting the normalized vector into a mapping network f for spatial mapping, and outputting an intermediate mapping vector W (W) { W }₁,W₂,…,W_mIn which W_mRepresents the mth intermediate mapping vector;

the synthetic network g consists of two composite networks; the first composite network is formed by sequentially connecting a first AdaIN module, a first convolution module and a second AdaIN module in series, and the output of each module is used as the input of the next module;

the second composite network is formed by sequentially connecting a second convolution module, a third AdaIN module, a third convolution module and a fourth AdaIN module, wherein the first convolution module, the second convolution module and the third convolution module are all composed of a 3 x 3 convolution layer, a batch normalization layer and a LeakyReLu activation function, and random noise is added behind each convolution module;

in the AdaIN modules, the inputs of the other AdaIN modules except the first AdaIN module are all intermediate mapping vectors W ═ W₁,W₂,…,W_mAnd the output of the previous module, wherein any AdaIN module obtains a corresponding output result AdaIN (x) by using the formula (1)_m)：

In the formula (1), y_s,mRepresenting the m-th image x_mScaling factor of, y_b,mRepresenting the m-th image x_mDeviation factor of, mu (x)_m) Is the m-th image x_mMean value of (a), σ (x)_m) Is the m-th image x_mIf the variance of (a) is the first AdaIN block, then x_mIs a constant tensor.

Taking a constant tensor with the dimension of 64 × 64 × 128 as an input of a first AdaIN module in a first composite network, and setting an intermediate mapping vector W to { W ═ W₁,W₂,…,W_mAs another input to a first AdaIN module in the first composite network, and outputting a set of images having dimensions 64 × 64 × 128 via the first composite network;

inputting an image set with the dimension of 64 × 64 × 128 into a second convolution module of a second composite network, obtaining convolution results, inputting the convolution results into a third AdaIN module and a third convolution module respectively, and outputting a repaired image data set with the dimension of 128 × 128 × 128 by a fourth AdaIN module, wherein G (Z) ═ G (Z) (Z is₁),G(Z₂),…,G(Z_m)}；

Step 3.2, constructing and generating a discrimination network in the countermeasure network;

the method comprises the steps that a set discrimination network is formed by sequentially connecting 4 convolution modules, 1 full-connection layer and a Sigmoid activation function in series, each convolution module comprises a convolution layer with a convolution kernel of 3 x 3 and a LeakyReLu activation function, and the rest 3 convolution modules are respectively connected with a batch normalization layer except the first convolution module;

the repaired image data set G (Z) ═ G (Z)₁),G(Z₂),…,G(Z_m) X and image training set M ═ x₁,x₂,…,x_mInputting the data into a discrimination network for processing, outputting discrimination true and false values by a Sigmoid activation function, feeding the values back to a generation network to accelerate network training, and obtaining a repair result D (G (Z)) which passes through the discrimination network after the training is finished, wherein D (G (Z)) is equal to { D (G (Z)) }₁)),D(G(Z₂)),…,D(G(Z_m) In which D (G (Z)) is_m) Represents the mth image to be restored Z passing through the discrimination network_mThe repair result of (2);

step 4, establishing a confrontation target function L shown as the formula (2)_D：

In the formula (2), E represents a desired, D (x)_m) Representing the m-th original image x_mInputting the repair result output after the judgment network,

representing the distribution of a training set M from images

Taking out any image x;

representing a distribution from a data set Z to be repaired

Taking out any graph to be repairedAn image;

step 5, establishing a generation target function L shown in the formula (3)_G：

In the formula (3), L_CRepresents the content loss and is obtained by equation (4), λ is a regularization parameter;

L_C＝||D(G(Z_m))-x_m||₂ (4)

step 6, based on the confrontation objective function L_DAnd generating an objective function L_GUsing the encoded image data set Z '{ Z'₁,Z′₂,…,Z′_mTraining the generation countermeasure network until the output values of the judgment network are all true, thereby obtaining a trained image restoration model for restoring the damaged image.

And comparing the repaired image data with the image data of the database, and then storing the image data in the field, thereby facilitating the query.

Experimental effect or technical effect of comparison:

in order to verify the effectiveness, the invention evaluates the repairing result of the test data set by using PSNR (Peak Signal to Noise ratio) peak Signal-to-Noise ratio and SSIM (structural similarity) structure similarity. The peak signal-to-noise ratio of PSNR is the difference in measured pixel values, with larger values representing less distortion. The SSIM structural similarity represents the similarity between the repair result and the original image, and is a numerical value between 0 and 1, and a larger numerical value represents a smaller difference. The specific quantitative comparison results are shown in table 1:

TABLE 1

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention; the scope of the invention is defined by the appended claims and equivalents thereof.

Claims (2)

1. An image restoration method based on a generation countermeasure network is characterized by comprising the following steps:

step 1, preprocessing an original image and obtaining an image to be restored;

step 1.1, extracting M images from the original image set data to form an image training set M ═ x₁,x₂,…,x_mIn which x_mRepresenting the m-th image, for the m-th image x_mAfter selecting the key characteristic area, the mth image x_mThe non-key characteristic region is randomly divided into regions with fixed sizes and used as different damaged regions;

step 1.2, comparing the similarity of the neighborhood blocks around each damaged area, and finding the domain block with the highest similarity to fill the corresponding damaged area, thereby obtaining the mth image x_mTo-be-repaired image, and then obtaining a to-be-repaired image data set Z ═ Z₁,Z₂,…,Z_m}；

Step 2, coding the image data set Z to be repaired to obtain a coded image data set Z' ═ { Z ═ Z₁′,Z₂′,…,Z′_mWherein, Z'_mCoded image data representing an mth image to be restored;

step 3, constructing a generation countermeasure network consisting of a generation network and a judgment network;

step 3.1, reconstructing a generation network of the styleGAN network and using the reconstructed generation network as a generation network in a generation countermeasure network;

setting a generation network in the generation countermeasure network to be composed of a mapping network f and a synthesis network g;

the mapping network f is composed of a fully-connected layers, and each fully-connected layer comprises s neurons;

encoding image data set Z '{ Z'₁,Z′₂,…,Z′_mInputting the mapping after normalization processingSpatial mapping is performed in the network f, thereby outputting an intermediate mapping vector W ═ W₁,W₂,…,W_mIn which W_mRepresents the mth intermediate mapping vector;

the synthetic network consists of two composite networks; the first composite network is formed by sequentially connecting a first AdaIN module, a first convolution module and a second AdaIN module in series, and the output of each module is used as the input of the next module;

the second composite network is formed by sequentially connecting a second convolution module, a third AdaIN module, a third convolution module and a fourth AdaIN module, wherein the first convolution module, the second convolution module and the third convolution module are all composed of an NxN convolution layer, a batch normalization layer and a LeakyReLu activation function, and random noise is added behind each convolution module;

taking a constant tensor with the dimension of c × c × e as an input of a first AdaIN module in a first composite network, wherein an intermediate mapping vector W is { W ═ W₁,W₂,…,W_m-as another input to a first AdaIN module in a first composite network, and outputting a set of images of dimensions c × c × e via said first composite network;

inputting the image set with the dimension of c × c × e into a second convolution module in a second composite network, obtaining convolution results, inputting the convolution results into a third AdaIN module and a third convolution module respectively, and outputting a repaired image data set with the dimension of f × f × e by a fourth AdaIN module, wherein G (Z) ═ G (Z) (Z is the same as G (Z) × e)₁),G(Z₂),…,G(Z_m)}；

Step 3.2, constructing and generating a discrimination network in the countermeasure network;

the method comprises the steps that a discrimination network is arranged and composed of b convolution modules, d full-connection layers and a Sigmoid activation function which are sequentially connected in series, each convolution module comprises a convolution layer with a convolution kernel of NxN and a LeakyReLu activation function, and the other b-1 convolution modules are respectively connected with a batch normalization layer except a first convolution module;

the inpainted image dataset G (Z) ═ G (Z)₁),G(Z₂),…,G(Z_m) X and image training set M ═ x₁,x₂,…,x_mInputting the result into the discrimination network for processing, outputting discrimination true and false values by the Sigmoid activation function, feeding the values back to the generation network to accelerate network training, and obtaining a repair result D (G (Z)) which passes through the discrimination network after training is finished₁)),D(G(Z₂)),…,D(G(Z_m) In which D (G (Z)) is_m) Represents the mth image to be restored Z passing through the discrimination network_mThe repair result of (2);

step 4, establishing a confrontation target function L shown in the formula (1)_D：

In the formula (1), E represents a desired, D (x)_m) Representing the m-th original image x_mInputting the repair result output after the judgment network,

representing the distribution of a training set M from images

Taking out any image x;

representing a distribution from a data set Z to be repaired

Taking out any image to be repaired;

step 5, establishing a generation target function L shown in the formula (2)_G：

In the formula (2), L_CRepresents the content loss and is obtained by the formula (3), and lambda is a regular parameter;

L_C＝||D(G(Z_m))-x_m||₂ (3)

step 6, based on the confrontation objective function L_DAnd generating an objective function L_GUsing the encoded image data set Z '{ Z'₁,Z′₂,…,Z′_mAnd training the generated countermeasure network until the output values of the network are judged to be true, so as to obtain a trained image restoration model for restoring the damaged image.

2. The method as claimed in claim 1, wherein the AdaIN modules except the first AdaIN module have intermediate mapping vector W ═ W₁,W₂,…,W_mAnd the output of the previous module, wherein any AdaIN module obtains a corresponding output result AdaIN (x) by using the formula (4)_m)：

In the formula (4), y_s,mRepresenting the m-th image x_mScaling factor of, y_b,mRepresenting the m-th image x_mDeviation factor of, mu (x)_m) Is the m-th image x_mMean value of (a), σ (x)_m) Is the m-th image x_mIf the variance of (a) is the first AdaIN block, then x_mIs a constant tensor.

Images