CN113160081A - Depth face image restoration method based on perception deblurring - Google Patents
Depth face image restoration method based on perception deblurring Download PDFInfo
- Publication number
- CN113160081A CN113160081A CN202110408739.6A CN202110408739A CN113160081A CN 113160081 A CN113160081 A CN 113160081A CN 202110408739 A CN202110408739 A CN 202110408739A CN 113160081 A CN113160081 A CN 113160081A
- Authority
- CN
- China
- Prior art keywords
- layer
- layers
- picture
- resolution
- deblurring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000008447 perception Effects 0.000 title claims abstract description 44
- 238000012549 training Methods 0.000 claims abstract description 35
- 238000012360 testing method Methods 0.000 claims abstract description 31
- 230000006870 function Effects 0.000 claims description 39
- 230000004913 activation Effects 0.000 claims description 33
- 238000013528 artificial neural network Methods 0.000 claims description 22
- 238000010586 diagram Methods 0.000 claims description 20
- 238000010606 normalization Methods 0.000 claims description 15
- 230000001902 propagating effect Effects 0.000 claims description 14
- 238000012113 quantitative test Methods 0.000 claims description 14
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 12
- 238000013527 convolutional neural network Methods 0.000 claims description 10
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 9
- 239000011800 void material Substances 0.000 claims description 9
- 239000011265 semifinished product Substances 0.000 claims description 8
- 230000000694 effects Effects 0.000 claims description 6
- 230000000644 propagated effect Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 3
- 230000008439 repair process Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000010276 construction Methods 0.000 claims description 2
- 230000001815 facial effect Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 abstract description 3
- 230000001427 coherent effect Effects 0.000 abstract 1
- 230000006872 improvement Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a depth face image restoration method based on perception deblurring, which comprises the following steps: step S1: and giving a face image restoration training set and a testing set. Step S2: designing coarse reconstruction generator G1Global discriminator D1And a local discriminator D2. Step S3: using G1、D1And D2And data are transmitted forward, loss is calculated and transmitted backward, and model parameters are updated. Step S4: and repeating the step 3 until the coarse reconstruction training is finished. Step S5: design aware deblurring generator G2And a discriminator D3. Step S6: using G2And D3And data are transmitted forward, loss is calculated and transmitted backward, and model parameters are updated. Step S7: and repeating the step 6 until the training of the perception deblurring stage is finished. Step S8: use the surveyAnd (5) testing and judging the model, and storing the optimal model. The invention can quickly give the prediction result when the human face picture with missing information is given. The prediction result has good visual effect, keeps coherent content and has correct semantic information.
Description
Technical Field
The invention belongs to the field of image restoration, and particularly relates to a human face image restoration algorithm based on CNN and GAN, which better solves the problems of insufficient consistency of restoration contents and discontinuous contents of restoration boundary areas of the existing human face image.
Background
The problem of image restoration, which aims at filling missing pixels in damaged images, has attracted a lot of attention and has been a valuable and active research topic for decades, since high quality images can benefit from a wide range of applications, such as restoration of old photographs, object removal, etc. High quality image inpainting generally requires filling missing regions with not only visually realistic, but also semantically reasonable content. Existing methods can be broadly divided into two categories. The first category attempts texture synthesis techniques at the fill area. In particular, these methods are typically sample and paste the full image resolution patch of the source image into the missing region, allowing the synthesis of detailed results. However, due to the lack of high-level understanding of the images, such methods often fail to produce semantically reasonable results. To solve this problem, a second group of methods proposes to encode the semantic context of an image into a potential feature space through a deep neural network, and then generate semantically relevant information by generating a model. However, generating visually realistic results from a compact latent feature remains a challenge, as full image resolution detail is typically convolved and pooled by stacking. The face restoration is a branch of image restoration, is an important work in computer vision, and is widely applied to image editing, image rendering and computer photography.
Disclosure of Invention
The invention provides a depth face image restoration method based on perception deblurring, aiming at the problems of low efficiency and poor robustness of a depth learning face image restoration technology and a traditional face image restoration algorithm and the problems of insufficient content consistency and discontinuous content of a restoration boundary region of a face image detection method in the prior art. And in the second stage, the neural network with a perception deblurring method is used for carrying out detail refinement on the semi-finished product generated in the first stage to generate a completely repaired picture.
The technical scheme of the invention is a depth face image restoration method based on perception deblurring, which comprises the following steps:
step S1: a facial image restoration data set is given and divided into a training set and a test set;
step S2: coarse reconstruction stage generator G in deep face image restoration method based on perception deblurring construction1Global arbiter D1And a local discriminator D2Constructing an ADADELTA optimizer;
step S3: in each iteration training of the coarse reconstruction stage, sampling a picture x in the training set, randomly generating a binary mask m, wherein the value of a region to be repaired in m is 1, the value of the rest regions is 0, and taking x (1-m) as G1Inputting an iterative training; an operation for multiplying corresponding elements in the matrix; d1,2Is D1And D2Connecting the last characteristic layer; input via G1Forward propagation to obtain a semi-finished product x', using D1,2Calculating L according to the probability of judging x' as a real pictureadv1And calculating L by using the error between x' and xpsnrAnd Lssim(ii) a At fixed position D1And D2When the parameters of (1) are used, L is usedadv1、LpsnrAnd LssimCounter propagating pair G1Updating parameters; at the fixed position G1When the parameters of (1) are D1,2To judge the probability that x is a real picture, the probability is calculated by BCELoss, the judgment error is propagated reversely, and the pair D1And D2Updating parameters; by D1,2The probability that x' is judged to be a false picture is calculated by using BCELoss, judgment error is calculated and propagated backwards, and the pair D1And D2Updating parameters;
step S4: at G1And D1,2In the alternate updating of parameters, G1Go on and off to generate more realistic samples, D1,2Then an effort is made to determine whether the generated sample is true or false, and the two are played against each other until nash balance is achievedRepeating the step S3 until the training of one stage is finished;
step S5: method for constructing generator G in perception deblurring stage in depth face image restoration method based on perception deblurring2And a discriminator D3(ii) a Constructing an Adam optimizer;
step S6: in each iteration training of the perceptual deblurring stage, a pair of pictures (x, y) is sampled, wherein y ═ x [ (1-m) + x' ] m, and y is used as G2Through G2Forward propagation yields the result y', with D3To distinguish y 'as a true picture, i.e. the probability that y' passes through D3Calculating L by the difference between the result after forward propagation and Trueadv2(ii) a Calculating L by using the error between y' and xper(ii) a At fixed position D3When the parameters of (1) are used, L is usedadv2And LperCounter propagating pair G2Updating parameters; at the fixed position G2When the parameters of (1) are D3To judge the probability that x is a real picture, namely calculating the judgment error back propagation by using the difference value, and to D3Updating parameters; by D3To judge y' as false picture probability, i.e. to calculate the judgment error back propagation by difference value, for D3Updating parameters;
step S7: at G2And D3In the alternate updating of parameters, G2Go on and off to generate more realistic samples, D3The method tries to judge whether the generated sample is real or false, and the two samples are played mutually until Nash balance is achieved, namely the step S6 is repeated until the two-stage training is finished;
step S8: in the testing stage, selecting the picture of the test set, the picture passing through G1And G2Forward propagating according to the sequence to obtain a prediction result; by calculating z ═ x ÷ y '<' > m, a final prediction z is obtained; judging the model and storing the optimal model through qualitative test and quantitative test of z;
in the step S3, the input is via G1Forward propagation yields x', specifically:
sampling from a training set to obtain a batch of original data, wherein each real picture x in the original data comprises three RGB channels; firstly, preprocessing the pictureC, processing; then randomly generating a single-channel mask m, wherein the method comprises the steps of initializing a single-channel two-dimensional matrix with the same size as the original image, setting the value of the matrix in a region to be repaired to be 1, and setting the value of the matrix in the rest regions to be 0; then, take x [ -1-m ] as input and normalize it to keep all pixel values [ -1,1 [ ]]To (c) to (d); input via G1Obtaining x' after forward propagation; by D1,2To determine the probability that x 'is a true picture, i.e. x' passes through D1,2After forward propagation, L is calculated using the result and the BCELoss value of Trueadv1(ii) a The calculation method comprises the following steps:
Ladv1＝-log[D1,2(G1(x,m))]
meanwhile, calculating L through the error of x' and x in the mask areapsnrAnd Lssim(ii) a Wherein L ispsnrBy
Obtaining that k is the side length of the input picture; l isssimBy
The subscripts a and b are obtained as mask regions, μaAnd mubIs a value calculated by a and b from a Gaussian filter, σaIs the standard deviation of a, σbIs the standard deviation of b, σabIs the covariance of a and b; to prevent the denominator and numerator from being zero, M1And M2Taking the smaller value of 0.0001 and 0.0009; at fixed position D1And D2When the parameters of (1) are used, L is usedadv1、LpsnrAnd LssimCounter propagating pair G1Updating parameters; having a loss function of
Wherein λpsnrAnd λadv11.16 and 0.0003 respectively, in the fixed position G1When the parameters of (1) are D1,2To judge the probability that x is a real picture, i.e. calculating the judgment error by BCELoss and back-propagating, for D1And D2Updating parameters; by D1,2To judge the probability that x' is false picture, i.e. using BCELoss to calculate the judgment error and back-propagate, for D1And D2Updating parameters;
in step S6, y is passed through G2Forward propagation to obtain a prediction result y', specifically:
in each iteration training of the perceptual deblurring stage, a pair of pictures (x, y) is sampled, wherein y ═ x [ (1-m) + x' ] m, and y is used as G2The input of (1); y passes through G2After forward propagation, the result y' is obtained, using D3To distinguish y 'as a true picture, i.e. the probability that y' passes through D3Calculating L by the difference between the result after forward propagation and Trueadv2(ii) a Therefore Ladv2＝-D3(G2(y)); calculating the perceptual loss L by the error of y' and xperTo obtain
Wherein
Is a characteristic diagram obtained after passing through the jth convolutional layer and activation function of VGG19 network, wherein WjAnd HjThe dimension of the j-th layer feature graph is obtained, and the VGG19 network is trained on an Imagenet data set in advance; at fixed position D3When the parameters of (1) are used, L is usedperAnd Ladv2Counter propagating pair G2Performing parameter update, G2Loss function of
Wherein λ isadv2Is 0.01; in the fixingLive G2When the parameters of (1) are D3To judge the probability that x is a real picture, namely calculating the judgment error back propagation by using the difference value, and to D3Updating parameters; by D3To determine the probability that y' is a false picture, i.e. to calculate the judgment error back propagation by using the difference value, for D3And updating the parameters.
As a further improvement, in step S2, a coarse reconstruction stage generator G in the depth human face image restoration method based on perceptual deblurring is constructed1Global arbiter D1And a local discriminator D2Constructing an ADADELTA optimizer; the method specifically comprises the following steps:
G1the coarse reconstruction generator is a deep neural network based on a convolutional neural network and a generation countermeasure network; in the coarse reconstruction stage, the features of the whole face picture are extracted by adopting a convolution layer and a cavity convolution layer with rich receptive fields; and recovering the characteristic diagram with low resolution to the original resolution by using the deconvolution layer; g1A total of 17 layers, 11 convolutional layers, 4 void convolutional layers and 2 deconvolution layers; the first 6 layers are convolution layers, the output characteristic diagram of the 3 rd layer is one half of the resolution of the original image, and the output characteristic diagram of the 5 th layer is one fourth of the resolution of the original image; the 7 th to 10 th layers are void convolution layers, and the void coefficients are respectively 2, 4, 8 and 16; layers 11 and 12 are convolutional layers; the 13 th layer is a deconvolution layer, and the resolution of the output feature map is twice of that of the input feature map; the 14 th layer is a convolution layer; the 13 th layer is a deconvolution layer, the resolution of the output feature map is twice of that of the input feature map, and the output resolution of the 13 th layer is the same as that of the input picture; layers 16 and 17 are convolution layers, and the output of the layer 17 is a coarse reconstruction result x' with three channels and the same resolution as the input picture; except the network layer with the changed resolution of the output characteristic diagram, the output resolution of other network layers is not changed; after the operation of the first 16 layers in each layer, the BatchNormailization normalization method and the ReLU activation function are used; layer 17 uses Sigmoid activation function after operation;
D1is a global discriminator, is a convolution neural network and is used for discriminating in a global angleThe probability that the picture x' is a real picture is generated at the other stage; d1The total number of the layers is 6, wherein 5 convolution layers and 1 full-connection layer are provided; the output characteristic resolution of the first 4 layers of convolution layers is one half of the input resolution; the output resolution of the 5 th convolution layer is unchanged; after each of the 5 convolutional layers was operated, the BatchNormailization normalization method and the ReLU activation function were used;
D2the local discriminator is a convolution neural network and is used for discriminating the probability that the generated picture x' is a real picture at a local angle in one stage; d2The total number of the layers is 5, wherein 4 convolution layers and 1 full-connection layer are arranged; the output characteristic resolution of the first 3 layers of convolution layers is one half of the input resolution; the output resolution of the 4 th convolution layer is unchanged; after each of the 4 convolutional layers was operated, the BatchNormailization normalization method and the ReLU activation function were used; layer 5 output feature map and D1The 6 th layer of the graph is used for outputting a characteristic diagram, a connection operation is carried out to form a full connection layer, and a ReLU activation function is used after the full connection layer;
in the first stage, an ADADELTA optimizer is used, so that the rapid fitting of the network is facilitated; in the initialization parameters, the learning rate is 1.0, the coefficient rho for calculating the running average is 0.9, the term epsilon added to the denominator to improve the numerical stability is 1e-6, and the weight attenuation coefficient is 0.
As a further improvement, in step S5, a perception deblurring stage generator G in the depth human face image restoration method based on perception deblurring is constructed2And a discriminator D3And constructing an Adam optimizer, specifically comprising the following steps:
G2the perception deblurring generation network is a deep neural network based on CNN and GAN; in a perception deblurring stage, a reconstructed picture with a roughly reconstructed structure needs to be subjected to feature extraction by adopting a neural network with residual connection, and a feature map with low resolution is restored to the original resolution by using an deconvolution layer; g2A total of 13 layers, 11 convolutional layers and 2 anti-convolutional layers; the first 3 layers are convolution layers, and the output resolutions of the 2 nd layer and the 3 rd layer are half and one fourth of a real picture respectively; first, theThe 1 residual block contains the 4 th and 5 th convolutional layers, the 2 nd residual block contains the 6 th and 7 th convolutional layers, and the 3 rd residual block contains the 8 th and 9 th convolutional layers; the residual blocks all contain residual links; the 10 th layer and the 11 th layer are deconvolution layers, the resolution of an output feature map of each layer is twice of that of the input, and the output resolution of the 11 th layer is the same as that of the input picture; the 12 th layer is a convolution layer, and the output of the 12 th layer is a perception deblurring result y' with three channels and the same resolution as the input picture; except the network layer with the changed resolution of the output characteristic diagram, the output resolution of other network layers is not changed; after the operation of each layer of the first 11 layers, the BatchNormailization normalization method and the ReLU activation function are used; after the operation of layer 12, a Tanh activation function is used;
D3the perceptive deblurring discriminator is a convolution neural network and is used for discriminating the probability that the two-stage prediction y' is a real picture; d3A total of 5 convolutional layers; the resolution of the output characteristic graphs of the 5 convolutional layers is one half of the input resolution, and finally, the discrimination probability is obtained through a Sigmoid activation function; after each layer of the first 4 convolutional layers was operated, the BatchNormailization normalization method and LeakyReLU activation function were used; after the layer 5 operation, a Sigmoid activation function is used;
in the second stage, an Adam optimizer is used to accelerate network convergence; in the initialization parameters, the learning rate is 1e-3, the coefficient beta for calculating the running average value and the square of the gradient is (0.9,0.999), the term epsilon added to the denominator to improve the numerical stability is 1e-8, and the weight attenuation coefficient is 0.
As a further improvement, in step S8, the model is judged and the optimal model is saved through qualitative and quantitative tests on z, specifically:
sampling from the test set to obtain a batch of real pictures, cutting and scaling the real pictures to achieve the target resolution, hollowing out the area to be repaired through a single-channel mask for representing the area to be repaired in RGB three channels, and normalizing the three channels to be used as input; input via G1And G2After forward propagation in sequence, the result is completeRepairing the result; repeating the operation until the whole test set is sampled without repetition; the qualitative test is to visually judge all finally obtained complete restoration results, including whether the contents in the picture restoration area and at the boundary are consistent or not and whether the color, the brightness and the contrast of the picture are consistent or not; the quantitative test is to calculate the PSNR value and the SSIM value of all finally obtained complete restoration results and corresponding original images, and the higher the value is, the better the value is; when the steps S2 to S6 are repeated, a plurality of trained models can be obtained; and performing qualitative test and quantitative test on different models, and selecting a final model with excellent effects of the qualitative test and the quantitative test, wherein the model with better qualitative test effect should be selected preferentially.
Compared with the prior art, the depth face image restoration method based on perception deblurring provided by the invention has the following beneficial effects:
the invention provides a depth face image restoration method based on perception deblurring, which uses a two-stage training mode, wherein one stage is used for finishing rough reconstruction by combining a neural network with large receptive field cavity convolution and information of a known region and a region to be restored, and generating a semi-finished product. And in the second stage, the neural network with a perception deblurring method is used for carrying out detail refinement on the semi-finished product generated in the first stage to generate a completely repaired picture. When the face picture with missing information is given, the model can quickly give a prediction result. The prediction result has good visual effect, has semantic information of the human face, and keeps continuous content in the result and at the boundary. The human face image restoration model provided by the invention has good robustness on the input image of the human face image with large internal color difference of the front face, the side face, the makeup and the like, and has good value and prospect in practical application.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is within the scope of the present invention for those skilled in the art to obtain other drawings based on the drawings without inventive exercise.
Fig. 1 is a flowchart of a depth face image restoration method based on perceptual deblurring according to an embodiment of the present invention;
fig. 2 is an architecture diagram of a depth human face image repairing method based on perceptual deblurring according to an embodiment of the present invention.
Detailed Description
For completeness and clarity of description of technical solutions in the embodiments of the present invention, the following detailed description will be further developed with reference to the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
As shown in fig. 1, the present invention provides a depth face image repairing method based on perception deblurring, which comprises the following steps:
in step S1, a face image inpainting data set is given and divided into a training set and a test set with a ratio of 9: 1.
In step S2, a coarse reconstruction stage generator G in the depth face image restoration method based on perception deblurring is constructed1Global arbiter D1And a local discriminator D2. An ADADELTA optimizer was constructed. The method specifically comprises the following steps:
g in the deep face image restoration method training flow chart based on perception deblurring given in FIG. 21The coarse reconstruction generator is a deep neural network based on a Convolutional Neural Network (CNN) and a generation countermeasure network (GAN). In the coarse reconstruction stage, the features of the whole face picture need to be extracted by using the convolution layer and the cavity convolution layer with rich receptive fields. And recovering the characteristic diagram with low resolution to the original resolution by using the deconvolution layer. G1A total of 17 layers were included, of which there were 11 convolutional layers, 4 void convolutional layers and 2 deconvolution layers. The first 6 layers are convolutional layers, the output characteristic map of the 3 rd layer is one half of the resolution of the original, and the output characteristic map of the 5 th layer is one quarter of the resolution of the original. The 7 th to 10 th layers are void convolution layers, and the void coefficients are 2, 4, 8 and 16, respectively.Layers 11 and 12 are convolutional layers. Layer 13 is a deconvolution layer, and the resolution of the output feature map is twice that of the input. The 14 th layer is a convolutional layer. The 13 th layer is a deconvolution layer, the resolution of the output feature map is twice that of the input, and the output resolution of the 13 th layer is the same as the input picture resolution. Layers 16 and 17 are convolutional layers, and the output of layer 17 is a coarse reconstruction result x' with three channels and the same resolution as the input picture. Except the network layer with the changed resolution of the output characteristic diagram, the output resolution of the other network layers is not changed. After the first 16 layers have been operated on each layer, the BatchNormailification normalization method and the ReLU activation function are used. Layer 17 uses the Sigmoid activation function after operation. Table 1 below shows G in the depth face image restoration method based on perceptual deblurring according to the present invention1Network detail parameters.
Table 1: g in depth face image restoration method based on perception deblurring1Network detail parameters
D1The global discriminator is a convolutional neural network and is used for discriminating the probability that the generated picture x' is a real picture in a global angle. D1The total of 6 layers were included, of which there were 5 convolutional layers and 1 fully-connected layer. The output characteristic resolution of the first 4 convolutional layers is half of the input resolution. The 5 th convolutional layer output resolution is unchanged. After each of the 5 convolutional layers was operated, the BatchNormailization normalization method and the ReLU activation function were used. As shown in the following Table 2, D in the depth human face image restoration method based on perception deblurring of the invention1Network detail parameters.
Table 2: depth face image restoration method based on perception deblurring1Network detail parameters
D2The local discriminator is a convolution neural network and is used for generating the probability that the picture x' is a real picture at a local angle discrimination stage. D2The total of 5 layers were included, of which there were 4 convolutional layers and 1 fully-connected layer. The output characteristic resolution of the first 3 convolutional layers is half of the input resolution. The 5 th convolutional layer output resolution is unchanged. After each of the 4 convolutional layers was operated, the BatchNormailization normalization method and the ReLU activation function were used. Layer 5 output feature map and D1The layer 6 of (2) outputs a feature map, a join operation is performed to form a fully connected layer, and a ReLU activation function is used after the fully connected layer. As shown in the following Table 3, D in the depth human face image restoration method based on perception deblurring of the present invention2Network detail parameters.
Table 3: depth face image restoration method based on perception deblurring2Network detail parameters
The ADADELTA is an extension of the Adagarad algorithm, aims to solve the problem that the learning rate in Adagarad is continuously reduced, and can be adaptive to the learning rate. One stage uses an ADADELTA optimizer, facilitating fast fitting of the network. In the initialization parameters, the learning rate is 1.0, the coefficient rho for calculating the running average is 0.9, the term epsilon added to the denominator to improve the numerical stability is 1e-6, and the weight attenuation coefficient is 0.
In step S3, in each iteration training of the coarse reconstruction stage, sampling a picture x in the training set, randomly generating a binary mask m in which the value of the region to be repaired is 1 and the value of the remaining region is 0, and taking x [ (1-m) ] as G1And inputting one iteration of training. As a multiplication operation of corresponding elements in the matrix. D1,2Is D1And D2And finally, connecting the characteristic layers. Input via G1Forward propagation to obtain a semi-finished product x', using D1,2Calculating L according to the probability of judging x' as a real pictureadv1And calculating L by using the error between x' and xpsnrAnd Lssim. At fixed position D1And D2When the parameters of (1) are used, L is usedadv1、LpsnrAnd LssimCounter propagating pair G1And updating the parameters. At the fixed position G1When the parameters of (1) are D1,2To judge the probability that x is a real picture, the probability is calculated by BCELoss, the judgment error is propagated reversely, and the pair D1And D2Updating parameters; by D1,2To judge the probability that x' is false, the judgment error is calculated by BCELoss and propagated reversely for D1And D2And updating the parameters. The method specifically comprises the following steps:
a batch of original data is obtained by sampling from a training set, and each real picture x in the original data comprises three RGB channels. The picture is preprocessed, for example, if the target resolution of the present invention is 256 × 256, the original image is randomly cropped and scaled to reach the target resolution. And then randomly generating a single-channel mask m, wherein the method is to initialize a single-channel two-dimensional matrix with the same size as the original image, the value of the matrix in the area to be repaired is set to be 1, and the value of the matrix in the rest areas is set to be 0. For example, if the area to be repaired is a rectangular area of the original image with a size of one fourth, a coordinate is randomly selected as an upper left corner in m, and the upper right corner, the lower left corner and the lower right corner of the area to be repaired are formed according to the coordinate. When the coordinate position of the upper left corner is randomly selected, the lower right corner is ensured not to exceed the range of the original image, the area to be repaired meeting the requirement can be constructed, the value of m in the rectangular area is set to be 1, and the value of m outside the rectangular area is set to be 0. Take x [ -1-m ] as input and normalize it to keep all pixel values [ -1,1 [ ]]In the meantime. Input via G1X' is obtained after forward propagation. By D1,2To determine the probability that x 'is a true picture, i.e. x' passes through D1,2After forward propagation, L is calculated using the result and the BCELoss value of Trueadv1. Therefore Ladv1＝-log[D1,2(G1(x,m))]. Calculating L through the error of x' and x in the mask areapsnrAnd Lssim. Wherein L ispsnrBy
Obtained, k is the side length of the input picture. L isssimBy
The subscripts a and b are obtained as mask regions, μaAnd mubIs a value calculated by a and b from a Gaussian filter, σaIs the standard deviation of a, σbIs the standard deviation of b, σabIs the covariance of a and b. To prevent the denominator and numerator from being zero, M1And M2Taking the smaller value of 0.0001 and 0.0009. At fixed position D1And D2When the parameters of (1) are used, L is usedadv1、LpsnrAnd LssimCounter propagating pair G1And updating the parameters. Loss function thereof
Where lambda ispsnrAnd λadv11.16 and 0.0003 respectively, in the fixed position G1When the parameters of (1) are D1,2To judge the probability that x is a real picture, i.e. calculating the judgment error by BCELoss and back-propagating, for D1And D2Updating parameters; by D1,2To judge the probability that x' is false picture, i.e. using BCELoss to calculate the judgment error and back-propagate, for D1And D2And updating the parameters.
In step S4, in G1And D1,2In the alternate updating of parameters, G1Go on and off to generate more realistic samples, D1,2Then, the method tries to judge whether the generated sample is true or false, and the two samples are played mutually until Nash balance is achieved, namely, the step S3 is repeated until one-stage training is finished, specifically:
first, the preliminary training is carried out to G1Performing several iterative training times, i.e. oneUpdate alone G1Several times. To D1,2Performing several iterative trainings, i.e. updating D separately1,2Several times. And then both are trained together, i.e. alternately updating G at the same time1And D1,2And (4) parameters. Wherein G is updated1While parameter, keep D1,2Fixing parameters; to update D1,2When parameters are satisfied, maintain G1And (5) fixing the parameters.
In step S5, a perception deblurring stage generator G in the depth human face image restoration method based on perception deblurring is constructed2And a discriminator D3. Constructing an Adam optimizer, specifically comprising the following steps:
g in the deep face image restoration method training flow chart based on perception deblurring given in FIG. 22To perceive the deblurring generator, it is a deep neural network based on CNN and GAN. In the stage of perception deblurring, a reconstructed picture with a roughly reconstructed structure needs to be subjected to feature extraction by adopting a neural network with residual connection, and a feature map with low resolution is restored to the original resolution by using an deconvolution layer. G2The total of 13 layers were included, 11 convolutional layers and 2 anti-convolutional layers. The first 3 layers are convolutional layers, and the output resolutions of the 2 nd and 3 rd layers are half and quarter of the real picture, respectively. The 1 st residual block contains the 4 th and 5 th convolutional layers, the 2 nd residual block contains the 6 th and 7 th convolutional layers, and the 3 rd residual block contains the 8 th and 9 th convolutional layers. The residual blocks each contain a residual link. The 10 th and 11 th layers are deconvolution layers, the output feature map resolution of each layer is twice that of the input, and the output resolution of the 11 th layer is the same as the input picture resolution. Layer 12 is a convolutional layer, and the output of layer 12 is a perceptual deblurring result y' with three channels and the same resolution as the input picture. Except the network layer with the changed resolution of the output characteristic diagram, the output resolution of the other network layers is not changed. After the operation of each of the first 11 layers, the BatchNormailization normalization method and the ReLU activation function were used. After layer 12 operation, the Tanh activation function is used. As shown in Table 4 below, G is the depth human face image restoration method based on perception deblurring in the invention2Network detail parameters.
Table 4: g in depth face image restoration method based on perception deblurring2Network detail parameters
D3The perceptive deblurring discriminator is a convolution neural network and is used for discriminating the probability that two-stage prediction y' is a real picture. D3A total of 5 convolutional layers were included. The resolution of the output characteristic graphs of the 5 convolutional layers is one half of the input resolution, and finally, the discrimination probability is obtained through a Sigmoid activation function. After each of the first 4 convolutional layers was operated, the BatchNormailization normalization method and LeakyReLU activation function were used. After layer 5 operations, the Sigmoid activation function is used. As shown in the following Table 5, D in the depth human face image restoration method based on perception deblurring of the present invention3Network detail parameters.
TABLE 5 perception deblurring-based depth face image restoration method D3Network detail parameters
Adam is an effective random optimization method, and the adaptive learning rates of different parameters are calculated through the estimation of the first gradient and the second gradient. Adam optimizer is used in the second stage, and network convergence is accelerated. In the initialization parameters, the learning rate is 1e-3, the coefficient beta for calculating the running average value and the square of the gradient is (0.9,0.999), the term epsilon added to the denominator to improve the numerical stability is 1e-8, and the weight attenuation coefficient is 0.
In step S6, a pair of pictures (x, y) is sampled in each iteration of the perceptual deblurring stage, wherein y ═ x [ (1-m) + x' ] m, andwith y as G2Through G2Forward propagation yields the result y', with D3To distinguish y 'as a true picture, i.e. the probability that y' passes through D3Calculating L by the difference between the result after forward propagation and Trueadv2. Calculating L by using the error between y' and xper. At fixed position D3When the parameters of (1) are used, L is usedadv2And LperCounter propagating pair G2And updating the parameters. At the fixed position G2When the parameters of (1) are D3To judge the probability that x is a real picture, namely calculating the judgment error back propagation by using the difference value, and to D3Updating parameters; by D3To judge y' as false picture probability, i.e. to calculate the judgment error back propagation by difference value, for D3Updating parameters, specifically:
calculating y ═ x ∑ m) + x '<' > m, sampling a pair of pictures (x, y), and using y as G2Is input. y passes through G2After forward propagation, the result y' is obtained, using D3To distinguish y 'as a true picture, i.e. the probability that y' passes through D3Calculating L by the difference between the result after forward propagation and Trueadv2. Therefore Ladv2＝-D3(G2(y)). Calculating L by the error of y' with xperTo obtain
Wherein
Is a characteristic diagram obtained after passing through the jth convolutional layer and activation function of VGG19 network, wherein WjAnd HjAnd the dimension of the j-th layer feature graph is obtained, and the VGG19 network is trained on an Imagenet data set in advance. At fixed position D3When the parameters of (1) are used, L is usedperAnd Ladv2Counter propagating pair G2Performing parameter update, G2Loss function of
Wherein λ isadv2Is 0.01. At the fixed position G2When the parameters of (1) are D3To judge the probability that x is a real picture, namely calculating the judgment error back propagation by using the difference value, and to D3Updating parameters; by D3To determine the probability that y' is a false picture, i.e. to calculate the judgment error back propagation by using the difference value, for D3And updating the parameters.
In step S7, in G2And D3In the alternate updating of parameters, G2Go on and off to generate more realistic samples, D3Then, the method tries to judge whether the generated sample is true or false, and the two samples are played mutually until Nash balance is achieved, namely, the step S6 is repeated until the two-stage training is finished, specifically:
both training together, i.e. alternately updating G at the same time2And D3And (4) parameters. Wherein G is updated2While parameter, keep D3Fixing parameters; to update D3When parameters are satisfied, maintain G2And (5) fixing the parameters.
In step S8, a picture of the test set is selected, and the picture passes through G1And G2And carrying out forward propagation in sequence to obtain a prediction result. By calculating z ═ x ÷ (1-m) + y '<' > m, the final prediction z is obtained. Judging the model and storing the optimal model through qualitative test and quantitative test of z, which specifically comprises the following steps:
and (3) obtaining a batch of real pictures from the test set, cutting and scaling the pictures to achieve the target resolution, hollowing the area to be repaired through a single-channel mask for representing the area to be repaired in RGB three channels, and normalizing the pictures to be used as input. Input via G1And G2And after forward propagation in sequence, a complete repair result is obtained. And repeating the operation until the whole test set is sampled without repetition. The qualitative test is to visually judge all the finally obtained complete restoration results, including whether the contents inside and at the boundary of the picture restoration area are consistent or not, and whether the color, the brightness and the contrast of the picture are consistent or not. Quantitative test is to obtain all the final complete repair results and correspondencesThe PSNR value and the SSIM value of the original image are calculated, and the higher the PSNR value and the higher the SSIM value, the better the PSNR value and the SSIM value should be. When the steps S2 to S6 are repeated, a plurality of trained models can be obtained. And performing qualitative test and quantitative test on different models, and selecting a final model with excellent effects of the qualitative test and the quantitative test, wherein the model with better qualitative test effect should be selected preferentially.
In summary, the present invention provides a depth face image restoration method based on perception deblurring, which uses a two-stage approach. And in the first stage, the neural network with the large receptive field cavity convolution is combined with the information of the known region and the region to be repaired, so that rough reconstruction is completed, and a semi-finished product is generated. And in the second stage, the neural network with a perception deblurring method is used for carrying out detail refinement on the semi-finished product generated in the first stage to generate a completely repaired picture. When a human face picture with missing information is given, the model provided by the invention can quickly give a prediction result. The prediction result has good visual effect, has semantic information of the human face, and keeps continuous content in the result and at the boundary. The human face image restoration model provided by the invention has good robustness on the input image of the human face image with large internal color difference of the front face, the side face, the makeup and the like, and has good value and prospect in practical application.
It will be appreciated by persons skilled in the art that the invention is not limited to details of the foregoing embodiments, and that the invention can be embodied in other specific forms without departing from the spirit or scope of the invention. In addition, various modifications and alterations of this invention may be made by those skilled in the art without departing from the spirit and scope of this invention, and such modifications and alterations should also be viewed as being within the scope of this invention.
Claims (4)
1. A depth face image restoration method based on perception deblurring is characterized by comprising the following steps:
step S1: a facial image restoration data set is given and divided into a training set and a test set;
step S2: coarse reconstruction stage generator G in deep face image restoration method based on perception deblurring construction1Global arbiter D1And a local discriminator D2Constructing an ADADELTA optimizer;
step S3: in each iteration training of the coarse reconstruction stage, sampling a picture x in the training set, randomly generating a binary mask m, wherein the value of a region to be repaired in m is 1, the value of the rest regions is 0, and taking x (1-m) as G1Inputting an iterative training; an operation for multiplying corresponding elements in the matrix; d1，2Is D1And D2Connecting the last characteristic layer; input via G1Forward propagation to obtain a semi-finished product x', using D1，2Calculating L according to the probability of judging x' as a real pictureadv1And calculating L by using the error between x' and xpsnrAnd Lssim(ii) a At fixed position D1And D2When the parameters of (1) are used, L is usedadv1、LpsnrAnd LssimCounter propagating pair G1Updating parameters; at the fixed position G1When the parameters of (1) are D1，2To judge the probability that x is a real picture, the probability is calculated by BCELoss, the judgment error is propagated reversely, and the pair D1And D2Updating parameters; by D1，2The probability that x' is judged to be a false picture is calculated by using BCELoss, judgment error is calculated and propagated backwards, and the pair D1And D2Updating parameters;
step S4: at G1And D1，2In the alternate updating of parameters, G1Go on and off to generate more realistic samples, D1，2Then, the method tries to judge whether the generated sample is real or false, and the two samples are played mutually until Nash balance is achieved, namely, the step S3 is repeated until one-stage training is finished;
step S5: method for constructing generator G in perception deblurring stage in depth face image restoration method based on perception deblurring2And a discriminator D3(ii) a Constructing an Adam optimizer;
step S6: in each iteration training of the perceptual deblurring stage, a pair of pictures (x, y) is sampled, wherein y ═ x [ (1-m) + x' ] m, and y is used as G2Through G2Forward propagation yields the result y', with D3Probability instant use for judging y' as real picturey' passing through D3Calculating L by the difference between the result after forward propagation and Trueadv2(ii) a Calculating L by using the error between y' and xper(ii) a At fixed position D3When the parameters of (1) are used, L is usedadv2And LperCounter propagating pair G2Updating parameters; at the fixed position G2When the parameters of (1) are D3To judge the probability that x is a real picture, namely calculating the judgment error back propagation by using the difference value, and to D3Updating parameters; by D3To judge y' as false picture probability, i.e. to calculate the judgment error back propagation by difference value, for D3Updating parameters;
step S7: at G2And D3In the alternate updating of parameters, G2Go on and off to generate more realistic samples, D3The method tries to judge whether the generated sample is real or false, and the two samples are played mutually until Nash balance is achieved, namely the step S6 is repeated until the two-stage training is finished;
step S8: in the testing stage, selecting the picture of the test set, the picture passing through G1And G2Forward propagating according to the sequence to obtain a prediction result; by calculating z ═ x ÷ y '<' > m, a final prediction z is obtained; judging the model and storing the optimal model through qualitative test and quantitative test of z;
in the step S3, the input is via G1Forward propagation yields x', specifically:
sampling from a training set to obtain a batch of original data, wherein each real picture x in the original data comprises three RGB channels; preprocessing a picture; then randomly generating a single-channel mask m, wherein the method comprises the steps of initializing a single-channel two-dimensional matrix with the same size as the original image, setting the value of the matrix in a region to be repaired to be 1, and setting the value of the matrix in the rest regions to be 0; then, take x [ -1-m ] as input and normalize it to keep all pixel values [ -1,1 [ ]]To (c) to (d); input via G1Obtaining x' after forward propagation; by D1，2To determine the probability that x 'is a true picture, i.e. x' passes through D1，2After forward propagation, L is calculated using the result and the BCELoss value of Trueadv1(ii) a The calculation method comprises the following steps:
Ladv1＝-log[D1，2(G1(x，m))]
meanwhile, calculating L through the error of x' and x in the mask areapsnrAnd Lssim(ii) a Wherein L ispsnrBy
Obtaining that k is the side length of the input picture; l isssimBy
The subscripts a and b are obtained as mask regions, μaAnd mubIs a value calculated by a and b from a Gaussian filter, σaIs the standard deviation of a, σbIs the standard deviation of b, σabIs the covariance of a and b; to prevent the denominator and numerator from being zero, M1And M2Taking the smaller value of 0.0001 and 0.0009; at fixed position D1And D2When the parameters of (1) are used, L is usedadv1、LpsnrAnd LssimCounter propagating pair G1Updating parameters; having a loss function of
Wherein λpsnrAnd λadv11.16 and 0.0003 respectively, in the fixed position G1When the parameters of (1) are D1，2To judge the probability that x is a real picture, i.e. calculating the judgment error by BCELoss and back-propagating, for D1And D2Updating parameters; by D1，2To judge the probability that x' is false picture, i.e. using BCELoss to calculate the judgment error and back-propagate, for D1And D2Updating parameters;
in step S6, y is passed through G2Forward direction of rotationPropagating to obtain a prediction result y', specifically:
in each iteration training of the perceptual deblurring stage, a pair of pictures (x, y) is sampled, wherein y ═ x [ (1-m) + x' ] m, and y is used as G2The input of (1); y passes through G2After forward propagation, the result y' is obtained, using D3To distinguish y 'as a true picture, i.e. the probability that y' passes through D3Calculating L by the difference between the result after forward propagation and Trueadv2(ii) a Therefore Ladv2＝-D3(G2(y)); calculating the perceptual loss L by the error of y' and xperTo obtain
Wherein
Is a characteristic diagram obtained after passing through the jth convolutional layer and activation function of VGG19 network, wherein WjAnd HjThe dimension of the j-th layer feature graph is obtained, and the VGG19 network is trained on an Imagenet data set in advance; at fixed position D3When the parameters of (1) are used, L is usedperAnd Ladv2Counter propagating pair G2Performing parameter update, G2Loss function of
Wherein λ isadv2Is 0.01; at the fixed position G2When the parameters of (1) are D3To judge the probability that x is a real picture, namely calculating the judgment error back propagation by using the difference value, and to D3Updating parameters; by D3To determine the probability that y' is a false picture, i.e. to calculate the judgment error back propagation by using the difference value, for D3And updating the parameters.
2. The perceptual deblurring deep human face image inpainting method according to claim 1Wherein, in step S2, a coarse reconstruction stage generator G in the depth human face image restoration method based on perception deblurring is constructed1Global arbiter D1And a local discriminator D2Constructing an ADADELTA optimizer; the method specifically comprises the following steps:
G1the coarse reconstruction generator is a deep neural network based on a convolutional neural network and a generation countermeasure network; in the coarse reconstruction stage, the features of the whole face picture are extracted by adopting a convolution layer and a cavity convolution layer with rich receptive fields; and recovering the characteristic diagram with low resolution to the original resolution by using the deconvolution layer; g1A total of 17 layers, 11 convolutional layers, 4 void convolutional layers and 2 deconvolution layers; the first 6 layers are convolution layers, the output characteristic diagram of the 3 rd layer is one half of the resolution of the original image, and the output characteristic diagram of the 5 th layer is one fourth of the resolution of the original image; the 7 th to 10 th layers are void convolution layers, and the void coefficients are respectively 2, 4, 8 and 16; layers 11 and 12 are convolutional layers; the 13 th layer is a deconvolution layer, and the resolution of the output feature map is twice of that of the input feature map; the 14 th layer is a convolution layer; the 13 th layer is a deconvolution layer, the resolution of the output feature map is twice of that of the input feature map, and the output resolution of the 13 th layer is the same as that of the input picture; layers 16 and 17 are convolution layers, and the output of the layer 17 is a coarse reconstruction result x' with three channels and the same resolution as the input picture; except the network layer with the changed resolution of the output characteristic diagram, the output resolution of other network layers is not changed; after the operation of the first 16 layers in each layer, the BatchNormailization normalization method and the ReLU activation function are used; layer 17 uses Sigmoid activation function after operation;
D1the global discriminator is a convolutional neural network and is used for discriminating the probability that the generated picture x' is a real picture at a global angle in one stage; d1 contained 6 layers in total, of which there were 5 convolutional layers and 1 fully-connected layer; the output characteristic resolution of the first 4 layers of convolution layers is one half of the input resolution; the output resolution of the 5 th convolution layer is unchanged; after each of the 5 convolutional layers was operated, the BatchNormailization normalization method and the ReLU activation function were used;
D2the local discriminator is a convolution neural network and is used for discriminating the probability that the generated picture x' is a real picture at a local angle in one stage; d2The total number of the layers is 5, wherein 4 convolution layers and 1 full-connection layer are arranged; the output characteristic resolution of the first 3 layers of convolution layers is one half of the input resolution; the output resolution of the 4 th convolution layer is unchanged; after each of the 4 convolutional layers was operated, the BatchNormailization normalization method and the ReLU activation function were used; performing a connection operation between the layer 5 output characteristic diagram and the layer 6 output characteristic diagram of D1 to form a fully connected layer, wherein a ReLU activation function is used after the fully connected layer;
in the first stage, an ADADELTA optimizer is used, so that the rapid fitting of the network is facilitated; in the initialization parameters, the learning rate is 1.0, the coefficient rho for calculating the running average is 0.9, the term epsilon added to the denominator to improve the numerical stability is 1e-6, and the weight attenuation coefficient is 0.
3. The method for restoring a deep face image with perceptual deblurring as defined in claim 1, wherein in step S5, a perceptual deblurring stage generator G in the method for restoring a deep face image based on perceptual deblurring is constructed2And a discriminator D3And constructing an Adam optimizer, specifically comprising the following steps:
G2the perception deblurring generation network is a deep neural network based on CNN and GAN; in a perception deblurring stage, a reconstructed picture with a roughly reconstructed structure needs to be subjected to feature extraction by adopting a neural network with residual connection, and a feature map with low resolution is restored to the original resolution by using an deconvolution layer; g2A total of 13 layers, 11 convolutional layers and 2 anti-convolutional layers; the first 3 layers are convolution layers, and the output resolutions of the 2 nd layer and the 3 rd layer are half and one fourth of a real picture respectively; the 1 st residual block contains the 4 th and 5 th convolutional layers, the 2 nd residual block contains the 6 th and 7 th convolutional layers, and the 3 rd residual block contains the 8 th and 9 th convolutional layers; the residual blocks all contain residual links; layers 10 and 11 are deconvolution layers, and the resolution of the output feature map of each layer is outputTwice as large as the input, the output resolution of layer 11 is the same size as the input picture resolution; the 12 th layer is a convolution layer, and the output of the 12 th layer is a perception deblurring result y' with three channels and the same resolution as the input picture; except the network layer with the changed resolution of the output characteristic diagram, the output resolution of other network layers is not changed; after the operation of each layer of the first 11 layers, the BatchNormailization normalization method and the ReLU activation function are used; after the operation of layer 12, a Tanh activation function is used;
D3the perceptive deblurring discriminator is a convolution neural network and is used for discriminating the probability that the two-stage prediction y' is a real picture; d3A total of 5 convolutional layers; the resolution of the output characteristic graphs of the 5 convolutional layers is one half of the input resolution, and finally, the discrimination probability is obtained through a Sigmoid activation function; after each layer of the first 4 convolutional layers was operated, the BatchNormailization normalization method and LeakyReLU activation function were used; after the layer 5 operation, a Sigmoid activation function is used;
in the second stage, an Adam optimizer is used to accelerate network convergence; in the initialization parameters, the learning rate is 1e-3, the coefficient beta for calculating the running average value and the square of the gradient is (0.9,0.999), the term epsilon added to the denominator to improve the numerical stability is 1e-8, and the weight attenuation coefficient is 0.
4. The perceptual deblurring deep human face image restoration method according to claim 1, wherein in the step S8, the model is judged and the optimal model is saved through qualitative and quantitative tests on z, specifically:
sampling from the test set to obtain a batch of real pictures, cutting and scaling the real pictures to achieve the target resolution, hollowing out the area to be repaired through a single-channel mask for representing the area to be repaired in RGB three channels, and normalizing the three channels to be used as input; input via G1And G2After forward propagation in sequence, a complete restoration result is obtained; repeating the operation until the whole test set is sampled without repetition; the qualitative test is to look at all the complete repair results obtainedJudging perceptually, including whether the contents inside and at the boundary of the picture repairing area are consistent or not, and whether the color, the brightness and the contrast of the picture are consistent or not; the quantitative test is to calculate the PSNR value and the SSIM value of all finally obtained complete restoration results and corresponding original images, and the higher the value is, the better the value is; when the steps S2 to S6 are repeated, a plurality of trained models can be obtained; and performing qualitative test and quantitative test on different models, and selecting a final model with excellent effects of the qualitative test and the quantitative test, wherein the model with better qualitative test effect should be selected preferentially.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110408739.6A CN113160081A (en) | 2021-04-16 | 2021-04-16 | Depth face image restoration method based on perception deblurring |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110408739.6A CN113160081A (en) | 2021-04-16 | 2021-04-16 | Depth face image restoration method based on perception deblurring |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113160081A true CN113160081A (en) | 2021-07-23 |
Family
ID=76868098
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110408739.6A Withdrawn CN113160081A (en) | 2021-04-16 | 2021-04-16 | Depth face image restoration method based on perception deblurring |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113160081A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113592736A (en) * | 2021-07-27 | 2021-11-02 | 温州大学 | Semi-supervised image deblurring method based on fusion attention mechanism |
-
2021
- 2021-04-16 CN CN202110408739.6A patent/CN113160081A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113592736A (en) * | 2021-07-27 | 2021-11-02 | 温州大学 | Semi-supervised image deblurring method based on fusion attention mechanism |
| CN113592736B (en) * | 2021-07-27 | 2024-01-12 | 温州大学 | Semi-supervised image deblurring method based on fused attention mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113240580B (en) | Lightweight image super-resolution reconstruction method based on multi-dimensional knowledge distillation | |
| CN111784602B (en) | Method for generating countermeasure network for image restoration | |
| CN110363716B (en) | High-quality reconstruction method for generating confrontation network composite degraded image based on conditions | |
| CN112149459B (en) | Video saliency object detection model and system based on cross attention mechanism | |
| CN110175986B (en) | Stereo image visual saliency detection method based on convolutional neural network | |
| CN111429347A (en) | Image super-resolution reconstruction method and device and computer-readable storage medium | |
| CN110728219A (en) | 3D face generation method based on multi-column multi-scale graph convolution neural network | |
| CN112541864A (en) | Image restoration method based on multi-scale generation type confrontation network model | |
| CN111563418A (en) | Asymmetric multi-mode fusion significance detection method based on attention mechanism | |
| CN109005398B (en) | Stereo image parallax matching method based on convolutional neural network | |
| CN110675339A (en) | Image restoration method and system based on edge restoration and content restoration | |
| CN112801914A (en) | Two-stage image restoration method based on texture structure perception | |
| CN112949553A (en) | Face image restoration method based on self-attention cascade generation countermeasure network | |
| CN115049556A (en) | StyleGAN-based face image restoration method | |
| CN112149662A (en) | Multi-mode fusion significance detection method based on expansion volume block | |
| Chen et al. | MFMAM: Image inpainting via multi-scale feature module with attention module | |
| CN114549387A (en) | Face image highlight removal method based on pseudo label | |
| CN117197627B (en) | Multi-mode image fusion method based on high-order degradation model | |
| CN113160081A (en) | Depth face image restoration method based on perception deblurring | |
| CN116523985B (en) | Structure and texture feature guided double-encoder image restoration method | |
| CN115526891B (en) | Training method and related device for defect data set generation model | |
| CN116524307A (en) | Self-supervision pre-training method based on diffusion model | |
| CN114708586A (en) | Method for extracting three-dimensional face representation from image and video | |
| CN113298814A (en) | Indoor scene image processing method based on progressive guidance fusion complementary network | |
| Kumar et al. | Underwater Image Enhancement using deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210723 |