CN111986104B - Face image deblurring method based on deep learning - Google Patents
Face image deblurring method based on deep learning Download PDFInfo
- Publication number
- CN111986104B CN111986104B CN202010717179.8A CN202010717179A CN111986104B CN 111986104 B CN111986104 B CN 111986104B CN 202010717179 A CN202010717179 A CN 202010717179A CN 111986104 B CN111986104 B CN 111986104B
- Authority
- CN
- China
- Prior art keywords
- image
- fuzzy
- face image
- blurred
- clear
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000013135 deep learning Methods 0.000 title claims abstract description 11
- 230000008447 perception Effects 0.000 claims abstract description 5
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 8
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 abstract description 2
- 238000011160 research Methods 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Images
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Abstract
The invention discloses a face image deblurring method based on deep learning, and belongs to the technical field of image processing. The invention realizes the deblurring of the image with high quality by using the content encoder and the blur encoder to distinguish the content of the blurred image from the blur characteristics while keeping the content structure of the original image. Fuzzy branches and cycle consistency loss are added in the framework, further restoration is carried out through cycle consistency to ensure the robustness of the model, and meanwhile the added perception loss is beneficial to removing unrealistic artifacts from the blurred image and finally added countermeasure loss, so that the deblurring effect of the method is better and the speed is higher.
Description
Technical Field
The invention relates to a face image deblurring method based on deep learning, and belongs to the technical field of image processing.
Background
The subject of image deblurring has received much attention and there has been a long history of research not only because digital images have become an important way for people to acquire, exchange and understand information, but also because images contain a large amount of information and other signals are not comparable. The imaging system has inherent defects or is affected by various interferences in the shooting process to cause image blurring, which brings troubles to the application of people. Human faces have been receiving great attention from ancient times as the most discriminative part of the human body. Portrait and self portrait are popular subjects in pictorial work. After the era of mobile internet, various applications based on human faces are more endless, such as self-shooting, cartoon human face images, face recognition, automatic face makeup changing, style migration and the like. The applications bring great pleasure to people and provide great convenience for the life of people. A typical example is face recognition. After the face recognition based on deep learning reaches the recognition accuracy exceeding that of human eyes, the access control system using the face to perform identity authentication is more widely applied in actual life. Considering the natural sensitivity of human face, the blurring effect on the face image is much larger than that on the general image. For face images, generally the higher the sharpness, the better. Experiments show that although the general image deblurring algorithm has great success, the existing face deblurring algorithm has a considerable distance from practical application. The deblurring of the human face image is an important component in the deblurring of the image, and the deblurring of the human face image has wide application in life, such as video monitoring, human face recognition and the like.
The face image deblurring, also commonly called image deconvolution, is based on the mechanism of face image blur, and reconstructs a clear image similar to a real face from one or more degraded face images by means of certain image prior knowledge. Most conventional methods formulate the image deblurring task as a fuzzy kernel estimation problem, and in the past decade various natural images and prior kernels have been developed to normalize the solution space of potentially sharp images, including heavy-tailed gradient prior, sparse kernel prior, gradient prior, normalized sparsity and dark channels. However, these priors are estimated by limited observations and are not accurate enough. As a result, deblurred images are typically under-deblurred (the image is still blurred) or over-deblurred (the image contains many artifacts). Therefore, deblurring of a face image by a deep learning method is a popular research topic in the field of digital images.
Disclosure of Invention
The invention provides a face image deblurring method based on deep learning, which aims to improve the face image deblurring method.
The invention adopts the following technical scheme for solving the technical problems:
a face image deblurring method based on deep learning comprises the following steps:
(1) Selecting a clear face image and a fuzzy face image, extracting content information from the clear face image, and estimating fuzzy information from the fuzzy face image by using a fuzzy image content encoder;
(2) Normalizing fuzzy features Z by adding a KL divergence penalty b ＝E b (b) Such that the distribution of (A) is close to the normal distribution p (z) to N (0, 1);
(3) Sending the clear image content code and the fuzzy code into a fuzzy image generator to reconstruct to obtain a fuzzy face image;
(4) Sending the fuzzy image content code and the fuzzy code into a clear image generator to reconstruct to obtain a deblurred clear face image;
(5) Application of countermeasure loss in sharp face image and blurred face image
And
(6) Introducing cycle consistency loss, re-blurring the deblurred clear face image, and converting the blurred clear face image into an original clear image;
(7) Adding perception loss between the deblurred face image and the original blurred face image, and removing artifacts in the deblurred face image;
(8) The fuzzy image content encoder and the fuzzy encoder are used for extracting the content and the fuzzy characteristics of the test image, and the image generator of the clear image obtains output and then generates a deblurred face image.
The KL divergence loss in step (2) is defined as follows:
wherein Z is b For fuzzy features, Z is a feature matrix, p (Z) represents the probability distribution of Z, q (Z) b ) Represents Z b Approximate probability distribution, KL (q (Z) b ) P (z)) represents the expected value of the logarithmic difference of the probability of the original distribution and the approximate distribution; the specific KL divergence loss function is further written as:
wherein: mu.s i Represents Z b Is the average value of i, σ i Represents Z b Standard deviation at i, N is Z b Dimension of, Z b Sampling is Z b = μ + z ° σ, wherein: mu and sigma are Z b Average and standard deviation, ° represents the element-by-element multiplication.
Applying the clear face image in the step (5) to resist loss L DS Comprises the following steps:
wherein：D S A sharp image discriminator; e s～p(s) Is to encode a sharp image, E b～p(b) Is a blur encoding of a blurred image,
is the encoding of the content of a blurred image, G S () Is a sharp image generator;
applying the blurred face image domain to combat loss
Comprises the following steps:
wherein: d B () A blurred image discriminator;
is to encode the content of a sharp image, G B() Is a blurred image generator.
The cycle consistency loss in step (6) is defined as:
wherein: s denotes a sharp image, b denotes a blurred image,
the estimated amount of the signal s is represented,
the estimated amount of b is represented by,
to represent
The norm of the matrix of (a) is,
to represent
The matrix norm of (c).
The perception loss in the step (7) is as follows:
wherein:
representing a generated sharp image s b At the l-th layer of the pre-trained CNN,
features representing a true blurred image at the ith layer of the pre-trained CNN,
indicating that the euclidean distance between corresponding layers is calculated.
The invention has the following beneficial effects:
the method is used for deblurring the blurred face image, so that a clear face image which is better than the deblurring effect of other blurred face images can be obtained, and the deblurring efficiency is higher than that of the deblurring invention of other face images. The invention realizes the deblurring of the image with high quality by using the content encoder and the blur encoder to distinguish the content of the blurred image from the blur characteristics while keeping the content structure of the original image. Fuzzy branches and cycle consistency loss are added in a framework, further restoration is carried out through cycle consistency to ensure the robustness of the model, and meanwhile the added perception loss is beneficial to removing unrealistic artifacts from a fuzzy image and finally added countermeasure loss, so that the de-blurring effect of the invention is better and the speed is higher.
Drawings
FIG. 1 is a flow chart of the present invention.
FIG. 2 is a deblurring framework diagram of the present invention.
Fig. 3 (a) shows a blurred face image, and fig. 3 (b) shows the effect of deblurring the face image according to the present invention.
Detailed Description
The invention is described in further detail below with reference to the accompanying drawings.
As shown in fig. 1, it is a flowchart of the present invention, which extracts content information of a clear face image and a blurred face image, and reconstructs the blurred and clear face images according to the obtained image content information. And introducing cycle consistency loss, re-blurring the deblurred clear face image to reconstruct an original blurred image, and converting the blurred clear face image into an original clear image. And finally, generating the deblurred face image through a clear image generator.
As shown in fig. 2, which is the deblurring framework of the present invention, S is a sharp face image, b is a blurred face image,
content encoder that is a blurred image and
content encoder being a sharp image, E b Is a fuzzy encoder, G B Is an image generator of a blurred image and is G S An image generator for sharp images. Encoding sharp image content
And fuzzy coding E b Into the blurred image generator G B Reconstructing to obtain a fuzzy face image b S The sharp image to blurred image is to optimize the effect of the blur coding and the content coding of the blurred image. Finally, testing is carried out, and a fuzzy face image b is given t Blurred image content coding
And fuzzy coding E b Image generator G for extracting content and fuzzy characteristic and clear image S Obtaining output and then generating deblurred face image
As shown in fig. 3, (a) in fig. 3 is a blurred face image, and (b) in fig. 3 is an effect of deblurring a face image by using the present invention. The method ensures the robustness of the model, removes unrealistic artifacts in blurred images and finally added countermeasure loss, and ensures that the deblurring effect is better and the speed is higher. As can also be seen from fig. 3, the present invention can better deblur the blurred face image to obtain a clear face image.
Claims (5)
1. A face image deblurring method based on deep learning is characterized by comprising the following steps:
(1) Selecting a clear face image and a fuzzy face image, extracting content information from the clear face image, and estimating fuzzy information from the fuzzy face image by using a fuzzy image content encoder;
(2) Normalizing fuzzy features Z by adding a KL divergence penalty b ＝E b (b) Such that the distribution of (A) is close to the normal distribution p (z) to N (0, 1);
(3) Sending the clear image content code and the fuzzy code into a fuzzy image generator to reconstruct to obtain a fuzzy face image;
(4) Sending the fuzzy image content code and the fuzzy code into a clear image generator to reconstruct to obtain a deblurred clear face image;
(5) Applying counter-loss in sharp and blurred face images
And
(6) Introducing cycle consistency loss, re-blurring the deblurred clear face image, and simultaneously converting the blurred clear face image into an original clear image;
(7) Adding perception loss between the deblurred face image and the original blurred face image, and removing artifacts in the deblurred face image;
(8) The fuzzy image content encoder and the fuzzy encoder are used for extracting the content and the fuzzy characteristics of the test image, and the image generator of the clear image obtains output and then generates a deblurred face image.
2. The method according to claim 1, wherein the KL divergence loss in step (2) is defined as follows:
wherein, Z b For fuzzy features, Z is a feature matrix, p (Z) represents the probability distribution of Z, q (Z) b ) Represents Z b Approximate probability distribution, KL (q (Z) b ) | p (z)) represents the expected value of the logarithmic difference of the probability of the original distribution and the approximate distribution; the specific KL divergence loss function is further written as:
wherein: mu.s i Represents Z b Is the average value of i, σ i Represents Z b Standard deviation at i, N is Z b Dimension of (c), Z b Sampling is Z b = μ + z ° σ, wherein: mu and sigma are Z b Average and standard deviation, ° stands for element-by-element multiplication.
3. The method of claim 1 for deblurring human face images based on deep learningMethod, characterized in that in step (5) said sharp face image is applied against loss
Comprises the following steps:
wherein: d S A sharp image discriminator; e s～p(s) Is to encode a sharp image, E b～p(b) Is a blur encoding of a blurred image,
is the encoding of the content of a blurred image, G S () Is a sharp image generator;
the blurred face image is applied to resist loss
Comprises the following steps:
wherein: d B () A blurred image discriminator;
is to encode the content of a sharp image, G B() Is a blurred image generator.
4. The method for deblurring a human face image based on deep learning of claim 3, wherein the cycle consistency loss in the step (6) is defined as:
wherein: s denotes a sharp image, b denotes a blurred image,
the estimated amount of the signal s is represented,
the estimated amount of b is represented by,
to represent
The norm of the matrix of (a) is,
represent
The matrix norm of (d).
5. The method for deblurring a human face image based on deep learning of claim 1, wherein the perceptual loss in the step (7) is:
wherein:
representing a generated sharp image s b At the l-th layer of the pre-trained CNN,
features representing the true blurred image at layer i of the pre-trained CNN,
representing the calculation of Euclidean distances between corresponding layersAnd (5) separating.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010717179.8A CN111986104B (en) | 2020-07-23 | 2020-07-23 | Face image deblurring method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010717179.8A CN111986104B (en) | 2020-07-23 | 2020-07-23 | Face image deblurring method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111986104A CN111986104A (en) | 2020-11-24 |
| CN111986104B true CN111986104B (en) | 2023-02-28 |
Family
ID=73438983
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010717179.8A Active CN111986104B (en) | 2020-07-23 | 2020-07-23 | Face image deblurring method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111986104B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140307950A1 (en) * | 2013-04-13 | 2014-10-16 | Microsoft Corporation | Image deblurring |
| CN107730458A (en) * | 2017-09-05 | 2018-02-23 | 北京飞搜科技有限公司 | A kind of fuzzy facial reconstruction method and system based on production confrontation network |
| CN108573479A (en) * | 2018-04-16 | 2018-09-25 | 西安电子科技大学 | The facial image deblurring and restoration methods of confrontation type network are generated based on antithesis |
| CN110223245B (en) * | 2019-05-16 | 2021-07-16 | 华南理工大学 | Method and system for processing blurred picture in sharpening mode based on deep neural network |
-
2020
- 2020-07-23 CN CN202010717179.8A patent/CN111986104B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111986104A (en) | 2020-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Xu et al. | Learning to restore low-light images via decomposition-and-enhancement | |
| CN112233038B (en) | True image denoising method based on multi-scale fusion and edge enhancement | |
| CN111028177B (en) | Edge-based deep learning image motion blur removing method | |
| CN111028163B (en) | Combined image denoising and dim light enhancement method based on convolutional neural network | |
| CN110969589A (en) | Dynamic scene fuzzy image blind restoration method based on multi-stream attention countermeasure network | |
| CN110796622B (en) | Image bit enhancement method based on multi-layer characteristics of series neural network | |
| Anvari et al. | Dehaze-GLCGAN: unpaired single image de-hazing via adversarial training | |
| Li | Deep learning for image denoising | |
| Deora et al. | Structure preserving compressive sensing MRI reconstruction using generative adversarial networks | |
| CN111861894A (en) | Image motion blur removing method based on generating type countermeasure network | |
| CN113066034A (en) | Face image restoration method and device, restoration model, medium and equipment | |
| CN112614061A (en) | Low-illumination image brightness enhancement and super-resolution method based on double-channel coder-decoder | |
| CN114331886A (en) | Image deblurring method based on depth features | |
| CN114723630A (en) | Image deblurring method and system based on cavity double-residual multi-scale depth network | |
| Zhou et al. | Sparse representation with enhanced nonlocal self-similarity for image denoising | |
| Wu et al. | Multiple-operation image anti-forensics with WGAN-GP framework | |
| CN114202473A (en) | Image restoration method and device based on multi-scale features and attention mechanism | |
| CN116452469B (en) | Image defogging processing method and device based on deep learning | |
| CN117274059A (en) | Low-resolution image reconstruction method and system based on image coding-decoding | |
| CN111986104B (en) | Face image deblurring method based on deep learning | |
| Krishnan et al. | A novel underwater image enhancement technique using ResNet | |
| CN114565528A (en) | Remote sensing image noise reduction method and system based on multi-scale and attention mechanism | |
| EP3913572A1 (en) | Loss function for image reconstruction | |
| CN113902647A (en) | Image deblurring method based on double closed-loop network | |
| Jeyaprakash et al. | Linearly uncorrelated principal component and deep convolutional image deblurring for natural images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |