CN107633483A - The face image super-resolution method of illumination robustness - Google Patents
The face image super-resolution method of illumination robustness Download PDFInfo
- Publication number
- CN107633483A CN107633483A CN201710842317.3A CN201710842317A CN107633483A CN 107633483 A CN107633483 A CN 107633483A CN 201710842317 A CN201710842317 A CN 201710842317A CN 107633483 A CN107633483 A CN 107633483A
- Authority
- CN
- China
- Prior art keywords
- mrow
- resolution
- illumination
- face image
- msubsup
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005286 illumination Methods 0.000 title claims abstract description 95
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 27
- 239000011159 matrix material Substances 0.000 claims abstract description 25
- 239000013598 vector Substances 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 238000003786 synthesis reaction Methods 0.000 claims description 2
- 230000001815 facial effect Effects 0.000 abstract description 2
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000002194 synthesizing effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Landscapes
- Image Processing (AREA)
Abstract
A kind of face image super-resolution method of illumination robustness, comprises the following steps：Step 1: the low-resolution face image I that certain illumination f is inputtedfThe training set sample linear combination shone with same light；Step 2: solve the reconstruction coefficients W of linear combinationf, local covariance matrix X is diagonally loaded in solution procedure；Step 3: the low-resolution face image I by different illumination ccWith the training set sample linear combination of corresponding illumination, with reconstruction coefficients WfThe coefficient of linear combination is replaced, produces the low-resolution face image I of a variety of different illuminationc；Step 4: using the method for reconstructing based on overall facial image, by the low-resolution face image I of a variety of different illuminationcWeighting synthesizes the high-definition picture H of different illuminationc.Certain illumination low resolution input picture can be reconstructed into the high-resolution human face image of a variety of different illumination by the present invention when illumination changes.
Description
Technical Field
The invention relates to an image processing method, in particular to a face image super-resolution method with illumination robustness.
Background
The face image detected in video monitoring is usually a low-resolution image and is easily influenced by illumination change, the illumination change is an important factor influencing the performance analysis of the face image, most of the existing face super-resolution algorithms are only suitable for the condition that illumination is uniformly distributed or the same illumination is output, and when the illumination is changed, the effect of the face super-resolution method is weakened or even cannot be realized, so that the face super-resolution method with illumination robustness is very necessary to design.
According to the literature, a face super-resolution method related to illumination robustness is not found yet and is proposed.
Disclosure of Invention
The invention aims to provide a super-resolution method of a face image with illumination robustness, which can convert a certain illumination low-resolution face image into a plurality of high-resolution face images with different illumination when the illumination is changed, and further solve the influence of the illumination on a low-resolution face reconstruction result.
In order to achieve the purpose, the technical scheme adopted by the invention comprises the following steps:
step one, inputting a certain illumination f into a low-resolution face image IfLinearly combining training set samples with the same illumination;
step two, solving the linear combined reconstruction coefficient W in the step onefDiagonal loading is carried out on the local covariance matrix X in the solving process;
step three, the low-resolution face image I with different illumination ccLinearly combining the training set samples corresponding to the illumination and using the reconstruction coefficient W in the step twofSubstituting linear combined coefficient to generate low-resolution face image I with multiple different illuminationsc；
Step four, utilizing a reconstruction method based on the whole face image to reconstruct the low-resolution face image I with various different illuminations generated in the step threecWeighted synthesis of high-resolution images H of different illuminationc。
The human face image I with certain illumination f and low resolution input in the step onefAnd the training set samples are represented in the form of column vectors.
Solving the linear combination reconstruction coefficient W in the second stepfThe method comprises the following steps:
wherein,respectively representing a certain illumination fMth and kth low resolution training set face image samples, WfThe reconstruction coefficient representing a certain illumination f, N being the maximum of the number of training set samples, X being the local covariance matrix, X-1An inverse matrix of X is represented by,representing an inverse matrix X-1The m-th row of (a), the k-th column of (b),representing an inverse matrix X-1The element of row a and column b, XmkAn element representing the m-th row and the k-th column of the matrix X; a, b, k and m are all positive integers; t denotes transposition.
The method for diagonal loading of the local covariance matrix X in the second step comprises the following steps:
X＝X+λY
where λ is a constant whose value can be set experimentally and Y is an N × N unity diagonal matrix.
Step three, the low-resolution training set samples corresponding to different illumination cExpressed as column vector form, according to the approximate linear mapping relation existing between the redundant linear combination of certain illumination f and different illumination c, the low resolution face image IcThe calculation formula is as follows:
step four, the high-resolution images H with different illuminations are weighted and synthesized by using the reconstruction method based on the whole face imagecThe specific method comprises the following steps:
4.1) calculating the optimal weight Wc：
In the formula, WcRepresents each reconstruction weightThe N-dimensional weight column vector of (1);
n is the number of high-low resolution image pairs corresponding to different illumination c in the training set;
4.2) weighting and synthesizing high-resolution images H with different illumination by utilizing the similarity of the facial image structurescComprises the following steps:
whereinRepresenting high resolution training set samples corresponding to a plurality of different illuminations c.
Compared with the prior art, the invention has the following beneficial effects: the method comprises the steps of linearly combining low-resolution face images with different illuminations by using training set samples corresponding to the illuminations, replacing coefficients of the linear combination by reconstruction coefficients to generate low-resolution face images with various illuminations, weighting and synthesizing the low-resolution face images with various illuminations into high-resolution face images with different illuminations by using a reconstruction method based on the whole face image, and reconstructing a certain illumination low-resolution input image into high-resolution face images with various illuminations when the illuminations are changed. Experiments prove that the reconstruction effect is better. The invention overcomes the problem that the existing face hyper-resolution algorithm is only suitable for the condition of uniform illumination distribution, and has wide application prospect.
Drawings
FIG. 1 is a flow chart of a super resolution method of the present invention;
FIG. 2 is a graph of results of comparative experiments with conversion effects:
(a) inputting a low-resolution face image with certain illumination;
(b) - (f) is a graph of the results of the low resolution spatial illumination conversion of the present invention;
(g) - (k) is a graph of the results after treatment by the method of the invention;
(l) - (p) are real face images.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The super-resolution method of the face image with illumination robustness, disclosed by the invention, is shown in figure 1, and specifically comprises the following steps:
step 1: inputting a low-resolution face image I of certain illumination ffInputting image I by linear combination of training set samples with same illuminationfAnd the training set are both represented in the form of column vectors.
Step 2: solving the reconstruction coefficient W of the linear combination of step 1fDiagonal loading is carried out on the local covariance matrix X in the solving process;
solving for linear combined reconstruction coefficient WfThe process of (2) is as follows:
wherein,respectively representing the mth and kth low-resolution training set face image samples W of a certain illumination ffThe reconstruction coefficient representing a certain illumination f, N being the maximum of the number of training set samples, X being the local covariance matrix, X-1An inverse matrix of X is represented by,representing an inverse matrix X-1The m-th row of (a), the k-th column of (b),representing an inverse matrix X-1Row a, column b elements; xmkThe element representing the m-th row and k-th column of the matrix X. a, b, k and m are all positive integers; t denotes transposition.
Diagonal loading is carried out on the local covariance matrix X:
X＝X+λY
where λ is a constant whose value can be set experimentally and Y is an N × N unity diagonal matrix.
And step 3: the low-resolution face image I with different illumination ccLinearly combining the training set samples corresponding to the illumination and using the reconstruction coefficient W obtained in the step 2fReplacing the combination coefficient to generate a plurality of low-resolution face images I with different illuminationc；
The low-resolution training set samples corresponding to different illumination cRepresented in column vector form.
According to the redundant linear combination of the human face image under a certain illumination f and a plurality of different illuminations cThere exists approximate linear mapping relation between them, low resolution face image I of different illumination ccThe calculation formula is as follows:
and 4, step 4: for the different illumination low-resolution face image I generated in the step 3cWeighting and synthesizing different illumination high-resolution images H by using a reconstruction method based on the whole face imagecThe realization process is as follows:
firstly, the optimal weight W is calculated according to the following methodc：
W in the formulacRepresents each reconstruction weightN is the number of high and low resolution image pairs corresponding to different illuminations c in the training set. Obtaining the optimal weight WcThen, the high-resolution face image H with a plurality of different illuminations c is weighted and synthesized by utilizing the similarity of the face image structurecThe calculation formula is as follows:
whereinRepresenting high resolution training set samples corresponding to a plurality of different illuminations c.
Referring to fig. 2, the invention performs experiments on the face database of CMU PIE in the united states, compares the input low-resolution face image (a) with a certain illumination, and compares the result images (b) - (f) of the illumination conversion in the low-resolution space, the result images (g) - (k) processed by the super-resolution method of the invention and the real face images (l) - (p), and it can be seen from the images that when the illumination changes, the invention can not only generate a plurality of low-resolution face images with different illumination for one low-resolution face image in the low-resolution space, but also reconstruct high-resolution face images with higher quality under different illumination.
Claims (6)
1. A super-resolution method for a face image with illumination robustness is characterized by comprising the following steps:
step one, inputting a certain illumination f into a low-resolution face image IfLinearly combining training set samples with the same illumination;
step two, solving the linear combined reconstruction coefficient W in the step onefDiagonal loading is carried out on the local covariance matrix X in the solving process;
step three, the low-resolution face image I with different illumination ccTraining with corresponding lightingTraining the linear combination of samples, using the reconstruction coefficient W in step twofSubstituting linear combined coefficient to generate low-resolution face image I with multiple different illuminationsc；
Step four, utilizing a reconstruction method based on the whole face image to reconstruct the low-resolution face image I with various different illuminations generated in the step threecWeighted synthesis of high-resolution images H of different illuminationc。
2. The illumination-robust face image super-resolution method according to claim 1, wherein the low-resolution face image I inputted in the first stepfAnd the training set samples are represented in the form of column vectors.
3. The illumination robustness face image super-resolution method according to claim 1, wherein the second step of solving the linear combination reconstruction coefficient WfThe method comprises the following steps:
<mrow> <msub> <mi>W</mi> <mi>f</mi> </msub> <mo>=</mo> <mrow> <mo>(</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msubsup> <mi>X</mi> <mrow> <mi>m</mi> <mi>k</mi> </mrow> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>)</mo> </mrow> <mo>/</mo> <mrow> <mo>(</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>a</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <munderover> <mo>&Sigma;</mo> <mrow> <mi>b</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msubsup> <mi>X</mi> <mrow> <mi>a</mi> <mi>b</mi> </mrow> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>)</mo> </mrow> </mrow>
<mrow> <msub> <mi>X</mi> <mrow> <mi>m</mi> <mi>k</mi> </mrow> </msub> <mo>=</mo> <msup> <mrow> <mo>(</mo> <msub> <mi>I</mi> <mi>f</mi> </msub> <mo>-</mo> <msubsup> <mi>R</mi> <mi>f</mi> <mi>m</mi> </msubsup> <mo>)</mo> </mrow> <mi>T</mi> </msup> <mrow> <mo>(</mo> <msub> <mi>I</mi> <mi>f</mi> </msub> <mo>-</mo> <msubsup> <mi>R</mi> <mi>f</mi> <mi>k</mi> </msubsup> <mo>)</mo> </mrow> </mrow>
wherein,respectively representing the mth and kth low-resolution training set face image samples W of a certain illumination ffThe reconstruction coefficient representing a certain illumination f, N being the maximum of the number of training set samples, X being the local covariance matrix, X-1An inverse matrix of X is represented by,representing an inverse matrix X-1The m-th row of (a), the k-th column of (b),representing an inverse matrix X-1The element of row a and column b, XmkAn element representing the m-th row and the k-th column of the matrix X; a, b, k and m are all positive integers; t denotes transposition.
4. The illumination robustness face image super-resolution method according to claim 1, wherein the second step of diagonal loading of the local covariance matrix X comprises:
X＝X+λY
where λ is a constant whose value can be set experimentally and Y is an N × N unity diagonal matrix.
5. The illumination robustness face image super-resolution method according to claim 1, wherein in the third step, samples of low resolution training sets corresponding to different illumination c are sampledExpressed as column vector form, according to the approximate linear mapping relation existing between the redundant linear combination of certain illumination f and different illumination c, the low resolution face image IcThe calculation formula is as follows:
<mrow> <msub> <mi>I</mi> <mi>c</mi> </msub> <mo>=</mo> <msub> <mi>W</mi> <mi>f</mi> </msub> <msubsup> <mi>R</mi> <mi>c</mi> <mi>n</mi> </msubsup> <mo>.</mo> </mrow>
6. the illumination robustness face image super-resolution method according to claim 1, wherein in the fourth step, the high-resolution images H with different illumination are synthesized by weighting using the reconstruction method based on the whole face imagecThe specific method comprises the following steps:
(1) calculating the optimal weight Wc；
<mrow> <msub> <mi>W</mi> <mi>c</mi> </msub> <mo>=</mo> <munder> <mi>argmin</mi> <msubsup> <mi>w</mi> <mi>c</mi> <mi>n</mi> </msubsup> </munder> <mo>|</mo> <mo>|</mo> <msub> <mi>I</mi> <mi>c</mi> </msub> <mo>-</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>n</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msubsup> <mi>w</mi> <mi>c</mi> <mi>n</mi> </msubsup> <msubsup> <mi>R</mi> <mi>c</mi> <mi>n</mi> </msubsup> <mo>|</mo> <msup> <mo>|</mo> <mn>2</mn> </msup> </mrow>
<mrow> <mi>s</mi> <mo>.</mo> <mi>t</mi> <mo>.</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>n</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msubsup> <mi>w</mi> <mi>c</mi> <mi>n</mi> </msubsup> <mo>=</mo> <mn>1</mn> </mrow>
In the formula, WcRepresents each reconstruction weightThe N-dimensional weight column vector of (1);
n is the number of high-low resolution image pairs corresponding to different illumination c in the training set;
(2) high-resolution images H with different illumination are weighted and synthesized by utilizing the similarity of the human face image structurescComprises the following steps:
<mrow> <msub> <mi>H</mi> <mi>c</mi> </msub> <mo>=</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>n</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msubsup> <mi>H</mi> <mi>c</mi> <mi>n</mi> </msubsup> <msub> <mi>W</mi> <mi>c</mi> </msub> </mrow>
whereinRepresenting high resolution training set samples corresponding to a plurality of different illuminations c.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710842317.3A CN107633483A (en) | 2017-09-18 | 2017-09-18 | The face image super-resolution method of illumination robustness |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710842317.3A CN107633483A (en) | 2017-09-18 | 2017-09-18 | The face image super-resolution method of illumination robustness |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107633483A true CN107633483A (en) | 2018-01-26 |
Family
ID=61102933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710842317.3A Pending CN107633483A (en) | 2017-09-18 | 2017-09-18 | The face image super-resolution method of illumination robustness |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107633483A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110838084A (en) * | 2019-09-24 | 2020-02-25 | 咪咕文化科技有限公司 | Image style transfer method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101794441A (en) * | 2010-04-08 | 2010-08-04 | 西安交通大学 | Method for reestablishing multi-frame different-angle high-resolution facial image by using single-frame low-resolution facial image |
| CN102521810A (en) * | 2011-12-16 | 2012-06-27 | 武汉大学 | Face super-resolution reconstruction method based on local constraint representation |
| CN102930510A (en) * | 2012-09-24 | 2013-02-13 | 长安大学 | Multi-angle mutual transformation method for facial image |
| CN102968775A (en) * | 2012-11-02 | 2013-03-13 | 清华大学 | Low-resolution face image rebuilding method based on super-resolution rebuilding technology |
| WO2015077926A1 (en) * | 2013-11-27 | 2015-06-04 | 苏州大学 | Super-resolution microscopy imaging method and system for continuously adjustable structured light illumination |
-
2017
- 2017-09-18 CN CN201710842317.3A patent/CN107633483A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101794441A (en) * | 2010-04-08 | 2010-08-04 | 西安交通大学 | Method for reestablishing multi-frame different-angle high-resolution facial image by using single-frame low-resolution facial image |
| CN102521810A (en) * | 2011-12-16 | 2012-06-27 | 武汉大学 | Face super-resolution reconstruction method based on local constraint representation |
| CN102930510A (en) * | 2012-09-24 | 2013-02-13 | 长安大学 | Multi-angle mutual transformation method for facial image |
| CN102968775A (en) * | 2012-11-02 | 2013-03-13 | 清华大学 | Low-resolution face image rebuilding method based on super-resolution rebuilding technology |
| WO2015077926A1 (en) * | 2013-11-27 | 2015-06-04 | 苏州大学 | Super-resolution microscopy imaging method and system for continuously adjustable structured light illumination |
Non-Patent Citations (1)
| Title |
|---|
| 刘良辰: "基于整体到局部的分离式人脸超分辨率重建方法研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110838084A (en) * | 2019-09-24 | 2020-02-25 | 咪咕文化科技有限公司 | Image style transfer method and device, electronic equipment and storage medium |
| CN110838084B (en) * | 2019-09-24 | 2023-10-17 | 咪咕文化科技有限公司 | Method and device for transferring style of image, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102902961B (en) | Face super-resolution processing method based on K neighbor sparse coding average value constraint | |
| Sun et al. | Regularized fourier ptychography using an online plug-and-play algorithm | |
| CN101556690B (en) | Image super-resolution method based on overcomplete dictionary learning and sparse representation | |
| CN105631807B (en) | The single-frame image super-resolution reconstruction method chosen based on sparse domain | |
| CN112818764B (en) | Low-resolution image facial expression recognition method based on feature reconstruction model | |
| US20150269708A1 (en) | Method for Generating High-Resolution Images using Regression Patterns | |
| CN106097253B (en) | A kind of single image super resolution ratio reconstruction method based on block rotation and clarity | |
| CN106651772B (en) | Super-resolution reconstruction method of satellite cloud picture | |
| Zhang et al. | A guaranteed convergence analysis for the projected fast iterative soft-thresholding algorithm in parallel MRI | |
| CN104680502B (en) | Infrared image super-resolution reconstruction method based on sparse dictionary and non-downsampling Contourlet conversion | |
| CN104574456A (en) | Graph regularization sparse coding-based magnetic resonance super-undersampled K data imaging method | |
| CN110830043A (en) | Image compressed sensing reconstruction method based on mixed weighted total variation and non-local low rank | |
| CN110139046A (en) | A kind of video frame synthetic method based on tensor | |
| Singh et al. | Sub-band energy constraints for self-similarity based super-resolution | |
| CN111754598B (en) | Local space neighborhood parallel magnetic resonance imaging reconstruction method based on transformation learning | |
| CN104915935B (en) | Compressed spectrum imaging method with dictionary learning is perceived based on non-linear compression | |
| CN106296583B (en) | Based on image block group sparse coding and the noisy high spectrum image ultra-resolution ratio reconstructing method that in pairs maps | |
| Fan et al. | Deep geometric distillation network for compressive sensing MRI | |
| Li et al. | D 3 C 2-Net: Dual-Domain Deep Convolutional Coding Network for Compressive Sensing | |
| CN107633483A (en) | The face image super-resolution method of illumination robustness | |
| CN111260556A (en) | Fourier laminated microscopic reconstruction method based on deep convolutional neural network | |
| CN112950473B (en) | Super-resolution method for MR image | |
| CN107945131B (en) | Distributed image reconstruction method for radio interference array | |
| Huang et al. | DF3Net: Dual frequency feature fusion network with hierarchical transformer for image inpainting | |
| Taniguchi et al. | Image super-resolution based on locality-constrained linear coding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180126 |