CN103971332A - HR (restrict restriction)-LLE (locally linear embedding) weight constraint based face image super-resolution restoration method - Google Patents

Abstract

The invention belongs to the filed of digital image processing and particularly relates to an HR (restrict restriction)-LLE (locally linear embedding) weight constraint based face image super-resolution restoration method. The restoration method includes: calculating a great many of HR face samples and average restoration weight constraint, relative to neighboring samples, of residual HR faces; during restoration, subjecting traditional LLE based face super-resolution restoration weight calculation method to weight constraint. The restoration method includes overall restoration and local detail compensation. The overall restoration aims to restore basic characteristics of a standard face as required, and the local detail compensation aims to restore the face image so as to enable the face to be provided with personality characteristics different from other faces. The HR-LLE weight constraint is added in the method in estimating the LLE based restoration weight of the target HR image, so that the weight is closer to the true restoration weight of the HR image in 12 norms. By the method, better image restoration results can be acquired.

Description

A kind of face image super-resolution restored method based on the constraint of HR-LLE weights

Technical field

The invention belongs to digital image processing field, particularly a kind of face image super-resolution restored method based on the constraint of HR-LLE weights.

Background technology

In recent years, the technology such as human face detection and recognition is being brought into play more and more important effect in the multimedia application such as video monitoring, mobile terminal and network retrieval.The quality of facial image has a great impact the performance tool of these multimedia application.Yet owing to being subject to the impact of image capture device and collection environment, particularly, under uncontrollable physical environment, the facial image getting is conventionally second-rate, is difficult to directly apply to follow-up detection and Identification.After man face image acquiring, adopt Super-Resolution (SuperResolution, SR) technology to improve quality of human face image and seem particularly important.

Existing human face super-resolution recovery technique can be divided into two classes: the method based on rebuilding and the method based on study.In recent years, the method based on study becomes the focus of research.Its main thought is to set up the mapping relations between low resolution and high-definition picture by learning method, and the prior imformation of being obtained by machine learning replaces the constraint condition of the artificial definition based in method for reconstructing.Along with the development of manifold learning theory, researcher has proposed a series of face image super-resolution restoration algorithms based on stream shape hypothesis.It is a kind of Method of Nonlinear Dimensionality Reduction representative in manifold learning that local linear embeds (Locally Linear Embedding, LLE), is used for carrying out in recent years the Super-Resolution of image by Many researchers, has obtained certain achievement.Human face super-resolution restoration algorithm based on LLE is to be all based upon on the basis of stream shape hypothesis, be high resolving power (High Resolution, HR) image (or image block) and corresponding low resolution (Low Resolution, LR) image (or image block) have similar local geometry.Specifically in LLE, show as pixel corresponding in LR and HR space or when image block is put by its surrounding pixel or piece carries out linear expression, weighted vector equates.This hypothesis is applied in image super-resolution recovery, a large amount of paired LR-HR learning sample storehouses of model; Then for LR image to be reconstructed, utilize the LR sample in Sample Storehouse to carry out linear expression, obtain LR weights coefficient.Process of reconstruction adopts LR weights coefficient directly to replace HR weights coefficient, predicts HR image with HR image linear combination corresponding in Sample Storehouse.Yet because LR is the mapping relations of one-to-many to the mapping in HR space, there is non-isometry in this mapping therefore.Directly with the weights replacement HR space weights in LR space, will introduce error.

Summary of the invention

The object of the invention is to, by a kind of face image super-resolution restored method based on the constraint of HR-LLE weights, low-resolution image is redeveloped into the image that resolution is higher.The high resolving power here refers to that spatial resolution amplification is more than 4 times or 4 times.The present invention mainly for be facial image.

The present invention adopts following technological means to realize:

First obtain a large amount of HR people's face samples and residual error HR people appearance for the average reconstruction weights constraint of its neighbour's sample; In process of reconstruction, utilize respectively the overall situation and local average to rebuild weights constraint traditional human face super-resolution reconstruction weights method for solving based on LLE is carried out to weights constraint.Overall flow figure as shown in Figure 1.Reconstruction algorithm is divided into overall situation reconstruction and two parts of local detail compensation.Here the object that the overall situation is rebuild is the essential characteristic that backout criterion people face should possess; The object of local detail compensation is to rebuild facial image should have the personal characteristics of distinguishing other people face;

The method specifically comprises the following steps:

(1) overall HR-LLE weights constraint reestablishing

(1) the average weights of rebuilding of the overall situation retrain

HR people's face sample image storehouse that model is paired and LR people's face sample image storehouse; Utilize Euclidean distance as criterion, to find out successively the K of each LR people's face sample ₁individual arest neighbors LR people face sample and K corresponding thereto ₁individual HR people's face sample image; Utilize traditional weights method for solving based on LLE to calculate LR people's face sample with respect to its K ₁the reconstruction weights of individual LR nearest samples; Utilize traditional weights method for solving based on LLE to show that HR people's face sample is with respect to K ₁the reconstruction weights of individual HR nearest samples, weights result is as shown in Figure 2; Between LR weights and HR weights, exist difference, this difference shows that the integral body fluctuating trend of weights coefficient is consistent, but variance is different; Reconstruction weights to LR sample and HR sample are asked l ₂norm, result as shown in Figure 3.The l of the reconstruction weights of HR sample ₂norm is to fluctuate in a very little scope, gets the l that a large amount of HR samples are rebuild weights ₂the mean value of norm is on average rebuild weights constraint W as the overall situation _g.

(2) overall situation is rebuild

Input LR facial image, utilizes Euclidean distance to find out K in LR people's face Sample Storehouse ₁individual LR nearest samples.Utilize the reconstruction weights optimization method based on HR-LLE that the present invention proposes to show that input LR facial image is with respect to its K ₁the reconstruction weights of individual LR nearest samples; Find out K ₁the HR sample that individual LR nearest samples is corresponding; Utilize and rebuild weights and K ₁individual HR sample carries out linear combination and obtains the facial image that the overall situation is rebuild.

(2) local detail compensation

(1) local average is rebuild weights constraint

First, set up HR residual error people face Sample Storehouse and corresponding LR residual error people face Sample Storehouse; LR imagery exploitation overall situation HR-LLE weights bounding algorithm in sample generates one group of initial HR image amplifying; In calculating Sample Storehouse, HR image and these initial residual errors of amplifying between HR image, obtain one group of HR residual sample storehouse; Then HR people's face sample image of initial amplification is carried out to down-sampling, the subimage after calculating down-sampling and the residual error between LR people's face sample image, obtain one group of LR residual error people face sample image.

While compensating due to local detail, be that piecemeal carries out, so asking for the method that also adopts piecemeal when local average is rebuild weights constraint.The method for solving of on average rebuilding weights constraint to the overall situation is similar, the l of the reconstruction weights of HR residual sample image block ₂norm is also to fluctuate in a very little scope, therefore gets the l that a large amount of HR residual sample pieces are rebuild weights ₂the mean value of norm is rebuild weights constraint W as local average _l.

(2) local detail compensation

The facial image that the overall situation is rebuild carries out down-sampling processing; Down-sampled images and input LR facial image are done to the poor LR residual error facial image that obtains; LR residual error facial image is carried out to piecemeal processing; Utilize Euclidean distance in LR residual error people face Sample Storehouse, to find out successively the K of each LR residual image piece ₂individual arest neighbors LR residual sample image block; Find out this K ₂the corresponding HR residual sample piece of individual arest neighbors LR residual sample image block; Utilize the reconstruction weights optimization method based on HR-LLE that the present invention proposes to show that input LR residual error facial image piece is with respect to K ₂the reconstruction weights of individual LR arest neighbors residual sample piece; Utilize this reconstruction weights and K ₂the linear combination of individual HR residual sample piece obtains the HR residual image piece of rebuilding.

Rebuild successively HR residual image piece, finally reconstruct view picture HR residual error facial image; This HR residual error facial image and the initial facial image that amplifies are added, obtain final output and amplify facial image.

The technique effect that the present invention is useful is: proposed a kind of HR image reconstruction weights Forecasting Methodology based on the constraint of HR-LLE weights.The method, when the reconstruction weights based on LLE of estimating target HR image, has added the constraint of HR-LLE weights, makes weights at l ₂in norm, more approach real HR image reconstruction weights.On this basis, a kind of face image super-resolution restored method based on the constraint of HR-LLE weights has been proposed.With respect to traditional images method for reconstructing, the method can obtain good image restoration result.

feature of the present invention:

(1) a kind of target HR image reconstruction weights Forecasting Methodology based on the constraint of HR-LLE weights has been proposed.The method, when the reconstruction weights based on LLE of estimating target HR image, has added the constraint of HR-LLE weights, makes weights at l ₂in norm, more approach true HR weights.

(2) a kind of face image super-resolution restored method based on the constraint of HR-LLE weights has been proposed.The method retrains from global characteristics, local feature two aspects respectively, rebuilds high-resolution human face image.Experiment shows, this paper method can obtain good high-resolution human face image reconstruction result.

Accompanying drawing explanation:

Fig. 1, the inventive method entire block diagram

The reconstruction weights comparison diagram of Fig. 2, HR face image weights and 4 times of down-sampling LR facial images

Fig. 3, face image weights l ₂the weights l of norm and 4 times of down-sampling LR people faces ₂norm contrast

Fig. 4, input low resolution facial image

Fig. 5, tentatively amplify facial image

Fig. 6, the final output facial image that amplifies

Fig. 7, the inventive method and the comparison of traditional interpolation amplification result

Embodiment:

Below in conjunction with Figure of description, embodiment of the present invention is illustrated.

(1) overall HR-LLE weights constraint reestablishing

(1) the average weights of rebuilding of the overall situation retrain

First, need to set up two people's face image pattern storehouses, i.e. paired HR facial image Sample Storehouse and corresponding LR facial image Sample Storehouse.In this invention, select CAS-PEAL facial image database and self-built China second-generation identity card image library to test; Select altogether 1470 width front face images, be all normalized to 140 * 160 pixel sizes as the HR people's face sample in Sample Storehouse; 4 times of down-sampling HR people face samples, generate corresponding LR people's face sample.

Utilize traditional weights method for solving based on LLE to calculate each LR people's face sample with respect to its K ₁the reconstruction weights of nearest samples, as shown in Equation (1).

Wherein for one of them LR people's face sample image, its K ₁individual nearest samples and that it is with respect to K ₁the reconstruction weights of individual nearest samples.In this invention, we are to K ₁value test, find when it is greater than 800, rebuilding the effect still time complexity that do not have clear improvement but increases a lot.In this invention, get K ₁=800.

Utilize traditional weights method for solving based on LLE to calculate the reconstruction weights of corresponding HR people's face sample, as shown in Equation (2).

Wherein be with corresponding HR people's face sample image, be with corresponding K ₁individual arest neighbors HR sample, that it is with respect to K ₁the reconstruction weights of individual arest neighbors HR sample.

Separate the reconstruction weights that optimization problem (1) and (2) draws LR sample and corresponding HR sample, example results as shown in Figure 2.Wherein wLR represents the reconstruction weights of LR sample, and wHR represents the reconstruction weights of HR sample; Found that, between LR weights and HR weights, exist difference, this difference shows that the integral body fluctuating trend of weights coefficient is consistent, and variance is different.Reconstruction weights to LR sample and HR sample are asked l ₂norm, example results as shown in Figure 3.Wherein star dotted line represents the l of LR sample reconstruction weights ₂norm, circular dashed line represents the l of HR sample reconstruction weights ₂norm; HR sample is rebuild the l of weights ₂norm is to fluctuate in a very little scope, therefore gets a large amount of HR samples and rebuilds weights l ₂the mean value of norm is on average rebuild weights constraint W as the overall situation _g, in this invention, get W _g=0.85.

(2) overall situation is rebuild

Inputting a width LR facial image x(is not included in Sample Storehouse), size is 35 * 40 pixels, as shown in Figure 4; Utilize Euclidean distance from LR people's face Sample Storehouse, to find out the K of x ₁individual nearest neighbour's face sample utilize formula (3) draw x with respect to reconstruction weights recycling formula (4) is rebuild the preliminary facial image y that amplifies of the overall situation _g, as shown in Figure 5.

$\left\{\begin{array}{c}\min \mathrm{\ϵ}={\left|\right|x-\underset{j=1}{\overset{K_1}{\mathrm{\Σ}}}{w}_{G_j}{\stackrel{\→}{l}}_{G_j}\left|\right|}_2+\mathrm{\α}|{\left|\right|W_G\left|\right|}_2-W_g|\\ {\mathrm{\Σ}}_{j=1}^{K_1}{w}_{G_j}=1;\\ w_{G_j}=0,\mathrm{if}({\stackrel{\→}{l}}_{G_j}\∉[{\stackrel{\→}{l}}_{G_1},{\stackrel{\→}{l}}_{G_2},{\stackrel{\→}{l}}_{G_3}{\stackrel{\→}{l}}_{G_4}......{\stackrel{\→}{l}}_{G_{K_1}}\left]\right)\end{array}\right.---\left(3\right)$ $y_G={\mathrm{\Σ}}_{j=1}^{K_1}{w}_{G_j}{\stackrel{\→}{h}}_{G_j}---\left(4\right)$

Wherein be corresponding HR people's face sample in HR people's face Sample Storehouse.In (3), utilize the average weights l that rebuilds of the overall situation ₂norm constraint W _gto rebuilding weights W _gretrain, separate so just the connect true weights of close-target HR facial image of the reconstruction weights that draw after optimization problem (3).ɑ is a customized parameter, and the reconstructed results obtaining while getting different value is different, and in this invention, we get ɑ=0.01.

(2) local HR-LLE weights constraint details compensation

(1) local average is rebuild weights constraint

Model HR residual error people face sample and corresponding LR residual error people face Sample Storehouse.LR imagery exploitation overall situation HR-LLE weights bounding algorithm in sample generates one group and initially amplifies HR image pattern, HR image and these initial residual errors of amplifying between HR sample in calculating Sample Storehouse, obtain one group of HR residual sample storehouse, then HR people's face sample image of initial amplification is carried out to down-sampling, subimage after calculating down-sampling and the residual error between LR people's face sample image, obtain LR residual error people face sample image.

While compensating due to local detail, be that piecemeal carries out, so asking for the method that also adopts piecemeal when local average is rebuild weights constraint; In this invention, HR residual error facial image is divided into 8 * 8 block of pixels, corresponding LR residual error facial image is divided into 2 * 2 block of pixels; The method for solving of on average rebuilding weights constraint to the overall situation is similar, the l of the reconstruction weights of HR residual sample image block ₂norm is also to fluctuate in a very little scope, gets a large amount of HR residual sample pieces and rebuilds weights l ₂the mean value of norm is on average rebuild weights constraint W as the overall situation _l, W in this invention _l=0.8.

(2) local detail compensation generates the final people of amplification face

The overall situation is initially amplified to people's face y _gcarry out 4 times of down-samplings, it is poor then down-sampling people face and input LR people face x to be done, and obtains inputting LR residual error people face diffLR.LR residual error people face diffLR is carried out to piecemeal processing (in order to guarantee the blocking effect between partial reconstruction image block, all having the overlapping of a pixel between image block), and block size is 2 * 2 pixels; From LR residual error people face Sample Storehouse, find successively each residual image piece k ₂individual arest neighbors residual block k in this invention ₂during >600, recovery effect is not improved, and therefore gets K ₂=600; Utilize formula (5) to draw LR residual image piece with respect to reconstruction weights $W_L=[w_{L_1},w_{L_2},w_{L_3},...,w_{L_{K_2}}].$ Recycling formula (6) is rebuild HR residual image piece

$\left\{\begin{array}{c}\min \mathrm{\ϵ}={\left|\right|{\mathrm{LR}}_{L_i}-\underset{j=1}{\overset{K_2}{\mathrm{\Σ}}}{w}_{L_j}{\stackrel{\→}{l}}_{L_j}\left|\right|}_2+\mathrm{\α}|{\left|\right|W_L\left|\right|}_2-W_l|\\ {\mathrm{\Σ}}_{j=1}^{K_2}{w}_{L_j}=1;\\ w_{L_j}=0,\mathrm{if}({\stackrel{\→}{l}}_{L_j}\∉[{\stackrel{\→}{l}}_{L_1},{\stackrel{\→}{l}}_{L_2},{\stackrel{\→}{l}}_{L_3}......{\stackrel{\→}{l}}_{L_{K_2}}\left]\right)\end{array}\right.---\left(5\right)$

${\mathrm{HR}}_{L_i}={\mathrm{\Σ}}_{j=1}^{K_2}{w}_{L_j}{\stackrel{\→}{h}}_{L_j}---\left(6\right)$

Wherein be corresponding HR residual error facial image piece in HR residual error people face sample image piece storehouse.In (5), utilize local average to rebuild weights constraint W _lto rebuilding weights W _lretrain, separate like this reconstruction weights that draw after optimization problem (5) and just connect and be bordering on the true reconstruction weights of target HR residual error facial image piece.

Rebuild successively HR residual image piece, finally reconstruct view picture HR residual image diffHR.By HR residual error facial image diffHR and the initial facial image y that amplifies _gbe added, obtain final output and amplify facial image y, as shown in Figure 6.

Fig. 7 is the inventive method and traditional interpolation amplification methods and results comparison diagram.

Claims (1)

1. the face image super-resolution restored method based on the constraint of HR-LLE weights, is characterized in that comprising the steps:

(1) overall HR-LLE weights constraint reestablishing

(1) the average weights of rebuilding of the overall situation retrain

First, need to set up two people's face image pattern storehouses, i.e. paired HR facial image Sample Storehouse and corresponding LR facial image Sample Storehouse;

The weights method for solving of utilization based on LLE calculates each LR people's face sample with respect to its K ₁the reconstruction weights of nearest samples, as shown in Equation (1);

Wherein for one of them LR people's face sample image, its K ₁individual nearest samples and that it is with respect to K ₁the reconstruction weights of individual nearest samples; K ₁=800;

The weights method for solving of utilization based on LLE calculates the reconstruction weights of corresponding HR people's face sample, as shown in Equation (2);

Wherein be with corresponding HR people's face sample image, be with corresponding K ₁individual arest neighbors HR sample, that it is with respect to K ₁the reconstruction weights of individual arest neighbors HR sample;

Separate the reconstruction weights that optimization problem (1) and (2) draws LR sample and corresponding HR sample, W _g=0.85;

(2) overall situation is rebuild

Input a width LR facial image x, size is 35 * 40 pixels; Utilize Euclidean distance from LR people's face Sample Storehouse, to find out the K of x ₁individual nearest neighbour's face sample utilize formula (3) draw x with respect to reconstruction weights $W_G=[w_{G_1},w_{G_2},w_{G_3},...,w_{G_{K_1}}];$ Recycling formula (4) is rebuild the preliminary facial image y that amplifies of the overall situation _g;

$\left\{\begin{array}{c}\min \mathrm{\ϵ}={\left|\right|x-\underset{j=1}{\overset{K_1}{\mathrm{\Σ}}}{w}_{G_j}{\stackrel{\→}{l}}_{G_j}\left|\right|}_2+\mathrm{\α}|{\left|\right|W_G\left|\right|}_2-W_g|\\ {\mathrm{\Σ}}_{j=1}^{K_1}{w}_{G_j}=1;\\ w_{G_j}=0,\mathrm{if}({\stackrel{\→}{l}}_{G_j}\∉[{\stackrel{\→}{l}}_{G_1},{\stackrel{\→}{l}}_{G_2},{\stackrel{\→}{l}}_{G_3}{\stackrel{\→}{l}}_{G_4}......{\stackrel{\→}{l}}_{G_{K_1}}\left]\right)\end{array}\right.---\left(3\right)$

$y_G={\mathrm{\Σ}}_{j=1}^{K_1}{w}_{G_j}{\stackrel{\→}{h}}_{G_j}---\left(4\right)$

Wherein be corresponding HR people's face sample in HR people's face Sample Storehouse; In (3), utilize the average weights l that rebuilds of the overall situation ₂norm constraint W _gto rebuilding weights W _gretrain, separate so just the connect true weights of close-target HR facial image of the reconstruction weights that draw after optimization problem (3); ɑ=0.01;

(2) local HR-LLE weights constraint details compensation

(1) local average is rebuild weights constraint

Model HR residual error people face sample and corresponding LR residual error people face Sample Storehouse; LR imagery exploitation overall situation HR-LLE weights bounding algorithm in sample generates one group and initially amplifies HR image pattern, HR image and these initial residual errors of amplifying between HR sample in calculating Sample Storehouse, obtain one group of HR residual sample storehouse, then HR people's face sample image of initial amplification is carried out to down-sampling, subimage after calculating down-sampling and the residual error between LR people's face sample image, obtain LR residual error people face sample image;

While compensating due to local detail, be that piecemeal carries out, so asking for the method that also adopts piecemeal when local average is rebuild weights constraint; HR residual error facial image is divided into 8 * 8 block of pixels, and corresponding LR residual error facial image is divided into 2 * 2 block of pixels; The method for solving of on average rebuilding weights constraint to the overall situation is similar, W _l=0.8;

(2) local detail compensation generates the final people of amplification face

The overall situation is initially amplified to people's face y _gcarry out 4 times of down-samplings, it is poor then down-sampling people face and input LR people face x to be done, and obtains inputting LR residual error people face diffLR; To LR residual error people face, diffLR carries out piecemeal processing, and block size is 2 * 2 pixels; From LR residual error people face Sample Storehouse, find successively each residual image piece k ₂individual arest neighbors residual block k ₂=600; Utilize formula (5) to draw LR residual image piece with respect to reconstruction weights $W_L=[w_{L_1},w_{L_2},w_{L_3},...,w_{L_{K_2}}];$ Recycling formula (6) is rebuild HR residual image piece

$\left\{\begin{array}{c}\min \mathrm{\ϵ}={\left|\right|{\mathrm{LR}}_{L_i}-\underset{j=1}{\overset{K_2}{\mathrm{\Σ}}}{w}_{L_j}{\stackrel{\→}{l}}_{L_j}\left|\right|}_2+\mathrm{\α}|{\left|\right|W_L\left|\right|}_2-W_l|\\ {\mathrm{\Σ}}_{j=1}^{K_2}{w}_{L_j}=1;\\ w_{L_j}=0,\mathrm{if}({\stackrel{\→}{l}}_{L_j}\∉[{\stackrel{\→}{l}}_{L_1},{\stackrel{\→}{l}}_{L_2},{\stackrel{\→}{l}}_{L_3}......{\stackrel{\→}{l}}_{L_{K_2}}\left]\right)\end{array}\right.---\left(5\right)$

${\mathrm{HR}}_{L_i}={\mathrm{\Σ}}_{j=1}^{K_2}{w}_{L_j}{\stackrel{\→}{h}}_{L_j}---\left(6\right)$

Wherein be corresponding HR residual error facial image piece in HR residual error people face sample image piece storehouse; In (5), utilize local average to rebuild weights constraint W _lto rebuilding weights W _lretrain, separate like this reconstruction weights that draw after optimization problem (5) and just connect and be bordering on the true reconstruction weights of target HR residual error facial image piece;

Rebuild successively HR residual image piece, finally reconstruct view picture HR residual image diffHR; By HR residual error facial image diffHR and the initial facial image y that amplifies _gbe added, obtain final output and amplify facial image y.