CN102968775B

CN102968775B - Low-resolution face image rebuilding method based on super-resolution rebuilding technology

Info

Publication number: CN102968775B
Application number: CN201210435373.2A
Authority: CN
Inventors: 苏光大; 任小龙; 苏楠
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2012-11-02
Filing date: 2012-11-02
Publication date: 2015-04-15
Anticipated expiration: 2032-11-02
Also published as: CN102968775A

Abstract

The invention relates to a method for reconstructing a low-resolution face image based on a super-resolution reconstruction technology, and belongs to the field of image processing. Then, by generating a training set, the low-resolution image to be reconstructed is denoised and the learning samples are determined by blur estimation, and the final reconstructed high-resolution image is obtained through LLE learning. The normalization method of the human face image adopted by the present invention is not only accurate, but also consistent with the normalization method of the training set, which improves the accuracy of the low-resolution human face image. Then, the super-resolution face image reconstruction method based on low-frequency components is used to produce a large-scale reconstruction image of the low-resolution face image, which better solves the reconstruction problem of the low-resolution face image.

Description

Reconstruction method of low-resolution face image based on super-resolution reconstruction technology

技术领域 technical field

本发明属于图像处理领域，特别涉及一种制作低分辨率人脸重建像的方法。The invention belongs to the field of image processing, in particular to a method for making low-resolution human face reconstruction images.

背景技术 Background technique

当前，视频监控得到了迅速的发展，在安防工作中发挥了越来越重要的作用。在视频监控的图像中，记录了许多与案件有关的人像信息。但是，由于视频监控的涉案人脸图像往往很小，因此无法分辨涉案人，致使许多案件的办案工作陷入了困境。这种小人脸图像的本质问题是人脸分辨率低的问题。低分辨率人脸图像的重建技术是公安部门办案工作急需的关键技术。本发明所指人脸图像，限定为正面人脸图像（下同）。包括人的头顶之下、颏底线之上、左耳到右耳之间的部分（参见中华人民共和国公共安全行业标准GA/T893-2010“安防生物特征识别应用术语”）。At present, video surveillance has developed rapidly and played an increasingly important role in security work. In the images of video surveillance, many portrait information related to the case were recorded. However, since the face images of the people involved in the video surveillance are often very small, it is impossible to distinguish the people involved, which has caused many cases to run into difficulties. The essential problem of this small face image is the problem of low resolution of the face. The reconstruction technology of low-resolution face images is a key technology urgently needed by the public security department to handle cases. The human face image referred to in the present invention is limited to the front face image (the same below). Including the part below the top of the head, above the bottom line of the chin, and between the left ear and the right ear (see the People's Republic of China public security industry standard GA/T893-2010 "Security biometrics application terms").

专利号为ZL 2005 1 0067692.X的专利公开一种人脸几何尺寸归一化的方法：即基于人脸几何尺寸归一化的人脸图像识别方法中的人脸三点归一化方法。确定图像中人脸的左、右眼和颌下点的坐标，并按三点归一化方法生成标准的尺寸的归一化低分辨率人脸图像。The patent No. ZL 2005 1 0067692.X discloses a method of normalizing the geometric size of the face: that is, the three-point normalization method of the face in the face image recognition method based on the normalization of the geometric size of the face. Determine the coordinates of the left, right eye and submandibular points of the face in the image, and generate a normalized low-resolution face image of a standard size according to the three-point normalization method.

目前常用的超分辨率人脸图像重建的方法为：利用一个包含成对的高分辨率图像及其对应的低分辨率图像的数据集作为训练库。将待重建的含有低分辨率人脸的图像分为若干小块，对待重建的低分辨率图像中的每一个小块，通过LLE表示成训练库中所有低分辨率图像对应位置小块的线性组合，然后使用相同的线性组合权重对训练库中相应高分辨率图像对应位置的小块进行组合，得到重建后的超分辨图像对应位置的小块。将得到的重建后的超分辨图像对应位置的小块进行拼接，块与块之间有少量的重叠，重叠的部分取平均值。最后得到整个重建后的超分辨大图。该方法虽然能对超低分辨率图像进行重建，但是，其直接在时域中进行流形学习，局部几何结构相似度不高，流行学习效果较差。此外其重建过程中受光照及噪声影响较大，未对超低分辨率人脸图像进行配准，对实际场景中的超分辨率图像重建效果较差。The currently commonly used method for super-resolution face image reconstruction is to use a data set containing pairs of high-resolution images and their corresponding low-resolution images as a training library. The image containing the low-resolution face to be reconstructed is divided into several small blocks, and each small block in the low-resolution image to be reconstructed is represented by LLE as the linearity of the corresponding small blocks of all low-resolution images in the training library Then use the same linear combination weight to combine the small blocks corresponding to the corresponding positions of the corresponding high-resolution images in the training library to obtain the small blocks corresponding to the reconstructed super-resolution images. The small blocks corresponding to the corresponding positions of the obtained reconstructed super-resolution image are spliced, and there is a small amount of overlap between the blocks, and the overlapping parts are averaged. Finally, the entire reconstructed super-resolution large image is obtained. Although this method can reconstruct ultra-low-resolution images, it directly performs manifold learning in the time domain, and the similarity of local geometric structures is not high, and the popular learning effect is poor. In addition, the reconstruction process is greatly affected by illumination and noise, and the ultra-low resolution face images are not registered, so the super-resolution image reconstruction effect in the actual scene is poor.

发明内容 Contents of the invention

本发明的目的是为克服已有技术的不足之处，提出了一种基于超分辨率重建技术的低分辨率人脸图像的重建方法，较好地解决了低分辨率人脸图像的重建难题。The purpose of the present invention is to overcome the deficiencies of the prior art, and propose a reconstruction method for low-resolution human face images based on super-resolution reconstruction technology, which better solves the reconstruction problem of low-resolution human face images .

一种基于超分辨率重建技术的低分辨率人脸图像的重建方法，其特征在于，包括以下步骤：A method for reconstructing low-resolution face images based on super-resolution reconstruction technology, characterized in that it comprises the following steps:

1）对原始图像进行基于人脸三点定位的归一化，得到待重建的归一化人脸图像，具体包括：1) Normalize the original image based on three-point positioning of the face to obtain a normalized face image to be reconstructed, including:

1.1）对含有低分辨率人脸图像的原始图像进行裁剪，裁剪出包含低分辨率人脸图像的矩形区域，得到低分辨率人脸图像；该区域左上、左下、右上、右下4个顶点在原始图像中坐标分别为：（m₁，n₁）、（m₁，n₂）、（m₂，n₁）、（m₂，n₂），m、n分别表示坐标系中的点的横坐标和纵坐标；1.1) Crop the original image containing the low-resolution face image, cut out a rectangular area containing the low-resolution face image, and obtain the low-resolution face image; the upper left, lower left, upper right, and lower right vertices of this area The coordinates in the original image are: (m ₁ , n ₁ ), (m ₁ , n ₂ ), (m ₂ , n ₁ ), (m ₂ , n ₂ ), and m and n represent points in the coordinate system respectively The abscissa and ordinate of ;

1.2）对1.1）中得到的低分辨率人脸图像进行等比放大，放大倍率为K×K，放大后图像的高度为L，L＝K×(n₂-n₁)，k满足：K≥150/(n₂-n₁)；1.2) Scale up the low-resolution face image obtained in 1.1), the magnification is K×K, the height of the enlarged image is L, L=K×(n ₂ -n ₁ ), k satisfies: K ≥150/(n ₂ -n ₁ );

1.3）对放大后的低分辨率人脸图像采用基于人脸三点定位的归一化方法进行归一化，得到一幅待重建的标准尺寸的归一化人脸图像；1.3) Normalize the enlarged low-resolution face image using a normalization method based on three-point positioning of the face to obtain a normalized face image of a standard size to be reconstructed;

1.31）在放大后的低分辨率人脸图像a上确定左眼球上的一点A的坐标（x₁₁,y₁₁）、右眼球上的一点B的坐标位置（x₂₁,y₂₁），通过A、B两点做直线L₁，并确定下颌点C₀坐标（x₀₁,y₀₁），按放大倍率K将三点的坐标对应至原裁剪图像坐标，分别为（x₁,y₁），（x₂,y₂），（x₀,y₀）；1.31) Determine the coordinates (x ₁₁ , y ₁₁ ) of a point A on the left eyeball and the coordinate position (x ₂₁ , y ₂₁ ) of a point B on the right eyeball on the enlarged low-resolution face image a, and pass A and B to make a straight line L ₁ , and determine the coordinates (x ₀₁ , y ₀₁ ) of the mandibular point C ₀ , and correspond the coordinates of the three points to the coordinates of the original cropped image according to the magnification K, respectively (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₀ , y ₀ );

1.32）对原始图像进行基于人脸几何尺寸归一化的人脸图像识别方法中的人脸三点归一化方法，得到尺寸为H×W的待重建的归一化人脸图像，其中H，W满足下式：1.32) Perform the face three-point normalization method in the face image recognition method based on face geometric size normalization on the original image to obtain a normalized face image to be reconstructed with a size of H×W, where H , W satisfies the following formula:

$H h = = \{\begin{matrix} 6464,, & (({n no}_{22} - - {n no}_{11})) &GreaterEqual; &Greater Equal; 6464 \\ 3232,, & 6464 > > (({n no}_{22} - - {n no}_{11})) &GreaterEqual; &Greater Equal; 3232 \\ 1616,, & 3232 > > (({n no}_{22} - - {n no}_{11})) &GreaterEqual; &Greater Equal; 1616 \end{matrix},, - - - - - - ((11))$

W=H×3/4，H₀=5×H/12,H₁=63×H/120，H₂=7×H/120；W=H×3/4, H ₀ =5×H/12, H ₁ =63×H/120, H ₂ =7×H/120;

2）对标准的待重建的归一化人脸图像进行基于低频分量的超分辨率人脸图像的重建得到512×384高分辨率图像：2) Reconstruct the standard normalized face image to be reconstructed based on the super-resolution face image of the low-frequency component to obtain a 512×384 high-resolution image:

2.1）生成训练集:2.1) Generate training set:

2.11）将不低于100幅的清晰正面人脸图像作为训练集的图像；2.11) Use no less than 100 clear frontal face images as the images of the training set;

2.12）采用人脸三点定位归一化方法将训练集中的每一幅图像变为512×384的标准分辨率人脸图像；2.12) Use the face three-point positioning normalization method to convert each image in the training set into a 512×384 standard resolution face image;

2.13）对每一幅512×384的标准分辨率图像使用9-7滤波器下采样生成32×24，64×48，128×96，256×192，512×384的五种清晰的分辨率样本，再对每一幅512×384的标准分辨率图像使用高斯金字塔滤波器下采样生成32×24的模糊的分辨率样本；2.13) Use a 9-7 filter to downsample each standard resolution image of 512×384 to generate five clear resolution samples of 32×24, 64×48, 128×96, 256×192, and 512×384 , and then use the Gaussian pyramid filter to downsample each standard resolution image of 512×384 to generate a blurred resolution sample of 32×24;

2.14）对64×48，128×96，256×192，512×384的四种清晰的分辨率样本用9-7滤波器组进行拉普拉斯金字塔分解提取高频分量并保存，作为高分辨率图像小波系数样本；2.14) For four clear resolution samples of 64×48, 128×96, 256×192, and 512×384, use a 9-7 filter bank to perform Laplacian pyramid decomposition to extract high-frequency components and save them as high-resolution rate image wavelet coefficient samples;

2.15）对32×24，64×48，128×96，256×192四种清晰的分辨率样本进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解提取它们的高频分量并保存，作为清晰的低分辨率图像小波系数样本，同时对模糊的32×24分辨率样本进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解提取其高频分量并保存，作为模糊的32×24低分辨率图像小波系数样本；2.15) Perform nearest neighbor interpolation and bicubic linear interpolation on four clear resolution samples of 32×24, 64×48, 128×96, and 256×192, take the average of the two, and perform Lap with a 9-7 filter bank Las Pyramid decomposes to extract their high-frequency components and save them as clear low-resolution image wavelet coefficient samples. At the same time, the nearest neighbor interpolation and bicubic linear interpolation are performed on the fuzzy 32×24 resolution samples, and the average of the two is used. The 9-7 filter bank performs Laplacian pyramid decomposition to extract its high-frequency components and save them as wavelet coefficient samples of fuzzy 32×24 low-resolution images;

2.2）对待重建的低分辨率图像进行去噪并通过模糊度估计确定学习样本：2.2) Denoise the low-resolution image to be reconstructed and determine the learning samples by blur estimation:

2.21）对归一化后的待重建的32×24低分辨率图像进行去噪；步骤如下：2.21) Denoise the normalized 32×24 low-resolution image to be reconstructed; the steps are as follows:

对待重建图像进行拉普拉斯金字塔分解，计算出其所有高频系数绝对值的中位数σ作为噪声标准差估计值，根据经典去噪算法以λσ为阈值进行去噪，λ为经验系数，取值范围为0.05＜λ＜0.3；The image to be reconstructed is decomposed into a Laplace pyramid, and the median σ of the absolute values of all high-frequency coefficients is calculated as the estimated value of the noise standard deviation. According to the classic denoising algorithm, λσ is used as the threshold for denoising, and λ is the empirical coefficient. The value range is 0.05<λ<0.3;

2.22）进行模糊程度估计，算出模糊程度系数：2.22) Estimate the degree of fuzziness and calculate the coefficient of fuzziness degree:

$u u = = \frac{{Σ Σ}_{h h}}{{Σ Σ}_{i i}},, - - - - - - ((22))$

∑_h为待重建的低分辨率图像的高频系数绝对值之和，∑_i为去噪后的待重建的32×24低分辨率图像像素值之和；∑ _h is the sum of the absolute values of the high-frequency coefficients of the low-resolution image to be reconstructed, and ∑ _i is the sum of the pixel values of the 32×24 low-resolution image to be reconstructed after denoising;

2.23）根据模糊程度系数u选取学习样本：2.23) Select learning samples according to the fuzzy degree coefficient u:

u＜0.05，时学习采用模糊的32×24样本，u≥0.05时采用清晰的32×24样本；When u<0.05, fuzzy 32×24 samples are used for learning, and clear 32×24 samples are used when u≥0.05;

2.3）通过LLE学习得到最终重建出的高分辨率图像：2.3) The final reconstructed high-resolution image is obtained through LLE learning:

2.31）从32×24到64×48放大:2.31) Zoom in from 32×24 to 64×48:

2.31.1）将于2.21）中得到的去噪后的32×24低分辨率图像的像素值直接作为重建出的64×48高分辨率图像的低频系数；2.31.1) The pixel values of the denoised 32×24 low-resolution image obtained in 2.21) are directly used as the low-frequency coefficients of the reconstructed 64×48 high-resolution image;

2.31.2）对去噪后的32×24低分辨率图像进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解得到其高频系数；2.31.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the denoised 32×24 low-resolution image, take the average of the two, and use the 9-7 filter bank to decompose the Laplacian pyramid to obtain its high-frequency coefficients ;

2.31.3）将去噪后的32×24低分辨率图像分割为大小为s×t的小块，s取值范围为4~8，t取值范围为8~12，相邻小块间有（s-1）×（t-1）的重叠；2.31.3) Divide the denoised 32×24 low-resolution image into small blocks with a size of s×t, where the value of s ranges from 4 to 8, and the value of t ranges from 8 to 12. There is an overlap of (s-1) × (t-1);

2.31.4）利用LLE学习得到2.31.3）中分割出的小块对应位置的64×48高分辨率小块的高频系数：2.31.4) Use LLE learning to obtain the high-frequency coefficients of the 64×48 high-resolution small blocks corresponding to the small blocks segmented in 2.31.3):

令y_LR为单个于2.31.3）中得到小块对应的于2.31.2）中得到的高频系数组成的向量，x_LR(i)为第i个在2.23）确定的样本集中相应位置小块的高频系数组成的向量，在2.23）确定的样本集相应位置的小块中取前k个（5<k<8）与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.31.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the corresponding position of the i-th sample set determined in 2.23). The vector composed of the high-frequency coefficients of the block, in the small blocks of the corresponding position of the sample set determined in 2.23), take the first k (5<k<8) sample small blocks with the smallest y _LR mean square error, and calculate the following extreme value problem :

$W W = = \underset{w w ((i i))}{arg arg min min} {| | | | ϵ ϵ {y the y}_{LR LR} - - {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{LR LR} ((i i)) | | | |}^{22},,$ ${Σ Σ}_{i i = = 11}^{k k} w w ((i i)) = = 11 - - - - - - ((33))$

令G＝(εy_LR1^T-X)^T(εy_LR1^T-X)，其中X的每列为x_LR(i)，共k列，1为元素皆为1的向量；w(i)为k个待求系数，此极值问题的解为W为w(i)组成的向量；进而计算出学习得到的64×48高分辨率图像小块的高频系数向量为：Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1; w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i); and then the high-frequency coefficient vector of the learned 64×48 high-resolution image block is calculated as:

${y the y}_{HR HR} = = \frac{{Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{HR HR} ((i i))}{ϵ ϵ} - - - - - - ((44))$

其中x_HR(i)为x_LR(i)对应的64×48清晰分辨率样本小块的高频系数向量；Where x _HR (i) is the high-frequency coefficient vector of the 64×48 clear resolution sample block corresponding to x _LR (i);

若进行光照处理，（3）、（4）两式中若不进行光照处理，（3）、（4）两式中ε=1；If light treatment is carried out, in (3) and (4) two formulas If no light treatment is performed, ε=1 in the two formulas (3) and (4);

2.31.5）重复2.31.4）得到所有2.31.3）中分割出的小块对应的64×48高分辨率小块的高频系数2.31.5) Repeat 2.31.4) to get the high-frequency coefficients of all 64×48 high-resolution small blocks corresponding to the small blocks segmented in 2.31.3)

2.31.6）将2.31.5）中学习得到的64×48高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为结果，得到64×48高分辨率图像的高频系数；2.31.6) Splice the 64×48 high-resolution small blocks learned in 2.31.5), and take the average value of all overlapping small blocks as the result for the overlap between adjacent small blocks, and obtain a 64×48 high-resolution High frequency coefficient of the image;

2.31.7）将得到的64×48高分辨率图像的高、低频系数进行反变换便得到64×48高分辨率图像；2.31.7) Inversely transform the high and low frequency coefficients of the obtained 64×48 high-resolution image to obtain a 64×48 high-resolution image;

2.32）从64×48到128×96放大：2.32) Zoom in from 64×48 to 128×96:

2.32.1）将从2.31）步得到的64×48高分辨率图像作为此步的64×48低分辨率图像，其像素值直接作为重建出的128×96高分辨率图像的低频系数；2.32.1) The 64×48 high-resolution image obtained from step 2.31) is used as the 64×48 low-resolution image in this step, and its pixel value is directly used as the low-frequency coefficient of the reconstructed 128×96 high-resolution image;

2.32.2）对64×48低分辨率图像进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解得到其高频系数；2.32.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the 64×48 low-resolution image, take the average of the two, and use the 9-7 filter bank to decompose the Laplacian pyramid to obtain its high-frequency coefficients;

2.32.3）将64×48低分辨率图像分割为大小为s×t的小块，s取值范围为4~8，t取值范围为8~12，相邻小块间有（s-1）×（t-1）的重叠；2.32.3) Divide the 64×48 low-resolution image into small blocks with a size of s×t. The value range of s is 4~8, and the value range of t is 8~12. There are (s- 1) × (t-1) overlap;

2.32.4）利用LLE学习得到2.32.3）中分割出的小块对应位置的128×96高分辨率小块的高频系数：2.32.4) Use LLE learning to obtain the high-frequency coefficient of the 128×96 high-resolution small block corresponding to the small block segmented in 2.32.3):

令y_LR为单个于2.31.3）中得到小块对应的于2.32.2）中得到的高频系数组成的向量，x_LR(i)为第i个64×48清晰的分辨率样本集相应位置小块的高频系数组成的向量，在64×48清晰的分辨率样本集相应位置的小块中取前k个（5<k<8）与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.32.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the i-th 64×48 clear resolution sample set corresponding to The vector composed of the high-frequency coefficients of the position small blocks, the first k (5<k<8) sample blocks with the smallest y _LR mean square error are selected from the small blocks at the corresponding positions of the 64×48 clear resolution sample set, Calculate the following extremum problem:

令G＝(εy_LR1^T-X)^T(εy_LR1^T-X)，其中X的每列为x_LR(i)，共k列，1为元素皆为1的向量，w(i)为k个待求系数，此极值问题的解为W为w(i)组成的向量，进而计算出学习得到的128×96高分辨率图像小块的高频系数向量为：Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1, w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i), and then the high-frequency coefficient vector of the learned 128×96 high-resolution image block is calculated as:

其中x_HR(i)为x_LR(i)对应的128×96清晰分辨率样本小块的高频系数向量；Where x _HR (i) is the high-frequency coefficient vector of the 128×96 clear resolution sample block corresponding to x _LR (i);

2.32.5）重复2.32.4）得到所有2.32.3）中分割出的小块对应的128×96高分辨率小块的高频系数2.32.5) Repeat 2.32.4) to get the high-frequency coefficients of all 128×96 high-resolution small blocks corresponding to the small blocks segmented in 2.32.3)

2.32.6）将2.32.5）中学习得到的128×96高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为结果，得到128×96高分辨率图像的高频系数；2.32.6) Splice the 128×96 high-resolution small blocks learned in 2.32.5), and take the average value of all overlapping small blocks as the result for the overlap between adjacent small blocks, and obtain 128×96 high-resolution High frequency coefficient of the image;

2.32.7）将得到的128×96高分辨率图像的高、低频系数进行反变换便得出128×96高分辨率图像；2.32.7) Inversely transform the high and low frequency coefficients of the obtained 128×96 high-resolution image to obtain a 128×96 high-resolution image;

2.33）从128×96到256×192放大：2.33) Zoom in from 128×96 to 256×192:

2.33.1）将从2.32）步得到的128×96高分辨率图像作为此步的128×96低分辨率图像，其像素值直接作为重建出的256×192高分辨率图像的低频系数；2.33.1) The 128×96 high-resolution image obtained from step 2.32) is used as the 128×96 low-resolution image in this step, and its pixel value is directly used as the low-frequency coefficient of the reconstructed 256×192 high-resolution image;

2.33.2）对128×96低分辨率图像进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解提取得到其高频系数；2.33.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the 128×96 low-resolution image, take the average of the two, and use the 9-7 filter bank to perform Laplacian pyramid decomposition to extract its high-frequency coefficients;

2.33.3）将128×96低分辨率图像分割为大小为s×t的小块，s取值范围为4~8，t取值范围为8~12，相邻小块间有（s-1）×（t-1）的重叠；2.33.3) Divide the 128×96 low-resolution image into small blocks with a size of s×t. The value of s ranges from 4 to 8, and the value of t ranges from 8 to 12. There are (s- 1) × (t-1) overlap;

2.33.4）利用LLE学习得到2.33.3）中分割出的小块对应位置的256×192高分辨率小块的高频系数：2.33.4) Use LLE learning to obtain the high-frequency coefficients of the 256×192 high-resolution small blocks corresponding to the small blocks segmented in 2.33.3):

令y_LR为单个于2.31.3）中得到小块对应的于2.33.2）中得到的高频系数组成的向量，x_LR(i)为第i个128×96清晰的分辨率样本相应位置小块的高频系数组成的向量，在128×96清晰的分辨率样本相应位置的小块中取前k个（5<k<8）与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.33.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the corresponding position of the ith 128×96 clear resolution sample The vector composed of the high-frequency coefficients of the small blocks, in the small blocks corresponding to the 128×96 clear resolution samples, take the first k (5<k<8) sample small blocks with the smallest y _LR mean square error, and the calculation is as follows Extreme value problem:

令G＝(εy_LR1^T-X)^T(εy_LR1^T-X)，其中X的每列为x_LR(i)，共k列，1为元素皆为1的向量，w(i)为k个待求系数，此极值问题的解为W为w(i)组成的向量，进而计算出学习得到的256×192高分辨率图像小块的高频系数向量为：Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1, w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i), and then the high-frequency coefficient vector of the learned 256×192 high-resolution image block is calculated as:

其中x_HR(i)为x_LR(i)对应的256×192清晰分辨率样本小块的高频系数向量；Where x _HR (i) is the high-frequency coefficient vector of the 256×192 clear resolution sample block corresponding to x _LR (i);

2.33.5）重复2.33.4）得到所有2.33.3）中分割出的小块对应的256×192高分辨率小块的高频系数2.33.5) Repeat 2.33.4) to get the high-frequency coefficients of all 256×192 high-resolution small blocks corresponding to the small blocks segmented in 2.33.3)

2.33.6）将2.33.5）中学习得到的256×192高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为最终结果，得到256×192高分辨率图像的高频系数；2.33.6) The 256×192 high-resolution small blocks learned in 2.33.5) are stitched together, and the overlapping parts between adjacent small blocks are taken as the average value of all overlapping small blocks as the final result to obtain 256×192 high-resolution High-frequency coefficients of high-rate images;

2.33.7）将得到的256×192高分辨率图像的高、低频系数进行反变换便得出256×192高分辨率图像；2.33.7) Inversely transform the high and low frequency coefficients of the obtained 256×192 high-resolution image to obtain a 256×192 high-resolution image;

2.34）从256×192到512×384放大：2.34) Zoom in from 256×192 to 512×384:

2.34.1）将从2.33）步得到的256×192高分辨率图像作为此步的256×192低分辨率图像，其像素值直接作为重建出的512×384高分辨率图像的低频系数；2.34.1) The 256×192 high-resolution image obtained from step 2.33) is used as the 256×192 low-resolution image in this step, and its pixel value is directly used as the low-frequency coefficient of the reconstructed 512×384 high-resolution image;

2.34.2）对256×192低分辨率图像进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解提取得到其高频系数；2.34.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the 256×192 low-resolution image, take the average of the two, and use the 9-7 filter bank to perform Laplacian pyramid decomposition to extract its high-frequency coefficients;

2.34.3）将256×192低分辨率图像分割为大小为s×t的小块，s取值范围为4~8，t取值范围为8~12，相邻小块间有（s-1）×（t-1）的重叠；2.34.3) Divide the 256×192 low-resolution image into small blocks with a size of s×t. The value range of s is 4~8, and the value range of t is 8~12. There are (s- 1) × (t-1) overlap;

2.34.4）利用LLE学习得到2.34.3）中分割出的小块对应位置的512×384高分辨率小块的高频系数：2.34.4) Use LLE learning to obtain the high-frequency coefficient of the 512×384 high-resolution small block corresponding to the small block segmented in 2.34.3):

令y_LR为单个于2.34.3）中得到小块对应的于2.34.2）中得到的高频系数组成的向量，x_LR(i)为第i个256×192清晰的分辨率样本相应位置小块的高频系数组成的向量，在256×192清晰的分辨率样本相应位置小块中取前k个（5<k<8）与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.34.2) corresponding to a small block obtained in 2.34.3), and x _LR (i) be the corresponding position of the ith 256×192 clear resolution sample The vector composed of the high-frequency coefficients of the small blocks, the first k (5<k<8) sample small blocks with the smallest y _LR mean square error are selected from the corresponding position small blocks of the 256×192 clear resolution samples, and the calculation is as follows: Value question:

令G＝(εy_LR1^T-X)^T(εy_LR1^T-X)，其中X的每列为x_LR(i)，共k列，1为元素皆为1的向量，w(i)为k个待求系数，此极值问题的解为W为w(i)组成的向量，进而计算出学习得到的512×384高分辨率图像小块的高频系数向量为：Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1, w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i), and then the high-frequency coefficient vector of the learned 512×384 high-resolution image block is calculated as:

其中x_HR(i)为x_LR(i)对应的512×384清晰分辨率样本小块的高频系数向量；Where x _HR (i) is the high-frequency coefficient vector of the 512×384 clear resolution sample block corresponding to x _LR (i);

2.34.5）重复2.34.4）得到所有2.34.3）中分割出的小块对应的512×384高分辨率小块的高频系数2.34.5) Repeat 2.34.4) to obtain high-frequency coefficients of 512×384 high-resolution small blocks corresponding to all small blocks segmented in 2.34.3)

2.34.6）将2.34.5）中学习得到的512×384高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为最终结果，得到512×384高分辨率图像的高频系数；2.34.6) The 512×384 high-resolution small blocks learned in 2.34.5) are stitched together, and the overlapping parts between adjacent small blocks are taken as the average value of all the overlapping small blocks as the final result to obtain a 512×384 high-resolution High-frequency coefficients of high-rate images;

将得到的512×384高分辨率图像的高、低频系数进行反变换便得到最终重建出的512×384高分辨率图像。The high- and low-frequency coefficients of the obtained 512×384 high-resolution image are inversely transformed to obtain the final reconstructed 512×384 high-resolution image.

本发明的特点及效果：Features and effects of the present invention:

本发明首先采用低分辨率人脸图像的归一化方法，特别是采用了基于人脸三点定位归一化方法的低分辨率人脸图像的归一化方法（见发明专利：基于人脸几何尺寸归一化的人脸图像识别方法，发明专利号：ZL 2005 1 0067692.X，），这种人脸图像的归一化方法不仅准确，而且和训练集的归一化方法一致，提升了低分辨率人脸图像的准确性。获得归一化的低分辨率人脸图像后，再采用基于低频分量的超分辨率人脸图像的重建方法，制作出低分辨率人脸图像的大尺寸重建像，较好地解决了低分辨率人脸图像的重建难题。The present invention first adopts the normalization method of low-resolution face images, especially the normalization method of low-resolution face images based on the normalization method of three-point positioning of the face (see invention patent: based on face Face image recognition method with geometric size normalization, invention patent number: ZL 2005 1 0067692.X,), the normalization method of this face image is not only accurate, but also consistent with the normalization method of the training set, improving improved the accuracy of low-resolution face images. After obtaining the normalized low-resolution face image, the super-resolution face image reconstruction method based on low-frequency components is used to produce a large-scale reconstruction image of the low-resolution face image, which better solves the problem of low-resolution face image reconstruction. The reconstruction problem of human face images.

附图说明 Description of drawings

图1为本发明的低分辨率人脸实施例图像。Fig. 1 is an embodiment image of a low-resolution face of the present invention.

图2为本实施例裁剪的低分辨率人脸图像。FIG. 2 is a low-resolution human face image cropped in this embodiment.

图3为本实施例的裁剪的低分辨率人脸的放大图像。FIG. 3 is an enlarged image of a cropped low-resolution human face in this embodiment.

图4为本实施例的低分辨率人脸的放大图像的三点定位图像。FIG. 4 is a three-point positioning image of an enlarged image of a low-resolution human face in this embodiment.

图5为本实施例的基于人脸三点定位的低分辨率人脸图像的归一化图像，图像尺寸为32×24。FIG. 5 is a normalized image of a low-resolution face image based on three-point face positioning in this embodiment, and the image size is 32×24.

图6为本实施例的低分辨率人脸图像的重建像。FIG. 6 is a reconstructed image of the low-resolution face image of this embodiment.

图7为本实施例的低分辨率人脸对应的真实清晰人脸图像。FIG. 7 is a real and clear face image corresponding to a low-resolution face in this embodiment.

具体实施方式 Detailed ways

本发明提出的基于超分辨率重建技术的低分辨率人脸图像的重建方法结合附图及实施例详细说明如下：The reconstruction method of the low-resolution face image based on the super-resolution reconstruction technology proposed by the present invention is described in detail as follows in conjunction with the accompanying drawings and embodiments:

本发明提出的一种基于超分辨率重建技术的低分辨率人脸图像的重建方法，包括低分辨率人脸图像的归一化和基于低频分量的超分辨率重建；具体包括以下步骤：A method for reconstructing a low-resolution face image based on a super-resolution reconstruction technology proposed by the present invention includes normalization of the low-resolution face image and super-resolution reconstruction based on low-frequency components; specifically comprises the following steps:

1）对原始图像进行基于人脸三点定位的归一化，得到待重建的归一化人脸图像1) Normalize the original image based on three-point positioning of the face to obtain a normalized face image to be reconstructed

1.1）对含有低分辨率人脸图像的原始图像进行裁剪，裁剪出包含低分辨率人脸图像的矩形图像区域，得到低分辨率人脸图像；该区域左上、左下、右上、右下4个顶点坐标分别为：（m₁，n₁）、（m₁，n₂）、（m₂，n₁）、（m₂，n₂），m、n分别表示坐标系中点的横坐标和纵坐标；1.1) Crop the original image containing the low-resolution face image, cut out the rectangular image area containing the low-resolution face image, and obtain the low-resolution face image; the area has 4 upper left, lower left, upper right, and lower right Vertex coordinates are: (m ₁ , n ₁ ), (m ₁ , n ₂ ), (m ₂ , n ₁ ), (m ₂ , n ₂ ), m and n represent the abscissa and Y-axis;

1.2）对1.1）得到的低分辨率人脸图像进行等比放大，放大倍率为K×K，放大后图像的高度为L，L＝K×(n₂-n₁)，k满足：K≥150/(n₂-n₁)；1.2) The low-resolution face image obtained in 1.1) is enlarged proportionally, the magnification is K×K, the height of the enlarged image is L, L=K×(n ₂ -n ₁ ), k satisfies: K≥ 150/(n ₂ -n ₁ );

1.3）对放大后的低分辨率人脸图像采用基于人脸三点定位的归一化方法进行归一化，得到一幅待重建的标准尺寸的人脸图像；1.3) Normalize the enlarged low-resolution face image using a normalization method based on three-point positioning of the face to obtain a face image of a standard size to be reconstructed;

1.31）在放大后的低分辨率人脸图像a上确定左眼球上的一点A的坐标（x₁₁,y₁₁）、右眼球上的一点B的坐标位置（x₂₁,y₂₁），通过A、B两点做直线L₁，并确定下颌点C₀坐标（x₀₁,y₀₁），，按放大倍率k将三点的坐标对应至原裁剪图像坐标，分别为（x₁,y₁），（x₂,y₂），（x₀,y₀）；1.31) Determine the coordinates (x ₁₁ , y ₁₁ ) of a point A on the left eyeball and the coordinate position (x ₂₁ , y ₂₁ ) of a point B on the right eyeball on the enlarged low-resolution face image a, and pass A and B to make a straight line L ₁ , and determine the coordinates of the mandibular point C ₀ (x ₀₁ , y ₀₁ ), and correspond the coordinates of the three points to the coordinates of the original cropped image according to the magnification k, respectively (x ₁ , y ₁ ) , (x ₂ , y ₂ ), (x ₀ , y ₀ );

1.32）对原始图像进行基于人脸几何尺寸归一化的人脸图像识别方法中的人脸三点归一化方法，得到尺寸为H×W的待重建的归一化人脸图像，具体包括(属于已有技术):1.32) Perform the face three-point normalization method in the face image recognition method based on face geometric size normalization on the original image to obtain a normalized face image with a size of H×W to be reconstructed, specifically including (Belongs to prior art):

1.32.1）计算直线L₁和水平线的夹角α；直线L1和水平线的夹角α由下式求得，其中(x₁,y₁)，(x₂,y₂)分别对应左右眼球坐标：1.32.1) Calculate the angle α between the straight line _L1 and the horizontal line; the angle α between the straight line L1 and the horizontal line is obtained by the following formula, where (x ₁ , y ₁ ), (x ₂ , y ₂ ) correspond to the coordinates of the left and right eyeballs respectively :

对该人脸图像a进行旋转角度为α的旋转处理，得到人脸图像b；旋转表达式如下： The face image a is rotated at a rotation angle of α to obtain the face image b; the rotation expression is as follows:

$[\begin{matrix} {x x}^{' '} \\ {y the y}^{' '} \end{matrix}] = = [\begin{matrix} cos cos α α & sin sin α α \\ - - sin sin α α & cos cos α α \end{matrix}] [\begin{matrix} x x \\ y the y \end{matrix}]$

式中，x、y为输入人脸图像的坐标，x、y为人脸图像b的坐标；In the formula, x, y are the coordinates of the input face image, and x, y are the coordinates of the face image b;

1.32.2）在人脸图像b上计算确定出左眼球上的一点C的坐标位置（x₃,y₃）、右眼球上的一点D的坐标位置（x₄,y₄），通过C、D两点做直线L₂，确定出旋转后的人脸图像的下颌点E的坐标位置（x₅,y₅）；1.32.2) Calculate and determine the coordinate position (x ₃ , y ₃ ) of a point C on the left eyeball and the coordinate position (x ₄ , y ₄ ) of a point D on the right eyeball on the face image b. Through C, Two points in D make a straight line L ₂ to determine the coordinate position (x ₅ , y ₅ ) of the mandible point E of the rotated face image;

1.32.3）规定几何尺寸归一化的人脸图像的几何尺寸的数值，其中宽度的尺寸为W，高度的尺寸为H；规定颌下线上的任何一点到两眼连线的垂直距离的标准值为H₀，到图像下边框的垂直距离的标准值为H₁，两眼连线到图像上边框的垂直距离的标准值为H₂；H，W满足下式：1.32.3) Specify the numerical value of the geometric dimension of the normalized face image, where the width dimension is W and the height dimension is H; specify the vertical distance from any point on the submandibular line to the line connecting the two eyes The standard value is H ₀ , the standard value of the vertical distance to the lower border of the image is H ₁ , the standard value of the vertical distance from the line connecting the two eyes to the upper border of the image is H ₂ ; H, W satisfy the following formula:

W=H×3/4，H₀=5×H/12，H₁=63×H/120，H₂=7×H/120；W=H×3/4, H ₀ =5×H/12, H ₁ =63×H/120, H ₂ =7×H/120;

1.32.4）求出E点到直线L2的垂直距离h_y，并计算图像放缩系数K＝h_y/H₀；其中，E点到直线L2的垂直距离h_y为1.32.4) Calculate the vertical distance h _y from point E to straight line L2, and calculate the image scaling factor K=h _y /H ₀ ; where, the vertical distance h _y from point E to straight line L2 is

${h h}_{y the y} = = {y the y}_{55} - - \frac{{y the y}_{33} + + {y the y}_{44}}{22}$

1.32.5）对人脸图像b按照放缩系数K进行放大或缩小处理，得到满足标准距离H₀的人脸图像c；1.32.5) Enlarge or reduce the face image b according to the scaling factor K, and obtain the face image c satisfying the standard distance _H0 ;

1.32.6）在人脸图像c上确定出左眼球上的一点M的坐标位置（x₆,y₆）、右眼球上的一点N的坐标位置（x₇,y₇），以及下颌点P的纵坐标y₈位置；y₈＝MidPoint.y+H₀；1.32.6) Determine the coordinate position (x ₆ , y ₆ ) of a point M on the left eyeball, the coordinate position (x ₇ , y ₇ ) of a point N on the right eyeball, and the mandibular point P on the face image c The vertical coordinate y ₈ position; y ₈ =MidPoint.y+H ₀ ;

1.32.7）对人脸图像c进行裁减得到标准的待重建的归一化人脸图像d：裁去人脸图像c中x坐标小于(x₆+x₇)/2-W/2、大于(x₆+x₇)/2+W/2的部分，以及y坐标小于(y₇-H₂)、大于(y₈+H₁)的部分；若裁减后图像的宽度小于W或者高度小于H，则采用插值的方法，将宽度补到W或者高度补到H；1.32.7) Cut the face image c to obtain the standard normalized face image d to be reconstructed: the x coordinate in the cut face image c is less than (x ₆ +x ₇ )/2-W/2, greater than The part of (x ₆ +x ₇ )/2+W/2, and the part whose y coordinate is less than (y ₇ -H ₂ ) and greater than (y ₈ +H ₁ ); if the width of the cropped image is less than W or the height is less than H, the interpolation method is used to fill the width to W or the height to H;

2.1）生成训练集:2.1) Generate training set:

2.15）对32×24，64×48，128×96，256×192四种清晰的分辨率样本进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解提取它们的高频分量并保存，作为清晰的低分辨率图像小波系数样本，同时对模糊的32×24分辨率样本进行最邻近插值与双三次线性插值，取两者平均，用9-7滤波器组进行拉普拉斯金字塔分解提取其高频分量并保存，作为模糊的32×24低分辨率图像小波系数样本2.15) Perform nearest neighbor interpolation and bicubic linear interpolation on four clear resolution samples of 32×24, 64×48, 128×96, and 256×192, take the average of the two, and perform Lap with a 9-7 filter bank Las Pyramid decomposes to extract their high-frequency components and save them as clear low-resolution image wavelet coefficient samples. At the same time, the nearest neighbor interpolation and bicubic linear interpolation are performed on the fuzzy 32×24 resolution samples, and the average of the two is used. The 9-7 filter bank performs Laplacian pyramid decomposition to extract its high-frequency components and save them as wavelet coefficient samples of fuzzy 32×24 low-resolution images

$u u = = \frac{{Σ Σ}_{h h}}{{Σ Σ}_{i i}} - - - - - - ((22))$

2.3）通过LLE学习得到氟化钙的高分辨率图像：2.3) High-resolution images of calcium fluoride obtained through LLE learning:

2.31）从32×24到64×48放大:2.31) Zoom in from 32×24 to 64×48:

2.31.3）将32×24低分辨率图像分割为大小为s×t的小块，s取值范围为4~8，t取值范围为8~12，相邻小块间有（s-1）×（t-1）的重叠；2.31.3) Divide the 32×24 low-resolution image into small blocks with a size of s×t. The value range of s is 4~8, and the value range of t is 8~12. There are (s- 1) × (t-1) overlap;

2.32）从64×48到128×96放大：2.32) Zoom in from 64×48 to 128×96:

2.32.4）利用LLE学习得到2.32.3）中分割出的小块对应位置的128×96高分辨率小块的高频系数：2.32.4) Use LLE learning to obtain the high-frequency coefficients of the 128×96 high-resolution small blocks corresponding to the small blocks segmented in 2.32.3):

令y_LR为单个于2.3.1.3）中得到小块对应的于2.32.2）中得到的高频系数组成的向量，x_LR(i)为第i个64×48清晰的分辨率样本集相应位置小块的高频系数组成的向量，在64×48清晰的分辨率样本集相应位置的小块中取前k个（5<k<8）与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.32.2) corresponding to a small block obtained in 2.3.1.3), and x _LR (i) be the i-th 64×48 clear resolution sample set corresponding to The vector composed of the high-frequency coefficients of the position small blocks, the first k (5<k<8) sample blocks with the smallest y _LR mean square error are selected from the small blocks at the corresponding positions of the 64×48 clear resolution sample set, Calculate the following extremum problem:

2.32.6）将2.32.5）中学习得到的128×96高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为结果，得到128×96高分辨率图像的高频系数；2.32.6) The 128×96 high-resolution small blocks learned in 2.32.5) are spliced, and the overlap between adjacent small blocks is taken as the average value of all small blocks participating in the overlap, and a 128×96 high-resolution High frequency coefficient of the image;

2.33）从128×96到256×192放大：2.33) Zoom in from 128×96 to 256×192:

2.33.3）将128×96低分辨率图像分割为大小为s×t的小块，s取值范围为4~8，t取值范围为8~12，相邻小块间有（s-1）×（t-1）的重叠；2.33.3) Divide the 128×96 low-resolution image into small blocks with a size of s×t, the value range of s is 4~8, the value range of t is 8~12, and there are (s- 1) × (t-1) overlap;

2.34）从256×192到512×384放大：2.34) Zoom in from 256×192 to 512×384:

2.34.7）将得到的512×384高分辨率图像的高、低频系数进行反变换便得到最终重建出的512×384高分辨率图像；2.34.7) Inversely transform the high and low frequency coefficients of the obtained 512×384 high-resolution image to obtain the final reconstructed 512×384 high-resolution image;

实施例Example

1.1）对含有低分辨率人脸图像的原始图像进行裁剪，裁剪出包含低分辨率人脸图像的矩形图像区域，得到低分辨率人脸图像，如图2所示，该区域左上、左下、右上、右下4个顶点坐标分别为：（898，526）、（898，556）、（924，526）、（924，556）；1.1) Cut out the original image containing the low-resolution face image, cut out the rectangular image area containing the low-resolution face image, and obtain the low-resolution face image, as shown in Figure 2, the upper left, lower left, and The coordinates of the upper right and lower right vertices are: (898,526), (898,556), (924,526), (924,556);

1.2）对1.1）得到的低分辨率人脸图像进行等比放大，放大倍率为6×6，放大后图像的高度为L=180，如图3所示；1.2) The low-resolution face image obtained in 1.1) is enlarged proportionally, the magnification is 6×6, and the height of the enlarged image is L=180, as shown in Figure 3;

1.3）对放大后的低分辨率人脸图像采用基于人脸三点定位归一化的方法进行归一化，得到一幅待重建的标准尺寸的人脸图像如图5所示；1.3) Normalize the amplified low-resolution face image using the normalization method based on three-point positioning of the face, and obtain a standard-sized face image to be reconstructed, as shown in Figure 5;

1.31）在放大后的低分辨率人脸图像a上确定左眼球上的一点A的坐标（63，104）、右眼球上的一点B的坐标位置（112，105），通过A、B两点做直线L₁，并确定下颌点C₀坐标（87，174），如图4所示，按放大倍率6将三点的坐标对应至原裁剪图像坐标，分别为（908，543），（916，543），（912，555）；1.31) Determine the coordinates (63, 104) of a point A on the left eyeball and the coordinates (112, 105) of a point B on the right eyeball on the enlarged low-resolution face image a, and pass through the two points A and B Make a straight line L ₁ , and determine the coordinates (87, 174) of the mandibular point C ₀ , as shown in Figure 4, and correspond the coordinates of the three points to the coordinates of the original cropped image according to the magnification 6, which are (908, 543), (916 , 543), (912, 555);

1.32）对原始图像进行基于人脸几何尺寸归一化的人脸图像识别方法中的人脸三点归一化方法，得到尺寸为32×24的待重建的归一化人脸图像，具体包括(属于已有技术)1.32) Perform the face three-point normalization method in the face image recognition method based on face geometric size normalization on the original image to obtain a normalized face image with a size of 32×24 to be reconstructed, specifically including (belongs to existing technology)

1.32.1）计算直线L₁和水平线的夹角α；1.32.1) Calculate the angle α between the straight line _L1 and the horizontal line;

直线L1和水平线的夹角α由下式求得，其中(x₁,y₁)，(x₂,y₂)分别对应左右眼球坐标：The angle α between the straight line L1 and the horizontal line is obtained by the following formula, where (x ₁ , y ₁ ), (x ₂ , y ₂ ) correspond to the coordinates of the left and right eyeballs respectively:

$α α = = arctan arctan ((\frac{{y the y}_{22} - - {y the y}_{11}}{{x x}_{22} - - {x x}_{11}}))$

1.32.2）对该人脸图像a进行旋转角度为α的旋转处理，得到人脸图像b；旋转表达式如下：1.32.2) Rotate the face image a with a rotation angle of α to obtain face image b; the rotation expression is as follows:

1.32.3）在人脸图像b上计算确定出左眼球上的一点C的坐标位置（x₃,y₃）、右眼球上的一点D的坐标位置（x₄,y₄），通过C、D两点做直线L₂，确定出旋转后的人脸图像的下颌点E的坐标位置（x₅,y₅）；1.32.3) Calculate and determine the coordinate position (x ₃ , y ₃ ) of a point C on the left eyeball and the coordinate position (x ₄ , y ₄ ) of a point D on the right eyeball on the face image b. Through C, Two points in D make a straight line L ₂ to determine the coordinate position (x ₅ , y ₅ ) of the mandible point E of the rotated face image;

1.32.4）规定几何尺寸归一化的人脸图像的几何尺寸的数值，其中宽度的尺寸为24，高度的尺寸为32；规定颌下线上的任何一点到两眼连线的垂直距离的标准值为H₀，到图像下边框的垂直距离的标准值为H₁，两眼连线到图像上边框的垂直距离的标准值为H₂；1.32.4) Specify the value of the geometric size of the normalized face image, where the width is 24 and the height is 32; the vertical distance between any point on the submandibular line and the line connecting the eyes The standard value is H ₀ , the standard value of the vertical distance to the lower border of the image is H ₁ , the standard value of the vertical distance from the line connecting the two eyes to the upper border of the image is H ₂ ;

1.32.5）求出E点到直线L2的垂直距离h_y，并计算图像放缩系数K＝h_y/H₀；其中，E点到直线L2的垂直距离h_y为1.32.5) Calculate the vertical distance h _y from point E to straight line L2, and calculate the image scaling factor K=h _y /H ₀ ; where, the vertical distance h _y from point E to straight line L2 is

1.32.6）对人脸图像b按照放缩系数K进行放大或缩小处理，得到满足标准距离H₀的人脸图像c；1.32.6) Enlarge or reduce the face image b according to the scaling factor K, and obtain the face image c satisfying the standard distance _H0 ;

1.32.7）在人脸图像c上确定出左眼球上的一点M的坐标位置（10，17）、右眼球上的一点N的坐标e位置（18，17），以及下颌点P的纵坐标y₈位置；y₈＝29；1.32.7) Determine the coordinate position (10, 17) of a point M on the left eyeball, the coordinate e position (18, 17) of a point N on the right eyeball, and the vertical coordinate of the mandibular point P on the face image c y ₈ position; y ₈ =29;

1.32.8）对人脸图像c进行裁减得到标准的待重建的归一化人脸图像d：裁去人脸图像c中x坐标小于(x₆+x₇)/2-W/2、大于(x₆+x₇)/2+W/2的部分，以及y坐标小于(y₇-H₂)、大于(y₈+H₁)的部分；若裁减后图像的宽度小于W或者高度小于H，则采用插值的方法，将宽度补到W或者高度补到H；1.32.8) Cut the face image c to obtain the standard normalized face image d to be reconstructed: the x coordinate in the cut face image c is less than (x ₆ +x ₇ )/2-W/2, greater than The part of (x ₆ +x ₇ )/2+W/2, and the part whose y coordinate is less than (y ₇ -H ₂ ) and greater than (y ₈ +H ₁ ); if the width of the cropped image is less than W or the height is less than H, the interpolation method is used to fill the width to W or the height to H;

2.1）生成训练集:2.1) Generate training set:

2.11）用100幅的清晰正面人脸图像作为训练集的图像；2.11) Use 100 clear frontal face images as the images of the training set;

对待重建图像进行拉普拉斯金字塔分解，计算出其所有高频系数绝对值的中位数σ作为噪声标准差估计值，根据经典去噪算法以λσ为阈值进行去噪，λ=0.15；Laplacian pyramid decomposition is performed on the image to be reconstructed, and the median σ of the absolute value of all high-frequency coefficients is calculated as the estimated value of the noise standard deviation. According to the classic denoising algorithm, λσ is used as the threshold for denoising, λ=0.15;

$u u = = \frac{{Σ Σ}_{h h}}{{Σ Σ}_{i i}} = = 0.03 0.03,, - - - - - - ((11))$

因为u＝0.03＜0.05，学习采用模糊的32×24样本；Because u=0.03<0.05, learning uses fuzzy 32×24 samples;

2.3）通过LLE学习得到最终重建的高分辨率图像：2.3) Get the final reconstructed high-resolution image through LLE learning:

2.31）从32×24到64×48放大:2.31) Zoom in from 32×24 to 64×48:

2.31.1）将于2.2.1）中得到的去噪后的32×24低分辨率图像的像素值直接作为重建出的64×48高分辨率图像的低频系数；2.31.1) The pixel values of the denoised 32×24 low-resolution image obtained in 2.2.1) are directly used as the low-frequency coefficients of the reconstructed 64×48 high-resolution image;

2.31.3）将32×24低分辨率图像分割为大小为5×10的小块，相邻小块间有4×9的重叠；2.31.3) Divide the 32×24 low-resolution image into small blocks with a size of 5×10, and there is a 4×9 overlap between adjacent small blocks;

令y_LR为单个于2.31.3）中得到小块对应的于2.31.2）中得到的高频系数组成的向量，x_LR(i)为第i个在模糊的32×24样本集中相应位置小块的高频系数组成的向量，在模糊的32×24样本集相应位置的小块中取前k=6个与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.31.2) corresponding to the small block obtained in 2.31.3), and x _LR (i) be the i-th corresponding position in the fuzzy 32×24 sample set The vector composed of the high-frequency coefficients of the small blocks, in the small blocks corresponding to the fuzzy 32×24 sample set, take the first k=6 sample small blocks with the smallest y _LR mean square error, and calculate the following extreme value problem:

2.32）从64×48到128×96放大：2.32) Zoom in from 64×48 to 128×96:

2.32.3）将64×48低分辨率图像分割为大小为5×10的小块，相邻小块间有4×9的重叠；2.32.3) Divide the 64×48 low-resolution image into small blocks with a size of 5×10, and there is a 4×9 overlap between adjacent small blocks;

令y_LR为单个于2.31.3）中得到小块对应的于2.32.2）中得到的高频系数组成的向量，x_LR(i)为第i个64×48清晰的分辨率样本集相应位置小块的高频系数组成的向量，在64×48清晰的分辨率样本集相应位置的小块中取前k=6个与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.32.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the i-th 64×48 clear resolution sample set corresponding to The vector composed of the high-frequency coefficients of the position small blocks, in the small blocks of the corresponding position of the 64×48 clear resolution sample set, take the first k=6 sample blocks with the smallest mean square error of y _LR , and calculate the following extreme value problem :

2.33）从128×96到256×192放大：2.33) Zoom in from 128×96 to 256×192:

2.33.3）将128×96低分辨率图像分割为大小为5×10的小块，相邻小块间有4×9的重叠；2.33.3) Divide the 128×96 low-resolution image into small blocks with a size of 5×10, and there is a 4×9 overlap between adjacent small blocks;

令y_LR为单个于2.31.3）中得到小块对应的于2.33.2）中得到的高频系数组成的向量，x_LR(i)为第i个128×96清晰的分辨率样本相应位置小块的高频系数组成的向量，在128×96清晰的分辨率样本相应位置的小块中取前k=6个与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.33.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the corresponding position of the ith 128×96 clear resolution sample The vector composed of the high-frequency coefficients of small blocks, in the small blocks corresponding to the 128×96 clear resolution samples, take the first k=6 small sample blocks with the smallest mean square error of y _LR , and calculate the following extreme value problem:

2.33.6）将2.33.5）中学习得到的256×192高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为最终结果，得到256×192高分辨率图像的高频系数；2.33.6) The 256×192 high-resolution small blocks learned in 2.33.5) are stitched together, and the overlap between adjacent small blocks is taken as the average value of all overlapping small blocks as the final result to obtain a 256×192 high-resolution High-frequency coefficients of high-rate images;

2.34）从256×192到512×384放大：2.34) Zoom in from 256×192 to 512×384:

2.34.3）将256×192低分辨率图像分割为大小为5×10的小块，相邻小块间有4×9的重叠；2.34.3) Divide the 256×192 low-resolution image into small blocks with a size of 5×10, and there is an overlap of 4×9 between adjacent small blocks;

令y_LR为单个于2.34.3）中得到小块对应的于2.34.2）中得到的高频系数组成的向量，x_LR(i)为第i个256×192清晰的分辨率样本相应位置小块的高频系数组成的向量，在256×192清晰的分辨率样本相应位置小块中取前k=6个与y_LR均方误差最小的样本小块，计算如下极值问题：Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.34.2) corresponding to a small block obtained in 2.34.3), and x _LR (i) be the corresponding position of the ith 256×192 clear resolution sample The vector composed of the high-frequency coefficients of the small blocks, in the small blocks of the corresponding positions of the 256×192 clear resolution samples, take the first k=6 small sample blocks with the smallest mean square error of y _LR , and calculate the following extreme value problem:

2.34.6）将2.34.5）中学习得到的512×384高分辨率小块进行拼接，相邻小块间重叠部分取所有参与重叠的小块的均值作为最终结果，得到512×384高分辨率图像的高频系数；2.34.6) The 512×384 high-resolution small blocks learned in 2.34.5) are stitched together, and the overlapping parts between adjacent small blocks are taken as the average value of all overlapping small blocks as the final result to obtain a 512×384 high-resolution High-frequency coefficients of high-rate images;

2.34.7）将得到的512×384高分辨率图像的高、低频系数进行反变换便得到最终重建出的512×384高分辨率图像，如图6所示；2.34.7) Inversely transform the high- and low-frequency coefficients of the obtained 512×384 high-resolution image to obtain the final reconstructed 512×384 high-resolution image, as shown in Figure 6;

将将上述方法，按照低分辨率人脸图像的裁剪、低分辨率人脸图像的放大、三点定位归一化、重建、存储的步骤编制程序，实现超分辨率重建技术的低分辨率人脸图像的重建，得到最终重建结果如图6所示。低分辨率人脸对应的真实清晰人脸图像如图7所示。从图6、图7中可以看到，本发明效果明显。The above method will be programmed according to the steps of cropping of low-resolution face image, enlargement of low-resolution face image, normalization of three-point positioning, reconstruction and storage, so as to realize the low-resolution face image of super-resolution reconstruction technology. The face image is reconstructed, and the final reconstruction result is shown in Figure 6. The real and clear face image corresponding to the low-resolution face is shown in Figure 7. As can be seen from Fig. 6 and Fig. 7, the effect of the present invention is obvious.

Claims

1. A reconstruction method of a low-resolution face image based on super-resolution reconstruction technology, characterized in that, comprising the following steps:

1) Normalize the original image based on three-point positioning of the face to obtain a normalized face image to be reconstructed, specifically including:

1.1) Crop the original image containing the low-resolution face image, cut out a rectangular area containing the low-resolution face image, and obtain the low-resolution face image; the upper left, lower left, upper right, and lower right vertices of this area The coordinates in the original image are: (m ₁ , n ₁ ), (m ₁ , n ₂ ), (m ₂ , n ₁ ), (m ₂ , n ₂ ), and m and n respectively represent points in the coordinate system The abscissa and ordinate of ;

1.2) Enlarge the low-resolution face image obtained in 1.1), the magnification is K×K, the height of the enlarged image is L, L=K×(n ₂ -n ₁ ), K satisfies: K ≥150/(n ₂ -n ₁ );

1.3) Normalize the enlarged low-resolution face image using a normalization method based on three-point positioning of the face to obtain a normalized face image of a standard size to be reconstructed;

1.31) On the enlarged low-resolution face image a, determine the coordinates (x ₁₁ , y ₁₁ ) of a point A on the left eyeball and the coordinate position (x ₂₁ , y ₂₁ ) of a point B on the right eyeball, and pass A and B to make a straight line L ₁ , and determine the coordinates (x ₀₁ , y ₀₁ ) of the mandibular point C ₀ , and correspond the coordinates of the three points to the coordinates of the original cropped image according to the magnification K, respectively (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₀ , y ₀ );

1.32) Carry out the face three-point normalization method in the face image recognition method based on face geometric size normalization to the original image, and obtain a normalized face image to be reconstructed with a size of H×W, where H , W satisfies the following formula:

H h = = \{\begin{matrix} 6464,, & (({n no}_{22} - - {n no}_{11})) &GreaterEqual; &Greater Equal; 6464 \\ 3232,, & 6464 > > (({n no}_{22} - - {n no}_{11})) &GreaterEqual; &Greater Equal; 3232 \\ 1616,, & 3232 > > (({n no}_{22} - - {n no}_{11})) &GreaterEqual; &Greater Equal; 1616 \end{matrix},, - - - - - - ((11))

W=H×3/4;

2) Reconstruct the normalized face image of the standard size to be reconstructed based on the super-resolution face image of the low-frequency component to obtain a 512×384 high-resolution image:

2.1) Generate training set:

2.11) Use no less than 100 clear frontal face images as the images of the training set;

2.12) Use the normalization method of three-point positioning of the face to convert each image in the training set into a standard resolution face image of 512×384;

2.13) Use a 9-7 filter to downsample each 512×384 standard resolution image to generate five clear resolution samples of 32×24, 64×48, 128×96, 256×192, and 512×384 , and then use the Gaussian pyramid filter to downsample each standard resolution image of 512×384 to generate a blurred resolution sample of 32×24;

2.14) For four clear resolution samples of 64×48, 128×96, 256×192, and 512×384, use a 9-7 filter bank to perform Laplacian pyramid decomposition to extract high-frequency components and save them as high-resolution rate image wavelet coefficient samples;

2.15) Perform nearest neighbor interpolation and bicubic linear interpolation on four clear resolution samples of 32×24, 64×48, 128×96, and 256×192, take the average of the two, and use the 9-7 filter bank for Lapp Lass pyramid decomposes to extract their high-frequency components and save them as clear low-resolution image wavelet coefficient samples. At the same time, perform nearest neighbor interpolation and bicubic linear interpolation on blurred 32×24 resolution samples, take the average of the two, and use The 9-7 filter bank performs Laplacian pyramid decomposition to extract its high-frequency components and save them as wavelet coefficient samples of fuzzy 32×24 low-resolution images;

2.2) Denoise the low-resolution image to be reconstructed and determine the learning samples by ambiguity estimation:

2.21) Denoising the normalized 32×24 low-resolution image to be reconstructed; the steps are as follows:

The image to be reconstructed is decomposed into a Laplacian pyramid, and the median σ of the absolute values of all high-frequency coefficients is calculated as the estimated value of the noise standard deviation. According to the classic denoising algorithm, λ _σ is used as the threshold for denoising, and λ is the empirical coefficient , the value range is 0.05<λ<0.3;

2.22) Estimate the degree of fuzziness and calculate the degree of fuzziness coefficient:

u u = = \frac{{Σ Σ}_{h h}}{{Σ Σ}_{i i}},, - - - - - - ((22))

Σ _h is the sum of the absolute values of the high-frequency coefficients of the low-resolution image to be reconstructed, and Σ _i is the sum of the pixel values of the 32×24 low-resolution image to be reconstructed after denoising;

2.23) Select learning samples according to the fuzzy degree coefficient u:

When u<0.05, learning uses fuzzy 32×24 samples, and when u≥0.05 uses clear 32×24 samples;

2.3) The final reconstructed high-resolution image is obtained through LLE learning:

2.31) Zoom in from 32×24 to 64×48:

2.31.1) The pixel values of the denoised 32×24 low-resolution image obtained in 2.21) are directly used as the low-frequency coefficients of the reconstructed 64×48 high-resolution image;

2.31.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the denoised 32×24 low-resolution image, take the average of the two, and use the 9-7 filter bank to decompose the Laplacian pyramid to obtain its high-frequency coefficients ;

2.31.3) Divide the denoised 32×24 low-resolution image into small blocks with a size of s×t. The range of s is 4 to 8, and the range of t is 8 to 12. Between adjacent small blocks There is an overlap of (s-1)×(t-1);

2.31.4) Use LLE learning to obtain the high-frequency coefficients of the 64×48 high-resolution small blocks corresponding to the small blocks segmented in 2.31.3):

Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.31.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the corresponding position of the ith sample set determined in 2.23). The vector composed of the high-frequency coefficients of the block, in the small block of the corresponding position of the sample set determined in 2.23), take the first k (5<k<8) sample small blocks with the smallest y _LR mean square error, and calculate the following extreme value problem :

W W = = \underset{w w ((i i))}{arg arg min min} {| | | | {ϵy ϵy}_{LR LR} - - {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{LR LR} ((i i)) | | | |}^{22},, {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) = = 11 - - - - - - ((33))

Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1; w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i); and then the high-frequency coefficient vector of the learned 64×48 high-resolution image block is calculated as:

{y the y}_{HR HR} = = \frac{{Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{HR HR} ((i i))}{ϵ ϵ} - - - - - - ((44))

Where x _HR (i) is the high-frequency coefficient vector of the 64×48 clear resolution sample block corresponding to x _LR (i);

If light treatment is carried out, in (3) and (4) two formulas If no light treatment is carried out, ε=1 in the two formulas (3) and (4);

2.31.5) Repeat 2.31.4) to obtain the high-frequency coefficients of the 64×48 high-resolution small blocks corresponding to all the small blocks segmented in 2.31.3);

2.31.6) Splice the 64×48 high-resolution small blocks learned in 2.31.5), and take the average value of all overlapping small blocks as the result for the overlap between adjacent small blocks, and obtain a 64×48 high-resolution High frequency coefficient of the image;

2.31.7) Inversely transform the high and low frequency coefficients of the obtained 64×48 high-resolution image to obtain a 64×48 high-resolution image;

2.32) Zoom in from 64×48 to 128×96:

2.32.1) The 64×48 high-resolution image obtained from step 2.31) is used as the 64×48 low-resolution image of this step, and its pixel value is directly used as the low-frequency coefficient of the reconstructed 128×96 high-resolution image;

2.32.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the 64×48 low-resolution image, take the average of the two, and use the 9-7 filter bank to decompose the Laplacian pyramid to obtain its high-frequency coefficients;

2.32.3) Divide the 64×48 low-resolution image into small blocks with a size of s×t. The value of s ranges from 4 to 8, and the value of t ranges from 8 to 12. There are (s- 1) Overlap of ×(t-1);

2.32.4) Use LLE learning to obtain the high-frequency coefficient of the 128×96 high-resolution small block corresponding to the small block segmented in 2.32.3):

Let y _LR be a vector composed of the high-frequency coefficients obtained in 2.32.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the i-th 64×48 clear resolution sample set corresponding to The vector composed of the high-frequency coefficients of the position small blocks, the first k (5<k<8) sample blocks with the smallest y _LR mean square error are selected from the small blocks at the corresponding positions of the 64×48 clear resolution sample set, Calculate the following extremum problem:

W W = = \underset{w w ((i i))}{arg arg min min} {| | | | {ϵy ϵy}_{LR LR} - - {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{LR LR} ((i i)) | | | |}^{22},, {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) = = 11 - - - - - - ((33))

Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1, w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i), and then the high-frequency coefficient vector of the learned 128×96 high-resolution image block is calculated as:

{y the y}_{HR HR} = = \frac{{Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{HR HR} ((i i))}{ϵ ϵ} - - - - - - ((44))

Where x _HR (i) is the high-frequency coefficient vector of the 128×96 clear resolution sample block corresponding to x _LR (i);

2.32.5) Repeat 2.32.4) to obtain high-frequency coefficients of 128×96 high-resolution small blocks corresponding to all the small blocks segmented in 2.32.3);

2.32.6) The 128×96 high-resolution small blocks learned in 2.32.5) are spliced, and the overlap between adjacent small blocks is taken as the average value of all small blocks participating in the overlap, and a 128×96 high-resolution High frequency coefficient of the image;

2.32.7) Inversely transform the high and low frequency coefficients of the obtained 128×96 high-resolution image to obtain a 128×96 high-resolution image;

2.33) Zoom in from 128×96 to 256×192:

2.33.1) The 128×96 high-resolution image obtained from step 2.32) is used as the 128×96 low-resolution image of this step, and its pixel value is directly used as the low-frequency coefficient of the reconstructed 256×192 high-resolution image;

2.33.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the 128×96 low-resolution image, take the average of the two, and use a 9-7 filter bank to perform Laplacian pyramid decomposition to extract its high-frequency coefficients;

2.33.3) Divide the 128×96 low-resolution image into small blocks with a size of s×t. The value of s ranges from 4 to 8, and the value of t ranges from 8 to 12. There are (s- 1) Overlap of ×(t-1);

2.33.4) Use LLE learning to obtain the high-frequency coefficients of the 256×192 high-resolution small blocks corresponding to the small blocks segmented in 2.33.3):

Let y _LR be a vector composed of high-frequency coefficients obtained in 2.33.2) corresponding to a small block obtained in 2.31.3), and x _LR (i) be the corresponding position of the ith 128×96 clear resolution sample The vector composed of the high-frequency coefficients of small blocks, the first k (5<k<8) sample small blocks with the smallest mean square error of y _LR are taken from the small blocks corresponding to the 128×96 clear resolution samples, and the calculation is as follows Extreme value problem:

W W = = \underset{w w ((i i))}{arg arg min min} {| | | | {ϵy ϵy}_{LR LR} - - {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{LR LR} ((i i)) | | | |}^{22},, {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) = = 11 - - - - - - ((33))

Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1, w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i), and then the high-frequency coefficient vector of the learned 256×192 high-resolution image block is calculated as:

{y the y}_{HR HR} = = \frac{{Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{HR HR} ((i i))}{ϵ ϵ} - - - - - - ((44))

Where x _HR (i) is the high-frequency coefficient vector of the 256×192 clear resolution sample block corresponding to x _LR (i);

2.33.5) Repeat 2.33.4) to obtain the high-frequency coefficients of 256×192 high-resolution small blocks corresponding to all small blocks segmented in 2.33.3);

2.33.6) Splice the 256×192 high-resolution small blocks learned in 2.33.5), and take the average value of all overlapping small blocks as the final result for the overlap between adjacent small blocks, and obtain a 256×192 high-resolution High-frequency coefficients of high-rate images;

2.33.7) Inversely transform the high and low frequency coefficients of the obtained 256×192 high-resolution image to obtain a 256×192 high-resolution image;

2.34) Zoom in from 256×192 to 512×384:

2.34.1) The 256×192 high-resolution image obtained from step 2.33) is used as the 256×192 low-resolution image of this step, and its pixel value is directly used as the low-frequency coefficient of the reconstructed 512×384 high-resolution image;

2.34.2) Perform nearest neighbor interpolation and bicubic linear interpolation on the 256×192 low-resolution image, take the average of the two, and use a 9-7 filter bank to perform Laplacian pyramid decomposition to extract its high-frequency coefficients;

2.34.3) Divide the 256×192 low-resolution image into small blocks with a size of s×t. The value of s ranges from 4 to 8, and the value of t ranges from 8 to 12. There are (s- 1) Overlap of ×(t-1);

2.34.4) Use LLE learning to obtain the high-frequency coefficients of the 512×384 high-resolution small block corresponding to the small block segmented in 2.34.3):

Let y _LR be a vector composed of high-frequency coefficients obtained in 2.34.2) corresponding to a small block obtained in 2.34.3), and x _LR (i) be the corresponding position of the ith 256×192 clear resolution sample The vector composed of the high-frequency coefficients of the small blocks, the first k (5<k<8) sample small blocks with the smallest y _LR mean square error in the corresponding position small blocks of the 256×192 clear resolution sample are taken, and the calculation is as follows: Value question:

W W = = \underset{w w ((i i))}{arg arg min min} {| | | | {ϵy ϵy}_{LR LR} - - {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{LR LR} ((i i)) | | | |}^{22},, {Σ Σ}_{i i = = 11}^{k k} w w ((i i)) = = 11 - - - - - - ((33))

Let G=(εy _LR 1 ^T -X) ^T (εy _LR 1 ^T -X), where each column of X is x _LR (i), a total of k columns, 1 is a vector whose elements are all 1, w(i) is the k coefficients to be sought, the solution of this extreme value problem is W is a vector composed of w(i), and then the high-frequency coefficient vector of the learned 512×384 high-resolution image block is calculated as:

{y the y}_{HR HR} = = \frac{{Σ Σ}_{i i = = 11}^{k k} w w ((i i)) {x x}_{HR HR} ((i i))}{ϵ ϵ} - - - - - - ((44))

Where x _HR (i) is the high-frequency coefficient vector of the 512×384 clear resolution sample block corresponding to x _LR (i);

2.34.5) Repeat 2.34.4) to get the high-frequency coefficients of 512×384 high-resolution small blocks corresponding to all small blocks segmented in 2.34.3)

2.34.6) The 512×384 high-resolution small blocks learned in 2.34.5) are stitched together, and the overlapping parts between adjacent small blocks are taken as the average value of all overlapping small blocks as the final result to obtain a 512×384 high-resolution High-frequency coefficients of high-rate images;

2.34.7) Inversely transform the high- and low-frequency coefficients of the obtained 512×384 high-resolution image to obtain the final reconstructed 512×384 high-resolution image.