CN109712069B - Face image multilayer reconstruction method based on CCA space - Google Patents

Face image multilayer reconstruction method based on CCA space Download PDF

Info

Publication number
CN109712069B
CN109712069B CN201811322383.9A CN201811322383A CN109712069B CN 109712069 B CN109712069 B CN 109712069B CN 201811322383 A CN201811322383 A CN 201811322383A CN 109712069 B CN109712069 B CN 109712069B
Authority
CN
China
Prior art keywords
resolution
low
dictionary
face image
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811322383.9A
Other languages
Chinese (zh)
Other versions
CN109712069A (en
Inventor
郭立君
李小宝
张�荣
姚正元
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ningbo University
Original Assignee
Ningbo University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ningbo University filed Critical Ningbo University
Priority to CN201811322383.9A priority Critical patent/CN109712069B/en
Publication of CN109712069A publication Critical patent/CN109712069A/en
Application granted granted Critical
Publication of CN109712069B publication Critical patent/CN109712069B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Image Analysis (AREA)

Abstract

The invention discloses a face image multilayer reconstruction method based on CCA space, which adopts large size to block a trained low-resolution face image, a trained high-resolution face image and a tested low-resolution face image to obtain a low-resolution dictionary and a high-resolution dictionary; secondly, performing CCA mapping on the two types of dictionaries once, and performing sparse updating on the two types of once-mapped dictionaries; then, performing inverse mapping on the two types of updated dictionaries, and performing CCA mapping on the two types of reflection dictionaries again; then, sorting the dictionaries by calculating Euclidean distances between each column vector in the two types of re-mapped dictionaries and the column vector of the corresponding image block in the tested low-resolution image, and obtaining a layer of reconstructed high-resolution face image by a super-resolution reconstruction method based on smoothness; then selecting a small size for blocking, repeating the process, introducing the constraint of a layer of reconstructed high-resolution image, and obtaining a high-resolution face image reconstructed by two layers; the advantage is that the reconstruction is effective.

Description

Face image multilayer reconstruction method based on CCA space
Technical Field
The invention relates to a face image reconstruction technology, in particular to a face image multilayer reconstruction method based on a CCA (Canonical Correlation Analysis) space.
Background
The most important face recognition problem in video surveillance is a difficult and complex one. For a face image in a monitoring video, edge structure information is unclear and details are blurred due to insufficient light, too far distance between a face and a monitoring device and the like. In order to solve the problems of unclear edge structure information and blurred details of a face image, it is necessary to use prior information to improve the resolution of the face image, that is, to reconstruct a high-resolution (HR) face image by using an observed low-resolution (LR) face image, which is a super-resolution (SR) reconstruction problem of the face image, and the reconstructed high-resolution face image provides more detailed information for the recognition and analysis of the face.
The super-resolution reconstruction technology of the face image has been widely concerned in the field of computer vision, and how to better reconstruct the face image is deeply researched by related organizations at home and abroad. Such as: jiang et al, J.Jiang, R.Hu, Z.Wang, and Z.Han, "Noise robust surface localization-constrained representation", IEEE trans.multimedia, vol.16, no.5, pp.1268-1281, aug.2014 (Jiang Junjun, hu Ruimin, wang Zhongyuan, han Zhen, noise robust face reconstruction [ J ], IEEE, multimedia, 2014, 1268-1281) propose a block-based local constraint model (LCR) with which reconstruction results show that the effect of Noise on the super-resolution reconstruction process can be reduced. On the basis, jiang J, ma J, chen C, et al. Noise Robust Face Image Super-Resolution [ J ]. IEEE Transactions on Cybernetics,2017, PP (99): 1-12. (Jiang Junjun, ma Jiayi, chen Chen, jiang Xinwei, wang Zheng), noise Robust Face reconstruction [ J ] based on Smooth Sparse Representation, IEEE, cybernetics,2017,1-12) provides a Super-Resolution reconstruction method (SSR) based on smoothness, which achieves certain smoothing and denoising effects.
The reconstruction technique proposed by Jiang et al is based on the assumption that the high-resolution dictionary and the low-resolution dictionary are highly correlated and have similar structural distribution, and the reconstruction of the face image can be performed based on the assumption. On one hand, however, although there is a certain similarity in structure and content between face images, a high-resolution dictionary and a low-resolution dictionary which are directly constructed based on a face image space cannot meet a highly relevant condition, so that a reconstruction effect is not ideal; on the other hand, the above reconstruction techniques all adopt a single-layer reconstruction mode, that is, the reconstruction is performed based on the size of a fixed block, in the block-based reconstruction techniques, the block size of the observed low-resolution face image is very important, if the block size is small, the number of blocks is large, the detailed information of the reconstructed high-resolution face image is abundant, but the structural information of the reconstructed high-resolution face image is not easy to grasp; if the size of the block is large, the number of blocks is small, the structural information of the reconstructed high-resolution face image is easy to grasp, but the detailed information of the reconstructed high-resolution face image is not rich. Therefore, a new face image reconstruction method needs to be researched for a low-resolution face image with the problems of unclear edge structure information and blurred details.
Disclosure of Invention
The invention aims to solve the technical problem of providing a human face image multilayer reconstruction method based on a CCA space, and the edge structure information of a high-resolution human face image obtained by utilizing the reconstruction method is clear, the details are clear, and the reconstruction effect is good.
The technical scheme adopted by the invention for solving the technical problems is as follows: a human face image multilayer reconstruction method based on CCA space is characterized by comprising the following steps:
the method comprises the following steps: selecting a face image database, wherein the face image database comprises at least two low-resolution face images and a high-resolution face image corresponding to each low-resolution face image, and correspondingly recording the nth low-resolution face image and the corresponding high-resolution face image in the face image database as
Figure BDA0001857838540000021
And &>
Figure BDA0001857838540000022
And the tested low-resolution face image is recorded as ^ or ^ based on>
Figure BDA0001857838540000023
Wherein N is a positive integer, N is more than or equal to 1 and less than or equal to N, N represents the total number of low-resolution face images contained in the face image database, N is more than or equal to 2, and/or is greater than or equal to N>
Figure BDA0001857838540000024
And &>
Figure BDA0001857838540000025
Are all W in width>
Figure BDA0001857838540000026
And &>
Figure BDA0001857838540000027
All the heights of (A) are H;
step two: dividing each low-resolution face image in face image database into S by adopting sliding window technology 1 Each overlapping with a dimension of k 1 ×k 1 Image blocks of
Figure BDA0001857838540000028
S of (1) 1 An image block is marked as>
Figure BDA0001857838540000029
Similarly, a sliding window technology is adopted to divide the high-resolution face image corresponding to each low-resolution face image in the face image database into S 1 Each overlapping with a dimension of k 1 ×k 1 The image block of (a) is selected, will->
Figure BDA0001857838540000031
S of (1) 1 Each image block is recorded as>
Figure BDA0001857838540000032
Will be picked up using a sliding window technique>
Figure BDA0001857838540000033
Is divided into 1 Each overlapping with a dimension of k 1 ×k 1 The image block of (2) is selected, will be/are>
Figure BDA0001857838540000034
S of (1) 1 Each image block is recorded as>
Figure BDA0001857838540000035
Wherein the size of the sliding window is k 1 ×k 1 ，k 1 =5,7,9,11, sliding step of sliding window is 1 pixel point, S 1 ＝(W-k 1 +1)×(H-k 1 +1)，s 1 Is a positive integer, s is not less than 1 1 ≤S 1 ；
Step three: arranging the pixel values of all pixel points in each image block in each low-resolution face image in a face image database to form corresponding column vectors, and arranging the pixel values of all pixel points in each image block in each low-resolution face image in the face image database to form a column vector
Figure BDA0001857838540000036
The corresponding column vector is marked +>
Figure BDA0001857838540000037
Similarly, arranging the pixel values of all pixel points in each image block in the high-resolution face image corresponding to each low-resolution face image in the face image database to form a corresponding column vector, and combining->
Figure BDA0001857838540000038
The corresponding column vector is marked +>
Figure BDA0001857838540000039
Will->
Figure BDA00018578385400000310
The pixel values of all the pixel points in each image block are arranged to form a corresponding column vector, and the ^ is greater than or equal to>
Figure BDA00018578385400000311
The corresponding column vector is marked +>
Figure BDA00018578385400000312
Then, column vectors corresponding to image blocks at the same position in all low-resolution face images in the face image database form a low-resolution dictionary to form S 1 A low resolution dictionary for mapping the s-th image of all low resolution face images in the face image database 1 A low-resolution dictionary formed by column vectors corresponding to image blocks is recorded as->
Figure BDA00018578385400000313
Similarly, column vectors corresponding to image blocks at the same position in all high-resolution face images in the face image database form a high-resolution dictionary, and form S 1 A high resolution dictionary for comparing the s-th of all high resolution face images in the face image database 1 The high-resolution dictionary formed by the column vectors corresponding to the image blocks is marked as->
Figure BDA00018578385400000314
Wherein it is present>
Figure BDA00018578385400000315
Figure BDA00018578385400000316
Are all (k) 1 ×k 1 )×1，/>
Figure BDA00018578385400000317
Are all (k) 1 ×k 1 )×N，/>
Figure BDA00018578385400000318
Is/is for the nth column vector of>
Figure BDA00018578385400000319
Is/is->
Figure BDA00018578385400000320
Step four: calculating the projection matrix corresponding to each low-resolution dictionary and each high-resolution dictionary respectively
Figure BDA00018578385400000321
The corresponding projection matrix is marked as->
Figure BDA00018578385400000322
Will->
Figure BDA00018578385400000323
The corresponding projection matrix is recorded as +>
Figure BDA00018578385400000324
Wherein it is present>
Figure BDA00018578385400000325
And &>
Figure BDA00018578385400000326
All dimensions of (c) are L × (k) 1 ×k 1 ) L represents the dimension of CCA space, and L is equal to {1,2, …, k 1 ×k 1 }；
Step five: mapping each low-resolution dictionary from an image space to a CCA space to obtain a corresponding primary mapping low-resolution dictionary, and mapping each low-resolution dictionary to a CCA space
Figure BDA0001857838540000041
The corresponding one-time mapped low resolution dictionary is &>
Figure BDA0001857838540000042
Similarly, mapping each high-resolution dictionary from the image space to the CCA space to obtain a corresponding once-mapped high-resolution dictionary, and combining>
Figure BDA0001857838540000043
Corresponding one-time mapped high resolution dictionary +>
Figure BDA0001857838540000044
Wherein it is present>
Figure BDA0001857838540000045
And &>
Figure BDA0001857838540000046
The dimensions of (A) are all L multiplied by N;
step six: calculating the sparse coefficient vector of each primary mapping low-resolution dictionary
Figure BDA0001857838540000047
Is marked as->
Figure BDA0001857838540000048
By counting->
Figure BDA0001857838540000049
Obtaining; then carrying out once sparse updating on the once mapped low-resolution dictionary by using the sparse coefficient vector of each once mapped low-resolution dictionary to obtain the updated dictionary of each once mapped low-resolution dictionary, and then judging whether the updated dictionary is the same as the updated dictionary or not>
Figure BDA00018578385400000410
The updated dictionary is recorded as +>
Figure BDA00018578385400000411
If/or>
Figure BDA00018578385400000412
The nth element of (a) is a non-zero element, will then >>
Figure BDA00018578385400000413
The nth column vector of (a) is extracted and the slave is then taken>
Figure BDA00018578385400000414
All column vectors extracted in (a) are formed in original order->
Figure BDA00018578385400000415
Similarly, performing sparse update once on each once-mapped high-resolution dictionary to obtain a dictionary updated by each once-mapped high-resolution dictionary, and judging whether the updated dictionary is matched with the updated dictionary or not>
Figure BDA00018578385400000416
Updated dictionary record>
Figure BDA00018578385400000417
If/or>
Figure BDA00018578385400000418
Will be a non-zero element, will->
Figure BDA00018578385400000419
The nth column vector of (a) is extracted and the slave is then taken>
Figure BDA00018578385400000420
All column vectors extracted in (a) are formed in original order->
Figure BDA00018578385400000421
Wherein it is present>
Figure BDA00018578385400000422
Dimension of (a) is Nx 1, argmin () represents solving the residual minimum value, the symbol "| | | | non-conducting phosphor 2 Is "is 2 Norm regular term operation symbol, symbol "| | | | non-woven phosphor 1 Is "as 1 Norm regular term operator sign, λ 1 Is a constant, λ 1 ∈(0,1)，/>
Figure BDA00018578385400000423
And &>
Figure BDA00018578385400000424
Has dimension L × M, M denotes->
Figure BDA00018578385400000425
The total number of the non-zero elements in the alloy is more than or equal to 1 and less than N;
step seven: the updated dictionary of each once-mapped low-resolution dictionary is reversely mapped back to the image space from the CCA space to obtain a corresponding reflection low-resolution dictionary, and the dictionary is subjected to image matching
Figure BDA00018578385400000426
The corresponding retroreflection low resolution dictionary is &>
Figure BDA00018578385400000427
Similarly, each high resolution word is mapped onceThe dictionary updated by the dictionary is reversely mapped back to the image space from the CCA space to obtain a corresponding reflection high-resolution dictionary, and the dictionary is/are>
Figure BDA0001857838540000051
The corresponding reflection high resolution dictionary is recorded as>
Figure BDA0001857838540000052
Wherein it is present>
Figure BDA0001857838540000053
And
Figure BDA0001857838540000054
has a dimension of (k) 1 ×k 1 )×M；
Step eight: calculating the projection matrix corresponding to each reverse mapping low-resolution dictionary and each reverse mapping high-resolution dictionary respectively
Figure BDA0001857838540000055
The corresponding projection matrix is recorded as +>
Figure BDA0001857838540000056
Will->
Figure BDA0001857838540000057
The corresponding projection matrix is recorded as +>
Figure BDA0001857838540000058
Wherein it is present>
Figure BDA0001857838540000059
And
Figure BDA00018578385400000510
all dimensions of (a) are L × (k) 1 ×k 1 ) L represents the dimension of CCA space, and L is equal to {1,2, …, k 1 ×k 1 }；
Step nine: mapping each reverse mapping low-resolution dictionary from the image space to the CCA space to obtain pairsShould remap the low resolution dictionary again, will
Figure BDA00018578385400000511
Corresponding remap low resolution dictionary noted in>
Figure BDA00018578385400000512
/>
Figure BDA00018578385400000513
Similarly, mapping each reflection high-resolution dictionary from the image space to the CCA space to obtain a corresponding re-mapping high-resolution dictionary, and combining>
Figure BDA00018578385400000514
Corresponding remapped high resolution dictionary
Figure BDA00018578385400000515
Will->
Figure BDA00018578385400000516
Each image block in (1) is mapped to CCA space from image space to obtain
Figure BDA00018578385400000517
Will ∑ be based on a corresponding primary mapped block of each image block in the image block>
Figure BDA00018578385400000518
The corresponding one-time mapping block is marked as->
Figure BDA00018578385400000519
Will->
Figure BDA00018578385400000520
The pixel values of all pixel points in the primary mapping block corresponding to each image block are arranged to form a corresponding column vector, and the column vector is obtained
Figure BDA00018578385400000521
The corresponding column vector is noted as/>
Figure BDA00018578385400000522
Wherein it is present>
Figure BDA00018578385400000523
And &>
Figure BDA00018578385400000524
Are all LxM->
Figure BDA00018578385400000525
Has dimension of L × 1;
step ten: computing the vector sum of each column in each remapped low resolution dictionary
Figure BDA00018578385400000526
Is used for ^ ing the Euclidean distance of the column vector corresponding to the primary mapping block corresponding to each image block in the>
Figure BDA00018578385400000527
Calculate->
Figure BDA00018578385400000528
Each column vector of (1)
Figure BDA00018578385400000529
Is based on the Euclidean distance of->
Figure BDA00018578385400000530
Obtaining M Euclidean distances; then, aiming at each M Euclidean distances obtained by re-mapping the low-resolution dictionary, sequencing the M Euclidean distances from large to small; then according to the magnitude sequence of M Euclidean distances obtained by aiming at each remapped low-resolution dictionary, carrying out position adjustment on all column vectors in each remapped low-resolution dictionary, recombining to obtain a corresponding recombined low-resolution dictionary, and then combining>
Figure BDA00018578385400000531
The corresponding recombined low resolution dictionary is marked>
Figure BDA00018578385400000532
The 1 st column vector and->
Figure BDA00018578385400000533
Has the largest Euclidean distance and is greater than or equal to>
Figure BDA0001857838540000061
And the last column vector and->
Figure BDA0001857838540000062
Has the smallest euclidean distance; wherein it is present>
Figure BDA0001857838540000063
Dimension of (d) is L × M;
similarly, calculate the sum of each column vector in each remapped high resolution dictionary
Figure BDA0001857838540000064
Is used for ^ ing the Euclidean distance of the column vector corresponding to the primary mapping block corresponding to each image block in the>
Figure BDA0001857838540000065
Calculate->
Figure BDA0001857838540000066
Each column vector of (1)
Figure BDA0001857838540000067
The distance in degrees of euclidean of (c), for +>
Figure BDA0001857838540000068
Obtaining M Euclidean distances; then, aiming at each M Euclidean distances obtained by re-mapping the high-resolution dictionary, sequencing the M Euclidean distances from large to small; then according to the magnitude sequence of M Euclidean distances obtained by mapping the high-resolution dictionary again for each time,adjusting the position of all column vectors in each remapped high-resolution dictionary, recombining to obtain a corresponding recombined high-resolution dictionary, and combining>
Figure BDA0001857838540000069
The corresponding recombined high-resolution dictionary is marked>
Figure BDA00018578385400000610
The 1 st column vector and->
Figure BDA00018578385400000611
Has the largest Euclidean distance and is greater than or equal to>
Figure BDA00018578385400000612
And the last column vector and->
Figure BDA00018578385400000613
Has the smallest euclidean distance; wherein it is present>
Figure BDA00018578385400000614
Dimension of (d) is L × M;
step eleven: computing
Figure BDA00018578385400000615
Will @, for each image block of the first sparse coefficient vector>
Figure BDA00018578385400000616
Is marked as £ the first sparse coefficient vector of>
Figure BDA00018578385400000617
Figure BDA00018578385400000618
By passing
Figure BDA00018578385400000633
Calculating to obtain; will then->
Figure BDA00018578385400000619
The high-resolution face image obtained after reconstruction in one layer is recorded as ^ er>
Figure BDA00018578385400000620
Will->
Figure BDA00018578385400000621
In and->
Figure BDA00018578385400000622
The area corresponding to the position is marked as->
Figure BDA00018578385400000623
Will->
Figure BDA00018578385400000624
The column vector formed by arranging the pixel values of all the pixel points is recorded as
Figure BDA00018578385400000625
Wherein it is present>
Figure BDA00018578385400000626
Has dimension of M × 1,m which is a positive integer, M is more than or equal to 1 and less than or equal to M, and λ 2 And λ 3 Are all constant, λ 2 ∈(0,1)，λ 3 ∈(0,1)，/>
Figure BDA00018578385400000627
Represents->
Figure BDA00018578385400000628
The mth element of (4), is selected>
Figure BDA00018578385400000629
Represents->
Figure BDA00018578385400000630
The mth column vector of (4), based on the number of cells in the column->
Figure BDA00018578385400000631
Represents->
Figure BDA00018578385400000632
M-1 element of (1);
step twelve: the size of the sliding window is changed to k 2 ×k 2 (ii) a Then S is obtained in the same manner according to the process from step two to step ten 2 A recombined low resolution dictionary and S 2 Recombining the high resolution dictionary to give 2 Reorganize the low resolution dictionary as
Figure BDA0001857838540000071
Will be(s) 2 Recombined high-resolution dictionary is marked as>
Figure BDA0001857838540000072
Then calculates->
Figure BDA0001857838540000073
Will ∑ the second sparse coefficient vector of each image block of>
Figure BDA0001857838540000074
S of (1) 2 Image block>
Figure BDA0001857838540000075
Is recorded as a second sparse coefficient vector
Figure BDA0001857838540000076
By passing
Figure BDA0001857838540000077
Calculating to obtain; then will
Figure BDA0001857838540000078
The high-resolution face image obtained after the two-layer reconstruction is recorded as ^ er>
Figure BDA0001857838540000079
Will->
Figure BDA00018578385400000710
In and>
Figure BDA00018578385400000711
the area corresponding to the position is marked as->
Figure BDA00018578385400000712
Will be/are>
Figure BDA00018578385400000713
The column vector formed by arranging the pixel values of all the pixel points is recorded as ^ er>
Figure BDA00018578385400000714
Figure BDA00018578385400000715
Wherein it is present>
Figure BDA00018578385400000716
And &>
Figure BDA00018578385400000717
Dimension of L × M, k 2 =3,5,7,9 and 1 < k 2 ＜k 1 ，S 2 Representing that each low-resolution face image in a face image database and the corresponding high-resolution face image are judged and judged by adopting a sliding window technology>
Figure BDA00018578385400000718
Divided into mutually overlapping dimensions of k 2 ×k 2 Total number of image blocks, S 2 ＝(W-k 2 +1)×(H-k 2 +1)，1≤s 2 ≤S 2 ，/>
Figure BDA00018578385400000719
Has dimension of Mx 1->
Figure BDA00018578385400000720
To be according to capturing>
Figure BDA00018578385400000721
Is obtained in the same manner, the projection matrix is evaluated>
Figure BDA00018578385400000722
Represents->
Figure BDA00018578385400000723
The pixel values of all the pixel points in the image are arranged to form a corresponding column vector,
Figure BDA00018578385400000724
represents->
Figure BDA00018578385400000725
The mth element of (4), is selected>
Figure BDA00018578385400000726
Represents->
Figure BDA00018578385400000727
The m-1 th element of (4), is selected>
Figure BDA00018578385400000728
Represents->
Figure BDA00018578385400000729
M-th column vector of (1) 4 Is a constant, λ 4 ∈(0,1)，/>
Figure BDA00018578385400000730
To be according to capturing>
Figure BDA00018578385400000731
Is obtained in the same manner, the projection matrix is evaluated>
Figure BDA00018578385400000732
Represents->
Figure BDA00018578385400000733
In and->
Figure BDA00018578385400000734
The pixel values of all the pixel points in the corresponding area are arranged to form a corresponding column vector, and the corresponding column vector is greater than or equal to>
Figure BDA00018578385400000735
Represents->
Figure BDA00018578385400000736
The m-th column vector of (1).
Compared with the prior art, the invention has the advantages that:
1) According to the method, the low-resolution dictionary and the high-resolution dictionary are both mapped to the CCA space from the image space, so that the correlation between the low-resolution dictionary and the high-resolution dictionary is enhanced; meanwhile, redundant information and noise information exist in the low-resolution dictionary and the high-resolution dictionary, and the anti-noise performance and the reconstruction effect of the reconstruction method are influenced, so that the dictionary updated by mapping the low-resolution dictionary once and the dictionary updated by mapping the high-resolution dictionary once are back-mapped from the CCA space to the image space, and then the reflection low-resolution dictionary and the reflection high-resolution dictionary are mapped from the image space to the CCA space, namely, two CCA mappings are adopted, and the anti-noise performance and the reconstruction effect of the method are improved.
2) The method adopts a two-layer reconstruction mode, firstly divides the tested low-resolution face image into larger image blocks to perform one-layer reconstruction so as to grasp the structural information of the reconstructed high-resolution face image, then reconstructs the next reconstruction work with smaller blocks by taking the high-resolution face image reconstructed from the tested low-resolution face image as constraint to reconstruct the detail information of the high-resolution face image of the tested low-resolution face image, and the edge structural information of the high-resolution face image obtained by two-layer reconstruction is clear and has clear details and good reconstruction effect.
Drawings
FIG. 1 is a general flow diagram of the process of the present invention;
FIG. 2 is a noisy face image;
FIG. 3 is a high-resolution face image obtained by reconstructing the noise face image shown in FIG. 2 using a conventional smoothing-based super-resolution reconstruction method (SSR);
fig. 4 is a high-resolution face image obtained by reconstructing the noise face image shown in fig. 2 by adding CCA mapping once based on the existing smoothing-based super-resolution reconstruction method (SSR) and single-layer reconstruction, i.e., one-layer reconstruction;
fig. 5 is a high-resolution face image obtained by reconstructing the noise face image shown in fig. 2 by adding two CCA mappings based on the existing smoothing-based super-resolution reconstruction method (SSR) and performing single-layer reconstruction, i.e., one-layer reconstruction;
FIG. 6 is a high-resolution face image reconstructed from the noise face image shown in FIG. 2 by the method of the present invention;
fig. 7 is a real high resolution face image corresponding to the noise face image shown in fig. 2.
Detailed Description
The invention is described in further detail below with reference to the accompanying examples.
The invention provides a human face image multilayer reconstruction method based on a CCA space, the general flow block diagram of which is shown in figure 1, and the method comprises the following steps:
the method comprises the following steps: selecting a face image database, wherein the face image database comprises at least two low-resolution face images and a high-resolution face image corresponding to each low-resolution face image, and correspondingly recording the nth low-resolution face image and the corresponding high-resolution face image in the face image database as
Figure BDA0001857838540000091
And &>
Figure BDA0001857838540000092
And testing the low resolution faceImage is recorded as->
Figure BDA0001857838540000093
Wherein N is a positive integer, N is more than or equal to 1 and less than or equal to N, N represents the total number of low-resolution face images contained in the face image database, N is more than or equal to 2, and if N =360 ″, then ^ N is selected>
Figure BDA0001857838540000094
And &>
Figure BDA0001857838540000095
Are all W in width>
Figure BDA0001857838540000096
And
Figure BDA0001857838540000097
all heights of (2) are H.
Step two: dividing each low-resolution face image in face image database into S by adopting sliding window technology 1 Each overlapping with a dimension of k 1 ×k 1 Image blocks of
Figure BDA0001857838540000098
S of (1) 1 Each image block is recorded as>
Figure BDA0001857838540000099
Similarly, a sliding window technology is adopted to divide the high-resolution face image corresponding to each low-resolution face image in the face image database into S 1 Each overlapping with a dimension of k 1 ×k 1 The image block of (2) is selected, will be/are>
Figure BDA00018578385400000910
S of (1) 1 Each image block is recorded as>
Figure BDA00018578385400000911
Will be picked up using a sliding window technique>
Figure BDA00018578385400000912
Is divided into 1 Each overlapping with a dimension of k 1 ×k 1 Will->
Figure BDA00018578385400000913
S of (1) 1 Each image block is recorded as>
Figure BDA00018578385400000914
Wherein the size of the sliding window is k 1 ×k 1 ，k 1 =5,7,9,11, in this example k 1 =5, sliding step length of sliding window is 1 pixel point, S 1 ＝(W-k 1 +1)×(H-k 1 +1)，s 1 Is a positive integer, s is not less than 1 1 ≤S 1 。
Step three: arranging the pixel values of all pixel points in each image block in each low-resolution face image in a face image database to form corresponding column vectors, and arranging the pixel values of all pixel points in each image block in each low-resolution face image in the face image database to form a column vector
Figure BDA00018578385400000915
The corresponding column vector is marked +>
Figure BDA00018578385400000916
Similarly, arranging the pixel values of all pixel points in each image block in the high-resolution face image corresponding to each low-resolution face image in the face image database to form a corresponding column vector, and combining->
Figure BDA00018578385400000917
The corresponding column vector is marked +>
Figure BDA00018578385400000918
Will->
Figure BDA00018578385400000919
The pixel values of all the pixel points in each image block are arranged to form a corresponding column vector, and the ^ is greater than or equal to>
Figure BDA00018578385400000920
The corresponding column vector is marked +>
Figure BDA00018578385400000921
Then, column vectors corresponding to image blocks at the same position in all low-resolution face images in the face image database form a low-resolution dictionary to form S 1 A low resolution dictionary for mapping the s-th image of all low resolution face images in the face image database 1 A low-resolution dictionary formed by column vectors corresponding to image blocks is recorded as->
Figure BDA0001857838540000101
Similarly, column vectors corresponding to image blocks at the same position in all high-resolution face images in the face image database form a high-resolution dictionary to form S 1 A high resolution dictionary for comparing the s-th of all high resolution face images in the face image database 1 The high-resolution dictionary formed by the column vectors corresponding to the image blocks is marked as->
Figure BDA0001857838540000102
Wherein it is present>
Figure BDA0001857838540000103
Figure BDA0001857838540000104
Are all (k) 1 ×k 1 )×1，/>
Figure BDA0001857838540000105
Are all (k) 1 ×k 1 )×N，/>
Figure BDA0001857838540000106
Is/is for the nth column vector of>
Figure BDA0001857838540000107
The nth column vector ofIs->
Figure BDA0001857838540000108
Step four: calculating the projection matrix corresponding to each low-resolution dictionary and each high-resolution dictionary respectively, and calculating the projection matrix
Figure BDA0001857838540000109
The corresponding projection matrix is recorded as +>
Figure BDA00018578385400001010
Will->
Figure BDA00018578385400001011
The corresponding projection matrix is recorded as +>
Figure BDA00018578385400001012
Wherein it is present>
Figure BDA00018578385400001026
And &>
Figure BDA00018578385400001013
All dimensions of (a) are L × (k) 1 ×k 1 ) L represents the dimension of CCA space, and L is equal to {1,2, …, k 1 ×k 1 }；/>
Figure BDA00018578385400001014
And &>
Figure BDA00018578385400001015
Reference may be made to David R.Hardoon et al.Canonical Correlation Analysis: an Overview with Application to Learning Methods [ J]Neural Computation,2004,2639-2664 (David R-Hayton et al, canonical correlation analysis: overview applied to learning methods [ J]Neural computation,2004, 2639-2664).
Step five: mapping each low-resolution dictionary from image space to CCA space to obtain corresponding primary mapping low-resolution dictionary, and mapping each low-resolution dictionary to CCA space
Figure BDA00018578385400001016
The corresponding one-time mapped low resolution dictionary is &>
Figure BDA00018578385400001017
Similarly, mapping each high-resolution dictionary from the image space to the CCA space to obtain a corresponding primary mapping high-resolution dictionary, and mapping each high-resolution dictionary to the CCA space
Figure BDA00018578385400001018
Corresponding one-time mapped high resolution dictionary +>
Figure BDA00018578385400001019
Wherein, is +>
Figure BDA00018578385400001020
And
Figure BDA00018578385400001021
are all L N.
Step six: calculating the sparse coefficient vector of each primary mapping low-resolution dictionary
Figure BDA00018578385400001022
Is marked as->
Figure BDA00018578385400001023
Figure BDA00018578385400001024
By counting->
Figure BDA00018578385400001025
Obtaining; then carrying out once sparse updating on the once mapped low-resolution dictionary by using the sparse coefficient vector of each once mapped low-resolution dictionary to obtain the updated dictionary of each once mapped low-resolution dictionary, and then judging whether the updated dictionary is the same as the updated dictionary or not>
Figure BDA0001857838540000111
Updated dictionary record>
Figure BDA0001857838540000112
If/or>
Figure BDA0001857838540000113
Will be a non-zero element, will->
Figure BDA0001857838540000114
The nth column vector of (a) is extracted and the slave is then taken>
Figure BDA0001857838540000115
All column vectors extracted in (are) formed in the original order->
Figure BDA0001857838540000116
Similarly, performing sparse update once on each once-mapped high-resolution dictionary to obtain a dictionary updated by each once-mapped high-resolution dictionary, and judging whether the updated dictionary is matched with the updated dictionary or not>
Figure BDA0001857838540000117
The updated dictionary is recorded as +>
Figure BDA0001857838540000118
If/or>
Figure BDA0001857838540000119
Will be a non-zero element, will->
Figure BDA00018578385400001110
The nth column vector of (a) is extracted and the slave is then taken>
Figure BDA00018578385400001111
All column vectors extracted in (are) formed in the original order->
Figure BDA00018578385400001112
Wherein the content of the first and second substances,
Figure BDA00018578385400001113
dimension of (a) is Nx 1, argmin () represents solving the residual minimum value, the symbol "| | | | non-conducting phosphor 2 Is "is 2 Norm regular term operation symbol, symbol "| | | | non-woven phosphor 1 Is "as 1 Norm regular term operator sign, λ 1 Is a constant, λ 1 E (0,1), generally given as λ 1 =0.1,0.3,0.5, in this example λ 1 ＝0.3，/>
Figure BDA00018578385400001114
And &>
Figure BDA00018578385400001115
Has dimension L × M, M denotes->
Figure BDA00018578385400001116
The total number of the non-zero elements in the alloy is more than or equal to 1, and M is less than N.
Step seven: the updated dictionary of each once-mapped low-resolution dictionary is reversely mapped back to the image space from the CCA space to obtain a corresponding reflection low-resolution dictionary, and the dictionary is subjected to image matching
Figure BDA00018578385400001117
The corresponding retroreflection low resolution dictionary is &>
Figure BDA00018578385400001118
Similarly, each updated dictionary with the once mapping high-resolution dictionary is inversely mapped from the CCA space to the image space to obtain a corresponding reflection high-resolution dictionary, and the dictionary is combined>
Figure BDA00018578385400001119
The corresponding reflection high resolution dictionary is recorded as>
Figure BDA00018578385400001120
Wherein it is present>
Figure BDA00018578385400001121
And
Figure BDA00018578385400001122
has a dimension of (k) 1 ×k 1 )×M。
Step eight: calculating the projection matrix corresponding to each back mapping low-resolution dictionary and each back mapping high-resolution dictionary respectively, and calculating the projection matrix corresponding to each back mapping low-resolution dictionary and each back mapping high-resolution dictionary
Figure BDA00018578385400001123
The corresponding projection matrix is recorded as +>
Figure BDA00018578385400001124
Will->
Figure BDA00018578385400001125
The corresponding projection matrix is recorded as +>
Figure BDA00018578385400001126
Wherein it is present>
Figure BDA00018578385400001127
And &>
Figure BDA00018578385400001128
All dimensions of (a) are L × (k) 1 ×k 1 ) L represents the dimension of CCA space, and L is equal to {1,2, …, k 1 ×k 1 }；/>
Figure BDA00018578385400001129
And
Figure BDA00018578385400001130
reference may be made to David R.Hardoon et al.Canonical Correlation Analysis: an Overview with Application to Learning Methods [ J]Neural Computation,2004,2639-2664 (David R-Hayton et al, canonical correlation analysis: overview applied to learning methods [ J]Neural computation,2004, 2639-2664).
Step nine: each is reflectedMapping the low-resolution dictionary from the image space to the CCA space to obtain a corresponding remapped low-resolution dictionary, and mapping the remapped low-resolution dictionary to the CCA space
Figure BDA0001857838540000121
Corresponding remap low resolution dictionary is recorded as>
Figure BDA0001857838540000122
Figure BDA0001857838540000123
Mapping each reflection high-resolution dictionary from the image space to the CCA space to obtain a corresponding re-mapping high-resolution dictionary, and combining>
Figure BDA0001857838540000124
Corresponding remapped high resolution dictionary
Figure BDA0001857838540000125
Will->
Figure BDA0001857838540000126
Each image block in (1) is mapped to CCA space from image space to obtain
Figure BDA0001857838540000127
Will ∑ be based on a corresponding primary mapped block of each image block in the image block>
Figure BDA0001857838540000128
The corresponding one-time mapping block is marked as->
Figure BDA0001857838540000129
Will->
Figure BDA00018578385400001210
The pixel values of all pixel points in the primary mapping block corresponding to each image block are arranged to form a corresponding column vector, and the column vector is obtained
Figure BDA00018578385400001211
The corresponding column vector is marked +>
Figure BDA00018578385400001212
Wherein it is present>
Figure BDA00018578385400001213
And &>
Figure BDA00018578385400001214
Are all LxM->
Figure BDA00018578385400001215
Has dimension of L × 1.
Step ten: computing the sum of vectors per column in each remapped low resolution dictionary
Figure BDA00018578385400001216
Is used for ^ ing the Euclidean distance of the column vector corresponding to the primary mapping block corresponding to each image block in the>
Figure BDA00018578385400001217
Calculate->
Figure BDA00018578385400001218
Each column vector of (1)
Figure BDA00018578385400001219
Is based on the Euclidean distance of->
Figure BDA00018578385400001220
Obtaining M Euclidean distances; then, aiming at each M Euclidean distances obtained by re-mapping the low-resolution dictionary, sequencing the M Euclidean distances from large to small; then according to the magnitude sequence of M Euclidean distances obtained by mapping the low-resolution dictionaries again, carrying out position adjustment on all column vectors in the low-resolution dictionaries mapped again, recombining to obtain corresponding recombined low-resolution dictionaries, and combining>
Figure BDA00018578385400001221
The corresponding recombined low resolution dictionary is marked>
Figure BDA00018578385400001222
The 1 st column vector and->
Figure BDA00018578385400001223
Has the largest Euclidean distance and is greater than or equal to>
Figure BDA00018578385400001224
And the last column vector and->
Figure BDA00018578385400001225
Has the smallest euclidean distance of (c); wherein it is present>
Figure BDA00018578385400001226
Dimension (d) is L M.
Similarly, calculate the sum of each column vector in each remapped high resolution dictionary
Figure BDA00018578385400001227
The euclidean distance of the column vector corresponding to the primary mapping block corresponding to each image block in (b), for +>
Figure BDA0001857838540000131
Calculate->
Figure BDA0001857838540000132
Each column vector of (1)
Figure BDA0001857838540000133
Is based on the Euclidean distance of->
Figure BDA0001857838540000134
Obtaining M Euclidean distances; then, sequencing the M Euclidean distances from large to small according to the M Euclidean distances obtained by re-mapping the high-resolution dictionary; according to each secondMapping the high-resolution dictionary to obtain the magnitude sequence of M Euclidean distances, performing position adjustment on all column vectors in each re-mapped high-resolution dictionary, recombining to obtain a corresponding recombined high-resolution dictionary, and then combining>
Figure BDA0001857838540000135
The corresponding recombined high-resolution dictionary is marked>
Figure BDA0001857838540000136
The 1 st column vector and->
Figure BDA0001857838540000137
Has the largest Euclidean distance and is greater than or equal to>
Figure BDA0001857838540000138
And/or the last column vector in (b)>
Figure BDA0001857838540000139
Has the smallest euclidean distance; wherein it is present>
Figure BDA00018578385400001310
Dimension (d) is L M.
Step eleven: computing
Figure BDA00018578385400001311
Will @, for each image block of the first sparse coefficient vector>
Figure BDA00018578385400001312
Is marked as £ the first sparse coefficient vector of>
Figure BDA00018578385400001313
Figure BDA00018578385400001314
By passing
Figure BDA00018578385400001315
Calculating to obtain; will then->
Figure BDA00018578385400001316
The high-resolution face image obtained after reconstruction in one layer is recorded as ^ er>
Figure BDA00018578385400001317
Will->
Figure BDA00018578385400001318
In and->
Figure BDA00018578385400001319
The area corresponding to the position is marked as->
Figure BDA00018578385400001320
Will->
Figure BDA00018578385400001321
The column vector formed by arranging the pixel values of all the pixel points is recorded as
Figure BDA00018578385400001322
Wherein it is present>
Figure BDA00018578385400001323
The dimension of M is multiplied by 1,m is a positive integer, M is more than or equal to 1 and less than or equal to M, and lambda is 2 And λ 3 Are all constant, λ 2 E (0,1), generally given as λ 2 =0.1,0.3,0.5, in this example λ 2 ＝0.3，λ 3 E (0,1), in this example λ 3 ＝0.001，/>
Figure BDA00018578385400001324
Represents->
Figure BDA00018578385400001325
The mth element of (1), in>
Figure BDA00018578385400001326
Represents->
Figure BDA00018578385400001334
The mth column vector of (4), based on the number of cells in the column->
Figure BDA00018578385400001327
Represents->
Figure BDA00018578385400001328
The m-1 th element in (1).
Step twelve: the size of the sliding window is changed to k 2 ×k 2 (ii) a Then S is obtained in the same manner according to the process from step two to step ten 2 Reorganized low resolution dictionary and S 2 Reorganize the high resolution dictionary into s 2 The recombined low resolution dictionary is noted
Figure BDA00018578385400001329
Will be(s) 2 Recombined high-resolution dictionary is marked as>
Figure BDA00018578385400001330
Then count>
Figure BDA00018578385400001331
Will ∑ the second sparse coefficient vector of each image block of>
Figure BDA00018578385400001332
S of (1) 2 Image block>
Figure BDA00018578385400001333
Is recorded as a second sparse coefficient vector
Figure BDA0001857838540000141
Figure BDA0001857838540000142
By passing
Figure BDA0001857838540000143
Calculating to obtain; then will be
Figure BDA0001857838540000144
The high-resolution face image obtained after the two-layer reconstruction is recorded as ^ er>
Figure BDA0001857838540000145
Will->
Figure BDA0001857838540000146
In and->
Figure BDA0001857838540000147
The area corresponding to the position is marked as->
Figure BDA0001857838540000148
Will->
Figure BDA0001857838540000149
The column vector formed by arranging the pixel values of all the pixel points is recorded as ^ er>
Figure BDA00018578385400001410
Figure BDA00018578385400001411
Wherein it is present>
Figure BDA00018578385400001412
And &>
Figure BDA00018578385400001413
Dimension of L × M, k 2 =3,5,7,9 and 1 < k 2 ＜k 1 If k is 1 K is taken out if =5 2 =3, if k 1 K is taken out if =7 2 =3 or k 2 If k is =5 1 K is taken out of the equation of k =9 2 =3 or k 2 =5 or k 2 =7, if k 1 K is taken out of the equation of k =11 2 =3 or k 2 =5 or k 2 =7 or k 2 ＝9，S 2 The method comprises the steps of representing, by adopting a sliding window technology, each low-resolution face image in a face image database and a corresponding high-resolution face image,/>
Figure BDA00018578385400001414
Divided into mutually overlapping dimensions of size k 2 ×k 2 Total number of image blocks, S 2 ＝(W-k 2 +1)×(H-k 2 +1)，1≤s 2 ≤S 2 ，/>
Figure BDA00018578385400001415
Has dimension of Mx 1->
Figure BDA00018578385400001416
To be according to capturing>
Figure BDA00018578385400001417
Is obtained in the same manner, the projection matrix is evaluated>
Figure BDA00018578385400001418
Represents->
Figure BDA00018578385400001419
The pixel values of all the pixel points in the column are arranged to form a corresponding column vector, and then the column vector is compared with the pixel value of the corresponding pixel point>
Figure BDA00018578385400001420
Represents->
Figure BDA00018578385400001421
The mth element of (4), is selected>
Figure BDA00018578385400001422
Represents->
Figure BDA00018578385400001423
The m-1 th element of (4), is selected>
Figure BDA00018578385400001424
Represents->
Figure BDA00018578385400001425
M-th column vector of (1) 4 Is a constant, λ 4 E (0,1), generally given as λ 4 =0.1,0.3,0.5, in this example λ 4 ＝0.3，/>
Figure BDA00018578385400001426
In accordance with the obtaining>
Figure BDA00018578385400001427
In the same manner, a projection matrix obtained in the same manner>
Figure BDA00018578385400001428
Represents->
Figure BDA00018578385400001429
In and->
Figure BDA00018578385400001430
The pixel values of all the pixel points in the corresponding area are arranged to form a corresponding column vector, and the corresponding column vector is greater than or equal to>
Figure BDA00018578385400001431
Represents->
Figure BDA00018578385400001432
The m-th column vector of (1).
To further illustrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.
Here, the method of the present invention was tested using FEI face data sets. The FEI face data set contains two high-resolution face images of 200 different persons (100 men and 100 women), one of which is a high-resolution face image with a normal expression, and the other is a high-resolution face image with a smile expression. Each high-resolution face image in the FEI face data set is subjected to down-sampling to obtain a corresponding low-resolution face image, the size of the down-sampled low-resolution face image is 30 × 25, and gaussian noises with standard deviations σ =10 and σ =30 are respectively added to all the low-resolution face images. In the experiment, 360 high-resolution face images of 180 persons in total and low-resolution face images corresponding to each high-resolution face image are randomly selected to form a training set, and 40 low-resolution face images of the rest 20 persons in total are used as tested low-resolution face images.
In order to verify the effectiveness of the method, the method is compared with other existing excellent Face Super-Resolution methods, such as Jiang J, ma J, chen C, et al. Noise Robust Face Image Super-Resolution Through Smooth Sparse reconstruction [ J ] IEEE Transactions on Cybernetics,2017, PP (99): 1-12. The method based on Smooth Super-Resolution reconstruction (SSR) proposed in the paper is compared with a method based on Smooth Super-Resolution reconstruction (SSR) in which CCA mapping is added once and reconstruction is performed as a layer (only one division of the size of an Image block is considered in the reconstruction process), and a method based on Smooth Super-Resolution reconstruction (SSR) in which CCA mapping is added twice and reconstruction is performed as a layer (only one division of the size of the Image block is considered in the reconstruction process) is performed as a layer. The method is to add two CCA mapping and two-layer (reconstruction, two image block sizes are considered in the reconstruction process, the image blocks with large sizes are firstly reconstructed, then the reconstruction result of the image blocks with large sizes is used for constraint, and the image blocks with small sizes are secondarily reconstructed) reconstruction on the basis of a smooth super-resolution reconstruction method (SSR). The sizes of the image blocks of the first sub-blocks of the low-resolution face images and the high-resolution face images in the training set and the tested low-resolution face images are 5 multiplied by 5, and the sizes of the image blocks of the second sub-blocks are 3 multiplied by 3.
The method comprises the steps of respectively adopting a super-resolution reconstruction method (SSR) based on smoothness, adding a method (simply referred to as CCA single layer) of CCA mapping and single-layer reconstruction forming on the basis of the super-resolution reconstruction method (SSR) based on smoothness, adding a method (simply referred to as 2CCA single layers) of CCA mapping and single-layer reconstruction forming on the basis of the super-resolution reconstruction method (SSR) based on smoothness, reconstructing the tested low-resolution face image, and giving an average PSNR and an SSIM of the high-resolution face image obtained after 40 tested low-resolution face images are reconstructed by adopting the methods under different noise environments (sigma =10 and sigma = 30) in table 1. As can be seen from the data listed in table 1, under the condition of severe noise, the method of the present invention has 0.75 improvement on PSNR compared to the SSR method, and has 0.0675 improvement on SSIM compared to the SSR method; meanwhile, the method is superior to a single-layer reconstruction method, namely a CCA single-layer method and a 2CCA single-layer method, in both PSNR indexes and SSIM indexes.
Table 1 shows the average PSNR and SSIM of the 40 tested low-resolution face images obtained by reconstructing the high-resolution face images by the above methods under different noise environments (σ =10 and σ = 30), respectively
Figure BDA0001857838540000161
FIG. 2 shows a noisy face image; FIG. 3 shows a high resolution face image reconstructed from the noisy face image shown in FIG. 2 using a conventional smoothing-based super-resolution reconstruction method (SSR); fig. 4 shows a high-resolution face image obtained by reconstructing the noise face image shown in fig. 2 by adding CCA mapping once and performing single-layer reconstruction, i.e., one-layer reconstruction, on the basis of the existing smoothing-based super-resolution reconstruction method (SSR); fig. 5 shows a high-resolution face image obtained by reconstructing the noise face image shown in fig. 2 by adding two CCA mappings based on the existing smoothing-based super-resolution reconstruction method (SSR) and performing single-layer reconstruction, i.e., one-layer reconstruction; FIG. 6 shows a high resolution face image reconstructed from the noise face image shown in FIG. 2 by the method of the present invention; fig. 7 shows a real high resolution face image corresponding to the noise face image shown in fig. 2. Comparing fig. 3, fig. 4, fig. 5, fig. 6 with fig. 7, it is obvious that the edge structure information of the high resolution face image shown in fig. 6 is clear, the details are clear, the reconstruction effect is good, and the edge structure information is closer to the real high resolution face image shown in fig. 7.

Claims (1)

1. A human face image multilayer reconstruction method based on CCA space is characterized by comprising the following steps:
the method comprises the following steps: selecting a face image database which contains at least two low-resolution face images and a high-resolution face image corresponding to each low-resolution face image, and recording the nth low-resolution face image and the corresponding high-resolution face image in the face image database as corresponding
Figure FDA0001857838530000011
And &>
Figure FDA0001857838530000012
And recording the tested low-resolution face image as->
Figure FDA0001857838530000013
Wherein N is a positive integer, N is more than or equal to 1 and less than or equal to N, N represents the total number of low-resolution face images contained in the face image database, N is more than or equal to 2, and/or is greater than or equal to N>
Figure FDA0001857838530000014
And &>
Figure FDA0001857838530000015
Are all W in width>
Figure FDA0001857838530000016
And &>
Figure FDA0001857838530000017
All the heights of (A) are H;
step two: dividing each low-resolution face image in face image database into S by adopting sliding window technology 1 Each overlapping with a dimension of k 1 ×k 1 Image blocks of
Figure FDA0001857838530000018
S of (1) 1 Each image block is recorded as>
Figure FDA0001857838530000019
Similarly, a sliding window technology is adopted to divide the high-resolution face image corresponding to each low-resolution face image in the face image database into S 1 Each overlapping with a dimension of k 1 ×k 1 Will->
Figure FDA00018578385300000110
S of (1) 1 Each image block is recorded as>
Figure FDA00018578385300000111
Will be picked up using a sliding window technique>
Figure FDA00018578385300000112
Is divided into 1 Each overlapping with a dimension of k 1 ×k 1 Will->
Figure FDA00018578385300000113
S of (1) 1 Each image block is recorded as>
Figure FDA00018578385300000114
Wherein the size of the sliding window is k 1 ×k 1 ，k 1 =5,7,9,11, sliding step of sliding window is 1 pixel point, S 1 ＝(W-k 1 +1)×(H-k 1 +1)，s 1 Is a positive integer, s is not less than 1 1 ≤S 1 ；
Step three: arranging the pixel values of all pixel points in each image block in each low-resolution face image in a face image database to form corresponding column vectors, and arranging the pixel values of all pixel points in each image block in each low-resolution face image in the face image database to form a column vector
Figure FDA00018578385300000115
The corresponding column vector is marked +>
Figure FDA00018578385300000116
Similarly, arranging the pixel values of all pixel points in each image block in the high-resolution face image corresponding to each low-resolution face image in the face image database to form a corresponding column vector, and then judging whether the pixel values of all pixel points in each image block in the high-resolution face image correspond to the low-resolution face images in the face image database or not>
Figure FDA00018578385300000117
The corresponding column vector is marked as +>
Figure FDA00018578385300000118
Will be/are>
Figure FDA00018578385300000119
The pixel values of all the pixel points in each image block are arranged to form a corresponding column vector, and the ^ is greater than or equal to>
Figure FDA00018578385300000120
The corresponding column vector is marked as +>
Figure FDA00018578385300000121
Then, column vectors corresponding to image blocks at the same position in all low-resolution face images in the face image database form a low-resolution dictionary to form S 1 A low resolution dictionary for mapping the s-th image to all low resolution face images in the face image database 1 A low-resolution dictionary formed by column vectors corresponding to image blocks is recorded as->
Figure FDA0001857838530000021
Similarly, column vectors corresponding to image blocks at the same position in all high-resolution face images in the face image database form a high-resolution dictionary, and form S 1 A high resolution dictionary for comparing the s-th of all high resolution face images in the face image database 1 The high-resolution dictionary formed by the column vectors corresponding to the image blocks is marked as->
Figure FDA0001857838530000022
Wherein it is present>
Figure FDA0001857838530000023
Figure FDA0001857838530000024
Are all (k) 1 ×k 1 )×1，/>
Figure FDA0001857838530000025
Are all (k) 1 ×k 1 )×N，/>
Figure FDA0001857838530000026
Is/is->
Figure FDA0001857838530000027
Is/is->
Figure FDA0001857838530000028
Step four: calculating the projection matrix corresponding to each low-resolution dictionary and each high-resolution dictionary respectively, and calculating the projection matrix
Figure FDA0001857838530000029
The corresponding projection matrix is recorded as +>
Figure FDA00018578385300000210
Will->
Figure FDA00018578385300000211
The corresponding projection matrix is recorded as +>
Figure FDA00018578385300000212
Wherein it is present>
Figure FDA00018578385300000213
And &>
Figure FDA00018578385300000214
All dimensions of (a) are L × (k) 1 ×k 1 ) L represents the dimension of CCA space, and L is equal to {1,2, …, k 1 ×k 1 }；
Step five: mapping each low-resolution dictionary from an image space to a CCA space to obtain a corresponding primary mapping low-resolution dictionary, and mapping each low-resolution dictionary to a CCA space
Figure FDA00018578385300000215
The corresponding one-time mapped low resolution dictionary is &>
Figure FDA00018578385300000216
Figure FDA00018578385300000217
Similarly, mapping each high-resolution dictionary from the image space to the CCA space to obtain a corresponding once-mapped high-resolution dictionary, and combining>
Figure FDA00018578385300000218
Corresponding one-time mapped high resolution dictionary +>
Figure FDA00018578385300000219
Figure FDA00018578385300000220
Wherein +>
Figure FDA00018578385300000221
And &>
Figure FDA00018578385300000222
The dimensions of (A) are all L multiplied by N;
step six: calculate each primary mapSparse coefficient vectors of the low resolution dictionary
Figure FDA00018578385300000223
Is marked as->
Figure FDA00018578385300000224
By counting->
Figure FDA00018578385300000225
Obtaining; then carrying out once sparse updating on the once mapped low-resolution dictionary by using the sparse coefficient vector of each once mapped low-resolution dictionary to obtain the updated dictionary of each once mapped low-resolution dictionary, and then judging whether the updated dictionary is the same as the updated dictionary or not>
Figure FDA00018578385300000226
The updated dictionary is recorded as
Figure FDA00018578385300000227
If/or>
Figure FDA00018578385300000228
Will be a non-zero element, will->
Figure FDA00018578385300000229
The nth column vector of (a) is extracted and the slave is then taken>
Figure FDA00018578385300000230
All column vectors extracted in (are) formed in the original order->
Figure FDA00018578385300000231
Similarly, sparsely updating each once-mapped high-resolution dictionary to obtain a dictionary updated by each once-mapped high-resolution dictionary, and combining>
Figure FDA0001857838530000031
The updated dictionary is recorded as +>
Figure FDA0001857838530000032
If/or>
Figure FDA0001857838530000033
Will be a non-zero element, will->
Figure FDA0001857838530000034
The nth column vector of (a) is extracted and the slave is then taken>
Figure FDA0001857838530000035
All column vectors extracted in (a) are formed in original order->
Figure FDA0001857838530000036
Wherein it is present>
Figure FDA0001857838530000037
Dimension of (a) is Nx 1, argmin () represents solving the residual minimum value, the symbol "| | | | non-conducting phosphor 2 Is "is 2 Norm regular term operation symbol, symbol "| | | | non-woven phosphor 1 Is "is 1 Norm regular term operator sign, λ 1 Is a constant, λ 1 ∈(0,1)，/>
Figure FDA0001857838530000038
And &>
Figure FDA0001857838530000039
Has dimension L × M, M denotes->
Figure FDA00018578385300000310
The total number of the non-zero elements in the alloy is more than or equal to 1, and M is less than N;
step seven: the updated dictionary of each once-mapped low-resolution dictionary is reversely mapped from the CCA space back to the image spaceTo obtain a corresponding reflection low resolution dictionary
Figure FDA00018578385300000311
The corresponding retroreflection low resolution dictionary is &>
Figure FDA00018578385300000312
Similarly, the updated dictionary of each once-mapping high-resolution dictionary is back-mapped to the image space from the CCA space to obtain a corresponding reflection high-resolution dictionary, and the dictionary is/are>
Figure FDA00018578385300000313
The corresponding reflection high resolution dictionary is recorded as>
Figure FDA00018578385300000314
Wherein it is present>
Figure FDA00018578385300000315
And &>
Figure FDA00018578385300000316
Has a dimension of (k) 1 ×k 1 )×M；
Step eight: calculating the projection matrix corresponding to each back mapping low-resolution dictionary and each back mapping high-resolution dictionary respectively, and calculating the projection matrix corresponding to each back mapping low-resolution dictionary and each back mapping high-resolution dictionary
Figure FDA00018578385300000317
The corresponding projection matrix is recorded as +>
Figure FDA00018578385300000318
Will->
Figure FDA00018578385300000319
The corresponding projection matrix is recorded as +>
Figure FDA00018578385300000320
Wherein it is present>
Figure FDA00018578385300000321
And &>
Figure FDA00018578385300000322
All dimensions of (a) are L × (k) 1 ×k 1 ) L represents the dimension of CCA space, and L is equal to {1,2, …, k 1 ×k 1 }；
Step nine: mapping each reverse mapping low-resolution dictionary from the image space to the CCA space to obtain a corresponding re-mapping low-resolution dictionary, and mapping each reverse mapping low-resolution dictionary from the image space to the CCA space
Figure FDA00018578385300000323
Corresponding remap low resolution dictionary is recorded as>
Figure FDA00018578385300000324
Figure FDA00018578385300000325
Similarly, mapping each reflection high-resolution dictionary from the image space to the CCA space to obtain a corresponding re-mapping high-resolution dictionary, and combining>
Figure FDA00018578385300000326
Corresponding remapped high resolution dictionary @>
Figure FDA00018578385300000327
Figure FDA00018578385300000328
Will->
Figure FDA00018578385300000329
Is mapped from image space to CCA space, resulting in ∑>
Figure FDA00018578385300000330
A primary mapping block corresponding to each image block in the image data will be
Figure FDA00018578385300000331
The corresponding one-time mapping block is marked as->
Figure FDA00018578385300000332
Will->
Figure FDA0001857838530000041
The pixel values of all pixel points in the primary mapping block corresponding to each image block are arranged to form a corresponding column vector, and the pixel values are compared with the pixel values in the primary mapping block corresponding to each image block to determine whether the pixel values are greater than or equal to the pixel values in the primary mapping block>
Figure FDA0001857838530000042
The corresponding column vector is marked +>
Figure FDA0001857838530000043
Figure FDA0001857838530000044
Wherein it is present>
Figure FDA0001857838530000045
And &>
Figure FDA0001857838530000046
Are all LxM->
Figure FDA0001857838530000047
Dimension of (a) is L × 1;
step ten: computing the sum of vectors per column in each remapped low resolution dictionary
Figure FDA0001857838530000048
The euclidean distance of the column vector corresponding to the primary mapping block corresponding to each image block in (b), for->
Figure FDA0001857838530000049
Calculate->
Figure FDA00018578385300000410
And/or for each column vector in>
Figure FDA00018578385300000411
Is based on the Euclidean distance of->
Figure FDA00018578385300000412
Obtaining M Euclidean distances; then, aiming at each M Euclidean distances obtained by re-mapping the low-resolution dictionary, sequencing the M Euclidean distances from large to small; then according to the magnitude sequence of M Euclidean distances obtained by aiming at each remapped low-resolution dictionary, carrying out position adjustment on all column vectors in each remapped low-resolution dictionary, recombining to obtain a corresponding recombined low-resolution dictionary, and then combining>
Figure FDA00018578385300000413
The corresponding recombined low resolution dictionary is marked>
Figure FDA00018578385300000414
The 1 st column vector and +>
Figure FDA00018578385300000415
Has a maximum Euclidean distance, and>
Figure FDA00018578385300000416
and/or the last column vector in (b)>
Figure FDA00018578385300000417
Has the smallest euclidean distance; wherein +>
Figure FDA00018578385300000418
Dimension of (d) is L × M;
similarly, calculate the sum of each column vector in each remapped high resolution dictionary
Figure FDA00018578385300000419
Is used for ^ ing the Euclidean distance of the column vector corresponding to the primary mapping block corresponding to each image block in the>
Figure FDA00018578385300000420
Calculate->
Figure FDA00018578385300000421
And/or for each column vector in>
Figure FDA00018578385300000422
Is based on the Euclidean distance of->
Figure FDA00018578385300000423
Obtaining M Euclidean distances; then, aiming at each M Euclidean distances obtained by re-mapping the high-resolution dictionary, sequencing the M Euclidean distances from large to small; then according to the sequence of the M Euclidean distances obtained by aiming at each high-resolution dictionary re-mapped, carrying out position adjustment on all column vectors in each high-resolution dictionary re-mapped, recombining to obtain a corresponding recombined high-resolution dictionary, and then combining>
Figure FDA00018578385300000424
Corresponding reorganized high resolution dictionary
Figure FDA00018578385300000425
The 1 st column vector and +>
Figure FDA00018578385300000426
Has the largest Euclidean distance and is greater than or equal to>
Figure FDA00018578385300000427
And the last column vector and->
Figure FDA00018578385300000428
Has the smallest euclidean distance; wherein it is present>
Figure FDA00018578385300000429
Dimension of (d) is L × M;
step eleven: computing
Figure FDA0001857838530000051
Will @, for each image block of the first sparse coefficient vector>
Figure FDA0001857838530000052
Is marked as £ the first sparse coefficient vector of>
Figure FDA0001857838530000053
By passing
Figure FDA0001857838530000054
Calculating to obtain; will then->
Figure FDA0001857838530000055
The high-resolution face image obtained after reconstruction in one layer is recorded as ^ er>
Figure FDA0001857838530000056
Will->
Figure FDA0001857838530000057
In and->
Figure FDA0001857838530000058
The area corresponding to the position is marked as->
Figure FDA0001857838530000059
Will->
Figure FDA00018578385300000510
The column vector formed by arranging the pixel values of all the pixel points is recorded as ^ er>
Figure FDA00018578385300000511
Figure FDA00018578385300000512
Wherein it is present>
Figure FDA00018578385300000513
The dimension of M is multiplied by 1,m is a positive integer, M is more than or equal to 1 and less than or equal to M, and lambda is 2 And λ 3 Are all constant, λ 2 ∈(0,1)，λ 3 ∈(0,1)，/>
Figure FDA00018578385300000514
Represents->
Figure FDA00018578385300000515
The mth element of (4), is selected>
Figure FDA00018578385300000516
Represents->
Figure FDA00018578385300000517
The mth column vector of (4), based on the number of cells in the column->
Figure FDA00018578385300000518
Represents->
Figure FDA00018578385300000519
M-1 element of (1); />
Step twelve: the size of the sliding window is changed to k 2 ×k 2 (ii) a Then S is obtained in the same manner according to the process from step two to step ten 2 Has low recombinationResolution dictionary and S 2 Reorganize the high resolution dictionary into s 2 The recombined low resolution dictionary is noted
Figure FDA00018578385300000520
Will be(s) 2 Recombined high-resolution dictionary is marked as>
Figure FDA00018578385300000521
Then calculates->
Figure FDA00018578385300000522
Will ∑ the second sparse coefficient vector of each image block of>
Figure FDA00018578385300000523
S of (1) 2 Image block->
Figure FDA00018578385300000524
Is recorded as a second sparse coefficient vector
Figure FDA00018578385300000525
By passing
Figure FDA00018578385300000526
Calculating to obtain; then will be
Figure FDA00018578385300000527
The high-resolution face image obtained after the two-layer reconstruction is recorded as ^ er>
Figure FDA00018578385300000528
Will->
Figure FDA00018578385300000529
In and->
Figure FDA00018578385300000530
Zone corresponding to positionField is recorded as->
Figure FDA00018578385300000531
Will->
Figure FDA00018578385300000532
The column vector formed by arranging the pixel values of all the pixel points is recorded as ^ er>
Figure FDA00018578385300000533
Figure FDA00018578385300000534
Wherein +>
Figure FDA00018578385300000535
And &>
Figure FDA00018578385300000536
Dimension of L × M, k 2 =3,5,7,9 and 1 < k 2 ＜k 1 ，S 2 Representing that each low-resolution face image in a face image database and the corresponding high-resolution face image are combined by adopting a sliding window technology>
Figure FDA00018578385300000537
Divided into mutually overlapping dimensions of size k 2 ×k 2 Total number of image blocks, S 2 ＝(W-k 2 +1)×(H-k 2 +1)，1≤s 2 ≤S 2 ，/>
Figure FDA0001857838530000061
Has dimension of Mx 1->
Figure FDA0001857838530000062
To be according to capturing>
Figure FDA0001857838530000063
In (2)Projection matrices obtained in the same manner>
Figure FDA0001857838530000064
Represents->
Figure FDA0001857838530000065
The pixel values of all the pixel points in the image are arranged to form a corresponding column vector,
Figure FDA0001857838530000066
represents->
Figure FDA0001857838530000067
The mth element of (4), is selected>
Figure FDA0001857838530000068
Represents->
Figure FDA0001857838530000069
The m-1 th element in (a), a>
Figure FDA00018578385300000610
Represents->
Figure FDA00018578385300000611
M-th column vector of (1) 4 Is a constant, λ 4 ∈(0,1)，/>
Figure FDA00018578385300000612
To be according to capturing>
Figure FDA00018578385300000613
In the same manner, a projection matrix obtained in the same manner>
Figure FDA00018578385300000614
Represents->
Figure FDA00018578385300000615
In and->
Figure FDA00018578385300000616
The pixel values of all the pixel points in the corresponding area are arranged to form a corresponding column vector, and the corresponding column vector is greater than or equal to>
Figure FDA00018578385300000617
Represents->
Figure FDA00018578385300000618
The m-th column vector of (1). />
CN201811322383.9A 2018-11-08 2018-11-08 Face image multilayer reconstruction method based on CCA space Active CN109712069B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811322383.9A CN109712069B (en) 2018-11-08 2018-11-08 Face image multilayer reconstruction method based on CCA space

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811322383.9A CN109712069B (en) 2018-11-08 2018-11-08 Face image multilayer reconstruction method based on CCA space

Publications (2)

Publication Number Publication Date
CN109712069A CN109712069A (en) 2019-05-03
CN109712069B true CN109712069B (en) 2023-04-07

Family

ID=66254200

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811322383.9A Active CN109712069B (en) 2018-11-08 2018-11-08 Face image multilayer reconstruction method based on CCA space

Country Status (1)

Country Link
CN (1) CN109712069B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110942495A (en) * 2019-12-12 2020-03-31 重庆大学 CS-MRI image reconstruction method based on analysis dictionary learning

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101615290A (en) * 2009-07-29 2009-12-30 西安交通大学 A kind of face image super-resolution reconstruction method based on canonical correlation analysis
CN101697197A (en) * 2009-10-20 2010-04-21 西安交通大学 Method for recognizing human face based on typical correlation analysis spatial super-resolution
CN107169928A (en) * 2017-05-12 2017-09-15 武汉华大联创智能科技有限公司 A kind of human face super-resolution algorithm for reconstructing learnt based on deep layer Linear Mapping

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10430922B2 (en) * 2016-09-08 2019-10-01 Carnegie Mellon University Methods and software for generating a derived 3D object model from a single 2D image

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101615290A (en) * 2009-07-29 2009-12-30 西安交通大学 A kind of face image super-resolution reconstruction method based on canonical correlation analysis
CN101697197A (en) * 2009-10-20 2010-04-21 西安交通大学 Method for recognizing human face based on typical correlation analysis spatial super-resolution
CN107169928A (en) * 2017-05-12 2017-09-15 武汉华大联创智能科技有限公司 A kind of human face super-resolution algorithm for reconstructing learnt based on deep layer Linear Mapping

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
姚正元 ; 郭立君 ; 张荣 ; .基于CCA空间的平滑稀疏超分辨率人脸重构.传感器与微系统.2018,(第04期),全文. *

Also Published As

Publication number Publication date
CN109712069A (en) 2019-05-03

Similar Documents

Publication Publication Date Title
CN110020989B (en) Depth image super-resolution reconstruction method based on deep learning
Zhang et al. Adaptive residual networks for high-quality image restoration
CN106952228B (en) Super-resolution reconstruction method of single image based on image non-local self-similarity
CN110111256B (en) Image super-resolution reconstruction method based on residual distillation network
CN109087258B (en) Deep learning-based image rain removing method and device
CN106952317B (en) Hyperspectral image reconstruction method based on structure sparsity
CN113962893A (en) Face image restoration method based on multi-scale local self-attention generation countermeasure network
CN111127374A (en) Pan-sharing method based on multi-scale dense network
CN111222519B (en) Construction method, method and device of hierarchical colored drawing manuscript line extraction model
CN111415323B (en) Image detection method and device and neural network training method and device
CN110210282A (en) A kind of moving target detecting method decomposed based on non-convex low-rank sparse
CN112634120A (en) Image reversible watermarking method based on CNN prediction
CN114693577B (en) Infrared polarized image fusion method based on Transformer
Chetty et al. Digital video tamper detection based on multimodal fusion of residue features
CN109712069B (en) Face image multilayer reconstruction method based on CCA space
CN109146785A (en) A kind of image super-resolution method based on the sparse autocoder of improvement
CN115439325A (en) Low-resolution hyperspectral image processing method and device and computer program product
Shi et al. Exploiting multi-scale parallel self-attention and local variation via dual-branch transformer-cnn structure for face super-resolution
CN111611962A (en) Face image super-resolution identification method based on fractional order multi-set partial least square
CN116523985B (en) Structure and texture feature guided double-encoder image restoration method
CN110569763B (en) Glasses removing method for fine-grained face recognition
CN115375537A (en) Nonlinear sensing multi-scale super-resolution image generation system and method
CN115619681A (en) Image reconstruction method based on multi-granularity Vit automatic encoder
CN111275624B (en) Face image super-resolution reconstruction and identification method based on multi-set typical correlation analysis
CN115035170A (en) Image restoration method based on global texture and structure

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant