CN105550988A - Super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity - Google Patents

Abstract

The invention discloses a super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity, comprising the following steps: first, the neighborhood embedding method is improved by use of structure similarity, more accurate high-frequency initial estimation is obtained, and an initial estimation algorithm based on neighborhood embedding is realized; and then, the local self-similarity and multi-scale structure similarity of a low-resolution image are used to construct a reconstruction constraint for the purpose of reconstructing high resolution, and a sparse representation dictionary is established. Compared with the prior art, on the basis that the algorithm put forward by the invention solves the problem that the learning-based super-resolution reconstruction algorithm of predecessors needs a lot of training sets, the neighborhood embedding method is improved, the method is adopted to solve the problem of inaccurate high-frequency initial estimation in a super-resolution algorithm based on local self-similarity and multi-scale similarity, and the super-resolution reconstruction effect of images is enhanced; and the saw-tooth effect and the ringing effect are suppressed better, and a reconstructed high-resolution image is closer to the real image and is of better subjective and objective quality.

Description

Super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity

Technical Field

The invention relates to the field of image processing, in particular to a super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity.

Background

Since the concept of super-resolution was proposed by Tsai et al in 1984, the super-resolution reconstruction technique has attracted much attention, and numerous super-resolution reconstruction algorithms have been proposed. These algorithms can be broadly classified into 3 types of interpolation-based methods, reconstruction-based methods, and learning-based methods. The interpolation-based method has low operation complexity and high running speed, but the interpolated image usually lacks high-frequency details and is easy to cause edge blurring; the super-resolution reconstruction is carried out by utilizing a degraded model of an image and specific priori knowledge based on a reconstruction method, the commonly used priori models in the general method comprise Gaussian prior hypothesis (Gaussian Process models), HuberMRF (HuberMRF), total variation (TotalVarioration) models, smooth boundary (Softedge) models, gradient profile (GradientProfile) models and the like, but due to the limitation of the priori knowledge, structural information and texture information cannot be effectively recovered, and the reconstruction effect is not obvious; learning-Based methods include an Example-Based method, a neighborhood embedding method (neighbor embedding) and a sparse representation method (sparse representation), and their basic idea is to establish a corresponding relationship between a large number of low-resolution blocks and high-resolution image blocks corresponding to the low-resolution blocks by learning them, and then guide the high-resolution image block reconstruction by the relationship. Among the 3 super-resolution reconstruction methods, the learning-based method can introduce more high-frequency information than the other two methods, and has stronger robustness to noise, so that the method becomes a research hotspot in recent years.

In the learning-based super-resolution reconstruction method, if training images are adopted, a huge external image training library is required, so that the memory consumption of the algorithm is large. To address this problem, many super-resolution reconstruction methods using non-local self-similar structures have emerged. This structural similarity is often prevalent in natural images in apparent or latent form, which provides rich additional information for image super-resolution reconstruction. Suetaken et al propose to utilize additional information provided by similar image blocks of different scales of the image itself to establish an internal dictionary, and then use a learning-based method to perform super-resolution reconstruction; GlasnerD and the like add additional information provided by similar image blocks with the same scale and different scales into a reconstruction model at the same time, and search similar structure image blocks in the whole image; ZhangK combines an image multi-scale structure self-similarity model with a compressed sensing theory, so that the image reconstruction effect is further improved. However, when the super-resolution reconstruction algorithm adopting the structure self-similarity is used for constructing the sparse representation model, the fact that image blocks with non-local similar properties have the same or similar sparse representation coefficients is ignored. Aiming at the problem, DongW and the like propose a method for comprehensively considering the multi-scale similarity and the non-local similarity of the images and integrating the multi-scale similarity and the non-local similarity into a compressed sensing model to establish a new image super-resolution, so as to obtain a good reconstruction effect.

Disclosure of Invention

In order to solve the problem of difficult fault isolation of a direct current system in the prior art, the invention provides an image super-resolution reconstruction algorithm which effectively combines neighborhood embedding and structure self-similarity in consideration of the fact that whether initial estimation is accurate or not directly influences the quality and the iteration times of image reconstruction.

The invention provides a super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity, which comprises the following steps:

step (1), based on initial estimation algorithm of neighborhood embedding, firstly extracting high-resolution training image I_HThen a down-sampling operation is performed on the high resolution image to obtain a low resolution image I_LDividing the high-resolution and low-resolution images into small blocks with mutually overlapped regions; let X be { X ═ X^mM 1.. p } is a set of low resolution blocks that are trained, Y ═ Y ·^mM 1.. p } is a set of high resolution blocks corresponding thereto, where p is the number of small blocks segmented from the training image; similarly, noteA set of blocks segmented for the low resolution image to be reconstructed,a set of blocks of a high resolution image to be estimated, wherein q is the number of segmented small blocks in the test image; image block x^mAnd image blockIs defined as D_pqWherein D is_pqIs listed as nAnd X represents a column vector formed by distances of all image blocks in the training low-resolution block set; then traverse D_pqEach column in the image block search unit finds K maximum values, and the index of the maximum value is corresponding to the image block, i.e. the image block to be searchedK neighboring blocks in the training set, denoted as Mapping of representations to a desired image blockT1, t 2.. tK denotes the sequence numbers of the K neighboring blocks;

then for each test image blockMethod for obtaining optimal weight vector omega by minimizing local reconstruction error_n＝[ω_n1,ω_n2,...ω_np]N1, n 2.. nP indicates the subscript of the sequence number, n indicates the nth test image block:

${\mathrm{\ω}}_n=\arg \underset{{\mathrm{\ω}}_n}{\min }\left|\right|{\hat{x}}^n-\underset{m=1}{\overset{p}{\Σ}}{\mathrm{\ω}}_{nm}{x}^m|{|}_2^2---\left(7\right)$

wherein each weight value omega_nmThe following constraints are satisfied:

$\left\{\begin{array}{ccc}\underset{m=1}{\overset{p}{\Σ}}{\mathrm{\ω}}_{nm}=1& & \\ {\mathrm{\ω}}_{nm}\≠0& if& x^m\∈S_l^n\\ {\mathrm{\ω}}_{nm}=0& if& x^m\∉S_l^m\end{array}\right.---\left(8\right)$

method for solving omega by using nuclear regression mode_nmObtaining each high resolution image blockEstimated value of (a):

${\hat{y}}^n=\underset{m=1}{\overset{p}{\Σ}}{\mathrm{\ω}}_{nm}{y}^m---\left(9\right)$

taking the average of all overlapped pixel values of the overlapped areas of all high-resolution image blocks;

the residual error e is defined by_lFor image blocksWith another image blockThe smaller the residual error, the more similar the two image blocks:

$e_l=\left|\right|{\hat{y}}_l-{\tilde{y}}_l|{|}_2^2---\left(10\right)$

for each image blockComputing image blocksAnd searching residual errors of all image blocks in the neighborhood, and finding L image blocks with the minimum residual errors as the image blocksL most similar block sets:

$\tilde{Y}=\{{\tilde{y}}^{ni},i=1,\mathrm{2...}L\},$

with residual errors of respectivelyL represents the number of image blocks; then the image block to be estimatedIs shown asRepresents a linear combination of one of the L image blocks found to have the smallest residual error:

${\hat{y}}^n=\underset{i=1}{\overset{L}{\Σ}}{\mathrm{\ρ}}_n^i{\tilde{y}}^{ni}---\left(11\right)$

wherein,each element in (1)Andsimilarity weight betweenCalculated from the following formula:

${\mathrm{\ρ}}_n^i=\frac{\exp (-\frac{e_l^{ni}}{h})}{\underset{i=1}{\overset{L}{\Σ}}\exp (-\frac{e_l^{ni}}{h})}---\left(12\right)$

h is a control factor of the weight;andeach element ofVegetable extractThe similarity weight between them is defined as:

$T_n^j=\left\{\begin{array}{cc}{\mathrm{\ρ}}_n^i& \begin{array}{cc}if& {\hat{y}}^j\∈\tilde{Y},{\hat{y}}^j={\tilde{y}}^{ni}\end{array}\\ 0& otherwise\end{array}\right.---\left(13\right)$

representing the jth block in the set of high-resolution image blocks to be estimated,for image blocksL most similar block sets;

order toIs composed ofThe vector of the composition is then calculated,expression (11) is:

adding (14) serving as a non-local self-similarity regular term into a neighborhood embedding method, and constructing a neighborhood super-resolution reconstruction method model:

$\hat{Y}=\underset{\hat{Y}}{\arg \min }\left|\right|\hat{Y}-WY|{|}_2^2+\mathrm{\λ}||\hat{Y}-\mathrm{\Φ}\hat{Y}|{|}_2^2---\left(15\right)$

wherein W is ω of the lexicographic order_nI.e. W ═ ω₁,ω₂,...,ω_q]^TPhi is in lexical orderNamely, it is

The formula (15) is solved by a gradient descent method and simplified into

${\hat{Y}}^{t+1}={\hat{Y}}^t-\mathrm{\λ}{(I-\mathrm{\Φ})}^T(I-\mathrm{\Φ}){\hat{Y}}^t---\left(16\right)$

t is iteration times, and lambda is a regularization coefficient constant; is provided withFor the iterative initial value obtained by the neighborhood embedding method, the method_tThe iteration is repeated to obtain accurate high-frequency initial estimation

Step (2) establishing a sparse representation dictionary, and after the high-resolution initial estimation image obtained in the step (1) is subjected to blocking operation, for each high-resolution input block x to be reconstructed_iThen, it is compared with the trained cluster center { C }₁,C₂,...,C_nAre compared to find the cluster center C with the smallest Euclidean distance from the input block_iCluster K of which it is located_iThe corresponding sub-dictionary Ψ_iI.e. the high resolution block x to be reconstructed_iThe dictionary used;

obtaining a sub-dictionary psi corresponding to each image block to be reconstructed_iThen, the model is reconstructed by using the non-local autoregressive super-resolution $\{\mathrm{\α},x\}=\arg \underset{\mathrm{\α},x}{\min }\left|\right|y-DMx|{|}_2^2+\underset{i=1}{\overset{N}{\Σ}}{\mathrm{\λ}}_i|\left|{\mathrm{\α}}_i\right|{|}_1+\underset{i=1}{\overset{N}{\Σ}}{\mathrm{\γ}}_i\left|\right|{\mathrm{\α}}_i-{\mathrm{\α}}_i^{*}|{|}_2^2+\mathrm{\θ}\underset{i=1}{\overset{N}{\Σ}}||R_{i}x-{\mathrm{\Ψ}}_i{\mathrm{\α}}_i|{|}_2^2---\left(6\right)$ Converting into two formulas (17) and (18) by Lagrange multiplier method,

$\mathrm{\α}=\arg \underset{\mathrm{\α}}{\min }\underset{i=1}{\overset{N}{\Σ}}{\mathrm{\λ}}_i\left|\right|{\mathrm{\α}}_i|{|}_1+\underset{i=1}{\overset{N}{\Σ}}{\mathrm{\γ}}_i||{\mathrm{\α}}_i-{\mathrm{\α}}_i^{*}|{|}_2^2+\mathrm{\θ}\underset{i=1}{\overset{N}{\Σ}}\left|\right|R_{i}x-{\mathrm{\Ψ}}_i{\mathrm{\α}}_i|{|}_2^2---\left(17\right)$

$x=\arg \underset{x}{\min }\left|\right|y-DWx|{|}_2^2+\mathrm{\θ}\underset{i=1}{\overset{N}{\Σ}}||R_{i}x-{\mathrm{\Ψ}}_i{\mathrm{\α}}_i|{|}_2^2---\left(18\right)$

wherein y represents an input low-resolution image, x represents a high-resolution image to be reconstructed, and D is a down-sampling matrix; m is a non-local self-similarity weight matrix and is used for describing the non-local self-similarity relation between the image blocks; Ψ_iA sub-dictionary corresponding to the ith image block, α is a sparse representation sparse matrix with each row α_iIn the sub-dictionary Ψ for the ith image block_iSparse representation of coefficient vectors under, α_iIs α_ij；α_ijSparse representation coefficients of a non-local self-similarity block for an ith image block; r_iThe function of the extraction matrix is to extract the ith image block from the image; lambda [ alpha ]_i、γ_iIs a weighted vector, each element of which ${\mathrm{\λ}}_{ij}=\frac{c_1}{\left|{\mathrm{\α}}_{ij}\right|+\mathrm{\ϵ}},{\mathrm{\γ}}_{ij}=\frac{c_2}{{({\mathrm{\α}}_{ij}-{{\mathrm{\α}}_{ij}}^{*})}^2+\mathrm{\ϵ}};$ Theta is a weighting coefficient;

and solving equations (17) and (18) iteratively; obtaining a final solution by an iterative contraction algorithm in the formula (17); to solve equation (18), the lagrange equation is constructed as:

$L(X,Z,\mathrm{\&tau;})=\left|\right|Y-DWX|{|}_2^2+\mathrm{\&theta;}||R_{i}X-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2+<Z,Y-DX>+\mathrm{\&tau;}\left|\right|Y-DX|{|}_2^2---\left(19\right)$

where Z is a lagrange multiplier, τ is a constant, equation (19) can be solved by iterating inter _ num times as follows, iter represents the current number of iterations:

$x^{(iter+1)}=\arg \underset{x}{\min }L(x,Z^{\left(iter\right)},{\mathrm{\&tau;}}^{\left(iter\right)})---\left(20\right)$

Z^(iter+1)＝Z^(iter)+τ^iter(y-Dx^(iter+1))(21)

τ^(iter+1)＝τ^(iter)(22)

order toThe solution is performed by reducing equation (20) to equation (23):

$x^{(iter+1)}={\&lsqb;{\left(DW\right)}^{T}DW+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}{R_i}^T{R}_i+{\mathrm{\&tau;}}^{\left(iter\right)}{D}^{T}D\&rsqb;}^{-1}\&lsqb;{\left(DW\right)}^{T}y+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}{R_i}^T{R}_i\left({\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i\right)+D^T{Z}^{\left(iter\right)}/2+{\mathrm{\&tau;}}^{\left(iter\right)}{D}^{T}y\&rsqb;---\left(23\right);$

and x is a high-resolution image to be reconstructed, and is an update multiple of the update constant tau, and is a constant.

Compared with the prior art, the algorithm provided by the invention improves the neighborhood embedding method on the basis of solving the defect that the learning-based super-resolution reconstruction algorithm of the predecessor needs a large number of training sets, and is used for solving the problem of inaccurate high-frequency initial estimation in the super-resolution algorithm based on local self-similarity and multi-scale similarity, thereby improving the super-resolution reconstruction effect of the image. Experimental results show that the algorithm provided by the invention can better inhibit the sawtooth effect and the ringing effect, and the reconstructed high-resolution image is closer to a real image and has better subjective and objective quality.

Drawings

Fig. 1 is a comparison of reconstruction results of different non-local self-similar block numbers: (1a) an original drawing; (1 b-1 f) respectively carrying out reconstruction results on the algorithm when the number of the non-local self-similar blocks is 5,10,15,20 and 25;

FIG. 2 is a result of the "Leaves" super-resolution reconstruction; : (2a) an original drawing; (2b) and Bicubic super-resolution reconstruction; (2c) NE; (2d) NARM; (2e) a text algorithm;

FIG. 3 shows the super-resolution reconstruction result of "Cameraman": (3a) an original drawing; (3b) and Bicubic super-resolution reconstruction; (3c) NE; (3d) NARM; (3e) a text algorithm;

FIG. 4 is a model diagram of the super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity.

Detailed Description

The technical solution of the present invention is further described below with reference to the accompanying drawings and the detailed description.

In the existing methods for super-resolution reconstruction by using compressed sensing, initial estimation needs to be performed on a high-resolution image. Based on the characteristic, and considering whether the accuracy of initial estimation directly influences the quality and the iteration times of image reconstruction, the invention provides an image super-resolution reconstruction algorithm which effectively combines neighborhood embedding and structure self-similarity. Firstly, a neighborhood embedding method is improved by using structural similarity, and then more accurate high-frequency initial estimation is obtained; and then, constructing a reconstruction constraint term to reconstruct high resolution by using the local self-similarity and the multi-scale structural similarity of the low-resolution image.

The technical scheme is as follows:

1. image super-resolution reconstruction model based on sparse representation

Let U and S be vectors and R be real space, thenThe image blocks of size constitute a column vector x ∈ R^U×1If there is a matrix Ψ and a vector α, then:

x＝Ψα，||α||₀＝k＜＜U(1)

then vector x is said to be sparse at Ψ, where Ψ ∈ R^U×SCalled a dictionary, each column in the dictionary Ψ called an atom in the dictionary, α∈ R^S×1Is a sparse representation coefficient and satisfies | | | α | non-calculation₀＝k＜＜U，||α||₀Representing α the number of non-zero elements according to sparse representation theory, an image block x can be represented as a linear combination of a few elements of the dictionary Ψ, and the dictionary representation can be described as an optimization problem as follows:

$\underset{\mathrm{\&alpha;}}{\min }\left|\right|x-\mathrm{\&Psi;}\mathrm{\&alpha;}|{|}_2^2+\mathrm{\&lambda;}|\left|\mathrm{\&alpha;}\right|{|}_1---\left(2\right)$

where λ is a regularization coefficient that balances the sparse term and the fidelity term, and (2) is referred to as the basic model of the sparse representation.

Assuming that x represents a vector composed of a high resolution image, y represents a vector composed of a low resolution image obtained by down-sampling the high resolution image, and the down-sampling matrix is denoted as D, the relationship between the high resolution image and the low resolution image can be expressed as:

y＝Dx(3)

reconstructing a high resolution image from a low resolution image solves the following least squares problem:

$\hat{x}=\arg \underset{x}{\min }\left|\right|y-Dx|{|}_2^2---\left(4\right)$

combining (2) and (4) to obtain a super-resolution image reconstruction model based on sparse representation:

$\mathrm{\&alpha;}=\arg \underset{\mathrm{\&alpha;}}{\min }\left|\right|y-Dx|{|}_2^2+\mathrm{\&lambda;}|\left|\mathrm{\&alpha;}\right|{|}_1++\mathrm{\&theta;}\left|\right|x-\mathrm{\&Psi;}\mathrm{\&alpha;}|{|}_2^2---\left(5\right)$

2. improved image super-resolution reconstruction based on structural self-similarity

In the invention, a non-local autoregressive super-resolution reconstruction model is utilized:

$\{\mathrm{\&alpha;},x\}=\arg \underset{\mathrm{\&alpha;},x}{\min }\left|\right|y-DMx|{|}_2^2+\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&lambda;}}_i|\left|{\mathrm{\&alpha;}}_i\right|{|}_1+\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&gamma;}}_i\left|\right|{\mathrm{\&alpha;}}_i-{\mathrm{\&alpha;}}_i^{*}|{|}_2^2+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}||R_{i}x-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2---\left(6\right)$

wherein y is an input low-resolution image, x is a high-resolution image to be reconstructed, and D is a down-sampling matrix; m is a non-local self-similarity weight matrix and is used for describing the non-local self-similarity relation between the image blocks; Ψ_iA sub-dictionary corresponding to the ith image block, α is a sparse representation sparse matrix with each row α_iIn the sub-dictionary Ψ for the ith image block_iSparse representation of coefficient vectors under, α_iIs α_ij；Sparse representation coefficients of a non-local self-similarity block for an ith image block; r_iThe function of the extraction matrix is to extract the ith image block from the image; lambda [ alpha ]_i、γ_iIs a weighted vector, each element of which ${\mathrm{\&lambda;}}_{ij}=\frac{c_1}{\left|{\mathrm{\&alpha;}}_{ij}\right|+\mathrm{\&epsiv;}},{\mathrm{\&gamma;}}_{ij}=\frac{c_2}{{({\mathrm{\&alpha;}}_{ij}-{{\mathrm{\&alpha;}}_{ij}}^{*})}^2+\mathrm{\&epsiv;}};$ And theta is a weighting coefficient.

In the model, α needs to perform iterative solution on x two quantities to be solved, so that the initial estimation of x is directly related to the iteration number of model solution, and meanwhile, the initial estimation of x also influences the sub-dictionary psi_iIn turn, affects the quality of the reconstructed image. But if the Bicubic interpolation method is simply adopted as the initial estimation of the high-resolution image, the edge aliasing effect of the image is caused. Based on the model, an initial estimation algorithm based on neighborhood embedding is provided, non-local self-similarity is used as prior information, a regular term of a neighborhood embedding method is added, high-frequency initial estimation with robustness is constructed, and super-resolution reconstruction is carried out on the basis.

2.1 initial estimation Algorithm based on neighborhood embedding

The neighborhood embedding algorithm assumes that the low-resolution image block and the high-resolution image block have similar local manifold, and estimates the missing high-frequency details in the low-resolution image by using the corresponding relation between the low-resolution image block and the high-resolution image block to perform super-resolution reconstruction. In the invention, a non-local self-similarity (NL) regular term is introduced to enhance the effect of the neighborhood embedding super-resolution reconstruction. The basic idea of the NL method is: similar blocks with the same scale are searched in a search range, the search range can be as large as the whole image because the similar blocks are possibly far away from the block to be estimated in geometric position, and the high-resolution image is reconstructed by utilizing complementary information provided by the similar image blocks. The specific algorithm herein is as follows:

firstly, extracting a high-resolution training image I_HThen a down-sampling operation is performed on the high resolution image to obtain a low resolution image I_LAnd the high resolution and low resolution images are divided into tiles having overlapping regions with each other. Let X be { X ═ X^mM 1.. p } is a set of low resolution blocks that are trained, Y ═ Y ·^mAnd m is 1, p is a high-resolution block set corresponding to the training image, wherein p is the number of small blocks segmented from the training image. Similarly, noteA set of blocks segmented for the low resolution image to be reconstructed, thenIs the set of blocks of the high resolution image to be estimated, where q is the number of segmented out tiles in the test image. Image block x^mAnd image blockIs defined as D_pqWherein the n-th column of D isThe distances to all image blocks in X constitute a column vector. Then, each column in D is traversed to find K maximum values, and the image block corresponding to the index is the image block to be foundK neighboring blocks in the training set, denoted as $S_l^n=\{{\hat{x}}^{n\left(t1\right)},{\hat{x}}^{n\left(t2\right)},\mathrm{...},{\hat{x}}^{n\left(tK\right)}\}.$

Then for each test image blockMethod for obtaining optimal weight vector omega by minimizing local reconstruction error_n＝[ω_n1,ω_n2,...ω_np]：

${\mathrm{\&omega;}}_n=\arg \underset{{\mathrm{\&omega;}}_n}{\min }\left|\right|{\hat{x}}^n-\underset{m=1}{\overset{p}{\&Sigma;}}{\mathrm{\&omega;}}_{nm}{x}^m|{|}_2^2---\left(7\right)$

Wherein each weight value omega_nmThe following constraints are satisfied:

$\left\{\begin{array}{ccc}\underset{m=1}{\overset{p}{\&Sigma;}}{\mathrm{\&omega;}}_{nm}=1& & \\ {\mathrm{\&omega;}}_{nm}\&NotEqual;0& if& x^m\&Element;S_l^n\\ {\mathrm{\&omega;}}_{nm}=0& if& x^m\&NotElement;S_l^m\end{array}\right.---\left(8\right)$

method for solving omega by using nuclear regression mode_nmEach high resolution block (pixel point values of the image in the block after blocking) can be derivedEstimated value of (a):

${\hat{y}}^n=\underset{m=1}{\overset{p}{\&Sigma;}}{\mathrm{\&omega;}}_{nm}{y}^m---\left(9\right)$

for the overlapping areas of the respective high resolution image blocks, the respective overlapping pixel values thereof are averaged to avoid severe blocking artifacts.

To introduce non-local self-similarity information, the residual error e is defined by_lFor image blocksWith another image blockThe smaller the residual error, the more similar the two image blocks:

$e_l=\left|\right|{\hat{y}}_l-{\tilde{y}}_l|{|}_2^2---\left(10\right)$

for each image blockComputing image blocksAnd searching residual errors of all image blocks in the neighborhood, and finding L image blocks with the minimum residual errors as the image blocksL most phases ofSet of similar blocksWith residual errors of respectivelyThen the image block to be estimatedCan be expressed asLinear combination of (a):

${\hat{y}}^n=\underset{i=1}{\overset{L}{\&Sigma;}}{\mathrm{\&rho;}}_n^i{\tilde{y}}^{ni}---\left(11\right)$

wherein,each element in (1)Andsimilarity weight betweenCan be calculated from the following formula:

${\mathrm{\&rho;}}_n^i=\frac{\exp (-\frac{e_l^{ni}}{h})}{\underset{i=1}{\overset{L}{\&Sigma;}}\exp (-\frac{e_l^{ni}}{h})}---\left(12\right)$

in (12), h is a control factor of the weight. ThenAndeach element in (1)The similarity weight between them can be defined as:

$T_n^j=\left\{\begin{array}{cc}{\mathrm{\&rho;}}_n^i& \begin{array}{cc}if& {\hat{y}}^j\&Element;\tilde{Y},{\hat{y}}^j={\tilde{y}}^{ni}\end{array}\\ 0& otherwise\end{array}\right.---\left(13\right)$

order toIs composed ofThe vector of the composition is then calculated,then (11) can be described as:

adding (14) as a non-local self-similarity regular term into a neighborhood embedding method, and constructing an improved neighborhood super-resolution reconstruction method model:

$\hat{Y}=\underset{\hat{Y}}{\arg \min }\left|\right|\hat{Y}-WY|{|}_2^2+\mathrm{\&lambda;}||\hat{Y}-\mathrm{\&Phi;}\hat{Y}|{|}_2^2---\left(15\right)$

wherein W is ω of the lexicographic order_nI.e. W ═ ω₁,ω₂,...,ω_q]^TPhi is in lexical orderNamely, it isThe formula (15) can be solved by a gradient descent method and simplified into

${\hat{Y}}^{t+1}={\hat{Y}}^t-\mathrm{\&lambda;}{(I-\mathrm{\&Phi;})}^T(I-\mathrm{\&Phi;}){\hat{Y}}^t---\left(16\right)$

t is the number of iterations and λ is the regularization coefficient constant. Is provided withThe iteration initial value obtained by the neighborhood embedding method can obtain accurate high-frequency initial estimation after t iterations

2.2 building sparse representation dictionary

In order to solve (6), an effective sparse representation dictionary needs to be established on the basis of an accurate initial estimation. Herein, a PCA (principal component analysis) dictionary learning method is employed. PCA dictionary learning method [30]The method is an effective dictionary construction method, and firstly carries out blocking operation on a trained high-resolution sample to obtain a high-resolution original subset A_sThen using K-means algorithm to convert A_sInto n clusters { K₁,K₂,...,K_nThe center of each cluster is marked as { C }₁,C₂,...,C_nThen for each cluster K_mCalculate its covariance matrix omega_mBy the pair omega_mCarrying out PCA transformation to obtain a final orthogonal transformation matrix which is used as a sub-dictionary psi_mFurther, the sub-dictionary set { Ψ is obtained₁,Ψ₂,...,Ψ_n}。

The PCA dictionary learning method generally adopts an external dictionary, and a large amount of memory is consumed by a huge external image training library. To address this problem, rather than using an external dictionary, the multi-scale similarity of images, atomic set A, is exploited herein_sThe multi-scale self-similarity sub-dictionary is trained by block generation of multi-scale down sampling of an input low-resolution image.

After the high-resolution initial estimation image obtained in the step 2.1 is subjected to blocking operation, each high-resolution input block x to be reconstructed_iIt is then compared with the trained cluster center{C₁,C₂,...,C_nAre compared to find the cluster center C with the smallest Euclidean distance from the input block_iCluster K of which it is located_iThe corresponding sub-dictionary Ψ_iI.e. the high resolution block x to be reconstructed_iThe dictionary used. Obtaining a sub-dictionary psi corresponding to each image block to be reconstructed_iAnd (6) can be converted into two formulas (17) and (18) by a Lagrange multiplier method, and the two formulas (17) and (18) are solved iteratively.

$\mathrm{\&alpha;}=\arg \underset{\mathrm{\&alpha;}}{\min }\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&lambda;}}_i\left|\right|{\mathrm{\&alpha;}}_i|{|}_1+\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&gamma;}}_i||{\mathrm{\&alpha;}}_i-{\mathrm{\&alpha;}}_i^{*}|{|}_2^2+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}\left|\right|R_{i}x-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2---\left(17\right)$

$x=\arg \underset{x}{\min }\left|\right|y-DWx|{|}_2^2+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}||R_{i}x-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2---\left(18\right)$

(17) The final solution may be obtained by an iterative contraction algorithm. To solve equation (18), the lagrange equation is constructed as:

$L(X,Z,\mathrm{\&tau;})=\left|\right|Y-DWX|{|}_2^2+\mathrm{\&theta;}||R_{i}X-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2+<Z,Y-DX>+\mathrm{\&tau;}\left|\right|Y-DX|{|}_2^2---\left(19\right)$

where Z is the lagrange multiplier, τ is a constant, (19) can be solved by iterating inter _ num times as follows:

$x^{(iter+1)}=\arg \underset{x}{\min }L(x,Z^{\left(iter\right)},{\mathrm{\&tau;}}^{\left(iter\right)})---\left(20\right)$

Z^(iter+1)＝Z^(iter)+τ^iter(y-Dx^(iter+1))(21)

τ^(iter+1)＝τ^(iter)(22)

order to(20) The solution can be simplified to equation (23):

$x^{(iter+1)}={\&lsqb;{\left(DW\right)}^{T}DW+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}{R_i}^T{R}_i+{\mathrm{\&tau;}}^{\left(iter\right)}{D}^{T}D\&rsqb;}^{-1}\&lsqb;{\left(DW\right)}^{T}y+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}{R_i}^T{R}_i\left({\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i\right)+D^T{Z}^{\left(iter\right)}/2+{\mathrm{\&tau;}}^{\left(iter\right)}{D}^{T}y\&rsqb;---\left(23\right).$

in the method, the size of a low-resolution image block is set to be 3 × 3, the size of a corresponding high-resolution image block is 9 × 9, the number K of neighborhood similar blocks is 5, the number L of non-local similar blocks is 10, the size of the non-local similar blocks is 3 × 3, a weight control factor h is 65, and overlapping is carried out on the low-resolution image block, the corresponding high-resolution image block and the corresponding high-resolution image block, wherein the original image is downsampled, the original image is subjected to 3-time scale super-resolution reconstruction amplification, and the super-resolution reconstruction is compared with Bicubic super-resolution reconstruction, NE and NARM (network object model)Generation number t of 30, regularization factors lambda, c₁、c₂θ, and inter _ num are 0.05, 0.15, 2.5, 0.15, 0.3, 1.2, and 180, respectively. In order to evaluate the reconstruction effect, in addition to subjective visual evaluation, peak signal-to-noise ratio (PSNR) and image Structure Similarity (SSIM) are also used for evaluating the super-resolution reconstruction performance of different algorithms. PSNR is an objective image quality evaluation method based on statistical characteristics, and a larger peak signal-to-noise ratio indicates that the similarity between a reconstructed image and an original image is higher. SSIM is a method for evaluating image quality by sensing image structure information, and a larger SSIM value indicates a more similar structure between images.

Because the method provided by the invention introduces non-local self-similarity constraint, in order to verify the influence of the number of non-local self-similarity blocks on the reconstruction effect, 256 multiplied by 256Cameraman images are adopted for carrying out experiments. In the experiment, non-local self-similar blocks L are respectively selected as 5,10,15,20 and 25 to be reconstructed. As can be seen from table 1 and fig. 1, as the number of the non-local self-similar blocks increases, the discontinuity at the edge is relieved to some extent, but the PSNR value decreases gradually, and meanwhile, an over-smooth blur in a detail area is caused, so that the detail factor and the edge effect of the image are considered comprehensively, and a good reconstruction effect can be achieved by selecting L as 10.

Fig. 2 and fig. 3 show experimental comparison diagrams of Leaves and Cameraman, respectively, from which it can be seen that the Bicubic method causes discontinuity phenomenon and ringing effect of the boundary while amplifying the image; the NE method can basically keep the edge structure of the image, but simultaneously blurs most details of the image; the NARM method can better recover image details, but the reconstruction effect is still not ideal in some places, such as edge details of leaves, lens frames and the like; the method improves the defects of the NARM algorithm better, achieves better effects in reconstructing texture details and eliminating false edges and saw teeth, and ensures that the recovery result is more vivid. Therefore, the super-resolution reconstruction result of the algorithm is most similar to the original high-resolution image in visual effect.

Table 2 shows PSNR and SSIM of different algorithms, and it can be seen that introduction of accurate high-frequency initial estimation to improve the super-resolution algorithm based on local and non-local similarities can effectively improve reconstruction quality, and compared with some representative reconstruction methods, the PSNR average value is improved by 0.47-1.84 dB, and the SSIM average value is improved by 0.0114-0.0438. Therefore, the algorithm provided by the invention is obviously superior to other algorithms in evaluation index, and the objective evaluation and the subjective evaluation result are completely consistent.

TABLE 1 influence of the number of non-local self-similar blocks on the reconstruction effect

TABLE 2 PSNR, SSIM for different super-resolution reconstruction algorithms

Claims (1)

1. A super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity is characterized by comprising the following steps:

step (1), based on initial estimation algorithm of neighborhood embedding, firstly extracting high-resolution training image I_HThen a down-sampling operation is performed on the high resolution image to obtain a low resolution image I_LDividing the high-resolution and low-resolution images into small blocks with mutually overlapped regions; let X be { X ═ X^mM 1.. p } is a set of low resolution blocks that are trained, Y ═ Y ·^mM 1.. p } is a set of high resolution blocks corresponding thereto, where p is the number of small blocks segmented from the training image; similarly, noteA set of blocks segmented for the low resolution image to be reconstructed,a set of blocks of a high resolution image to be estimated, wherein q is the number of segmented small blocks in the test image; image block x^mAnd image blockIs defined as D_pqWherein D is_pqIs listed as nAnd X represents a column vector formed by distances of all image blocks in the training low-resolution block set; then traverse D_pqEach column in the image block search unit finds K maximum values, and the index of the maximum value is corresponding to the image block, i.e. the image block to be searchedK neighboring blocks in the training set, denoted as Mapping of representations to a desired image blockT1, t 2.. tK denotes the sequence numbers of the K neighboring blocks;

then for each test image blockMethod for obtaining optimal weight vector omega by minimizing local reconstruction error_n＝[ω_n1,ω_n2,...ω_np]N1, n 2.. nP indicates the subscript of the sequence number, n indicates the nth test image block:

${\mathrm{\&omega;}}_n=\arg \underset{{\mathrm{\&omega;}}_n}{min}\left|\right|{\hat{x}}^n-\underset{m=1}{\overset{p}{\&Sigma;}}{\mathrm{\&omega;}}_{nm}{x}^m|{|}_2^2---\left(7\right)$

wherein each weight value omega_nmThe following constraints are satisfied:

$\left\{\begin{array}{ccc}\underset{m=1}{\overset{p}{\&Sigma;}}{\mathrm{\&omega;}}_{nm}=1& & \\ {\mathrm{\&omega;}}_{nm}\&NotEqual;0& if& x^m\&Element;S_l^n\\ {\mathrm{\&omega;}}_{nm}=0& if& x^m\&NotElement;S_l^n\end{array}\right.---\left(8\right)$

method for solving omega by using nuclear regression mode_nmObtaining each high resolution image blockEstimated value of (a):

${\hat{y}}^n=\underset{m=1}{\overset{p}{\&Sigma;}}{\mathrm{\&omega;}}_{nm}{y}^m---\left(9\right)$

taking the average of all overlapped pixel values of the overlapped areas of all high-resolution image blocks;

the residual error e is defined by_lFor image blocksWith another image blockThe smaller the residual error, the more similar the two image blocks:

$e_l=\left|\right|{\hat{y}}_l-{\tilde{y}}_l|{|}_2^2---\left(10\right)$

for each image blockComputing image blocksAnd searching residual errors of all image blocks in the neighborhood, and finding L image blocks with the minimum residual errors as the image blocksL most similar block sets:

$\tilde{Y}=\{{\tilde{y}}^{ni},i=1,2...L\},$

with residual errors of respectivelyLL denotes the number of image blocks; then the image block to be estimatedIs shown asRepresents a linear combination of one of the L image blocks found to have the smallest residual error:

${\hat{y}}^n=\underset{i=1}{\overset{L}{\&Sigma;}}{\mathrm{\&rho;}}_n^i{\tilde{y}}^{ni}---\left(11\right)$

wherein,each element in (1)Andsimilarity weight betweenCalculated from the following formula:

${\mathrm{\&rho;}}_n^i=\frac{\exp (-\frac{e_l^{ni}}{h})}{\underset{i=1}{\overset{L}{\&Sigma;}}\exp (-\frac{e_l^{ni}}{h})}---\left(12\right)$

h is a control factor of the weight;andeach element in (1)The similarity weight between (j ═ 1,2., q) is defined as:

$T_n^j=\left\{\begin{array}{cc}{\mathrm{\&rho;}}_n^i& if{\hat{y}}^j\&Element;\tilde{Y},{\hat{y}}^j={\tilde{y}}^{ni}\\ 0& otherwise\end{array}\right.---\left(13\right)$

representing the jth block in the set of high-resolution image blocks to be estimated,for image blocksL most similar block sets;

order toIs composed ofThe vector of the composition is then calculated,expression (11) is:

adding (14) serving as a non-local self-similarity regular term into a neighborhood embedding method, and constructing a neighborhood super-resolution reconstruction method model:

$\hat{Y}=\underset{\hat{Y}}{\arg \min }\left|\right|\hat{Y}-WY|{|}_2^2+\mathrm{\&lambda;}||\hat{Y}-\mathrm{\&Phi;}\hat{Y}|{|}_2^2---\left(15\right)$

wherein W is ω of the lexicographic order_nI.e. W ═ ω₁,ω₂,...,ω_q]^TPhi is in lexical orderNamely, it isThe formula (15) is solved by a gradient descent method and simplified into

${\hat{Y}}^{t+1}={\hat{Y}}^t-\mathrm{\&lambda;}{(I-\mathrm{\&Phi;})}^T(I-\mathrm{\&Phi;}){\hat{Y}}^t---\left(16\right)$

t is iteration times, and lambda is a regularization coefficient constant; is provided withFor the iterative initial value obtained by the neighborhood embedding method, the method_tThe iteration is repeated to obtain accurate high-frequency initial estimation

Step (2) establishing a sparse representation dictionary, and after the high-resolution initial estimation image obtained in the step (1) is subjected to blocking operation, for each high-resolution input block x to be reconstructed_iThen, it is compared with the trained cluster center { C }₁,C₂,...,C_nAre compared to find the cluster center C with the smallest Euclidean distance from the input block_iCluster K of which it is located_iThe corresponding sub-dictionary Ψ_iI.e. the high resolution block x to be reconstructed_iDictionary used；

Obtaining a sub-dictionary psi corresponding to each image block to be reconstructed_iThen, the model is reconstructed by using the non-local autoregressive super-resolution $\{\mathrm{\&alpha;},x\}=\arg \underset{\mathrm{\&alpha;},x}{min}\left|\right|y-DMx|{|}_2^2+\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&lambda;}}_i|\left|{\mathrm{\&alpha;}}_i\right|{|}_1+\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&gamma;}}_i\left|\right|{\mathrm{\&alpha;}}_i-{\mathrm{\&alpha;}}_i^{*}|{|}_2^2+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}||R_{i}x-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2---\left(6\right)$ Converting into two formulas (17) and (18) by Lagrange multiplier method,

$\mathrm{\&alpha;}=\arg \underset{\mathrm{\&alpha;}}{min}\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&lambda;}}_i\left|\right|{\mathrm{\&alpha;}}_i|{|}_1+\underset{i=1}{\overset{N}{\&Sigma;}}{\mathrm{\&gamma;}}_i||{\mathrm{\&alpha;}}_i-{\mathrm{\&alpha;}}_i^{*}|{|}_2^2+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}\left|\right|R_{i}x-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2---\left(17\right)$

$x=\arg \underset{x}{min}\left|\right|y-DWx|{|}_2^2+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\&Sigma;}}||R_{i}x-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2---\left(18\right)$

wherein y represents an input low-resolution image, x represents a high-resolution image to be reconstructed, and D is a down-sampling matrix; m is a non-local self-similarity weight matrix and is used for describing the non-local self-similarity relation between the image blocks; Ψ_iA sub-dictionary corresponding to the ith image block, α is a sparse representation sparse matrix with each row α_iIn the sub-dictionary Ψ for the ith image block_iSparse representation of coefficient vectors under, α_iIs α_ij；α_ijSparse representation coefficients of a non-local self-similarity block for an ith image block; r_iFor extracting matrix, the function is to extract the ith image block from imageTo the process; lambda [ alpha ]_i、γ_iIs a weighted vector, each element of which $\begin{array}{cc}{\mathrm{\&lambda;}}_{ij}=\frac{c_1}{\left|{\mathrm{\&alpha;}}_{ij}\right|+\mathrm{\&epsiv;}},& {\mathrm{\&gamma;}}_{ij}=\frac{c_2}{{(a_{ij}-{{\mathrm{\&alpha;}}_{ij}}^{*})}^2+\mathrm{\&epsiv;}}\end{array};$ Theta is a weighting coefficient;

and solving equations (17) and (18) iteratively; obtaining a final solution by an iterative contraction algorithm in the formula (17); to solve equation (18), the lagrange equation is constructed as:

$L(X,Z,\mathrm{\&tau;})=\left|\right|Y-DWX|{|}_2^2+\mathrm{\&theta;}||R_{i}X-{\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i|{|}_2^2+<Z,Y-DX>+\mathrm{\&tau;}\left|\right|Y-DX|{|}_2^2---\left(19\right)$ where Z is a lagrange multiplier, τ is a constant, equation (19) can be solved by iterating inter _ num times as follows, iter represents the current number of iterations:

$x^{(iter+1)}=\arg \underset{x}{\min }L(x,Z^{\left(iter\right)},{\mathrm{\&tau;}}^{\left(iter\right)})---\left(20\right)$

Z^(iter+1)＝Z^(iter)+τ^iter(y-Dx^(iter+1))(21)

τ^(iter+1)＝τ^(iter)(22)

order toThe solution is performed by reducing equation (20) to equation (23):

$x^{(iter+1)}={\&lsqb;{\left(DW\right)}^{T}DW+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{R_i}^T{R}_i+{\mathrm{\&tau;}}^{\left(iter\right)}{D}^{T}D\&rsqb;}^{-1}\&lsqb;{\left(DW\right)}^{T}y+\mathrm{\&theta;}\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{R_i}^T{R}_i\left({\mathrm{\&Psi;}}_i{\mathrm{\&alpha;}}_i\right)+D^T{Z}^{\left(iter\right)}/2+{\mathrm{\&tau;}}^{\left(iter\right)}{D}^{T}y\&rsqb;---\left(23\right);$

and x is a high-resolution image to be reconstructed, and is an update multiple of the update constant tau, and is a constant.