CN105550988A

CN105550988A - Super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity

Info

Publication number: CN105550988A
Application number: CN201510900809.4A
Authority: CN
Inventors: 周圆; 冯丽洋; 陈莹; 陈阳; 侯春萍
Original assignee: Tianjin University
Current assignee: Tianjin University
Priority date: 2015-12-07
Filing date: 2015-12-07
Publication date: 2016-05-04

Abstract

The invention discloses a super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity. Firstly, the structural similarity is used to improve the neighborhood embedding method, and then a more accurate high-frequency initial estimate is obtained, and the initial estimation based on neighborhood embedding is realized. Estimation algorithm; then use the local self-similarity and multi-scale structural similarity of the low-resolution image to construct the reconstruction constraint item to reconstruct the high-resolution, and establish a sparse representation dictionary. Compared with the prior art, the algorithm proposed in the present invention improves the neighborhood embedding method on the basis of solving the defect that the previous learning-based super-resolution reconstruction algorithm needs a large number of training sets, and uses it to solve the problem based on The inaccurate high-frequency initial estimation problem in the super-resolution algorithm of local self-similarity and multi-scale similarity improves the super-resolution reconstruction effect of the image; it can better suppress the aliasing effect and ringing effect, and reconstruct the The high-resolution images are closer to real images with better subjective and objective quality.

Description

Super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity

技术领域technical field

本发明涉及图像处理领域，更具体地，涉及一种基于改进邻域嵌入和结构自相似性的超分辨率重建算法。The present invention relates to the field of image processing, more specifically, to a super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity.

背景技术Background technique

自1984年Tsai等提出超分辨率这一概念以来，超分辨率重建技术得到的广泛关注，提出了众多超分辨率重建算法。这些算法大致可以分为基于插值的方法、基于重建的方法和基于学习的方法3类。基于插值的方法运算复杂度低、运行速度快，但是插值后的图像通常缺少高频细节，易造成边缘模糊；基于重建的方法利用图像的降质模型以及特定的先验知识进行超分辨率重建，一般方法中常用的先验模型包括高斯先验假设(GaussianProcessPriors)、胡伯马尔可夫随机场(HuberMRF)、全变差(TotalVariation)模型、平滑边界(SoftEdge)模型和梯度轮廓(GradientProfile)模型等，但是由于先验知识的局限性，结构信息和纹理信息无法得到有效的恢复，重建效果不明显；基于学习的方法包括Example-Based方法、邻域嵌入方法(NeighborEmbedding)和稀疏表示法(SparseRepresentation)，他们的基本思想是通过对大量低分辨率块和与之对应的高分辨率图像块的学习，在它们之间建立一种对应关系，然后通过这种关系指导高分辨率图像块重建。在这3种超分辨率重建的方法中，基于学习的方法较其它两类方法能够引入更多的高频信息，对噪声的鲁棒性更强，因此成为近年的研究热点。Since Tsai et al. proposed the concept of super-resolution in 1984, super-resolution reconstruction technology has received widespread attention, and many super-resolution reconstruction algorithms have been proposed. These algorithms can be roughly divided into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods. The interpolation-based method has low computational complexity and fast operation speed, but the interpolated image usually lacks high-frequency details, which is easy to cause blurred edges; the reconstruction-based method uses the degraded model of the image and specific prior knowledge for super-resolution reconstruction , the prior models commonly used in the general method include Gaussian prior assumption (GaussianProcessPriors), Huber Markov random field (HuberMRF), total variation (TotalVariation) model, smooth boundary (SoftEdge) model and gradient profile (GradientProfile) model etc., but due to the limitation of prior knowledge, structure information and texture information cannot be effectively restored, and the reconstruction effect is not obvious; learning-based methods include Example-Based method, NeighborEmbedding method (NeighborEmbedding) and sparse representation (SparseRepresentation ), their basic idea is to establish a correspondence between them by learning a large number of low-resolution patches and their corresponding high-resolution image patches, and then guide the reconstruction of high-resolution image patches through this relationship. Among the three super-resolution reconstruction methods, the learning-based method can introduce more high-frequency information than the other two methods, and is more robust to noise, so it has become a research hotspot in recent years.

在基于学习的超分辨率重建方法中，如采用训练图像，则需要庞大的外部图像训练库，因此会导致算法的内存消耗较大。针对这一问题，出现了众多利用非局部自相似结构的超分辨率重建方法。这种结构相似性通常以显在或潜在的形式普遍存在于自然图像中，这便为图像超分辨率重建提供了丰富的附加信息。SuetakeN等提出利用图像自身不同尺度相似图像块所提供的附加信息建立内部字典，然后使用基于学习的方法进行超分辨率重建；GlasnerD等将相同尺度和不同尺度相似图像块所提供的附加信息同时加入重建模型，在整幅图像中进行相似结构图像块的搜索；ZhangK等将图像多尺度结构自相似模型与压缩感知理论相结合，使图像重构效果进一步提升。但是上述采用结构自相似性的超分辨率重建算法在构建稀疏表示模型时，忽略了拥有非局部相似性质的图像块应具有相同或相近的稀疏表示系数。针对此问题，DongW等提出综合考虑图像的多尺度相似性与非局部相似性，并将其融入压缩感知模型建立新的图像超分辨率方法，得到良好的重建效果。In the learning-based super-resolution reconstruction method, if training images are used, a huge external image training library is required, which will lead to a large memory consumption of the algorithm. In response to this problem, many super-resolution reconstruction methods using non-local self-similar structures have emerged. This structural similarity usually exists in natural images in obvious or latent forms, which provides rich additional information for image super-resolution reconstruction. SuetakeN et al. proposed to use the additional information provided by similar image blocks of different scales in the image itself to establish an internal dictionary, and then use a learning-based method for super-resolution reconstruction; GlasnerD et al. The reconstruction model searches for similar structural image blocks in the entire image; ZhangK et al. combined the multi-scale structural self-similar model of the image with the compressive sensing theory to further improve the image reconstruction effect. However, when the above-mentioned super-resolution reconstruction algorithm using structural self-similarity builds a sparse representation model, it ignores that image blocks with non-local similarity properties should have the same or similar sparse representation coefficients. In response to this problem, DongW et al. proposed to comprehensively consider the multi-scale similarity and non-local similarity of the image, and integrate it into the compressed sensing model to establish a new image super-resolution method, and obtain a good reconstruction effect.

发明内容Contents of the invention

为了克服上述现有技术存在的直流系统故障隔离难的问题，本发明提出了考虑到初始估计的准确与否直接影响图像重建的质量与迭代次数，提出一种将邻域嵌入和结构自相似性有效结合的图像超分辨率重建算法。In order to overcome the problem of difficult isolation of DC system faults in the prior art, the present invention proposes a method that combines neighborhood embedding and structural self-similarity Effectively combined image super-resolution reconstruction algorithms.

本发明提出一种基于改进邻域嵌入和结构自相似性的超分辨率重建算法，该算法包括以下步骤：The present invention proposes a super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity, which comprises the following steps:

步骤(1)、基于邻域嵌入的初始估计算法，首先提取高分辨率训练图像I_H的亮度分量，接着对高分辨率图像进行a倍的下采样操作得到低分辨率图像I_L，并将高分辨率和低分辨率图像分成相互有重叠区域的小块；记X＝{x^m,m＝1,...,p}为训练的低分辨率块集合，Y＝{y^m,m＝1,...,p}为与之对应的高分辨率块集合，其中p是从训练图像中分割出的小块的数；同样的，记为待重建的低分辨率图像分割出的块集合，为待估计的高分辨率图像的块集合，其中q是测试图像中分割出小块的数量；将图像块x^m和图像块的距离矩阵定义为D_pq，其中D_pq的第n列为与X，X表示训练的低分辨率块集合中所有图像块的距离构成的列向量；然后遍历D_pq中的每一列找到K个最大值，其索引对应的图像块即所求图像块在训练集中的K个近邻块，记为表示的对应所求的图像块的K个近邻块，t1,t2,...tK表示K个近邻块的序号；Step (1), the initial estimation algorithm based on neighborhood embedding, firstly extract the luminance component of the high-resolution training image I _H , then perform a downsampling operation on the high-resolution image to obtain the low-resolution image I _L , and set The high-resolution and low-resolution images are divided into small blocks with overlapping areas; record X={x ^m ,m=1,...,p} as the set of low-resolution blocks for training, Y={y ^m ,m =1,...,p} is the corresponding set of high-resolution blocks, where p is the number of small blocks segmented from the training image; similarly, record A set of blocks segmented for the low-resolution image to be reconstructed, is the block set of the high-resolution image to be estimated, where q is the number of small blocks in the test image; the image block x ^m and the image block The distance matrix of is defined as D _pq , where the nth column of D _pq is and X, X represents the column vector formed by the distances of all image blocks in the training low-resolution block set; then traverse each column in D _pq to find K maximum values, and the image block corresponding to its index is the requested image block The K nearest neighbor blocks in the training set are denoted as Represented corresponding to the requested image block The K neighboring blocks of , t1, t2,...tK represent the sequence numbers of the K neighboring blocks;

接着对于每一个测试图像块利用最小化局部重建误差的方式求得最佳权值向量ω_n＝[ω_n1,ω_n2,...ω_np]，n1,n2,...nP表示序号的脚标，n表示第n个测试图像块：Then for each test image block The optimal weight vector ω _n =[ω _n1 ,ω _n2 ,...ω _np ] is obtained by minimizing the local reconstruction error, n1,n2,...nP represent the subscripts of the serial numbers, and n represents the nth test image blocks:

${ω ω}_{n no} = = arg arg \underset{{ω ω}_{n no}}{min min} | | | | {\overset{^^}{x x}}^{n no} - - {Σ Σ}_{m m = = 11}^{p p} {ω ω}_{n no m m} {x x}^{m m} | | {| |}_{22}^{22} - - - - - - ((77))$

其中每一个权值ω_nm需满足下述约束条件：Each of the weights ω _nm needs to meet the following constraints:

$\{\begin{matrix} {Σ Σ}_{m m = = 11}^{p p} {ω ω}_{n no m m} = = 11 \\ {ω ω}_{n no m m} &NotEqual; &NotEqual; 00 & i i f f & {x x}^{m m} &Element; &Element; {S S}_{l l}^{n no} \\ {ω ω}_{n no m m} = = 00 & i i f f & {x x}^{m m} &NotElement; &NotElement; {S S}_{l l}^{m m} \end{matrix} - - - - - - ((88))$

利用核回归方式求解ω_nm，得到每个高分辨率图像块的估计值：Use kernel regression to solve ω _nm to get each high-resolution image block Estimated value of :

${\overset{^^}{y the y}}^{n no} = = {Σ Σ}_{m m = = 11}^{p p} {ω ω}_{n no m m} {y the y}^{m m} - - - - - - ((99))$

对于各个高分辨率图像块的重叠区域，取其各个重叠像素值的平均；For the overlapping area of each high-resolution image block, the average value of each overlapping pixel value is taken;

利用下式定义残余误差e_l为图像块与另一图像块的相似程度，残余误差越小说明两图像块越相似：Use the following formula to define the residual error e _l as the image block with another image block The smaller the residual error, the more similar the two image blocks are:

${e e}_{l l} = = | | | | {\overset{^^}{y the y}}_{l l} - - {\overset{~ ~}{y the y}}_{l l} | | {| |}_{22}^{22} - - - - - - ((1010))$

对于每一个图像块计算图像块与其搜索邻域内的所有图像块的残余误差，找到残余误差最小的L个图像块即为图像块的L个最相似块集合：for each image block Computing Image Blocks Instead of searching the residual errors of all image blocks in the neighborhood, find the L image blocks with the smallest residual error as the image block The L most similar block sets of :

$\overset{~ ~}{Y Y} = = {{{\overset{~ ~}{y the y}}^{n no i i},, i i = = 11,, 2... 2... L L}},,$

其残余误差分别为L表示的是图像块的个数；则待估计图像块表示为表示找到的L个残余误差最小的图像块中的一个图像块的线性组合：Its residual error is L represents the number of image blocks; the image block to be estimated Expressed as Represents a linear combination of an image block among the L image blocks with the smallest residual error found:

${\overset{^^}{y the y}}^{n no} = = {Σ Σ}_{i i = = 11}^{L L} {ρ ρ}_{n no}^{i i} {\overset{~ ~}{y the y}}^{n no i i} - - - - - - ((1111))$

其中，中的每个元素与之间的相似度权值由下式计算：in, each element in and The similarity weight between Calculated by the following formula:

${ρ ρ}_{n no}^{i i} = = \frac{exp exp ((- - \frac{{e e}_{l l}^{n no i i}}{h h}))}{{Σ Σ}_{i i = = 11}^{L L} exp exp ((- - \frac{{e e}_{l l}^{n no i i}}{h h}))} - - - - - - ((1212))$

h是权值的控制因子；与中的每个元素之间的相似度权值定义为：h is the control factor of the weight; and each element in The similarity weight between is defined as:

${T T}_{n no}^{j j} = = \{\begin{matrix} {ρ ρ}_{n no}^{i i} & \begin{matrix} i i f f & {\overset{^^}{y the y}}^{j j} &Element; &Element; \overset{~ ~}{Y Y},, {\overset{^^}{y the y}}^{j j} = = {\overset{~ ~}{y the y}}^{n no i i} \end{matrix} \\ 00 & o o t t h h e e r r w w i i s the s e e \end{matrix} - - - - - - ((1313))$

表示待估计的高分辨率图像块集合中的第j个块，为图像块的L个最相似块集合； Indicates the jth block in the set of high-resolution image blocks to be estimated, for image block The L most similar block sets;

令为组成的向量，将式(11)表示为：make for composed of vectors, Formula (11) is expressed as:

将(14)作为非局部自相似正则项加入邻域嵌入方法中，构造邻域超分辨率重建方法模型：Add (14) as a non-local self-similar regular term to the neighborhood embedding method to construct a neighborhood super-resolution reconstruction method model:

$\overset{^^}{Y Y} = = \underset{\overset{^^}{Y Y}}{arg arg min min} | | | | \overset{^^}{Y Y} - - W W Y Y | | {| |}_{22}^{22} + + λ λ | | | | \overset{^^}{Y Y} - - Φ Φ \overset{^^}{Y Y} | | {| |}_{22}^{22} - - - - - - ((1515))$

其中W为字典序的ω_n，即W＝[ω₁,ω₂,...,ω_q]^T，Φ为字典序的即 Where W is the ω _n of lexicographical order, that is, W=[ω ₁ ,ω ₂ ,...,ω _q ] ^T , and Φ is the lexicographical order which is

公式(15)通过梯度下降法求解，化简为Formula (15) is solved by the gradient descent method, which is simplified as

${\overset{^^}{Y Y}}^{t t + + 11} = = {\overset{^^}{Y Y}}^{t t} - - λ λ {((I I - - Φ Φ))}^{T T} ((I I - - Φ Φ)) {\overset{^^}{Y Y}}^{t t} - - - - - - ((1616))$

t为迭代次数，λ为正则化系数常量；设为邻域嵌入法得到的迭代初始值，经过_t次迭代，得到准确的高频初始估计 t is the number of iterations, λ is the regularization coefficient constant; let is the iterative initial value obtained by the neighborhood embedding method, after _t iterations, an accurate high-frequency initial estimate is obtained

步骤(2)、建立稀疏表示字典，将步骤(1)得到的高分辨率初始估计图像进行分块操作后，对于每一个待重建高分辨率输入块x_i，将其与已经训练得到的簇中心{C₁,C₂,...,C_n}进行比较，找到与输入块欧氏距离最小的簇中心C_i，其所在的簇K_i所对应的子字典Ψ_i即待重建高分辨率块x_i所使用的字典；Step (2), establish a sparse representation dictionary, divide the high-resolution initial estimated image obtained in step (1) into blocks, and for each high-resolution input block x _i to be reconstructed, combine it with the trained cluster Centers {C ₁ , C ₂ ,...,C _n } are compared to find the cluster center C _i with the smallest Euclidean distance to the input block, and the sub-dictionary Ψ _i corresponding to the cluster K _i where it is located is to be reconstructed with high resolution The dictionary used by rate block x _i ;

得到每个待重建图像块所对应的子字典Ψ_i后，将利用非局部自回归超分辨率重建模型 ${α, x} = \arg \min_{α, x} | | y - D M x | |_{2}^{2} + Σ_{i = 1}^{N} λ_{i} | | α_{i} | |_{1} + Σ_{i = 1}^{N} γ_{i} | | α_{i} - α_{i}^{*} | |_{2}^{2} + θ Σ_{i = 1}^{N} | | R_{i} x - Ψ_{i} α_{i} | |_{2}^{2} - - - (6)$ 通过拉格朗日乘子法转化为(17)、(18)两式，After obtaining the sub-dictionary Ψ _i corresponding to each image block to be reconstructed, the non-local autoregressive super-resolution reconstruction model will be used ${α, x} = \arg \min_{α, x} | | the y - D. m x | |_{2}^{2} + Σ_{i = 1}^{N} λ_{i} | | α_{i} | |_{1} + Σ_{i = 1}^{N} γ_{i} | | α_{i} - α_{i}^{*} | |_{2}^{2} + θ Σ_{i = 1}^{N} | | R_{i} x - Ψ_{i} α_{i} | |_{2}^{2} - - - (6)$ Transformed into (17) and (18) by Lagrange multiplier method,

$α α = = arg arg \underset{α α}{min min} {Σ Σ}_{i i = = 11}^{N N} {λ λ}_{i i} | | | | {α α}_{i i} | | {| |}_{11} + + {Σ Σ}_{i i = = 11}^{N N} {γ γ}_{i i} | | | | {α α}_{i i} - - {α α}_{i i}^{* *} | | {| |}_{22}^{22} + + θ θ {Σ Σ}_{i i = = 11}^{N N} | | | | {R R}_{i i} x x - - {Ψ Ψ}_{i i} {α α}_{i i} | | {| |}_{22}^{22} - - - - - - ((1717))$

$x x = = arg arg \underset{x x}{min min} | | | | y the y - - D D. W W x x | | {| |}_{22}^{22} + + θ θ {Σ Σ}_{i i = = 11}^{N N} | | | | {R R}_{i i} x x - - {Ψ Ψ}_{i i} {α α}_{i i} | | {| |}_{22}^{22} - - - - - - ((1818))$

其中y表示输入低分辨率图像，x表示待重建高分辨率图像，D为下采样矩阵；M为非局部自相似权重矩阵，用来描述图像块之间的非局部自相似关系；Ψ_i为第i个图像块对应的子字典；α为稀疏表示稀疏矩阵，其每一行α_i为第i个图像块在子字典Ψ_i下的稀疏表示系数向量，α_i的每个元素为α_ij；α_ij为第i个图像块的非局部自相似性块的稀疏表示系数；R_i为抽取矩阵，其作用是将第i个图像块从图像中抽取出来；λ_i、γ_i为加权向量，其每个元素 $λ_{i j} = \frac{c_{1}}{| α_{i j} | + ϵ}, γ_{i j} = \frac{c_{2}}{{(α_{i j} - {α_{i j}}^{*})}^{2} + ϵ};$ θ为加权系数；where y represents the input low-resolution image, x represents the high-resolution image to be reconstructed, D is the downsampling matrix; M is the non-local self-similar weight matrix, which is used to describe the non-local self-similar relationship between image blocks; Ψ _i is The sub-dictionary corresponding to the i-th image block; α is a sparse representation sparse matrix, and each row α _i is the sparse representation coefficient vector of the i-th image block under the sub-dictionary Ψ _i , and each element of α _i is α _ij ; α _ij is the sparse representation coefficient of the non-local self-similarity block of the i-th image block; R _i is the extraction matrix, and its function is to extract the i-th image block from the image; λ _i and γ _i are weighted vectors, each of its elements $λ_{i j} = \frac{c_{1}}{| α_{i j} | + ϵ}, γ_{i j} = \frac{c_{2}}{{(α_{i j} - {α_{i j}}^{*})}^{2} + ϵ};$ θ is the weighting coefficient;

并对式(17)、(18)迭代求解；式(17)由迭代收缩算法得到最终解；为解决式(18)，构造拉格朗日方程为：And iteratively solve equations (17) and (18); the final solution of equation (17) is obtained by iterative contraction algorithm; to solve equation (18), construct the Lagrangian equation as:

$L L ((X x,, Z Z,, τ τ)) = = | | | | Y Y - - D D. W W X x | | {| |}_{22}^{22} + + θ θ | | | | {R R}_{i i} X x - - {Ψ Ψ}_{i i} {α α}_{i i} | | {| |}_{22}^{22} + + < < Z Z,, Y Y - - D D. X x > > + + τ τ | | | | Y Y - - D D. X x | | {| |}_{22}^{22} - - - - - - ((1919))$

其中，Z是拉格朗日乘子，τ为常量，式(19)可由下式迭代inter_num次进行求解，iter表示当前的迭代次数：Among them, Z is the Lagrangian multiplier, τ is a constant, formula (19) can be solved by iterating inter_num times as follows, and iter represents the current number of iterations:

${x x}^{((i i t t e e r r + + 11))} = = arg arg \underset{x x}{min min} L L ((x x,, {Z Z}^{((i i t t e e r r))},, {τ τ}^{((i i t t e e r r))})) - - - - - - ((2020))$

Z^(iter+1)＝Z^(iter)+τ^iter(y-Dx^(iter+1))(21)Z ^(iter+1) = Z ^(iter) +τ ^iter (y-Dx ^(iter+1) )(21)

τ^(iter+1)＝δτ^(iter)(22)τ ^(iter+1) = δτ ^(iter) (22)

令将式(20)化简为式(23)进行求解：make Simplify formula (20) into formula (23) for solution:

${x x}^{((i i t t e e r r + + 11))} = = {[[{((D D. W W))}^{T T} D D. W W + + θ θ {Σ Σ}_{i i = = 11}^{N N} {R R}_{i i}^{T T} {R R}_{i i} + + {τ τ}^{((i i t t e e r r))} {D D.}^{T T} D D.]]}^{- - 11} [[{((D D. W W))}^{T T} y the y + + θ θ {Σ Σ}_{i i = = 11}^{N N} {R R}_{i i}^{T T} {R R}_{i i} (({Ψ Ψ}_{i i} {α α}_{i i})) + + {D D.}^{T T} {Z Z}^{((i i t t e e r r))} / / 22 + + {τ τ}^{((i i t t e e r r))} {D D.}^{T T} y the y]] - - - - - - ((23 twenty three));;$

x为待重建高分辨率图像，δ为更新常量τ的更新倍数，为常数。x is the high-resolution image to be reconstructed, and δ is the update multiple of the update constant τ, which is a constant.

与现有技术中相比，本发明所提出的算法在解决前人的基于学习的超分辨率重建算法需要大量训练集的缺陷的基础上，改进了邻域嵌入方法，并将其用于解决基于局部自相似性和多尺度相似性的超分辨率算法中存在的不准确高频初始估计问题，提升了图像的超分辨率重建效果。实验结果表明，本发明提出的算法能够更好地抑制了锯齿效应和振铃效应，重建出的高分辨率图像更接近于真实图像，具有更好的主观和客观质量。Compared with the prior art, the algorithm proposed in the present invention improves the neighborhood embedding method on the basis of solving the defect that the previous learning-based super-resolution reconstruction algorithm requires a large number of training sets, and uses it to solve The problem of inaccurate high-frequency initial estimation in the super-resolution algorithm based on local self-similarity and multi-scale similarity improves the super-resolution reconstruction effect of the image. Experimental results show that the algorithm proposed by the invention can better suppress the jagged effect and the ringing effect, and the reconstructed high-resolution image is closer to the real image, and has better subjective and objective quality.

附图说明Description of drawings

图1为不同非局部自相似块数量重建结果比较：(1a)、原图；(1b～1f)、本文算法分别在非局部自相似块数量为5,10,15,20,25时的重建结果；Figure 1 is the comparison of the reconstruction results of different non-local self-similar block numbers: (1a), the original image; (1b-1f), the reconstruction of the algorithm in this paper when the number of non-local self-similar blocks is 5, 10, 15, 20, 25 result;

图2为“Leaves”超分辨率重建结果；：(2a)、原图；(2b)、Bicubic超分辨率重建；(2c)、NE；(2d)、NARM；(2e)、本文算法；Figure 2 is the super-resolution reconstruction result of "Leaves"; (2a), the original image; (2b), Bicubic super-resolution reconstruction; (2c), NE; (2d), NARM; (2e), the algorithm of this paper;

图3为“Cameraman”超分辨率重建结果：(3a)、原图；(3b)、Bicubic超分辨率重建；(3c)、NE；(3d)、NARM；(3e)、本文算法；Figure 3 shows the super-resolution reconstruction results of "Cameraman": (3a), the original image; (3b), Bicubic super-resolution reconstruction; (3c), NE; (3d), NARM; (3e), the algorithm of this paper;

图4为本发明的基于改进邻域嵌入和结构自相似性的超分辨率重建算法模型示意图。Fig. 4 is a schematic diagram of a super-resolution reconstruction algorithm model based on improved neighborhood embedding and structural self-similarity of the present invention.

具体实施方式detailed description

以下结合附图及具体实施方式，进一步详述本发明的技术方案。The technical solution of the present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

在现有的利用压缩感知进行超分辨率重建的方法中，均需要对高分辨率图像进行初始估计。基于这一特点，并考虑到初始估计的准确与否直接影响图像重建的质量与迭代次数，本发明提出一种将邻域嵌入和结构自相似性有效结合的图像超分辨率重建算法。首先，用结构相似性改进邻域嵌入方法，进而获得更加准确的高频初始估计；接着，利用低分辨率图像的局部自相似性和多尺度结构相似性构建重建约束项重建高分辨。In the existing super-resolution reconstruction methods using compressed sensing, an initial estimation of high-resolution images is required. Based on this feature, and considering that the accuracy of the initial estimation directly affects the quality and number of iterations of image reconstruction, the present invention proposes an image super-resolution reconstruction algorithm that effectively combines neighborhood embedding and structural self-similarity. First, the neighborhood embedding method is improved with structural similarity to obtain a more accurate high-frequency initial estimate; then, the local self-similarity and multi-scale structural similarity of low-resolution images are used to construct reconstruction constraints to reconstruct high-resolution images.

技术方案如下：The technical solution is as follows:

1、基于稀疏表示的图像超分辨率重建模型1. Image super-resolution reconstruction model based on sparse representation

设U和S为向量，R为实空间，那么大小的图像块组成列向量x∈R^U×1，若存在矩阵Ψ以及向量α使得：Let U and S be vectors, and R be a real space, then The size of the image block constitutes a column vector x∈R ^U×1 , if there is a matrix Ψ and a vector α such that:

x＝Ψα，||α||₀＝k＜＜U(1)x＝Ψα，||α|| ₀ ＝k＜＜U(1)

则称向量x在Ψ下具有稀疏性，其中Ψ∈R^U×S称为字典，字典Ψ中的每一列称为字典中的一个原子，α∈R^S×1为稀疏表示系数且满足||α||₀＝k＜＜U，||α||₀表示α中非零元的个数。根据稀疏表示理论，图像块x可以表示为字典Ψ中少数几个元素的线性组合，则字典表示可描述为下述最优化问题：Then the vector x is said to have sparsity under Ψ, where Ψ∈R ^U×S is called a dictionary, each column in the dictionary Ψ is called an atom in the dictionary, α∈R ^S×1 is a sparse representation coefficient and satisfies || α|| ₀ ＝k<<U, ||α|| ₀ represents the number of non-zero elements in α. According to the sparse representation theory, the image block x can be expressed as a linear combination of a few elements in the dictionary Ψ, then the dictionary representation can be described as the following optimization problem:

$\underset{α α}{min min} | | | | x x - - Ψ Ψ α α | | {| |}_{22}^{22} + + λ λ | | | | α α | | {| |}_{11} - - - - - - ((22))$

其中λ为平衡稀疏项和保真项的正则化系数，(2)式被称为稀疏表示的基本模型。Where λ is the regularization coefficient to balance the sparse term and the fidelity term, and (2) is called the basic model of sparse representation.

设x表示高分辨率图像组成的向量，y表示高分辨率图像下采样得到的低分辨率图像组成的向量，下采样矩阵记为D，则高分辨率图像与低分辨率图像之间的关系可表示为：Suppose x represents a vector composed of high-resolution images, y represents a vector composed of low-resolution images obtained by downsampling high-resolution images, and the downsampling matrix is denoted as D, then the relationship between high-resolution images and low-resolution images Can be expressed as:

y＝Dx(3)y=Dx(3)

从低分辨率图像中重建出高分辨率图像便是解决如下最小二乘问题：Reconstructing a high-resolution image from a low-resolution image is to solve the following least squares problem:

$\overset{^^}{x x} = = arg arg \underset{x x}{min min} | | | | y the y - - D D. x x | | {| |}_{22}^{22} - - - - - - ((44))$

结合(2)和(4)可以得到基于稀疏表示的图像超分辨率重建模型：Combining (2) and (4) can get the image super-resolution reconstruction model based on sparse representation:

$α α = = arg arg \underset{α α}{min min} | | | | y the y - - D D. x x | | {| |}_{22}^{22} + + λ λ | | | | α α | | {| |}_{11} + + + + θ θ | | | | x x - - Ψ Ψ α α | | {| |}_{22}^{22} - - - - - - ((55))$

2、改进的基于结构自相似性的图像超分辨率重建2. Improved image super-resolution reconstruction based on structural self-similarity

在本发明中，利用非局部自回归超分辨率重建模型：In the present invention, the non-local autoregressive super-resolution reconstruction model is used:

${{α α,, x x}} = = arg arg \underset{α α,, x x}{min min} | | | | y the y - - D D. M m x x | | {| |}_{22}^{22} + + {Σ Σ}_{i i = = 11}^{N N} {λ λ}_{i i} | | | | {α α}_{i i} | | {| |}_{11} + + {Σ Σ}_{i i = = 11}^{N N} {γ γ}_{i i} | | | | {α α}_{i i} - - {α α}_{i i}^{* *} | | {| |}_{22}^{22} + + θ θ {Σ Σ}_{i i = = 11}^{N N} | | | | {R R}_{i i} x x - - {Ψ Ψ}_{i i} {α α}_{i i} | | {| |}_{22}^{22} - - - - - - ((66))$

其中y为输入低分辨率图像，x为待重建高分辨率图像，D为下采样矩阵；M为非局部自相似权重矩阵，用来描述图像块之间的非局部自相似关系；Ψ_i为第i个图像块对应的子字典；α为稀疏表示稀疏矩阵，其每一行α_i为第i个图像块在子字典Ψ_i下的稀疏表示系数向量，α_i的每个元素为α_ij；为第i个图像块的非局部自相似性块的稀疏表示系数；R_i为抽取矩阵，其作用是将第i个图像块从图像中抽取出来；λ_i、γ_i为加权向量，其每个元素 $λ_{i j} = \frac{c_{1}}{| α_{i j} | + ϵ}, γ_{i j} = \frac{c_{2}}{{(α_{i j} - {α_{i j}}^{*})}^{2} + ϵ};$ θ为加权系数。where y is the input low-resolution image, x is the high-resolution image to be reconstructed, D is the downsampling matrix; M is the non-local self-similar weight matrix, which is used to describe the non-local self-similar relationship between image blocks; Ψ _i is The sub-dictionary corresponding to the i-th image block; α is a sparse representation sparse matrix, and each row α _i is the sparse representation coefficient vector of the i-th image block under the sub-dictionary Ψ _i , and each element of α _i is α _ij ; is the sparse representation coefficient of the non-local self-similarity block of the i-th image block; R _i is an extraction matrix, and its function is to extract the i-th image block from the image; λ _i and γ _i are weighted vectors, each of which elements $λ_{i j} = \frac{c_{1}}{| α_{i j} | + ϵ}, γ_{i j} = \frac{c_{2}}{{(α_{i j} - {α_{i j}}^{*})}^{2} + ϵ};$ θ is the weighting coefficient.

在该模型中，需要α对x两个待求量进行迭代求解，因此对x的初始估计将与模型求解的迭代次数直接相关；同时，对x的初始估计也通过影响子字典Ψ_i的选择，进而影响重建图像的质量。但是如果简单采用Bicubic插值方法作为高分辨率图像的初始估计，这将引起图像的边缘锯齿效应。以上述模型为基础，提出了一种基于邻域嵌入的初始估计算法，将非局部自相似性作为先验信息，加入邻域嵌入法的正则项，构建更加具有鲁棒性的高频初始估计，并在此基础上进行超分辨率重建。In this model, α is required to iteratively solve the two quantities of x to be sought, so the initial estimate of x will be directly related to the number of iterations of the model solution; at the same time, the initial estimate of x also affects the selection of the sub-dictionary Ψ _i , thereby affecting the quality of the reconstructed image. However, if the Bicubic interpolation method is simply used as the initial estimate of the high-resolution image, it will cause the edge jagged effect of the image. Based on the above model, an initial estimation algorithm based on neighborhood embedding is proposed, which uses non-local self-similarity as prior information, and adds the regular term of the neighborhood embedding method to construct a more robust high-frequency initial estimation , and perform super-resolution reconstruction on this basis.

2.1基于邻域嵌入的初始估计算法2.1 Initial Estimation Algorithm Based on Neighborhood Embedding

邻域嵌入算法假设低分辨率图像块和高分辨率图像块具有相似的局部流形，利用低分辨率图像块和高分辨率图像块之间的对应关系来估计低分辨率图像中所缺失的高频细节，进行超分辨率重建。在本发明中，引入非局部自相似性(NL)正则项来增强邻域嵌入超分辨率重建的效果。NL方法的基本思想是：在搜索范围内搜索相同尺度的相似块，由于这些相似块可能在几何位置上距离待估计块较远，故搜索范围可能大到整幅图像，利用这些相似图像块所提供的互补信息重构高分辨率图像。本文的具体算法如下：The neighborhood embedding algorithm assumes that low-resolution image patches and high-resolution image patches have similar local manifolds, and uses the correspondence between low-resolution image patches and high-resolution image patches to estimate the missing High-frequency details for super-resolution reconstruction. In the present invention, a non-local self-similarity (NL) regular term is introduced to enhance the effect of neighborhood embedding super-resolution reconstruction. The basic idea of the NL method is to search for similar blocks of the same scale within the search range. Since these similar blocks may be geometrically far away from the block to be estimated, the search range may be as large as the entire image. High-resolution images are reconstructed from the complementary information provided. The specific algorithm in this paper is as follows:

首先提取高分辨率训练图像I_H的亮度分量，接着对高分辨率图像进行a倍的下采样操作得到低分辨率图像I_L，并将高分辨率和低分辨率图像分成相互有重叠区域的小块。记X＝{x^m,m＝1,...,p}为训练的低分辨率块集合，Y＝{y^m,m＝1,...,p}为与之对应的高分辨率块集合，其中p是从训练图像中分割出的小块的数量。同样的，记为待重建的低分辨率图像分割出的块集合，那么为待估计的高分辨率图像的块集合，其中q是测试图像中分割出小块的数量。将图像块x^m和图像块的距离矩阵定义为D_pq，其中D的第n列为与X中所有图像块的距离构成的列向量。然后遍历D中的每一列找到K个最大值，其索引对应的图像块即所求图像块在训练集中的K个近邻块，记为 $S_{l}^{n} = {{\hat{x}}^{n (t 1)}, {\hat{x}}^{n (t 2)}, ..., {\hat{x}}^{n (t K)}} .$ First extract the luminance component of the high-resolution training image I _H , then perform a downsampling operation on the high-resolution image to obtain the low-resolution image I _L , and divide the high-resolution and low-resolution images into overlapping regions small pieces. Note that X={x ^m ,m=1,...,p} is the set of low-resolution blocks for training, and Y={y ^m ,m=1,...,p} is the corresponding high-resolution block A set of blocks, where p is the number of small blocks segmented from the training image. same, remember The set of blocks segmented for the low-resolution image to be reconstructed, then is the block set of the high-resolution image to be estimated, where q is the number of small blocks in the test image. Combine image patch x ^m and image patch The distance matrix for is defined as D _pq , where the nth column of D is Column vector of distances to all image patches in X. Then traverse each column in D to find K maximum values, and the image block corresponding to its index is the requested image block The K nearest neighbor blocks in the training set are denoted as $S_{l}^{no} = {{\hat{x}}^{no (t 1)}, {\hat{x}}^{no (t 2)}, ..., {\hat{x}}^{no (t K)}} .$

接着对于每一个测试图像块利用最小化局部重建误差的方式求得最佳权值向量ω_n＝[ω_n1,ω_n2,...ω_np]：Then for each test image block The optimal weight vector ω _n =[ω _n1 ,ω _n2 ,...ω _np ] is obtained by minimizing the local reconstruction error:

利用核回归方式求解ω_nm可以得到每个高分辨率块(分块后的块中图像的像素点值)的估计值：Using the kernel regression method to solve ω _nm can get each high-resolution block (the pixel value of the image in the block after block) Estimated value of :

对于各个高分辨率图像块的重叠区域，取其各个重叠像素值的平均以避免严重的块效应。For the overlapping area of each high-resolution image block, the average value of each overlapping pixel value is taken to avoid serious block effects.

为引入非局部自相似性信息，利用下式定义残余误差e_l为图像块与另一图像块的相似程度，残余误差越小说明两图像块越相似：In order to introduce non-local self-similarity information, the following formula is used to define the residual error e _l as the image block with another image block The smaller the residual error, the more similar the two image blocks are:

对于每一个图像块计算图像块与其搜索邻域内的所有图像块的残余误差，找到残余误差最小的L个图像块即为图像块的L个最相似块集合其残余误差分别为则待估计图像块可以表示为的线性组合：for each image block Computing Image Blocks Instead of searching the residual errors of all image blocks in the neighborhood, find the L image blocks with the smallest residual error as the image block The L most similar block sets of Its residual error is Then the image block to be estimated It can be expressed as A linear combination of:

其中，中的每个元素与之间的相似度权值可由下式计算：in, each element in and The similarity weight between It can be calculated by the following formula:

在(12)中，h是权值的控制因子。那么与中的每个元素之间的相似度权值可定义为：In (12), h is the control factor of the weight. So and each element in The similarity weight between can be defined as:

令为组成的向量，则(11)可以描述为：make for composed of vectors, Then (11) can be described as:

将(14)作为非局部自相似正则项加入邻域嵌入方法中，可构造改进的邻域超分辨率重建方法模型：Adding (14) as a non-local self-similar regular term to the neighborhood embedding method can construct an improved neighborhood super-resolution reconstruction method model:

其中W为字典序的ω_n，即W＝[ω₁,ω₂,...,ω_q]^T，Φ为字典序的即公式(15)可通过梯度下降法求解，化简为Where W is the ω _n of lexicographical order, that is, W=[ω ₁ ,ω ₂ ,...,ω _q ] ^T , and Φ is the lexicographical order which is Formula (15) can be solved by gradient descent method, which can be simplified as

t为迭代次数，λ为正则化系数常量。设为邻域嵌入法得到的迭代初始值，经过t次迭代，可以得到准确的高频初始估计 t is the number of iterations, and λ is the regularization coefficient constant. Assume is the iterative initial value obtained by the neighborhood embedding method, after t iterations, an accurate high-frequency initial estimate can be obtained

2.2建立稀疏表示字典2.2 Establish sparse representation dictionary

为了进行(6)的求解，在一个准确的初始估计的基础上，还需要建立一个有效的稀疏表示字典。在本文中，采用PCA(主成分分析)字典学习方法。PCA字典学习方法[30]是一种有效的字典构建方法，该方法首先对训练的高分辨率样本进行分块操作，得到高分辨率原子集A_s，接着用K-means算法将A_s分成n簇{K₁,K₂,...,K_n}，每个簇的中心记为{C₁,C₂,...,C_n}，然后对于每一个簇K_m计算它的协方差矩阵Ω_m，通过对Ω_m进行PCA变换得到最终的正交变换矩阵，将其作为子字典Ψ_m，进而得到子字典集合{Ψ₁,Ψ₂,...,Ψ_n}。In order to solve (6), on the basis of an accurate initial estimate, it is also necessary to establish an effective sparse representation dictionary. In this paper, PCA (Principal Component Analysis) dictionary learning method is adopted. The PCA dictionary learning method [30] is an effective dictionary construction method. This method first divides the training high-resolution samples into blocks to obtain the high-resolution atomic set A _s , and then uses the K-means algorithm to divide A _s into n clusters {K ₁ ,K ₂ ,...,K _n }, the center of each cluster is denoted as {C ₁ ,C ₂ ,...,C _n }, and then for each cluster K _m calculate its correlation Variance matrix Ω _m , the final orthogonal transformation matrix is obtained by performing PCA transformation on Ω _m , which is used as a sub-dictionary Ψ _m , and then a sub-dictionary set {Ψ ₁ , Ψ ₂ ,...,Ψ _n } is obtained.

PCA字典学习方法一般采用外部字典，庞大的外部图像训练库会消耗大量内存。针对这个问题，本文中不使用外部字典，而是充分利用图像的多尺度相似性，原子集A_s通过对输入低分辨率图像的多尺度下采样分块产生，从而训练出多尺度自相似子字典。The PCA dictionary learning method generally uses an external dictionary, and a huge external image training library will consume a lot of memory. To solve this problem, this paper does not use an external dictionary, but makes full use of the multi-scale similarity of the image. The atomic set A _s is generated by multi-scale downsampling of the input low-resolution image, thereby training a multi-scale self-similar subclass dictionary.

将2.1中得到的高分辨率初始估计图像进行分块操作后，对于每一个待重建高分辨率输入块x_i，将其与已经训练得到的簇中心{C₁,C₂,...,C_n}进行比较，找到与输入块欧氏距离最小的簇中心C_i，其所在的簇K_i所对应的子字典Ψ_i即待重建高分辨率块x_i所使用的字典。得到每个待重建图像块所对应的子字典Ψ_i后，(6)可以通过拉格朗日乘子法转化为(17)(18)两式，并对(17)(18)迭代求解。After the high-resolution initial estimation image obtained in 2.1 is divided into blocks, for each high-resolution input block x _i to be reconstructed, compare it with the cluster center {C ₁ ,C ₂ ,..., C _n } for comparison, find the cluster center C _i with the smallest Euclidean distance to the input block, and the sub-dictionary Ψ _i corresponding to the cluster K _i where it is located is the dictionary used to reconstruct the high-resolution block _xi . After obtaining the sub-dictionary Ψ _i corresponding to each image block to be reconstructed, (6) can be transformed into (17) (18) through the Lagrangian multiplier method, and (17) (18) can be solved iteratively.

(17)可由迭代收缩算法得到最终解。为解决(18)式，构造拉格朗日方程为：(17) The final solution can be obtained by iterative contraction algorithm. To solve equation (18), the Lagrangian equation is constructed as:

其中，Z是拉格朗日乘子，τ为常量，(19)可由下式迭代inter_num次进行求解：Among them, Z is the Lagrangian multiplier, τ is a constant, and (19) can be solved by iterating inter_num times as follows:

τ^(iter+1)＝δτ^(iter)(22)τ ^(iter+1) = δτ ^(iter) (22)

令(20)可化简为式(23)进行求解：make (20) can be simplified to formula (23) for solution:

${x x}^{((i i t t e e r r + + 11))} = = {[[{((D D. W W))}^{T T} D D. W W + + θ θ {Σ Σ}_{i i = = 11}^{N N} {R R}_{i i}^{T T} {R R}_{i i} + + {τ τ}^{((i i t t e e r r))} {D D.}^{T T} D D.]]}^{- - 11} [[{((D D. W W))}^{T T} y the y + + θ θ {Σ Σ}_{i i = = 11}^{N N} {R R}_{i i}^{T T} {R R}_{i i} (({Ψ Ψ}_{i i} {α α}_{i i})) + + {D D.}^{T T} {Z Z}^{((i i t t e e r r))} / / 22 + + {τ τ}^{((i i t t e e r r))} {D D.}^{T T} y the y]] - - - - - - ((23 twenty three)) . .$

在实验中，分别采用彩色图像和灰度图像作为测试图像，先对原始图像进行下采样，然后进行3倍尺度的超分辨率重建放大，并与Bicubic超分辨率重建、NE、NARM进行对比。本文方法中，低分辨率图像块的大小设为3×3，其对应的高分辨率图像块的大小为9×9，邻域相似块数量K为5，非局部相似块数量L为10，非局部相似块大小为3×3，权值控制因子h为65，迭代次数t为30，正则化因子λ、c₁、c₂、θ、ε、δ和inter_num分别为0.05、0.15、2.5、0.15、0.3、1.2和180。为了评价重建效果，除了采用主观的视觉评价外，还采用峰值信噪比(PSNR)和图像结构相似度(SSIM)评价不同算法的超分辨率重建性能。PSNR是一种基于统计特性的客观图像质量评价方法，较大的峰值信噪比表示重构图像与原始图像之间的相似性较高。SSIM是一种通过感知图像结构信息来进行图像质量评价的方法，SSIM值越大说明图像间的结构越相似。In the experiment, the color image and the grayscale image were used as the test image respectively, the original image was down-sampled first, and then the super-resolution reconstruction of 3 times the scale was enlarged, and compared with the Bicubic super-resolution reconstruction, NE, and NARM. In the method of this paper, the size of the low-resolution image block is set to 3×3, and the size of the corresponding high-resolution image block is 9×9, the number K of similar blocks in the neighborhood is 5, and the number L of non-local similar blocks is 10. The non-local similarity block size is 3×3, the weight control factor h is 65, the number of iterations t is 30, and the regularization factors λ, c ₁ , c ₂ , θ, ε, δ and inter_num are 0.05, 0.15, 2.5, 0.15, 0.3, 1.2, and 180. To evaluate the reconstruction performance, in addition to subjective visual evaluation, peak signal-to-noise ratio (PSNR) and image structure similarity (SSIM) are used to evaluate the super-resolution reconstruction performance of different algorithms. PSNR is an objective image quality evaluation method based on statistical properties. A larger peak signal-to-noise ratio indicates a higher similarity between the reconstructed image and the original image. SSIM is a method for evaluating image quality by perceiving image structure information. The larger the SSIM value, the more similar the structure between images.

由于本发明提出的方法引入非局部自相似性约束，为验证非局部自相似块数量对重建效果的影响，采用256×256Cameraman图像进行实验。实验中，分别选取非局部自相似块L为5,10,15,20,25进行重构。从表1和图1可以看出，随着非局部自相似块数量的增加，边缘处的不连续现象得到一定缓解，但是PSNR值逐渐下降，同时造成了细节区域的过平滑模糊，因此综合考虑图像的细节因素和边缘效果，本文选取L为10，可以达到良好的重建效果。Since the method proposed in the present invention introduces non-local self-similarity constraints, in order to verify the influence of the number of non-local self-similar blocks on the reconstruction effect, a 256×256 Cameraman image is used for experiments. In the experiment, non-local self-similar blocks L are selected as 5, 10, 15, 20, and 25 for reconstruction. It can be seen from Table 1 and Figure 1 that as the number of non-local self-similar blocks increases, the discontinuity phenomenon at the edge is alleviated to a certain extent, but the PSNR value gradually decreases, and at the same time it causes over-smoothing blur in the detail area, so comprehensive consideration For the detail factor and edge effect of the image, this paper selects L as 10, which can achieve a good reconstruction effect.

图2和图3分别展示了Leaves和Cameraman的实验比较图，从中可以看出Bicubic方法在放大图像的同时引起了边界的不连续现象与振铃效应；NE方法基本可以保持图像的边缘结构，但与此同时也模糊了图像的大部分细节；NARM方法能够较好地恢复图像细节，但在某些地方，如树叶的边缘细节、镜头架等处重建效果仍不理想；本文方法较好地改进了NARM算法的缺陷，在重建纹理细节和消除伪边缘与锯齿两方面均做到了较好的效果，复原结果更加逼真。因此从视觉效果上看，文中算法的超分辨率重建结果与原始高分辨率图像最为相似。Figure 2 and Figure 3 show the experimental comparison diagrams of Leaves and Cameraman respectively, from which it can be seen that the Bicubic method causes boundary discontinuity and ringing effects while enlarging the image; the NE method can basically maintain the edge structure of the image, but At the same time, most of the details of the image are blurred; the NARM method can restore the image details better, but in some places, such as the edge details of the leaves, the lens frame, etc., the reconstruction effect is still not ideal; the method in this paper is better improved The defects of the NARM algorithm have been overcome, and good results have been achieved in terms of reconstructing texture details and eliminating false edges and aliasing, and the restoration results are more realistic. Therefore, from the perspective of visual effect, the super-resolution reconstruction result of the algorithm in this paper is most similar to the original high-resolution image.

表2展示了不同算法的PSNR和SSIM，可以看出，引入准确高频初始估计改进基于局部和非局部相似性的超分辨率算法能够有效提升重建质量，与一些具有代表性的重建方法相比，PSNR平均值提高了0.47～1.84dB，SSIM平均值提高了0.0114～0.0438。因此从评价指标上看，本发明中提出的算法明显优于其余几种算法，客观评价与主观评价结果完全一致。Table 2 shows the PSNR and SSIM of different algorithms. It can be seen that the introduction of accurate high-frequency initial estimation to improve the super-resolution algorithm based on local and non-local similarity can effectively improve the reconstruction quality. Compared with some representative reconstruction methods , the average PSNR increased by 0.47-1.84dB, and the average SSIM increased by 0.0114-0.0438. Therefore, from the perspective of evaluation indicators, the algorithm proposed in the present invention is obviously superior to other algorithms, and the objective evaluation and subjective evaluation results are completely consistent.

表1、非局部自相似块数量对重建效果的影响Table 1. The influence of the number of non-local self-similar blocks on the reconstruction effect

表2、不同超分辨率重建算法的PSNR、SSIMTable 2. PSNR and SSIM of different super-resolution reconstruction algorithms

Claims

1. A super-resolution reconstruction algorithm based on improved neighborhood embedding and structural self-similarity is characterized by comprising the following steps:

step (1), based on initial estimation algorithm of neighborhood embedding, firstly extracting high-resolution training image I_HThen a down-sampling operation is performed on the high resolution image to obtain a low resolution image I_LDividing the high-resolution and low-resolution images into small blocks with mutually overlapped regions; let X be { X ═ X^mM 1.. p } is a set of low resolution blocks that are trained, Y ═ Y ·^mM 1.. p } is a set of high resolution blocks corresponding thereto, where p is the number of small blocks segmented from the training image; similarly, noteA set of blocks segmented for the low resolution image to be reconstructed,a set of blocks of a high resolution image to be estimated, wherein q is the number of segmented small blocks in the test image; image block x^mAnd image blockIs defined as D_pqWherein D is_pqIs listed as nAnd X represents a column vector formed by distances of all image blocks in the training low-resolution block set; then traverse D_pqEach column in the image block search unit finds K maximum values, and the index of the maximum value is corresponding to the image block, i.e. the image block to be searchedK neighboring blocks in the training set, denoted as Mapping of representations to a desired image blockT1, t 2.. tK denotes the sequence numbers of the K neighboring blocks;

then for each test image blockMethod for obtaining optimal weight vector omega by minimizing local reconstruction error_n＝[ω_n1,ω_n2,...ω_np]N1, n 2.. nP indicates the subscript of the sequence number, n indicates the nth test image block:

ω_{n} = \arg \underset{ω_{n}}{m i n} | | {\hat{x}}^{n} - Σ_{m = 1}^{p} ω_{n m} x^{m} | |_{2}^{2} - - - (7)

wherein each weight value omega_nmThe following constraints are satisfied:

\{\begin{matrix} Σ_{m = 1}^{p} ω_{n m} = 1 \\ ω_{n m} &NotEqual; 0 & i f & x^{m} &Element; S_{l}^{n} \\ ω_{n m} = 0 & i f & x^{m} &NotElement; S_{l}^{n} \end{matrix} - - - (8)

method for solving omega by using nuclear regression mode_nmObtaining each high resolution image blockEstimated value of (a):

{\hat{y}}^{n} = Σ_{m = 1}^{p} ω_{n m} y^{m} - - - (9)

taking the average of all overlapped pixel values of the overlapped areas of all high-resolution image blocks;

the residual error e is defined by_lFor image blocksWith another image blockThe smaller the residual error, the more similar the two image blocks:

e_{l} = | | {\hat{y}}_{l} - {\tilde{y}}_{l} | |_{2}^{2} - - - (10)

for each image blockComputing image blocksAnd searching residual errors of all image blocks in the neighborhood, and finding L image blocks with the minimum residual errors as the image blocksL most similar block sets:

\tilde{Y} = {{\tilde{y}}^{n i}, i = 1, 2 ... L},

with residual errors of respectivelyLL denotes the number of image blocks; then the image block to be estimatedIs shown asRepresents a linear combination of one of the L image blocks found to have the smallest residual error:

{\hat{y}}^{n} = Σ_{i = 1}^{L} ρ_{n}^{i} {\tilde{y}}^{n i} - - - (11)

wherein,each element in (1)Andsimilarity weight betweenCalculated from the following formula:

ρ_{n}^{i} = \frac{\exp (- \frac{e_{l}^{n i}}{h})}{Σ_{i = 1}^{L} \exp (- \frac{e_{l}^{n i}}{h})} - - - (12)

h is a control factor of the weight;andeach element in (1)The similarity weight between (j ═ 1,2., q) is defined as:

T_{n}^{j} = \{\begin{matrix} ρ_{n}^{i} & i f {\hat{y}}^{j} &Element; \tilde{Y}, {\hat{y}}^{j} = {\tilde{y}}^{n i} \\ 0 & o t h e r w i s e \end{matrix} - - - (13)

representing the jth block in the set of high-resolution image blocks to be estimated,for image blocksL most similar block sets;

order toIs composed ofThe vector of the composition is then calculated,expression (11) is:

adding (14) serving as a non-local self-similarity regular term into a neighborhood embedding method, and constructing a neighborhood super-resolution reconstruction method model:

\hat{Y} = \underset{\hat{Y}}{\arg \min} | | \hat{Y} - W Y | |_{2}^{2} + λ | | \hat{Y} - Φ \hat{Y} | |_{2}^{2} - - - (15)

wherein W is ω of the lexicographic order_nI.e. W ═ ω₁,ω₂,...,ω_q]^TPhi is in lexical orderNamely, it isThe formula (15) is solved by a gradient descent method and simplified into

{\hat{Y}}^{t + 1} = {\hat{Y}}^{t} - λ {(I - Φ)}^{T} (I - Φ) {\hat{Y}}^{t} - - - (16)

t is iteration times, and lambda is a regularization coefficient constant; is provided withFor the iterative initial value obtained by the neighborhood embedding method, the method_tThe iteration is repeated to obtain accurate high-frequency initial estimation

Step (2) establishing a sparse representation dictionary, and after the high-resolution initial estimation image obtained in the step (1) is subjected to blocking operation, for each high-resolution input block x to be reconstructed_iThen, it is compared with the trained cluster center { C }₁,C₂,...,C_nAre compared to find the cluster center C with the smallest Euclidean distance from the input block_iCluster K of which it is located_iThe corresponding sub-dictionary Ψ_iI.e. the high resolution block x to be reconstructed_iDictionary used；

Obtaining a sub-dictionary psi corresponding to each image block to be reconstructed_iThen, the model is reconstructed by using the non-local autoregressive super-resolution

{α, x} = \arg \underset{α, x}{m i n} | | y - D M x | |_{2}^{2} + Σ_{i = 1}^{N} λ_{i} | | α_{i} | |_{1} + Σ_{i = 1}^{N} γ_{i} | | α_{i} - α_{i}^{*} | |_{2}^{2} + θ Σ_{i = 1}^{N} | | R_{i} x - Ψ_{i} α_{i} | |_{2}^{2} - - - (6)

Converting into two formulas (17) and (18) by Lagrange multiplier method,

α = \arg \underset{α}{m i n} Σ_{i = 1}^{N} λ_{i} | | α_{i} | |_{1} + Σ_{i = 1}^{N} γ_{i} | | α_{i} - α_{i}^{*} | |_{2}^{2} + θ Σ_{i = 1}^{N} | | R_{i} x - Ψ_{i} α_{i} | |_{2}^{2} - - - (17)

x = \arg \underset{x}{m i n} | | y - D W x | |_{2}^{2} + θ Σ_{i = 1}^{N} | | R_{i} x - Ψ_{i} α_{i} | |_{2}^{2} - - - (18)

wherein y represents an input low-resolution image, x represents a high-resolution image to be reconstructed, and D is a down-sampling matrix; m is a non-local self-similarity weight matrix and is used for describing the non-local self-similarity relation between the image blocks; Ψ_iA sub-dictionary corresponding to the ith image block, α is a sparse representation sparse matrix with each row α_iIn the sub-dictionary Ψ for the ith image block_iSparse representation of coefficient vectors under, α_iIs α_ij；α_ijSparse representation coefficients of a non-local self-similarity block for an ith image block; r_iFor extracting matrix, the function is to extract the ith image block from imageTo the process; lambda [ alpha ]_i、γ_iIs a weighted vector, each element of which

\begin{matrix} λ_{i j} = \frac{c_{1}}{| α_{i j} | + ϵ}, & γ_{i j} = \frac{c_{2}}{{(a_{i j} - {α_{i j}}^{*})}^{2} + ϵ} \end{matrix};

Theta is a weighting coefficient;

and solving equations (17) and (18) iteratively; obtaining a final solution by an iterative contraction algorithm in the formula (17); to solve equation (18), the lagrange equation is constructed as:

L (X, Z, τ) = | | Y - D W X | |_{2}^{2} + θ | | R_{i} X - Ψ_{i} α_{i} | |_{2}^{2} + < Z, Y - D X > + τ | | Y - D X | |_{2}^{2} - - - (19)

where Z is a lagrange multiplier, τ is a constant, equation (19) can be solved by iterating inter _ num times as follows, iter represents the current number of iterations:

x^{(i t e r + 1)} = \arg \min_{x} L (x, Z^{(i t e r)}, τ^{(i t e r)}) - - - (20)

Z^(iter+1)＝Z^(iter)+τ^iter(y-Dx^(iter+1))(21)

τ^(iter+1)＝τ^(iter)(22)

order toThe solution is performed by reducing equation (20) to equation (23):

x^{(i t e r + 1)} = {[{(D W)}^{T} D W + θ Σ_{i = 1}^{N} {R_{i}}^{T} R_{i} + τ^{(i t e r)} D^{T} D]}^{- 1} [{(D W)}^{T} y + θ Σ_{i = 1}^{N} {R_{i}}^{T} R_{i} (Ψ_{i} α_{i}) + D^{T} Z^{(i t e r)} / 2 + τ^{(i t e r)} D^{T} y] - - - (23);

and x is a high-resolution image to be reconstructed, and is an update multiple of the update constant tau, and is a constant.