CN105046672A - Method for image super-resolution reconstruction - Google Patents

Abstract

The invention discloses a method for image super-resolution reconstruction. A two-dimensional sparse representation model is employed under a super-resolution sparse reconstruction basic framework, so that spatial characteristics of two-dimensional images can be effectively used. The problem that dictionary storage space is large and parameters to be estimated are too much in one-dimensional sparse expression can be solved. The dictionary storage space can be reduced while the image super-resolution reconstruction result is guaranteed.

Description

A kind of image super-resolution rebuilding method

Technical field

The invention belongs to the technical field of the super-resolution of rarefaction representation, relate to a kind of image super-resolution rebuilding method particularly, mainly apply two-dimentional sparse representation model and carry out super-resolution rebuilding in single-frame images.

Background technology

Image is as the important information form in the human perception world, and the abundant and details of its content, directly determines that these mankind experience the level of detail of content.When the picture element density on image as unit yardstick is higher, then image is more clear, and its details ability expressed is stronger, and the information of human perception is abundanter, and this is high-definition picture namely.The super-resolution rebuilding of image has had corresponding research as remote sensing images, satellite imagery field, field of medical images in a lot, and some high-resolution display fields etc.

The method improving the resolution of image mainly reduces pixel dimension by improving sensor manufacturing process, or increases the pixel quantity in unit area.But minimizing pixel elements can reduce the electromagnetic energy that pixel can accept, thus can cause easily by noise, cause picture quality not high.And the size increasing surface-mounted integrated circuit can cause electric capacity to increase.This easily causes electric capacity to be difficult to transfer charge, and its application is reduced.Another kind of method adopts signal processing technology to rebuild single frames or multiframe low-resolution image, obtains high-definition picture, namely the super-resolution rebuilding technology of image processing field.

The super-resolution rebuilding of image refers to the deteriroation of image quality utilizing the method for signal transacting and computer software to eliminate to focus on due to imaging system the factors such as inaccurate, motion blur and imperfect sampling to cause, thus obtains high-resolution picture rich in detail.It is merge the technology from multiple low resolution (LowResolution, LR) images of identical dynamic scene that multiple image is rebuild, and has wherein used the complementary information of multiple low-resolution image.The super-resolution image reconstruction of single-frame images is then the high-frequency information estimated from single-frame images beyond cutoff frequency, reconstructs high-resolution image.

The concept and methodology of super-resolution is proposed the sixties in last century by Harris and Goodman the earliest, and proposes multiple method, as prolate ellipsoid Wave function method, linear extrapolation, the sinusoidal template of superposition etc.These methods are all based on single image.At present certain exploration is carried out to super-resolution technique, but be not very extensive in reality.Last century is for the eighties, Tsai and Huang proposes multiframe super resolution ratio reconstruction method, its main thought is under the prerequisite not changing existing imaging system, if there is the situation of the low-resolution image of several Same Scene, and can in conjunction with multiple image information reconstruction high quality graphic.While the technical development of multiframe super-resolution rebuilding, single-frame images method for reconstructing have also been obtained tremendous development.But in fact, be difficult to obtain several qualified low-resolution images, therefore main it is considered that the Problems of Reconstruction of single frames super-resolution here.

The model being deteriorated to low-resolution image by high resolving power is as follows: Y=SHX+N.Here, Y is low-resolution image, and S is down-sampling operator, and H is fuzzy filter, and X is high-definition picture, and N is noise image.The object of Super-resolution Reconstruction is exactly how will be recovered original X image by Y.

The method of single-frame images super-resolution mainly contains with under type: 1. based on the method for interpolation.Mainly produce high-definition picture by the interpolating function of structure smooth curve or curved surface.I.e. " image interpolation ".Interpolation method conventional at present repeats interpolation, bilinear interpolation and spline interpolation just like arest neighbors.2., based on the method for reconstruct, solve the inverse problem of above image degradation model.Mainly introduce some prioris, under the guidance of priori, solve inverse problem, and due to the solution of inverse problem not unique, so need in certain constraint, as up, edge constraint etc., finally find the solution of error minimize, therefore reconstructing method is a kind of method solving optimization problem.Similar method has maximum a posteriori probability (Maximizeaposterior, MAP) etc.But because the constraint condition in model needs some prioris with it, and in fact the understanding of priori be may not be certain accurate, so reconstruction situation is not fine.3. based on the method for study.Freemanetal. be the super-resolution rebuilding of the instance-based learning proposed the earliest, can by the information in external image storehouse, by the relation that study high-low resolution image is right, to the single frames low-resolution image of input, estimate the detail of the high frequency in high-definition picture, finally obtain rebuilding image, obviously after outside image library information adds, the reconstruction image obtained, the high-definition picture obtained by interpolation than the past is more clear.And the introducing of sample along with the prior imformation of image, add more effective constraint, improve the accuracy of rebuilding image.

The method of current representational study is exactly the corresponding relation between training high-low resolution.Not only ensure edge, and increase the detailed information of texture.As utilized Markov field to image block modeling, using the method for belief propagation (BeliefPropagation, BP) simultaneously, eliminating the problem that border is repeated.Its main thought is exactly using the image block be of moderate size as input unit by the low-resolution image obtained, then the low-resolution image block the most similar to it is found in a database, and the most high-definition picture block corresponding to similar image block found just serves the effect of supplementary high frequency detail, also just can be used for rebuilding high-definition picture.But this based in learning process between high-low resolution image block, if sample is a lot, computation complexity can be made to increase, and if use contiguous image block reconstruction current block also to there is poor fitting and Expired Drugs.

Therefore along with the development of sparse representation theory, the people such as Yang propose the paradigm learning super-resolution reconstruction method based on rarefaction representation, when setting up high-low resolution dictionary, require that sparse coefficient corresponding to high-low resolution image block is the same.Like this for the low resolution image block of input, first can obtain its rarefaction representation coefficient under low resolution base space, be then multiplexed into high resolving power base spatially, thus realize the super-resolution rebuilding of image.But require that sparse coefficient corresponding to high-low resolution image block is that constraint is equally stronger.In fact certain mapping relations should be there are, therefore different scholars is in succession had to propose pre-filtering interpolation, the algorithm that the features such as the sparse deblurring of non-blind strengthen, carrys out rebuilding super resolution about the mapping relations between support vector regression model training high-low resolution image block in addition.Zeyde etc. by reducing the dimension of LR, and utilize the dictionary calculated difference image of LR to obtain the dictionary of HR, instead of directly training obtains the dictionary of LR and HR.The people such as Wang loosen original sparse coefficient need consistent tight constraint by setting up mapping function between sparse coefficient corresponding to LR and HR dictionary.And at present along with the effective application of non local self-similarity characteristic in image restoration problem, also have a lot of people to be incorporated in dictionary learning by this non local similarity characteristic and cluster, thus improve the quality of image super-resolution rebuilding.

The present invention is mainly under the basic framework of the super-resolution sparse reconstruction Super-ResolutionSparseRepresentation (SRSR) of Yang proposition, introduce two-dimentional sparse representation model, make it possible to the spatial character effectively utilizing two dimensional image, reduce in one dimension rarefaction representation simultaneously and there is dictionary storage space comparatively greatly, need the problem that the parameter of estimation is more.The storage space of dictionary finally can be reduced while ensureing image super-resolution rebuilding result.

Summary of the invention

Technology of the present invention is dealt with problems and is: overcome the deficiencies in the prior art, a kind of image super-resolution rebuilding method is provided, it effectively can utilize the spatial character of two dimensional image, reduce in one dimension rarefaction representation simultaneously and there is the problem that dictionary storage space is comparatively large, need the parameter of estimation more, thus the storage space of dictionary can be reduced while ensureing image super-resolution rebuilding result.

Technical solution of the present invention is: this image super-resolution rebuilding method, comprises the following steps:

(1) the RGB image of the low resolution of input is converted into YCbCr image, wherein Y is non-linear luma component, and Cb is blue color difference component, and Cr is red color difference component; Bilinear interpolation is utilized to carry out super-resolution rebuilding to Cb, Cr;

(2) Y-component is designated as lIm, utilizes bilinear interpolation to carry out 2 times of up-samplings, obtain the image mIm of corresponding intermediate resolution, solve the characteristic image of mIm image simultaneously, obtain the characteristic image of different directions and different order;

(3) with the lIm upper left corner for starting point, the image block Y of the 3x3 that samples successively _i, wherein each direction has the repeated region of 1 pixel, calculates the average M of current block;

(4) respectively from the characteristic image of mIm and different directions and different order, the image block of the 6x6 on relevant position is extracted and solve $vec\left(Z_i\right)=\left[\begin{array}{c}vec\left({\tilde{X}}_i\right)\\ vec\left(F^{\left(1\right)}{Y}_i\right)\\ \·\\ \·\\ \·\\ vec\left(F^{\left(k\right)}{Y}_i\right)\\ \·\\ \·\\ \·\\ vec\left(F^{\left(N\right)}{Y}_i\right)\end{array}\right],$ Wherein represent the value of the high-definition picture that repeat region has been rebuild;

(5) dictionary trained is utilized with solve

$\text{D}=\left[\begin{array}{c}{\text{D}}_2^h\⊗D_1^h\\ {\text{C}}_2^{\left(1\right)}\⊗{\text{C}}_1^{\left(1\right)}\\ \·\\ \·\\ \·\\ {\text{C}}_2^{\left(k\right)}\⊗{\text{C}}_1^{\left(k\right)}\\ \·\\ \·\\ \·\\ {\text{C}}_2^{\left(N\right)}\⊗{\text{C}}_1^{\left(N\right)}\end{array}\right];$

(6) sparse coefficient of current block is solved:

$\underset{vec\left(B_i^T\right)}{min}\{{||vec\left(Z_i\right)-Dvec\left(B_i^T\right)||}_F^2+\mathrm{\λ}{||vec\left(B_i^T\right)||}_1\};$

(7) the high-resolution features image block of relevant position is rebuild

(8) by X _i+ M is as the high-definition picture block of the reconstruction of relevant position;

(9) judge whether all to have carried out sampling and super-resolution rebuilding to low-resolution image, if the low-resolution image block of all 3x3 completes all, then perform step (10), otherwise perform step (3)-(9);

(10) final super-resolution image X is solved according to formula (6) ^*:

$\underset{x}{min}\{{||SHX-Y||}_F^2+c{||X-X_0||}_F^2\}---\left(6\right)$

Wherein c is the error that parameter is used for balancing the overall situation and local;

(11) high-definition picture X Y-component super-resolution rebuilding obtained ^*super-resolution rebuilding image combining with utilizing bilinear interpolation to obtain to Cb, Cr, obtains the high-definition picture in YCbCr space; Then color space conversion is carried out to it, be transformed into RGB color space, finally obtain colored super-resolution rebuilding image.

The present invention is under the basic framework of super-resolution sparse reconstruction Super-ResolutionSparseRepresentation (SRSR), introduce two-dimentional sparse representation model, make it possible to the spatial character effectively utilizing two dimensional image, reduce in one dimension rarefaction representation simultaneously and there is the problem that dictionary storage space is comparatively large, need the parameter of estimation more, thus the storage space of dictionary can be reduced while ensureing image super-resolution rebuilding result.

Accompanying drawing explanation

Fig. 1 shows the process flow diagram according to dictionary training method of the present invention;

Fig. 2 shows the process flow diagram according to image super-resolution rebuilding method of the present invention.

Embodiment

As shown in Figure 2, this image super-resolution rebuilding method, comprises the following steps:

(1) the RGB image of the low resolution of input is converted into YCbCr image, wherein Y is non-linear luma component, and Cb is blue color difference component, and Cr is red color difference component; Bilinear interpolation is utilized to carry out super-resolution rebuilding to Cb, Cr;

(2) Y-component is designated as lIm, utilizes bilinear interpolation to carry out 2 times of up-samplings, obtain the image mIm of corresponding intermediate resolution, solve the characteristic image of mIm image simultaneously, obtain the characteristic image of different directions and different order;

(3) with the lIm upper left corner for starting point, the image block Y of the 3x3 that samples successively _i, wherein each direction has the repeated region of 1 pixel, calculates the average M of current block;

(4) respectively from the characteristic image of mIm and different directions and different order, the image block of the 6x6 on relevant position is extracted and solve $vec\left(Z_i\right)=\left[\begin{array}{c}vec\left({\tilde{X}}_i\right)\\ vec\left(F^{\left(1\right)}{Y}_i\right)\\ \·\\ \·\\ \·\\ vec\left(F^{\left(k\right)}{Y}_i\right)\\ \·\\ \·\\ \·\\ vec\left(F^{\left(N\right)}{Y}_i\right)\end{array}\right],$ Wherein represent the value of the high-definition picture that repeat region has been rebuild;

(5) dictionary trained is utilized with solve

$\text{D}=\left[\begin{array}{c}{\text{D}}_2^h\⊗D_1^h\\ {\text{C}}_2^{\left(1\right)}\⊗{\text{C}}_1^{\left(1\right)}\\ \·\\ \·\\ \·\\ {\text{C}}_2^{\left(k\right)}\⊗{\text{C}}_1^{\left(k\right)}\\ \·\\ \·\\ \·\\ {\text{C}}_2^{\left(N\right)}\⊗{\text{C}}_1^{\left(N\right)}\end{array}\right];$

(6) sparse coefficient of current block is solved:

$\underset{vec\left(B_i^T\right)}{min}\{{||vec\left(Z_i\right)-Dvec\left(B_i^T\right)||}_F^2+\mathrm{\λ}{||vec\left(B_i^T\right)||}_1\};$

(7) the high-resolution features image block of relevant position is rebuild

(8) by X _i+ M is as the high-definition picture block of the reconstruction of relevant position;

(9) judge whether all to have carried out sampling and super-resolution rebuilding to low-resolution image, if the low-resolution image block of all 3x3 completes all, then perform step (10), otherwise perform step (3)-(9);

(10) utilize gradient descent method (gradientdescentalgorithm), solve final super-resolution image X according to formula (6) ^*:

$\underset{x}{min}\{{||SHX-Y||}_F^2+c{||X-X_0||}_F^2\}---\left(6\right)$

Wherein c is the error that parameter is used for balancing the overall situation and local;

(11) high-definition picture X Y-component super-resolution rebuilding obtained ^*super-resolution rebuilding image combining with utilizing bilinear interpolation to obtain to Cb, Cr, obtains the high-definition picture in YCbCr space; Then color space conversion is carried out to it, be transformed into RGB color space, finally obtain colored super-resolution rebuilding image.

The present invention is under the basic framework of super-resolution sparse reconstruction Super-ResolutionSparseRepresentation (SRSR), introduce two-dimentional sparse representation model, make it possible to the spatial character effectively utilizing two dimensional image, reduce in one dimension rarefaction representation simultaneously and there is the problem that dictionary storage space is comparatively large, need the parameter of estimation more, thus the storage space of dictionary can be reduced while ensureing image super-resolution rebuilding result.

Preferably, the characteristic image obtaining different directions and different order in described step (2) is single order horizontal gradient, single order VG (vertical gradient), second order horizontal gradient, second order VG (vertical gradient), is respectively f ₁=[-101], f ₂=f ₁ ^t, f ₃=[10-201], f ₄=f ₃ ^t, after filtering, obtain the characteristic image f of corresponding intermediate resolution image ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm.

Preferably, as shown in Figure 1, the dictionary of described step (5) comprises step by step following:

(5.1) set up high-resolution image data base, if image itself is coloured image, then first coloured image is converted into gray level image;

(5.2) image data base of low resolution is set up: to the image in outside image library, carry out 3 times of down-samplings, obtain corresponding low-resolution image lIm, then to all low-resolution image lIm, carry out 2 times of up-samplings, obtain the image mIm of intermediate resolution, solve the characteristic image of mIm image simultaneously, namely different filtering process is adopted, obtain the characteristic image of different directions and different order, the characteristic image of corresponding intermediate resolution image is obtained after filtering, utilize mIm, the characteristic image of intermediate resolution image is as the image data base of low resolution,

(5.3) to the image data base of the high-low resolution set up, sampling obtains the training sample set of paired high-definition picture block and low resolution characteristic image block; (if image itself is coloured image, then need first color RGB image to be converted into gray level image (gray) image, otherwise directly as the image Im in high resolution image data storehouse)

(5.4) initialization dictionary with the image block X of Stochastic choice same index _i, with be averaging by row respectively and obtain hx _i, and be averaging by row and obtain vx _i ^t, and by hx _ias dictionary a row primitive, and vx _ias dictionary a row primitive, can construct respectively by this way a row primitive;

(5.5) to dictionary with carry out sparse coding: first utilize with fabric tensor dictionary respectively, thus obtain the dictionary for sparse coding reconstruction $D=\left[\begin{array}{c}{D}_2^h\⊗D_1^h\\ C_2^{\left(1\right)}\⊗C_1^{\left(1\right)}\\ C_2^{\left(2\right)}\⊗C_1^{\left(2\right)}\\ C_2^{\left(3\right)}\⊗C_1^{\left(3\right)}\\ C_2^{\left(4\right)}\⊗C_1^{\left(4\right)}\\ C_2^{\left(5\right)}\⊗C_1^{\left(5\right)}\end{array}\right],$ And operation is normalized, simultaneously to sample set to each column signal of D in every pair of sample, construct new vector $vec\left(Z_i\right)=\left[\begin{array}{c}vec\left(X_i\right)\\ vec\left(Y_i^{\left(1\right)}\right)\\ vec\left(Y_i^{\left(2\right)}\right)\\ vec\left(Y_i^{\left(3\right)}\right)\\ vec\left(Y_i^{\left(4\right)}\right)\\ vec\left(Y_i^{\left(5\right)}\right)\end{array}\right],$ Then formula (5) is utilized to solve i-th sparse coefficient B _iso, obtain all sparse coefficient set successively

$\underset{vec\left(B_i^T\right)}{min}\{{||vec\left(Z_i\right)-Dvec\left(B_i^T\right)||}_F^2+\mathrm{\λ}{||vec\left(B_i^T\right)||}_1\}---\left(5\right);$

(5.6) to dictionary with carry out dictionary updating: upgrade dictionary respectively with

(5.7) judge whether to reach iteration stopping condition: if do not meet iteration stopping condition, jump to step (5.3); If meet iteration stopping condition, then export dictionary with

Preferably, in described step (5.2), the characteristic image of different directions and different order comprises single order horizontal gradient, single order VG (vertical gradient), second order horizontal gradient, second order VG (vertical gradient), is respectively f ₁=[-101], f ₂=f ₁ ^t, f ₃=[10-201], f ₄=f ₃ ^t, after filtering, obtain the characteristic image f of corresponding intermediate resolution image ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm, utilizes mIm, f ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm is as the image data base of low resolution.

Preferably, in described step (5.3), take lIm as benchmark, the image block of sampling 3x3, and the image block of Im up-sampling 9x9 in correspondence, and corresponding mIm, f ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm then distinguishes the image block of corresponding sampling 6x6, M block of sampling respectively; After image block on sampled I m and mIm, the average deducting current block, as the image block in sample, finally obtains paired sample set wherein N=5, wherein X _ifor the 9x9 image block on Im image, for the image block of the 6x6 on mIm image, be respectively f ₁mIm, f ₂mIm, f ₃mIm, and f ₄the image block of the 6x6 on mIm image.

Preferably, in described step (5.6), upgrade solution formula (9)

$\begin{array}{c}\underset{\{D_1^h,D_2^h\},}{\min }\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{||X_i-D_1^h{B}_i^T{D}_2^{h^T}||}_F^2\\ s.t.{||D_1^h(:,r)||}_2^2\≤1,1\≤r\≤K_1;\\ {||D_2^h(:,t)||}_2^2\≤1,1\≤r\≤K_2.\end{array}---\left(9\right)$

Dictionary solution procedure adopt Aries In The Block By Block Relaxation method to solve,

First solve upgrade by formula (10):

Given solve time, first solve

then upgrade by formula (11):

The solution procedure of formula (10) and (11), adopts Lagrange duality method to solve, thus realizes the renewal of dictionary.

Preferably, in described step (5.7), iteration stopping condition is: iterations reaches upper limit num; Or fidelity error reaches ${||X_i-D_1^h{B}_i^T{D}_2^{h^T}||}_F^2+\underset{k=1}{\overset{N}{\Σ}}{||Y_i^{\left(k\right)}-C_1^{\left(k\right)}{B}_i^T{C}_2^{{\left(k\right)}^T}||}_F^2\≤{\mathrm{\ϵ}}_1,$ ε ₁for default error; Or degree of rarefication reaches || B _i|| ₁≤ ε ₂, ε ₂for default error.

The embodiment of this method is described in more detail below.

Understand for the ease of hereinafter formula and the convenient of symbol, first provide the explanation of a little symbolic formula here.Hereinafter black upper case character representing matrix: as matrix X, black lowercase character represents vector, as vector x.And vec (X) ordinary representation matrix X press column weight arrangement after vector form.X ^tthe transposed matrix of representing matrix X. representing matrix set.X _jit is the element of the jth in vector x.X _ijthe element of the i-th row jth row of representing matrix X.The l of vector ₀, l ₁, l _pnorm is defined as respectively, || x|| ₀=#{x _j≠ 0}, || x|| ₁=∑ _j| x _j| with the l of matrix ₀, l ₁, l _pnorm defines respectively || X|| ₀=#{x _ij≠ 0}, || X|| ₁=∑ _ij| x _ij| with symbol represent Kroneker tensor operator.

Based on the super-resolution rebuilding algorithm of two-dimentional sparse representation model

Image super-resolution rebuilding is from fuzzy, original high-definition picture X is recovered in the low-resolution image Y of down-sampling, the process that degrades can be denoted as Y=SHX, wherein H is fuzzy filter, S is down-sampling operator, and the image super-resolution rebuilding model that therefore the present invention proposes based on two-dimentional sparse representation model is as follows:

$\underset{X,\left\{B_i\right\}}{min}\{{||SHX-Y||}_F^2+\mathrm{\λ}\underset{i}{\Σ}{||B_i||}_0+\mathrm{\γ}{||D_1^h{B}_i^T{D}_2^{h^T}-R_{i}X||}_F^2\}---\left(1\right)$

Here the right Section 1 is the overall situation reconstruction bound term between low-resolution image Y and high-definition picture X, and Section 2 and Section 3 are the two-dimentional sparse constraint items in local, namely guarantee each image block can by two-dimensional level dictionary with second vertical dictionary rarefaction representation.Wherein R _ibe used to the operator of i-th image block extracting image X, be corresponding rarefaction representation matrix, wherein λ and γ is used to two parameters balancing fidelity and degree of rarefication.

Introduce the concrete method for solving of the image super-resolution rebuilding algorithm for two-dimentional sparse representation model below.Following two steps are divided to solve.

1. given X, solves a series of sparse coefficient { B _i}

$\underset{\left\{B_i\right\}}{min}\{\mathrm{\λ}\underset{i}{\Σ}{||B_i||}_0+\mathrm{\γ}{||D_1^h{B}_i^T{D}_2^{h^T}-R_{i}X||}_F^2\}---\left(2\right)$

2. by obtain solve final X

$\underset{x}{min}\{{||SHX-Y||}_F^2+\mathrm{\λ}\underset{i}{\Σ}{||D_1^h{B}_i^T{D}_2^{h^T}-R_{i}X||}_F^2\}---\left(3\right)$

First, we discuss how when given X, solve a series of sparse coefficient { B _i.Direct solution objective function (2) is more difficult because in Section 2 X and { B _iall unknown, therefore, we utilize the reconstruction error item of the characteristic image of a series of low resolution to the error term of high-definition picture estimate, here F ^(k)represent a kth Linear feature extraction operator, be used for extracting the corresponding feature of low-resolution image block, and with be respectively kth class low resolution characteristic image F ^(k)the horizontal and vertical dictionary of Y, F ^(k)y comprises a series of characteristic image block F ^(k)y _i.

If but direct solution $\underset{\left\{B_i\right\}}{min}\{\mathrm{\λ}\underset{i}{\Σ}{||B_i||}_0+\underset{k=1}{\overset{N}{\Σ}}{||C_1^{\left(k\right)}{B}_i^T{C}_2^{{\left(k\right)}^T}-F^{\left(k\right)}{Y}_i||}_F^2\},$ And the compatibility between adjacent image block can be ensured.Therefore, by introducing one ensure the compatibility between adjacent image block, so { B can be solved as follows _iexpression formula:

$\underset{B_i}{min}\{{||D_1^h{B}_i^T{D}_2^{h^T}-{\tilde{X}}_i||}_F^2+\mathrm{\λ}{||B_i||}_1+\underset{k=1}{\overset{N}{\Σ}}{||C_1^{\left(k\right)}{B}_i^T{C}_2^{{\left(k\right)}^T}-F^{\left(k\right)}{Y}_i||}_F^2\}---\left(4\right)$

Here, represent the value of the high-definition picture that repeat region has been rebuild.λ is used to the parameter balancing fidelity and degree of rarefication.L in addition in formula (2) ₀norm has been replaced by l ₁norm, thus make original non-convex problem be converted into convex problem, thus be convenient to solve.

When given high resolving power dictionary with a series of low-resolution dictionary time, for solving of (4), (4) can be converted into following problem

$\underset{vec\left(B_i^T\right)}{min}\{{||vec\left(Z_i\right)-Dvec\left(B_i^T\right)||}_F^2+\mathrm{\λ}{||vec\left(B_i^T\right)||}_1\}---\left(5\right)$

Wherein dictionary D and vec (Z _i) respectively by $D=\left[\begin{array}{c}{D}_2^h\⊗D_1^h\\ C_2^{\left(1\right)}\⊗C_1^{\left(1\right)}\\ .\\ .\\ .\\ C_2^{\left(k\right)}\⊗C_1^{\left(k\right)}\\ .\\ .\\ .\\ C_2^{\left(N\right)}\⊗C_1^{\left(N\right)}\end{array}\right],\mathrm{vec}\left(Z_i\right)=\left[\begin{array}{c}\mathrm{vec}\left({\tilde{X}}_i\right)\\ \mathrm{vec}\left(F^{\left(1\right)}{Y}_i\right)\\ .\\ .\\ .\\ \mathrm{vec}\left(F^{\left(k\right)}{Y}_i\right)\\ .\\ .\\ .\\ \mathrm{vec}\left(F^{\left(N\right)}{Y}_i\right)\end{array}\right]$ Solve.(5) that obtain are classical Lasso problems, and (feature-signsearch) method of can being searched for by characteristic symbol is solved.

B is obtained when solving _iafter, high-definition picture X can be rebuild ₀, wherein rebuild image X ₀each image block meet $R_i{X}_0=D_1^h{B}_i^T{D}_2^{h^T}.$

In above-mentioned solution procedure, only consider the two-dimentional rarefaction representation constraint of local, next, solve objective function (3), not only consider local sparse constraint, consider the global restriction of image simultaneously, therefore, objective function (3) can be converted into following problem

$\underset{x}{min}\{{||SHX-Y||}_F^2+c{||X-X_0||}_F^2\}---\left(6\right)$

Wherein c is used to the parameter balancing global restriction and local restriction, and this problem can directly utilize gradient descent method to solve, thus obtains the final high-definition picture X rebuild ^*.

The dictionary training algorithm of super-resolution rebuilding

In above-mentioned discussion process, give high-low resolution dictionary given when carry out the algorithm of Image Super-resolution Reconstruction.Next, the present invention provides and how to train from high-low resolution image pair the planar dictionary obtaining high-low resolution image.

When given paired sample set wherein the set of high-definition picture block, the set of low-resolution image (feature) block, wherein target of the present invention trains the high resolving power dictionary obtained with a series of low-resolution dictionary high-definition picture block and low-resolution image (feature) block can be made to share identical sparse coefficient.Wherein training sample is concentrated and is comprised M high-definition picture block with N × M low resolution image block here M ₁=d × M, M ₂=b × M.And there is N class low-resolution image (feature).Therefore the model of dictionary is trained can be defined as following form:

$\begin{array}{c}\underset{{\left\{B_i\right\}}_{i=1}^M,\{D_1^h,D_2^h\},{\{C_1^{\left(k\right)},C_2^{\left(k\right)}\}}_{k=1}^N,}{\min }\underset{i=1}{\overset{M}{\mathrm{\Σ}}}\left\{{||X_i-D_1^h{B}_i^T{D}_2^{h^T}||}_F^2+\underset{k=1}{\overset{M}{\mathrm{\Σ}}}{||Y_i^{\left(k\right)}-C_1^{\left(k\right)}{B}_i^T{C}_2^{{\left(k\right)}^T}||}_F^2+\mathrm{\λ}{||B_i||}_1\right\}\\ s.t.{||D_1^h(:,r)||}_2^2\≤1,{||C_1^{\left(k\right)}(:,r)||}_2^2\≤1,1\≤r\≤K_1;\\ {||D_2^h(:,t)||}_2^2\≤1,{||C_2^{\left(k\right)}(:,t)||}_2^2\≤1,1\≤r\≤K_2;\end{array}---\left(7\right)$

Wherein sparse coefficient set is m ₃=K ₂× M, wherein i-th high-definition picture block X _iwith low-resolution image (feature) block the sparse coefficient shared.

Objective function (7) is non-convex optimization problem, can be solved, two key link: 1. sparse coding: at given dictionary by two benches Aries In The Block By Block Relaxation method with solve the rarefaction representation of high-low resolution image block 2. dictionary updating: according to obtaining above-mentioned sparse coefficient time, as with renewal dictionary with

The sparse coding stage

Given high-low resolution dictionary with can by solving each high-low resolution image block to X separately _iwith sparse coefficient, thus obtain sparse coefficient set the sparse coefficient B that each high-low resolution image block is right _ican combine to solve by following objective function and obtain:

$\underset{B_i}{min}\{{||X_i-D_1^h{B}_i^T{D}_2^{h^T}||}_F^2+\underset{k=1}{\overset{N}{\Σ}}{||Y_i^{\left(k\right)}-C_1^{\left(k\right)}{B}_i^T{C_2^{{\left(k\right)}^T}}_i||}_F^2+\mathrm{\λ}{||B_i||}_1\}---\left(8\right)$

Be similar to the method for solving of formula (4), one dimension sparse representation model can be translated into and solve.

The dictionary updating stage

Next, discuss according to sparse coefficient obtained above time, upgrade dictionary with because generally the not of uniform size of high-low resolution image block causes, therefore directly high-low resolution image can not be carried out cascade, even and if high-low resolution tile size is consistent, can be cascaded into $\left[\begin{array}{c}{X}_i\\ Y_i^{\left(k\right)}\end{array}\right],$ But also can exist $\left[\begin{array}{c}{X}_i\\ Y_i^{\left(k\right)}\end{array}\right]\≠\left[\begin{array}{c}{D}_1\\ C_1^{\left(k\right)}\end{array}\right]B^T{\left[\begin{array}{c}{D}_2\\ C_2^{\left(k\right)}\end{array}\right]}^T,$ Therefore $\left[\begin{array}{c}{D}_1\\ C_1^{\left(k\right)}\end{array}\right]$ With $\left[\begin{array}{c}{D}_2\\ C_2^{\left(k\right)}\end{array}\right]$ Can not regard as $\left[\begin{array}{c}{X}_i\\ Y_i^{\left(k\right)}\end{array}\right]$ Horizontal dictionary and vertical dictionary.Therefore dictionary can not be upgraded respectively $\left[\begin{array}{c}{D}_1\\ C_1^{\left(k\right)}\end{array}\right]$ With $\left[\begin{array}{c}{D}_2\\ C_2^{\left(k\right)}\end{array}\right],$ But upgrade dictionary respectively with owing to upgrading with method similar, the present invention here with upgrade for example illustrates the concrete solution procedure upgrading dictionary, its objective function is as follows:

$\begin{array}{c}\underset{\{D_1^h,D_2^h\},}{\min }\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{||X_i-D_1^h{B}_i^T{D}_2^{h^T}||}_F^2\\ s.t.{||D_1^h(:,r)||}_2^2\≤1,1\≤r\≤K_1;\\ {||D_2^h(:,t)||}_2^2\≤1,1\≤r\≤K_2.\end{array}---\left(9\right)$

Same dictionary solution procedure adopt Aries In The Block By Block Relaxation method to solve, namely fix solve vice versa.First solve for us therefore upgrade can following equation be passed through:

Similarly, given solve time, first solve then upgrade can by solving following equation:

For problem (10) and (11), can not adopt again during the renewal dictionary mentioned in two-dimentional sparse representation model and adopt svd (svd) to upgrade line by line dictionary.Because the dictionary of the dictionary of high-definition picture and low-resolution image upgrades respectively, if therefore according to upgrading dictionary line by line, the corresponding relation between high-low resolution image block can be destroyed.The thing followed is, the dictionary obtained is difficult to keep high-low resolution to share constraining of sparse coefficient again, and therefore in order to keep the corresponding relation between high-low resolution image block, the present invention will (or ) upgrade as a whole, therefore, use Lagrange duality method to solve quadratically constrained quadratic programming (QCQP) problem (10) and (11) of standard.

Finally, the present invention, to train the dictionary of high-definition picture block, provides the algorithm complex in dictionary updating process, for a given training set there is M high-definition picture block training two-dimensional level dictionary with second vertical dictionary the then sparse coefficient of each piece and the corresponding sparse coefficient in order to obtain same number, then one dimension dictionary wherein c=d ², K=K ₁× K ₂.Then upgrade D with M image block ^halgorithm complex be O (K ²m) (K < < M), upgrades planar dictionary algorithm complex be therefore planar dictionary is upgraded with algorithm complex required is altogether work as K ₁=K ₂=K ^1/2time, be then O (Kc ^1/2m).And one dimension dictionary and the storage space required for planar dictionary are respectively c × K=c × K ₁× K ₂and c ^1/2× (K ₁+ K ₂).No matter obviously from storage space, or from algorithm complex, the dictionary training method that the present invention proposes all is less than one dimension sparse representation model.

In order to validity of the present invention is described, first the present invention obtains the planar dictionary of high-definition picture and the planar dictionary of corresponding low-resolution image (and characteristic image) by training from external image storehouse.And carry out super-resolution rebuilding by training the planar dictionary that obtains to be applied to a few width image, eventually through comparing the subjective quality of Super-resolution Reconstruction image and objective quality so that validity of the present invention to be described.Wherein objective quality is by Y-PSNR (PeakSignaltoNoiseRatio, PSNR) measure, and subjective quality is measured by structural similarity amount (structuralsimilarity (SSIM) indexmeasurement, SSIM).

In order to verify the quality to above image super-resolution rebuilding, mainly by Y-PSNR (PeakSignaltoNoiseRatio, PSNR) tolerance, unit is decibel (dB).Its computing formula is as follows:

$PSNR=10\&CenterDot;{\log }_{10}\left(\frac{{255}^2}{MSE}\right)---\left(11\right)$

Two width sizes are that the square error MSE of the image of m*n is defined as follows:

$MSE=\frac{1}{m\&times;n}\underset{x=0}{\overset{m-1}{\&Sigma;}}\underset{y=0}{\overset{n-1}{\&Sigma;}}{||I(x,y)-J(x,y)||}^2---\left(12\right)$

Wherein I, J represents the image of original not Noise respectively and utilizes sparse coding method to rebuild image, and I (x, y), J (x, y) is for corresponding to position (x, y) pixel value at place, then square error is less, then PSNR is higher, then the denoising effect of the method is higher.

The judgment criteria of subjective quality is in addition: structural similarity amount (structuralsimilarity (SSIM) indexmeasurement, SSIM).It is a kind of evaluation method of structure based distorted image quality, and brightness and contrast is separated by the method from image structure information, and integrated structure information is evaluated picture quality. define:

SSIM(i,j)＝[L(i,j)] ^α·[C(i,j)] ^β[S(i,j)] ^γ

Wherein:

$L(i,j)=\frac{2{\mathrm{\&mu;}}_i{\mathrm{\&mu;}}_j+{\mathrm{\&lambda;}}_1}{{\mathrm{\&mu;}}_i^2+{\mathrm{\&mu;}}_j^2+{\mathrm{\&lambda;}}_1},\text{C}(i,j)=\frac{2{\mathrm{\&sigma;}}_i{\mathrm{\&sigma;}}_j+{\mathrm{\&lambda;}}_2}{{\mathrm{\&sigma;}}_i^2+{\mathrm{\&sigma;}}_j^2+{\mathrm{\&lambda;}}_2},\text{S}(i,j)=\frac{{\mathrm{\&sigma;}}_{ij}+{\mathrm{\&lambda;}}_3}{{\mathrm{\&sigma;}}_i{\mathrm{\&sigma;}}_j+{\mathrm{\&lambda;}}_3}$

L represents brightness (Lightness), with average (μ _i, μ _j) as the estimation of brightness, C represents contrast (Contrast), with standard deviation (σ _i, σ _j) estimation of spending as a comparison, covariance sigma _ijas the tolerance of structure similarity degree.α, beta, gamma is used to adjust brightness, the weight of contrast and structural information, in order to prevent denominator from occurring zero or close to zero, and produces wild effect, and introduces λ ₁, λ ₂, λ ₃.When α=β=γ=1, λ ₃=λ ₂when/2, simplified formula is

$SSIM(i,j)=\frac{(2{\mathrm{\&mu;}}_i{\mathrm{\&mu;}}_j+{\mathrm{\&lambda;}}_1)(2{\mathrm{\&sigma;}}_{ij}+{\mathrm{\&lambda;}}_2)}{({\mathrm{\&mu;}}_i^2+{\mathrm{\&mu;}}_j^2+{\mathrm{\&lambda;}}_1)({\mathrm{\&sigma;}}_i^2+{\mathrm{\&sigma;}}_j^2+{\mathrm{\&lambda;}}_2)}$

SSIM has symmetry, boundedness, uniqueness, and therefore it can reflect the subjective quality of image well, and especially because observed value is more paid close attention to the details of certain regional area in a flash at certain, therefore SSIM more can reflect the quality of image in conjunction with PSNR.When general PSNR is larger, the discrimination of SSIM is less, and namely PSNR is high, and SSIM is also high, and when PSNR is less, SSIM just has good discrimination.

Present invention employs the external image storehouse identical with SRSR method, and amplify the basic reconstruction parameters such as 3 times, then sampled 100000 high-definition picture blocks and low-resolution image (feature) block, dictionary primitive of the present invention is 32.The present invention have selected the single frames low-resolution image of 15 width standard testing images as input, comprises respectively: Flower, Kodim07, Pallon, Kodim19, Barche, Parrots, Donna, Girl, Parthenon, Raccon, Lena, Tulips, Peppers, Voit and Papav.

Main and bicubic interpolation algorithm in the present invention, the SRSR algorithm that the scholars such as Yang propose, and the Scale-up algorithm that Elad proposes compares.Wherein, in order to the validity of dictionary storage space is described, the one dimension dictionary size arranging Yang is the one dimension dictionary size of the scholars such as 225x60, Zeyde is (324*30+111*5), and this dictionary size is 2x39x32.That is the dictionary size that SRSR and Scale-up method uses is 5 times and 4 times of this dictionary respectively.Last in order to better compare the quality of super-resolution rebuilding, only provide the comparative result of PSNR and SSIM of the gray level image of reconstruction here:

The PSNR results contrast of table 1 super-resolution rebuilding image

The SSIM results contrast of table 2 super-resolution rebuilding image

The above; it is only preferred embodiment of the present invention; not any pro forma restriction is done to the present invention, every above embodiment is done according to technical spirit of the present invention any simple modification, equivalent variations and modification, all still belong to the protection domain of technical solution of the present invention.

Claims (7)

1. an image super-resolution rebuilding method, is characterized in that: comprise the following steps:

(1) the RGB image of the low resolution of input is converted into YCbCr image, wherein Y is non-linear luma component, and Cb is blue color difference component, and Cr is red color difference component; Bilinear interpolation is utilized to carry out super-resolution rebuilding to Cb, Cr;

(2) Y-component is designated as lIm, utilizes bilinear interpolation to carry out 2 times of up-samplings, obtain the image mIm of corresponding intermediate resolution, solve the characteristic image of mIm image simultaneously, obtain the characteristic image of different directions and different order;

(3) with the lIm upper left corner for starting point, the image block Y of the 3x3 that samples successively _i, wherein each direction has the repeated region of 1 pixel, calculates the average M of current block;

(4) respectively from the characteristic image of mIm and different directions and different order, the image block of the 6x6 on relevant position is extracted and solve $vec\left(Z_i\right)=\left[\begin{array}{c}vec\left({\tilde{X}}_i\right)\\ vec\left(F^{\left(1\right)}{Y}_i\right)\\ .\\ .\\ .\\ vec\left(F^{\left(k\right)}{Y}_i\right)\\ .\\ .\\ .\\ vec\left(F^{\left(N\right)}{Y}_i\right)\end{array}\right],$ Wherein represent the value of the high-definition picture that repeat region has been rebuild;

(5) dictionary trained is utilized with solve

$D=\left[\begin{array}{c}{D}_2^h\&CircleTimes;D_1^h\\ C_2^{\left(1\right)}\&CircleTimes;C_1^{\left(1\right)}\\ .\\ .\\ .\\ C_2^{\left(k\right)}\&CircleTimes;C_1^{\left(k\right)}\\ .\\ .\\ .\\ C_2^{\left(N\right)}\&CircleTimes;C_1^{\left(N\right)}\end{array}\right];$

(6) sparse coefficient of current block is solved:

$\underset{vec\left(B_i^T\right)}{\min }\{{||vec\left(Z_i\right)-Dvec\left(B_i^T\right)||}_F^2+\mathrm{\&lambda;}{||vec\left(B_i^T\right)||}_1\};$

Rebuild the high-resolution features image block of relevant position

(7) by X _i+ M is as the high-definition picture block of the reconstruction of relevant position;

(8) judge whether all to have carried out sampling and super-resolution rebuilding to low-resolution image, if the low-resolution image block of all 3x3 completes all, then perform step (10), otherwise perform step (3)-(9);

(9) final super-resolution image X is solved according to formula (6) ^*:

$\underset{x}{\min }\left\{{||SHX-Y||}_F^2+c{||X-X_0||}_F^2\right\}---\left(6\right)$

Wherein c is the error that parameter is used for balancing the overall situation and local;

The high-definition picture X that Y-component super-resolution rebuilding is obtained ^*super-resolution rebuilding image combining with utilizing bilinear interpolation to obtain to Cb, Cr, obtains the high-definition picture in YCbCr space; Then color space conversion is carried out to it, be transformed into RGB color space, finally obtain colored super-resolution rebuilding image.

2. image super-resolution rebuilding method according to claim 1, it is characterized in that: the characteristic image obtaining different directions and different order in described step (2) is single order horizontal gradient, single order VG (vertical gradient), second order horizontal gradient, second order VG (vertical gradient), is respectively f ₁=[-101], f ₃=[10-201], the characteristic image f of corresponding intermediate resolution image is obtained after filtering ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm.

3. image super-resolution rebuilding method according to claim 2, is characterized in that: the dictionary of described step (5) comprises step by step following:

(5.1) set up high-resolution image data base, if image itself is coloured image, then first coloured image is converted into gray level image;

(5.2) image data base of low resolution is set up: to the image in outside image library, carry out 3 times of down-samplings, obtain corresponding low-resolution image lIm, then to all low-resolution image lIm, carry out 2 times of up-samplings, obtain the image mIm of intermediate resolution, solve the characteristic image of mIm image simultaneously, obtain the characteristic image of different directions and different order, the characteristic image of corresponding intermediate resolution image is obtained after filtering, utilize mIm, the characteristic image of intermediate resolution image is as the image data base of low resolution;

(5.3) to the image data base of the high-low resolution set up, sampling obtains the training sample set of paired high-definition picture block and low resolution characteristic image block;

(5.4) initialization dictionary with the image block X of Stochastic choice same index _i, with be averaging by row respectively and obtain hx _i, and be averaging by row and obtain vx _i ^t, and by hx _ias dictionary a row primitive, and vx _ias dictionary a row primitive, can construct respectively by this way a row primitive;

(5.5) to dictionary with carry out sparse coding: first utilize with fabric tensor dictionary respectively, thus obtain the dictionary for sparse coding reconstruction $D=\left[\begin{array}{c}{D}_2^h\&CircleTimes;D_1^h\\ C_2^{\left(1\right)}\&CircleTimes;C_1^{\left(1\right)}\\ C_2^{\left(2\right)}\&CircleTimes;C_1^{\left(2\right)}\\ C_2^{\left(3\right)}\&CircleTimes;C_1^{\left(3\right)}\\ C_2^{\left(4\right)}\&CircleTimes;C_1^{\left(4\right)}\\ C_2^{\left(5\right)}\&CircleTimes;C_1^{\left(5\right)}\end{array}\right],$ And operation is normalized, simultaneously to sample set to each column signal of D in every pair of sample, construct new vector $vec\left(z\right)=\left[\begin{array}{c}vec\left(X_i\right)\\ vec\left(Y_i^{\left(1\right)}\right)\\ vec\left(Y_i^{\left(2\right)}\right)\\ vec\left(Y_i^{\left(3\right)}\right)\\ vec\left(Y_i^{\left(4\right)}\right)\\ vec\left(Y_i^{\left(5\right)}\right)\end{array}\right],$ Then formula (5) is utilized to solve i-th sparse coefficient B _iso, obtain all sparse coefficient set successively

$\underset{vec\left(B_i^T\right)}{\min }\{{||vec\left(Z_i\right)-Dvec\left(B_i^T\right)||}_F^2+\mathrm{\&lambda;}{||vec\left(B_i^T\right)||}_1\}---\left(5\right);$

(5.6) to dictionary with carry out dictionary updating: upgrade dictionary respectively with

(5.7) judge whether to reach iteration stopping condition: if do not meet iteration stopping condition, jump to step (5.3); If meet iteration stopping condition, then export dictionary with ${\{C_1^{\left(k\right)},C_2^{\left(k\right)}\}}_{k=1}^N.$

4. image super-resolution rebuilding method according to claim 3, it is characterized in that: in described step (5.2), the characteristic image of different directions and different order comprises single order horizontal gradient, single order VG (vertical gradient), second order horizontal gradient, second order VG (vertical gradient), is respectively f ₁=[-101], f ₃=[10-201], the characteristic image f of corresponding intermediate resolution image is obtained after filtering ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm, utilizes mIm, f ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm is as the image data base of low resolution.

5. image super-resolution rebuilding method according to claim 4, is characterized in that: in described step (5.3), take lIm as benchmark, the image block of sampling 3x3, and the image block of Im up-sampling 9x9 in correspondence, and corresponding mIm, f ₁mIm, f ₂mIm, f ₃mIm, and f ₄mIm then distinguishes the image block of corresponding sampling 6x6, M block of sampling respectively; After image block on sampled I m and mIm, the average deducting current block, as the image block in sample, finally obtains paired sample set wherein N=5, wherein X _ifor the 9x9 image block on Im image, for the image block of the 6x6 on mIm image, be respectively f ₁mIm, f ₂mIm, f ₃mIm, and f ₄the image block of the 6x6 on mIm image.

6. image super-resolution rebuilding method according to claim 5, is characterized in that: in described step (5.6), upgrades solution formula (9)

$\underset{\left\{D_1^h,D_2^h\right\},}{\min }\underset{i=1}{\overset{M}{\mathrm{\&Sigma;}}}{||X_i-D_1^h{B}_i^T{D_2^h}^T||}_F^2$

$s.t.{||D_1^h(:,r)||}_2^2\&le;1,1\&le;r\&le;K_1;---\left(9\right)$

${||D_2^h(:,t)||}_2^2\&le;1,1\&le;t\&le;K_2.$

Dictionary solution procedure adopt Aries In The Block By Block Relaxation method to solve,

First solve upgrade by formula (10):

Given solve time, first solve then upgrade by formula (11):

The solution procedure of formula (10) and (11), adopts Lagrange duality method to solve, thus realizes the renewal of dictionary.

7. image super-resolution rebuilding method according to claim 6, is characterized in that: in described step (5.7), iteration stopping condition is: iterations reaches upper limit num; Or fidelity error reaches ${||X_i-D_1^h{B}_i^T{D}_2^{h^T}||}_F^2+\underset{k=1}{\overset{N}{\&Sigma;}}{||Y_i^{\left(k\right)}-C_1^{\left(k\right)}{B}_i^T{C}_2^{{\left(k\right)}^T}||}_F^2\&le;{\mathrm{\&epsiv;}}_1,$ ε ₁for default error; Or degree of rarefication reaches || B _i|| ₁≤ ε ₂, ε ₂for default error.