CN106296586A - Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode - Google Patents

Abstract

Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode, the invention belongs to computer vision field, mainly solve image procossing precision problem and propose optimized algorithm, the inventive method step includes multiframe low resolution (the low resolution to input, LR) image sequence enforcement associated movement estimation obtains with the method for super-resolution rebuilding " combining high-resolution (high resolution, HR) image I_joint”；The method for reconstructing implementing the multiframe LR image sequence of input based on improving kernel regression obtains " kernel regression HR image I_kernel”；Use the convex combination framework that the present invention proposes, the region merged is determined according to spatial domain fusion criterion, the frequency content needing to merge is determined according to frequency domain criteria, and overall situation weight determines the weight distribution of the fusion to two kinds of methods, the HR image obtained upper two steps by these three aspect is merged, and obtains the HR image of last reconstruction.

Description

Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode

Technical field

The invention belongs to computer vision field, relate to image procossing and optimum theory problem, be specifically related to a kind of new Super resolution ratio reconstruction method based on the convex combination improving kernel regression and Combined estimator.

Background technology

Along with the intensification of information age, multimedia messages has obtained explosive growth, around people without time without Carve and be not flooded with various digital information (such as image, audio frequency, video etc.).And along with image, video technique wide General application with reach its maturity, people are more and more higher for the requirement of digital picture, video quality.In actual application scenarios (such as the photo of mobile phone shooting, or monitoring video), usually require that compare clearly image or video for digital picture, The post processing of video and analysis, and a concept closely-related with image definition is exactly the resolution of image.Resolution Definition is the ability that imaging device represents the good details of object.So a kind of thinking is directly from imaging device hardware aspect Hands, this method improving resolution has two kinds.A kind of pixel quantity being to increase gathered image, but this method can be brought More sound pollution thus reduce signal noise ratio (snr) of image but also acquisition time can be increased, second method is to increase wafer chi Very little, but expensive cost performance is the highest.Therefore, the technology of the lifting image resolution ratio of a kind of alternative said method occurs in that, I.e. image super-resolution rebuilding technology, this technology relatively easily realizes and will not bring and outer expense, moreover it is possible to bring good Good visual effect.This method has a lot of application in computer vision and image processing field, and represents important at needs The occasion of details is particularly important, and such as offer a clear explanation from low-quality monitoring or mobile video text or other are little During details.

In existing image super-resolution rebuilding method, it is broadly classified as single frames according to the quantity of input low-resolution image Super-resolution rebuilding (Single Frame Super Resolution, SFSR) and multiframe super-resolution rebuilding (Multi- Frame Super Resolution,MFSR).SFSR technology is to obtain height by estimation from a lot of training set HR images mostly Frequently the high-frequency information that composition is lost in making up input LR image, and MFSR technology is the information acquisition by merging all LR images HR image.The invention belongs to the category of MFSR technology.The performance of MFSR technology depends primarily on the accuracy of action reference variable. MFSR method can be divided into frequency domain method and spatial domain method two class.Frequency domain method is actually the image interpolation problem that solves in frequency domain, its Observing and nursing is shift characteristics based on Fourier transformation, but frequency domain method is only limitted in global translation motion and model, and frequency domain Lack data dependence, it is very difficult to comprise spatial domain priori, therefore the most no longer become research main flow at present.Spatial domain method is in pixel On yardstick, by the conversion of pixel, constraint are improved spatial resolution.Main spatial domain method has non-homogeneous interpolation method, repeatedly Mixed for backprojection algorithm, projections onto convex sets, and method based on Bayesian frame-maximum a posteriori probability method and various method Close.

Three during the last ten years, and MFSR method is conducted in-depth research by Chinese scholars, achieves great successes, including Discussion to priori model, proposes new method for estimating, to the analysis of existing scheme and improvement etc..But, multiframe The reconstruction research of low-resolution image and application there is problems:

(1) needing more precisely effective image registration (i.e. accurate estimation), the reconstruction that elimination registration brings is by mistake Difference；

(2) need more quickly to reconstruct, owing to mostly the method reconstructed based on multiframe is iterative, in real-time also Have much room for improvement, after its interpolation reconstruction, there is the problem that the deviation of pixel overall gray value is bigger and empty；

(3) need more sane robustness, reduce quantization error or noise to rebuilding image contributions.

Summary of the invention

The goal of the invention of the present invention is: for the problem of above-mentioned existence, it is provided that a kind of to noise robustness and can be accurately Estimate the new super resolution ratio reconstruction method based on the convex combination improving kernel regression and Combined estimator of movable information.

A kind of based on the convex combination improving kernel regression and Combined estimator the super resolution ratio reconstruction method that the present invention proposes is Utilize the significant two kinds of methods of Performance comparision in current MFSR method: method based on kernel regression and associated movement estimation and oversubscription The method that resolution is rebuild.Method based on kernel regression is to be difficult to, for kinematic parameter between image sequence frame, the difficult problem accurately estimated Proposing, it need not accurate estimation just can carry out super-resolution rebuilding to multiple image.The method is in super-resolution Reconstruction field has outstanding performance, it is possible to while interpolation reconstruction, it is achieved the denoising of image.Application scenarios due to project It is monitor video, and monitor video exists mostly, and resolution is relatively low and ambiguous problem, so using the kernel regression improved Method carries out super-resolution rebuilding.Associated movement estimation has the biggest with the method for super-resolution rebuilding in terms of precise motion estimation Advantage.This combined estimation method mainly has the process of sequencing to put estimation and super-resolution rebuilding the two Complete together in a framework, optimized the result of estimation by the estimated value of super-resolution rebuilding, it is thus possible to obtain Compared to the higher precision of existing motion estimation algorithm, promote the effect of super-resolution rebuilding the most in turn.Both sides Method respectively has superiority, and is also respectively arranged with shortcoming, for method based on kernel regression, because it does not has motion-estimation step, so working as When needing the objects in images rebuild to have Large Amplitude Motion, it is rebuild precision and can decline.And for combined estimation method For, it is to have that interactive estimation and super-resolution rebuilding the two process can mutually bring the information of another process Limit, so it is not unlimited to the lifting of image super-resolution rebuilding effect, namely for rebuilding the lifting of effect It is not the most notable.

For problem above, the present invention proposes the framework of a kind of convex combination, can effectively merge both approaches, mutually Make up respective inferior position, the most mutually promote respective advantage.Object is had significantly to transport in needing the image rebuild Time dynamic, combined estimation method at this time just can provide the advantage in estimation accurately, and rebuilds the preferable core of effect Homing method can also promote the reconstruction effect of associated movement method of estimation further.

The new method that the present invention proposes is embodied as comprising the following steps:

Step 1: the multiframe LR image sequence enforcement associated movement estimation of input is obtained with the method for super-resolution rebuilding " associating high-resolution HR image I_joint”；

Step 2: the multiframe LR image sequence enforcement to input obtains " kernel regression HR based on the method for reconstructing improving kernel regression Image I_kernel”；

Step 3: use the convex combination framework that the present invention proposes, determines the region merged according to spatial domain fusion criterion, according to Frequency domain criteria determines the frequency content needing to merge, and overall situation weight determines the weight distribution of the fusion to two kinds of methods, The HR image obtained upper two steps by these three aspect is merged, and obtains the HR image of last reconstruction；Convex combination mode Can be expressed as:

I_com=(1-A^(r,s))·I_kernel+A^(r,s)·I_joint (1)

A^(r,s)=α V W^(r,s) (2)

Wherein V and W^(r,s)Determine the Mixed Zone, spatial domain to two images and frequency domain hybrid frequency blending constituent respectively, entirely Office's parameter alpha ∈ [0,1] determines associating HR image I_jointWith kernel regression HR image I_kernelMixing is how many.

Main novel point due to the present invention is the proposition of convex combination framework, so for two kinds of basic MFSR methods: Based on improve kernel regression method for reconstructing and associated movement estimation and super resolution ratio reconstruction method be embodied as only do simple introduction Do not do and go into seriously, the determination of convex combination framework is mainly discussed in detail.

The concrete process of multi-frame image super-resolution reconstruction method substantially can be divided into three basic steps: registration, interpolation, Rebuild.According to different algorithms, three basic links can be carried out simultaneously, it is also possible to the most independent realization.Generally, multiframe oversubscription Resolution is rebuild and is registrated by each low-resolution image firstly the need of by motion estimation process, recycling frequency domain or spatial domain weight Build algorithm to be permeated by the low-resolution image registrated panel height image in different resolution.And associated movement estimation and Super-resolution reconstruction The principal character of construction method is exactly that estimation link is carried out with super-resolution rebuilding link simultaneously, two process iterative estimates, Interact, improve motion-estimation precision thus improve reconstruction effect.

Kernel regression is as a kind of non-parametric homing method needing not rely on data modeling itself, due to the spirit that it is stronger Activity and ductility, be widely used in image processing field, such as: image interpolation, denoising, super-resolution rebuilding etc..Based on changing Entering kernel regression method for reconstructing is the further improvement theoretical for classical kernel regression.Classical kernel regression is rebuild and is built upon multinomial On the basis of interpolation reconstruction method, only considered the half-tone information of image.On the basis of classical kernel regression theory, utilize image Partial structurtes feature also been proposed the multiframe super-resolution reconstruction method of a kind of controlled kernel regression.Controlled kernel regression method for reconstructing exists While considering half-tone information, incorporate the structural information of image also by the controlled kernel function of structure, although to a certain extent Improve the visual quality of image, but the calculating of local covariance matrix be to rebuild from classical kernel regression to calculate gradient, Computation complexity is high, and the problem that the deviation that there is pixel overall gray value after interpolation reconstruction is bigger and empty.The present invention uses A kind of based on improve kernel regression multiframe super resolution ratio reconstruction method.It uses local structure tensor to replace covariance matrix, The geometry feature of image is described, it is possible to decrease the computation complexity of kernel regression algorithm.Further, in rebuilding for controlled kernel regression The problem that image pixel overall gray value deviation is big and empty, the geometric distance function correction core letter of structure image neighborhood pixels Number so that the image reconstructed has preferable visual quality.

Convex combination framework set up on above two method, the image utilizing kernel regression method to rebuild have preferable details and The advantage that combined estimation method precise motion is estimated, the fusion of convex combination mode in addition.Make up mutually the bad of two kinds of methods existence Gesture: kernel regression method lacks exercise estimation, and combined estimation method reconstructed image quality is relatively poor.Convex combination mode relies primarily on In the reconstruction image I that kernel regression method obtains_kernel, and in place of mixing for needs, the mainly spatial domain that compound mode needs is mixed Close criterion V, frequency-domain frequency composition mixed criteria W^(r,s)And the determination of overall situation mixed criteria α.The purpose of spatial domain mixed criteria V exists In finding out the local needing two width image blend, say, that kernel regression method rebuilds HR image the place of bigger error, also The multiframe LR image i.e. rebuild occurs in that the place of the object of Large Amplitude Motion.Briefly, V needs to lock kernel regression method The appearance certain error region of HR image mixes with integrated processes HR image.Similarly, frequency-domain frequency composition mixed criteria W^(r,s)Purpose be to find out the frequency components needing two width image blend, need exist for using image carried out steerable pyramid Decompose.W^(r,s)Task be lock kernel regression method HR image frequency domain occur the bigger loss of signal primary bands, the most just It is to say that the information that the loss of these band signals needs integrated processes HR image to bring makes up.For overall situation mixed criteria α, After mainly ensureing two width image blend, still there is this feature having of normal picture, say, that make mixed image inclined Minimum from the degree with natural image.

In sum, owing to have employed technique scheme, the invention has the beneficial effects as follows:

(1) present invention proposes the convex combination mode of a kind of innovation, can obtain high-quality SR and rebuild image；

(2) due to the determination of three kinds of principles of convexity compound mode of the present invention, for improving the HR image that kernel regression method is rebuild Bring the information of loss；

(3) due to the present invention, two kinds of MFSR methods of current main flow have been carried out effective mixing so that this SR algorithm for reconstructing Have robustness to interframe Large Amplitude Motion and noise, reconstructed image quality is high.

Accompanying drawing explanation

To illustrate by example and with reference to the appended drawing, wherein:

Fig. 1 is the algorithm flow general introduction of the present invention；

Fig. 2 is the mutual relation schematic diagram of the low-resolution image motion estimation parameter of the present invention；

Fig. 3 is the mathematical model schematic diagram of the kernel regression method of the present invention；

Fig. 4 is the D S obel gradient operator template that the present invention uses；

Fig. 5 is that the steerable pyramid algorithm that the present invention uses is applied to showing of the input picture multiple dimensioned multi-direction frequency band of generation It is intended to (example being 3 yardsticks and 3 direction frequency bands)；

Fig. 6 is the schematic diagram that in the present invention, image block mixes according to convex combination mode.

Detailed description of the invention

All features disclosed in this specification, or disclosed all methods or during step, except mutually exclusive Feature and/or step beyond, all can combine by any way.

Any feature disclosed in this specification (including any accessory claim, summary and accompanying drawing), unless chatted especially State, all can be by other equivalences or there is the alternative features of similar purpose replaced.I.e., unless specifically stated otherwise, each feature is only It it is an example in a series of equivalence or similar characteristics.

With reference to Fig. 1, the present invention to implement process as follows:

Embodiment 1

Process 1, implements combined estimation method and generates high-definition picture I_joint

(1.1) data such as the method for super-resolution rebuilding is firstly the need of obtaining motion estimation parameter, image blurring parameter are Can rebuild.But these parameters owing to obtaining cannot ensure the most accurately, and the capital of error tiny in parameter Process of reconstruction is exaggerated.Therefore, if the build-up effect of parameter error can be tried every possible means to reduce, then super-resolution rebuilding Effect will be improved.So integrated processes to rebuild super-resolution rebuilding framework as follows:

$\hat{X}=\underset{\hat{X}}{\mathrm{argmin}}\{\underset{k=1}{\overset{N}{\Σ}}f(Y_k-H_k\hat{X})+P\left(\hat{X}\right)+Q\left(H_k\right)\}---\left(3\right)$

WhereinIt is the HR image rebuild, H_kIt is degenerate matrix,It is the prior information about unknown HR image, Q (H_k) function adjusts degenerate matrix H_kSo that degenerate matrix is more accurate.

(1.2) combined estimation method thinks it is not completely self-contained between low-resolution image degenerate matrix, by setting up Relation between degenerate matrix is as restrictive condition, thus ensures that degenerate matrix is the most accurate.The process rebuild is divided into two steps: the One step, does not consider the factors such as fuzzy noise, associated movement estimation and data fusion, the information of all low-resolution images is all melted It is combined in piece image；Second step, carries out post processing to the image after merging.Then the model that degrades of image becomes:

Y_k=F_kZ+E_k (4)

Wherein Y_kIt is low-resolution image, F_kIt is motion estimation parameter matrix, E_kIt it is noise.Z represents low-resolution image warp Cross the image after data fusion:

Z=B_kX+σ_k (5)

Wherein B_kIt is fuzzy operator, σ_kIt it is noise.In formula (4), only consider low-resolution image and rebuild between image Movement relation and noise, then set up the model of associated movement estimation and data fusion, solves the fusion image in formula (4) Z_k, solve last high-definition picture X finally according to formula (5).

Associated movement estimation includes two with the model of data fusion, data fit term and degenerate matrix limit entry.Data Fit term employing robust iterative data fitting function:

$Q\left(Z\right)=\underset{k=1}{\overset{N}{\Σ}}(-e^{-a\left|\right|Y_k-F_k{\hat{Z}}^n|{|}^2}+1)---\left(6\right)$

The restrictive condition of degenerate matrix is as follows:

$Q\left(F\right)={\mathrm{\ϵ}}^{\left|k-i\right|}\underset{k=1}{\overset{N}{\Σ}}\left|\right|F_k^{-1}{Y}_k-F_i^{-1}{Y}_i\left|\right|---\left(7\right)$

WhereinIt it is kth frame low-resolution image motion estimation parameter inverse of a matrix matrix.Wherein ε is constant, 0 to 1 Between.Weight in view of different low-resolution frames is distinguishing, this difference and the time gap of low-resolution image Relevant.Adjacent frame correlation is relatively big, and the degenerate matrix therefore from consecutive frame is more credible, so needing to give bigger power Being worth and frame apart from each other, possible picture material changes greatly, and degenerate matrix credibility is slightly smaller.Therefore there is employed herein index letter Number as weight function, when two width low-resolution image time gaps farther out time, give less weights.(7) formula thinks each Width low-resolution image can obtain the image needing to rebuild, different low-resolution images after " reversely " estimation The approximation for reconstruction image after " reversely " estimation should be close.As shown in Figure 2.

(1.3) according to analysis above, the model such as following formula of associated movement estimation and data fusion is set up:

$Q(Z,F_1,F_2,\mathrm{...},F_N)=\underset{k=1}{\overset{N}{\Σ}}(-e^{-a\left|\right|Y_k-F_k{\hat{Z}}^n|{|}^2}+1)+{\mathrm{\λ\ϵ}}^{\left|k-i\right|}\underset{k=1}{\overset{N}{\Σ}}\underset{i=1}{\overset{N}{\Σ}}\left|\right|F_k^{-1}{Y}_k-F_i^{-1}{Y}_i\left|\right|---\left(8\right)$

Then solve the problems referred to above with alternative iteration method, first carry out estimation, obtain F₁,…F_NInitial estimate, so Rear hypothesis F₁,…F_NConstant, withIt is iterated for independent variable:

${\hat{Z}}^{n+1}={\hat{Z}}^n-\mathrm{\ζ}\frac{\∂}{\∂{\hat{Z}}^n}\left(\underset{k=1}{\overset{N}{\Σ}}\right(-e^{-a\left|\right|Y_k-F_k{\hat{Z}}^n|{|}^2}+1)+{\mathrm{\λ\ϵ}}^{\left|k-i\right|}\underset{k=1}{\overset{N}{\Σ}}\underset{i=1}{\overset{N}{\Σ}}||F_k^{-1}{Y}_k-F_i^{-1}{Y}_i|\left|\right)---\left(9\right)$

Wherein, ζ represents the step-length of iteration.According to formulaJudge the no end covered with clouds of whole iterative process.Wherein, η is threshold value.After above-mentioned iteration stopping, the image Z after can being merged, sets up with drag:

$\hat{X}=\arg min\underset{k=1}{\overset{N}{\Σ}}(-e^{-a\left|\right|Z-B\hat{X}|{|}^2}+1)+\mathrm{\β}\underset{m,l=-q}{\overset{q}{\Σ}}{\mathrm{\α}}^{\left|m\right|+\left|l\right|}(\hat{X}-S_x^l{S}_y^m\hat{X})---\left(10\right)$

Using gradient descent method to solve (10) formula, the cut-off condition of iteration is:The X finally obtained It it is i.e. the HR image rebuild.

Process 2, implements the method for reconstructing based on improving kernel regression and generates high-definition picture I_kernel

It is the improvement theoretical to controlled kernel regression based on the method for reconstructing improving kernel regression, and controlled kernel regression theory can Review in classical kernel regression theoretical.Classical kernel regression method, independent of the exponent number N returned, its essence is the local of pixel Weighting procedure, this local linear filtering characteristic is the inherent limitation that traditional core returns.The mathematical model signal of kernel regression method Figure is as shown in Figure 3.The kernel function of classical kernel regression theory contains only the spatial information of sampled point pixel, does not accounts for pixel The impact on weights of the gray value distance.And in real image, usually contained different image texture details, the limit of image The zoness of different such as edge, smooth, angle point reflect the variation characteristic that data are different.Therefore, on the basis of classical kernel regression theory, Propose the controlled kernel regression method of estimation adding image Self-variation.The method not only relies on the space length of sampled point, also Consider the Gray homogeneity between neighborhood sampled point, thus make controlled kernel regression method be provided with nonlinear filtering characteristic. Obtained the partial signal construction of image by the half-tone information robustness analyzing local pixel value, this information incorporated kernel function, The shape of the adaptive change kernel function of energy and size so that the shape of core and the partial structurtes feature being sized to image Join.

But controlled kernel function theory is when calculating the structural information of each pixel, needs the picture interested to each The local neighborhood of vegetarian refreshments is done covariance and is calculated；And the calculating of covariance has the phenomenon of double counting, calculate process more complicated, And it is computationally intensive.Structure tensor, as the powerful of the local geometry analyzing image, has been successfully applied figure The fields such as the estimation of the field of direction of picture, removal picture noise.Structure tensor is one and comprises each element neighborhood side in image To the symmetrical matrix with strength information, it is possible to accurately portray direction and the structural strength of partial gradient, and amount of calculation is little, Therefore the kernel regression method for reconstructing three dimensional structure tensor that improves replaces covariance matrix, not only can the structure letter of picture engraving Breath, distinguishes different structural regions, and can reduce amount of calculation, effectively maintains the detailed information such as edge of image.

(2.1) due in kernel regression method the calculating of directional information need to calculate gradient, so improve kernel regression calculate First method carries out gradient estimation.The operator of rim detection generally utilizes the direction template convolved image of image space to detect figure The vertical edge of picture and horizontal edge, and edge direction is perpendicular to gradient direction, therefore can be carried out by edge detection operator Gradient is estimated.Sobel operator comprises gaussian derivative and smooth characteristic simultaneously, therefore to making an uproar in the presence of low-resolution video Sound has robustness, will not have undesirable effect subsequent result because of the interference of noise, and has and quickly calculate gradient Characteristic.So using 3D-Sobel operator permissible with multiple image sequence frame convolutional calculation Grad in improving kernel regression method The effective time reducing the big moment matrix of calculating.Fig. 4 is used 3D-Sobel operator template.

(2.2) technology on the basis of kernel regression reconstruction is built upon gradient and interpolation, it is thus achieved that after frame sequence gradient, needs The direction of computation structure tensor obtains the interpolation kernel adapting to picture structure, and structure tensor matrix is from partial gradient derivation Coming, the characteristic vector of structure tensor and eigenvalue can describe the energy principal direction of concrete neighborhood of pixel points, at sampling location P The three dimensional structure tensor of point is as follows:

$\begin{array}{c}S\left(P\right)=\▿I\left(P\right)\·\▿I{\left(P\right)}^T\\ =\left[\begin{array}{ccc}{\Σ}_W{I}_x^2\left(P\right)& {\Σ}_W{I}_x\left(P\right)\·I_y\left(P\right)& {\Σ}_W{I}_x\left(P\right)\·I_t\left(P\right)\\ {\Σ}_W{I}_x\left(P\right)\·I_y\left(P\right)& {\Σ}_W{I}_y^2\left(P\right)& {\Σ}_W{I}_x\left(P\right)\·I_t\left(P\right)\\ {\Σ}_W{I}_x\left(P\right)\·I_t\left(P\right)& {\Σ}_W{I}_y\left(P\right)\·I_t\left(P\right)& {\Σ}_W{I}_t^2\left(P\right)\end{array}\right]\end{array}---\left(11\right)$

Wherein,The gradient of representative image, I representing input images, subscript x, y, t represent three gradient directions respectively, and W is Partial analysis window, three dimensional structure tensor S (P) is symmetrical matrix, and after eigenvalue solves, three dimensional structure tensor is expressed as:

S (P)=λ₁e₁e₁ ^T+λ₂e₂e₂ ^T+λ₃e₃e₃ ^T (12)

Wherein, λ₁≥λ₂≥λ₃>=0 is the eigenvalue being arranged in decreasing order of matrix, e₁, e₂, e₃It is that they characteristics of correspondence are vowed Amount, the intensity of they representative image partial structurtes and direction.The kernel function then improved is:

$K(\·)=\frac{\sqrt{\det \left(S_i\right)}}{2{\mathrm{\πh}}^2}\exp \{-\frac{{(X_i-X)}^T{S}_i(X_i-X)}{2h^2}\}---\left(13\right)$

X_iRepresenting noise samples point, X is intended to the sampled point interested estimated, h referred to as overall situation smoothing parameter.

(2.3), in process of reconstruction, image intensity value deviation is excessive, utilizes the pixel interdependence principle of image, by image slices The kernel function of the neighbouring geometric distance function above-mentioned improvement of insertion of element can suppress the generation of this phenomenon, and this function calculates The value gone out can save in advance.Adjacency function is:

$\begin{array}{c}K\left(X_i\right)=\exp \{-\frac{{(X_i-X_i(i,2\left)\right)}^T\left(X_i\right(i,1)-X(i,2\left)\right)}{{\mathrm{\σ}}^2}\}+\mathrm{\α}\\ \mathrm{\α}=\left\{\begin{array}{cc}1& if{X}_i(i,1)=X_i(i,2)\\ 0& \end{array}\right.\end{array}---\left(14\right)$

α is correction factor, the controlled kernel function expression formula that finally improve mutually multiplied with formula (13):

$K_H(\·)=\frac{\sqrt{\det \left(S_i\right)}}{2{\mathrm{\πh}}^2}\exp \{-\frac{{(X_i-X)}^T{S}_i(X_i-X)}{2h^2}\}K\left(X_i\right)---\left(15\right)$

It is final that to solve form as follows:

$z\left(\hat{X}\right)={\widehat{\mathrm{\β}}}_0=e_1^T{\left(X_X^T{W}_X{X}_X\right)}^{-1}{X}_X^T{W}_{X}y---\left(16\right)$

Wherein,Represent the pixel value of point-of-interest to be estimated.Y be each sampled point gray value row to Amount, X_XRepresent sampled point each rank distance matrix, e₁Be header element be 1, remaining is the column vector of 0.W_XFor the controlled kernel function improved K_HThe diagonal matrix of ().

W_X=diag [K_H(X₁-X),K_H(X₂-X),...,K_H(X_P-X)]×diag[K(X₁),K(X₂),...,K(X_P)] (17) (2.4) are analyzed by above, and the key step of the kernel regression multiframe super-resolution rebuilding improved is summarized as follows:

(1) input multiple image sequence

(2) 3D sobel operator is used to calculate the gradient of each interested pixel point；

(3) utilize formula (11) to calculate three-dimensional partial gradient structure tensor S (P), try to achieve three dimensional structure directional information；

(4) formula (14) is utilized to calculate the geometric distance functional value of the neighborhood territory pixel revising kernel function；

(5) solution step (3) and (4) obtained substitutes into formula (15), draws kernel function value K of improvement_H(·)；

(6) kernel function value K that step (5) is drawn_H() substitutes into formula (17) and draws W_X；

(7) W that step (6) is drawn_XSubstitution formula (16), obtains final output image I_kernel。

Process 3, the Frank-Wolfe algorithm implementing the present invention generates high-definition picture I_com

Convex combination framework is set up on above two method, utilizes respective advantage, and the shortcoming making up the other side obtains more High-quality reconstruction effect.With reference to Fig. 4, generate I_comTo implement process as follows:

(3.1) spatial domain mixed criteria V is determined:

Calculating the high-definition picture I that kernel regression method produces_kernelTime, need to calculate sense with 3D sobel operator emerging The gradient of interest pixel, can obtain the gradient of area-of-interest in the multiframe LR image sequence of input in the process, by Grad in image reflects the place of the notable change of image intensity value to a certain extent.If so LR figure to be reconstructed Area-of-interest exists the wing acutely agitated in the object of strenuous exercise, such as penguin picture, then every two field picture in Xiang The Grad of corresponding region is compared other static regions and is had certain deviation.Therefore, kernel regression method is being used to calculate gradient During value, whether the variance of multiple Grad of multiframe LR image same ROI pixel can exist as this region is significantly transported One metric of animal body.Calculate the variance of the corresponding Grad of the LR image ROI region of all frames, and normalize to 0, Between 1, the most just obtain a variance map (variance map) V.The region that variance map intermediate value is higher is corresponding It is the biggest to there is the probability significantly moved in region, and the convex combination framework the most just proposed with greater need for the present invention adds more containing essence The really information of estimation, also further illustrates and proposes the reasonability that variance this concept of map V mixes in spatial domain.Method exists When spatial domain is implemented, choose V-value in variance map and exceed the region of certain threshold value and mix, and other regions keep constant.

(3.2) frequency-domain frequency composition mixed criteria W is determined^(r,s):

Frequency-domain frequency composition mixed criteria W^(r,s)Purpose determine that and need to carry out, at frequency domain, the primary bands that mixes.Due to Will process image at frequency domain, the present invention is to improving the high-resolution rate image I that kernel regression method obtains_kernelUse controlled gold word Tower (steerable pyramid) algorithm decomposes frequency domain, obtains image I_kernelDifferent scale and different directions frequency band point Solve.Steerable pyramid algorithm can be provided by the decomposition of multiple dimensioned multi-direction frequency band image is combined localization (in space or Frequency domain) expression.As shown in Figure 5.Represent direction number of frequency bands r ∈ (0, R) decomposed with R, S represents yardstick quantity s decomposed ∈ (0, S), on different scale S, can obtain the decomposition result of R different directions band, it is simple to we select more accurately The direction frequency band of mixing is needed on different scale.By variance map V obtained in the previous step, choose the district having Large Amplitude Motion to change Territory (assume V > 0.5), implement controllable pyramid decomposition and obtain the decomposition of R direction frequency band of S yardstick, then calculate these image districts Territory average energy under the frequency band of different scale direction, is distributed one according to region energy under the frequency band of different scale direction Important conclusion: signal be lost in that frequency band containing higher-energy occurs the most universal.And these contain the frequency band of higher-energy Preferably corresponding to a certain partial-band in R direction band decomposition under different scale S.Introduce new variableRepresent There is under yardstick S the part frequency band of higher-energy.λ span is between 0 to 1, controls the frequency chosen from total frequency band number R The number of band.By analysis above, for obtaining at the location of pixels x in ROI region:

$W^{(r,s)}\left(x\right)=\left\{\begin{array}{cc}1& ifr\∈B_{\mathrm{\λ}}^{\left(s\right)}\left(x\right)\\ 0& else\end{array}\right.---\left(18\right)$

W in above formula^(r,s)Choose and can effectively make two width image I_kernelAnd I_jointAlong image I_kernelIn need most volume Those frequency bands of external information mix.The mixing schematic diagram in block region is as shown in Figure 6.

(3.3) overall hybrid parameter α is determined:

The Main Function of overall situation hybrid parameter α is to ensure that mixed two width images are the width meeting natural image character Real image.The statistical property of band reduction of fractions to a common denominator solution is analyzed by these needs by natural image: natural image is for bandpass filtering The skirt response of device is height non-gaussian.And the departure degree for Gauss model can be with the peak value (Kurtosis) of response Measure.And the most really have the peak value of investigation display natural image to be kept approximately constant over different frequency bands.Certain point The peak value of cloth is defined as:

$k=\frac{{\mathrm{\μ}}_4}{{\mathrm{\σ}}_4}-3---\left(19\right)$

Wherein μ₄Being the Fourth-order moment of distribution average, σ is the standard deviation of distribution.Defined by this, Gauss distribution Peak value is 0.So choosing of optimum alpha make last mixing HR image I_comPeak value on different frequency bands has minimum Change.I.e. α should meet:

${\mathrm{\α}}^{*}=\underset{0\≤\mathrm{\α}\≤1}{\mathrm{argmin}}\underset{r,s}{\Σ}{\[k_{com}^{(r,s)}\left(\mathrm{\α}\right)-{\stackrel{\‾}{k}}_{new}\left(\mathrm{\α}\right)\]}^2---\left(20\right)$

WhereinIt is last mixing HR image I_comR direction is contained for comprising S yardstick and each yardstick The peak value of the skirt response distribution of the band filter of frequency band.It it is the average of peak value on all frequency bands.Above formula uses The fminsearch function of MATLAB is readily available solution.Finally implement mixing, (2) can be brought into according to the analysis of above two steps Formula obtains A^(r,s)(1) formula of bringing into again obtains last reconstruction HR image I_com。

The invention is not limited in aforesaid detailed description of the invention.The present invention expands to any disclose in this manual New feature or any new combination, and the arbitrary new method that discloses or the step of process or any new combination.

Claims (5)

1. multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode, it is characterised in that include following step Suddenly,

Step 1, to input multiframe LR image sequence implement associated movement estimation combined with the method for super-resolution rebuilding High-resolution HR image I_joint；

Step 2, the method for reconstructing implementing the multiframe LR image sequence inputted based on improving kernel regression obtain kernel regression HR image I_kernel；

Step 3, the region merged according to the decision of spatial domain fusion criterion, determine the frequency content needing to merge according to frequency domain criteria, Determine the weight distribution merged further according to overall situation weight；

Step 4, according to convex combination model, according to the region merged, the frequency content of fusion, fusion weight distribution to associating height Resolution HR image I_jointWith kernel regression HR image I_kernelCarry out merging the HR image obtaining rebuilding.

Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode the most according to claim 1, its Being characterised by, step 1 comprises the following steps,

Step 1.1, the model that degrades according to image, utilize the motion between multiframe LR image sequence and super-resolution rebuilding image Relation and noise, set up the model of associated movement estimation and data fusion, and alternating iteration goes out fusion image；

Step 1.2, utilizing super-resolution rebuilding framework, substitute into fusion image, iteration goes out associating high-resolution HR image I_joint。

Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode the most according to claim 1, its Being characterised by, step 2 comprises the following steps,

Step 2.1, input multiframe LR image sequence

Step 2.2, employing 3D sobel operator calculate the gradient of each interested pixel point；

Step 2.3, three-dimensional partial gradient structure tensor S (P) of calculating, try to achieve three dimensional structure directional information；

Step 2.4, calculate the geometric distance functional value of neighborhood territory pixel revising kernel function；

Step 2.5, solution step 2.3 and step 2.4 obtained substitute into improves kernel function and by its interpolation adjacent to geometric distance letter The product of the adjacency function that number draws, draws kernel function value K () of improvement；

Step 2.6, utilize kernel function value K () build diagonal matrix W_X；

Step 2.7, according to diagonal matrix W_X, sampled point is set, obtains kernel regression HR image I_kernel。

Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode the most according to claim 1, its Being characterised by, step 3 comprises the steps,

Step 3.1, the variance of multiple Grad of corresponding multiframe LR interesting image regions, normalized between 0 to 1 also As the element of variance map, predetermined threshold value, it is judged that more than threshold value all elements V (i, j), so that it is determined that merge district Territory；

Step 3.2, utilize the area-of-interest of steerable pyramid algorithm decomposition step 3.1, find out in each area-of-interest and have There is the main band of higher-energy, thus determine the frequency content needing to merge；

Step 3.3, according to the natural image statistical property to band reduction of fractions to a common denominator solution, the weight distribution of fusion should be chosen so that super-resolution Rebuild image peak value on different frequency bands and there is the change of minimum, thus determine the weight distribution merged.

Multiframe low-resolution image super resolution ratio reconstruction method based on convex combination mode the most according to claim 1, its Being characterised by, step 4, its convex combination model is

I_com=(1-A^(r,s))·I_kernel+A^(r,s)·I_joint

A^(r,s)=α V W^(r,s)

Wherein, V and W^(r,s)Determine correspondence associating high-resolution HR image I respectively_jointWith kernel regression HR image I_kernelSpatial domain mix Closing region and the blending constituent of frequency domain hybrid frequency, global parameter α ∈ [0,1] determines associating HR image I_jointScheme with kernel regression HR As I_kernelMixed proportion.