CN106127688A - A kind of super-resolution image reconstruction method and system thereof - Google Patents

Abstract

The open a kind of super-resolution image reconstruction method of the present invention, including: picture breakdown step, wherein pass through picture breakdown, input picture is resolved into structure division and texture part, wherein structure division relative smooth, and there is sharp keen edge, and texture part comprises texture and the details of image；Image amplification procedure, is wherein amplified respectively to described structure division and described texture part；And image combining process, the structural images after wherein amplifying and texture image combination, generate final super-resolution image.

Description

A kind of super-resolution image reconstruction method and system thereof

Technical field

The present invention relates to technical field of video image processing, particularly relate to one and divide and non-linear enhancing based on total variance The super resolution ratio reconstruction method of filtering and system thereof.

Background technology

Along with popularizing of digital product, image obtains the main source of information as the mankind, has obtained the most widely Application.Meanwhile, digital image processing techniques have also been obtained and develop rapidly.And the collection of video image is digital image processing system In a crucial step.During digital collection, affected by following factor, image resolution ratio and picture quality Can decline.Sample frequency, lack sampling makes the spectral aliasing of image, degrades because of anamorphic effect.Atmospheric perturbation, decoking, Relative motion between size sensor and image capture device and subject, can cause the fuzzy of image.And at figure In the acquisition of picture, transmission and storing process, also can introduce noise, such as Gaussian noise, image also can be made to degrade.

Therefore, resolution and the quality of image how are improved so that it is become in recent years as close as original image One of study hotspot of image processing field in the world.And along with the development of image processing techniques and computer computation ability not Disconnected lifting, the reconstruction that super-resolution rebuilding technology is low-resolution image of video image provides good solution.It The image of a series of low resolution can be amplified according to a certain percentage, final generation one width or several high-resolution figures Picture, and well keep the structure of artwork.

Existing super resolution ratio reconstruction method is broadly divided into three major types: the first kind is super-resolution technique based on interpolation； Equations of The Second Kind is based on the super-resolution technique rebuild；3rd class is super-resolution technique based on study.

But simple linear interpolation techniques such as bilinearity and bicubic interpolation, calculate simple can produce sawtooth effect, Simultaneously also can fuzzy edge.In order to preferably keep the acutance at edge, the interpolation method much instructed based on edge is carried in succession Go out.Researcher is had to propose to estimate on low-resolution image the covariance of high-definition picture in calendar year 2001, then with this association side Difference carries out interpolation.Separately there is researcher to propose a kind of autoregression model based on piecemeal in 2008, once estimate monoblock pixel. Separately there is researcher to propose the soft decision interpolation technique of a kind of robust in 2012, in the estimation of parameter and pixel, all use and add Power method of least square.Then, these methods all only considered the reconstruction of marginal portion, does not accounts for the reconstruction of texture part.

Based on the super-resolution technique rebuild, it is the inverse process that degrades of analog image, goes to solve an optimization method.Image drops Matter process is, a panel height image in different resolution, after fuzzy, down-sampled obtains low-resolution image.Researcher is separately had to exist The method divided based on image total variance proposed for 2005 is in such method the most representational one.In the method, figure The total variance of picture is allocated as into bound term, being added in optimization method, thus the solution of restricted problem.It is while keeping edge sharpness Artificial effect can be suppressed greatly.Researcher is separately had to propose to estimate high score by the gradient of low-resolution image in 2011 The gradient of resolution image border, the gradient then estimation obtained, as bound term, joins in optimization method.

In recent years, some super resolution ratio reconstruction methods based on study are the most constantly suggested.Separately there is researcher in 2010 A kind of super resolution ratio reconstruction method based on rarefaction representation is proposed.The method proposes, and image block can be by a super complete word Element in allusion quotation represents by the way of linear combination, and wherein, the number of nonzero coefficient can be lacked as far as possible.So, first produce Raw two super complete dictionary set, the image block in the two set is one to one, be respectively low-resolution image and High-definition picture.For the arbitrarily low image in different resolution block of input, low-resolution dictionary is found a kind of rarefaction representation, so In high-resolution dictionary, high-definition picture block is generated afterwards with this group is sparse.Researcher is separately had to propose to use deeply in 2016 The method of degree study rebuilds high-definition picture.Basic skills is, generates many group low resolution and high-definition picture pair, so Afterwards using low-resolution image as the input of convolutional neural networks, using high-definition picture as the output of convolutional neural networks, Training network.For the network trained, using arbitrarily low image in different resolution as input, produce high-definition picture as reconstruction Result.

Existing algorithm is method based on interpolation, and amount of calculation is low, but the weak effect rebuild.Based on the method rebuild, Edge and two parts of texture the most well can not be rebuild simultaneously.In method based on study, computer complexity is high, and Selection for training storehouse also has the strongest dependency.

The present invention provides a kind of and divides and the super resolution ratio reconstruction method of non-linear boostfiltering based on total variance, can be fine The marginal texture to image and texture structure rebuild.First, by picture breakdown, input picture is resolved into structural portion Divide and texture part, wherein structure division relative smooth, and there is sharp keen edge, and texture part comprises the texture of image And details.Then, these two parts are amplified respectively.For structure division, first it is amplified by linear interpolation, then with one Individual non-linear sharpening filter is sharpened, and uses the non-local mean filtering of improvement to carry out post processing.For texture part, Use pulse sharpening wave filter, texture image is strengthened.Finally, the structural images after amplifying and texture image combination, Generate final super-resolution image.

The technical problem to be solved in the present invention is: realize image super-resolution rebuilding, keeps edge and the stricture of vagina of image simultaneously Reason structure, and reduce computational complexity, meet the requirement of real-time.

Summary of the invention

The present invention provides a kind of super-resolution image reconstruction method, wherein, including: picture breakdown step, wherein by figure As decomposing, input picture is resolved into structure division and texture part, wherein structure division relative smooth, and has sharp keen Edge, and texture part comprises texture and the details of image；Image amplification procedure, wherein to described structure division and described texture Part is amplified respectively；And image combining process, the structural images after wherein amplifying and texture image combination, generate Whole super-resolution image.

The super-resolution image reconstruction method of the present invention is preferably, and described image amplification procedure includes: structural images is amplified Sub-step, for described structure division, is first amplified by linear interpolation, then carries out sharp with a non-linear sharpening filter Change, and use the non-local mean filtering of improvement to carry out post processing；Sub-step is amplified, for described texture portion with texture image Point, use pulse sharpening wave filter, texture image is strengthened.

The super-resolution image reconstruction method of the present invention is preferably, in described picture breakdown step, following by solving Minimize equation and carry out picture breakdown:

WhereinBeing the gradient of image u, gradient is the least, and explanatory diagram picture is the most smooth, and λ is Lagrange multiplier, is used for balancing This two-part weight.

The super-resolution image reconstruction method of the present invention is preferably, in described picture breakdown step, in equation (1) λ takes 0.85.

$\underset{u}{\min }\∫\[\▿u\]+\mathrm{\λ}\∫|f-u|---\left(1\right)$

The super-resolution image reconstruction method of the present invention is preferably, and in described structural images amplifies sub-step, carries out base Structural images in impact filtering sharpens, and uses traditional bicubic interpolation to process the structural images of input, it is thus achieved that initially to amplify After structural images Iu⁰, then use shock filter that edge is sharpened operation, the iteration of pixel is grasped by shock filter Make as follows:

${\mathrm{Iu}}^{n+1}={\mathrm{Iu}}^n-sign\left({\mathrm{\ΔIu}}^n\right)\left|\right|\▿{\mathrm{Iu}}^n\left|\right|t\mathrm{......}\left(2\right)$

Wherein, t is iteration step length, and n is iterations, is initially 0, Δ IuⁿWithEnter in the following manner

$\mathrm{\Δ}Iu={\mathrm{Iu}}_{xx}\·{\mathrm{Iu}}_x^2+2\·{\mathrm{Iu}}_{xx}{\mathrm{Iu}}_x{\mathrm{Iu}}_y+{\mathrm{Iu}}_{yy}\·{\mathrm{Iu}}_y^2---\left(3\right)$

Row calculates:

$\▿Iu=\sqrt{{\mathrm{Iu}}_x^2+{\mathrm{Iu}}_y^2}---\left(4\right)$

Wherein, Iu_xAnd Iu_yIt is that image is in first derivative both horizontally and vertically.

The super-resolution image reconstruction method of the present invention is preferably, and described iteration total degree is 50, and iteration step length is 0.1.

The super-resolution image reconstruction method of the present invention is preferably, and also includes: similarity measure step, based on gray-scale intensity With intensity profile, pixel similarity is measured.

The super-resolution image reconstruction method of the present invention is preferably, and the gray-scale intensity difference between two image blocks is by public affairs Formula (5) defines:

$d(m,n)=\left|\right|N(i,j)-N(m,n)|{|}_2^2---\left(5\right)$

Wherein,It it is second normal form operator.For the intensity profile of image block, the method for perception Hash is used to carry out Tolerance.

The super-resolution image reconstruction method of the present invention is preferably, and also includes: revises step, for each pixel, makes It is weighted averagely, interpolation result being modified with pixel in periphery 21 × 21 window, thus obtains revised gray scale Value.

Super-resolution image reconstruction system involved in the present invention, including: picture breakdown module, wherein divided by image Solve, input picture is resolved into structure division and texture part, wherein structure division relative smooth, and there is sharp keen limit Edge, and texture part comprises texture and the details of image；Image amplification module, wherein to described structure division and described texture portion Divide and be amplified respectively；And image combination module, the structural images after wherein amplifying and texture image combination, generate final Super-resolution image.

Accompanying drawing explanation

Fig. 1 is the flow chart of the super-resolution image reconstruction method representing the present invention.

Fig. 2 is to represent the sub-process figure being amplified structure chart.

Fig. 3 is to represent the sub-process figure being amplified texture maps.

Fig. 4 is the result figure representing and carrying out picture breakdown, and wherein Fig. 4 (a) represents original graph, and Fig. 4 (b) represents structure chart, Fig. 4 (c) represents texture maps.

Fig. 5 is to represent the effect contrast figure before and after being modified image, before wherein Fig. 5 (a) represents correcting process The figure of effect, Fig. 5 (b) is the figure of the effect after correcting process.

Fig. 6 is to represent the effect contrast figure before and after rebuilding image, and wherein Fig. 6 (a) is the low resolution before rebuilding Image, Fig. 6 (b) be rebuild after after high resolution graphics.

Detailed description of the invention

In order to make the purpose of the present invention, technical scheme and advantage clearer, below in conjunction with the embodiment of the present invention In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it will be appreciated that described herein Specific embodiment only in order to explain the present invention, is not intended to limit the present invention.Described embodiment is only the present invention one Divide embodiment rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making The all other embodiments obtained under creative work premise, broadly fall into the scope of protection of the invention.

As shown in the flowchart of fig.1, the dividing and the super-resolution rebuilding of non-linear sharp filtering based on total variance of the present invention Method includes: picture breakdown step S1, wherein by picture breakdown, input picture resolves into structure division and texture part, Wherein structure division relative smooth, and there is sharp keen edge, and texture part comprises the texture of image with thin；Image amplifies Step S2, is wherein amplified respectively to described structure division and described texture part；And image combining process S3, wherein will Structural images after amplification and texture image combination, generate final super-resolution image.

In picture breakdown step S1, piece image f is resolved into structure division u, and texture part v.F=u+v.Use base In the method that total variance is divided.Total variance is divided, and refers to the intensity of variation sum of signal, and for two dimensional image, total variance is divided and schemed exactly The gradient sum of picture.The problem of picture breakdown solves by minimizing equation below solving:

$\underset{u}{\min }\∫\[\▿u\]+\mathrm{\λ}\∫|f-u|---\left(1\right)$

WhereinBeing the gradient of image u, gradient is the least, and explanatory diagram picture is the most smooth, and λ is Lagrange multiplier, is used for balancing This two-part weight.λ is the biggest, illustrate u closer to f, the most smooth.λ is the least, and the weight of total variance subitem is just The biggest, illustrate that u is the most smooth.Being found through experiments, when λ takes 0.85, discomposing effect is preferable, and decomposition result is as shown in Figure 4.

In image amplification procedure S2, carrying out structural images based on impact filtering and sharpen, structural images comprises image Edge and smooth, therefore, for the acutance at amplification artwork the to be kept edge of this part.Flow process such as Fig. 2 Shown in figure, in the step s 21, the structural images of input is processed initially with traditional bicubic interpolation, it is thus achieved that after initially amplifying Structural images Iu⁰, then use shock filter that edge is sharpened operation.The shock filter iterative operation to pixel As follows:

${\mathrm{Iu}}^{n+1}={\mathrm{Iu}}^n-sign\left({\mathrm{\ΔIu}}^n\right)\left|\right|\▿{\mathrm{Iu}}^n\left|\right|t\mathrm{......}\left(2\right)$

Wherein, t is iteration step length, and n is iterations, is initiallyWithCarry out in the following manner

$\mathrm{\Δ}Iu={\mathrm{Iu}}_{xx}\·{\mathrm{Iu}}_x^2+2\·{\mathrm{Iu}}_{xx}{\mathrm{Iu}}_x{\mathrm{Iu}}_y+{\mathrm{Iu}}_{yy}\·{\mathrm{Iu}}_y^2---\left(3\right)$

Calculate:

$\▿Iu=\sqrt{{\mathrm{Iu}}_x^2+{\mathrm{Iu}}_y^2}---\left(4\right)$

Wherein, Iu_xAnd Iu_yIt is that image is in first derivative both horizontally and vertically.Test result indicate that, when iteration is the most secondary Number is 50, can obtain and preferably sharpen effect when that iteration step length being 0.1.

It follows that carry out pixel similarity measurement based on gray-scale intensity and intensity profile.The similarity of pixel leads to The similarity crossing image block defines, the most accurate the most more robustness.Assume current pixel point be y (i, j), its periphery N × N pixel composition image block be N (i, j).Assume in image another pixel be y (m, n), the picture of its periphery N × N Vegetarian refreshments composition image block be N (m, n).Pixel y (i, j) and y (m, n) between similarity by the gray scale of respective image block The similarity of intensity and intensity profile is estimated.Because structural images is mainly based on smooth region and marginal area, therefore, add Enter gray scale intensity distributions this, the similarity between two image blocks can be estimated more accurately.Between two image blocks Gray-scale intensity difference is defined by formula (5):

$d(m,n)=\left|\right|N(i,j)-N(m,n)|{|}_2^2---\left(5\right)$

Wherein,It it is second normal form operator.For the intensity profile of image block, the method for perception Hash is used to carry out Tolerance.(i, j) (m, (i, j) with N (m, cryptographic Hash n) n) to be respectively image block N with H to assume binary picture block H.For figure As block N, (i j), first calculates its average gray.Then for N (i, j) in each pixel, if its gray value is big In draw gray value, then at H, (i, j) relevant position is entered as 1, is otherwise entered as 0.In like manner calculate image block N (m, Hash n) Value.Intensity distribution difference between two image blocks is defined by formula (6):

$h(m,n)=\left|\right|H(i,j)-H(m,n)|{|}_1^1---\left(6\right)$

Wherein,It it is first normal form operator.The two class similaritys defined according to formula (5) and formula (6), to pixel Point y (m, n) gives weights, is used for measuring similarity, as shown in formula (7):

$\mathrm{\ω}(m,n)=\frac{1}{Z(i,j)}{e}^{-\frac{d(m,n)}{\mathrm{\σ}1}}{e}^{-\frac{h(m,n)}{\mathrm{\σ}2}}---\left(7\right)$

Wherein, (i, is j) normalization constant to Z, represents the summation of all weights, declining of parameter σ 1 and σ 2 control characteristic equation Deceleration.Difference between image block is the biggest, and the weights giving respective pixel point are the least, otherwise, then weights are the biggest.Here, will Window block size N × N is set to 7 × 7, and σ 1 size takes the variance of 7 × 7 image blocks, and σ 2 takes 0.1.

It follows that be modified interpolation result, above sharpening operation also can introduce one while keeping edge sharpness A little sawtooth, so being modified the result of interpolation in this step.For each pixel, use periphery 21 × 21 window Interior pixel is weighted averagely obtaining revised gray value.In weights use step 3, the similarity between pixel enters Row is estimated.Revise before and revise after Comparative result figure as shown in Figure 5.

It follows that carry out the pulse sharpening filtering kept based on intensity.Above step is mainly and structural images carries out weight Build.In this step, texture image is rebuild.Image after decomposition is divided into structural images and texture image.Wherein tie Composition picture is in marginal portion close to the rectangular signal of phase step type, and therefore, employing impact filtering can well be to the type Signal strengthens, and keeps edge step response, allows again both sides of edges the most smooth simultaneously.And texture image, working as proparea Territory, closer to isolated pulse signal, therefore, processes unsuitable for impact filtering.Therefore, as it is shown on figure 3, in step In rapid S22, use the pulse sharpening filtering kept based on intensity.First, by the texture image of low resolution by linear double Cubic interpolation is amplified to the high-definition picture size specified, then for the texture image pixel under each high-resolution, Search for the gray-scale intensity maximum of pixel in its periphery 11 × 11 window, it is assumed that for G_max.Pixel gray value is by such as lower section Formula updates:

Y=E × | G_max|^1-N×|x|^N×sign(x) (8)

Wherein, x is current pixel value, and E is enhancer, in order to keep one in structure and brightness with low-resolution image Causing, enhancer E calculates in the following way.Fall Current high resolution location of pixels is mapped to low point in the ratio used by amplifying In resolution image pixel positions, in the window of low-resolution image location of pixels periphery 5 × 5, then search for gray-scale intensity maximum Value, obtains G_max1, then take E=max (G_max1/G_max,1).Fig. 6 represents the low-resolution image before and after reconstruction and high resolution graphics Picture.

The super-resolution image reconstruction system of the present invention, including: picture breakdown module, wherein by picture breakdown, by defeated Enter picture breakdown and become structure division and texture part, wherein structure division relative smooth, and there is sharp keen edge, and texture Part comprises texture and the details of image；Image amplification module, wherein enters respectively to described structure division and described texture part Row amplifies；And image combination module, the structural images after wherein amplifying and texture image combination, generate final super-resolution Rate image.

The a kind of of the present invention divides in the super resolution ratio reconstruction method with non-linear sharp filtering and system thereof based on total variance, Picture breakdown is become structural images and texture image, and carries out weight by different methods respectively according to the feature of both images Build.Wherein, for the reconstruction of structural images, first use simple bicubic linear interpolation techniques, be amplified to and target sizes one Causing, then use a nonlinear scale spaces wave filter, being sharpened image of iteration, afterwards, with the non-local mean improved Wave filter, as post processing, makes the structural images after finally amplifying can have sharp keen edge, also can suppress sawtooth effect Produce.For the reconstruction of texture structure, nonlinear pulse sharp filtering is utilized to carry out enhancement process.This method can be simultaneously to figure Edge and the texture of picture are well rebuild, and lay a good foundation for follow-up application, meanwhile, the most relatively low operand, can To meet the requirement of real-time.

The above, the only detailed description of the invention of the present invention, but protection scope of the present invention is not limited thereto, and any Those familiar with the art in the technical scope that the invention discloses, the change that can readily occur in or replacement, all answer Contain within protection scope of the present invention.

Claims (10)

1. a super-resolution image reconstruction method, it is characterised in that

Including:

Picture breakdown step, wherein by picture breakdown, resolves into structure division and texture part, wherein structure by input picture Part relative smooth, and there is sharp keen edge, and texture part comprises texture and the details of image；

Image amplification procedure, is wherein amplified respectively to described structure division and described texture part；And

Image combining process, the structural images after wherein amplifying and texture image combination, generate final super-resolution image.

Super-resolution image reconstruction method the most according to claim 1, it is characterised in that

Include at described image amplification procedure:

Structural images amplify sub-step, for described structure division, be first amplified by linear interpolation, then with one non-linear Sharpening filter is sharpened, and uses the non-local mean filtering of improvement to carry out post processing；With

Texture image amplifies sub-step, for described texture part, uses pulse sharpening wave filter, increases texture image By force.

Super-resolution image reconstruction method the most according to claim 1 and 2, it is characterised in that

In described picture breakdown step, carry out picture breakdown by solving the following equation that minimizes:

$\underset{u}{min}\∫\[\▿u\]+\mathrm{\λ}\∫|f-u|---\left(1\right)$

WhereinBeing the gradient of image u, gradient is the least, and explanatory diagram picture is the most smooth, and λ is Lagrange multiplier, be used for balance this two The weight of part.

Super-resolution image reconstruction method the most according to claim 3, it is characterised in that

In described picture breakdown step, the λ in equation (1) takes 0.85.

Super-resolution image reconstruction method the most according to claim 2, it is characterised in that

In described structural images amplifies sub-step, carry out structural images based on impact filtering and sharpen, use traditional double three The structural images of secondary interpolation processing input, it is thus achieved that structural images Iu after initial amplification⁰, then use shock filter to edge Being sharpened operation, shock filter is as follows to the iterative operation of pixel:

${\mathrm{Iu}}^{n+1}={\mathrm{Iu}}^n-sign\left({\mathrm{\ΔIu}}^n\right)\left|\right|\▿{\mathrm{Iu}}^n\left|\right|t......\left(2\right)$

Wherein, t is iteration step length, and n is iterations, and initial value is 0, Δ IuⁿWithBy with lower section

$\mathrm{\Δ}Iu={\mathrm{Iu}}_{xx}\·{\mathrm{Iu}}_x^2+2\·{\mathrm{Iu}}_{xx}{\mathrm{Iu}}_x{\mathrm{Iu}}_y+{\mathrm{Iu}}_{yy}\·{\mathrm{Iu}}_y^2---\left(3\right)$

Formula calculates:

$\▿Iu=\sqrt{{\mathrm{Iu}}_x^2+{\mathrm{Iu}}_y^2}---\left(4\right)$

Wherein, Iu_xAnd Iu_yIt is that image is in first derivative both horizontally and vertically.

Super-resolution image reconstruction method the most according to claim 5, it is characterised in that

Described iteration total degree is 50, and iteration step length is 0.1.

Super-resolution image reconstruction method the most according to claim 5, it is characterised in that

Also include:

Similarity measure step, measures pixel similarity based on gray-scale intensity and intensity profile.

Super-resolution image reconstruction method the most according to claim 7, it is characterised in that

Gray-scale intensity difference between two image blocks is defined by formula (5):

$d(m,n)=\left|\right|N(i,j)-N(m,n)|{|}_2^2---\left(5\right)$

Wherein,It it is second normal form operator.For the intensity profile of image block, the method for perception Hash is used to measure.

Super-resolution image reconstruction method the most according to claim 8, it is characterised in that

Also include:

Revising step, for each pixel, in using periphery 21 × 21 window, pixel is weighted averagely, to interpolation result It is modified, thus obtains revised gray value.

10. a super-resolution image reconstruction system, it is characterised in that

Including:

Picture breakdown module, wherein by picture breakdown, resolves into structure division and texture part, wherein structure by input picture Part relative smooth, and there is sharp keen edge, and texture part comprises texture and the details of image；

Image amplification module, is wherein amplified respectively to described structure division and described texture part；And

Image combination module, the structural images after wherein amplifying and texture image combination, generate final super-resolution image.