CN106934769A - Motion blur method is gone based on close shot remote sensing - Google Patents

Abstract

Motion blur method is gone based on close shot remote sensing the present invention relates to a kind of, is comprised the following steps：1st, the streetscape video of selection area is obtained, is then extracted by framing, obtain the sequence blurred picture of streetscape video, then by sequence blurred picture digitized processing, obtain Serial No. blurred picture；2nd, the RGB three-dimensional data matrix of Serial No. blurred picture is obtained, according to the preceding bilateral symmetry feature to light stream and backward light stream, based on Serial No. blurred picture brightness change, data regularization estimation is carried out using the two-way light stream estimation technique to process, and then the Serial No. blurred picture light stream estimated；3rd, the image restored using light stream restored method to the Serial No. blurred picture light stream estimated, removes the edge blurry of objects in images.The present invention is processed by street view image, and the fuzzy field in streetscape is restored using optical flow algorithm, Pyramid technology algorithm and Fourier Cumulate algorithm, can reach the level of streetscape application.

Description

Motion blur method is gone based on close shot remote sensing

Technical field

It is more particularly to a kind of that motion blur is gone based on close shot remote sensing the invention belongs to remote sensing, technical field of image processing Method.

Background technology

The photogrammetric or virtual reality system based on mobile platform is in the ascendant in recent years, and Li Deren etc. takes the lead in proposing simultaneously Realize the vehicle-mounted streetscape acquisition system (Ning Yongqiang of the companies such as vehicle mounted road measurement system (Li Deren 2008), Tengxun, Baidu 2016) it is also gradually ripe and achieve large-scale commercial application, these measurements for being based on mobile platform or virtual reality system Depend on clearly image.But light is good, in the case of excessive velocities, mobile platform image is it is possible that motion mould Paste, removal motion blur turns into the important need of extension mobile platform remote sensing application applicable elements.

With the high speed development of sensor technology, informational geomatics is entered into the new epoch, quickly and conveniently obtain close shot Remote sensing video, has in streetscape application work and is of great significance, but air draught is easily received during IMAQ Influence, the relative motion surveyed between thing of the mechanical shock of photographic platform and photographic platform and ginseng cause the motion mould of image Paste, has had a strong impact on the application in various scenes.Some scholars by using sparse fuzzy core restored image method (Cai Et al propose multiple image deblurring algorithm), using novel energy formula optimization light stream method (Kim and Lee propose Dynamic video deblurring algorithm), using many images Fourier domain conversion accumulation method (Delbracio and Sapiro Propose Fourier domain Cumulate algorithm) and by using method (the cho et al propositions of clearly image restoring blurred picture Use the method for lucky pixel).The above method all have it is certain remove motion blur energy, but exist very big uncertain Property, disturbing factor is more, as a result not accurate enough.

The content of the invention

Motion blur method is gone based on close shot remote sensing it is an object of the invention to provide a kind of, the method will be by that will obscure Streetscape video, sequence blurred picture is extracted as by frame, while blurred picture is digitized, and is pre-processed by spatial domain light stream, is obtained To preliminary fuzzy streetscape deblurring effect, then by Image-matching registration process, obtain the sequence with spatial coherence Row, then carry out image pyramid layered shaping to the image sequence with spatial coherence, then by entering image pyramid Data after reason carry out Fourier accumulation, after processing herein, can obtain the street view image of deblurring.

In order to solve the above technical problems, it is disclosed by the invention it is a kind of motion blur method is gone based on close shot remote sensing, it is special Levy and be, it comprises the following steps：

Step 1：The streetscape video of selection area is obtained, is then extracted by framing, obtain the sequence of the streetscape video Blurred picture, then by sequence blurred picture digitized processing, obtain Serial No. blurred picture；

Step 2：Obtain RGB (color of red, green, blue three) three-dimensional data matrix, Ran Hougen of above-mentioned Serial No. blurred picture According to the preceding bilateral symmetry feature to light stream and backward light stream, based on Serial No. blurred picture brightness change, using double Data regularization estimation is carried out to the light stream estimation technique to process, and then the Serial No. blurred picture light stream estimated；

Step 3：Serial No. blurred picture light stream to estimating obtains preliminary restored image using light stream restored method, and Image segmentation treatment is carried out to preliminary restored image, preliminary restored image is divided into some image blocks, then image block is entered Row registration process, and Pyramid technology treatment is carried out to the image after registration, Fourier then is carried out to image pyramid layering Accumulation obtains final restored image, and the image of the recovery improves image entirety clearly relative to above-mentioned Serial No. blurred picture Clear degree, removes the edge blurry of objects in images.

The present invention is processed by street view image, is accumulated using optical flow algorithm, Pyramid technology algorithm and Fourier and calculated Method is restored to the fuzzy field in streetscape, can reach the level of streetscape application, and the method calculates restoration result more Accurately, more streetscape occasions are adapted to, there is preferable applicability.

Specific embodiment

Below in conjunction with specific embodiment, the present invention is described in further detail：

Of the invention to go motion blur method based on close shot remote sensing, it comprises the following steps：

Step 1：The streetscape video of selection area is obtained, is then extracted by framing, obtain the sequence of the streetscape video Blurred picture, then sequence blurred picture digitized processing is obtained into Serial No. blurred picture by MATLAB softwares；

Step 2：Matlab softwares obtain the RGB three-dimensional data matrix of above-mentioned Serial No. blurred picture, then according to preceding To the bilateral symmetry feature of light stream and backward light stream, based on Serial No. blurred picture brightness change, using bi-directional light The stream estimation technique carries out data regularization estimation treatment, and then the Serial No. blurred picture light stream estimated, by the method The edge blurry of preliminary lifting image entirety definition and removal objects in images；

Step 3：Serial No. blurred picture light stream to estimating obtains preliminary restored image using light stream restored method, and Image segmentation treatment is carried out to preliminary restored image, preliminary restored image is divided into some image blocks, then image block is entered Row registration process, and Pyramid technology treatment is carried out to the image after registration, Fourier then is carried out to image pyramid layering Accumulation obtains final restored image, and the image of the recovery improves image entirety clearly relative to above-mentioned Serial No. blurred picture Clear degree, removes the edge blurry of objects in images.

In the step of above-mentioned technical proposal 2 based on Serial No. blurred picture brightness change, estimated using two-way light stream Meter method carries out data regularization estimation treatment, and the Serial No. blurred picture light stream estimated is answered using light stream in step 3 The specific calculation that former method obtains preliminary restored image is：

E=E_data+E_temporal+E_spatial (1)

Wherein, E is energy theorem, E_dataIt is Serial No. blurred picture energy in itself, E_temporalIt is two width consecutive numbers The energy of word sequence blurred picture conversion, E_spatialThe energy of trend is moved forward and backward for a secondary Serial No. blurred picture；B_iRepresent Fuzzy street view image, τ_iRepresent numerical value of the fuzzy street view image time for exposure divided by acquisition time, generally 1, L_iRepresent multiple Former image, u_i→i+1And u_i→i-1It is two-way light stream, H (L_i,t·u_i→i+1) it is positive restored image L_iEntered deformation matrix H Treatment after obtain L_i+1Image, H (L_i,t·u_i→i-1) it is reverse restored image L_iObtained after the treatment for entering deformation matrix H L_i-1Image, t is blurred picture variation tendency time parameter；Formula 7 is to make the value of its this formula minimum, there is variable in formula, This is equivalent to a constraints；

E_dataIt is a data matrix, λ is the weight distribution value of restored image, and L, u, B represent restored image, light stream respectively And blurred picture,Represent the corresponding linear operator of Toeplitz matrix, K_iIt is fuzzy core；

E_temporalIt is a time data, n is sequence image number, and N is image sequence, μ_nFor a Stationary Parameter is represented The weight that different object views are accounted in blurred picture, L_iX () is the restored image for representing each pixel, L_i+n(x+u_i→i+n) it is recovery Variation tendency of each pixel of image plus light stream side-play amount；

E_spatialIt is a spatial data, in formulaIt is spatial regularization term, has used TV (total Variation) method is optimized,It is the space smoothing to light stream, uses coupling TV methods Optimize,Regularization, g are done in expression to restored image_iX () represents blurred picture edge,Expression is employed The light stream of coupling TV method treatment.

ν represents restored image edge equilibrium, parameter σ_IBlurred picture edge weights are represented,It is on restored image First iteration, exp is exponential function；

By the iteration optimization of formula 7, the preliminary restored image of acquisition is improved in overall definition and removal image The edge blurry of object.

Preliminary restored image is handled as follows in the step 3 obtained the method for final restored map and is：

Hereafter, blurred picture reaches deblurring effect generality by light stream process, in order to improve operational efficiency and expire The demand of sufficient frequency domain accumulation, image segmentation treatment is carried out to preliminary restored image, and preliminary restored image is divided into some images Block；

Image sequence to image block is matched, by frame on the basis of the intermediate frame for setting image block, therewith with image block Other frame figures are matched, and obtain the characteristic point based on reference frame, and image block other frames are realized using the method for following affine transformation Figure is registering with reference frame；

G=L_Reference⊙L_Registration (8)

L'_Reference(x, y)=g (L_Reference(x,y)) (9)

L_RegistrationIt is the intermediate frame of each image block, L_ReferenceIt is the reference frame of each image block, during reference frame is Between other frames beyond frame, g is the affine transformation matrix of reference frame, and what ⊙ symbols referred to carries out locus analysis to the two, obtains Obtain g affine transformations, the transverse and longitudinal coordinate in making (x, y) to represent each image block respectively, L'_ReferenceIt is by affine transformation registration Reference frame, by the image sequence after registration, be provided with stronger spatial coherence.

In the step 3, in order to adapt to more scenes, pyramid is carried out according to the image after 10 pairs of registrations of equation below Layered shaping, i.e. Interpolating transform；

In order to obtain i+1 layers of images after registration, Gaussian kernel convolution is carried out to images after registration i layers first, then will All even number row and column removals, it is possible to obtain i+1 layers of images after registration, result figure is a quarter of artwork；

In formula 10, w (m, n) is the Gaussian convolution core of length 5, g_i(2i-m, 2j-n) is to remove matching somebody with somebody for even number row and column Image after standard, g_i+1(i, j) is the images after registration by reducing.

In the step 3, deblurring is done to the image by Interpolating transform using in the following manner and is processed, first, by right Fourier transform is done by the image sequence after Pyramid technology algorithm, if being by the image sequence after Pyramid technology algorithm {P_i, (i=1 ..., 2M), M is to refer to the number in image sequence, and the image block after registration is { P_i,k, k=1 ..., n, n are Block number after fragmental image processing, sequence image pyramidal layer is { P_i,k,l, (l=1 ..., m), m is represented at Pyramid technology The number of plies after reason, to reach deblurring effect, is obtained by doing Fourier transform to sequence image pyramidal layerAnd be Stabilization Fourier weight is then rightSmoothing processing, obtains Represent by after Fourier transform Data, G_σTo set the Gaussian filter of standard deviation,It is right to representData after smoothing processing；

F^-1Represent inverse fourier transform, the number of the image sequence that M refers to, the norm parameter that P refers to, ω_i,k,lRepresent weight Distribution Value, by the numerical transformation to P values, ω_i,k,lWeight distribution changes, and when P values increase, causes on different segments Weighted value changes, and when P values are 0, as weighted average, in order to obtain more preferable effect, was entered experimental results demonstrate selecting p =11, effect is pretty good, and afterwards, the data after to smoothing processing are accumulated (Fourier accumulation) according to weight distribution ratio, Final restored image is obtained, deblurring effect is reached.The restored image for finally obtaining can reach the ring effect for slowing down border Answer, and obtain the edge of strong and fairing, compensate for the defect that optical flow method causes error due to light conversion, so as to improve frequency The overall definition of domain accumulative.

Contrast and other a few class methods, this method have more preferable effect, and its corresponding numeric reference is table 1：

Tab.1 Compare the GMG/SMD/EVA values for the different methods

Experiment uses image detail Data Comparison, is processed by distinct methods, obtains the de-blurred image of different-effect.By Change excessively sensitive to light in the method for Kim and Lee, cause restored image signboard edge unsmooth effect occur, and Although FBA methods obtain good object edge effect, but image overall noise lays particular stress on, and causes image integrally unintelligible, but Context of methods, by combining the characteristics of the two, weakens image overall noise and preliminary smooth object edge, secondly profit first With the advantage of Fourier domain, so smooth object edge, and also control image overall noise, reach more good Image deblurring effect.

The content that this specification is not described in detail belongs to prior art known to professional and technical personnel in the field.

Claims (5)

1. it is a kind of that motion blur method is gone based on close shot remote sensing, it is characterised in that it comprises the following steps：

Step 1：The streetscape video of selection area is obtained, is then extracted by framing, the sequence for obtaining the streetscape video is obscured Image, then by sequence blurred picture digitized processing, obtain Serial No. blurred picture；

Step 2：The RGB three-dimensional data matrix of above-mentioned Serial No. blurred picture is obtained, then according to preceding to light stream and backward light The bilateral symmetry feature of stream, based on Serial No. blurred picture brightness change, line number is entered using the two-way light stream estimation technique Processed according to regularization estimation, and then the Serial No. blurred picture light stream estimated；

Step 3：Serial No. blurred picture light stream to estimating obtains preliminary restored image using light stream restored method, and to first Step restored image carries out image segmentation treatment, and preliminary restored image is divided into some image blocks, and then image block is matched somebody with somebody Quasi- treatment, and Pyramid technology treatment is carried out to the image after registration, Fourier accumulation then is carried out to image pyramid layering Final restored image is obtained, the image of the recovery improves image entirety clearly relative to above-mentioned Serial No. blurred picture Degree, removes the edge blurry of objects in images.

It is 2. according to claim 1 that motion blur method is gone based on close shot remote sensing, it is characterised in that：In the step 2 Based on Serial No. blurred picture brightness change, carry out data regularization estimation using the two-way light stream estimation technique and process, with And obtain the specific meter of preliminary restored image in step 3 using light stream restored method to the Serial No. blurred picture light stream estimated Calculation mode is：

E=E_data+E_temporal+E_spatial (1)

$B_i=\frac{1}{2{\mathrm{\τ}}_i}\underset{0}{\overset{{\mathrm{\τ}}_i}{\∫}}H(L_i,t\·u_{i\→i+1})+H(L_i,t\·u_{i\→i-1})dt---\left(2\right)$

$E_{data}(L,u,B)=\mathrm{\λ}\underset{i}{\Σ}\underset{{\∂}_{*}}{\Σ}\left|\right|{\∂}_{*}{K}_i({\mathrm{\τ}}_i,u_{i\→i+1},u_{i\→i-1})L_i-{\∂}_{*}{B}_i|{|}^2---\left(3\right)$

$E_{temporal}(L,u)=\underset{i}{\Σ}\underset{n=-N}{\overset{N}{\Σ}}{\mathrm{\μ}}_n|L_i\left(x\right)-L_{i+n}(x+u_{i\→i+n})|---\left(4\right)$

$E_{spatial}(L,u)=\underset{i}{\Σ}|\▿L_i|+\underset{n=-N}{\overset{N}{\Σ}}{g}_i\left(x\right)|\▿u_{i\→i+n}|---\left(5\right)$

$g_i\left(x\right)=v\exp \left({\left(-\frac{|\▿\stackrel{\‾}{L_i}|}{{\mathrm{\σ}}_I}\right)}^2\right)---\left(6\right)$

$\begin{array}{c}\underset{L,u}{\min }\mathrm{\λ}\underset{i}{\Σ}\underset{{\∂}_{*}}{\Σ}\left|\right|{\∂}_{*}{K}_i({\mathrm{\τ}}_i,u_{i\→i+1},u_{i\→i-1})L_i-{\∂}_{*}{B}_i|{|}^2+\\ \underset{i}{\Σ}\underset{n=-N}{\overset{N}{\Σ}}{\mathrm{\μ}}_n|L_i\left(x\right)-L_{i+n}(x+u_{i\→i+n})|+\\ \underset{i}{\Σ}|\▿L_i|+\underset{n=-N}{\overset{N}{\Σ}}{g}_i\left(x\right)|\▿u_{i\→i+n}|\end{array}---\left(7\right)$

Wherein, E is energy theorem, E_dataIt is Serial No. blurred picture energy in itself, E_temporalIt is two adjacent digital sequences The energy of row blurred picture conversion, E_spatialThe energy of trend is moved forward and backward for a secondary Serial No. blurred picture；B_iRepresent fuzzy Street view image, τ_iRepresent numerical value of the fuzzy street view image time for exposure divided by acquisition time, L_iThe image for restoring is represented, u_i→i+1And u_i→i-1It is two-way light stream, H (L_i,t·u_i→i+1) it is positive restored image L_iAfter entering the treatment of deformation matrix H Obtain L_i+1Image, H (L_i,t·u_i→i-1) it is reverse restored image L_iL is obtained after the treatment for entering deformation matrix H_i-1Image, t It is blurred picture variation tendency time parameter；

E_dataIt is a data matrix, λ is the weight distribution value of restored image, and L, u, B represent restored image, light stream and mould respectively Paste image,Represent the corresponding linear operator of Toeplitz matrix, K_iIt is fuzzy core；

E_temporalIt is a time data, n is sequence image number, and N is image sequence, μ_nFor a Stationary Parameter represents fuzzy The weight that different object views are accounted in image, L_iX () is the restored image for representing each pixel, L_i+n(x+u_i→i+n) it is restored image Variation tendency of each pixel plus light stream side-play amount；

E_spatialIt is a spatial data, in formulaIt is spatial regularization term, has used TV methods to optimize,It is the space smoothing to light stream, uses coupling TV methods and optimize,Represent to restoring Image does Regularization, g_iX () represents blurred picture edge,Expression employs the light stream of coupling TV method treatment；

ν represents restored image edge equilibrium, parameter σ_IBlurred picture edge weights are represented,It is on the first of restored image Iteration, exp is exponential function；

By the iteration optimization of formula 7, the preliminary restored image of acquisition improves overall definition and removal objects in images Edge blurry.

It is 3. according to claim 2 that motion blur method is gone based on close shot remote sensing, it is characterised in that：In the step 3 Preliminary restored image is handled as follows and is obtained the method for final restored map and is：

Image segmentation treatment is carried out to preliminary restored image, preliminary restored image is divided into some image blocks；

Image sequence to image block is matched, by frame on the basis of the intermediate frame for setting image block, therewith with image block other Frame figure match, obtain based on reference frame characteristic point, using the method for following affine transformation realize other frame figures of image block with The registration of reference frame；

G=L_Reference⊙L_Registration (8)

L'_Reference(x, y)=g (L_Reference(x,y)) (9)

L_RegistrationIt is the intermediate frame of each image block, L_ReferenceIt is the reference frame of each image block, reference frame is intermediate frame Other frames in addition, g is the affine transformation matrix of reference frame, and what ⊙ symbols referred to carries out locus analysis to the two, obtains g and imitates Conversion is penetrated, the transverse and longitudinal coordinate in making (x, y) to represent each image block respectively, L'_ReferenceIt is the reference by affine transformation registration Frame, by the image sequence after registration, is provided with stronger spatial coherence.

It is 4. according to claim 3 that motion blur method is gone based on close shot remote sensing, it is characterised in that：In the step 3, In order to adapt to more scenes, Pyramid technology treatment, i.e. Interpolating transform are carried out according to the image after 10 pairs of registrations of equation below；

$g_{i+1}(i,j)=\underset{m=-2}{\overset{2}{\Σ}}\underset{n=-2}{\overset{2}{\Σ}}w(m,n)g_i(2i-m,2j-n)---\left(10\right)$

In order to obtain i+1 layers of images after registration, Gaussian kernel convolution is carried out to images after registration i layers first, then will be all Even number row and column is removed, it is possible to obtain i+1 layers of images after registration, and result figure is a quarter of artwork；

In formula 10, w (m, n) is the Gaussian convolution core of length 5, g_i(2i-m, 2j-n) is figure after the registration for removing even number row and column Picture, g_i+1(i, j) is the images after registration by reducing.

It is 5. according to claim 4 that motion blur method is gone based on close shot remote sensing, it is characterised in that：In the step 3, Deblurring is done to the image by Interpolating transform using in the following manner to process, first, by by after Pyramid technology algorithm Image sequence do Fourier transform, if by the image sequence after Pyramid technology algorithm be { P_i, (i=1 ..., 2M), M To refer to the number in image sequence, the image block after registration is { P_i,k, k=1 ..., n, n are the block after fragmental image processing Number, sequence image pyramidal layer is { P_i,k,l, (l=1 ..., m), m represents the number of plies after Pyramid technology treatment, to reach Blur effect, is obtained by doing Fourier transform to sequence image pyramidal layerAnd to stablize Fourier weight, then It is rightSmoothing processing, obtains Represent by the data after Fourier transform, G_σTo set standard The Gaussian filter of deviation,It is right to representData after smoothing processing；

F^-1Represent inverse fourier transform, the number of the image sequence that M refers to, the norm parameter that P refers to, ω_i,k,lRepresent weight distribution Value, by the numerical transformation to P values, ω_i,k,lWeight distribution changes, and when P values increase, causes the weight on different segments Value changes, when P values are 0, as weighted average, and afterwards, the data after to smoothing processing are entered according to weight distribution ratio Row accumulation, obtains final restored image, reaches deblurring effect.