CN104796623B - Splicing video based on pyramid Block-matching and functional optimization goes structural deviation method - Google Patents

Abstract

The invention belongs to Image Information Processing field, provide a kind of splicing video based on pyramid Block-matching and functional optimization and go structural deviation method, on the basis of global registration, for the overlapping region structural deviation problem that parallax in video-splicing causes, pyramid block matching method is utilized to obtain the corresponding relation of two width images at some image blocks of overlapping region, and represent its position deviation with the displacement of two dimension, then minimized by energy functional and discrete displacement is diffused into whole image range, the displacement that last foundation obtains does deformation operation to image, obtain the frame of video of overlapping region non-structure deviation.Meanwhile, consider in the frame that deformation operates in the process of establishing of energy functional expression formula and interframe constraint, the video spliced there will not be abnormal distortion and shake when playing simultaneously.

Description

Splicing video based on pyramid Block-matching and functional optimization goes structural deviation method

Technical field:

The invention belongs to Image Information Processing field, relate to video-splicing method, the splicing video particularly related to based on pyramid Block-matching and functional optimization goes structural deviation method.

Background technology

Image mosaic, as an important research direction of computer vision subject, receives and pays close attention to widely and deep research.Up to the present, existing a series of Theories and methods for different application demand is proposed by scholars, and achieves good experiment effect, and the algorithm of some of them maturation has been integrated in the business software of image processing field.Therefore, can think that image mosaic has been a more ripe research field.And for video-splicing problem, although just handling object is replaced with the video of multichannel video camera sync pulse jamming by image, but the scale of problem and complexity all sharply promote, be limited to the disposal ability of theory basis and hardware device, be also in the starting stage to the research of video-splicing.Existing video-splicing method is the simple expansion of certain image split-joint method on time dimension mostly, and the new features bring video-splicing problem and constraint lack to be discussed fully, causes splicing effect not good.

Because the photocentre of each video camera not exclusively overlaps, there is certain dislocation and distortion in the imaging of Same Scene in different cameras shooting, claims this difference to be parallax.The existence of parallax makes each road picture being mapped to same viewing plane by projective transformation not overlap completely, namely there is structural deviation.Therefore, in order to improve the joining quality of each frame video pictures, the impact of structural deviation must be eliminated.

Conventional images joining method utilizes optimum splicing seams technology to address this problem usually, and namely avoiding structural deviation affects serious region, and carries out mixing operation near optimum piece, obtains the good stitching image of quality.But the scene for video-splicing, in most cases, in overlapping region is dynamic, the i.e. optimum piece position of consecutive frame inconsistent, due to the impact of parallax, if the image in same region is always alternately got from different original image, the exception of video pictures can be caused to glimmer.Even if increase space-time restriction, make the position of front and back frame piece relatively stable, when there being moving object to pass from piece both sides, always there is piece only cannot adjust the situation just can avoiding structural deviation by a small margin.To sum up, by the structural deviation problem of avoiding in single image splicing, in video-splicing saliency out, joining quality can be affected.

Summary of the invention:

The structural deviation that the present invention is directed to overlapping region in video-splicing can affect the reality of joining quality, propose a kind of splicing video based on pyramid Block-matching and functional optimization and go structural deviation method, keeping the impact eliminating structural deviation between frame of video under conforming prerequisite, high-quality video-splicing can be completed.Strategy of the present invention is operated by image deformation, attempts to eliminate instead of avoid structural deviation, and concrete grammar is after global registration, and according to the result of overlapping region Block-matching, counterweight projected image carries out local directed complete set.In addition, owing to introducing deformation operation, for guaranteeing that the splicing video exported there will not be abnormal distortion when playing and rocks, the present invention adds interframe constraint when calculating deformation.

First the present invention utilizes pyramid block matching method to obtain the corresponding relation of two width images at some image blocks of overlapping region, and represents its position deviation with the displacement of two dimension.Then minimized by energy functional, solve the displacement obtaining whole image range.The last displacement according to each position carries out deformation operation to image, obtains the frame of video of overlapping region non-structure deviation.

Particularly, the technical solution used in the present invention is:

Splicing video based on pyramid Block-matching and functional optimization goes a structural deviation method, it is characterized in that comprising the following steps:

S1. after extracting re-projection, the lap of two width images carries out pyramid Block-matching, obtains discrete structure corresponding relation and the motion vector of discrete point;

S2., on the discrete displacement vector basis that step S1 obtains, each item constraint of comprehensive video splicing, sets up the energy functional of deformation diffusion, obtains the displacement in whole image range by minimization of energy functional;

S3. the displacement obtained according to step S2 carries out deformation operation to input picture, carries out re-projection and fusion, obtain the stitching image of current each frame of video to the image after deformation.

Further, the concrete grammar of described step S1 comprises:

S11. the image pyramid of two width image laps after re-projection is set up;

With I _s, I _tthe lap of two width images after expression re-projection, first to I _sand I _tcarry out multiple dimensioned convergent-divergent:

$\left\{\begin{array}{cc}{I}_{s,k+1}=g_{\mathrm{\σ}}\⊗I_{s,k}& k=\mathrm{0,1},...L-1\\ I_{s,k+1}(i,j)=I_{s,k+1}^{\′}& (2i-\mathrm{1,2}j-1)\end{array}\right.i=\mathrm{1,2},...H_k,j=\mathrm{1,2},...W_k$

Wherein k is current layer number, and L is convolution operation the total degree performed, and I _{s, 0}=I _s, g _σrepresent that variance is the gaussian kernel function of σ, H _kwith W _kbe respectively the height and the width of kth tomographic image; To I _tdo same operation, set up pyramid I _{t, 0}, I _{t, 1}... I _t,L;

S12. pass through successively Block-matching, obtain the corresponding relation between current viewing plane epigraph block;

Descending with yardstick, namely number of plies k is by the order of L to 0, to one group of correspondence image I of every one deck _s,kwith I _s,koperation block matching algorithm, obtains the matching relationship of current layer, and with the displacement of two dimension represent the position deviation obtained, wherein V _k,x() and V _k,y() is vector component in level and vertical both direction, numbering m=1,2 ... M _kwith n=1,2 ... N _kbe respectively the Position Number of match block vertical and horizontal, M _kwith N _kbe respectively the maximum numbering of kth layer vertical and horizontal; With for as lower one deck (2m-1,2n-1), (2m-1,2n), (2m, 2n-1) and (2m, 2n) initial value of the four pairs of match block displacement search in place, successively searches for, finally obtains the original image i.e. Block-matching displacement of the 0th tomographic image and have:

$\left\{\begin{array}{c}{\hat{x}}_t(m,n)={\hat{x}}_s(m,n)+V_x(m,n)\\ {\hat{y}}_t(m,n)={\hat{y}}_s(m,n)+V_y(m,n)\end{array}\right.$

Wherein, with be respectively the coordinate at each match block center under the re-projection image place viewing plane coordinate system of source images and target image; The result of Block-matching is discrete, the result of Block-matching is represented with following one dimension numbering form:

$\left\{\begin{array}{c}{\hat{x}}_{t,p}={\hat{x}}_{s,p}+V_{x,p}\\ {\hat{y}}_{t,p}={\hat{y}}_{s,p}+V_{y,p}\end{array}\right.p=\mathrm{1,2},...P_m$

Wherein, P _mfor the quantity that match block is right;

S13. under each match block center being transformed to original image coordinate system to be deformed according to known global mapping relation, and displacement discrete under calculating original image coordinate;

For obtaining I _scorresponding original image I _{s, org}in the deformation quantity of each pixel, need viewing plane coordinate i is mapped to according to the mapping relations that the global registration stage obtains _{s, org}image coordinate on, obtain corresponding coordinates of original image coordinates (x _s,p, y _s,p) and (x _t,p, y _t,p), now, I _{s, org}middle target location (x _t,p, y _t,p) corresponding displacement can be obtained by following formula:

$\left\{\begin{array}{c}{U}_{x,p}=x_{t,p}-x_{s,p}\\ U_{y,p}=y_{t,p}-y_{s,p}\end{array}\right..$

Further, the concrete grammar of described step S2 comprises:

S21. combined block mates in frame that deformation action need in the discrete prior information and video-splicing obtained meets and interframe retrain, and setting up need minimized energy functional expression formula;

Deformation operation is carried out to input picture, needs to obtain I _{s, org}displacement function in full figure scope Ω wherein the separate and computational process of two components consistent, unifiedly represent one of them with scalar function U at this; Three constraints below function U demand fulfillment:

(1) U is at (x _t,p, y _t,p) place value should with U _pclose as much as possible;

(2) in image, the U value of adjacent position should be close as much as possible;

(3) the U value of consecutive frame should be close as much as possible;

Wherein retrain (1) and represent the discrete displacement prior information obtained by Block-matching in step S1, the continuity constraint that constraint (2) is frame intrinsic displacement amount, the continuity constraint that constraint (3) is interframe displacement;

The energy functional component of constraint (1) correspondence is:

$E_D=\underset{\mathrm{\Ω}}{\∫\∫}{(U-\tilde{U})}^2\mathrm{dxdy}$

Wherein, for be defined on Ω by the reference displacement function that gridding interpolation obtains;

The energy functional component of constraint (2) correspondence is:

$E_S=\underset{\mathrm{\Ω}}{\∫\∫}{|\▿U|}^2\mathrm{dxdy}=\underset{\mathrm{\Ω}}{\∫\∫}({\left(\frac{\mathrm{dU}}{\mathrm{dx}}\right)}^2+{\left(\frac{\mathrm{dU}}{\mathrm{dy}}\right)}^2)\mathrm{dxdy}$

The energy functional component of constraint (3) correspondence is:

$E_C=\underset{\mathrm{\Ω}}{\∫\∫}{(U-U_{\mathrm{pre}})}^2\mathrm{dxdy}$

Comprehensively above-mentioned three constraints, need minimized total energy functional to be:

$\begin{array}{c}E=E_D+{\mathrm{\λ}}_1{E}_S+{\mathrm{\λ}}_2{E}_C\\ =\underset{\mathrm{\Ω}}{\∫\∫}({(U-\tilde{U})}^2+{\mathrm{\λ}}_1{|\▿U|}^2+{\mathrm{\λ}}_2{(U-U_{\mathrm{pre}})}^2)\mathrm{dxdy}\end{array}$

S22. according to variation principle, energy functional minimization problem is converted to solving of partial differential equation;

According to variation principle, according to Euler-Lagrange (Euler-Language) formula, can be the Solve problems of following partial differential equation by the problem equivalent minimizing E:

Above-mentioned partial differential equation are ellipse partial differential equation of second order;

S23. the partial differential equation that difference method numerical solution obtains in step S22 are utilized

The partial differential equation obtained in step S22 are conitnuous forms, carry out discretization, and solve numerical solution to it;

Used first-order difference form is as follows:

$\frac{\∂U}{\∂x}=\frac{U(x+h,...)-U(x,...)}{h}+O\left(h\right)$

Finite Difference Scheme of Second Order is:

$\frac{{\∂}^{2}U}{{\∂x}^2}=\frac{U(x+h,...)-2U(x,...)+U(x-h,...)}{2h}+O\left(h^2\right)$

Especially, the difference scheme of Laplacian Δ is:

$\mathrm{\ΔU}=\frac{U(x+h,y)+U(x-h,y)+U(x,y+h)+U(x,y-h)-4U(x,y)}{2h}+O\left(h^2\right)$

Wherein, difference interval h gets the width of a pixel, h=1 in image coordinate system, by substituting into the difference scheme of above-mentioned differential operator, the partial differential equation obtained in S22 is converted to difference equation; This difference equation is linear, and it solves and realizes by overrelaxation alternative manner;

Displacement vector components U is solved respectively according to above-mentioned steps _xand U _ythe partial differential equation obtained in corresponding S22, can obtain original image I _{s, org}the motion vector of each pixel

Further, the concrete grammar of described step S3 comprises:

S31. according to the displacement obtained in step S2, deformation operation is carried out to input picture;

According to I _{s, org}displacement function within the scope of full images can to I _{s, org}carry out deformation operation, image I ' after distortion _{s, org}middle position (x ' _s,p, y ' _s,p) pixel value at place can from I _{s, org}as upper/lower positions value:

$\left\{\begin{array}{c}{x}_{s,p}=x_{s,p}^{\′}-U_{x,p}\\ y_{s,p}=y_{s,p}-U_{y,p}\end{array}\right.$

Usually (x _s,p, y _s,p) be non-integer position, the value of this position needs to do bilinear interpolation by neighbouring pixel and obtains;

S32. demarcated the projective parameter obtained according to video camera itself, the image projection after distortion has been merged to unified viewing plane;

By the image I ' after distortion _{s, org}with target image I _{t, org}according to projecting to same viewing plane, now, the overlapping region of two images is non-structure deviations, merges, namely obtain the stitching image of present frame to two width re-projection image applications Multiscale Fusions or gradient field fusion method; Step S1 to S3 is suitable for each group synchronization frame of video, the video spliced can be obtained.

The method of the invention is on the basis of global registration, for the overlapping region structural deviation problem that parallax in video-splicing causes, utilize pyramid Block-matching and energy functional to minimize and certain deformation operation is carried out to input picture, the non-structure deviation after re-projection to viewing plane again of the image after distortion can be made, finally coordinate re-projection and mixing operation, complete high-quality video-splicing.Meanwhile, consider in the frame that deformation operates in the process of establishing of energy functional expression formula and interframe constraint, the video spliced there will not be abnormal distortion and shake when playing simultaneously.

Accompanying drawing illustrates:

Fig. 1 is overview flow chart of the present invention.

Fig. 2 is the matching result schematic diagram of each layer of pyramid Block-matching.

Embodiment

Below in conjunction with accompanying drawing and example, the specific embodiment of the present invention is described in further detail.

Video-splicing is divided into global registration and two stages of local directed complete set by the present invention.In the global registration stage, extract and the result of mating based on SIFT feature, calculate the projective parameter of each road video camera, accordingly can by each road image projection to unified viewing plane, this one-phase is existing ripe solution now; The present invention is mainly devoted to second stage, the i.e. local directed complete set stage, pyramid block matching method is utilized to obtain the corresponding relation of two width images at some image blocks of overlapping region, and represent its position deviation with the displacement of two dimension, then minimized by energy functional and discrete displacement is diffused into whole image range, the displacement that last foundation obtains does deformation operation to image, and obtain the frame of video of overlapping region non-structure deviation, the overview flow chart in local directed complete set stage as shown in Figure 1.

Splicing video based on pyramid Block-matching and functional optimization provided by the invention goes structural deviation method to be realized by following steps:

S1. demarcated the projective parameter obtained according to video camera itself, two width source images are projected to same viewing plane, and after extracting re-projection, the lap of two width images carries out pyramid Block-matching, obtain discrete match block and corresponding motion vector.

S2. on the discrete displacement vector basis obtained in step sl, with interframe constraint in the frame of comprehensive video splicing, set up the energy functional of deformation diffusion, and minimization of energy functional is converted into the Solve problems of partial differential equation, solve by difference the deformation quantity that partial differential equation obtain in whole image range.

S3. the deformation quantity in the whole image range obtained according to step S2 carries out deformation operation to input picture, then the image after deformation carried out re-projection to unified viewing plane and do to merge, and obtains the stitching image of present frame.

Further, the concrete grammar of described step S1 comprises:

S11. the image pyramid of two width image laps after re-projection is set up.

With I _s, I _tthe lap of two width images after expression re-projection, first to I _sand I _tcarry out multiple dimensioned convergent-divergent:

$\left\{\begin{array}{cc}{I}_{s,k+1}=g_{\mathrm{\σ}}\⊗I_{s,k}& k=\mathrm{0,1},...L-1\\ I_{s,k+1}(i,j)=I_{s,k+1}^{\′}& (2i-\mathrm{1,2}j-1)\end{array}\right.i=\mathrm{1,2},...H_k,j=\mathrm{1,2},...W_k$

Wherein k is current layer number, and L is convolution operation the total degree performed, and I _{s, 0}=I _s, g _σrepresent that variance is the gaussian kernel function of σ, H _kwith W _kbe respectively the height and the width of kth tomographic image.To I _tdo same operation, set up pyramid I _{t, 0}, I _{t, 1}... I _t,L.

S12. pass through successively Block-matching, obtain the corresponding relation between current viewing plane epigraph block.

Descending with yardstick, namely number of plies k is by the order of L to 0, to one group of correspondence image I of every one deck _s,kwith I _s,koperation block matching algorithm, obtains the matching relationship of current layer, and with the displacement of two dimension represent the position deviation obtained, wherein V _k,x() and V _k,y() is vector component in level and vertical both direction, numbering m=1,2 ... M _kwith n=1,2 ... N _kbe respectively the Position Number of match block vertical and horizontal, M _kwith N _kbe respectively the maximum numbering of kth layer vertical and horizontal.With for as lower one deck (2m-1,2n-1), (2m-1,2n), the initial value that (2m, 2n-1) and four pairs, (2m, 2n) place match block displacement are searched for, successively search for, the matching result of each layer as shown in Figure 2.Finally obtain the original image i.e. Block-matching displacement of the 0th tomographic image and have:

$\left\{\begin{array}{c}{\hat{x}}_t(m,n)={\hat{x}}_s(m,n)+V_x(m,n)\\ {\hat{y}}_t(m,n)={\hat{y}}_s(m,n)+V_y(m,n)\end{array}\right.$

Wherein, with be respectively the coordinate at each match block center under the re-projection image place viewing plane coordinate system of source images and target image.The result of Block-matching is discrete, therefore for process is convenient, the result of Block-matching can be represented with following one dimension numbering form:

$\left\{\begin{array}{c}{\hat{x}}_{t,p}={\hat{x}}_{s,p}+V_{x,p}\\ {\hat{y}}_{t,p}={\hat{y}}_{s,p}+V_{y,p}\end{array}\right.p=\mathrm{1,2},...P_m$

Wherein, P _mfor the quantity that match block is right.

S13. under each match block center being transformed to original image coordinate system to be deformed according to known global mapping relation, and deformation quantity discrete under calculating original image coordinate.

For obtaining I _scorresponding original image I _{s, org}in the deformation quantity of each pixel, need viewing plane coordinate with i is mapped to according to the mapping relations that the global registration stage obtains _{s, org}image coordinate on, obtain corresponding coordinates of original image coordinates (x _s,p, y _s,p) and (x _t,p, y _t,p), now, I _{s, org}middle target location (x _t,p, y _t,p) corresponding displacement can be obtained by following formula:

$\left\{\begin{array}{c}{U}_{x,p}=x_{t,p}-x_{s,p}\\ U_{y,p}=y_{t,p}-y_{s,p}\end{array}\right.$

Further, the concrete grammar of described step S2 comprises:

S21. combined block mates in frame that deformation action need in the discrete prior information and video-splicing obtained meets and interframe retrain, and setting up need minimized energy functional expression formula.

Deformation operation is carried out to input picture, needs to obtain I _{s, org}displacement function in full figure scope Ω in the present invention the separate and computational process of two components consistent, therefore for simplifying statement, unifiedly represent one of them with scalar function U.Meanwhile, right and the two-dimension displacement vector of follow-up appearance does similar simplification.Three constraints below function U demand fulfillment:

(1) U is at (x _t,p, y _t,p) place value should with U _iclose as much as possible;

(2) in image, the U value of adjacent position should be close as much as possible;

(3) the U value of consecutive frame should be close as much as possible.

Wherein retrain (1) and represent the discrete displacement prior information obtained by Block-matching in step S1, the continuity constraint that constraint (2) is deformation quantity in frame, the continuity constraint that constraint (3) is interframe deformation quantity.

The energy functional component of constraint (1) correspondence is:

$E_D=\underset{\mathrm{\Ω}}{\∫\∫}{(U-\tilde{U})}^2\mathrm{dxdy}$

Wherein, for be defined on Ω by the reference displacement function that gridding interpolation obtains.

The energy functional component of constraint (2) correspondence is:

$E_S=\underset{\mathrm{\Ω}}{\∫\∫}{|\▿U|}^2\mathrm{dxdy}=\underset{\mathrm{\Ω}}{\∫\∫}({\left(\frac{\mathrm{dU}}{\mathrm{dx}}\right)}^2+{\left(\frac{\mathrm{dU}}{\mathrm{dy}}\right)}^2)\mathrm{dxdy}$

The energy functional component of constraint (3) correspondence is:

$E_C=\underset{\mathrm{\Ω}}{\∫\∫}{(U-U_{\mathrm{pre}})}^2\mathrm{dxdy}$

Comprehensively above-mentioned three constraints, need minimized total energy functional to be:

$\begin{array}{c}E=E_D+{\mathrm{\λ}}_1{E}_S+{\mathrm{\λ}}_2{E}_C\\ =\underset{\mathrm{\Ω}}{\∫\∫}({(U-\tilde{U})}^2+{\mathrm{\λ}}_1{|\▿U|}^2+{\mathrm{\λ}}_2{(U-U_{\mathrm{pre}})}^2)\mathrm{dxdy}\end{array}$

S22. according to variation principle, energy functional minimization problem is converted to solving of partial differential equation.

If with the energy functional obtained in optimal method direct iteration solution procedure S21, because unknown function U is defined in image range, by U according to after image resolution ratio discretization, the displacement of each pixel position is the component of the unknown state variable of demand solution.Existing optimal method is when solving large-scale optimization problem like this, and convergence rate cannot meet the requirement of video-splicing to processing speed.Therefore, need to transform problem, to accelerate to solve.

According to variation principle, according to Euler-Language formula, can be the Solve problems of following partial differential equation by the problem equivalent minimizing E:

Above-mentioned partial differential equation are ellipse partial differential equation of second order.

S23. the partial differential equation that difference method numerical solution obtains in step S22 are utilized.

The partial differential equation obtained in step S22 are conitnuous forms, and its analytic solutions normal conditions cannot be tried to achieve, and the present invention carries out discretization when implementing to it, and solves numerical solution.Used first-order difference form is as follows:

$\frac{\∂U}{\∂x}=\frac{U(x+h,...)-U(x,...)}{h}+O\left(h\right)$

Finite Difference Scheme of Second Order is:

$\frac{{\∂}^{2}U}{{\∂x}^2}=\frac{U(x+h,...)-2U(x,...)+U(x-h,...)}{2h}+O\left(h^2\right)$

Especially, the difference scheme of Laplacian Δ is:

$\mathrm{\ΔU}=\frac{U(x+h,y)+U(x-h,y)+U(x,y+h)+U(x,y-h)-4U(x,y)}{2h}+O\left(h^2\right)$

In the present invention, difference interval h gets the width of a pixel, h=1 in image coordinate system, by substituting into the difference scheme of above-mentioned differential operator, the partial differential equation obtained in S22 is converted to difference equation.Solving of difference equation is realized by overrelaxation (SOR, SuccessiveOverRelaxation) alternative manner.

Displacement vector components U is solved respectively according to above-mentioned steps _xand U _ythe partial differential equation obtained in corresponding S22, can obtain original image I _{s, org}the motion vector of each pixel

Further, the concrete grammar of described step S3 comprises:

S31. according to the deformation quantity obtained in step S2, deformation operation is carried out to input picture.

According to I _{s, org}deformation quantity within the scope of full images can to I _{s, org}carry out deformation operation, image I ' after distortion _{s, org}middle position (x ' _s,p, y ' _s,p) pixel value at place can from I _{s, org}as upper/lower positions value:

$\left\{\begin{array}{c}{x}_{s,p}=x_{s,p}^{\′}-U_{x,p}\\ y_{s,p}=y_{s,p}-U_{y,p}\end{array}\right.$

Usually (x _s,p, y _s,p) be non-integer position, the value of this position needs to do bilinear interpolation by neighbouring pixel and obtains.

S32. demarcated the projective parameter obtained according to video camera itself, the image projection after distortion has been merged to unified viewing plane.

By the image I ' after distortion _{s, org}with target image I _{t, org}according to projecting to same viewing plane, now, the overlapping region of two images is non-structure deviations.Two width re-projection image applications Multiscale Fusions or gradient field fusion method are merged, namely obtains the stitching image of present frame.Step S1 to S3 is suitable for each group synchronization frame of video, the video spliced can be obtained.

Claims (3)

1. the splicing video based on pyramid Block-matching and functional optimization goes a structural deviation method, it is characterized in that comprising the following steps:

S1. after extracting re-projection, the lap of two width images carries out pyramid Block-matching, obtains discrete structure corresponding relation and the motion vector of discrete point;

S11. the image pyramid of two width image laps after re-projection is set up;

With I _s, I _tthe lap of two width images after expression re-projection, first to I _sand I _tcarry out multiple dimensioned convergent-divergent:

$\left\{\begin{array}{c}\begin{array}{cc}{I}_{s,k+1}=g_{\mathrm{\σ}}\⊗I_{s,k}& k=0,1,...L-1\end{array}\\ \begin{array}{ccc}{I}_{s,k+1}(i,j)=I_{s,k+1}^{\′}(2i-1,2j-1)& i=1,2,...H_k& j=1,2,...W_k\end{array}\end{array}\right.$

Wherein k is current layer number, and L is convolution operation the total degree performed, and I _{s, 0}=I _s, g _σrepresent that variance is the gaussian kernel function of σ, H _kwith W _kbe respectively the height and the width of kth tomographic image; To I _tdo same operation, set up pyramid I _{t, 0}, I _{t, 1}... I _t,L;

S12. pass through successively Block-matching, obtain the corresponding relation between current viewing plane epigraph block;

Descending with yardstick, namely number of plies k is by the order of L to 0, to one group of correspondence image I of every one deck _s,kwith I _s,koperation block matching algorithm, obtains the matching relationship of current layer, and with the displacement of two dimension represent the position deviation obtained, wherein V _k,x() and V _k,y() is vector component in level and vertical both direction, numbering m=1,2 ... M _kwith n=1,2 ... N _kbe respectively the Position Number of match block vertical and horizontal, M _kwith N _kbe respectively the maximum numbering of kth layer vertical and horizontal; With for as lower one deck (2m-1,2n-1), (2m-1,2n), (2m, 2n-1) and (2m, 2n) initial value of the four pairs of match block displacement search in place, successively searches for, finally obtains the original image i.e. Block-matching displacement of the 0th tomographic image and have:

$\left\{\begin{array}{c}{\hat{x}}_t(m,n)={\hat{x}}_s(m,n)+V_x(m,n)\\ {\hat{y}}_t(m,n)={\hat{y}}_s(m,n)+V_y(m,n)\end{array}\right.$

Wherein, with be respectively the coordinate at each match block center under the re-projection image place viewing plane coordinate system of source images and target image; The result of Block-matching is discrete, the result of Block-matching is represented with following one dimension numbering form:

$\left\{\begin{array}{c}{\hat{x}}_{t,p}={\hat{x}}_{s,p}+V_{x,p}\\ {\hat{y}}_{t,p}={\hat{y}}_{s,p}+V_{y,p}\end{array}\right.,p=1,2,...P_m$

Wherein, P _mfor the quantity that match block is right;

S13. under each match block center being transformed to original image coordinate system to be deformed according to known global mapping relation, and displacement discrete under calculating original image coordinate;

For obtaining I _scorresponding original image I _{s, org}in the deformation quantity of each pixel, need viewing plane coordinate with i is mapped to according to the mapping relations that the global registration stage obtains _{s, org}image coordinate on, obtain corresponding coordinates of original image coordinates (x _s,p, y _s,p) and (x _t,p, y _t,p), now, I _{s, org}middle target location (x _t,p, y _t,p) corresponding displacement can be obtained by following formula:

$\left\{\begin{array}{c}{U}_{x,p}=x_{t,p}-x_{s,p}\\ U_{y,p}=y_{t,p}-y_{s,p}\end{array}\right.$

S2., on the discrete displacement vector basis that step S1 obtains, each item constraint of comprehensive video splicing, sets up the energy functional of deformation diffusion, obtains the displacement in whole image range by minimization of energy functional;

S3. the displacement obtained according to step S2 carries out deformation operation to input picture, carries out re-projection and fusion, obtain the stitching image of current each frame of video to the image after deformation.

2. the splicing video based on pyramid Block-matching and functional optimization according to claim 1 goes structural deviation method, it is characterized in that: the concrete grammar of described step S2 comprises:

S21. combined block mates in frame that deformation action need in the discrete prior information and video-splicing obtained meets and interframe retrain, and setting up need minimized energy functional expression formula;

Deformation operation is carried out to input picture, needs to obtain I _{s, org}displacement function in full figure scope Ω wherein the separate and computational process of two components consistent, unifiedly represent one of them with scalar function U at this; Three constraints below function U demand fulfillment:

(1) U is at (x _t,p, y _t,p) place value should with U _pclose as much as possible;

(2) in image, the U value of adjacent position should be close as much as possible;

(3) the U value of consecutive frame should be close as much as possible;

Wherein retrain (1) and represent the discrete displacement prior information obtained by Block-matching in step S1, the continuity constraint that constraint (2) is frame intrinsic displacement amount, the continuity constraint that constraint (3) is interframe displacement;

The energy functional component of constraint (1) correspondence is:

$E_D=\underset{\mathrm{\Ω}}{\∫\∫}{(U-\tilde{U})}^{2}dxdy$

Wherein, for be defined on Ω by the reference displacement function that gridding interpolation obtains;

The energy functional component of constraint (2) correspondence is:

$E_S=\underset{\mathrm{\Ω}}{\∫\∫}|\▿U{|}^{2}dxdy=\underset{\mathrm{\Ω}}{\∫\∫}({\left(\frac{dU}{dx}\right)}^2+{\left(\frac{dU}{dy}\right)}^2)dxdy$

The energy functional component of constraint (3) correspondence is:

$E_C=\underset{\mathrm{\Ω}}{\∫\∫}{(U-U_{pre})}^{2}dxdy$

Comprehensively above-mentioned three constraints, need minimized total energy functional to be:

$\begin{array}{c}E=E_D+{\mathrm{\λ}}_1{E}_S+{\mathrm{\λ}}_2{E}_C\\ =\underset{\mathrm{\Ω}}{\∫\∫}({(U-\tilde{U})}^2+{\mathrm{\λ}}_1|\▿U{|}^2+{\mathrm{\λ}}_2{(U-U_{pre})}^2)dxdy\end{array}$

S22. according to variation principle, energy functional minimization problem is converted to solving of partial differential equation;

According to variation principle, according to Euler-Language formula, can be the Solve problems of following partial differential equation by the problem equivalent minimizing E:

Above-mentioned partial differential equation are ellipse partial differential equation of second order;

S23. the partial differential equation that difference method numerical solution obtains in step S22 are utilized

The partial differential equation obtained in step S22 are conitnuous forms, carry out discretization, and solve numerical solution to it;

Used first-order difference form is as follows:

$\frac{\∂U}{\∂x}=\frac{U(x+h,...)-U(x,...)}{h}+O\left(h\right)$

Finite Difference Scheme of Second Order is:

$\frac{{\∂}^{2}U}{\∂x^2}=\frac{U(x+h,...)-2U(x,...)+U(x-h,...)}{2h}+O\left(h^2\right)$

Especially, the difference scheme of Laplacian Δ is:

$\mathrm{\Δ}U=\frac{U(x+h,y)+U(x-h,y)+U(x,y+h)+U(x,y-h)-4U(x,y)}{2h}+O\left(h^2\right)$

Wherein, difference interval h gets the width of a pixel, h=1 in image coordinate system, by substituting into the difference scheme of above-mentioned differential operator, the partial differential equation obtained in S22 is converted to difference equation; This difference equation is linear, and it solves and realizes by overrelaxation alternative manner;

Displacement vector components U is solved respectively according to above-mentioned steps _xand U _ythe partial differential equation obtained in corresponding S22, can obtain original image I _{s, org}the motion vector of each pixel

3. the splicing video based on pyramid Block-matching and functional optimization according to claim 2 goes structural deviation method, it is characterized in that: the concrete grammar of described step S3 comprises:

S31. according to the displacement obtained in step S2, deformation operation is carried out to input picture;

According to I _{s, org}displacement function within the scope of full images can to I _{s, org}carry out deformation operation, image I ' after distortion _{s, org}middle position (x ' _s,p, y ' _s,p) pixel value at place can from I _{s, org}as upper/lower positions value:

$\left\{\begin{array}{c}{x}_{s,p}=x_{s,p}^{\′}-U_{x,p}\\ y_{s,p}=y_{s,p}-U_{y,p}\end{array}\right.$

Usually (x _s,p, y _s,p) be non-integer position, the value of this position needs to do bilinear interpolation by neighbouring pixel and obtains;

S32. demarcated the projective parameter obtained according to video camera itself, the image projection after distortion has been merged to unified viewing plane;

By the image I ' after distortion _{s, org}with target image I _{t, org}according to projecting to same viewing plane, now, the overlapping region of two images is non-structure deviations, merges, namely obtain the stitching image of present frame to two width re-projection image applications Multiscale Fusions or gradient field fusion method; Step S1 to S3 is suitable for each group synchronization frame of video, the video spliced can be obtained.