CN105005964A - Video sequence image based method for rapidly generating panorama of geographic scene - Google Patents

Abstract

The present invention discloses a video sequence image based method for rapidly generating a panorama of a geographic scene. The method comprises: 1) performing grid division on video frames to obtain sub-image blocks, and establishing a one-to-one correspondence between each sub-image block of the continuous video frames; 2) performing key point detection and filtration on each sub-image by using a multi-thread parallel strategy, and constructing a to-be-matched point set, and performing matching search between each associated sub-image, and extracting matched point pairs between each sub-image; 3) merging matched point pairs between each sub-image, and thus calculating a geometric splice parameter of each video frame; 4) performing splicing on video sequences according to the spice parameter, to obtain a panorama; and 5) constructing a global rotation matrix for modifying a bending panorama generated due to shooting shake. Beneficial effects of the present invention are: the method is applicable to splice the video sequences rotarily shooted by a conventional video camera on the geographic scene into a panorama image without distortion, and multi-thread parallel processing is used, so that a splicing speed is fast.

Description

Based on the geographic scenes panorama sketch rapid generation of video sequence image

Technical field

The present invention relates to the fields such as geographical information visualization, Digital Image Processing, computer vision, particularly relate to a kind of method generating geographic scenes panorama sketch based on video sequence image fast.

Background technology

Video sequence owing to having time-space attribute, contain much information, resolution is high, express the features such as directly perceived, become fast generalization, the geography information source of socialization and modeling analysis and expression means.Panoramic picture allows the positional information around people's quick obtaining, therefore 360 ° of video sequences around shooting are spliced into the larger panoramic picture of field range, the integrality of geographic scenes can not only be expressed in all directions, make people get more information about the overall content of video sequence, the space-time redundancy of video sequence can be reduced simultaneously.

Some commercial companies develop Related product, and as the Google Street View of Google company exploitation, by using special device, and the mm professional camera special demarcated is to take the multi-angle video in street, and is spliced into panoramic picture; The Ladybug multi-angle video filming apparatus of GreyPoint company exploitation, the same position that adopts is fixed and the mm professional camera special demarcated, the video sequence splicing generating panorama image that the supporting SDK provided of company can will take.But the price of the professional video panorama that these commercial companies develop is too high, and be unfavorable for that 360 ° of video sequences around shooting are spliced into panoramic picture by domestic consumer.

Summary of the invention

The present invention proposes a kind of geographic scenes panorama sketch rapid generation based on video sequence image, the method adopts multi-threaded parallel mode to carry out graticule mesh piecemeal to video sequence, set up the local searching strategy of key point under piecemeal to control match search region, in the matching process, match search efficiency is improved by the number reducing key point, thus realize the rapid registering of video sequence and the seamless spliced of panoramic picture, decrease the time of video sequence splicing, improve splicing efficiency, especially panoramic picture is spliced into high-resolution video sequence and has good effect, the video sequence that can be adapted to domestic consumer's shooting is spliced into panoramic picture.

To achieve these goals, the present invention adopts following technical scheme:

Based on the geographic scenes panorama sketch rapid generation of video sequence image, comprise the steps:

1) subimage that grid partition becomes different is carried out to video frame images, and set up the one-to-one relationship of each subimage block between successive video frames;

2) adopt piecemeal multi-threaded parallel strategy to carry out critical point detection to each subimage, and the key point detected is filtered, retain a small amount of key point as point to be matched; Between each association subimage, set up match search, extract the matching double points between each subimage;

3) matching double points between each subimage is merged, utilize the coupling point set after merging to calculate the geometry splicing parameter of each frame of video;

4) utilize the geometry splicing parameter of each frame of video, each two field picture of video sequence is spliced, obtains a width panoramic picture;

5) shake for avoiding video camera to produce in shooting process, builds overall rotation matrix to revise bending panorama sketch, makes the last panorama sketch exported more meet the expression of objective world.

Described step 2) detailed process as follows:

(2-1) piecemeal multi-threaded parallel mode is adopted to carry out critical point detection to each subimage;

(2-2) its local message entropy is calculated to all key points, and carry out entropy filtration, retain front n larger key point of entropy and be used for characteristic matching;

(2-3) between each association subimage, extracting effective matching double points, for guaranteeing the accuracy that sub-block is mated, adopting symmetrical matching process to carry out mating between subimage;

(2-4) because noise and moving target exist, in coupling set, can Mismatching point be there is, finally adopt RANSAC (Random Sample Consensus, random sampling consistance) method to reject Mismatching point;

(2-5) between each association subimage, setting up the right coupling corresponding relation of subimage to rejecting the key point after filtering, extracting effective matching double points.

Described step 3) detailed process as follows:

(3-1) matching double points between each subimage extracted is merged, right as the coupling between two images;

(3-2) right according to coupling, homography matrix conversion between computed image, estimates rotational transform and the translation of camera parameters and image.

Described step 4) detailed process as follows:

(4-1) step 3 is utilized) geometry splicing parameter between the image of trying to achieve, image registration is carried out to all video frame images;

(4-2) by the image projection after registration on the face of cylinder, carry out the tone difference between removal of images by image co-registration, finally export revised panoramic picture.

Compared to prior art, method of the present invention has following features:

1, piecemeal is carried out to video frame images, and set up the one-to-one relationship between subimage, the strict match search region controlled between correspondence image, and adopt image block multi-threaded parallel strategy to accelerate match search speed;

2, by building overall rotation matrix, revise wavy panoramic picture, the video-splicing that the method is applicable to domestic consumer's shooting becomes panoramic picture.

Therefore, the present invention can be adapted to use common hand-held camera to carry out rotary taking video sequence to geographic scenes and be spliced into panoramic picture, splicing speed is fast, the shake occurred in shooting process is processed, avoiding spliced panorama sketch to produce distortion, is a kind of general panorama mosaic method being adapted to arbitrary geographic scene.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the inventive method;

Fig. 2 is image block and the multi-threaded parallel matching strategy of the embodiment of the present invention;

Fig. 3 is the wavy panoramic picture of the embodiment of the present invention;

Fig. 4 is the cylindrical surface projecting of the embodiment of the present invention;

Fig. 5 is the revised panoramic picture of the embodiment of the present invention.

Embodiment

Be described in further detail below in conjunction with drawings and Examples.

As

Shown in Fig. 1, based on the geographic scenes panorama sketch rapid generation of video sequence image, the method comprises following four parts:

The image of step 1 to the video sequence of input carries out graticule mesh piecemeal, after setting up the corresponding relation between subimage, adopts multi-threaded parallel optimisation strategy to carry out feature extracting and matching to subimage;

Step is 2-in-1 and subimage mates set;

Step 3 splices geometric parameter according to matching characteristic point to computed image;

Step 4 video sequence image merges;

The concrete implementation step of the present embodiment is as follows:

The image of step 1 to the video sequence of input carries out graticule mesh piecemeal, carries out the feature extracting and matching of subimage after setting up the corresponding relation between subimage;

(1-1) read two continuous frames image A and B in video sequence frame by frame, A and B is carried out to the graticule mesh piecemeal of M × N.

(1-2) multi-threaded parallel strategy (as shown in Figure 2) is adopted to adopt ORB algorithm to carry out critical point detection to each subimage;

(1-3) calculate its local message entropy descending sort to all key points, the key point selecting n entropy larger is used for Image Feature Matching.Computing method are as follows:

(1) for each key point arranges region-of-interest, being usually set to 3 × 3 or 5 × 5 neighborhoods centered by key point, for considering computing velocity, contiguous range being set to 3 × 3.

(2) local entropy of key point in 3 × 3 neighborhoods is calculated.If the gamma function of image block is f (x, y), and f (x, y) >=0, then have:

$H_f=-\underset{i=0}{\overset{2}{\Σ}}\underset{j=0}{\overset{2}{\Σ}}{p}_{ij}{\mathrm{logp}}_{ij}---\left(1\right)$

$p_{ij}=f(i,j)/\underset{i=0}{\overset{2}{\Σ}}\underset{j=0}{\overset{2}{\Σ}}f(i,j)---\left(2\right)$

Wherein: p _ijthe probability of pixel in image block i, j position.

(4) 2 ~ 3 are repeated until the local entropy of each key point has calculated.

(1-4) symmetrical matching process is adopted to carry out characteristic matching to subimage.

(1-5) because noise and moving target exist, can there is Mismatching point in coupling set, according to polar curve principle, two corresponding key points are distributed on respective polar curve, and expression formula is as follows:

$u_2^T{F}_{12}{u}_1=0---\left(3\right)$

Wherein, u ₁and u ₂be two image Corresponding matching point coordinate; F ₁₂based on matrix; According to formula (3), RANSAC method is adopted to reject Mismatching point.

Step 2 repeats the process of step 1, until often pair of subimage is to processing, and using all Match mergings in set of sub-images as the right coupling set of whole image.

Step 3 is according to the images match point pair extracted, and homography matrix conversion between computed image, estimates rotational transform and the translation of camera parameters and image.

(1) image homography matrix conversion

The n (>4) of two images is to match point can calculate homography matrix under the meaning of a difference constant factor, its algorithm is as follows:

Order $H=\left[\begin{array}{ccc}{h}_{01}& h_{02}& h_{03}\\ h_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\end{array}\right],$ Its vector form: h=(h ₀₁, h ₀₂, h ₀₃, h ₁₁, h ₁₂, h ₁₃, h ₂₁, h ₂₂, h ₂₃) ^t.For matching double points m=(x, y, 1) ^t, m'=(x', y', 1) ^t, from following two linear equations about h can be obtained:

(x,y,1,0,0,0,x'x,x'y)h＝x'

(4)

(0,0,0,x,y,1,y'x,y'y)h＝y'

$\begin{array}{cc}{x}^{\′}=\frac{h_{00}x+h_{01}y+h_{02}}{h_{20}x+h_{21}y+h_{22}}& y^{\′}=\frac{h_{10}x+h_{11}y+h_{12}}{h_{20}x+h_{21}y+h_{22}}\end{array}---\left(5\right)$

Choose 4 like this and can solve homography matrix H to above match point.

(2) focal length of camera is estimated

First calculate consecutive image in video sequence between homography matrix H _ij, according to plane homography definition, H=K (R+tn ^t) K ^-1, have:

$H_{ij}\≅K_i{R}_{ij}{K}_j^{-1}---\left(6\right)$

(6) formula is rewritten as:

$R_{10}~K_1^{-1}{H}_{10}{K}_0~\left[\begin{array}{ccc}{h}_{00}& h_{01}& f_0^{-1}{h}_{02}\\ h_{10}& h_{11}& f_0^{-1}{h}_{12}\\ f_1{h}_{20}& f_1{h}_{21}& f_0^{-1}{f}_1{h}_{22}\end{array}\right]---\left(7\right)$

Wherein, $H_{10}=\left[\begin{array}{ccc}{h}_{00}& h_{01}& h_{02}\\ h_{10}& h_{11}& h_{12}\\ h_{20}& h_{21}& h_{22}\end{array}\right],$ Utilize rotation matrix R ₁₀orthogonality,

Can obtain:

$h_{00}^2+h_{01}^2+f_0^{-2}{h}_{02}^2=h_{10}^2+h_{11}^2+f_0^{-2}{h}_{12}^2---\left(8\right)$

$h_{00}{h}_{10}+h_{01}{h}_{11}+f_0^{-2}{h}_{02}{h}_{12}=0---\left(9\right)$

Can be calculated by (8) and (9) two formulas:

$f_0^2=\frac{h_{12}^2-h_{02}^2}{h_{00}^2+h_{01}^2-h_{10}^2-h_{11}^2}$ And $h_{00}^2+h_{01}^2\≠h_{10}^2+h_{11}^2---\left(10\right)$

Or: $f_0^2=-\frac{h_{01}{h}_{12}}{h_{00}{h}_{10}+h_{01}{h}_{11}}$ And h ₀₀h ₁₀≠-h ₀₁h ₁₁(11)

F ₁the similar mode of same employing is obtained, and owing to supposing that video camera remains unchanged in shooting process mid-focal length herein, then the final estimation of focal length of camera f can pass through f ₀and f ₁geometric mean calculate, in panoramic image sequence, from different homography matrixs, obtain respective focal length estimate, final f should get the geometric mean of all focal lengths.The internal reference matrix K that last basis is obtained asks for the rotational transformation matrix R between two images.

(3) overall rotation matrix calculates

Be relative rotation matrices according to the rotation matrix between the image of homography transformation calculations between image, cause using the image of said method splicing to become wavy or skewed, as shown in Figure 3.In order to the image making last splicing is " straight ", need to set up overall rotating coordinate system.

By 3d space point x _ibe mapped to the some x in two dimensional image _ik, use following expression:

x _ik～K _kR _kx _i(12)

Consider may move and inclined camera in the shooting process of image, but camera horizontal edge (x-axis) generally can be kept to be parallel to ground level (vertical with y-axis direction), therefore the world coordinate system selected is vertical corresponding is y-axis, level is x-axis, and the 3rd axle is that z-axis is along optical axis direction.

In order to set up overall rotating coordinate system, matrix R need be built _g, make R _gr is taken advantage of on the right side _kcan ensure that overall y-axis j=(0,1,0) is perpendicular to image x-axis i=(1,0,0) afterwards, image z-axis is expressed as k=(0,0,1).

i ^TR _kR _gj＝0 (13)

Expression formula (13) is equivalent to and requires matrix R _kthe first row r _k0=i ^tr _kwith matrix R _gsecondary series r _g1=R _gj is mutually vertical.The constraint of all images forms a set, and is expressed as least square problem.

$r_{g1}=\arg \underset{r}{\min }\underset{k}{\Σ}{\left(r^T{r}_{k0}\right)}^2=\arg \underset{r}{\min }{r}^T\[\underset{k}{\Σ}{r}_{k0}{r}_{k0}^T\]r---\left(14\right)$

In order to determine the rotation matrix R of the overall situation completely _g, also need the constraint condition that one additional, i.e. the z-axis mean value of single rotation matrix r is met with the z-axis of overall rotation matrix _g2=R _gk.

Below, complete rotation matrix can be calculated by 3 steps:

1. $r_{g1}=\min eigenvector\left({\mathrm{\Σr}}_{k0}{r}_{k0}^T\right)$

2.r _g0＝N((∑R _k2)×r _g1)

3.r _g2＝r _g0×r _g1

Wherein, N ((∑ R _k2) × r _g1) be that vectorial normalization represents.

By overall rotation matrix R _globalconversion correction often opens the local rotation matrix of image wherein i representative image numbering:

$R_{newlocal}^i=R_{global}\×R_{local}^i---\left(15\right)$

(4) cylinder environment map builds

The present invention adopts the projecting method based on cylindrical coordinates, projects image onto on the face of cylinder.

Point on cylinder is determined by angle θ and height h parametrization, and as shown in Figure 4, corresponding relation is as follows:

(sinθ,h,cosθ)∝(x,y,f) (16)

According to (16) formula corresponding relation, the coordinate form being mapped as the face of cylinder from the plane of delineation can be calculated:

$x^{\′}=s\mathrm{\θ}=s{\tan }^{-1}\left(\frac{x}{f}\right)---\left(17\right)$

$y^{\′}=sh=s\frac{y}{\sqrt{x^2+y^2}}---\left(18\right)$

Wherein, (x, y) is plane picture coordinate, (x ', y ') be cylindrical coordinates, s is the radius of cylinder, and general s=f is with the deformation extent at minimizing image center.

Because cylinder is torse, image translation and rotation under cylindrical coordinates can keep shape invariance.Inverse mapping equation expression is as follows:

$\begin{array}{c}x=f\tan \mathrm{\θ}=f\tan \frac{x^{\′}}{s}\\ y=h\sqrt{x^2+f^2}=\frac{y^{\′}}{s}f\sqrt{1+{\tan }^2\frac{x^{\′}}{s}}=f\frac{y^{\′}}{s}\sec \frac{x^{\′}}{s}\end{array}---\left(19\right)$

Step 4 video sequence image merges, the tone difference phenomenon produced after adopting laplacian pyramid fusion method process image mosaic.Panoramic picture after merging as shown in Figure 5.

Claims (4)

1., based on the geographic scenes panorama sketch rapid generation of video sequence image, it is characterized in that, comprise the steps:

1) subimage that grid partition becomes different is carried out to video frame images, and set up the one-to-one relationship of each subimage block between successive video frames;

2) adopt piecemeal multi-threaded parallel strategy to carry out critical point detection to each subimage, and the key point detected is filtered, retain a small amount of key point as point to be matched; Between each association subimage, set up match search, extract the matching double points between each subimage;

3) matching double points between each subimage is merged, utilize the coupling point set after merging to calculate the geometry splicing parameter of each frame of video;

4) utilize the geometry splicing parameter of each frame of video, each two field picture of video sequence is spliced, obtains a width panoramic picture;

5) shake for avoiding video camera to produce in shooting process, builds overall rotation matrix to revise bending panorama sketch, makes the last panorama sketch exported more meet the expression of objective world.

2. the geographic scenes panorama sketch rapid generation based on video sequence image according to claim 1, is characterized in that, described step 2) detailed process as follows:

(2-1) piecemeal multi-threaded parallel mode is adopted to carry out critical point detection to each subimage;

(2-2) its local message entropy is calculated to all key points, and carry out entropy filtration, retain front n larger key point of entropy and be used for characteristic matching;

(2-3) between each association subimage, extracting effective matching double points, for guaranteeing the accuracy that sub-block is mated, adopting symmetrical matching process to carry out mating between subimage;

(2-4) because noise and moving target exist, in coupling set, can Mismatching point be there is, finally adopt RANSAC method to reject Mismatching point;

(2-5) between each association subimage, setting up the right coupling corresponding relation of subimage to rejecting the key point after filtering, extracting effective matching double points.

3. the geographic scenes panorama sketch rapid generation based on video sequence image according to claim 1, is characterized in that, described step 3) detailed process as follows:

(3-1) matching double points between each subimage extracted is merged, right as the coupling between two images;

(3-2) right according to coupling, homography matrix conversion between computed image, estimates rotational transform and the translation of camera parameters and image.

4. the geographic scenes panorama sketch rapid generation based on video sequence image according to claim 1, is characterized in that, described step 4) detailed process as follows:

(4-1) step 3 is utilized) geometry splicing parameter between the image of trying to achieve, image registration is carried out to all video frame images;

(4-2) by the image projection after registration on the face of cylinder, carry out the tone difference between removal of images by image co-registration, finally export revised panoramic picture.