CN105025287A - Method for constructing scene stereo panoramic image by utilizing video sequence images of rotary shooting - Google Patents

Abstract

The invention discloses a method for automatically generating an immersive stereo panoramic image by utilizing video sequence images of rotary shooting of a geographic scene. The method mainly comprises the steps that 1) characteristic matched pairs between successive frame images in the video sequence are extracted one by one, and internal and external parameters of a camera are estimated; 2) the virtual rotation speed and the strip size of the camera are calculated; 3) the strip position is calculated according an estimated parallax value in the principle of center circle projection; and 4) left and right strip images are extracted from video frame images according to the strip size and position, and panoramic stitching is carried out on the extracted left and right strip images to generate left and right panoramic images with parallax. The method has the advantages that the process of generating the stereo panoramic image is simple, the single digital camera carries out rotation shooting on the scene at the visual point which is approximately fixed to obtain the video sequence, and the left and right panoramic images with parallax can be automatically stitched, and real 3D observation can be carried out via different 3D display devices as red-green glasses.

Description

The video sequence image of rotary taking is utilized to build the method for scene stereoscopic full views figure

Technical field

The present invention relates to the fields such as the splicing of video image, computer vision, Digital Image Processing, geography information three-dimensional visualization, particularly relate to the full-view stereo drawing generating method utilizing video sequence image.

Background technology

Video sequence is as a kind of common mass media, and itself is containing very abundant information, not only obtain convenient and can more vivid, objectively express geographical space, form the real scene from " viewed from side the world ".With video sequence image for data basis, the method of research and utilization video sequence video generation panorama 3D stereopsis, the ability of GIS in data acquisition and video tour and modeling will be enriched widely, and namely panorama 3D stereopsis itself is a kind of modeling expression way, not only directly perceived, simple, convenient, and a kind of fabulous view expression can be provided for 3DGIS.

Some scholars generate panoramic stereo image for ordinary video sequence assembly and have carried out the research of related fields, the people such as Ishiguro adopt angularly rotary taking mode that the image sequence splicing of taking continuously is generated panoramic stereo image (Ishiguro H, Yamamoto M, Tsuji S.Omni-directional stereo [J] .IEEE Transaction on Pattern Analysis andMachine Intelligence, 1992,14 (2): 257-262).Peleg and Ben-Ezra utilizes circular projection's method on the basis of Ishiguro method, the image-forming condition of full-view stereo is simplified, do not need strictly angularly to take, tentatively achieve and utilize common camera to carry out full-view stereo imaging (Peleg S, Ben-Ezra M.Stereo panorama with a single camera [C] .IEEE Conference of Computer Vision and Pattern Recognition, 1999:395-401).The people such as JungukCho use single camera rotary taking image, calculate the band between superimposed images, image mosaic is formed panoramic stereo image (Junguk C, Joon-Hyuk C, Yongmin T, et al.Stereo Panoramic Image Stitching with a Single Camera [C] .IEEEInternational Conference on Consumer Electronics, 2013:256-257).The people such as K ang adopt image layered method to panoramic picture compute depth information, and generate panoramic stereo image (Kang S B.Layered DepthPanoramas [C] .IEEE Conference on Computer Vision and Pattern Recognition, 2007:1-8) on this basis.

The current full-view stereo generation method based on image mosaic formula requires strict or more to the restriction of style of shooting to shooting condition, and is not suitable for use general digital video camera to scene random capture video sequence construct real tri-dimension panorama sketch.

Summary of the invention

The present invention, based on video sequence image, utilizes center circle projection model, adopts self adaptation stripe size and position calculating method that video sequence image joint is generated stereoscopic panoramic image.The method makes full use of redundant information between video sequence image to generate real tri-dimension panoramic picture, can effectively avoid using multiple video camera to take and the inconvenience that brings, for ordinary populace provides a kind of method generating stereoscopic panoramic image easily.

To achieve these goals, the present invention adopts following technical scheme:

Utilize the video sequence image of rotary taking to build the method for scene stereoscopic full views figure, comprise the steps:

1) video frame images is read one by one, extract characteristic matching pair between consecutive image, and estimate the inside and outside parameter of video camera;

2) utilize the characteristic matching point between image to calculate the pseudo-velocity of video camera rotation, and determine the size of band according to the rotary speed of video camera;

3) according to center circle projection theory, conic projection is carried out to video image, utilize disparity estimation value, the position of computed image band;

4) according to the size of band and position, video frame images is extracted, extract left histogram picture and right histogram picture respectively, and all left histogram picture extracted and right histogram picture are carried out panoramic mosaic respectively and generate the left panoramic picture and right panoramic picture with parallax.

Described step 1) detailed process be:

(1-1) sequential frame image in video sequence is read one by one;

(1-2) ORB (oriented FAST and Rotated BRIEF) algorithm is adopted to carry out characteristic matching to video frame images;

(1-3) because noise and moving target exist, in coupling set, can Mismatching point be there is, according to polar curve principle, adopt RANSAC (Random Sample Consensus, random sampling consistency) method to reject Mismatching point;

(1-4) according to correct coupling point set, the homography matrix conversion between computed image, according to the inside and outside parameter matrix of homograph matrix computations video camera, and utilizes bundle adjustment principle to revise camera interior and exterior parameter matrix.

Described step 2) detailed process be:

(2-1) according to step 1) in the matching characteristic point of picture frame that extracts, calculate horizontal offset and the vertical offset of often pair of characteristic point;

(2-2) the horizontal offset summation of all feature point pairs is averaged, calculates camera horizon rotary speed;

(2-3) the vertical offset summation of all feature point pairs is averaged, calculates video camera vertical rotary speed;

(2-4) on the basis of first three step, the size of band is calculated.

Described step 3) detailed process is:

(3-1) video sequence image is projected to internal surface of column;

(3-2) the diameter viewpoint circle close with human eye cardinal distance is set at cylindrical inside, viewpoint circle builds left and right two virtual video cameras, virtual image is carried out to the video image projecting to the face of cylinder;

(3-3) according to virtual image by step 2) gyrobearing of the stripe size determined and virtual video camera and disparity estimation value, calculate left and right virtual camera respectively by center circle projection model and take position corresponding to band.

Described step 4) detailed process is:

(4-1) according to the stripe size calculated and position, from conic projection video image, left and right virtual camera histogram picture is extracted respectively;

(4-2) panoramic mosaic is carried out to the left and right virtual video camera image band extracted, obtain the left and right panoramic picture with parallax, form stereoscopic panoramic image pair.

Compared to prior art, the method for the invention has following features:

1, the inventive method adopts a video camera, self adaptation band is carried out to video sequence image in the basis of center circle projection and extracts to generate and have the left and right two width panoramic pictures of parallax, can effectively avoid using multiple video camera to take and the inconvenience that brings;

2, stereoscopic full views map generalization process of the present invention is without the need to manual intervention, and automaticity is high, can provide a kind of method generating panoramic stereo image easily for ordinary populace.

Therefore, stereoscopic panoramic image generation method of the present invention is simple, a video camera is only needed to carry out rotary taking to scene, being applicable to domestic consumer uses general video equipment to generate stereoscopic full views figure along approximate fixing viewpoint around the video sequence taken, and various true three-dimensional display apparatus (as mycyscope) all can carry out real tri-dimension observation.

Accompanying drawing explanation

Fig. 1 is the flow chart of the embodiment of the present invention;

The video image conic projection of Fig. 2 embodiment of the present invention;

The center circle projection theory of Fig. 3 embodiment of the present invention;

The center circle projection simplified mathematical model of Fig. 4 embodiment of the present invention;

Fig. 5 is the cylindrical surface projecting of video sequence of the present invention and left and right band image zooming-out.

Embodiment

Be described in further detail below in conjunction with drawings and Examples.

As shown in Figure 1, utilize the video sequence image of rotary taking to build scene stereoscopic full views drawing method, the method comprises following four parts:

Step 1 pair video frame images reads one by one, extracts characteristic matching pair between consecutive image, and estimates the inside and outside parameter of video camera;

Step 2 utilizes the characteristic matching point between image to calculate the pseudo-velocity of video camera rotation, and calculates stripe size according to the rotary speed of video camera;

Step 3 carries out conic projection according to center circle projection theory to video image, utilizes disparity estimation value, the position of computed image band;

Step 4 carries out band extraction according to the size of band and position to video frame images, extract left histogram picture and right histogram picture respectively, and carry out all left histogram picture and the right histogram picture that extract respectively splicing the left panoramic picture and right panoramic picture that generate and have parallax, and the use facilities for observation (for mycyscope be described) corresponding with display mode carries out stereovision and the experience with feeling of immersion;

The concrete implementation step of the present embodiment is as follows:

Step 1 pair video frame images reads one by one, extracts characteristic matching pair between consecutive image, and estimates the inside and outside parameter of video camera:

(1-1) read sequential frame image in video sequence one by one, adopting ORB algorithm to carry out characteristic matching to video frame images, for refining the correctness of matching double points between image further, adopting symmetrical matching principle to mate;

(1-2) because noise and moving target exist, can there is Mismatching point in coupling set, according to polar curve principle, two corresponding key points are distributed on respective polar curve, and expression formula is as follows:

$u_2^T{F}_{12}{u}_1=0---\left(1\right)$

Wherein, u ₁and u ₂be two image Corresponding matching point coordinates; F ₁₂based on matrix; According to formula (1), RANSAC method is adopted to reject Mismatching point.

(1-3) according to correct coupling point set, the homograph matrix between computed image, according to the inside and outside parameter matrix of homograph matrix computations video camera, according to the inside and outside parameter matrix of bundle adjustment principle correction video camera.

(1) image homography matrix conversion

The n (>4) of two images is to match point can calculate homography matrix under the meaning of a difference invariant, its algorithm is as follows:

Order $H=\left[\begin{array}{ccc}{h}_{01}& h_{02}& h_{03}\\ h_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\end{array}\right],$ Its vector form: h=(h ₀₁, h ₀₂, h ₀₃, h ₁₁, h ₁₂, h ₁₃, h ₂₁, h ₂₂, h ₂₃) ^t.For matching double points m=(x, y, 1) ^t, m'=(x', y', 1) ^t, from following two linear equations about h can be obtained:

(x,y,1,0,0,0,x'x,x'y)h＝x' (2)

(0,0,0,x,y,1,y'x,y'y)h＝y'

$\begin{array}{cc}{x}^{\′}=\frac{h_{00}x+h_{01}y+h_{02}}{h_{20}x+h_{21}y+h_{22}}& y^{\′}=\frac{h_{10}x+h_{11}y+h_{12}}{h_{20}x+h_{21}y+h_{22}}\end{array}---\left(3\right)$

Choose 4 like this and can solve homography matrix H to above match point.

(2) focal length of camera is estimated

First calculate consecutive image in video sequence between homography matrix H _ij, according to plane homography definition, H=K (R+tn ^t) K ^-1, have:

$H_{ij}\≅K_i{R}_{ij}{K}_j^{-1}---\left(4\right)$

(4) formula is rewritten as:

$R_{10}~K_1^{-1}{H}_{10}{K}_0~\left[\begin{array}{ccc}{h}_{00}& h_{01}& f_0^{-1}{h}_{02}\\ h_{10}& h_{11}& f_0^{-1}{h}_{12}\\ f_1{h}_{20}& f_1{h}_{21}& f_0^{-1}{f}_1{h}_{22}\end{array}\right]---\left(5\right)$

Wherein, $H_{10}=\left[\begin{array}{ccc}{h}_{00}& h_{01}& h_{02}\\ h_{10}& h_{11}& h_{12}\\ h_{20}& h_{21}& h_{22}\end{array}\right],$ Utilize spin matrix R ₁₀orthogonality,

Can obtain:

$h_{00}^2+h_{01}^2+f_0^{-2}{h}_{02}^2=h_{10}^2+h_{11}^2+f_0^{-2}{h}_{12}^2---\left(6\right)$

$h_{00}{h}_{10}+h_{01}{h}_{11}+f_0^{-2}{h}_{02}{h}_{12}=0---\left(7\right)$

Can be calculated by (6) and (7) two formulas:

$f_0^2=\frac{h_{12}^2-h_{02}^2}{h_{00}^2+h_{01}^2-h_{10}^2-h_{11}^2}$ And $h_{00}^2+h_{01}^2\≠h_{10}^2+h_{11}^2---\left(8\right)$

Or: $f_0^2=-\frac{h_{01}{h}_{12}}{h_{00}{h}_{10}+h_{01}{h}_{11}}$ And h ₀₀h ₁₀≠-h ₀₁h ₁₁(9)

F ₁the similar mode of same employing is obtained, and owing to supposing that video camera remains unchanged in shooting process mid-focal length herein, then the final estimation of focal length of camera f can pass through f ₀and f ₁geometric average calculate, in panoramic image sequence, from different homography matrixs, obtain respective focal length estimate, final f should get the geometric average of all focal lengths.The internal reference matrix K that last basis is obtained asks for the rotational transformation matrix R between two images.

Step 2 calculates the pseudo-velocity of video camera rotation according to the characteristic matching point between image, and calculates stripe size according to the rotary speed of video camera:

(2-1) the matching double points number detected according to two continuous frames image calculates the pseudo-velocity that video camera rotates:

$V_h=\frac{\underset{i}{\overset{n}{\Σ}}{p}_{1i}\left(x\right)-p_{2i}\left(x\right)}{n}---\left(10\right)$

$V_p=\frac{\underset{i}{\overset{n}{\Σ}}{p}_{1i}\left(y\right)-p_{2i}\left(y\right)}{n}---\left(11\right)$

Wherein, V _hthe pseudo-velocity of horizontal direction, V _pthe pseudo-velocity of vertical direction, p _1ix () represents the abscissa x of i-th match point in frame 1 image, p _2ix () represents the abscissa x of i-th match point in frame 2 image, p _1iy () represents the ordinate y of i-th match point in frame 1 image, p _2iy () represents the ordinate y of i-th match point in frame 2 image.

(2-2) stripe size △ D is calculated according to the rotary speed of video camera:

△D＝V _h(12)

Step 3 carries out conic projection according to center circle projection theory to video image, utilizes disparity estimation value, the position of computed image band:

(3-1) conic projection is carried out to video image, is projected to internal surface of column:

Point on cylinder is determined by angle θ and height h parametrization, and as shown in Figure 2, corresponding relation is as follows:

(sinθ,h,cosθ)∝(x,y,f) (13)

According to (13) formula corresponding relation, the coordinate form being mapped as the face of cylinder from the plane of delineation can be calculated:

$x^{\′}=s\mathrm{\θ}={\mathrm{stan}}^{-1}\left(\frac{x}{f}\right)---\left(14\right)$

$y^{\′}=sh=s\frac{y}{\sqrt{x^2+y^2}}---\left(15\right)$

Wherein, (x, y) is plane picture coordinate, (x ', y ') be cylindrical coordinates, s is the radius of cylinder, and general s=f is with the deformation extent at minimizing image center.

Because cylinder is developable surface, image translation and rotation under cylindrical coordinates can keep shape invariance.Inverse mapping equation expression is as follows:

$\begin{array}{c}x=ftan\mathrm{\θ}=ftan\frac{x^{\′}}{s}\\ y=h\sqrt{x^2+f^2}=\frac{y^{\′}}{s}f\sqrt{1+{\tan }^2\frac{x^{\′}}{s}}=f\frac{y^{\′}}{s}\sec \frac{x^{\′}}{s}\end{array}---\left(16\right)$

(3-2) in order to realize stereoscopic vision, according to center circle projection theory, as shown in Figure 3, in cylinder, set up diameter to be the center circle of human eye cardinal distance 2 times, respectively furnish a virtual video camera V in the arranged on left and right sides of center circle _l, V _r, distance is designated as 2d.Real camera is put in photocentre O place, and rotating shaft radius is r, and camera intrinsic rotating shaft carries out the shooting of rotating 360 degrees continuous videos, and each two field picture of shooting is projected to conic projection surface, and the light of virtual camera is projected to plane of projection of cylinder along photocentre O place.

(3-3) for ease of calculating pillar location, center circle projection model is simplified, as shown in Figure 4, left and right virtual video camera V _r, V _lbetween differ 2d, radius of turn is designated as r, focal length of camera by step 1 estimate obtain, be designated as f, the difference of left and right pillar location is designated as 2v.According to similar triangle theory, the relation (see Fig. 4) between band and parallax can be calculated:

$\frac{2v}{2d}=\frac{f}{r}---\left(17\right)$

In formula, d is the human eye cardinal distance of setting, and radius of turn mean value when r is video camera rotary taking, f is that step 1 calculates video camera internal reference.After obtaining v according to proportion expression, just can calculate position corresponding to band.

Step 4 carries out band extraction according to the stripe size calculated and position to video frame images, extract left histogram picture and right histogram picture respectively, and carry out all left histogram picture extracted and right histogram picture respectively splicing the left panoramic picture and right panoramic picture that generate and there is parallax, and use the facilities for observation (for mycyscope) corresponding with display mode to have the full-view stereo figure of feeling of immersion with regard to observable, its concrete step is as follows:

(4-1) for video sequence, the image of generation is more, needs all to carry out conic projection by center circule method to each two field picture, obtains projection video sequence as shown in Figure 5.

(4-2) by the stripe size calculated and position, band extraction is carried out to video frame images, extract left histogram picture and right histogram picture (see Fig. 5) respectively;

(4-3) respectively panoramic mosaic is carried out to all left and right histogram picture extracted, generate and there is the left panorama sketch of parallax and right panorama sketch;

(4-4) adopt red green synthetic method to carry out red green synthesis to panoramic picture respectively to left and right panoramic picture and generate red green stereoscopic panoramic image, synthetic method is as follows:

As follows to left panoramic picture processing procedure:

(1) select the product look mask image that onesize with left panoramic picture, the rgb value of each pixel of mask image is (255,0,0).

(2) each pixel of left panoramic picture is carried out inclusive-OR operation with each pixel of mask image respectively, obtain a new left panoramic picture.

Computational methods are as follows:

If certain pixel (i in left panoramic picture, j) colored RGB is (r, g, b), same position (the i of red mask image, j) colored RGB is (255,0,0), then carry out inclusive-OR operation, then obtain the colored RGB of new pixel for (255, g, b).

Similarly, also can adopt to use the same method to right panoramic picture and carry out inclusive-OR operation with green mask image (RGB (0,255,0)), obtain a new right panoramic picture.

After new left panoramic picture A and new right panoramic picture B generates, need by A, B two Zhang Quanjing image carry out synthesizing a red green panoramic picture and just can demonstrate stereoeffect, therefore, need to carry out AND operation to left and right two Zhang Quanjing images.

If the rgb value (255, g, b) of certain pixel (i, j) in A panoramic picture, in B panoramic picture respective pixel rgb value (r, 255, b), carry out AND operation, obtain a new rgb value for (r', g', b').AND operation is carried out to all pixels in A, B image, the red green panoramic picture after a new synthesis can be obtained.

Claims (5)

1. utilize the video sequence image of rotary taking to build the method for scene stereoscopic full views figure, it is characterized in that, the method comprises the steps:

1) video frame images is read one by one, extract characteristic matching pair between consecutive image, and estimate the inside and outside parameter of video camera;

2) utilize the characteristic matching point between image to calculate the pseudo-velocity of video camera rotation, and determine the size of band according to the rotary speed of video camera;

3) according to center circle projection theory, conic projection is carried out to video image, utilize disparity estimation value, the position of computed image band;

4) according to the size of band and position, video frame images is extracted, extract left histogram picture and right histogram picture respectively, and all left histogram picture extracted and right histogram picture are carried out panoramic mosaic respectively and generate the left panoramic picture and right panoramic picture with parallax.

2. the video sequence image utilizing rotary taking according to claim 1 builds the method for scene stereoscopic full views figure, it is characterized in that, described step 1) detailed process be:

(1-1) sequential frame image in video sequence is read one by one;

(1-2) ORB algorithm is adopted to carry out characteristic matching to video frame images;

(1-3) because noise and moving target exist, in coupling set, can Mismatching point be there is, according to polar curve principle, adopt RANSAC method to reject Mismatching point;

(1-4) according to correct coupling point set, the homography matrix conversion between computed image, according to the inside and outside parameter matrix of homograph matrix computations video camera, and utilizes bundle adjustment principle to revise camera interior and exterior parameter matrix.

3. the video sequence image utilizing rotary taking according to claim 1 builds the method for scene stereoscopic full views figure, it is characterized in that, described step 2) detailed process be:

(2-1) according to step 1) in the matching characteristic point of picture frame that extracts, calculate horizontal offset and the vertical offset of often pair of characteristic point;

(2-2) the horizontal offset summation of all feature point pairs is averaged, calculates camera horizon rotary speed;

(2-3) the vertical offset summation of all feature point pairs is averaged, calculates video camera vertical rotary speed;

(2-4) on the basis of first three step, the size of band is calculated.

4. the video sequence image utilizing rotary taking according to claim 1 builds the method for scene stereoscopic full views figure, it is characterized in that, described step 3) detailed process is:

(3-1) video sequence image is projected to internal surface of column;

(3-2) the diameter viewpoint circle close with human eye cardinal distance is set at cylindrical inside, viewpoint circle builds left and right two virtual video cameras, virtual image is carried out to the video image projecting to the face of cylinder;

(3-3) according to virtual image by step 2) gyrobearing of the stripe size determined and virtual video camera and disparity estimation value, calculate left and right virtual camera respectively by center circle projection model and take position corresponding to band.

5. the video sequence image of rotary taking that utilizes stated according to claim 4 builds the method for scene stereoscopic full views figure, it is characterized in that, described step 4) detailed process is:

(4-1) according to the stripe size calculated and position, from conic projection video image, left and right virtual camera histogram picture is extracted respectively;

(4-2) panoramic mosaic is carried out to the left and right virtual video camera image band extracted, obtain the left and right panoramic picture with parallax, form stereoscopic panoramic image pair.