CN104881869A - Real time panorama tracing and splicing method for mobile platform - Google Patents

Abstract

The invention discloses a real time panorama tracing and splicing method for mobile platform. First, each image frame captured by a camera is used for feature tracking and is then utilized for calculating rotation posture of the camera in three-dimensional space. Second, whether the center of the camera shifts or not is also detected during the process of estimating the posture of the camera in real time. If shift occurs, a corresponding feedback is sent to a user and help is offered to the user to reset the camera position to an initial one, so that translation error between captured images can be reduced and that the images meets constraint on a sphere is guaranteed. Third, key frames and three-dimensional feature points are added and related parameters are optimized and a Map is updated completely by utilizing a bundling adjustment method. Fourth, a plurality of pictures are spliced together in real time in an offline manner on a background, and exposure compensation and multi-frequency-band fusion of images are realized, so that a panorama with geometrical consistency is generated with high quality. The result indicates that a scene expression method based on multi-view panorama has very good scene expression capability.

Description

Real time panoramic on a kind of mobile platform is followed the tracks of and joining method

Technical field

The present invention relates to a kind of scene expression of panorama, the real time panoramic particularly related on a kind of mobile platform is followed the tracks of and joining method.

Technical background

Panorama sketch is a kind of common environment viewing mode, can be regarded as the popularization of common wide-angle lens, has larger visual angle and wider scene information.Panorama sketch can be widely used in scene walkthrough, virtual reality, texture, the every field such as augmented reality with its distinctive Visual exhibition form of stage design.

Traditional Panoramagram generation method, comprise David Lowe based on scale invariant feature (SIFT) (Lowe, David G.Distinctive Image Features from Scale-InvariantKeypoints.International Journal of Computer Vision 60.2, 2004:91-110.) photograph image joining method (Matthew Brown, David G.Lowe.Automatic Panoramic ImageStitching using Invariant Features.International Journal of ComputerVision, 2007, 74 (1): 59-73), Google based on gyrostatic pan-shot App-PhotoSphere (https: //www.google.com/maps/about/contribute/photosphere/), panorama App-the PhotoSynth (https: //photosynth.net/) based on characteristics of image shooting of Microsoft.Wherein the panorama generating algorithm of David is based on SIFT feature coupling, the feature interpretation ability powerful by means of SIFT algorithm and matching precision, and the algorithm of David under large scene, can generate without side seam, no color differnece, the panorama sketch that geometry is consistent.But because the calculating of SIFT is very consuming time, make this algorithm be not suitable for using on a mobile platform.The pan-shot App of Google is the rotation parameter obtained based on gyroscope, and on a mobile platform, the information of these sensors of gyroscope can accomplish the renewal frequency of 60 frames per second, because can accomplish the effect of live preview on a mobile platform.But built-in sensor has very large noise, especially after the noise of these sensors is mapped on image, image can see obvious border and gap, and some basic geometries can not keep consistency, and such as line segment can in image junction disconnection etc.The PhotoSynth of Microsoft uses the characteristic information of image, thus can keep obvious Geometrical consistency on image, and there is reasonable degrees of fusion marginal portion relative to the PhotoSphere of Google.But although PhotoSynth is the pure rotational motion of hypothesis camera, the operation of but not strict restriction handset user, even if user has obvious translation, namely the center of camera moves, or image can be added to come in, do Images uniting like this and just have obvious geometric error, especially in indoor scene, because the depth of field of object is smaller, it is all larger skew that movement faint on camera is reflected on image.Therefore be necessary very much to do a detection to the translation of camera, provide feedback timely simultaneously, ensure that the image adding panorama does not exist larger translation, only have the original image of input to meet the geometric attribute of pure rotation, the final panorama sketch generated may be just that geometry is consistent.The method of traditional process panorama sketch mostly visual cognitive ability how to go to eliminate image center move the image that causes between geometric error on, and nobody goes to check from source whether the image of input meets the constraint of pure rotation.

First this invention designs a kind of real time panoramic tracker realized on mobile terminals, achieve the estimation of quick camera attitude (3DOF), simultaneously can also in the process of following the tracks of, whether the camera of real-time each frame of detection has translation motion, and make feedback timely, instruct user to take according to the method for pure rotation.Experimental result shows: by the form of this real-time detection and feedback, domestic consumer can be good at shooting agonic panorama sketch.

Summary of the invention

The object of the invention is to the deficiency for conventional panoramic method for making, the real time panoramic proposed on a kind of mobile platform is followed the tracks of and joining method.

Real time panoramic on a kind of mobile platform is followed the tracks of and is comprised the steps: with joining method

1) based on the real-time follow-up of Sphere Measurement Model: the image that each frame is caught by camera, first do signature tracking, then calculate Current camera at three-dimensional rotation attitude;

2) translation of fast robust detects: while carrying out real-time follow-up camera rotation attitude, also whether the center of detection camera is subjected to displacement, in time being subjected to displacement, feed back accordingly to user, user is helped to get back to initial camera position, translation error between the image that reduction is caught, ensures that image meets the constraint on a sphere;

3) Map expansion and renewal: add key frame and three-dimensional feature point, the method utilizing boundling to adjust is optimized, and upgrades whole Map, and whole Mapping process is placed on background thread to calculate;

4) panorama off-line synthesis: multiple images are stitched together by backstage off-line in real time, carry out exposure compensating and multi-band blending to image, the panorama sketch that high-quality generation geometry is consistent.

Described step 1) be specially:

A) the RGBA image will obtained, builds 4 tomographic image pyramids, all detects FAST unique point on each layer after changing into gray-scale map;

B) before the signature tracking of each frame starts, based on the attitude of previous frame camera and the image information of present frame, the camera attitude of pre-estimation present frame;

C) by the camera attitude estimated, by the three-dimensional feature spot projection in Map on current frame image, as the initial position of feature point tracking, and unique point search is limited in distance current signature point radius is in the window ranges of 10 pixels;

D) select FAST to do feature detection, adopt image Patch to do signature tracking and coupling, and carry out brightness of image change and pattern distortion process;

E) define the increment that energy function estimates camera attitude, introduce MEstimator and reject outliers, then utilize weighted least-squares iteration optimization, solve the camera parameter of present frame;

F) image is divided into four layers of pyramid, in the highest two-layer estimation initial camera attitude, is two-layerly below optimized.

Described step 3) be specially:

1) selected characteristic coupling reaches more than 30%, with add before key frame have at least 20 frame pitches from key frame, and the strict image center that controls rotates, and utilizes the homography relational extensions Map of reference frame and present frame;

2) the Projection Constraint relation between multiple key frame and multiple unique point be put into inside an energy function, utilize the adjustment of the boundling of Sphere Measurement Model to carry out global optimization, upgrade whole Map, whole Mapping process is placed on background thread to calculate.

The invention has the advantages that: designed and Implemented real time panoramic on mobile terminals and followed the tracks of and generation system, to compare traditional panorama system, while carrying out real-time camera Attitude estimation, also whether the center of detection camera is subjected to displacement, in time being subjected to displacement, feed back accordingly to user, help user to get back to initial camera position, reduce the translation error between the image of catching to the full extent, ensure that image maximum possible meets the constraint on a sphere.The parameter that the method optimization utilizing boundling to adjust is relevant, upgrades whole Map, and the quality of the final panorama sketch generated also can be guaranteed.

Accompanying drawing explanation

Fig. 1 is real time panoramic tracker frame diagram on mobile platform;

Fig. 2 is that real time panoramic is followed the tracks of and Map expansion process.

Embodiment

Real time panoramic on a kind of mobile platform is followed the tracks of and is comprised the steps: with joining method

1) based on the real-time follow-up of Sphere Measurement Model: the image that each frame is caught by camera, first do signature tracking, then calculate Current camera at three-dimensional rotation attitude;

2) translation of fast robust detects: while carrying out real-time follow-up camera rotation attitude, also whether the center of detection camera is subjected to displacement, in time being subjected to displacement, feed back accordingly to user, user is helped to get back to initial camera position, translation error between the image that reduction is caught, ensures that image meets the constraint on a sphere;

3) Map expansion and renewal: add key frame and three-dimensional feature point, the method utilizing boundling to adjust is optimized, and upgrades whole Map, and whole Mapping process is placed on background thread to calculate;

4) panorama off-line synthesis: multiple images are stitched together by backstage off-line in real time, carry out exposure compensating and multi-band blending to image, the panorama sketch that high-quality generation geometry is consistent.

Fig. 1 is the real time panoramic tracker framework on mobile platform.

The described real-time follow-up based on Sphere Measurement Model: the image that each frame is caught by camera, first does signature tracking, then calculates Current camera in the concrete steps of three-dimensional rotation attitude to be:

1) Image Acquisition

First obtain the RGBA format-pattern of 640x480 resolution from equipment, image is transferred to the gray-scale map of 8 bits (256 grades), then build image pyramid, be divided into 4 levels, every one deck is the half of a tomographic image above respectively.All detect FAST unique point on each layer.

2) estimation

Before the signature tracking of each frame starts, based on the attitude of previous frame camera and the image information of present frame, the camera attitude of pre-estimation present frame.Here select a SmallBlurryImage after down-sampled of original image, size is 40x30, and namely length and width are 1/16th of original image respectively.Then utilize the gray-scale value of image to optimize the rotation of present frame relative to previous frame.For this problem, be defined as follows function:

$\mathrm{err}=\underset{i}{\overset{n}{\mathrm{\Σ}}}{\left|\right|I_r\left(W(x,p_i)\right)-I_c\left(p_i\right)\left|\right|}^2$

Wherein I _rrepresent the gray-scale map of previous frame (reference frame), I _crepresent the gray-scale map of present frame, p _irepresent i-th pixel coordinate on image, W represents and does perspective transform to image, and x is the parameter of W.This problem is followed the tracks of based on the panorama of sphere, so the parameter that needs are estimated is a rotation parameter of 3DOF.

3) projecting characteristic points

After estimation, just there is present frame camera initial attitude.By the camera attitude that this is estimated, by the three-dimensional feature point { X in Map _iproject to present frame image on obtain { x _i, as the initial position of feature point tracking.By the characteristic point position that this is initial, the search of each unique point is limited at x _iin a window around (being usually set to the window that radius is 10).This method significantly can reduce the calculated amount of unique point search, and the probability of error hiding.

4) based on the signature tracking of Patch

Select FAST to do feature detection, adopt image Patch to do signature tracking and coupling.Each unique point chooses the description amount of the image intensity value in the window of 8*8 around as this feature.The measuring similarity of feature is the SSD (mean square deviation) of calculating two pieces of Patch.Consider that the illumination of real scene has the change in brightness.Normalized Patch is used to be SSD.

${\mathrm{err}}_{\mathrm{SSD}}=\underset{i}{\overset{n}{\mathrm{\Σ}}}{\left|\right|(I_r\left(p_i\right)-M_r)-(I_c\left(p_i\right)-M_c)\left|\right|}^2$ Formula (0.1)

Wherein M _rthe mean value of reference frame pixel, M _cthe mean value of current frame pixel.Each Patch mean value of current Patch pixel does a normalization, namely uses the relative value of gray-scale map, instead of absolute value, can effectively avoid like this by brightness of image change cause do not mate.

Also consider the Geometrical consistency of image, do the motion of pure rotation at camera under, image has and clearly distorts, in order to process the matching problem that pattern distortion causes, when calculating the SSD between Patch, first will according to the relative motion of camera, the imagery exploitation homography matrix of reference frame, under changing to the visual angle of current frame.Similar PTAM, directly adopts homography conversion (Homography) to do deformation to image here.That is:

W(x,p)＝H(p)

Wherein H is homography matrix, and homography matrix has 8 degree of freedom, relative to the affined transformation of 6 degree of freedom, and the function of many process perspective transforms.Thus better descriptive power is had to the tracking of the pure rotation of process.

5) camera attitude is upgraded

After having carried out effective feature point tracking, just having had the new observation station of three-dimensional feature point on present frame, is next exactly follow the tracks of new observation station to upgrade the attitude of camera.Be defined as follows energy function:

f＝π(K*ΔR*initR*V _i)-(u _i,v _i) ^T

$g={\mathrm{\Σ}}_i^n{f}^2$

Wherein V _ibe three site coordinates on sphere, the initial camera attitude obtained time initR is estimation, Δ R is the increment needing the camera attitude estimated.π is a projection function, and a three-dimensional point is projected in the plane of Z=1, obtains the two-dimensional coordinate of normalized (x, a y) form.

Introduce MEstimator and reject outliers, to process the larger noise produced in unique point when practical application

$F={\mathrm{\Σ}}_i^n\mathrm{\ρ}\left(f\right)$

Wherein ρ is a MEstimater, and we select Tukey,

$\mathrm{Tukey}=\left\{\begin{array}{c}\frac{c^2}{6}(1-{[1-{\left(\frac{x}{c}\right)}^2]}^3)\mathrm{when}\left|x\right|\≤c\\ \left(\frac{c^2}{6}\right)\mathrm{when}\left|x\right|>c\end{array}\right.$

Tukey is comparatively large to the weight at the nearer point of decentering, and the weight of the point that decentering is far away is almost nil.Can well mark and inliers and outliers is distinguished.

According to the weight that Tukey calculates, utilize weighted least-squares iteration optimization, the camera parameter of present frame can be solved.

6) by the thick two step iterative process to essence

Because mobile hand-held device probably has the situation of movement fast in the middle of reality uses.Image is divided into which floor pyramid, the first step first does signature tracking at the highest two-layer (80*60 and 160*120), estimates an initial camera attitude.Then second step more below two-layer (320*240 and 640*480) do above and further go to optimize camera attitude, the search radius of patch also can be restricted to original half simultaneously.The general coupling that can obtain 40 ~ 60 feature point pairs in the first step, can obtain the Feature Points Matching of 800 ~ 1000 at second step.Through like this by thick to smart process, the tracking of camera can keep good robustness and accuracy.

The translation of described fast robust detects: while carrying out real-time camera Attitude estimation, also whether the center of detection camera is subjected to displacement, in time being subjected to displacement, feed back accordingly to user, user is helped to get back to initial camera position, reduce the translation error between the image of catching to the full extent, ensure that the constraint that image maximum possible meets on a sphere is specially:

When camera view is motionless, there is following geometric projection relation:

P _c＝(u _c,v _c,1) ^T＝(K*R _c*V _w)

P _r＝(u _r,v _r,1) ^T＝(K*R _r*V _w)

Wherein P _cthe subpoint (pixel coordinate) on present frame, P _rthe subpoint on reference frame, for same three-dimensional coordinate V _w, by the camera attitude of present frame and reference frame, project on the image of present frame and reference frame respectively, can derive according to formula above

P _c＝K*R _c*R _r ^-1*K ^-1*P _r

Make H=K*R _c* R _r ^-1* K ^-1, then have:

P _c＝H*P _r

Then H be one from P _rto P _chomography converts.Homography matrix H has at most 8 degree of freedom.And when image center invariant position time, this conversion only has three degree of freedom.Change between two frames only has a rotation amount R=R _c* R _r ^-1, whether the center that we utilize this characteristic to carry out detection camera moves.

Described Map expansion and renewal: add key frame and three-dimensional feature point, and the parameter that the method optimization utilizing boundling to adjust is relevant, upgrade whole Map.Whole Mapping process is placed on background thread:

1) choose matching characteristic better, with add before key frame have 20 frame pitches from key frame, and the strict image center that controls rotates.

Add the principle of key frame:

1. the tracking quality of camera is sufficiently good, and define all unique points projected to from Map image, what have 30% to match just thinks that tracking quality is reliable.

2. present frame nearest once interpolation key frame will more than 20 frames, and that avoids key frame to add is too concentrated.

3. the distance of key frame nearest in Current camera distance Map is enough large, and distance COS distance is here measured.The direction vector of key frame does inner product, is less than cos (5 °), just can adds Map.

4. for the model followed the tracks of based on spherical panorama, also to limit the movement of image center, if image center has translation, camera first will be waited to return to the position of initial center, just can add new key frame.

After with the addition of new key frame, new key frame can introduce new unique point.According to homography matrix, the projecting characteristic points of reference frame to present frame, then around subpoint 10x10 window in search characteristics point, the SSD of Patch around comparative feature point, choose be less than threshold value and SSD value minimum as optimum matching.The point of coupling can be found to add in Map, upgrade the related data of Map simultaneously.

Map expansion algorithm describes:

1. detect key frame KF _c

2. select key frame KF nearest in Map _r

3.KF _rwith KF _cbetween do the characteristic matching of discontinuous frame

A) calculating K F _rwith KF _cbetween homography matrix H

B) calculating K F _ron unique point at KF _con projection

C) at KF _rwith KF _clap do characteristic matching based on SSD, mate the unique point that obtains and be added in Map.

D) at KF _cin the unique point that do not match, the unique point choosing response maximum adds Map

4. by new KF _cadd Map with relevant unique point, upgrade Map

5. local boundling adjustment

Fig. 2 illustrates the process of this algorithm splicing in real time on Ipad, and wherein green box is the real-time pictures of current camera image.Along with the movement of camera, constantly add new image in Map and come in, the area of panorama sketch constantly expands, and the visual angle of covering is also increasing.

2) the Projection Constraint relation between multiple key frame and multiple unique point is put into inside an energy function, utilizes the adjustment of the boundling of Sphere Measurement Model to carry out global optimization.

If the information spinner key frame stored inside Map and three-dimensional feature point, because incremental computation has cumulative errors.Projection Constraint relation between multiple key frame and multiple unique point is put into global optimization inside an energy function:

$\underset{\underset{\left(R_j\right),j=1,...,N_C}{X_i,i=1,...,N_P}}{\arg \min }\underset{i=1}{\overset{N_P}{\mathrm{\Σ}}}\underset{j=1}{\overset{N_C}{\mathrm{\Σ}}}{b}_{\mathrm{ij}}\mathrm{\ρ}(\mathrm{\π}(K_j,R_j,X_i)-x_{\mathrm{ij}})$

Wherein ρ is Tukey function, b _ijrepresent that whether i-th point be visible at jth frame.If i-th point is visible at jth frame, b _ij=1, x _ijbe the projection of i-th point at jth frame; Otherwise b _ij=0.

Described panorama off-line synthesis: plurality of pictures is stitched together by backstage off-line in real time, realizes the exposure compensating to image and multi-band blending, the consistent panorama sketch of high-quality generation geometry is specially:

The method also adopts the method for David G.Lowe to do exposure compensating and multi-band blending to image, simultaneously by means of the computer vision storehouse OpenCV increased income to realize this process.

ExposureCompensator module in OpenCV achieves the exposure compensating function of image, and MultiBandBlender module achieves the multi-band blending function of image, calls these two modules realizations respectively to the exposure compensating of image and multi-band blending.

Several groups of panoramic pictures to same scene capture.From image results, no matter be PhotoSphere or PhotoSynth, owing to throwing the reins to the shooting behavior of photographer, photographer is easy in the process of shooting, image center is moved, catches the constraint that the image obtained just does not meet pure rotation like this.

Claims (3)

1. the real time panoramic on mobile platform is followed the tracks of and a joining method, it is characterized in that comprising the steps:

1) based on the real-time follow-up of Sphere Measurement Model: the image that each frame is caught by camera, first do signature tracking, then calculate Current camera at three-dimensional rotation attitude;

2) translation of fast robust detects: while carrying out real-time follow-up camera rotation attitude, also whether the center of detection camera is subjected to displacement, in time being subjected to displacement, feed back accordingly to user, user is helped to get back to initial camera position, translation error between the image that reduction is caught, ensures that image meets the constraint on a sphere;

3) Map expansion and renewal: add key frame and three-dimensional feature point, the method utilizing boundling to adjust is optimized, and upgrades whole Map, and whole Mapping process is placed on background thread to calculate;

4) panorama off-line synthesis: multiple images are stitched together by backstage off-line in real time, carry out exposure compensating and multi-band blending to image, the panorama sketch that high-quality generation geometry is consistent.

2. the real time panoramic on a kind of mobile platform according to claim 1 is followed the tracks of and joining method, it is characterized in that described step 1) is specially:

A) the RGBA image will obtained, builds 4 tomographic image pyramids, all detects FAST unique point on each layer after changing into gray-scale map;

B) before the signature tracking of each frame starts, based on the attitude of previous frame camera and the image information of present frame, the camera attitude of pre-estimation present frame;

C) by the camera attitude estimated, by the three-dimensional feature spot projection in Map on current frame image, as the initial position of feature point tracking, and unique point search is limited in distance current signature point radius is in the window ranges of 10 pixels;

D) select FAST to do feature detection, adopt image Patch to do signature tracking and coupling, and carry out brightness of image change and pattern distortion process;

E) define the increment that energy function estimates camera attitude, introduce MEstimator and reject outliers, then utilize weighted least-squares iteration optimization, solve the camera parameter of present frame;

F) image is divided into four layers of pyramid, in the highest two-layer estimation initial camera attitude, is two-layerly below optimized.

3. the real time panoramic on a kind of mobile platform according to claim 1 is followed the tracks of and joining method, it is characterized in that described step 3) be specially:

1) selected characteristic coupling reaches more than 30%, with add before key frame have at least 20 frame pitches from key frame, and the strict image center that controls rotates, and utilizes the homography relational extensions Map of reference frame and present frame;

2) the Projection Constraint relation between multiple key frame and multiple unique point be put into inside an energy function, utilize the adjustment of the boundling of Sphere Measurement Model to carry out global optimization, upgrade whole Map, whole Mapping process is placed on background thread to calculate.