CN103093460A - Moving camera virtual array calibration method based on parallel parallax - Google Patents

Abstract

The invention discloses a moving camera virtual array calibration method based on parallel parallax. The moving camera virtual array calibration method based on the parallel parallax is used for solving the technical problems that an existing fixed camera virtual array calibration method based on the parallel parallax needs a plurality of cameras. The technical scheme includes imitating a virtual camera array system by means of moving in parallel on a linear guide rail and tracking feature points of images shot at various viewpoints so as to estimate gestures of a multi-view camera. Firstly, a reference plane which is parallel to the direction of the linear guide rail is defined according to the obtained each viewpoint gesture and a homography matrix of each viewpoint induced by the plane and any one reference viewpoint is estimated. Secondly, parallaxes between each viewpoint and the reference viewpoint on the reference plane are calculated by the homography matrix. The relative position of each viewpoint is obtained by means of singular value decomposition (SVD) and calculation of a parallax matrix. When the moving camera virtual array calibration method based on the parallel parallax is used for conducting perspective imaging to a sheltered target in a static scene, a discovering effect is good.

Description

Mobile camera virtual array scaling method based on Parallax

Technical field

The present invention relates to a kind of mobile camera virtual array scaling method, particularly relate to a kind of mobile camera virtual array scaling method based on Parallax.

Background technology

In numerous many viewpoints collecting devices, dense camera array is widely used in and gathers in light field, free view-point imaging and synthetic aperture imaging.Traditional camera array is many to be made of dense fixedly camera of arranging several, and its scaling method mainly contains: complete camera inner parameter and the scaling method of calibrating external parameters and Parallax.

Document " Using Plane+Parallax for Calibrating Dense Camera Arrays; IEEE CVPR2004, Vol.1Page.I-2-I-9 " discloses a kind of structural form of dense camera array and has provided fixedly camera array scaling method based on Parallax (Plane+Parallax) method.This camera array right coplanar 128 of arranging of system are camera and image acquisition node formation fixedly, can be used for the application such as optical field acquisition, image-based drafting Rendering and synthetic aperture imaging.Yet, still there is following inferior position in this camera array system: at first, because the position of camera is fixed, can not take the visual angle or increase new viewpoint according to the characteristics adaptively modifying of captured scene, secondly, because camera quantity is more, the aspects such as system constructing, image acquisition and camera parameter control all need spend more manpower and materials.In the document utilization array system, the coplanar characteristics of each camera photocentre are demarcated camera array.Two the parallel virtual planes with each plane, camera place have been defined in document: reference planes and focussing plane.At first calibration process utilizes the common cooperation of taking of each camera to demarcate the mark estimation should (Homography) projection matrix by the list of being induced by reference planes between the reference camera of each camera and certain appointment.And then to being estimated to the projection matrix (Homology) of focussing plane by reference planes.Point out in document that this projection matrix deteriorates to pure flat moving when camera plane, reference planes and focussing plane are parallel to each other, and can be by the parallax of each camera in reference planes calculated.

Summary of the invention

In order to overcome the existing deficiency that needs many cameras based on the fixedly camera array scaling method of Parallax, the invention provides a kind of mobile camera virtual array scaling method based on Parallax.The method is utilized virtual camera array system of single camera simulation of parallel on line slideway, and the unique point in the captured image of each viewpoint is followed the tracks of and estimated various visual angles camera attitude with this.Each visual angle attitude that utilization obtains, at first define one with line slideway to parallel reference planes, each visual angle that plane is thus induced and the homography matrix of a certain reference viewing angle are estimated.Then this homography matrix calculates each visual angle and reference viewing angle at the parallax of reference planes.The relative position of each viewpoint is by the SVD decomposition computation gained to the parallax matrix.Utilize the inventive method, can carry out perspective imaging to the target that is blocked in static scene, reach the good occlusion effect of going.

The technical solution adopted for the present invention to solve the technical problems is: a kind of mobile camera virtual array scaling method based on Parallax is characterized in comprising the following steps:

Step 1, at first the image sequence of input carried out SIFT feature point detection and coupling, use the RANSAC robust estimation method to carry out exterior point to detected unique point and remove.Then utilize point in characteristics of image correspondence on each visual angle two dimensional image, recover the three-dimensional coordinate of some sparse points in attitude, camera intrinsic parameter and the scene at each visual angle; At last the estimated result of all frames carried out global optimization under the framework that the bundle collection is adjusted.After this step finishes, set up the projection equation of space three-dimensional point on each visual angle image:

x _i＝P _iX

Wherein X is the three-dimensional coordinate of putting under world coordinate system, P _iCamera attitude and Intrinsic Matrix for i visual angle estimating.

Step 2: establish in the motion process of camera on line slideway, each viewpoint photocentre plane of living in is camera plane, and the reference planes parallel with camera plane are ∏ _RBe located in the camera motion process, gathered image k different visual angles altogether, i.e. V ₁..., V _R..., V _k, V wherein _RIt is a reference viewing angle that is used for measuring parallax.If H _iExpression is by ∏ _RInduce from V _iTo V _RHomography matrix.J reference mark XR1 arranged on reference planes ..., X _Rj, utilize the estimated P that goes out of the first step _iMatrix is with X _R1..., X _RjBe projected on each visual angle image, utilize the golden standard algorithm to plane ∏ _RHomography induced is estimated.

Step 3: establish total I different plane and reference planes ∏ _RParallel, be respectively ∏ ₁..., ∏ _I, on each plane, J different reference mark is arranged all, j reference mark X on i plane _ijRepresent, utilize attitude and the camera intrinsic parameter of each viewpoint, with X _ijBe projected to visual angle V _kAnd reference viewing angle V _RImage on, establish this subpoint and be respectively x _ijkAnd x _ijRBy three-dimensional point X _ijThe visual angle V that induces _kWith V _RBetween parallax p _ijkAt reference planes ∏ _ROn be expressed as follows:

p _ijk＝H _ix _ijk-x _ijR

In formula, H _iBe the homography induced that estimates.J on a same plane parallax data is added up ask all, obtain at plane ∏ _iUpward angle of visibility V _kWith V _RBetween mean parallax p _ik

$p_{\mathrm{ik}}=\frac{1}{J}\underset{j=1}{\overset{J}{\mathrm{\Σ}}}{p}_{\mathrm{ijk}}$

Write the mean parallax of K different visual angles on I plane as matrix form, got the parallax matrix M, matrix M is carried out SVD decompose,

In formula, s is scalar factor, Δ x _kBe visual angle V _kWith reference viewing angle V _RRelative position, d _iBe plane ∏ _iWith reference planes ∏ _RRelative distance:

$d_i=\frac{{\mathrm{\Δz}}_i}{Z}$

In formula, Δ z _iBe ∏ _iTo ∏ _RDistance, Z is ∏ _iDistance to camera plane.

The invention has the beneficial effects as follows: due to the virtual camera array system of single camera simulation that utilizes parallel on line slideway, the unique point in the captured image of each viewpoint is followed the tracks of and estimated various visual angles camera attitude with this.Each visual angle attitude that utilization obtains, at first define one with line slideway to parallel reference planes, each visual angle that plane is thus induced and the homography matrix of a certain reference viewing angle are estimated.Then this homography matrix calculates each visual angle and reference viewing angle at the parallax of reference planes.The relative position of each viewpoint is by the SVD decomposition computation gained to the parallax matrix.Utilize the inventive method to carry out perspective imaging to the target that is blocked in static scene, go occlusion effect good.

Below in conjunction with embodiment, the present invention is elaborated.

Embodiment

The mobile camera virtual array scaling method concrete steps that the present invention is based on Parallax are as follows:

1, the various visual angles attitude is estimated.

By image sequence, the camera attitude is estimated and the sparse point in three-dimensional scenic is redeveloped into study hotspot in computer vision always, and the algorithm of existing many maturations, only sketch its estimation procedure here

At first, the image sequence of inputting is carried out automated characterization point detect, then the unique point of interframe is carried out the matching analysis.Owing to may having moving object in scene, some characteristic matching is not correct, therefore need to carry out outer point analysis and removal to the coupling that analyzes.Remaining match point becomes interior point, can be utilized to carry out the estimation of camera attitude.The estimated result of all frames carries out global optimization under the framework that the bundle collection is adjusted at last, obtains globally optimal solution.

2, each camera relative position is estimated.

If in the motion process of camera on line slideway, each viewpoint photocentre plane of living in is camera plane, establishes ∏ _RBe the reference planes parallel with camera plane, at first calibration process should be estimated the list that reference planes are thus induced.Be located in the camera motion process, gathered image k different visual angles altogether, i.e. V ₁..., V _R..., V _k, V wherein _RIt is a reference viewing angle that is used for measuring parallax.If H _iExpression is by ∏ _RInduce from V _iTo V _RHomography matrix.Suppose to have j reference mark on reference planes, as X _R1..., X _Rj, because attitude and the camera intrinsic parameter of each viewpoint are known, this j reference mark is projected to vision point _iWith V _RPixel coordinate on image can calculate by following equation:

x _Ri＝PX _Ri

Wherein P is camera attitude and the Intrinsic Matrix that estimates in the first step.By computer vision knowledge as can be known, when j 〉=4, the corresponding pixel points on available two width images is estimated homography matrix.

This homography matrix can be used for generating the synthetic aperture image that focuses in reference planes, when needs generate composograph on non-reference planes, need to carry out according to different visual angles after the translation of different parallax amount synthetic to image.This parallax amount can be calculated according to the relative position between each viewpoint.The method of estimation of the below's narration to the relative position between each viewpoint.

If total I different and reference planes ∏ _RParallel plane is respectively ∏ ₁..., ∏ _I, on each plane, J different reference mark is arranged all, j reference mark X on i plane _ijRepresent, utilize attitude and the camera intrinsic parameter of each viewpoint, can be with X _ijBe projected to visual angle V _kAnd reference viewing angle V _RImage on, establish this subpoint and be respectively x _ijkAnd x _ijRBy three-dimensional point X _ijThe visual angle V that induces _kWith V _RBetween parallax p _ijkCan be at reference planes ∏ _ROn be expressed as follows:

p _ijk＝H _ix _ijk-x _ijR

H wherein _iBe the homography induced that estimates.J on a same plane parallax data is added up ask all, obtain at plane ∏ _iUpward angle of visibility V _kWith V _RBetween mean parallax p _ik

$p_{\mathrm{ik}}=\frac{1}{J}\underset{j=1}{\overset{J}{\mathrm{\Σ}}}{p}_{\mathrm{ijk}}$

Write the mean parallax of K different visual angles on I plane as matrix form, can be got the parallax matrix M.Owing to there being the proportionate relationship of similar triangles between parallax and camera relative position, so the order of M is 1, matrix M carried out SVD decompose, and can get

Wherein s is scalar factor, Δ x _kBe visual angle V _kWith reference viewing angle V _RRelative position, d _iBe plane ∏ _iWith reference planes ∏ _RRelative distance:

$d_i=\frac{{\mathrm{\Δz}}_i}{Z}$

Δ z wherein _iBe ∏ _iTo ∏ _RDistance, Z is ∏ _iDistance to camera plane.Therefore work as d _iIn the time of all can be measured, the relative position between each visual angle can decompose gained by the SVD to the parallax matrix M.

Claims (1)

1. mobile camera virtual array scaling method based on Parallax is characterized in that comprising the following steps:

Step 1, at first the image sequence of input carried out SIFT feature point detection and coupling, use the RANSAC robust estimation method to carry out exterior point to detected unique point and remove; Then utilize point in characteristics of image correspondence on each visual angle two dimensional image, recover the three-dimensional coordinate of some sparse points in attitude, camera intrinsic parameter and the scene at each visual angle; At last global optimization is carried out in the estimated result of all frames under the framework that the bundle collection is adjusted; After this step finishes, set up the projection equation of space three-dimensional point on each visual angle image:

x _i＝P _iX

Wherein X is the three-dimensional coordinate of putting under world coordinate system, P _iCamera attitude and Intrinsic Matrix for i visual angle estimating;

Step 2: establish in the motion process of camera on line slideway, each viewpoint photocentre plane of living in is camera plane, and the reference planes parallel with camera plane are ∏ _RBe located in the camera motion process, gathered image k different visual angles altogether, i.e. V ₁..., V _R..., V _k, V wherein _RIt is a reference viewing angle that is used for measuring parallax; If H _iExpression is by ∏ _RInduce from V _iTo V _RHomography matrix; J reference mark X arranged on reference planes _R1.., X _Rj, utilize the estimated P that goes out of the first step _iMatrix is with X _R1..., X _RjBe projected on each visual angle image, utilize the golden standard algorithm to plane ∏ _RHomography induced is estimated;

Step 3: establish total I different plane and reference planes ∏ _RParallel, be respectively ∏ ₁..., ∏ _I, on each plane, J different reference mark is arranged all, j reference mark X on i plane _ijRepresent, utilize attitude and the camera intrinsic parameter of each viewpoint, with X _ijBe projected to visual angle V _kAnd reference viewing angle V _RImage on, establish this subpoint and be respectively x _ijkAnd x _ijRBy three-dimensional point X _ijThe visual angle V that induces _kWith V _RBetween parallax p _ijkAt reference planes ∏ _ROn be expressed as follows:

p _ijk＝H _ix _ijk-x _ijR

In formula, H _iBe the homography induced that estimates; J on a same plane parallax data is added up ask all, obtain at plane ∏ _iUpward angle of visibility V _kWith V _RBetween mean parallax p _ik

$p_{\mathrm{ik}}=\frac{1}{J}\underset{j=1}{\overset{J}{\mathrm{\Σ}}}{p}_{\mathrm{ijk}}$

Write the mean parallax of K different visual angles on I plane as matrix form, got the parallax matrix M, matrix M is carried out SVD decompose,

In formula, s is scalar factor, Δ x _kBe visual angle V _kWith reference viewing angle V _RRelative position, d _iBe plane ∏ _iWith reference planes ∏ _RRelative distance:

$d_i=\frac{{\mathrm{\Δz}}_i}{Z}$

In formula, Δ z _iBe ∏ _iTo ∏ _RDistance, Z is ∏ _iDistance to camera plane.