CN113763480A - Multi-lens panoramic camera combination calibration method - Google Patents
Multi-lens panoramic camera combination calibration method Download PDFInfo
- Publication number
- CN113763480A CN113763480A CN202110885803.XA CN202110885803A CN113763480A CN 113763480 A CN113763480 A CN 113763480A CN 202110885803 A CN202110885803 A CN 202110885803A CN 113763480 A CN113763480 A CN 113763480A
- Authority
- CN
- China
- Prior art keywords
- camera
- calibration
- checkerboard
- mpc
- view
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000005457 optimization Methods 0.000 claims description 22
- 230000005484 gravity Effects 0.000 claims description 5
- 238000004422 calculation algorithm Methods 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 10
- 230000000007 visual effect Effects 0.000 description 10
- 238000003384 imaging method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 239000013598 vector Substances 0.000 description 8
- 238000013461 design Methods 0.000 description 7
- 238000013519 translation Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000002370 liquid polymer infiltration Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 239000001993 wax Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Abstract
The invention relates to the technical field of computer vision, and particularly discloses a multi-lens panoramic camera combination calibration method, which comprises the following steps: placing the MPC on a two-dimensional rotary table to carry out rotation photography and calibration on the checkerboard; the calibration process is divided into two continuous stages, namely front-stage and back-stage, wherein the front stage is side-view camera rotation sequence image calibration, and the back stage is central camera multi-view geometric calibration. The method can realize high-precision calibration of the MPC camera external parameter only by using a common two-dimensional numerical control turntable and a single checkerboard, has low cost, simple operation and low requirement on real-time environment, successfully gets rid of the dependence on a 3D calibration field, and has good application value.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a multi-lens panoramic camera combination calibration method.
Background
As an effective monitoring means, cameras have long played an irreplaceable role in social public safety, management scheduling, and production control. The Multi-lens Panoramic Camera MPC (Multi-Head Panoramic Camera), or called Panoramic Camera System OMS (omni-directional Multi-Camera System), is formed by packaging a plurality of traditional monitoring cameras with different visual angles and independent physics, and high-resolution Panoramic videos with 180-degree (or 360-degree) visual field ranges and basically consistent directions are obtained by splicing sub-pictures in real time.
The existing MPC calibration process for acquiring internal and external parameters of a sub-camera relates to two parts of single-camera calibration and combined calibration. The MPC sub-cameras are generally traditional 'pinhole' cameras, and the former research is carried out deeply and has difficulty in obtaining relative external parameters among the sub-cameras through combined calibration. In order to save hardware cost, the MPC usually utilizes a small number of cameras and low-cost optical lenses to obtain a 180-degree or 360-degree visual angle, the overlapped visual angles of adjacent sub-cameras are small, an indoor large-scale calibration field needs to be established in the traditional MPC calibration process, or a small-scale space measuring device and a 2D reference object are jointly utilized to provide high-precision 3D control information for resolving the absolute spatial position and the attitude of the MPC sub-cameras, and further, relative external parameters among the sub-cameras are deduced, and the calibration mode depending on the 3D control information is high in implementation cost and needs professional operation, and undoubtedly forms a limit on MPC application development.
Disclosure of Invention
The invention aims to provide a multi-lens panoramic camera combined calibration method, and aims to solve the technical problems that a calibration mode depending on 3D control information in the prior art is high in implementation cost, needs to be operated by professional personnel, and forms limitation on MPC application development.
In order to achieve the above object, the present invention provides a multi-lens panoramic camera combination calibration method, which comprises the following steps:
placing the MPC on a two-dimensional rotary table to carry out rotation photography and calibration on the checkerboard;
the calibration process is divided into two continuous stages, namely front-stage and back-stage, wherein the front stage is side-view camera rotation sequence image calibration, and the back stage is central camera multi-view geometric calibration.
Wherein, the side-view camera rotation sequence image calibration comprises:
setting and establishing a strict equation of the rotational photography by combining the checkerboard and the coordinate system of the two-dimensional turntable, and giving an initial value of the external parameter of the side-looking camera on the basis of the strict equation;
and then, taking the checkerboard image angular points and the connection points between the adjacent side-view cameras as observed values, and carrying out adjustment optimization solution on the external parameters of the side-view cameras and the external parameters of the checkerboard as a whole.
Wherein the multi-view geometric calibration of the central camera comprises:
translating the origin of the two-dimensional turntable rotating coordinate system to the geometric gravity center of all the side-view camera photographing centers to establish an MPC space coordinate system;
and (3) carrying out adjustment optimization solution of the light beam method on external parameters of the central camera by using a multi-view geometric relation between the central camera and the side-view cameras and a large number of connecting points obtained by rotating photography as observed values on the basis of an MPC space coordinate system.
Wherein, the step of placing the MPC on a two-dimensional rotating table to perform rotation photography and calibration on the checkerboard comprises the following steps:
the MPC is placed on a two-dimensional turntable controlled by a computer, the platform is rotated for a circle at intervals of a fixed angle, checkerboard patterns displayed on an LCD are synchronously shot, a sequence image is obtained, and then calibration is carried out.
The invention discloses a multi-lens panoramic camera combination calibration method, which comprises the steps of placing an MPC (personal computer) on a two-dimensional rotary table to carry out rotary photography on a checkerboard and dividing the calibration process into two stages of side-view camera rotary sequence image calibration. Setting and establishing a strict rotational photography equation by combining the checkerboard and the coordinate system of the rotary table, providing an initial value of external parameters of the side-looking camera on the basis of the strict rotational photography equation, and further performing adjustment optimization solution on the external parameters of the side-looking camera and the whole external parameters of the checkerboard by taking the image corner points of the checkerboard and the connection points between adjacent side-looking cameras as observed values; and (4) multi-view geometric calibration of the central camera. And translating the origin of the rotating coordinate system of the turntable to the geometric center of gravity of the photographing center of all the side-view cameras to establish an MPC space coordinate system, and performing beam adjustment optimization solution on the external parameters of the center camera by using the multi-view geometric relationship between the center camera and the side-view cameras and a large number of connecting points obtained through rotation photographing as observed values on the basis. Experimental results show that the method can realize high-precision calibration of the MPC camera external parameters only by using a common two-dimensional numerical control turntable and a single checkerboard, is low in cost, simple to operate and low in requirements on a real-time environment, successfully gets rid of dependence on a 3D calibration field, and has good application value.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic representation of the MPC geometry of the present invention.
FIG. 2 is a schematic diagram of an MPC spherical panoramic imaging model of the present invention.
FIG. 3 is a flowchart illustrating the steps of the calibration method of the multi-lens panoramic camera.
FIG. 4 is a schematic diagram of MPC rotation photography of the present invention.
Fig. 5 is a schematic diagram of a five-lens panoramic camera and a numerical control 2D turntable of the present invention.
FIG. 6 is a schematic diagram of PanoCam checkerboard calibration image acquisition in accordance with the present invention.
Fig. 7 is a schematic diagram of the error between the set rotation angle and the calibration value of the turntable.
Fig. 8 is a schematic diagram of the inventive PanoCam side view camera rotation calibration residual variation.
Fig. 9 is a schematic diagram of the PanoCam center camera external parameter optimization calculation of the present invention.
Fig. 10 is a schematic view of the same-name image points of the center-side view camera at rest of the turret.
Fig. 11 is a PanoCam spherical panoramic video output comparison schematic diagram under different calibration parameters.
Detailed Description
Referring to fig. 1 to 11, the present invention provides a calibration method for a multi-lens panoramic camera assembly, which includes the following steps:
placing the MPC on a two-dimensional rotary table to carry out rotation photography and calibration on the checkerboard;
the calibration process is divided into two continuous stages, namely front-stage and back-stage, wherein the front stage is side-view camera rotation sequence image calibration, and the back stage is central camera multi-view geometric calibration.
The side-view camera rotation sequence image calibration comprises:
setting and establishing a strict equation of the rotational photography by combining the checkerboard and the coordinate system of the two-dimensional turntable, and giving an initial value of the external parameter of the side-looking camera on the basis of the strict equation;
and then, taking the checkerboard image angular points and the connection points between the adjacent side-view cameras as observed values, and carrying out adjustment optimization solution on the external parameters of the side-view cameras and the external parameters of the checkerboard as a whole.
The multi-view geometric calibration of the central camera comprises the following steps:
translating the origin of the two-dimensional turntable rotating coordinate system to the geometric gravity center of all the side-view camera photographing centers to establish an MPC space coordinate system;
and (3) carrying out adjustment optimization solution of the light beam method on external parameters of the central camera by using a multi-view geometric relation between the central camera and the side-view cameras and a large number of connecting points obtained by rotating photography as observed values on the basis of an MPC space coordinate system.
The step of placing the MPC on a two-dimensional rotating table to carry out rotating photography and calibration on the checkerboard comprises the following steps:
the MPC is placed on a two-dimensional turntable controlled by a computer, the platform is rotated for a circle at intervals of a fixed angle, checkerboard patterns displayed on an LCD are synchronously shot, a sequence image is obtained, and then calibration is carried out.
In the multi-lens panoramic camera combination calibration method,
the MPC monitoring visual angle adopting a hoisting (or wall-mounted) mode can cover the whole hemisphere, and the geometric structure design of the MPC monitoring visual angle mainly adopts a '1 + N' mode: "1" represents 1 central camera, whose main optical axis is designed (approximately) perpendicular to the mounting base (plane); "N" represents N side-view cameras, which are annularly and equally spaced around the center camera. Fig. 1 shows a schematic of a mode of "1 + 4" commonly adopted in the structural design of a commercial MPC camera in the current security market, and the subsequent spherical panoramic imaging model construction and the rotary calibration formula derivation work are all developed based on the mode, but can be expanded according to the number N of different side-looking cameras.
MPC sub-cameras are typically conventional "pinhole" cameras, with a field of view greater than (or equal to) 180 ° obtained by spherical perspective re-projection of each sub-camera video. As shown in FIG. 2, let Ci(I is 0,1,2,3,4) represents the MPC sub-camera shooting center, I is a spherical panoramic image, O and r are the projection spherical center and radius respectively, and assuming that the world coordinate system XYZ origin coincides with the spherical projection center, any space point P in the MPC view fieldWThe spatial transformation to spherical panoramic image pixel p' can be described as the following three-stage process:
a) single camera "pinhole" imaging, i.e. MPC to a sub-camera CiFor space point P under world coordinate systemW(X, Y, Z) obtaining an image point p (X, Y) by fluoroscopic imaging, the geometric imaging relationship being described by a photogrammetric collinearity equation having:
wherein: (X, Y, Z) is a point PWSpace coordinates (u, v) are PWThe actual pixel coordinates of the projection point p; (X)S,YS,ZS) For the position of the camera's camera center in the world coordinate system, (a)j,bj,cj) (j ═ 1,2,3) rotation matrix elements given for the camera principal optic attitude angle (phi, omega, kappa), (X) andS,YS,ZS) (phi, omega, kappa) together with camera extrinsic parameters; (u)0,v0) Is the coordinate of the principal point of the camera, f is the equivalent focal length of the camera, and (Deltax, Deltay) is the optical distortion error of the lens of the camera, which can be considered as the main radial distortion coefficient k1,k2The decision is as follows:
Δx＝(u-u0)×(k1r2+k2r4)，Δy＝(v-v0)×(k1r2+k2r4)，r2＝(u-u0)2+(v-v0)2
above (f, u)0,v0,k1,k2) Collectively called camera intrinsic parameters, when an optical distortion correction is performed on an image point p and a principal point of a camera is taken as a coordinate origin (here, called an ideal image point, the same applies), equation (1) can also be expressed as the following spatial transformation:
Wherein: lambda [ alpha ]PThe scaling coefficient of the image point p is, and R is a rotation matrix; [ x, y, -f ]]TImage space coordinates representing an ideal image point p[16]The origin of the space coordinate system is located at the photographing center S, the photographing main optical axis is taken as the Z axis, the image plane is taken as the xoy plane, the attitude angle (phi, omega, kappa) of the photographing main optical axis is defined as the (continuous) rotation angle around the world coordinate axes X, Y and Z, and the vector formed by the image space coordinate of the ideal image point p and the photographing center
Reflecting the incident ray (direction) of the image point.
b) Sphere remapping, i.e. MPC sub-camera CiIdeal image point P obtained by perspective imaging and backlight PWCiMapping to a set projection sphere to obtain a spatial point PO(XO,YO,ZO) The space point should satisfy the projection spherical equation and the space linear equation of the incident ray, namely:
r2＝XO 2+YO 2+ZO 2 (3)
wherein: (X)O,YO,ZO) Is the spherical space point coordinate of the point P in the world coordinate system; (V)X,VY,VZ)TThe vector of the space linear direction of the incident ray of the point P can be obtained by rotating the image space coordinate of the point P, LPIs the parameter of the linear parameter equation of the space where the incident ray of the point P is located, where the linear parameter equation of the space where the incident ray of the point P is located is defined as the sub-camera CiRay with the shooting center as the starting point and passing through the ideal image point P, and parameter LPThe calculation formula is as follows:
c) spherical panoramic image synthesis, i.e. spherical spatial point POAnd projecting the panoramic image to the plane according to the selected model. The existing spherical projection model is divided into four types: equidistant projection, equal solid angle projection, stereoscopic projection and orthogonal projection are adopted, and considering that an orthogonal projection model is simple to calculate and can establish a reversible transformation relation between a space point and a spherical panoramic image point, namely, as shown in figure 2, a point P is divided into two partsOOrthographic projection onto a panoramic image plane perpendicular to the Z-axis) to obtain pixel coordinates p ' (x ', y ') transformed as follows:
x′＝XO,y′＝YO (6)
specifically, in the video calibration of the video camera rotation sequence, because it is difficult to strictly satisfy the design condition that the photographing center of each sub-camera coincides with the center of the projection sphere, the projection of the homonymous pixels in the overlapping area of the MPC sub-camera onto the panoramic image plane will have a position error, the error size is related to the selection of the spherical projection parameters (center position and radius size), the geometric center of gravity of all the sub-camera photographing centers is usually taken as the spherical projection center, the spherical projection radius is an empirical value related to the depth of field, and the spherical projection parameters herein adopt the above setting mode.
As can be seen from the above spherical panoramic imaging process, MPC complete parameter calibration using the "1 + N" design mode should include two parts: 5(1+ N) camera internal parameters IOPs, 6(1+ N) camera external parameters EOPs. At present, the internal and external parameters of a single camera can be conveniently solved by utilizing 2D checkerboard multiview, and the difficulty lies in how to simply and quickly obtain the external parameters of the MPC sub-camera, and the reason is in two aspects: firstly, the common checkerboard is difficult to be simultaneously seen by adjacent sub-cameras of the MPC and cannot form effective calibration control, and secondly, the MPC has small overlapped view angle of each sub-camera and is difficult to provide a large amount of redundant observation (same-name characteristics) for the solution of the external reference optimization. In order to solve the above problems, it is assumed that the internal parameters of each sub-camera of the MPC are known, the MPC is placed on a two-dimensional rotating platform to perform combined calibration by performing rotating photography on a single checkerboard, and the MPC combined calibration is divided into two stages of side-view camera rotating sequence image calibration and central camera multi-view geometric calibration to be sequentially implemented by combining the structural design characteristics of the MPC and the rotating photography geometric characteristics thereof.
Rotating photography and setting a coordinate system:
as shown in FIG. 4(a) for the acquisition of calibration images, a sequence of images can be obtained by placing the MPC on a horizontal rotating platform controlled by a computer, rotating the platform at regular angular intervals for one revolution and simultaneously capturing checkerboard patterns displayed on the LCD
Wherein: i denotes the i-th sub-camera C of MPCi(ii) a j denotes the sub-camera CiThe j-th image is acquired at a rotation angle j β, where N is 360 °/β.
As shown in FIG. 4(b) representing a rotational coordinate system setup, the world coordinate system O is first established hereinW-XWYWZWAnd the coordinate system O of the checkerboard objectP-XPYPZPThe definition is as follows: world coordinate system origin OWAt the center of rotation of the turntable plane, ZWThe axis coincides with the axis of rotation (perpendicular to the plane of the turntable, upwards in direction), XWWith axes in the plane of the turntable and approximately parallel to the central camera imageLike the horizontal direction, YWThe axis is given according to the right hand rule; origin O of coordinate system of checkerboard objectPAt the lower left corner of the checkerboard, the checkerboard plane is defined as OP-XPYPPlane, XPAxis and YPThe axis is respectively parallel to the horizontal and vertical directions, Z of the checkerboardPAxes are given according to the right hand rule. The LCD checkerboard is placed so that its horizontal direction is approximately parallel to the horizontal direction of the central camera image (i.e., X) in view of the actual shooting conditions and parameter initializationPAxis and XWAxes are approximately parallel), then the checkerboard corner point (X) is found when the turntable is in a stationary stateP,YP) Spatial coordinates (X) in the world coordinate systemW,YW,ZW) Can be expressed as:
in the formula: (X)P,YP0) is the angular point of the checkerboard at OP-XPYPZPSpace coordinates of (1), R (phi, omega, K) and T (X)0,Y0,Z0) Respectively a rotation matrix and a translation vector from the checkerboard object coordinate system to the world coordinate system. It is easy to understand that MPC shoots static checkerboard when revolving stage rotates and MPC shoots around Z when revolving stage is staticWThe axis rotation is equivalent to a checkerboard if the horizontal plane O of the world coordinate system is usedW-XWYWZWAlong ZWThe axis is raised (or lowered) so that the lower left corner of the checkerboard falls on a horizontal plane, then equation (7) can be rewritten as:
further, in the formula:
when the rotating angle j is beta, the spatial coordinates of the checkerboard angular points in a world coordinate system;
representing a matrix of rotation of the checkerboard about the Z-axis of the turntable.
Without loss of generality, order
Representing MPC sub-camera CiThe exterior orientation element in the world coordinate system can be C if the checkerboard rotates by an angle j beta after the rotating platform rotates by an angleiComplete observation, then CiThe coordinate mapping relationship between the checkerboard image corner points and the space points thereof is as follows:
in the formula: (x, y, -f)TDefined by the formula (2), RiIs a camera CiAn exterior orientation corner matrix in the world coordinate system. The combination of formulae (8) and (9) gives:
equation (10) is the unified rotational photography equation for MPC herein with respect to checkerboard references. Under the static condition of the MPC, due to the problem of visual angles, the same checkerboard is difficult to shoot simultaneously for adjacent side-looking cameras of the MPC, so that control information cannot be provided for calibrating and calculating relative attitude parameters between the side-looking cameras, and the MPC unified rotation photography equation given by the formula (10) can be regarded as expanding the checkerboard control information under the fixed visual angles to a 360-degree spatial range through the rotation of the platform, thereby laying a foundation for the initialization of external parameters of different side-looking cameras of the subsequent MPC and the overall optimization estimation of the checkerboard control information.
Parameter initialization and bundle adjustment:
the MPC side-view camera external reference calibration under the world coordinate system (namely when the turntable is static) is obtained by minimizing the checkerboard corner point reprojection error and the homonymous pixel reprojection error of the overlapping area of the adjacent side-view cameras, and the key point is to provide an ideal external reference initial value. For simplicity of calculation and explanation, the side-view camera image pixels are defined as ideal image points and are obtained by camera internal reference calculation.
Under the known conditions of the internal parameters of the camera, the external parameters of the camera calibration image under the checkerboard object coordinate system can be estimated by utilizing direct linear transformation DLT and checkerboard corner point information, and the external parameter initial values of the side-looking camera under the world coordinate system can be given out by combining the parameters with an MPC unified rotation photographic equation. As can be seen from the MPC coordinate system setup, if the checkerboard plane is approximately perpendicular to the turntable plane, i.e. taking (Φ, Ω, K) ═ 0, pi/2, 0, equation (10) can be simplified as:
when the rotation angle of the platform is j × β, if one side of the MPC views the camera image
The checkerboard angular point space point (X) can be observedP,YP0) and the corresponding pixel is (x, y), then the following formula (11) is obtained:
in the formula: ri,jSide view camera C for MPCiCheckerboard image
Rotation matrix of attitude angles, Ti,jIs a checkerboard image
The translation vectors are the external parameters of the image output by Zhang Zhengyou calibration algorithm and under the coordinates of the checkerboard object, and then the formula (12) is used for deriving the side-looking camera CiInitial values of external parameters when the turntable is stationary. Angle j beta is known for CiAttitude angle element (phi)i,ωi,κi) From R may bei,jAnd
the product result is decomposed to obtain[16]Taking the average value of the decomposition results of the images of different checkerboards; for the photographic center
Then by Ti,jTranslation vector T (X) of the coordinate system of the checkerboard object relative to the world coordinate system0,Y0And 0) given jointly, the specific calculation formula is as follows:
equation (13) is a linear equation, and for a single side-view camera, the photographing center and corresponding rotation angle of at least 2 images in the checkerboard object coordinate system are known, and then the linear equation can be solved
For all 4 side-looking cameras, each camera provides at least 1 image of the photographing center and the corresponding rotation angle in the checkerboard coordinate system, and the photographing center extrinsic parameter initial value and the translation vector T of each side-looking camera can be simultaneously and linearly solved. Considering that any side-view camera can obtain a considerable amount of checkerboard images during MPC rotation photography, the ideal initial value of the checkerboard translation vector T externally referred to the photography center of the side-view camera is obtained through least square estimation.
Let VFRepresenting the reprojection error, V, of checkerboard space points on the sequence imageGRepresenting the reprojection error of the homonymous image points in the image overlapping area of the adjacent side-looking cameras, and the objective equation for carrying out global optimization on the external parameters of the side-looking cameras and the primary values of the checkerboard translation vectors is represented as follows:
in the formula
Is a side view camera CiExterior orientation element, pi,jIs the s corner point of the checkerboard
At camera CiThe image point coordinates on the sequence image j, M is the number of space points, and N is the sub-camera CiNumber of sequential images, p'i,jTo obtain checkerboard corner points at camera C according to calibration parametersiReprojection of coordinates, k, on sequential image ji,jIndicating that the jth image of the adjacent camera corresponds to the image point serial number of the same name,
representing adjacent cameras CiAnd Ci+1At a corner j*Beta time image pair
And
number of pixels of the same name, G (ζ)i,ζi+1) Representing the homonymous image point coplanarity equation residuals of adjacent cameras. VFFrom photogrammetry collinearity equation [16]Given, the calculation formula is:
equation (15) is nonlinear, and the error equation can be obtained by expanding the nonlinear equation in a Taylor series and taking one term:
VGaccording to the coplanar condition G between the homonymous image point and the photographing base line in the binocular vision, the method comprises the following steps:
in the formula:
are respectively images
And
the coordinates of the image points with the same name. Equation (17) is nonlinear, and the error equation can be obtained by expanding the nonlinear equation in a Taylor series and taking one term:
and (4) according to the minimization formula (14) of the indirect adjustment principle and iterating to a preset convergence condition, so that the optimal estimation of the rotation and translation parameters of all external parameters of the MPC side view camera and the coordinate system of the checkerboard object and the world coordinate system of the rotating platform can be obtained.
Multi-view geometric calibration of a central camera:
as can be seen from the above-mentioned rotational photography process, the MPC center camera cannot photograph the checkerboard; on the other hand, the MPC center camera and all the side-view cameras are overlapped, so that the outer parameters of the center camera can be optimized and solved by using multi-view geometric constraints between all the side-view cameras and the center camera on the premise of accurately calibrating the inner and outer parameters of the side-view cameras, the potential problem is that the overlapping area of the two is small and effective homonymous features (image points) are difficult to provide as redundant observation, and fortunately, the problem can be effectively solved by defining a new MPC world coordinate system and a rotating photography process under the coordinate system.Without loss of generality, let (x)0,y0) And (x)i,yi) Representing MPC center camera C0To a side view camera Ci(i is 1,2,3,4) shooting the same-name (ideal) image point in the image overlapping region under the world coordinate system (when the rotary table is still), and the same-name image point and the shooting baseline satisfy the coplanarity condition:
further, when the turntable rotates by an angle j beta, C0And CiThe same (ideal) image points of (b) also satisfy the coplanarity condition, should:
the finishing formula (20) gives:
due to the fact that
As shown in the formula (21), when the turntable is rotated, C0And CiThe coplanar condition between the two is only related to the coordinates of the homonymous image points and the shooting baseline (external reference) of the two in a static state, which means that more homonymous features (image points) can be obtained by rotating shooting as observed values for the purpose of estimating the parameters of the central camera under the coplanar constraint. The non-linearity of equation (22) is developed in Taylor series and a term is takenThe error equation can be obtained:
and (3) taking the same-name (actual) image points of the central camera and all side-view cameras under different rotation angles as observed values, and iterating to preset convergence conditions according to an indirect adjustment principle minimization formula (22) to obtain the optimal estimation of the external parameters of the central camera, wherein: the geometric barycenter coordinates of the shooting centers of the 4 side-view cameras are referred to outside the shooting center, and the initial value of the external reference angle element is 0 because the MPC world coordinate system is approximately parallel to the central camera coordinate system.
Here, a PC (Intel E5-1620 processor)&win10 operating system&VS2010 compilation environment) to implement the above calibration algorithm. The algorithm verification adopts a five-lens camera PanoCam and a common numerical control turntable (angle repositioning error [ -0.5, +0.5 ] of a certain company]Degree), see fig. 5(a) and (b), the checkerboard drawn on the LCD is rotated and photographed under the indoor environment to obtain the rotation sequence image required for algorithm processing, the MPC camera inside and outside parameters, the checkerboard orientation parameters and the turntable rotation angle β under the test conditionjAll the parameters are unknown parameters, but the internal parameters of the MPC sub-camera are calculated by Zhang Zhengyou algorithm and are regarded as known values, and all the calibration parameters are used for outputting MPC spherical panoramic video of an actual scene and comparing the MPC spherical panoramic video with the video output by the camera self-contained software. When the scheme is used for rotating photography, the MPC is approximately placed in the center of the (virtual) turntable, and each sub-camera of the MPC shoots 40 images when the turntable rotates for one circle
Shooting one image every 9 degrees (beta is 360 degrees/40), controlling the rotation speed to ensure that the overlapping degree of adjacent images in 40 sequence images of the same side-view sub-camera is not less than 60 percent, the number of images for completely observing the checkerboard by the side-view camera is 5, and the number of images for completely observing the checkerboard by the central camera is 0; the overlapping degree of the video images of the adjacent side-view cameras of the MPC is about 10%, and the overlapping degree of the video images of the side-view cameras and the central camera is about 10%.
Each sub-camera of PanoCam adopts the same module (the internal reference size is close), the image breadth size is the same (1280 × 960 pixels), the calibrated image required by the algorithm is obtained by the PanoCam on the LCD checkerboard (the grid size is 11 × 11, the grid distance is 20.32mm) rotating photography placed on the turntable shown in fig. 5(b), and the method is shown in fig. 6, wherein: FIG. 6(a) is a schematic view of a shooting scene; fig. 6(b) is a video image output by PanoCam when the turntable is stationary; fig. 6(C) and 6(d) are image sequences obtained by the rotational photography by the center camera C0 and the side camera C1, respectively. The LCD checkerboard object coordinate system and the world coordinate system are set according to the above, and the same-name image points of the calibration image required by optimization calculation are obtained by utilizing SIFT operators to automatically match and wild points are removed by adopting an RANSANC algorithm. Table 1 lists PanoCam camera internal parameters given by the zhangnyou algorithm, and table 2 lists external parameters of each side-view camera calibration image in the checkerboard object coordinate system and the checkerboard corner point average reprojection error RMSE _0 under the parameters.
TABLE 1 PanoCam Camera reference statistics
TABLE 2 statistics of PanoCam side-view camera calibration image external reference and angular point average re-projection error under checkerboard object coordinate system
PanoCam calibration first initializes the turn angle βjThe initial values of the external parameter and the checkerboard orientation parameter of the side-view camera in the world coordinate system are given in table 3, and table 3 simultaneously gives the reprojection error RMSE _0 of the checkerboard corner point under the parameter and the coplanar error (referred to as relative orientation error) RMSE _1 of the same-name image point of the adjacent side-view camera in table 3. As can be seen from Table 3, the accuracy of the turntable is not high and it is difficult to satisfy the setting conditions of the coordinate system of the checkerboard object (the checkerboard plane is perpendicular to the plane of the turntable, the X axis of the coordinate system of the checkerboard object is parallel to the world)Coordinate system X axis), the lateral-view camera external parameters and the checkerboard orientation parameter values given by equations (12) - (13) have inevitable errors, which are reflected in that RMSE _0 (about 1.7 pixels) is significantly higher than the average reprojection error of the checkerboard corner points of the calibration image of each lateral-view camera in table 2, and the high error RMSE _1 of about 9 pixels laterally emphasizes the necessity of camera calibration by using the same-name image point constraint of the calibration image.
TABLE 3 statistics of initial values and accuracies of external parameters and checkerboard orientation parameters of side-looking cameras before optimization
Taking projection coordinates of the checkerboard corner point image and coordinates of the same-name image points of the adjacent cameras as observation values, and giving an initial external reference optimization result in table 3 by the algorithm in table 4, it can be seen that the total error RMSE under the optimization parameters is 0.39, about one third of pixels reaches the requirement of high calibration precision, wherein: the reprojection error RMSE _0 of the checkerboard corner points is 0.30, which is close to the average reprojection error of the checkerboard corner points of the calibration image of each side-view camera in the table 2, and the relative orientation error RMSE _1 of the homonymous image points of the adjacent side-view cameras is greatly reduced to 1.38. Fig. 7 shows the error between the corner calibration value of all side view camera checkerboard images (4 × 5 — 20 pieces in total) of PanoCam and the rotational photography design angle thereof, the root mean square error of all image corners is about 0.004 radians (about 0.23 degrees), the maximum error is about 0.006 radians (about 0.35 degrees), the error is consistent with the angular repeated positioning accuracy of the turntable, and the effectiveness of the calibration algorithm is verified from the side; fig. 8 further shows the residual variation curve of the PanoCam side view camera rotation calibration, which can be converged to the preset condition after 9-10 iterations.
TABLE 4 optimized side view camera external parameters and checkerboard orientation parameters and precision statistics
Table 5 lists initial values of external parameters of the PanoCam center camera and the parameter optimization results thereof under the multi-view geometric constraint, and fig. 9 shows a specific parameter and residual optimization calculation process, which can be converged to a preset condition after 9-10 iterations, thus having good calculation efficiency. As can be seen from Table 5, the relative orientation error RMSE _1 between the same-name image points of the images of the central camera and the side-view camera during the parameter optimization convergence is less than 1 pixel, so that the calibration precision is higher and better than that of the side-view camera rotation calibration RMSE _1 in Table 4, and the conclusion consistent with the simulation imaging simulation test is obtained. Fig. 10 shows the homonymous image points between the images of the center camera and the side-view camera at the stationary time of the turntable in fig. 6(b), and it can be found that, because the image overlapping degree is small and the relative visual angle change is large, the number of homonymous image points obtained by the SIFT operator is rare (partial rotation angle image matching or even complete failure), so the observed values provided by the homonymous image points of the single-angle image are limited, fortunately, by defining a new MPC spatial coordinate system and a rotation photography process under the coordinate system, more homonymous image points of different rotation angle images can be used as redundant observed values, thereby laying a foundation for the application of the multi-view geometric constraint of the center camera and the smooth implementation of the adjustment optimization calculation by the beam method.
TABLE 5 PanoCam's Central Camera external parameter initial values and optimization results
When the PanoCam equipment leaves a factory, the internal and external parameters of the camera of the PanoCam equipment are obtained through high-precision three-dimensional calibration field calibration, and are stored in self-contained commercial software in an encrypted file mode for outputting a panoramic video in real time, and the panoramic video output by the calibration parameters is contrasted and analyzed because the parameters cannot be directly compared. The panoramic video is generated in the above manner and is not smoothed, the PanoCam self-contained software outputs the panoramic video for smoothing, see fig. 11, as can be seen from comparing fig. 11(a) and 11(b), the output panoramic video quality of the PanoCam calibration parameters is better, and the parallax artifact of the video overlapping region of the PanoCam self-contained software is more obvious (marked by a rectangular frame), the reason is that the output quality of the MPC panoramic video is affected by the misalignment of the camera shooting center, the change of the scene depth and the potential camera calibration error, one of the basis of the MPC external parameter optimization is to minimize the reprojection error of the same-name pixels of the adjacent cameras reflecting the change of the scene depth, so the parallax artifact of the video overlapping region can be effectively overcome, the PanoCam commercial software camera parameters are obtained under the calibration environment of the specific scene depth, and the camera shooting center of the MPC camera cannot well adapt to the new scene depth change, this also laterally demonstrates the accuracy of the calibration parameters herein.
The experimental and analytical results show that:
the strict equation of the rotational photography established by comprehensively considering the coordinate system of the checkerboard object and the coordinate system of the turntable world is reasonable, not only can the single checkerboard calibration control range be expanded, but also good initial values of the external parameters of the side-looking camera can be provided, and therefore stable and reliable solving of the external parameters of the MPC side-looking camera is ensured.
The method has the advantages that the method is effective in solving the external parameters of the central camera by using the MPC multi-view geometric relationship, and meanwhile, the defect that effective homonymous image points cannot be provided as observed values due to small overlapping visual fields of the central camera and a single side-view camera is overcome by rotary photography, so that the stable and high-precision solution of the external parameters of the MPC central camera is ensured.
The MPC external reference is combined and calibrated by combining a common two-dimensional turntable and a checkerboard calibration reference object, the application limit that the traditional MPC external reference estimation depends on a 3D calibration field is successfully eliminated, and the panoramic video output effect is comparable to the high-precision three-dimensional calibration field calibration parameters.
In summary, the conventional MPC combination calibration depends on high-precision 3D control information, is high in implementation cost and needs to be operated by a professional, and forms a limitation on MPC application development. The invention provides a novel MPC checkerboard combination calibration method combined with rotary photography by taking LCD checkerboards as calibration reference objects, wherein the calibration control range of a single 2D checkerboard reference object to MPC is expanded through controllable rotary photography and connection points between MPC sub-cameras are increased to serve as redundancy observation values, MPC is placed on a common two-dimensional turntable to realize strict solution of MPC external parameters only through one-circle rotary imaging, the method is low in cost, high in precision, simple to operate, small in dependence on implementation conditions, and comparable to high-precision three-dimensional calibration field calibration parameters in panoramic video output effect, and is an ideal MPC combination calibration method. In the calibration process, the internal parameters of the MPC sub-camera are given by using a Zhangyingyou calibration algorithm and serve as known values, but a multi-view chessboard calibration image meeting the requirements of the Zhangyingyou calibration algorithm can be obtained through reasonable photographing control design (such as increasing the number of rotation cycles of a turntable and changing the angle of an LCD relative to the plane of the turntable), and from this point, the method can realize the complete calibration of the internal and external parameters of the MPC camera.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (4)
1. A multi-lens panoramic camera combination calibration method is characterized by comprising the following steps:
placing the MPC on a two-dimensional rotary table to carry out rotation photography and calibration on the checkerboard;
the calibration process is divided into two continuous stages, namely front-stage and back-stage, wherein the front stage is side-view camera rotation sequence image calibration, and the back stage is central camera multi-view geometric calibration.
2. The multi-lens panoramic camera combination calibration method according to claim 1, wherein the side view camera rotation sequence image calibration comprises:
setting and establishing a strict equation of the rotational photography by combining the checkerboard and the coordinate system of the two-dimensional turntable, and giving an initial value of the external parameter of the side-looking camera on the basis of the strict equation;
and then, taking the checkerboard image angular points and the connection points between the adjacent side-view cameras as observed values, and carrying out adjustment optimization solution on the external parameters of the side-view cameras and the external parameters of the checkerboard as a whole.
3. The multi-lens panoramic camera combined calibration method according to claim 1, wherein the central camera multi-view geometric calibration comprises:
translating the origin of the two-dimensional turntable rotating coordinate system to the geometric gravity center of all the side-view camera photographing centers to establish an MPC space coordinate system;
and (3) carrying out adjustment optimization solution of the light beam method on external parameters of the central camera by using a multi-view geometric relation between the central camera and the side-view cameras and a large number of connecting points obtained by rotating photography as observed values on the basis of an MPC space coordinate system.
4. The method for calibrating a multi-lens panoramic camera assembly as claimed in claim 1, wherein said positioning the MPC on the two-dimensional rotating table to perform the rotating photography and calibration of the checkerboard comprises:
the MPC is placed on a two-dimensional turntable controlled by a computer, the platform is rotated for a circle at intervals of a fixed angle, checkerboard patterns displayed on an LCD are synchronously shot, a sequence image is obtained, and then calibration is carried out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110885803.XA CN113763480A (en) | 2021-08-03 | 2021-08-03 | Multi-lens panoramic camera combination calibration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110885803.XA CN113763480A (en) | 2021-08-03 | 2021-08-03 | Multi-lens panoramic camera combination calibration method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113763480A true CN113763480A (en) | 2021-12-07 |
Family
ID=78788413
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110885803.XA Pending CN113763480A (en) | 2021-08-03 | 2021-08-03 | Multi-lens panoramic camera combination calibration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113763480A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200298769A1 (en) * | 2017-12-01 | 2020-09-24 | Lg Innotek Co., Ltd. | Method and device for correcting camera |
| CN116977449A (en) * | 2023-09-25 | 2023-10-31 | 安徽大学 | Compound eye event camera active calibration method based on flicker checkerboard |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654502A (en) * | 2016-03-30 | 2016-06-08 | 广州市盛光微电子有限公司 | Panorama camera calibration device and method based on multiple lenses and multiple sensors |
| WO2018076154A1 (en) * | 2016-10-25 | 2018-05-03 | 成都通甲优博科技有限责任公司 | Spatial positioning calibration of fisheye camera-based panoramic video generating method |
| CN108171758A (en) * | 2018-01-16 | 2018-06-15 | 重庆邮电大学 | Polyphaser scaling method based on minimum time principle and transparent glass scaling board |
| CN110244282A (en) * | 2019-06-10 | 2019-09-17 | 于兴虎 | A kind of multicamera system and laser radar association system and its combined calibrating method |
-
2021
- 2021-08-03 CN CN202110885803.XA patent/CN113763480A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654502A (en) * | 2016-03-30 | 2016-06-08 | 广州市盛光微电子有限公司 | Panorama camera calibration device and method based on multiple lenses and multiple sensors |
| WO2018076154A1 (en) * | 2016-10-25 | 2018-05-03 | 成都通甲优博科技有限责任公司 | Spatial positioning calibration of fisheye camera-based panoramic video generating method |
| CN108171758A (en) * | 2018-01-16 | 2018-06-15 | 重庆邮电大学 | Polyphaser scaling method based on minimum time principle and transparent glass scaling board |
| CN110244282A (en) * | 2019-06-10 | 2019-09-17 | 于兴虎 | A kind of multicamera system and laser radar association system and its combined calibrating method |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200298769A1 (en) * | 2017-12-01 | 2020-09-24 | Lg Innotek Co., Ltd. | Method and device for correcting camera |
| US11548452B2 (en) * | 2017-12-01 | 2023-01-10 | Lg Innotek Co., Ltd. | Method and device for correcting vehicle view cameras |
| CN116977449A (en) * | 2023-09-25 | 2023-10-31 | 安徽大学 | Compound eye event camera active calibration method based on flicker checkerboard |
| CN116977449B (en) * | 2023-09-25 | 2024-03-15 | 安徽大学 | Compound eye event camera active calibration method based on flicker checkerboard |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2018076154A1 (en) | Spatial positioning calibration of fisheye camera-based panoramic video generating method | |
| CN111062873B (en) | Parallax image splicing and visualization method based on multiple pairs of binocular cameras | |
| TWI555379B (en) | An image calibrating, composing and depth rebuilding method of a panoramic fish-eye camera and a system thereof | |
| CN111028155B (en) | Parallax image splicing method based on multiple pairs of binocular cameras | |
| TWI555378B (en) | An image calibration, composing and depth rebuilding method of a panoramic fish-eye camera and a system thereof | |
| US6643396B1 (en) | Acquisition of 3-D scenes with a single hand held camera | |
| CN104537707B (en) | Image space type stereoscopic vision moves real-time measurement system online | |
| CN103115613B (en) | Three-dimensional space positioning method | |
| US20150116691A1 (en) | Indoor surveying apparatus and method | |
| CN107358633A (en) | Join scaling method inside and outside a kind of polyphaser based on 3 points of demarcation things | |
| CN110827392B (en) | Monocular image three-dimensional reconstruction method, system and device | |
| CN113763480A (en) | Multi-lens panoramic camera combination calibration method | |
| Yang et al. | A calibration method for binocular stereo vision sensor with short-baseline based on 3D flexible control field | |
| CN105374067A (en) | Three-dimensional reconstruction method based on PAL cameras and reconstruction system thereof | |
| CN113450416B (en) | TCSC method applied to three-dimensional calibration of three-dimensional camera | |
| CN108898550B (en) | Image splicing method based on space triangular patch fitting | |
| Nagy et al. | Development of an omnidirectional stereo vision system | |
| CN113487683B (en) | Target tracking system based on trinocular vision | |
| Benosman et al. | A new catadioptric sensor for the panoramic vision of mobile robots | |
| CN111583117A (en) | Rapid panoramic stitching method and device suitable for space complex environment | |
| US11943539B2 (en) | Systems and methods for capturing and generating panoramic three-dimensional models and images | |
| CN106934861B (en) | Object three-dimensional reconstruction method and device | |
| US20240046596A1 (en) | Image splicing method, computer-readable storage medium, and computer device | |
| JP2005275789A (en) | Three-dimensional structure extraction method | |
| CN111091595B (en) | Strabismus three-dimensional mapping method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |