DE10340023B3 - Self-calibration of camera system involves inserting matrices describing transformations into equation giving equation system linear in calibration parameters until it can be fully solved by computer - Google Patents

Abstract

The method involves using a number of images of a scene taken from different positions and/or viewing directions and measurement data for the relative rotation of the at least one camera between the individual records, whereby the camera parameters are determined with a linear equation system. It involves determining a first linear transformation from corresponding points in images selected in pairs, determining a second linear transformation representing the rotation, inserting matrices describing the transformations into an equation giving an equation system that is linear in the calibration parameters, inserting further matrices from further image pairs until the equation system can be uniquely solved by computer.

Description

The The invention relates to a method for self-calibration of a camera system, especially one whose image data is stored electronically be after the preamble of the main claim.

The Using cameras for the metric reconstruction of taken from different perspectives Scenes with the help of computers has always been in recent years wider distribution found. For however, such camera systems are the internal parameters, for example the focal length, generally not known a priori and during the In addition, operation partly variable, e.g. at the zoom. Your knowledge is for a metric reconstruction but indispensable, so the camera systems before or during to calibrate the image recording.

The Determination of the camera parameters is usually based on one of the camera recorded image sequence of a calibration pattern different perspectives. An extension is the calibration based on any rigid scene that the camera encounters during the Recording relatively moved. Rotation and translational degrees of freedom The camera forms the so-called external camera parameters, the Often not all are known. The automatic calculation the internal and external parameters from the image sequence is called Self-calibration called.

Around on as possible easy way to robust values for getting the calculated parameters are often constraints for the Camera movement and / or the scene formulates the extra Provide information and / or certain parameters from the outset establish. One differentiates with the procedures for the camera calibration Also, whether the internal camera parameters are unknown fixed sizes or at least partially continued to vary.

To the most common Constraints for the camera itself counts, that she is only allowed to rotate around her optical center. In this case can not provide depth information for the scene will be extracted from the image sequence. This camera movement but is found in many applications, such as video conferencing systems. Typically, panoramas are created during this camera movement (de Agapito, L .; Hartley, R.I; Hayman, E .: Linear Self-Calibration of a rotating and zooming camera, IEEE Computer Society conference on Computer Visison and pattern recognition, 1999, Volume 1, June 23-25 1999, pages 15-21).

A other restriction for simplified calibration is in the presence of a known reference object, z. B. a scale with a known position, in all images of a calibration sequence. This does however a requirement on the scene, at least not always meet leaves, if an automatic self-calibration to unknown, future Timing is planned or even unavoidable, for example in the reconstruction of historical sites.

Just few works deal with the self-calibration of freely moving Cameras that do not provide conditions to the scene except that it is a rigid scene, and the variation of the internal parameters possible (e.g., M. Pollefeys, R. Koch, L. Van Gool: Self Calibration and metric reconstruction in the process of varying and unknown internal camera parameters, In Proc. ICCV, p. 90-96, 1998). These methods are technically disadvantageous because they tend to be slightly disturbed image data already severely disturbed Calibrations to calculate or no more meaningful results to deliver. Furthermore, there are critical camera movements in these approaches, where not complete Camera calibration can be calculated without the occurrence This critical situation is recognized, for example, a pure Shifting the camera.

A suggestion for camera calibration for basically any camera movements and fixed camera parameters is from the GB 2 261 566 A known, the camera also needs to be moved and rotated necessarily. This method uses Kruppa's equations for iterative calculation of the camera parameters and does not impose any requirements on the fixed scene. However, the algorithm is complex and the obtained parameters are in many cases not or only partially suitable for scene reconstruction.

It is also known to equip a camera with a permanently mounted acceleration gyro-sensor in order to simultaneously record not only images but also the rotation of the camera (eg Suya You, Ulrich Neumann, and Ronald Azuma: Orientation Tracking for Outdoor Augmented Reality Registration, IEEE Computer Graphics and Applications 19, 6 (Nov / Dec 1999), 36-42). These data are used in the numerical Re construction of the scene shown, but not for camera calibration, used because so far no suitably formulated method for this is known.

It is therefore an object of the invention, a method for self-calibration to specify a camera. The problem is solved by a method with the features of claim 1. The dependent claims give advantageous embodiments at.

Especially it is advantageous to work with fixed rotation sensor, where the sensor measurement data directly into the determination of the internal Incorporate camera parameters, who are free to vary themselves.

there it is assumed that at least a camera and a plurality of images of a scene consisting of different Positions and / or directions of the camera system recorded and measured data of the relative rotation of at least one Camera between each shot are present. From corresponding Points in pairs Images can now be a first matrix determined from the relative Rotation of the at least one camera between the same selected images can then a second matrix as representation The twist can be determined, then both matrices can be in an equation be used, which is a linear in the calibration parameters system of equations indicates, and more matrices should also from other image pairs determined and used until the system of linear equations clearly solvable is and by computational solving of the system of equations determines all parameters of the camera system can be.

The Camera system can consist of more than one camera, with the fixed positions and orientations to each other are known or from one or more freely movable and freely rotatable cameras consist. Advantageously, at least one of the cameras with a rotation sensor for detecting the relative rotation of the Camera connected between two separate shots.

By Set up a statistical error function with the measurement data to the camera rotation, the pictures taken, the calculated Camera parameters and the preconditions and optimizing this can the orientation measurement are stabilized for error compensation of the orientation measuring system.

The The invention will be explained with reference to figures. Showing:

1 a sketch to explain the camera movements;

2 a scheme of the mathematical hole camera model;

3 a first sketch to explain the epipolar geometry;

4 a second sketch to explain the epipolar geometry;

5 a sketch to explain the homography of rotating cameras,

6 and b the results of fixed focal length focal length calibrations ( 6a ) and zooming camera ( 6b )

7a shows the average focal length, which is calculated by the self-calibration of the noisy images, depending on the degree of noise in pixels and rotation angle,

7b shows that too 7a ) associated variance of the focal length

7c shows the mean pixel aspect ratio.

7d shows that too 7c ) associated variance of the pixel aspect ratio.

8a - 8d the calculated results are analogous to 7 ,

9a and b the comparison with and without statistical calibration

10a and b the variance of f with a) linear calibration and b) statistical calibration.

First, the following camera model will be explained. The simplest model for this is a pinhole camera. In the 3D world coordinate system, the camera position is determined from the position of the camera center C and the rotation R of the optical camera axis against the coordinate axes, as in FIG 1 shown. In the case of freely moving cameras both can change with time τ. C is a three-dimensional column vector in the world coordinate system, R is a 3 × 3 orthogonal matrix containing the coordinate axes (optical axis, axes of the image plane) of the rotated camera coordinate system in the world coordinate system, hence a determinant one rotation matrix.

In the case of the pinhole camera, the camera center is identified with the focal point, which is ideal in the model, which can optionally be in front of or behind the image plane. It is common today to start from an image plane in front of the focal point, as in 2 you can see. In the camera-fixed coordinate system, the focal point is at the origin, the optical axis coincides with the z-axis, and the image plane is at z = f, where f denotes the focal length. Imagine the image plane now occupied with photosensitive, rectangular pixels of the edge lengths dx and dy. Since one practically performs the image evaluation in pixel coordinates, the origin of the pixel coordinate system may be different than the penetration point of the optical axis through the image plane, which should be located at c (vectors in pixel coordinates are indicated by small letters). This is called the main point of the camera.

A three-dimensional scene point M in world coordinates is now imaged via a linear image onto a pixel m in camera-fixed pixel coordinates. In this case, for a simple representation of the mapping, it is expedient to add a further coordinate (one) to the points, for example as M = (X, Y, Z, 1) ^T and m = (x, y, 1) ^T. You can then write m = PM (1) with a 3 × 4 matrix P, called a camera projection matrix. It is given for each camera position and orientation, ie at fixed time τ, by P = K [R T | - R T C] (2) with R and C off 1 and K as a priori unknown matrix containing the internal camera parameters. The real-valued 3 × 3 matrix K is called the camera calibration matrix. she has

The calibration matrix represents the properties of the recording sensor or system, ie usually a CCD chip. It contains the five internal camera parameters:
f is the focal length of the camera in pixels. If dx is the width of a pixel in mm, fdx will give the focal length of the camera in mm.
a is the aspect ratio of the pixels. It is defined as a = dy / dx, where dx and dy are the dimensions of the pixels in the x and y directions.
s describes the pixel shear. This depends on the angle between the lines and columns of the recording sensor.
c = (c _x , c _y ) describes the main point of the camera with two parameters.

The Determining the parameters in K is the task of self-calibration. With modern cameras it is often assumed that s = 0 in a good approximation become.

The The mapping described by the projection matrix P is not unique reversible. Rather, the central projection of the camera all object points of a line of flight are mapped to the same pixel. Therefore, no distances can be obtained from a single image determine the object points. There are at least two pictures for this from different perspectives (places or times τ1, τ2) of the same object needed for triangulation.

3 shows schematically the same camera at times τ1 and τ2 with mutually shifted and rotated image planes. This is equivalent to the scenario of recording with two different cameras at the locations τ1 and τ2, so in the following only the recording at different times will be discussed. Given all the camera parameters, the position of an object point M can be calculated from its pixels m1 and m2 in the image planes. For unknown parameters, the calibration is first searched for point correspondences. For a given pixel ml, however, one only knows from the object point M that it is on the line L. As a corresponding pixel m2 is thus any point on the line λ2 into consideration. First of all there are point-line correspondences, such as 4 clarified. Each object point M lies in a plane of the illustrated set of planes, which in turn must pass through both camera centers. The layer group is represented in each image plane in each case a group of lines, of which each one point on λ1 and λ2 can be unambiguously assigned to one another. If M has an image in λ1, then also in λ2. The lines of lines intersect in each image in exactly one point (e1 or e2), namely the image of the other camera center, which is called Epipol.

Using the initially introduced three-dimensional representation of the two-dimensional image points m1 = (m1 _x, m1 _y, 1) ^T, e1 = (e1 _x, e1 _y, 1) ^T, m2 = (m2 _x, m2 _y, 1) ^T, e2 = (e2 _x , e2 _y , 1) ^T can be used to form vector products n1 = e1 × m1 and n2 = e2 × m2, and these are normal vectors to the same plane, hence identical except for one factor. However, n1 and n2 are still represented in different, mutually twisted coordinate systems. Nonetheless, it is plausible (and this is also proven in the literature) that it is possible to set up a 3 × 3 matrix that satisfies the condition for any corresponding points in both images m1 T Fm2 = 0 (4) Fulfills. This so-called fundamental matrix F contains both information about the rotation of the cameras relative to one another or the rotation of a camera over time and about the displacement of the cameras or the translation of a camera in the form of vector product formation with an epipole. The fundamental matrix is calculable from point correspondences that can be determined with common tracking software. Because of (4) oBdA one can set one of its components to one. In particular, since Fe2 = 0, the map represented by F has a nonzero kernel, ie F has rank 2 and thus determinant 0. Thus, only seven independent components need to be calculated, requiring seven independent point correspondences. For more information on the structure and properties of the fundamental matrix, please refer to technical literature.

The the determination of F from point correspondences still delivers not the desired ones Camera parameters. Rather, the expert is now anxious from the Knowledge of F on the projection matrix P from equation (2) shut down. This approach is fundamentally disadvantageous because in In this case, the projection matrix can not be uniquely determined can. Without knowledge of the scene one gets only one with one invertible 4 × 4 matrix transformed projection matrix, as generally in the specialist literature is proved.

However, if additional rotation information is available, a completely different way can be pursued, which constitutes the core of the method according to the invention. It can be shown that a system of equations can be set up which is linear in the sought-after camera parameters, which are the only unknowns except for a scaling factor. If one then draws point correspondences from several images, if one sets several F _{j, i} from the requirements mi ^T _{Fj, i} mj = 0 for image pairs i and j, then the parameters can be obtained quickly, and the above disadvantages of access via the projection matrices do not occur.

mit R_j,i als Rotationsmatrix, die die Kameraorientierung des Bildes j in die des Bildes i überführt, also insbesondere R_j,i = R_iR_j ^T oder auch R_j,i = R(τi)R^T(τj). Mit ρ_j,i wird der unbekannte Skalierungsfaktor bezeichnet, der beim Aufstellen der Fundamentalmatrizen nicht bestimmt werden kann. Die erste Matrix auf der rechten Seite von (5) beschreibt die Bildung des Vektorprodukts mit dem Epipol in Bild i, also dem Zielbild der Abbildung analog zur Definition von R_j,i.The system of equations to solve is:

with R _{j, i} as a rotation matrix which converts the camera orientation of the image j into that of the image i, that is to say in particular R _{j, i} = R _i R _j ^T or also R _{j, i} = R (τ i) R ^T (τ j). Ρ _{j, i} denotes the unknown scaling factor that can not be determined when setting up the fundamental matrices. The first matrix on the right side of (5) describes the formation of the vector product with the epipole in image i, ie analogous to the target image of the image for the definition of R _{j, i} .

Under certain conditions, solutions of equation (5) result in an advantageous manner:
If the main points (c _{i, x} , c _{i, y} ) and (c _{j, x} , c _{j, y} ) are known, the focal lengths f _i and f _j , the pixel aspect ratios a _i and a _j and the pixel shifts s _i and determine s _j uniquely from two pairs of images.

are Given major points and pixel shearing, pixel aspect ratios can and focal lengths of a single fundamental matrix and rotation information be calculated.

One Special case is that of a rotating camera with a fixed center, i.e. without translation between the camera centers. For pictures, The result of a pure rotation of the camera is not an epipole defined so that equation (5) is not applicable.

For rotating cameras, the pixels mi and mj of an object point M in the rotated images i and j can be clearly mapped to each other, such as 5 clarified. The associated figure is called homography: mi = H ∞ j, i mj. (6)

wobei der Faktor ρ_j,i als Unbekannte eingeführt wird, wenn man die aus Punktkorrespondenzen ermittelten Homographien in (8) einsetzt. Im Übrigen ist (8) wieder linear in den Kameraparametern, und die Analogie zu (5) ist offensichtlich.H ^∞ _{j, i is} a 3 × 3 matrix and the vectors m are, as before, the three-dimensional correspondences to the pixels m in pixel coordinates. H ^∞ _{j, i} can be determined from four independent point correspondences up to a scaling factor. It applies generally H ∞ j, i = K i R j, i K j -1 (7) with the same matrices K _i and R _{j, i} from equation (5), and by multiplying (7) by K _j from the right one obtains immediately

where the factor ρ _{j, i is introduced} as an unknown, if one uses the homographies determined from point correspondences in (8). Incidentally, (8) is again linear in the camera parameters, and the analogy to (5) is obvious.

Around in all cases the camera movement a full Calibration and a simultaneous error correction for the sensor data To achieve this, a statistical calibration is connected to the above calibration procedures. The goal is to estimate the most likely camera parameters below proviso the measured measurement (images, rotational measurements) and formulated known preconditions. For this a mistake is minimized only determined by the prior knowledge, the images and the rotation data becomes. The error consists of a term for evaluation the camera parameter with regard to the pictures (maximum likelihood), a Term for evaluation of rotation data and multiple terms for evaluation compliance with the preconditions. The rating of the camera parameters Based on the image data is carried out using the formulas (6) and (4).

Of the Term for evaluating the improved rotation uses the error model the rotation measuring system to decide how likely the currently estimated Rotation has occurred during the measurement. The other terms to Formulation of prior knowledge can to stabilize badly determinable parameters, such as Main point to be used. This error function is then through minimized a nonlinear minimization method to be the most likely Camera parameters and rotation data to determine. Here are the camera parameters determined by the above linear methods and used the measured rotation data as starting values.

The Statistical calibration thus calculates a complete calibration and corrected rotation information. This corrected rotation information is used to compensate the system and measurement errors of the sensor.

The following is an example:
For a test of real-time calibration, still images were taken from a rotating camera with and without zoom. In both cases, the camera's main point, pixel shear, and pixel aspect ratio were assumed to be known. The rotation of the camera was controlled by computer commands, so that the rotation information from the program specifications was given with an uncertainty of about 0.5 °. A reference for the actual focal length adjustment was determined manually. The focal length varied during the zoom experiment between 875 and 1232 (measured in pixels). Zoom-related radial image distortions were corrected before self-calibration, which is possible without knowing the exact focal length.

Out Image sequence were determined by prior art methods the homographies are set up by means of equation (6) and by Reprojection checked. in this connection a mean pixel error of 0.8 was found.

Triples of images were used for focal length calibrations. The results are in 6 shown. The dashed lines represent the reference data, the solid lines the results of the self-calibrations. The average relative error is 3% in the case of fixed focal length ( 6a ) and 7% the zooming camera ( 6b ).

Furthermore, the robustness against noise was investigated:
To estimate the robustness of the calibration against pixel errors and inaccurate rotation data, synthetic data were generated and deliberately noisy.

In the case The rotating camera was a virtual camera with center in the Coordinate origin about x- and y-axis rotated by 6 °. The camera watched uniform distributed, arranged in a cube scene points and generated calculated images of 512 × 512 Pixels. The positions of the pixels became uniform up to pixels noisy. Furthermore, the rotation information became up to 2 ° per axis noisy. In the virtual camera were all camera parameters a priori known, in particular, f = 415 and a = 1.1 were set.

In the 7a shows at a medium focal length, which is calculated by the self-calibration of the noisy images, depending on the degree of noise in pixels and rotation angle. 7b ) shows that too 7a ) associated variance of the focal length. 7c ) shows the average pixel aspect ratio and 7d ) shows that too 7c ) associated variance of the pixel aspect ratio.

For the free moving camera were using six virtual cameras on a spherical surface Viewing direction arranged inside the ball. The scene points were fortuitously distributed inside the ball and from the cameras to 512 × 512 pixels displayed. Furthermore The pictures were noisy as described above.

8th shows the calculated results analogous to 7 , In particular, the same descriptions of the individual illustrations apply.

The 7 and 8th It can be seen that the calculated calibration parameters behave very robustly with respect to variations of the rotation angle <1 ° and pixel errors <1 pixel. The dependence on the angle error is consistently stronger than that of the pixel error. If you compare 7 and 8th directly, eg in particular 7b ) and 8b ), you can also see that the influence of pixel errors in the free-moving camera is greater than in the rotating one.

This results in the following effect of the statistical calibration:
To clarify the effect of the statistical calibration, the robustness measurement was repeated using synthetic data for the rotating camera with f = 415, whereby the y-component of the main point was now set to the value c _y = 201 (image center). The linear calibration did not compute from the noisy images a robust estimate for c _y how to 9a ) recognizes well. In fact, the results for angle errors> 0.5 ° are almost random. Similar susceptibility showed the parameters c _x and s (not shown).

During statistical calibration, an error function is set up which should be minimized. Among other things, the error function includes a summand that tends to zero, the closer the measurements for the main point are at the respective center of the image, as must be according to the selected specifications. Including the numerical minimization of this error function (with standard programs) of the linear parameter determination, one obtains improved estimates for the camera parameters. This should be understood by statistical calibration. In many practical applications, but you will be on this last Opti Even if you only need to update the focal length, you can do without

The statistical calibration determined the main point and the pixel shear even from noisy images very robust, as exemplified by 9b ) compared to 9a ) recognizes well. At the same time, the already robust results for focal length and pixel aspect ratio were further stabilized. This becomes clear, for example, from the variance of f in 10a ) (Result linear calibration) and 10b ) (Result statistical calibration) in direct comparison.

Claims (4)

A method of self-calibrating a camera system comprising at least one camera utilizing a plurality of images of a scene taken from different positions and / or viewing directions of the camera system and measurement data of the relative rotation of at least one camera between the individual images, the camera parameters being determined by a camera linear equation system can be determined, characterized in that - from corresponding points in pairwise selected images, a first linear transformation described by a matrix is determined, - from the relative rotation of the at least one camera between the same selected images, a second linear transformation describable by a matrix Representation of the rotation is determined, - both matrices are inserted into an equation that indicates a linear system of equations in the calibration parameters, - other linear transformations that can be described as matrices a us further image pairs are determined and used until the linear system of equations is uniquely solvable, and - the system of equations for all parameters of the camera system is solved by computation, wherein at least one camera with a rotation sensor for detecting the relative rotation of the camera between two separate temporal recordings connected is. Method according to claim 1, characterized in that that this Camera system consists of more than one camera, with the fixed Positions and orientations are known to each other. Method according to claim 1, characterized in that that this Camera system consisting of one or more freely movable and freely rotatable Cameras exists. Method according to one of the preceding claims, characterized characterized in that Set up a statistical error function with the measurement data to the camera rotation, the pictures taken, the calculated Camera parameters and the preconditions and optimizing these the Orientation measurement is stabilized to compensate for the error Orientation measurement system.