DE102010004233B3 - Method for determining position of camera system with respect to display, involves determining geometric dimensions of object and calibration element, and determining fixed spatial relation of calibration element to object - Google Patents

Abstract

The method involves recording an image with an image content by a camera system (2) for an undetermined position of a mirror (4). A parameter of an imaging function is determined from image information of a part of the image content, where the imaging function characterizes imaging of an object point in an image storage of the camera system. Geometric dimensions of an object i.e. display (3), and a calibration element i.e. calibration pattern (6), are determined, and fixed spatial relation of the calibration element to the object is determined.

Description

The invention relates to a method for determining the position of a camera system with respect to an object, wherein the object is arranged outside a field of view of the camera system. A position is understood to mean a spatial position and orientation. A layer expresses a rigid body transformation between coordinate systems, wherein a coordinate system of the object is referred to herein as a reference coordinate system. The determination of the position thus means the determination of at least one parameter of a transformation between a coordinate system of the camera system and the reference coordinate system. The inventive method and the device according to the invention are also applicable to more general mapping rules between points on the object and their images in the camera system.

For the mathematical representation of the situation, there are a variety of possibilities. Two commonly used are the specification of six parameters (three angles and three distances) or the specification of a transformation matrix.

In principle, there are various possibilities for determining the position of a camera system with respect to the object. A principal possibility is, using specifications and design data such as focal length, position of the optical sensor in the housing and position of the housing to the object to determine the position of the camera system to the object. Such an approach fails most often because important data is not or only vaguely known. For example, for cameras in mass production, the main point is never and the focal length of the lens is given only vaguely. In addition, assembly errors are inevitable within certain tolerances.

Another possibility is the complex measurement of the relative geometry with precision measuring means, for example by means of laser triangulation, but here only the position of the camera body to the object can be determined, but not internal parameters of the camera (such as distortion or focal length) or the position of the optical sensor.

Furthermore, photogrammetric calibrations and self-calibration are known, however, which presuppose that the object is in the field of vision of the camera system.

The DE 10 2007 030 378 A1 discloses a method for use in external calibration of a camera system to determine the location of the camera system relative to a frame of reference. Hereby, the method uses a main calibration element, wherein the main calibration element is in a previously known spatial relationship to the reference system. The method determines the location of the camera system relative to the reference system by detecting the main calibration element from the camera system. For this purpose, a separate from the camera system imaging system is used. Furthermore, the main calibration element is positioned relative to the reference system in such a way that it can be detected by the camera system by means of a suitable arrangement of the imaging system via the imaging system. In addition, the imaging system is arranged with respect to the main calibration element and the camera system such that the main calibration element is detectable by the camera system via the imaging system. As a further feature essential to the invention, it is disclosed that in the step of determining the position of the camera system relative to the reference system, this determination takes place taking into account the position of the imaging system with respect to the camera system, wherein the main calibration element is detected by the camera system via the imaging system. With regard to the consideration of the position of the imaging system with respect to the camera system, the document discloses that the spatial position of the imaging system with respect to the reference system must be known exactly. Alternatively, the document discloses that the determination of the position of the imaging system with respect to the camera system by at least one calibration mark is arranged on the imaging system, wherein the calibration mark is detected by the camera system. The imaging system can be a mirror here.

The DE 10 2007 032 471 A1 discloses a method for determining the location of a camera system relative to an object, wherein the object is located out of the field of view of the camera system. The determination is carried out by means of at least one mirror, wherein the position parameters of the mirror are known and / or can be determined by at least three mirror marks arranged on the mirror. The object is at least partially mirrored by the mirror into the camera system. Furthermore, an imaging function is set up which describes the image of an object point in the image memory of the camera system, wherein the unknown magnitudes of the imaging function are determined by numerical methods and the position between camera system and object is determined therefrom. Again, it follows that a spatial location of the mirror must be known in advance and / or explicitly determined by means of mirror marks and photogrammetric methods.

The invention is based on the technical problem of providing a method for determining the position of a camera system with respect to an object, wherein the object is arranged outside the field of view of the camera system, which without an a priori knowledge and / or without an explicit determination or measuring an exact spatial location of an imaging system or mirror with respect to the camera system.

The solution of the technical problem results from the objects having the features of claim 1. Further advantageous embodiments of the invention will become apparent from the dependent claims.

Proposed is a method for determining a position of a camera system with respect to an object, wherein the object is not in a field of view of the camera system.

Properties of the object, for example its geometric dimensions, can be known a priori here. Alternatively, properties of the object can also be determined during a performance of the method according to the invention. For example, it is conceivable that the object is reconstructed and determined by means of a method for 3D reconstruction during the execution of the method. In this case, the object and also the reference coordinate system are determined only at runtime of the method. For 3D reconstruction z. B. captured by the camera system images are used. Preferably, the position of the camera system to be determined with respect to the object does not change during the proposed method.

For example, the object may be a screen and the camera system may be a webcam mounted on the screen.

Furthermore, at least one mirror is arranged in such a way that the mirror images or reflects at least part of a calibration element into a field of view of the camera system. The mirror thus forms at least part of the calibration element in a mirror image of the calibration element. This mirror image of the calibration element can be detected by the camera system. The camera system thus captures at least parts of the calibration element via the mirror. In this case, a mirror is understood as meaning a single mirror or an arrangement of a plurality of mirrors. For example, a mirror may comprise a plurality of partial mirrors, which are arranged offset to one another, but which at the same time image at least parts of the calibration element. The term mirror also includes an arrangement of a plurality of mirrors arranged one after the other with respect to a beam path from the calibration element to the camera system. Preferably, the mirror is a plane mirror, but it is also conceivable to use a non-planar mirror.

The calibration element is in a fixed spatial relationship to the object. The fixed spatial relationship can be known a priori. Alternatively, geometric dimensions of the calibration element during the implementation of the method, for. B. also be determined via a 3D reconstruction. In this case also the fixed spatial relationship of the calibration element to the object is determined during the performance of the method. However, it is also conceivable in this case that the reconstructed calibration element replaces the original object as a new object.

As explained in more detail later, the calibration element can also be formed by the object. In this case, no spatial relationship between the object and the calibration element has to be determined.

Thus, the object is the calibration element or at least part of the object forms the calibration element. Alternatively, a spatial relationship of the calibration element to the object is previously known or geometric dimensions of the object and the calibration element are determined during the execution of the method, from which the fixed spatial relationship of the calibration element to the object is determined.

This results in an advantageous manner that the determination of the position of the camera system with respect to an object, such as a motor vehicle or a screen, can take place.

The camera system captures at least one image with an image content for at least one indeterminate position of the mirror. In this case, a position of the mirror is understood to be a spatial position and orientation of the mirror relative to the object or relative to the camera system. Under an indefinite position of the mirror is understood here that a spatial position and orientation of the mirror with respect to the object a priori are not known. Also, a spatial position and orientation of the mirror with respect to the camera system is not known a priori.

The image content of the image here comprises at least a first part. Here, the first part refers exclusively to the part of the image content that is imaged by the mirror in the field of view of the camera system. The first part hereby includes, inter alia or exclusively, the imaged, ie mirrored, part of the calibration element. The image content of the image may additionally comprise a further part. Consequently, the further part of the image content designates the part which, although in the field of view of the camera system, is not imaged by the mirror into the field of view of the camera system. So z. B. the first part mirrored by the mirror in the field of view of the camera system elements. Thus, the first part also comprises at least a part of the mirrored calibration element. The further part can be z. These include, but are not limited to, including a mirror frame that serves as an enclosure for a mirror surface of the mirror, or includes marks disposed on the mirror frame.

As explained later, the indeterminate position of the mirror with respect to the camera system is not explicit, z. B. determined by means of a detection of mirror marks, during the proposed method or measured.

In this case, the first part of the image content contains the mirror image of at least part of the calibration element. By means of numerical methods, at least one parameter of an imaging function is determined exclusively from image information of the first part of the image content of the at least one image. The determination of the at least one parameter of the mapping function can in this case be carried out, for example, by means of a calculation unit. Preferably, at least one parameter of an imaging function is determined exclusively from image information of a portion of the first part which exclusively comprises the mirror image of the calibration element. In this case, the imaging function describes a mapping of an object point into an image memory of the camera system. The object point can also be determined in the reference coordinate system.

Starting values for the numerical approaches can be determined, for example, via estimated design data.

Preferably, to determine the at least one parameter of the mapping function, the mapping of prominent object points, the so-called landmarks, is analyzed. Here, a distinctive object point or a landmark is understood to be a point which can be easily localized by means of digital image processing methods in the image content of the at least one image. Thus, image coordinates of the camera system can be assigned to these landmarks. For this purpose, the camera system z. B. include a device for image analysis.

In this case, a geometry of the calibration element does not have to be known in advance. Preferably, however, at least one geometric distance between two landmarks of the calibration is known. In this way, a scale factor can advantageously be determined, wherein a distance of the image coordinates of the detected landmarks into the known geometric distance of the landmarks can be transformed by means of the scale factor. Further preferably, more than just a geometric distance of landmarks of the calibration, for example, an entire geometry of the calibration, known. This advantageously results in a simplified determination of the parameters of the mapping function.

As previously explained, the mapping function may also describe a transformation between a coordinate system of the camera system to the reference coordinate system. It is of course conceivable that the mapping function describes only parts of the transformation, eg. B. a translation portion of the transformation. The mapping function can also describe the transformation down to a scale factor.

In general, the mirror can have previously known imaging properties. If imaging properties are known, a simplified determination of the at least one parameter of the imaging function results in an advantageous manner.

Also by known internal and / or external parameters of the camera system results in a simplified determination of the at least one parameter of the imaging function.

It should also be noted that the at least one parameter of the mapping function is determined exclusively from image information of the first part of the image content of the at least one image. This is understood to mean that the imaging function is determined exclusively on the basis of image information, which is mirrored by means of the mirror in the field of view of the camera system. For example, the mapping function is determined solely on the basis of landmarks of the mirrored by the mirror calibration element.

This makes it possible to determine the at least one parameter of the mapping function without an explicit calculation or measurement of the position of the mirror. An explicit determination here means that parameters of the position of the mirror with respect to the camera system are determined separately, for example in a separate calculation step. In particular, the determination of the mapping function does not have to be based on an evaluation of z. B. mirror marks done. The proposed method therefore avoids both the effort that would be necessary for a metrological or photogrammetric position detection, as well as their error influences on the results of the subsequent calculation for determining the position of the camera system with respect to the object.

Since the imaging function is determined on the one hand on the basis of an image taken in an indefinite position of the mirror and on the other hand solely on the basis of image information of the first part of the image content of the image, it is advantageously not necessary in the proposed method to determine the position of the mirror a priori to know or measure.

Thus, a determination of the mapping function thus takes place without taking into account the position of the mirror i. P. D. DE 10 2007 030 378 A1 since the position need not be known a priori nor must it be specified during the determination of the position of the camera system relative to the reference system. In the method according to the invention, parameters of the position of the mirror relative to the camera system are determined during the determination of the imaging function, but without z. B. to be calculated explicitly in a separate step.

In another embodiment, the camera system captures a first image having a first image content in a first indeterminate position and at least one second image having a second image content in a second indeterminate position. The first layer is different from the second layer. Furthermore, by means of numerical methods, the at least one parameter of the mapping function is determined from image information of the first part of the first image content and of the first part of the second image content.

As a rule, the determination of the mapping function from the first part of the picture content of only one picture is susceptible to error. For example, a minimization problem posed using detected landmarks may be poorly conditioned in this case. In the case of a planar mirror, the problem is even under-determined. The inclusion of two or more images advantageously reduces the susceptibility to errors in the determination of the imaging function or enables the clear solution of the problem.

A knowledge of a change in position between the first indefinite position and the second indefinite position of the mirror is not required here.

However, the change in position between the first and the second indefinite position of the mirror can also be determined and / or detected. A change in position can be detected, for example, by an optical, magnetic or mechanical position detection system, wherein a coordinate system of the position detection system is not registered to the reference coordinate system or a coordinate system of the camera system. This results in an advantageous manner that in knowledge of the change in position, a simplified determination of the mapping function can be done.

A change in the position of the mirror can be adjusted manually or automatically. If the change in position is set manually, it results in an advantageous manner that no additional automatic adjusting devices are provided for adjusting a position of the mirror. If the change in position is set automatically, eg. B. by means of an adjustment, so it is possible to determine the change in position in a particularly simple manner.

Since the position of the mirror is understood to be a position relative to the camera system, it is also possible for a position of the camera system with respect to the mirror to be changed to accommodate a first and at least one second image. If a change in position of the camera system is not detected and / or determined, it is in this case, however, imperative that the position of the camera system to be determined with respect to the object does not change during the change in the position of the camera system.

If a relative change in position of the camera system is detected and / or determined, then it is also conceivable to change the position of the camera system relative to the object between the taking of different images. By knowing the change in position between the camera system and the object, this can be computationally compensated in the determination of the imaging function.

In particular, several images are also used if there is no a priori knowledge of the geometry of the calibration element. Multiple recordings generally improve the accuracy of the estimate so that multiple recordings may also be used if individual a priori knowledge of the geometry of the calibration element is available.

In another embodiment, the object is a screen. In this case, the camera system can be arranged, for example, on the screen. In particular, this method is advantageous if the camera system is arranged on the screen in such a way that the camera system is able to detect the eyes of a user viewing the screen. Such an arrangement can be used, for example, for adapting an autostereoscopic display of images on the screen to the position of the user's eyes. Under screen is generally understood an optical output medium for computers such as display, 3D display, monitor, projector, projector or the like.

In a further embodiment, the calibration element is a calibration pattern, wherein the calibration pattern is displayed on the screen. By the screen which displays the calibration pattern itself forms the calibration element, the spatial position of the calibration element, here the calibration pattern, is fixed with respect to the object and is not subject to any possibly faulty attachment procedure. Also can be dispensed with an exact measurement of this spatial relationship. For example, the reference coordinate system or world coordinate system may be directly related to a pixel matrix of the screen. With knowledge of the screen pixel size, the spatial relationship between the calibration element and the reference coordinate system can then be precisely determined. Any calibration errors due to mechanical inaccuracies or measurement errors in the design can be reduced to an insignificant degree.

The camera system hereby comprises at least one camera. In a further preferred embodiment, the camera system comprises two or more cameras. In each case two or more cameras can also be calibrated to one another and form a stereo camera system.

The invention will be explained in more detail with reference to two embodiments. The figures show:

1 a schematic representation of a device for determining the position of a camera system relative to an object and

2 a schematic block diagram of a device for determining the position of a camera system relative to an object.

In the 1 is schematically a device 1 for determining a position of a camera system 2 relative to an object that serves as a display 3 is formed, shown. This is the display 3 not in the field of vision of the camera system 2 , Preferably, the camera system 2 and the display 3 firmly connected. On the display 3 is a calibration pattern 6 presented in the form of a checkerboard pattern. Chessboard surfaces in the center of the checkerboard pattern have circles, the circles having a color complementary to the respective checkerboard surface. Here it is shown that three checkerboard surfaces each have such a circle, wherein the connecting lines of centers of the circles form a right-angled triangle. Of course, alternative calibration patterns are conceivable. Furthermore, the device comprises 1 a mirror 4 that is so to the camera system 2 and the display 3 is positioned that of these illustrations of the display 3 (here the calibration pattern 6 ) in the field of vision of the camera system 2 reflects.

Furthermore, the mirror 4 For example, connected to an adjusting device, not shown, by means of which the position of the mirror 4 can be changed. The mirror 4 but can also be held and adjusted manually.

Next, in a graphics memory, not shown, of the display 3 a point P, for example a landmark, which is stored as the real pixel P 'of P on the display 3 is pictured. This real pixel P 'is reflected by the mirror 4 mirrored in a camera plane not shown. P "denotes the virtual mirror image of P '. By means of the camera system 2 the virtual mirror image P "is imaged into a pixel P"'in the camera stabilization plane and transmitted from there into a camera image memory. P '''' then designates a data point of P '''in the camera image memory.

The landmarks of the checkerboard surfaces serve as a landmark in particular. The circles are used here for a simplified referencing of landmarks by specifying an origin of the calibration pattern.

2 shows a schematic block diagram of the in 1 illustrated device 1 for determining the position of a camera system 2 relative to a display 3 , The device 1 this includes the camera system 2 , the display 3 and a mirror 4 , In addition, an adjusting device 5 and a calculation unit 8th shown. The camera system 2 is fixed to the display 3 connected. Here it is shown that a position of the mirror by means of the adjusting device 5 can be adjusted. Next will be on the display 3 a calibration pattern 6 displayed. An example is a beam path 7 from the calibration pattern 6 over the mirror 4 to the camera system 2 shown. The from the camera system 2 recorded images are sent to the calculation unit 8th transmitted by data technology. The calculation unit 8th includes a device 9 for image analysis and one unit 10 for calculating at least one parameter of a mapping function. Here are by means of the device 9 For image analysis, for example, landmarks in one of the camera system 2 recorded image detected. Information about these landmarks will then be technically available from the facility 9 for image evaluation to the unit 10 to transfer at least one parameter of a mapping function.

It will now be an abstract calibration pattern 6 (Data in graphics memory) with known dimensions in the unit pixels assumed. The illustration of this calibration pattern 6 takes place via the display 3 , the mirror 4 , the camera system 2 and finally into the camera image memory. There, the picture is in digital form again. The image from the graphics memory on the display 3 is usually a linear map and can be assumed to be known. The pictures of the mirror image in the camera system 2 depend on external parameters (position parameters) and internal parameters (eg focal length and distortion of the lens of the camera system 2 ). Preferably, the internal parameters are determined in advance and are then known for further calculation. However, the internal parameters can also be determined during the proposed method.

_Gp

Ortsvektor des Datenpunktes P in (homogenen) Grafikspeicherkoordinaten

_Bp

Ortsvektor des Datenpunktes P'''' in Bildspeicherkoordinaten

_Bp ~

Ortsvektor des gemessenen Datenpunktes P'''' in Bildspeicherkoordinaten

_DS D / k

Spiegelung im Displaykoordinatensystem

_KT^D

Transformationsmatrix: Display → Kamera

_DK^G

Matrix der Abbildung Grafikspeicher → Display

_BK^K

Matrix der linearen Abbildung Kamerabildebene → Bildspeicher

ν

Parameter der nichtlinearen Abbildung in die Kamerabildebene (Verzeichnungsparameter)

f(...)

Funktion, welche die Abbildungseigenschaft der Optik beschreibt.

Ψ

Menge der mitzubestimmenden Parameter

Ψ -

Menge der a priori bekannten Parameter

i

Index für Bezugspunkte des Kalibriermusters und dessen Abbildungen i = 1 ... L

k

Index für Lagen des Spiegels k = 1 ... N

The following is a mathematical model that can be used as the basis for determining the parameters of the mapping function. The following terms are used for this:

_G p

Position vector of the data point P in (homogeneous) graphics memory coordinates

_B p

Position vector of the data point P "" in image memory coordinates

_B p ~

Position vector of the measured data point P "" in image memory coordinates

_D S D / k

Mirroring in the display coordinate system

_K T ^D

Transformation Matrix: Display → Camera

_D K ^G

Matrix of figure Graphic memory → Display

_B K ^K

Matrix of linear imaging Camera Stabilization → Image Memory

ν

Parameters of the non-linear mapping into the camera plane (distortion parameter)

f (...)

Function describing the imaging characteristic of optics.

Ψ

Amount of parameters to be determined

Ψ -

Set of a priori known parameters

i

Index for reference points of the calibration pattern and its mappings i = 1 ... L

k

Index for positions of the mirror k = 1 ... N

If a plane mirror is used, the reflection _D S can be used here D / k using 3 parameters that describe the position and orientation of the mirror plane in space. Model:

Formula 1 describes the mapping of the graphics memory data points _G p to image memory coordinates _B p. Formula 2 includes all the parameters that occur in this figure, except for the explicitly searched location _K T ^{D of} the object relative to the camera system. In this case, the set Ψ includes all the parameters that are determined during the determination of the explicitly searched location _K T ^D , while the set Ψ includes all parameters that are known a priori. Both sets are disjoint as shown in formula 3. The mirroring _D S is also included in the parameters that are to be implicitly determined, that is, not sought after and not explicitly determined D / k contained by the mirror system (Formula 4). An error function may be established that defines a measure of the deviation of the measurement points from the data points expected by the imaging model and parameters (Formula 5). The position of the object relative to the camera system _k T ^D is determined by means of nonlinear minimization methods together with the parameters Ψ to be determined, also called nuisance parameters.

It is understood that the accuracy of the estimation of the parameters of the mapping function is facilitated, for example, by means of a priori knowledge, for example about the distortion.

Preferred applications of the invention are, for example, autostereoscopic devices, web cameras or on-board camera systems in motor vehicles.

LIST OF REFERENCE NUMBERS

1

contraption

2

camera system

3

display

4

mirror

5

adjustment

6

calibration

7

exemplary beam path

8th

calculation unit

9

Device for image analysis

10

Device for calculating at least one parameter of a mapping function

Claims (5)

Method for determining a position of a camera system ( 2 ) with respect to an object, wherein the camera system ( 2 ) comprises at least one camera, wherein at least one mirror ( 4 ) is arranged such that at least parts of a calibration element on the mirror ( 4 ) in a field of view of the camera system ( 2 ) and from the camera system ( 2 ), characterized in that the camera system ( 2 ) for at least one indeterminate position of the mirror ( 4 ) receives at least one image with an image content, wherein the image content comprises at least a first part, wherein the first part exclusively from that of the mirror ( 4 ) in the field of vision of the camera system ( 2 ), wherein by means of numerical methods at least one parameter of an imaging function is exclusively determined from image information of the first part, wherein the imaging function is a mapping of an object point into an image memory of the camera system ( 2 ), wherein the object is the calibration element or at least a part of the object forms the calibration element or a spatial relationship of the calibration element to the object is already known or geometric dimensions of the object and the calibration element are determined during the execution of the method, from which the fixed spatial relationship of the calibration element to the object is determined. Method according to claim 1, characterized in that the camera system ( 2 ) a first image having a first image content in a first indefinite position of the mirror ( 4 ) and at least one second image with a second image content in a second indefinite position of the mirror ( 4 ), wherein the first layer is different from the second layer, wherein by means of numerical methods the at least one parameter of the mapping function is determined from image information of the first part of the first image content and the first part of the second image content. A method according to claim 1 or 2, characterized in that the object is a screen. Method according to Claim 3, characterized in that the calibration element has a calibration pattern ( 6 ), the calibration pattern ( 6 ) is displayed on the screen. Method according to one of the preceding claims, characterized in that the camera system ( 2 ) comprises at least two cameras.

Images