DE102017221721A1 - Device for calibrating a multi-camera system - Google Patents

Abstract

The present invention relates to a device (100) for calibrating a multi-camera system for a motor vehicle, the device (100) comprising: a first calibration pattern; a second calibration pattern; a calibration frame, which is designed to visibly position the first calibration pattern in a field of view of a first camera of the multi-camera system and which is designed to visibly position the second calibration pattern in a field of view of a second camera of the multi-camera system.

Description

Technical area

The present invention relates to devices and display devices for storing and reading card-referenced vehicle data for a motor vehicle.

Technical background

Currently, the transformation between two cameras of a multi-camera system of a motor vehicle with overlapping field of view is determined, for example, by methods of relative pose determination - the term "relative pose" here means position and orientation -, for example by a stereo camera calibration, or one works with the so-called "PnP" - Algorithm, here "PnP" stands for English. "Perspective-n-Point" Camera Pose Estimation.

Therefore, the laser method is not suitable for fast and flexible use.

If the fields of view of the cameras do not overlap, then one way to perform a camera calibration is to use auxiliary cameras, which must be positioned so that the calibration patterns are always within the field of view of at least two cameras.

Over a sufficient number of such auxiliary cameras - and possibly additional calibration pattern - then all transformations can be calculated. The problem with this method, however, is that the calibration error can potentially increase with each additional auxiliary camera. In addition, this solution is very dependent on unrestricted fields of view.

The EP 2 375 376 B1 describes methods and arrangements for calibrating multiple cameras with overlapping fields of view.

The US 2016/343 136 A1 describes methods of operating a data processing system to calibrate a camera arrangement for a vehicle surround view.

The process described therein is based on the image capture of one or more 2D calibration patterns arranged at arbitrary positions in an overlapping field of view between two or more cameras mounted on the vehicle.

Summary of the invention

It is an object of the present invention to provide an improved apparatus for calibrating a multi-camera system for a motor vehicle.

This object is solved by the subject matters of the independent claims. Embodiments and further developments can be taken from the dependent claims, the description and the figures of the drawings.

A first aspect of the present invention relates to a device for calibrating a multi-camera system for a motor vehicle, the device comprising: a first calibration pattern, a second calibration pattern, a calibration frame.

The calibration frame is designed to visibly position the first calibration pattern in a field of view of a first camera of the multi-camera system, and further configured to visibly position the second calibration pattern in a field of view of a second camera of the multi-camera system.

In other words, the present invention makes it possible to convert the viewing direction of a camera by means of a calibration method and optimization method using a calibration rack into the coordinate system of another camera.

The calibration frame can be designed as a double tripod for stable installation and positioning of calibration samples.

The two cameras do not need an overlapping field of view. The methods of the present invention serve to calculate the transformation between at least two cameras of a multi-camera system, for example a driver camera and an environment camera of a motor vehicle.

The present invention enables calibration with increased and improved quality of the results. Furthermore, the present device and the method for calibrating a multi-camera system enable an increased degree of automation. The quality of the results of the calibration therefore also depends less on the person performing the calibration. It is also possible to fully automate the present calibration procedure.

The present invention advantageously makes it possible to eliminate the determination of the transformation via laser-based surveying equipment.

In such laser-based surveying instruments, the laser must be tuned very accurately and be able to detect and output a certain generally quite high number of coordinates, for example over 20 different coordinates.

Therefore, such a laser-based measurement is time-consuming, since many individual points of a checkerboard pattern must be measured by the laser system. Furthermore, the accuracy of such a laser-based method depends heavily on the person performing the calibration, since the points to be measured must be manually targeted.

The present invention advantageously enables laser-based methods for calibration to be avoided.

A second aspect of the present invention relates to a method for calibrating a multi-camera system for a motor vehicle, the method comprising the following method steps:

As a first step, fixing a first calibration pattern visible in the field of view of a first camera of the multi-camera system by means of a calibration rack and fixing a second calibration pattern visible in the field of view of a second camera of the multi-camera system by means of the calibration frame, wherein the first calibration pattern and the second calibration pattern in a relative pose are fixed to each other. The calibration rack thus serves as a tripod for the first calibration pattern and the second calibration pattern.

Furthermore, the method comprises performing a plurality of point measurements at different positions of the first calibration pattern visible in the field of view of the first camera and the second calibration pattern visible in the field of view of the second camera.

Furthermore, the method comprises determining a first transformation X between the first camera and the second camera and a second transformation Y between the first calibration pattern and the second calibration pattern.

The present invention advantageously makes it possible to provide the use of a solution method for determining the relative pose between both cameras using a suitable calibration device.

Advantageous embodiments of the present invention are characterized in the subclaims.

In an advantageous embodiment of the present invention it is provided that the device further comprises a robot arm device, which is designed to position the calibration frame.

In an advantageous embodiment of the present invention, it is provided that the device further comprises a robot arm device which has a first robot arm, which is designed to position the first calibration pattern; wherein the robotic arm device further comprises a second robotic arm configured to position the second calibration pattern.

This advantageously makes it possible to also calibrate cameras in a multi-camera system of the motor vehicle, which are on opposite sides of the vehicle or far away from each other, for example more than 2 m or which have different, opposite viewing directions, which have no overlap or which, for example, a field of view of over 30 °.

In a further advantageous embodiment of the present invention, it is provided that the determination of the first transformation X comprises determining a first rotation matrix and a first translation vector.

In a further advantageous embodiment of the present invention, it is provided that the determination of the first rotation matrix is performed by optimizing an initial estimate of a first rotation and determining the first translation vector based on the optimization of the initial estimate.

In a further advantageous embodiment of the present invention, it is provided that the determination of the second transformation Y comprises determining a second rotation matrix and a second translation vector.

In a further advantageous embodiment of the present invention, it is provided that the determination of the second rotation matrix is performed by optimizing an initial estimate of a second rotation and determining the second translation vector based on the optimization of the initial estimate.

The described embodiments and developments can be combined with each other as desired.

Other possible embodiments, developments and implementations of the present The invention also does not explicitly include combinations of features of the present invention described above or below with respect to the embodiments.

The accompanying drawings are intended to provide further understanding of the embodiments of the present invention. The accompanying drawings illustrate embodiments and, together with the description, serve to explain concepts of the present invention.

Other embodiments and many of the stated advantages will become apparent with reference to the figures of the drawings. The illustrated elements of the figures of the drawings are not necessarily shown to scale to each other.

list of figures

• 1: eine schematische Darstellung einer Vorrichtung zum Kalibrieren eines Multikamerasystems für ein Kraftfahrzeug gemäß einem Ausführungsbeispiel der vorliegenden Erfindung;
• 2: eine schematische Darstellung einer Vorrichtung zum Kalibrieren eines Multikamerasystems für ein Kraftfahrzeug gemäß einem Ausführungsbeispiel der vorliegenden Erfindung; und
• 3: eine schematische Darstellung eines Flussdiagramms eines Verfahrens zum Kalibrieren eines Multikamerasystems für ein Kraftfahrzeug gemäß einem weiteren Ausführungsbeispiel der vorliegenden Erfindung.

Show it:

• 1 a schematic representation of an apparatus for calibrating a multi-camera system for a motor vehicle according to an embodiment of the present invention;
• 2 a schematic representation of an apparatus for calibrating a multi-camera system for a motor vehicle according to an embodiment of the present invention; and
• 3 : is a schematic representation of a flowchart of a method for calibrating a multi-camera system for a motor vehicle according to another exemplary embodiment of the present invention.

Detailed description of the embodiments

In the figures of the drawings, like reference characters designate like or functionally identical elements, components, components or method steps, unless indicated otherwise.

The motor vehicle or vehicle is, for example, a motor vehicle or a hybrid vehicle, for example a hybrid vehicle with a sailing function, for example a motorcycle, a bus or a lorry or even a rail vehicle, a ship, an aircraft such as a helicopter or airplane or for example a bicycle.

The method of calibrating a multi-camera system may be used for areas that deal with multi-camera based security systems, such as multi-camera systems that include a driver and / or surround camera, or provide surround view functions, or a front camera and / or a rear Camera include.

For example, the term camera calibration as used by the present invention refers to a method that can determine the parameters of a camera or the camera model. The camera calibration allows the three-dimensional interpretation of two-dimensional image data. Based on the image data and the camera calibration, three-dimensional objects can be measured or their position in space determined.

The method for calibrating a multi-camera system can also be used for the camera-assisted tracking of objects due to the increasing use of multi-camera systems, since the relationship between two perspectives and possibly several perspectives can be established.

For example, the present invention allows a detected deviation of less than 1.5 ° in the rotation and less than 6.5 cm in translation to be achieved after the calibration has been performed.

The 1 shows a schematic representation of an apparatus for calibrating a multi-camera system for a motor vehicle according to an embodiment of the present invention.

The device is designed, for example, for calibrating a stereo camera system for a camera calibration based on the use of a calibration body in order to carry out a reconstruction of a - for example three-dimensional - environment model of a vehicle.

The device is based, for example, on a calibration method, for example a hand-eye calibration, based on an extrinsic transformation.

With the present calibration frame or calibration frame and the application of a solution method for solving the equation AX = YB, the desired extrinsic transformation can be determined.

Necessary for the application of the method is that the relative position of the calibration pattern K1 and K2 does not change in the course of calibration.

Furthermore, the calibration patterns can be moved to different positions in front of the respective cameras to be calibrated C1 . C2 be placed to get more point measurements.

This can be ensured for example by a suitable calibration rack.

In a further embodiment of the present invention it is provided that the calibration frame is attached to a robot arm or each calibration pattern is fixed by its own robot arm. In the latter case, when repositioning or repositioning the calibration patterns, it must again be ensured that the relative pose of the patterns does not change with respect to one another.

The calibration can be made possible, for example via camera calibration and robot hand-eye calibration, to carry out the transformation with a present setup of two calibration patterns fixed relative to one another.

drücken A und B jeweils die Transformationen von Kalibiermustern - etwa von den dargestellten Schachbrettern - zu den jeweiligen Kameras C1, C2 aus, wie in 1 dargestellt.In the equation: $$\text{AX}=\text{YB}$$

A and B respectively express the transformations of calibration patterns - such as the illustrated chess boards - to the respective cameras C1 . C2 out, like in 1 shown.

These can be determined using standard methods for determining poses (such as PnP) and are thus known.

Unknown are the sought transformation X between the two cameras and Y between the calibration patterns. The calibration frame is used to calibrate the samples K1 and K2 firmly connect with each other.

The calibration rack is shown in the in 1 is the sketch of the experimental setup and is used to maintain the position of both chessboards to each other during the performance of the calibration procedure based on a solution method of Equation (1).

The transformation X between the two cameras is also fixed, as the cameras are rigidly mounted to the vehicle.

Equation (1) can be solved by a method called a simultaneous solution method in which it simultaneously determines the transformation between the cameras and the calibration patterns. For every transformation A between chessboard K1 and camera C1 a transformation is taking place at the same time B between chessboard K2 and camera C2 ,

A transformation A is always a corresponding transformation B assigned. The program then calculates the transformations X and Y using some of these transformation pairs A and B ,

The method first optimizes the rotation matrices and then calculates the translation vector from them. The rotation matrices are called optimization because the process first requires initial estimates of the rotations between the cameras and the calibration patterns. These are then corrected by the procedure. An approximate approximation is enough for the initial rotation matrices.

The 2 shows a schematic representation of an apparatus for calibrating a multi-camera system for a motor vehicle according to an embodiment of the present invention.

The device 100 For calibrating a multi-camera system for a motor vehicle, a calibration frame KG comprises a first calibration pattern K1 and a second calibration pattern K2 ,

The first calibration pattern K1 and the second calibration pattern K2 For example, are designed as a black and white pattern, the device 100 can also have other calibration patterns, the patterns can be printed on paper and then mounted on a solid plate. Typically, the calibration patterns form a rectangular grid with identical spacings between the grid points.

The calibration frame KG is designed to be the first calibration pattern K1 visibly positioned in a field of view of a first camera of the multi-camera system and is further adapted to the second calibration pattern K2 visibly position in a field of view of a second camera of the multi-camera system.

The 3 a schematic representation of a flowchart of a method for calibrating a multi-camera system for a motor vehicle according to another embodiment of the present invention.

As a first method step, fixing takes place S1 a first calibration pattern K1 visible in the field of view of a first camera of the multi-camera system and fixing a second calibration pattern K2 visible in the field of view of a second camera of the multi-camera system by means of a calibration rack, wherein the first calibration pattern and the second calibration pattern are fixed relative to each other in a relative pose.

As a second method step is carried out S2 a plurality of point measurements at different positions of the first calibration pattern visible in the field of view of the first camera and the second calibration pattern visible in the field of view of the second camera.

As a third method step, a determination is made S3 a first transformation X between the first camera and the second camera and a second transformation Y between the first calibration pattern and the second calibration pattern.

Although the present invention has been described above with reference to preferred embodiments, it is not limited thereto but is modifiable in a variety of ways. In particular, the present invention can be varied or modified in many ways without departing from the gist of the invention.

In addition, it should be noted that "comprising" and "having" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality.

It should also be appreciated that features or steps described with reference to any of the above embodiments may also be used in combination with other features or steps of other embodiments described above. Reference signs in the claims are not to be considered as limiting.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• EP 2375376 B1 [0006]
• US 2016/343136 A1 [0007]

Claims (8)

Apparatus (100) for calibrating a multi-camera system for a motor vehicle, the apparatus (100) comprising: a first calibration pattern (K1); a second calibration pattern (K2); - A calibration frame (KG), which is adapted to visibly position the first calibration pattern in a field of view of a first camera of the multi-camera system and which is adapted to visibly position the second calibration pattern in a field of view of a second camera of the multi-camera system. Device (100) according to Claim 1 wherein the device further comprises a robot arm device which is adapted to position the calibration frame (KG). Device (100) according to Claim 1 wherein the apparatus further comprises robotic arm means having a first robotic arm adapted to position the first calibration pattern (K1); wherein the robotic arm device further comprises a second robot arm configured to position the second calibration pattern (K2). Method for calibrating a multi-camera system for a motor vehicle, the method comprising the following method steps: Fixing (S1) a first calibration pattern visible in the field of view of a first camera of the multi-camera system and a second calibration pattern visible in the field of view of a second camera of the multi-camera system by means of a calibration rack, wherein the first calibration pattern and the second calibration pattern are fixed relative to each other in a relative pose; - performing (S2) a plurality of point measurements at different positions of the first calibration pattern visible in the field of view of the first camera and the second calibration pattern visible in the field of view of the second camera; and Determining (S3) a first transformation X between the first camera and the second camera and a second transformation Y between the first calibration pattern and the second calibration pattern. Method according to Claim 4 wherein determining (S3) the first transformation X comprises determining a first rotation matrix and determining a first translation vector. Method according to Claim 5 wherein determining (S3) the first rotation matrix optimizes an initial estimate of a first rotation and determining (S3) the first translation vector based on optimizing the initial estimates of the first rotation. Procedure according to one of the previous Claims 4 to 6 wherein determining (S3) the second transformation Y comprises determining a second rotation matrix and a second translation vector. Method according to Claim 7 wherein determining (S3) the second rotation matrix optimizes an initial estimate of a second rotation; and wherein determining (S3) the second translation vector is made based on optimizing the initial estimate.

Images