EP4222705A1

EP4222705A1 - Method for capturing three-dimensional images with the aid of a stereo camera having two cameras, method for producing a redundant image of a measurement object, and device for carrying out the methods

Info

Publication number: EP4222705A1
Application number: EP21785832.3A
Authority: EP
Inventors: Jens Schick; Michael Scharrer
Original assignee: Tripleye GmbH
Current assignee: Tripleye GmbH
Priority date: 2020-09-29
Filing date: 2021-09-28
Publication date: 2023-08-09
Also published as: WO2022069424A1; JP2023543946A; US20230377196A1; KR20230078780A; DE102020212285A1

Abstract

To capture three-dimensional images with the aid of a stereo camera (55a) having two cameras (54, 55), an image of a three-dimensional scene is first captured (64) with the two cameras (54, 55) simultaneously. Characteristic signatures of scene objects within each captured image are determined (65) and assigned to each other in pairs (66). Characteristic position deviations of the assigned signature pairs from one another are determined (67). The position deviations are filtered in order to select assigned signature pairs. On the basis of the selected signature pairs, a triangulation calculation is carried out (69) to determine depth data for the respective scene objects. A 3D data map of the captured scene objects within the captured image of the three-dimensional scene is then created (70). This results in a method for capturing three-dimensional images, which is well adapted for practical use, in particular for capturing images to safeguard autonomous driving.

Description

Method for spatial image acquisition using a stereo camera having two cameras and method for generating a redundant image of a measurement object and device for carrying out the method

The present patent application claims the priority of German patent application DE 10 2020 212 285.7, the content of which is incorporated herein by reference.

The invention relates to a method for spatial image acquisition using a stereo camera having two cameras. Furthermore, the invention relates to a method for generating a redundant image of a measurement object and a device for carrying out this method.

An object detection device is known from WO 2013/020872 A1 and the references given there.

It is an object of the present invention to specify a method for spatial image acquisition which is well adapted to the practice of image acquisition in particular to ensure autonomous driving.

According to the invention, this object is achieved by a method having the features specified in claim 1 .

Positional deviations between signatures of scenery objects, which result due to positional or positional deviations of the cameras, can be precisely recognized and compensated for with this method.

Scenery objects can be comparatively small objects whose images are, for example, only individual pixels or less than ten pixels of the respective camera. Alternatively, the scenery objects can also be larger and encompass entire image areas. Examples of such larger scenery objects are individual vehicle components or vehicle sections or the entire vehicle in the example of vehicle image acquisition. Examples of characteristic positional deviations that are determined in the image acquisition method are a deviation or a distance of an imaging position of the imaging of scenery objects from an epipolar line of the respective camera. A positional deviation of the respective imaging position of a scenery object along the epipolar line can also be considered in the determination method. These positional deviations are also referred to below as vertical disparity and horizontal disparity.

When filtering the positional deviations, it can simply be checked whether the positional deviations are smaller than a specified positional deviation tolerance value. The specification of the position deviation tolerance value is fluid and can take place until the number of selected signature pairs is less than the specified limit value.

Instead of a stereo camera, more than two cameras can also interact in the image acquisition method.

Each of the cameras can in turn be constructed as a system of several cameras assigned to one another. One of these associated cameras can be a fisheye camera. The focal length of such a fisheye camera can be less than 20 mm. Another of these associated cameras may be a telephoto camera with a telephoto lens with a focal length of at least 80mm.

To determine the characteristic positional deviations of the associated signature pairs from one another and to filter the positional deviations, the respective vertical disparity for all signature pairs can be summed up squared and state variables of the stereo camera can be varied until this squared sum is minimized.

A positional deviation tolerance value in the form of a specification of a squared sum of the vertical disparity or also a standard deviation of the vertical disparity can be used as a termination criterion for reducing the number of selected signature pairs to less than a specified limit value.

In particular, relative movements between the cameras can be taken into account in the image acquisition method. One about determining the characteristic positional deviations associated signature pairs occurring relative position estimation of the cameras to each other can be done asynchronously to the triangulation calculation.

A position correction of a relative position of the cameras to one another, in particular on the basis of an estimate of the relative position, can also be carried out immediately before each triangulation calculation. There can then be a camera position calibration for each measurement process. With such a direct calibration, reliable stereo measurements can be carried out even when using image acquisition devices in which the position of the cameras relative to one another is constantly changing. Such a position correction based in particular on an estimation of the relative position of the camera can be carried out using data from inertial measuring units to which the cameras are permanently connected. In particular, the use of camera measuring devices with a very large distance between the cameras (long baseline stereoscopy) is possible.

Known rectification methods can be used in the triangulation calculation. Reference is made to the specialist books by Karl Kraus, Photogrammetry, Volumes 1 and 2, Dümmler, Bonn, 1996/1997 and Thomas Luhmann, Close-range photogrammetry fundamentals, methods and applications, 3rd edition, Berlin/Offenbach 2010.

In the triangulation calculation, an optical distance measurement is carried out by measuring angles within triangles defined within the framework of the method. These can each be formed by one of the two cameras and two pixels or by the two cameras and one pixel.

When filtering the offset deviations for the selection of associated signature pairs, those signature pairs can be selected which have a higher probability of belonging to the same scenery object of the three-dimensional scenery.

The filtering of the positional deviations for the selection of assigned signature pairs can be done according to claim 2 with a filter algorithm. A measure of the respective positional deviation that can be used for filtering is, for example, the horizontal disparity and/or the vertical disparity. With the image acquisition method according to claim 4 or 5, even spatial scenes that are difficult to acquire can be reliably assigned with regard to the scenery objects and corresponding 3D data maps can be created without errors. Deviations in the relative alignment of the cameras of the stereo camera to one another can be precisely taken into account. The angle correction values determined in the determination step can be used as state variables for the variation mentioned above, e.g. B. can be used to minimize the sum of squares of the vertical disparity.

The angle correction values are angles that are characteristic of the relative positional relationship of the cameras to one another, for example a tilting of a baseline plane, a baseline tilting, a relative tilting of the camera image recording directions or a tilting of camera coordinate axes to one another.

The determination method according to claim 6 leads to particularly precise 3D data values.

The method according to claim 7 allows the inclusion of instantaneous changes in the position of the cameras relative to one another, for example due to vibrations. This data collection then takes place in real time. A time constant of this data acquisition can be less than 500 ms and in particular can be less than 100 ms. The data recorded by the inertial measuring units can provide raw information for a necessary correction of the relative position of the cameras to one another, which is then optimized by the image recording. A corresponding position change detection can also be used to calibrate the positions of the cameras relative to one another. This can be done in particular before carrying out the respective triangulation step when carrying out the image acquisition method.

A further object of the invention is to improve the data acquisition security when imaging a measurement object.

According to the invention, this object is achieved by a method having the features specified in claim 8 .

According to the invention, it was recognized that the various cameras that are provided, for example, in a camera system to enable the autonomous method, with one another for redundant imaging of a measurement object. The cameras are linked accordingly so that the recording results of a pair of cameras can be compared and checked using a third camera. In this way, detection errors can be detected and genuine redundancy can even be generated by comparing three independent detection results with one another, with the detection data being qualified as correct if at least two of the three detection results match one another. The measurement object can be arbitrary, as long as the measurement object has textures or structures and contains depth information about them.

The advantages of a device according to claims 9 and 10 correspond to those which have already been explained above in connection with the methods. The device can have at least one camera permanently connected to an inertial measuring unit.

The three cameras can be arranged in the form of a triangle, in particular in the form of an isosceles triangle.

An arrangement according to claim 11 with six cameras leads to an even better imaging redundancy. The six cameras can be arranged in the form of a hexagon, in particular in the form of a regular hexagon. The cameras can be arranged in the camera arrangement level.

The use of an additional remote camera according to claim 12 enables a security comparison of the results of the image acquisition by the three cameras arranged adjacent to one another. The three cameras arranged adjacent to one another can be arranged in the camera arrangement plane. A distance factor to characterize the distance of the remote camera compared to the distances of the adjacent cameras can be greater than 2, can be greater than 3, can be greater than 4, can be greater than 5 and can also be 10, for example. This distance factor can be selected so that the adjacent cameras cover a close-up area up to a close-up range limit, and adding the remotely arranged camera from this close-up range limit then covers a far range of the field of view of the cameras.

An arrangement according to claim 13 simplifies the comparison of the triangulation measurements. The connection of at least one of the cameras to an inertial measuring unit also enables an acceleration measurement of the respective camera and a corresponding evaluation of recorded acceleration values.

Exemplary embodiments of the invention are explained in more detail below with reference to the drawing. In this show:

1 shows a top view of a device for calibrating a spatial position of a center of an entrance pupil of a camera, with an additional calibration surface being shown both in a neutral position outside a camera field of view and in an operating position in the camera field of view;

FIG. 2 shows a view from direction II in FIG. 1 with additional calibration surfaces in the neutral position;

FIG. 3 shows a schematic representation to clarify positional relationships between components of the calibration device; FIG.

4 shows a further detailed view of a movable reference camera of the calibration device including a camera displacement drive for moving the movable reference camera in several translational/rotational degrees of freedom;

5 shows schematically different alignments of the movable reference camera, namely a total of eight alignment variants;

6 shows a calibration panel with a calibration surface, having calibration structures that can be used as the main calibration surface and/or as an additional calibration surface in the calibration device; 7 shows an arrangement of a system for determining relative positions of centers of entrance pupils of at least two cameras, which are mounted on a common support frame, in a view from above;

8 shows a schematic of two cameras of a stereo camera for three-dimensional image acquisition, with coordinates and position parameters for determining angle correction values of the cameras relative to one another being illustrated;

FIG. 9 again schematically shows the two cameras of the stereo camera according to FIG. 8 when capturing scenery objects of a three-dimensional scenery, with positional deviation parameters of characteristic signatures of the images captured by the cameras being highlighted;

10 shows a block diagram to clarify a method for spatial image acquisition using the stereo camera according to FIGS. 8 and 9;

11 shows a device for carrying out a method for generating a redundant image of a measurement object using, for example, two groups of three cameras each assigned to common signal processing;

12 shows two cameras of a stereo camera for three-dimensional image acquisition in a representation similar to FIG.

A calibration device 1 is used to calibrate a spatial position of a center of an entrance pupil of a camera 2 to be calibrated. The camera 2 to be calibrated is arranged within a cuboid assembly volume 3, which is highlighted in FIGS. 1 and 2 by dashed lines. The camera 2 to be calibrated is fixedly mounted within the mounting volume 3 when the calibration method is carried out. A holder 4, which is only indicated in FIG. 1, is used for this purpose. The camera 2 to be calibrated is held with the holder 4 in such a way that it captures a predefined calibration field of view 5, the boundaries of which are shown in dashed lines in the side view of the device 1 according to FIG. The camera 2 to be calibrated can be, for example, a camera for a vehicle that is to be used to provide an “autonomous driving” function.

To simplify the description of positional relationships, in particular between cameras of the device 1 and in relation to the field of view 5, an xyz coordinate system is drawn in in each of FIGS. 1 to 3, unless otherwise stated. In FIG. 1, the x-axis runs perpendicular to the plane of the drawing and into it. The y-axis runs upwards in FIG. The z-axis runs to the right in FIG. In FIG. 2, the x-axis runs to the right, the y-axis runs upwards and the z-axis runs perpendicular to the plane of the drawing and out of it.

An entire field of view of the calibration field of view 5 can, for example, cover a detection angle of 100° in the xz plane. Other detection angles between, for example, 10° and 180° are also possible. In principle, it is also possible to calibrate cameras with a detection angle that is greater than 180°.

The holder 4 is fixed to a support frame 6 of the calibration device 1 .

The calibration device 1 has at least two and in the embodiment shown a total of four stationary reference cameras 7, 8, 9 and 10 (cf. FIG. 3), of which only two stationary reference cameras, namely the reference cameras 7 and 8, are visible in FIG . The stationary reference cameras 7 to 10 are also mounted on the support frame 6 . The stationary reference cameras 7 to 10 are used to record the calibration field of view 5 from different directions.

FIG. 3 shows exemplary dimension parameters that play a role in the calibration device 1.

Main lines of sight 11, 12, 13, 14 of the stationary reference cameras 7 to 10 are shown in broken lines in FIG.

These main lines of sight 11 to 14 intersect at a point C (cf. FIGS. 1 and 3). The coordinates of this point of intersection C are denoted by x _c , y _c and z _c in FIGS. An x-distance between the reference cameras 7 and 10 on the one hand and the reference cameras 8 and 9 on the other is denoted by dxh in FIG. An x-coordinate of the stationary reference cameras 7 and 8 on the one hand and 9 and 10 on the other hand is the same in each case.

A y-distance between the stationary reference cameras 7 and 8 on the one hand and 9 and 10 on the other is denoted by dyh in FIG. A y-coordinate of the stationary reference cameras 7 and 10 on the one hand and 8 and 9 on the other hand is the same in each case.

The calibration device 1 also has at least one stationary main calibration surface, in the exemplary embodiment shown three main calibration surfaces 15, 16 and 17, which are specified by corresponding calibration panels. In the arrangement according to FIGS. 1 and 2, the main calibration surface 15 extends parallel to the xy plane and at a z coordinate which is greater than z _c . In the arrangement according to FIGS. 1 and 2, the two further lateral main calibration surfaces 16, 17 extend parallel to the yz plane on both sides of the arrangement of the four stationary reference cameras 7 to 10. The main calibration surfaces 15 to 17 are also stationary mounted on the support frame 6.

The main calibration surfaces have stationary main calibration structures, examples of which are shown in FIG. In any case, some of these calibration structures are arranged in the calibration field of view 5 . The main calibration structures can have a regular pattern, for example arranged in the form of a grid. Corresponding grid points, which are part of the calibration structures, are shown at 18 in FIG. The calibration structures can have colored pattern elements, as illustrated at 19 in FIG. Furthermore, the calibration structures can have different sizes. In comparison to the raster points 18, pattern elements that are enlarged as the main calibration structures are emphasized at 20 in FIG. Furthermore, the main calibration structures can have encoded pattern elements, for example QR codes 21 (cf. FIG. 6).

An arrangement of the main calibration surfaces 15 to 17 aligned with the xyz coordinate system according to FIGS. 1 and 2 is not mandatory. FIG. 3 shows an example of a tilted arrangement of a main calibration surface 15′, which is at an angle to the xy plane, for example. In addition, FIG. 3 shows an external coordinate system XYZ, for example of a production hall in which the calibration device 1 is accommodated. The xyz coordinate system of the calibration device 1 on the one hand and the XYZ coordinate system of the production hall can be tilted relative to one another, as illustrated in FIG. 3 by a tilt angle rot _z .

The main calibration structures 15 to 17, 15' are therefore in a main calibration structure main plane (xy plane in the arrangement according to FIGS. 1 and 2) and additionally in a main calibration structure angle plane (yz plane in the arrangement according to FIGS. 1 and 2), the main calibration structure main plane xy being arranged at an angle greater than 5°, namely at an angle of 90°, to the main calibration structure angular plane yz. Depending on the design, this angle to the main calibration structure angle plane yz can be greater than 10°, can be greater than 20°, can be greater than 30°, can be greater than 45° and can also be greater than 60°. Small angles, for example in the range between 1° and 10°, can be used to approximate a curved calibration structure surface with the main calibration structures.

In the arrangement according to FIGS. 1 and 2 and additionally in the arrangement of the main calibration surface 15' according to FIG. 3, more than two main calibration structure surfaces 15 to 17, 15' can be arranged in different main calibration structure planes.

A position of the respective main calibration surface, for example the main calibration surface 15' in comparison to the xyz coordinate system can be defined via a position of a center of the main calibration surface and two tilt angles of the main calibration surface 15' to the xyz coordinates. A further parameter with which the respective main calibration surface 15 to 17 or 15' is characterized is a grid spacing grid of the grid points 18 of the calibration structure, which is illustrated in FIG. The two grid values that are given horizontally and vertically for the main calibration surface 15' in FIG. 6 do not necessarily have to be equal to one another, but they must be firmly specified and known.

The positions of the colored pattern elements 19, the enlarged pattern elements 20 and/or the coded pattern elements 21 within the grid of the grid points 18 are also fixed in each case for the main calibration area 15 to 17, 15'. These positional relationships of the various pattern elements 18 to 21 to one another serve to identify the respective main calibration surface, to determine the absolute position of the respective main calibration surface im Space. The enlarged pattern elements 20 can be used to support the respective position determination. Different sizes of the pattern elements 18 to 20 and also the encoded pattern elements 21 enable a calibration measurement in the near and far range as well as a measurement in which the main calibration surfaces 15 to 17, 15' are tilted sharply to the xy plane.

Furthermore, the calibration device 1 has at least one and, in the embodiment shown, three additional calibration surfaces 22, 23 and 24 with additional calibration structures 25. The additional calibration surfaces 22 to 24 are realized by shell-shaped calibration panels. The additional calibration structures 25 are arranged on the respective additional calibration area 22 to 24 in the form of a 3×3 grid. The additional calibration structures 25 can in turn each have pattern elements in the manner of the pattern elements 18 to 21 which were explained above in connection with the main calibration areas.

The additional calibration surfaces 22 to 24 are mounted together on a movable holding arm 26 . The latter can be pivoted about a pivot axis 28, which runs parallel to the x-direction, via a geared motor 27, ie a calibration surface displacement drive. The additional calibration structures 22 to 24 can be displaced between a neutral position and an operating position via the geared motor 27 . The neutral position of the additional calibration structures 22 to 24 is shown in solid lines in FIGS. 1 and 2, in which these are arranged outside of the calibration field of view 5. An operating position of the holding arm 26 and the additional calibration surfaces 22 to 24 which is swiveled up compared to the neutral position is shown in dashed lines in FIG. The additional calibration surfaces 22 to 24 are arranged in the calibration field of view 5 in the operating position.

In the operating position, a central additional calibration structure 25z (cf. also FIG. 3), for example, as shown in FIGS. 1 and 2, lies parallel to the xy plane. In the operating position, the 3×3 grid arrangement of the additional calibration structures 25 is then present in three rows 251, 252 and 25s running along the x-direction and in three columns running parallel to the y-direction. Each adjacent row and each adjacent column of the additional calibration structures 25 are tilted to one another by a tilt angle α, which can be in the range between 5° and 45°, for example in the range of 30°. For the four grid areas arranged in the corners of the 3×3 grid, this tilting by the tilting angle a is two on top of one another vertical axes. This then results in the shell-shaped basic structure of the respective additional calibration surface 22 to 24. The additional calibration structures 25 are present in a 3D arrangement that deviates from a flat surface.

The calibration device 1 also includes an evaluation unit 29 for processing camera data recorded by the camera 2 to be calibrated and the stationary reference cameras 7 to 10 as well as status parameters of the device, i.e. in particular the position of the additional calibration surfaces 22 to 24 and the main calibration surfaces 15 to 17 and of positions and line-of-sight courses of the reference cameras 7 to 10. The evaluation unit 29 can have a memory for image data.

The calibration device 1 also includes a movable reference camera 30 which is also used to record the calibration field of view 5 .

FIG. 3 illustrates degrees of freedom of movement of the movable reference camera 30, namely two degrees of freedom for tilting and one degree of freedom for translation.

Figure 4 shows details of the movable reference camera 30. This can be moved via a camera displacement drive 31 between a first field of view recording position and at least one other field of view recording position, which extends from the first field of view recording position in an image recording direction (cf. the recording direction 32 in Figure 1) differs.

The camera displacement drive 31 includes a first pan motor 33, a second pan motor 34 and a linear displacement motor 35. The camera head 36 can be pivoted about an axis parallel to the x-axis via the first pivoting motor 33 . The first swing motor 33 is mounted on a swing member of the second swing motor 34 via another support plate 38 . Pivoting of the camera head 36 about a pivoting axis parallel to the y-axis is possible via the second pivoting motor 34 . The second swivel motor 34 is mounted on a linear displacement unit 40 of the linear displacement motor 35 via a bracket 39 . A linear displacement of the camera head 36 parallel to the x-axis is possible via the linear displacement motor 35 .

The camera displacement drive 31 and also the camera head 36 of the reference camera 30 are in signal communication with the evaluation unit 29 . The position of the camera head 36 is transmitted exactly to the evaluation unit 29 as a function of the position of the motors 33 to 35 and also as a function of the installation situation of the camera head 36 with respect to the first swivel motor 33 .

The angular position of the camera head 36 that can be predetermined via the first swivel motor 33 is also referred to as the pitch angle. Instead of the first swivel motor 33, a pitch angle can also be changed via a joint connection of the camera head 36 via a joint axis parallel to the x-axis and a linear drive that can be displaced in the y-direction and has two stops for specifying two different pitch angles, which is connected to the camera head 36. be realised. The angular position of the camera head 36 that can be predetermined via the second swivel motor 34 is also referred to as the yaw angle.

FIG. 5 illustrates, by way of example, eight variants of positioning of the camera head 36 of the movable reference camera 30 using the three degrees of freedom of movement, which are illustrated in FIG.

The image recording direction 32 is shown in broken lines as a function of the respectively set pitch angle ax and yaw angle ay. In the upper line of FIG. 5, the camera head is at a small x-coordinate x _m in . In the bottom line of FIG. 5, the camera head 36 is located at an x coordinate x _max that is larger than this. The eight imaging directions according to FIG. 5 represent different parameter triples (position x; ax; ay) each with two discrete values for each of these three parameters.

In a variant of the calibration device, the movable reference camera 30 can also be dispensed with.

The calibration device 1 is used as follows to calibrate a spatial position of a center of an entrance pupil of the camera 2 to be calibrated: First, the camera 2 to be calibrated is held in the mount 4 .

The stationary main calibration surfaces 15 to 17 and 15' are then recorded with the camera 2 to be calibrated and the reference cameras 7 to 10 and 30, with the additional calibration surfaces 22 to 24 being in the neutral position.

The additional calibration surfaces 22 to 24 are then shifted between the neutral position and the operating position with the calibration surface displacement drive 27 . The additional calibration surfaces 22 to 24 are then recorded with the camera 2 to be calibrated and with the reference cameras 7 to 10 and 30, with the additional calibration structures 25 being in the operating position. The recorded image data of the camera 2 to be calibrated and of the reference cameras 7 to 10 and 30 are then evaluated with the evaluation unit 29 . This evaluation is carried out via a vector analysis of the recorded image data, taking into account the positions of the recorded calibration structures 18 to 21 and 25.

When capturing the main calibration surfaces 15 to 17 and the additional calibration surfaces 22 to 24, the main calibration surfaces 15 to 17, 15' on the one hand and the additional calibration surfaces 22 to 24 on the other hand can be captured with the movable camera 30 in the first field of view recording position and, after moving the movable reference camera 30 with the camera displacement drive 31, in the at least one further field of view recording position, the image data of the movable reference camera 30 in the at least two field of view recording positions also being taken into account when evaluating the recorded image data .

A detection order of the calibration areas 15 to 17 and 22 to 24 can be as follows: First, the main calibration areas 15 to 17 are recorded with the movable camera 30 in the first field of view recording position. Then the additional calibration surfaces 22 to 24 are shifted into the operating position and again recorded with the movable reference camera 30 in the first field of view recording position. The movable reference camera 30 is then shifted to the further field of view recording position, with the additional calibration surfaces 22 to 24 remaining in the operating position. The additional calibration surfaces 22 to 24 with the movable reference camera 30 in the further field of view recording position. The additional calibration surfaces 22 to 24 are then shifted to the neutral position and the main calibration surfaces 15 to 17 are further recorded with the movable reference camera in the further field of view recording position. During this sequence, in periods when the additional calibration surfaces 22 to 24 are in the neutral position, the main calibration surfaces 15 to 17 can also be recorded with the stationary reference cameras 7 to 10 and, insofar as the additional calibration surfaces 22 to 24 are in of the operating position, a detection of these additional calibration surfaces 22 to 24 with the stationary reference cameras 7 to 10.

A system 41 for determining relative positions of centers of entrance pupils of at least two cameras 42, 43, 44 to one another, which are mounted on a common support frame 45, is described below with reference to FIG.

The cameras 42 to 44 can be calibrated in advance with the help of the calibration device 1 with regard to the position of their respective entrance pupil center.

A nominal position of the cameras 42 to 44 relative to the support frame 45, that is to say a target installation position, is known when this relative position determination is carried out using the system 41.

The cameras 42 to 44 can be, for example, cameras on a vehicle that are to be used to provide an “autonomous driving” function.

The system 41 has several calibration structure support components 46, 47, 48 and 49. The calibration structure support component 46 is a master component for specifying a master coordinate system xyz. The x-axis of this master coordinate system runs to the right in FIG. 7, the y-axis runs upwards and the z-axis runs out of the plane of the drawing perpendicularly.

What was explained above in relation to the calibration structures 18 to 21 in connection in particular with FIG. 6 applies to the calibration structures which are applied to the calibration structure carrier components 46 to 49 . The calibration structure support components 46 to 49 are arranged around the support frame 45 in an operating position of the system 41 such that each of the cameras 42 to 44 captures at least calibration structures of two of the calibration structure support components 46 to 49 . Such an arrangement is not mandatory, it is therefore possible for at least one of the cameras 42 to 44 to record calibration structures of just exactly one of the calibration structure carrier components 46 to 49 . The arrangement of the calibration structure carrier components 46 to 49 is also such that at least one of the calibration structures on exactly one of the calibration structure carrier components 46 to 49 is recorded by two of the cameras 42 to 44 . In order to ensure these conditions, the support frame 45 can, if necessary, be displaced relative to the calibration structure support components 46 to 49, which do not change their respective positions.

Figure 7 illustrates a position example of the support frame 45 with the actual positions of the cameras 42, 43, 44 on the support frame (not shown again) such that a field of view 50 of the camera 42 captures the calibration structures of the calibration structure support components 46 and 47, while the camera 43 with its field of view 51 captures the calibration structures of the calibration structure carrier components 47 and 48 and while the further camera 44 captures the calibration structures of the calibration structure carrier components 48 and 49 with its field of view 52 .

A position of the calibration structure carrier components 46 to 49 relative to one another does not have to be strictly defined in advance, but must not change using the system 41 during the position determination method.

System 41 also includes an evaluation unit 53 for processing recorded camera data from cameras 42 to 44 and, if applicable, status parameters during position determination, i.e. in particular an identification of the respective support frame 45.

To determine the relative positions of the entrance pupil centers of cameras 42-44, system 41 is used as follows:

In a first preparatory step, the cameras 42 to 44 are mounted on the common support frame 45 . In a further preparatory step, the calibration structure carrier components 46 to 49 are arranged around the support frame 45 as a group of calibration structure support components. This can also be done in such a way that the group of calibration structure support components 46 to 49 is laid out in a preparatory step and the support frame is then positioned relative to this group. In addition, the xyz coordinate system is defined via the alignment of the master component 46 . The other calibration structure support components 47 to 49 do not have to be aligned with this xyz coordinate system.

The calibration structure support components 46 to 49 in the field of view of the cameras 42 to 44 are now detected in a predetermined relative position of the support frame 45 to the group of calibration structure support components 46 to 49, for example in the actual position of the cameras 42 to 44 according to Figure 7 The image data recorded by the cameras 42 to 44 are then evaluated with the evaluation unit 53 so that the exact positions of the centers of the entrance pupils and also the image recording directions of the cameras 42 to 44 in the coordinate system of the master component 46 are determined. These actual positions are then converted into coordinates of the support frame 45 and compared with the nominal target positions. This can be done as part of a best-fit procedure.

In the determination method, the support frame can also be shifted between different camera detection positions in such a way that at least one of the cameras whose relative position is to be determined detects a calibration structure support component that was not previously detected by this camera. This step of detecting and moving the support frame can be repeated until the condition is met for all cameras whose relative positions to one another are to be determined, that each of the cameras detects at least calibration structures of two of the calibration structure support components, with at least one of the Calibration structures is captured by two of the cameras.

A method for spatial image acquisition using a stereo camera 55a having two cameras 54, 55 is described below with reference to FIGS. These cameras 54 to 55 can have been calibrated in a preparatory step with the aid of the calibration device 1 and also measured with the aid of the system 41 with regard to their relative position. The cameras 54, 55 are in turn mounted on a support frame. The camera 54 shown on the left in FIG. 8 is used as the master camera to define a master coordinate system xm, ym and zm, where zm is the image recording direction of the master camera 54. The second camera 55 shown on the right in FIG then the slave camera.

The master camera 54 is permanently connected to an inertial master measurement unit 56 (IMU), which can be embodied as a yaw rate sensor, in particular in the form of a micro-electro-mechanical system (MEMS). The master measurement unit 56 measures angle changes in a pitch angle daxm, a yaw angle day _m and a roll angle daz _m of the master camera 54 and thus allows position changes in the master coordinate system xm, ym, zm to be monitored in real time. A time constant of this real-time attitude change detection can be better than 500 ms, can be better than 200 ms and can also be better than 100 ms.

The slave camera 55 is also permanently connected to an associated inertial slave measuring unit 57, via which angle changes in a pitch angle dax _s , a yaw angle day _s and a roll angle daz _s of the slave camera 55 can be recorded in real time, so that relative changes in the Slave coordinate system xs, ys, zs to the master coordinate system xm, ym, zm can in turn be detected in real time. Movements of the cameras 54, 55 of the stereo camera 55a relative to one another can be recorded in real time via the measuring units 56, 57 and included in the three-dimensional image recording method. The measuring units 56, 57 can be used to predict a change in the position of the cameras 54, 55 relative to one another. Image processing that takes place as part of the spatial image acquisition can then improve this relative position prediction. Even if, for example, due to a movement of a supporting frame on which the stereo camera 54a is accommodated, the cameras 54, 55 move continuously towards one another on an uneven surface, a stable result of the three-dimensional image acquisition still results.

A connecting line between the centers of the entrance pupils of the cameras 54, 55 is denoted by 58 in FIG. 8 and represents the baseline of the stereo camera 55a.

The following angles, which are relevant for the positional relationship of the slave camera 55 to the master camera 54, are recorded in the method for spatial image acquisition: the angle by _s , ie a tilting of a plane that is perpendicular to the xmzm plane and through which the baseline 58 runs, to the xmym plane about the tilting axis ym; bz _s : a tilting of the baseline 58 corresponding to the tilting by _s with respect to the xmzm plane about a tilting axis parallel to the slave coordinate axis zs; ax _s : a tilting of the slave coordinate axis zs, ie the image recording direction of the slave camera 55, to the xmzm plane about the slave coordinate axis xs; ay _s : a tilting of the slave coordinate axis xs relative to the master plane xmym about the slave coordinate axis ys, corresponding to the tilting ax _s , and az _s : a tilting of the slave coordinate axis ys to the master coordinate plane, corresponding to the tilting ax _s , ay _s ymzm around the slave coordinate axis zs.

In the spatial image acquisition with the aid of the two cameras 54, 55, taking into account on the one hand these angles by _s , bz _s , ax _s , ay _s , az _s , including the angle changes dax _m , day _m , daz _m , dax _s , day _s , daz _s proceed as follows:

First, an image of a three-dimensional scenery with scenery objects 59, 60, 61 (cf. FIG. 9) is captured simultaneously with the two cameras 54, 55 of the stereo camera. This image capture of the images 62, 63 takes place simultaneously for both cameras 54, 55 in a capture step 64 (cf. FIG. 10).

The image acquisition can be integrated over several acquisition cycles of the inertial measuring unit 56, 57, in particular over a period that corresponds to several time constants of the real-time position change measurement acquisitions.

The respective image 62, 63 of the cameras 54 and 55 is shown schematically in FIG.

The image of the scenery object 59 is shown in the image 62 of the master camera 54 at 59M, the image of the scenery object 60 at 60M.

The imaging of the scenery object 59 is shown in the image 63 of the slave camera 55 at 59s. The depiction of the scenery object 61 is shown in the image 63 of the slave camera 55 at öls. In addition, the images 59M, 60M of the master camera 54 are also found in the image 63 of the slave camera 55 at the corresponding x, y coordinates of the image frame. A y-deviation of the imaging positions 59M, 59S is referred to as the disparity perpendicular to the epipolar line of the respective camera or as the vertical disparity VD. Correspondingly, an x-deviation of the imaging positions 59M, 59S of the scenery object 59 is referred to as disparity along the epipolar line or as horizontal disparity HD. For this purpose, reference is made to the known terminology for epipolar geometry. The parameter "centre of camera entrance pupil" is referred to as "projection center" in this terminology.

The two images 60M, 61S show the same signature in images 62, 63, i.e. they are shown with the same image pattern in images 62, 63, but actually come from the two scenery objects 60 and 61, which are different within the spatial scenery.

The characteristic signatures of the scenery objects 59 to 61 in the images are now determined separately for each of the two cameras 54, 55 in a determination step 65 (cf. FIG. 10).

The signatures determined in step 65 are combined in a signature list and in an association step 66 the signatures of the captured images 62, 63 determined in step 65 are associated in pairs. Identical signatures are therefore associated with one another with regard to the captured scenery objects.

Depending on the detected spatial scenery, the result of the assignment step 66 can be a very large number of assigned signatures, for example several tens of thousands of assigned signatures and correspondingly several tens of thousands of characteristic position deviations that have been determined.

In a further determination step 67, characteristic positional deviations of the associated signature pairs from one another are now determined, ie for example the vertical and horizontal disparities VD, HD.

For example, the respectively ascertained vertical disparity VD is summed up squarely for all associated signature pairs. By varying the angle parameters by _s , bz _s , ax _s , ay _s , az _s described above in connection with FIG sum of squares. This sum of squares depends on the angles explained above in connection with FIG.

In a subsequent filtering step 68, the determined positional deviations are then filtered for the selection of associated signature pairs that have a higher probability of belonging to the same scenery object 59 to 61, using a filter algorithm. The simplest variant of such a filter algorithm is a selection by comparison with a specified tolerance value, with only those signature pairs passing through the filter for which the square sum is smaller than the specified tolerance value. This default tolerance value can, for example, be increased until, as a result of the filtering, the number of selected signature pairs is less than a default limit value.

In the case of an image algorithm variant, the respective positional deviation VD, HD itself can be used to select associated signature pairs. It is then checked whether the respective positional deviation is below a predetermined threshold.

In particular, several thresholds can be examined. For example, it can be checked for how many signature pairs the vertical disparity is below thresholds Si, S2, S3 and S4, where Si<S2<S3<S4. This then results in four lists of respectively accepted signature pair assignments (accepted correspondence) and rejected signature pair assignments (rejected correspondence). It is then examined how the number of the respectively bound accepted correspondences depends on the threshold value. The lowest threshold is used at which the correspondence count value remains approximately the same. The result is a heuristic method for filtering the signature pairs, so that there is a high probability that “wrong” signature pairs that do not belong to the same object are rejected.

As soon as the number of selected signature pairs is less than a predetermined limit value as a result of the filtering, for example less than a tenth of the signatures originally assigned in pairs or is less than five thousand signature pairs in absolute terms, for example, a triangulation calculation is carried out in step 69 for determination of depth data for the respective scenery objects 59 to 61. In addition to the number of selected signature pairs, a default tolerance value for the sum of squares of characteristic position deviations of the associated signature pairs, for example the vertical disparity VD, are used, in accordance with what has been explained above. A standard deviation of the characteristic positional deviation, for example the vertical disparity VD, can also be used as a termination criterion.

The triangulation can be carried out with the respectively accepted pairs of signatures, ie the accepted correspondences.

As a result of this triangulation calculation, a 3D data map of the captured scenery objects 59 to 61 within the captured image of the spatial scenery can be created and output as a result in a creation and output step 70 . An example of such a 3D data map is an assignment of all points of the respective scenery object to respective value triples Xi, yi, Zi, which reflect the position of this respective scenery point in Cartesian coordinates. A spatial reproduction of the respective scenery object is possible via appropriate access to this 3D data card.

If the filter step 68 shows that the number of selected signature pairs is still greater than the specified limit value, angle correction values are first determined in a determination step 71 between the various selected assigned signature pairs to check whether mapped raw objects, belonging to the various selected associated signature pairs can be arranged in the correct position relative to one another within the spatial scene. For this purpose, the angles described above in connection with FIG. 8 are used, these angles being corrected in real time due to the measurement monitoring via the measuring units 56, 57.

Based on a compensation calculation performed in determination step 71, the scenery objects 60, 61 can then be distinguished from one another, for example, despite their identical signatures 60M, 61s in the images 62, 63, so that a correspondingly assigned signature pair can be separated out as a mismatch, so that the number of the selected signature pairs is reduced accordingly.

After the angle correction has taken place, in a comparison step 72, the angle correction values determined for the signature pairs are compared with a predetermined correction value size performed. If the angle values of the signature pairs as a result of the comparison step 72 deviate more from each other than the specified correction value size, the filter algorithm used in the filter step 68 is adapted in an adaptation step 73 such that after filtering with the adapted filter algorithm, a The number of selected signature pairs is smaller than the number that resulted from the previous filter step 68 . This adaptation can take place in that signature pairs are discarded which differ in their disparities by more than a predetermined limit value. A standard of comparison, as of which point the signatures of a potential signature pair are judged to be the same and therefore assignable, can also be set more critically in the adjustment 73 .

This sequence of steps 73, 68, 71 and 72 is then carried out until the result is that the angle correction values of the remaining associated signature pairs deviate from one another at most by the predetermined correction value size. The triangulation calculation then takes place again in step 69, in which case the angle correction values of the selected signature pairs can be included, and the results obtained are created and output, in particular in the form of a 3D data card again.

A method for generating a redundant image of a measurement object is explained below with reference to FIG. To this end, several cameras are linked together, the centers of the entrance pupils of which define a camera arrangement plane. For this purpose, FIG. 11 shows two groups of three cameras each 74 to 76 on the one hand (group 74a) and 77, 78, 79 on the other hand (group 77a). The groups 74a on the one hand and 77a on the other hand each have an assigned data processing unit 80, 81 for processing and evaluating the image data recorded by the associated cameras. The two data processing units 80, 81 have a signal connection to one another via a signal line 82.

For example, the cameras 74 to 76 of the group 74a can be interconnected to capture a three-dimensional scene, so that a 3D capture of this three-dimensional scene is made possible, for example via an image capturing method that was explained above in connection with FIGS. 8 to 10. To create an additional redundancy in this spatial image acquisition, the image acquisition result can, for example, Camera 77 of the further group 77a can be used, which is made available to the data processing unit 80 of the group 74a via the data processing unit 81 of the group 77a and the signal line 82. Due to the spatial distance between the camera 77 and the cameras 74 to 76 of the group 74a, the viewing angle for the imaging of the spatial scenery is significantly different, which improves the redundancy of the spatial image acquisition.

A camera arrangement plane 83, which is defined by cameras 74 to 76 of group 74a and cameras 77 to 79 of group 77a, is indicated schematically in Figure 11 and is at an angle to the plane of the drawing in Figure 11.

Spatial image acquisition using the cameras of exactly one group 74a, 77a is also referred to as intra image acquisition. Spatial image acquisition involving the cameras of at least two groups is also referred to as inter-image acquisition.

A triangulation can, for example, take place independently with the stereo arrangements of the cameras 78, 79, the cameras 79, 77 and the cameras 77, 78. The triangulation points of these three arrangements must match.

A camera group like the groups 74a, 77a can be arranged in the form of a triangle, in particular in the form of an isosceles triangle. An arrangement of six cameras in the form of a hexagon is also possible.

Compared to the distance between the cameras in a group 74a, 77a, the cameras in the other group are at least a factor of 2 further away. A distance between cameras 76 and 77 is therefore at least twice the distance between cameras 75 and 76 or cameras 74 and 76. This distance factor can also be greater and can be greater than 3, for example, can be greater than 4 , can be greater than 5 and can also be greater than 10. A close camera range covered by the respective group 74a, 77a can be in the range between 80 cm and 2.5 m, for example. By adding at least one camera from the respective other group, a far range can then also be captured with the image capturing device beyond the close range limit. A method for spatial image acquisition is described below, which should be understood as a supplement to the above description.

For spatial image acquisition, a system of equations for pixel correspondences of known pixels (u,v) left with (u,v) _right , for example of the cameras 54, 55 with a known focal length f, is solved. This system of equations is:

This equation 1 is explained in more detail below with reference to FIG.

In Equation 1: ui, vi: image coordinates of a considered scenery feature, using the example of the scenery

Objects 59, 59i (=59M) in the left image;

Ur, v _r : image coordinates of the feature, using the example of the scenery object 59, 59 _r (=59s) in the right image; fi, fi: focal lengths of the left and right cameras 54, 55; i, A, _r : running variables along the ray from the center of the left and right cameras 54,

55 through the pixel of the left or right camera. At k=0 is the center of the left or right camera 54, 55, at =1 the point 59i, _59r of the scenery object 59 is on the respective image 62, 63 of the respective camera 54, 55. At X>1 are points on the further ray in front of the left or right camera 54, 55 after the respective image plane 62, 63 up to the point of intersection in the scenery object 59 and beyond; t _x , t _y , t _z : coordinates of the position of the center of the right camera 55 in the coordinate system of the left camera 54. The length of this vector t is the base length (bien) or baseline 58 (cf. FIG. 8). This length of the vector t can be measured and/or is known from the assembly situation of the two cameras 54, 55. After the length of t is known, two degrees of freedom remain for the three coordinates t _x , _ty and t _z ; Rxx, . . . , R _zz : parameters of the rotation matrix for describing the rotational position of the right camera 55 in the coordinate system of the left camera 54. This rotation has three rotational degrees of freedom.

The optical axes of the two cameras 54, 55 are again shown in broken lines in FIG. 12 and are comparable to FIG.

Equation 1 above can be written as three equations 1.1, 1.2 and 1.3 for the value triple ui (equation 1.1), vi (equation 1.2) and fi (equation 1.3). It is therefore a system of equations with three equations (Eq.1.1 to Eq.1.3) and two unknowns ( i, Xr). These equations can be transformed into an equation by eliminating the unknowns (Xi, X _r ).

Correspondence is a match of feature points when the same scenery object is captured by the different cameras 54, 55 (left 1, right r) of the image capturing device. If a feature point of a scenery object was captured by camera 54 at image coordinates ui, vi and the same feature point by camera 55 at image coordinates uj, vj, this is a (positive) correspondence. Each correspondence can be assigned a vertical disparity VD. Therefore, at least five correspondences per pair of cameras are necessary, the number can be somewhat higher, but it is advantageous to be significantly higher because of higher statistical stability.

The pixel correspondences can be stereo correspondences. For stereo correspondence, see the specialist article Daniel Scharstein, Richard Szeliski: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, in: International journal of computer vision, vol. 47, no. 1-3, 2002, p. 7 -42 and Alex Kendall et al.: End-to-end learning of geometry and context for deep stereo regression. CoRR, vol. abs/1703.04309, 2017. u, v are Cartesian coordinates of the respective pixel in GL 1, for example the respective Cartesian coordinates x, y or coordinates in the direction of the respective horizontal and vertical disparity HD, VD. The vectors (u,v,f) describe a point on the ray from the origin of a camera 54, 55 with run parameter .

Eq. 1 describes the change in position from the right camera (e.g. camera 55) to the left camera (e.g. camera 54) with three position parameters via the translation vector t and the rotation matrix R, ie with six degrees of freedom.

Because the length (baseline 58 of FIG. 8 = bien) of the displacement vector t cannot be estimated without scalar knowledge in the spatial scene, but is measured or otherwise previously known, there are 5 degrees of freedom, namely two normalized with bien to bien perpendicular Shifts (which can also be interpreted as rotations of the bien) and three rotations of the right camera to the left (ie 5 rotations).

An example of the 5 degrees of freedom are the aforementioned angles by _s , bz _s , ax _s , ay _s , az _s for the positional relationship of the slave camera 55 to the master camera 54.

The system of equations GL 1 consists of three equations with two unknowns, namely the two running parameters i, A, _r and five degrees of freedom of the translation matrix T and the rotation matrix R.

From these three equations, an equation can be formed in which there are no longer any unknown variables. By means of suitable transformations, exactly one equation is created without the X parameters, i.e. only dependent on the 5 degrees of freedom.

With at least five equations, the degrees of freedom can be calculated by an estimator, for example. Such an estimator solves the system of equations, which is usually overdetermined, by minimizing a residual error. Such an estimator is known in the literature as a "James-Stein estimator" (cf. the article "Stein's estimation rule and its competitors - an empirical bayes approach" by B. Efron and C. Morris, Journal of the American Statistical Association 68 (341, pp. 117-130, 1973).

The correspondences, ie the pixel correspondences according to Equation 1 above, must be linearly independent of one another. With a large number of correspondences, false positives can be filtered out by algorithms that are based on the weighting of the features via their size of offset to the respective epipolar line of the respective camera 54, 55.

The false-positive correspondences are normally distributed to a first approximation. The true positive correspondences have characteristic accumulations, i.e. they are not normally distributed. The filtering leads to the fact that the false-positive correspondences are filtered out, so that the positive correspondences remain within the framework of a convergent algorithm. The positives allow the estimator to converge to the actual value. This estimator also works with squinting cameras (directional deviation between zm and zs, cf. in particular the above description of FIG. 8) and converges more quickly with fisheye cameras.

If the image capturing device has two cameras 54, 55, five degrees of freedom must be determined. With three cameras that can be configured into two pairs of cameras, there are ten degrees of freedom to be determined.

In general, n cameras are available in an image capturing device. With a pair of cameras, 5 degrees of freedom can be estimated. Each camera beyond a first camera pair with five degrees of freedom contributes six additional degrees of freedom. With n cameras, there are then 6n - 1 estimable degrees of freedom.

In both cameras 54, 55 of the image capturing device, IMUs (inertial angular rate sensors) are installed. From the rate of rotation in a period (mono-period frame), the change in rotational position can be estimated and thus the five changes in rotational position of the cameras in relation to one another. The residual error is compensated by applying the method. When using IMUs, an IMU translation can also be used. When using IMUs, acceleration values can be recorded, which helps to improve prediction accuracy.

The prediction from the IMU data can also be integrated over several cycles for stabilization. The method is also calculated over these periods. Instead of two, the number of cameras can also be increased to three or more cameras for joint calculation. The different perspectives improve the estimation results, even when the cameras are lined up.

The estimation is further improved when the cameras span an area.

The estimate is further improved when the cameras squint more, thus opening up a larger field of view together. In this sense, a fisheye camera 54, 55 can be better than a normal lens.

Each of the scenery objects may be a larger corresponding blob, for example a component of a vehicle. The correspondences of larger blobs can be determined more precisely than the correspondences of smaller features.

With three cameras, the same spatial feature can be found in three images. If it is found three times, it is checked for plausibility. A corresponding plausibility check can be included in the position deviation filtering of the image acquisition method. If a particular feature is captured by two cameras, its position can be predicted when captured by the third camera. If it is actually imaged there when it is imaged by the third camera, the spatial feature in question is then checked for plausibility. If not, a mismatch is assumed.

As the number of cameras increases, so does the plausibility check, and thus the aforementioned rejection of false-positive correspondence.

Each estimation weights the correspondences via the weights of the offsets to the epipolar line of the respective camera 54, 55. The respective epipolar line is calculated before each estimation step. With multiple estimates, the epipolar line changes and the weights are changed as well. For example, the weight “1” can be assumed up to a threshold value. The weight can then be reduced linearly to zero up to a double threshold value. The change in the weights in the course of the multiple estimation in turn provides information about the possibility of a false positive, which in turn leads to new weights. The size of the respective weights can be derived by comparing the correspondence numbers when using the four differently sized threshold values Si to S4.

The weightings are described using a cost function depending on, for example, the magnitude of the threshold values.

The correspondence should be distributed as evenly as possible in the image. To do this, the image is divided into zones (e.g. nine evenly distributed zones). The number of features in the zones should ideally be even. A ZDF (Zone Distribution Function) describes the distribution of features in the image. If the distribution is unfavorable, the estimate is rejected.

An estimator is more robust with regard to the result obtained as long as there are correspondences distributed evenly over the respective image. When dividing into zones, it can be ensured that there is a minimum number of correspondences in all zones. Such a minimum number of correspondences must be present in the zones at least in the case of a last estimate carried out. Correspondences from zones in which there are many correspondences compared to, for example, an average value of the correspondences in the divided zones can then be weighted less by smaller weighting factors.

As an alternative to the reweighting described above, unequally distributed correspondences can also be better evenly distributed by deleting in overcrowded zones.

Unequally distributed correspondences can be discriminated by weighting in overcrowded zones.

The ZDF distribution can also be adjusted using the weights from close to the epipolar lines. This follows from a product, on the one hand, of a weight that depends on the proximity of the respective epipolar line and, on the other hand, of a weight that depends on the respective zone. The correspondences are weighted by their distances from the current epipolar line. In principle, a large number of correspondences, i.e. also larger distances from the respective epipolar line, can be considered. However, the number of correspondences with small distances to the respective epipolar line should be comparatively large. For this purpose, the curve of the number of correspondences over increasing maximum epipolar distances is considered. This typically s-shaped curve (first slightly increasing, then strongly increasing, then slightly increasing again), which indicates the number of correspondences as a function of the epipolar distance, is analyzed and the location of the greatest gradient is sought (inflection point). In the case of small changes of small distances, rejected correspondences statistically increase almost linearly.

Beyond any actual error, this increase remains nominally constant.

This place is a compromise between as much correspondence as possible and not too much correspondence by far.

Correspondence accumulates near an actual value, so the significance increases. The correspondences are generally statistically distributed (background noise) far from an actual value.

The results of the estimation can be averaged over a period of time. A number of images can therefore be recorded in a sequence and subjected to a corresponding estimation. A temporal filtering results.

The averaged estimated values are only averaged up to a maximum past. So there is a sliding average calculation in real time. Disturbances that happened a long time ago are forgotten again and no longer hinder the latest estimates.

The relative positions of the cameras used in the image capturing device can be estimated using the image capturing method. Estimation results of different image capture scenarios can be compared. If they deviate too much from each other, it is rejected and deactivated (fail-safe) or by a third one of the two results confirmed and the mission continued (fail-operational). In this case, the measurements with these recalibrated cameras are also confirmed with two matches within three measurements.

An extrinsic calibration with the image acquisition method is based on an intrinsic calibration (distortion or distortion, focal length, etc.) of the respective camera.

The intrinsic calibration model error can be integrated into the weights of the distances to the epipolar lines. A prediction of coordinates of the scenery objects 59 to 61 in the image of the second camera 55, depending on the epipolar line distances, ie the epipolar curve, can be made directly taking into account a distortion description. The weights of such a description can be increased according to an expected residual error of a distortion correction.

In order to capture the all-round view with as few cameras as possible, the cameras can be equipped with fish-eye lenses with a focal length of less than 20 mm, in particular less than 10 mm. At greater distances, the distance resolution will be lower than when using normal or telephoto lenses because of the small focal length. Estimation with standard or telephoto lenses is more unstable than estimation with fisheye lenses. If telecameras with a focal length of at least 80 mm, which can in particular be at least 150 mm or at least 200 mm, are mounted or assigned to fisheye cameras in a dimensionally stable manner in order to optimize the cameras 54, 55, 74-76, 77-79 described above the estimation of the fisheye camera can be transformed into the telecamera. If two fisheye cameras are estimated, new estimated values for the telecameras associated with them automatically result. However, the higher resolution of the telecameras can provide contributions to the estimation of the system from at least two fisheye cameras and the associated telecameras. The image capturing device then no longer only has fish-eye cameras, but also additional, associated telecameras. For example, two fisheye cameras can form a stereo pair (see the above example of cameras 54, 55). A telecamera can be permanently or rigidly assigned to each fisheye camera. From correspondences of the fisheye cameras on the one hand and correspondences of the telecameras on the other hand, an enlarged system of equations can then be built up corresponding to the above equation 1, which is then solved. The estimates can be averaged over multiple frames. The averaging can be done with IMU value normalized estimates. Larger movements are less disruptive.

The averaging of the IMU value normalized estimates can be smoothed and thus stabilized by rejecting IMU-detected jerk movements.

A motion model can support an estimation. A prediction of the angles with their probability can be balanced with the current measurement and the measurement errors and dynamic measurements can be supported. The motion model is based on the assumed stiffness of a support structure or the support frame of the cameras. The movement must be slower than the exposure time, i.e. the exposure time, but can be faster than the cycle time, i.e. the image repetition time.

The typical movements of a structure can be recorded and AI models (models of artificial intelligence) can be derived from this training set, which allow a better prediction. For example, engine vibrations that are introduced into the support structure for the cameras can be compensated for.

The Kl model can detect jerk-like movements and help reject estimates.

The estimation of the relative position of the cameras can be asynchronous with the measurement using triangulation. The measurement is based on classical stereo algorithms of rectified images.

The image acquisition method uses native features to form correspondences.

A classic stereo matching method compares gray value differences and searches for a location along the epipolar curve, typically a horizontal stretch, with the smallest gray value difference that must fall below a threshold. A signature is present when the gray values are the same. "Native" features are converted into a signature from a transformation of gray values of a feature environment. There is therefore a comparison of gray values within a feature environment in a camera image. Only features are used whose signatures occur comparatively rarely in the image, for example fewer than ten, fewer than eight or fewer than seven times. Such rare features are referred to as native features.

The correspondences can also be used for distance measurement with triangulation. The features are extracted and used for triangulation (native stereo). The same features are used as part of the image acquisition process and for estimation. Once the estimation is complete, the new results can be added to the measurements asynchronously. It is measured multiple times while only one estimate is made. This takes place when the relative position of the cameras changes slowly.

The image acquisition method can quickly discover strong deviations in coarser (binned) images, ie in images in which a plurality of pixels are combined. If these are recognized, the exact changes are determined in the finer images and made available for the measurements. Estimates with smaller deviations are not provided in order to avoid the computationally expensive calculation of the warp parameters for rectified images. Warp parameters are parameters for describing a desired equalization based on a distortion description, i.e. based on an image error description of the cameras 54, 55.

The rejected deviations must, of course, be smaller than the deviations that can be tolerated by DenseStereo, ie by the classic stereo matching method described above. Line errors of one pixel can usually be tolerated. This depends on a size of a compared environment. However, even if they are small, such line errors lead to errors in the distance measurement.

In the case of highly deformable (soft) mechanical constructions between the cameras under consideration, for example the cameras 54, 55, an estimate must be made before each measurement. The warp parameters are calculated from the estimated data, then the rectified images are calculated with the aim of line fidelity and the distance is determined via triangulation from the dense stereo correspondence, i.e. a location of a minimum distance. The calculations of the warp parameters for DenseStereo are computationally intensive. The native stereo, ie an assignment of native features without line fidelity and without comparison of gray values, where only a comparison of signatures takes place, can calculate the triangulation directly from the correspondences of the image acquisition process.

The image acquisition method also provides a measure of confidence for the correspondences via the weighting of the distances to the epipolar lines. The degree of confidence is, on the one hand, a probability of an association using more than two cameras and, on the other hand, an assessment of a magnitude of noise to be expected, ie ultimately a signal/noise ratio.

The synchronous sequence of estimation and triangulation with native features is therefore efficient and provides a measure of the trustworthiness of measurement points.

Native triangulation can measure distance from a correspondence. In this case, a center point of a distance of skewed rays is considered, which are generated from two corresponding image points of the two stereo cameras 54, 55. Such a distance must fall below a predetermined threshold.

However, if three or more cameras are used, there can be three or more correspondences per feature. The distance measurement becomes more accurate and reliable. However, the number of measuring points usually decreases. An intersection of successful triangulations between a camera pair 1/2 and a camera pair 2/3 is smaller than a union of the two successful triangulations.

If several identical signatures are found in pairs of cameras, multiple correspondences arise. From these correspondences of the same signatures, those with the lowest cost functions for the epipolar lines are selected. So the signature that is closest to an epipolar curve is selected and not necessarily the signature that is closest in space. Several can be selected, but each selected signature may only be used once. Once a correspondence has been selected, the two items in both screens should not be used for further correspondence.

Claims

- 37 -Claims

1. Method for spatial image acquisition using a stereo camera (55a) having two cameras (54, 55) with the following steps:

Capturing (64) an image (62, 63) of a spatial scene simultaneously with the two cameras (54, 55) of the stereo camera (55a),

Determination (65) of characteristic signatures of scenery objects (59 to 61) within the respectively captured image (62, 63), pairwise assignment (66) of the signatures of the captured images (62, 63) to one another, determination (67) of characteristic Positional deviations HD, VD of the associated signature pairs from each other,

Filtering (68) the positional deviations to select associated signature pairs, triangulation calculation (69) to determine depth data for the respective scenery objects (59 to 61) on the basis of the selected signature pairs,

creating (70) a 3D data map of the captured scenery objects (59 to 61) within the captured image of the spatial scenery.

2. The method according to claim 1, characterized by filtering (68) the position deviations to select the associated signature pairs that belong to the same scenery object of the spatial scenery with a filter algorithm.

3. The method according to claim 1 or 2, characterized in that the triangulation calculation (69) takes place as soon as the number of selected signature pairs is less than a predetermined limit value.

4. The method according to any one of claims 1 to 3, characterized in that as long as the number of selected signature pairs is greater than a predetermined limit value, the following steps are carried out:

Determining (71) angle correction values between the various selected associated signature pairs to check whether mapped raw objects (59 to 61) belonging to the various selected associated signature pairs can be arranged in the correct position within the spatial scenery, - 38 -

Comparing (72) the respective angle correction values determined for the signature pairs with a predetermined correction value size,

5. The method according to claim 4, characterized in that as long as the number of selected signature pairs is greater than a predetermined limit value, the following steps are carried out: if the angle correction values of the signature pairs differ from each other more than the predetermined correction value size:

- Adjusting (73) the filter algorithm such that after filtering (68) with the adjusted filter algorithm, a number of selected signature pairs results that is smaller than the number that resulted from the previous filter step, and repeating the comparison (72), as long as the angle correction values of the signature pairs differ from each other at most by the specified correction value size:

- triangulation calculation (69) to determine depth data for the respective scenery objects (59 to 61),

6. The method according to claim 4 or 5, characterized in that the angle correction values of the selected signature pairs are included in the triangulation calculation (69).

7. The method according to any one of claims 1 to 6, characterized in that when capturing (64) the images (62, 63) at the same time data from inertial measuring units (56, 57) are recorded, with which the cameras (54, 55) fixed are connected.

8. Method for generating a redundant image of a measurement object with the following steps:

Linking at least three cameras (74 to 76; 77 to 79) whose entrance pupil centers define a camera arrangement plane, Bringing the measurement object into the field of view of the cameras,

Carrying out triangulation measurements of at least one selected measurement point of the measurement object with at least three different camera pairs (74, 75; 74, 76; 75, 76; 77, 78; 77, 79; 78, 79) of the cameras (74 to 79) ,

Comparing the results of the triangulation measurements. Device for carrying out the method according to one of Claims 1 to 8. Device according to Claim 9, characterized by three cameras (74 to 76; 77 to 79) in the camera arrangement plane. Device according to Claim 9 or 10, characterized by six cameras (74 to 79). Device according to one of Claims 9 to 11, characterized by at least three cameras (74 to 76) arranged adjacent to each other and by a camera (77) arranged at a distance of at least a factor of 2 when comparing the distances between these cameras (74 to 76) arranged next to one another. Apparatus according to claim 8, characterized in that the remotely located camera (77) is located in the camera array plane (83) of the three adjacent cameras (74-76). Device according to one of Claims 9 to 13, characterized in that at least one of the cameras (54, 55; 74-76; 77-79) is permanently connected to an inertial measuring unit (56, 57).