WO2024251542A1 - Device for capturing a spatial image of a surrounding area which is moving relative to the device, and method for operating same - Google Patents

Abstract

The invention relates to a device (1) which is used to capture a spatial image of a surrounding area. The image capturing device (1) has at least one stereo camera (5, 14, 15, 20) for determining the object distance of the device (1) to the surrounding area using triangulation. At least one flash light source (7, 13) is used to illuminate the surrounding area. Each of the cameras of the stereo camera has a sensor array. A controller (10) is connected to the cameras so as to transmit signals and is used to specify image capturing parameters in a temporally synchronized manner, namely a flash illumination start, a flash illumination duration of the flash light source, and a detection start and a detection duration of the camera. The result is an image capturing device which is well adapted for the practice of capturing images in order to ensure a supporting or autonomous drive.

Description

Device for spatial image capture of an environment moving relative to the device and operating method therefor

The present patent application claims the priority of the German patent application DE 10 2023 205 343.8, the contents of which are incorporated herein by reference.

The invention relates to a device for spatial image capture of an environment moving relative to the device. The invention further relates to a method for operating such a device.

Such devices are known from WO 2022/069 424 A1, WO 2022/069 425 A2, WO 2022/179 998 A1 and DE 10 2011 080 702 B3.

WO 2014/009 945 A1 discloses a stereo imaging system and an associated imaging method. US 2014/0 282 224 A1 discloses a detection of a scroll gesture. WO 2017/189 185 A1 discloses 3D imaging systems.

It is an object of the present invention to provide a device for spatial image acquisition which is well adapted to the practice, in particular of image acquisition for ensuring assisted or autonomous driving.

This object is achieved according to the invention by a device having the features specified in claim 1.

According to the invention, it was recognized that the functions of at least one stereo camera on the one hand and of image capture using an illumination time-of-flight method on the other hand advantageously complement each other. Both the stereo camera and the components enabling a time-of-flight method, namely the flash light source and sensor array, via which the time-of-flight method image capture parameters can be specified, allow the determination of a distance from objects within the environment. The device can be used for spatial image capture of an environment that is moving relative to the device. The at least one stereo camera can be used for triangular object position determination within the captured environment in the forward direction of a relative movement of the device to the environment. This can be used to improve the reliability of the environment capture, particularly in connection with driver-assisted or autonomous driving. The device can be part of a vehicle, for example a car or truck.

The control device can in particular specify a delay period between the start of the flash illumination of the at least one flash light source and the start of detection of the respective sensor array of the cameras of the at least one stereo camera. The distance between the cameras of the stereo camera results in a length of a baseline, i.e. a distance between the centers of the entrance pupils of the cameras of the at least one stereo camera. The device can be used to combine the principles of a TOF (Time of Flight) camera with a triangulatory distance-measuring stereo camera. A corresponding TOF image acquisition is described, for example, in US 2019/056 498 AL. The flash light source can be designed in the same way as is known in principle from TOF cameras or LIDAR systems. The same applies to the sensor arrays of the cameras. A sensor array used in TOF image acquisition can be constructed as a Single Photon Avalanche Diode (SPAD) array. Such a SP AD array and its application in connection with a TOF measurement is known from a technical article by T. Swedish et al.: "Beyond the Line of Sight? What's New in Optical Perception", AUVSI XPONENTIAL 2022-SWEDISH (conference handout), pp. 1-8, https://www.xponential.org/xponential2022/Custom/Handout/Speaker51584_Session4220_l .pdf and from US 2022/0174255 AL

For triangulatory object distance determination, the two cameras of the stereo camera are synchronized with each other via a synchronization unit. This synchronization makes it possible to achieve an exactly simultaneous recording of the respective object via the cameras of the stereo camera. This distinguishes a stereo camera for triangulatory object distance determination from a stereoscopic camera, which only creates a spatial impression of an image capture and which is described, for example, in the US 2015/296 200 Al. The use of an SP AD array can lead to a reduction in motion blur and can be used to improve the contrast of an image acquisition.

The synchronization and in particular the simultaneous object detection in the triangulatory object distance determination makes it possible to precisely determine the distance of even externally moving objects, since the influence of an additional relative speed due to the external movement of the object does not then affect the distance determination.

A focal length f of the respective camera of a stereo camera, the baseline b of the stereo camera and a disparity D, i.e. a difference between image coordinates of imaging positions of the same point on the sensor arrays of the cameras of the stereo camera, are coordinated in such a way that, taking into account the relationship d = fb/D, the following applies to the object distance value d to be recorded:

3 m < d < 300 m

In addition, the device can be designed such that a corresponding object distance range is detected via the TOF image acquisition functionality of the device.

The flash light source can be a controlled flash lamp or a time-controlled LED or laser light source. Flash light sources can be used that are known for use in LID AR devices, for example.

A flash illumination duration of the flash light source can be in the range between 10 ns and 200 ns, for example in the range of 100 ns.

A detection time of the sensor arrays of at least one stereo camera is regularly set to be twice as long as the flash illumination time of the flash light source. The Detection time can be in the range between 20 ns and 400 ns, for example in the range of 200 ns.

Both the flash illumination duration and the detection duration can be set via the control device.

The synchronization unit can be part of the control device. Part of the control device can be a storage unit, in particular in the form of a flash memory. The synchronization unit can be part of a synchronization device for synchronizing the flash light source and the sensor arrays of the cameras of the at least one stereo camera. The synchronization device can in turn be part of the control device of the image capture device.

Synchronization that can be provided by the synchronization device may be better than 5 ns, may be better than 2 ns, may be better than 1 ns, and may be even better.

The cameras of the stereo camera can be spaced apart laterally to the forward direction. Depending on the detection angle of the cameras and depending on the application and main detection direction of the stereo camera, other camera arrangements of the stereo camera are also possible.

The cameras of at least one stereo camera can have a telephoto lens for detecting objects at a distance between 50 m and 300 m. Depending on the application, the cameras of the stereo camera can also be equipped with lenses of other focal lengths. These camera lenses are designed in such a way that an entire specified distance range is covered, in particular between 3 m and 300 m. In principle, even smaller distances can be covered.

The distance ranges of the triangulatory object detection on the one hand and the TOF distance detection on the other hand are coordinated and can, for example, be the same. Partial overlap of these distance ranges is also possible, so that in a In some distance ranges both distance determination principles “triangulatory” and “TOF” are available and in other distance ranges one of these principles is available.

Using the cameras of the at least one stereo camera, a correspondence analysis can be carried out to identify objects that are different from a background and are present in a field of view of interest. Methods of a corresponding correspondence analysis are known from WO 2022/069 424 A1 and WO 2022/069 425 A2. In preparation for operation of the device, a stereo calibration of the cameras of the at least one stereo camera can be carried out, which is basically known from WO 2022/069 424 A1 and DE 10 2011 080 702 B3.

The at least one stereo camera can have more than two cameras spaced apart from each other, which in particular enables switching between different baselines, which is also described, for example, in WO 2022/069 424 A1.

The device can have at least two stereo cameras with a relatively large baseline and at least one further stereo camera with a comparatively significantly smaller baseline, wherein the smaller baseline can be at most 50%, at most 25% or even at most 10% of the larger baseline. The device can also have several stereo cameras with the small baseline.

Further object information, in particular object position information, can be obtained by evaluating an object shadow that is cast by an object within the captured environment as a shadow of the flash lighting and is imaged differently by the two cameras of the at least one stereo camera that are spaced apart from each other. In particular, the height of the object can be determined because an image of an object shadow depends on the height and lateral position of the object.

The device can be used to capture spatial images of the environment, whether moving or stationary relative to the device, which enables a driver-assisting function or even the function of autonomous driving, for example when the device is used as part of a vehicle. The flash light source of the device or another light source of the device for ambient lighting can have a measuring light beam that is textured or structured across its beam cross-section. Such texturing/structuring of the measuring light beam can be achieved by constructing it as a plurality of individual beams that capture the environment in the form of a grid. Such a light source for generating a textured/structured measuring light beam can be designed as a laser. Examples of a detection device that works using such a textured/structured measuring light beam can be found in DE 10 2018 213 976 A1 and the references cited therein, in particular in WO 2012/095 258 A1, US 9,451,237 and WO 2013/020 872 A1. With the help of such textured/structured lighting, optically unstructured, in particular diffuse scattering media within the enclosed environment can be measured with regard to their distance. Such diffuse scattering media can be, for example, fog, smoke or dust.

Form redundancy and/or function redundancy can be achieved using the spatial image acquisition device. Form redundancy is achieved when different technologies are used to determine one and the same measurement variable. Function redundancy exists when several devices of the same technology are used to measure one and the same measurement variable. The terms "form redundancy" and "function redundancy" are mainly used in the aviation sector. There, a combination of form and function redundancy is required to avoid multiple effects of an error. Such redundancies can be required by law, depending on the application of the device.

At least one further flash light source and/or at least one further stereo camera according to claim 2 enable additional functional redundancy of the device. If at least one further flash light source is provided, it is possible to switch to the further flash light source after the failure of the initially used flash light source, so that a longer break in operation of the device is avoided. If at least one further stereo camera is used, for example with parallel triangulation measurements using two stereo cameras, further redundancy in the object distance determination can be achieved. The cameras of the additional stereo camera can each have a sensor array with a controlled specification of a detection start and a detection duration, so that the additional stereo camera can basically replace the function of the stereo camera originally used. A redundant TOF determination of object distances is also possible using two corresponding stereo cameras with a controlled specification of the detection start and the detection duration, which provides additional security when determining the object distance and helps to identify synchronization problems, for example.

When using multiple stereo cameras, it is also possible to assign cameras from different stereo cameras to a new "virtual" stereo camera with a correspondingly modified baseline, which allows for further redundancy or security when determining the distance to objects. For example, the camera pair that proves to be particularly suitable for triangulation determination in relation to a specific surrounding object can be selected as the baseline.

At least one fisheye stereo camera according to claim 3 enables the use of additional triangulation baselines, which can be used to detect additional objects in the environment and/or to check object distance results of the other stereo cameras and/or to calibrate the other stereo cameras, in particular stereo cameras designed with telephoto optics cameras. The main detection direction can be the forward direction of a relative movement of the device to the environment. The use of fisheye cameras is known from the above-mentioned references and from WO 2022/179 998 A1. A fisheye camera that is attached as rigidly as possible to a telephoto optics camera allows a more precise calibration of the stereo cameras to one another due to the larger opening angle of the fisheye camera. An inertial measurement unit (IMU) in the cameras of the respective stereo camera allows further optimization of the relative calibration. In particular, the IMU can be used to determine a rotation rate, i.e. a relative tilt of the respective camera around a defined tilt axis.

The device can contain several fish-eye stereo cameras, which allows additional redundancy. The cameras of the at least one fish-eye stereo camera can also each have sensor arrays with controlled specification of a detection start and a detection duration, so that a TOF determination of object distances is also possible. In principle, the at least one fisheye stereo camera can also be designed without a TOF function. If the device has several fisheye stereo cameras, it is also possible for at least one of the fisheye stereo cameras to be designed with a TOF function and at least one other of the fisheye stereo cameras to be designed without a TOF function.

In principle, the device according to the above claims can be used in such a way that a precise distance range is detected by means of the TOF components, for example in the distance range between 30 m and 300 m, in particular between 100 m and 200 m, from the device.

The object mentioned at the outset is also achieved by an operating method having the features specified in claim 5.

The operating method according to claim 5 enables real-time operation of the device, so that, for example, autonomous driving is possible when using the device as part of a vehicle.

In the operating method according to claim 5, a covered total distance range is divided into a plurality of distance ranges that are adjacent to one another or partially overlap one another. For example, three to 100 such distance ranges can be specified using corresponding image acquisition parameters, for example 50 distance ranges that lie within the total distance range. When the device is operating, the total distance range can be recorded over the specified individual distance ranges several times a second, for example at a rate of 10 Hertz. During operation, the most distant distance range within the total distance range can be started. This ensures that when overwriting images recorded in a previous recording step with a last recorded image, close objects overwrite more distant objects in non-filtered image areas of the last recorded image, so that the result of the operating method, namely the last updated, output image, in any case, the objects closest to the device are represented, which can then be responded to, for example, by intervening in a movement of the device. Buffering of images captured in previous capture steps can then be omitted, which increases the speed of the process.

The last updated output image is a 2D image with unfiltered objects in all distance ranges recorded within the total distance range. Information from distance ranges recorded earlier within the total distance range that was overwritten during the operating procedure can also be retained for independent spatial (3D) evaluation.

For the non-filtered out objects that are assigned to the respective detection step, the method results in a distance range in which the object is present relative to the device. This resulting object distance information, which is determined using the TOF image capture, enables a form redundancy of the operating method in conjunction with a triangular distance capture made possible by the at least one stereo camera.

The distance information can be output as the smallest distance of the detected object or as a distance per object pixel of the respective sensor array.

The overall result is an image capture that is both safe and predictive, and is well adapted to the requirements of driver assistance or autonomous driving in particular.

When digitally filtering to filter out structureless image areas, a stereo correspondence determination can be used, which is described, for example, in WO 2022/069 424 Al.

A temporal overlap according to claim 6 enables a rapid execution of the operating method. The temporal overlap can be such that at least two distance ranges with simultaneous detection start and the same detection duration with the sensor arrays, whereby these detection areas are specified by different specifications of the respective flash illumination start and/or the respective flash illumination duration with regard to their image acquisition parameters. It is also possible to design the repetition sequence of the operating procedure in such a way that a detection actually takes place at exactly one time, i.e. at a specified detection start and with a specified detection duration, for all specified distance ranges. This simultaneous detection of at least two detection areas can reduce the image processing effort and speed up the entire process.

Different distance extensions of the distance ranges according to claim 7 make it possible in particular to specify a distance sensitivity depending on the distance, which also helps to shorten the duration of the operating method. With increasing distance, for example, the distance extension of the respective distance range can be increased, in particular increased progressively.

An additional triangulation determination according to claim 8 enables a comparison of object distance values which were determined on the one hand via the distance-range-resolved detection using the TOF image acquisition parameters and on the other hand by means of triangulation.

A ROI specification according to claim 9 enables a reduction in the image data to be processed, which can also shorten the method. The specified ROI can be a road area in the forward direction.

An operating method according to claim 10 can be used as an alternative or in addition to the operating method explained above. Such an operating method enables redundant distance determination of a respective object distance. In particular, this can provide form redundancy, i.e. redundancy provided by using different technologies for distance detection.

The use of the detected TOF distance as an input variable for the triangulatory detection according to claim 11 can enable a correspondence analysis in the triangulatory detection facilitate. In particular, the correspondence analysis can be used to limit the disparity range to be covered. Conversely, the triangulatory distance can also be used as an input variable for a TOF detection. The use of a detected object distance can be used in the operation of the device for tracking detected objects. Exactly one such object can be tracked or a plurality of such objects can be tracked simultaneously. Object tracking steps can alternate with detection steps for an entire specified distance range. For object tracking, image acquisition parameters of the TOF distance detection and/or the triangulatory distance detection can be adapted to an object distance to be expected during tracking.

A flash intensity/distance range adjustment according to claim 12 makes it possible in particular to draw a conclusion about diffuse scattering media present in the flashlight and/or camera detection path. Safe device operation is then possible even in foggy, dusty or smoky environments.

A quadratic dependence of the flash intensity on the distance to be detected according to claim 13 enables, in particular when the illumination intensity increases quadratically with increasing distance range, a tuning such that with identical ambient backscattering and negligible scattering along a flash path and along a detection path between source and object, a control of a constant intensity illumination of the at least one sensor array is possible. This can be used to determine diffuse scattering media in the detected distance range.

A regulation of a pixel sensitivity according to claim 14 can in particular not be carried out linearly to the incident light intensity. In particular, a gamma correction (y) can be carried out. The sensitivity of the at least one sensor array can then be specifically adapted to the lighting and/or detection conditions present in the application.

An adapted distance range Z movement mode selection according to claim 15 enables efficient utilization in particular of a computing power of the device and can also increase operational reliability. When evaluating with triangulatory stereoscopy in binocular vision, two images are compared so that possible changes in brightness due to changes in density can be neglected in a first approximation. When comparing the images with trained data (neural networks, AI) in monocular vision, a homogeneous brightness distribution over the distance increases the detection performance, whereas changing density distributions make reliable object recognition more difficult.

Embodiments of the invention are explained in more detail below with reference to the drawing, in which:

Fig. 1 schematically shows a plan view of a device for spatial image capture of an environment moving relative to the device, wherein an environmental object in the forward direction of a relative movement of the device to the environment is illustrated by way of example and in perspective;

Fig. 2 is a timing diagram illustrating image acquisition parameters during operation of the device of Fig. 1;

Fig. 3 shows, in a representation similar to Fig. 1, a further embodiment of a device for spatial image capture of an environment moving relative to the device, wherein environmental objects in the forward direction of a relative movement of the device to the environment are illustrated by way of example and in perspective;

Fig. 4 shows, in a representation similar to Fig. 2, a time sequence of an operating method of the device according to Fig. 3; and

Fig. 5 shows a side view of the main components of an embodiment of the device for spatial imaging of an environment within three exemplary distance parameters, wherein a cloud-shaped diffuse scattering medium is present within the captured environment. Fig. 1 shows a plan view of a device 1 for spatial image capture of an environment moving relative to the device 1, which is illustrated by an environmental object 2 in the form of a cylindrical body.

The device 1 can be part of a vehicle 3, for example a car or a truck. A forward direction of the relative movement of the device 1 to the environment is illustrated in Fig. 1 by an arrow 4.

The device 1 has several stereo cameras for triangulatory object distance measurement of objects within the detected environment in the forward direction 4, i.e. in particular for determining the distance of the object 2.

A first stereo camera 5 of these stereo cameras has two cameras 5i, ₅₂ spaced apart from one another. A distance A between these cameras 5i, 52 of the stereo camera 5 results in a length of a baseline of the stereo camera 5, i.e. a length of a distance from the centers of entrance pupils of the cameras 5i, 52 of the stereo camera. The cameras 5i, 52 of the stereo camera 5 each have a sensor array 6i, 62 with controlled specification of a start of detection of the respective camera 5i and 52 and a detection duration of the respective camera 5i, 52. The sensor arrays 61, 62 are designed in the manner of corresponding sensors of time of flight (TOF) cameras and can realize a detection duration in the range between 1 ns and 500 ns. A typical detection time is in the range between 50 ns and 400 ns, for example between 150 ns and 250 ns.

The sensor arrays 61, 62 can each have 100 x 100 pixels or 200 x 200 pixels. In principle, a higher number of pixels is also possible, in particular a number of 640 x 480 or 1440 x 1080 pixels.

The sensor arrays 61, 62 can be designed as CMOS arrays.

The 5i, 52 cameras have a telephoto lens for detecting objects at distances between 50 m and 300 m. The device 1 also includes a flash light source 7, which can also be designed like the light source of a TOF camera and enables a controlled specification of a flash illumination start and a flash illumination duration of successive illumination flashes. A flash illumination duration can be in the range between 5 ns and 150 ns, for example in the range of 100 ns.

Fig. 2 illustrates a temporal sequence during operation of the device 1 with the stereo camera 5 and the flash light source 7. At a time t = 0, an illumination flash of the flash light source 7 begins. t = 0 thus represents the illumination flash start of an illumination flash 8 shown schematically as a signal box in Fig. 2. A flash illumination duration of the illumination flash 8 is indicated in Fig. 2 with Atfiash.

Also at time t = 0, a delay time Ati _ag starts, which is also called delay time. This delay time Ati _ag is in the range between 300 ns and 2 ms and can, for example, be in the range between 500 ns and 1 ms, for example in the range between 700 ns and 900 ns, for example 800 ns.

After the delay period Ati _ag has expired, a detection 9 by the respective sensor array 6i or 62 of the stereo camera 5 begins at time t = tß. tE thus represents the start of detection. The respective sensor array 61, 62 then detects incident light from the flash light source 7 during a detection period At _exp .

The detection 9 by the respective sensor array 61, 62 of the stereo camera 5 is again illustrated in Fig. 2 by a temporal box.

The image acquisition parameters explained above, namely the start of flash illumination (t = 0), the duration of flash illumination Atfiash, the start of detection (t = tß) and the duration of detection At _exp are specified in a time-synchronized manner by a control device 10 of the device 1. A synchronization unit of a synchronization device of the control device 10 can be used for this purpose. The control device 10 has an image processor for real-time data processing. This can be a processor with a field programmable gate array (FPGA). The image processor can be trimmed for ultra-fast image grabbing of the individual images captured by the sensor arrays 6i, triggered by the control device 10.

Due to the synchronized specification of the start of the flash illumination and the start of detection, the control device 10 necessarily also specifies the delay period Ati _ag between the start of the illumination flash (t = 0) and the start of detection (t = tß) of the device 1.

Correlating with the speed of light, this results in a distance range 11 recorded by the stereo camera 5 in the forward direction 4 of the device 1. The object 2 is at a distance from the device 1 along the forward direction 4 that lies within this distance range 11, so that the object 2 is recorded by the stereo camera 5. The recorded distance range 11 is approximately 100 m to 150 m away from the device 1 and has an extension of, for example, 30 m along the forward direction 4.

This distance is indicated in Fig. 1 at 12 and is not to scale compared to the distance range 11.

Due to the detection via the two cameras 5i, 52 of the stereo camera 5, a spatial image of the object 2 is recorded. A reference distance determination of the object 2 to the device 1 can then be carried out in a triangulatory manner redundantly to the TOF distance measurement over the delay period Ati _ag . The device 1 is therefore in principle a TOF camera with additional triangulation.

The stereo camera 5 can also perform a correspondence analysis of individual camera images recorded with the two cameras 5i, 52 in order to separate stereoscopically captured objects from background noise that is only present in one of the two cameras 5i, 52. Such a correspondence analysis is described, for example, in WO 2022/069424 A1 and in WO 2022/069425 A2. Initially, a stereo calibration of the stereo camera 5, in particular of the alignments and positions of the cameras 5i, 52 of the stereo camera 5, can be carried out using a method that is also described in WO 2022/069424 A1. With regard to the calibration, reference is also made to DE 10 2011 080 702 B3.

During spatial image capture using the device 1, a TOF distance of at least one captured object captured using the TOF principle can be compared with a triangular distance of the object captured using the at least one stereo camera using the triangular principle. The TOF distance can be used, for example, as an input variable for the triangular capture. This can facilitate the correspondence analysis during the triangular capture, for example by limiting a disparity range to be covered in the correspondence analysis.

Using the images of the object 2 illuminated by the flash light source 7 captured by the cameras 5i, 52 of the stereo camera 5, a shadow cast by the object 2 can also be captured for the illumination light of the flash light source 7. By means of such a shadow casting evaluation of the different shadows cast by the object 2, captured by the two cameras 5i, 52 of the stereo camera 5, an additional redundant determination of the distance of the object 2 to the device 1 is possible.

The device 1 has a further flash light source 13, the function of which corresponds to that of the flash light source 7, which is also used for the controlled specification of a flash lighting start and a flash lighting duration of successive lighting flashes. The further flash light source 13 can be used redundantly to the initially described flash light source 7, for example in the event of a failure of the initially described flash light source 7. During operation of the further flash light source 13, the initially described flash light source 7 can then be replaced, for example, so that uninterrupted operation of the device 1 is possible, for example during a journey to be monitored by the car or truck on which the device 1 is mounted. The device 1 has a further stereo camera 14 with cameras 14i, 142, the structure and function of which corresponds to what was already explained above with reference to the stereo camera 5. The stereo camera 14i is arranged directly adjacent to the stereo camera 5i and covers practically the same field of view in the forward direction 4 as the camera 5i. The camera 142 is in turn arranged directly adjacent to the camera 52 and covers the same field of view in the forward direction 4 as the camera 52.

The stereo camera 14 is in turn redundant to the stereo camera 5. The same applies here as was said for the flash sources 7 and 13.

The device 1 also has a fisheye stereo camera 15 with two pairs 15i, 152 on the one hand and 15s, 154 on the other. Each of these fisheye cameras 15i (i = 1 to 4) is used to capture the environment around the device 1 both in the forward direction 4 and in a lateral direction (see arrow 16 in Fig. 1). The two fisheye cameras 15i and 152 arranged on the left of the vehicle 3 in Fig. 1 have a main capture direction 17 which, like the main capture directions of the cameras 5i and 6i, runs horizontally, but in the case of the fisheye cameras 15i and 152, forms an angle of 45° to the forward direction 4. These fisheye cameras 15i, 152 therefore cover both the forward direction 4 and the lateral direction opposite to the lateral direction 16 (left in Fig. 1) of the vehicle 3 on which the device 1 is mounted. Accordingly, the two further fisheye cameras 153, 154, whose main detection directions 18 also form an angle of 45° to the forward direction 4, also cover a right-hand area in the lateral direction 16 in addition to the forward direction 4.

The fisheye cameras 15i have a detection angle around the main detection directions 17, 18 of 180°. The limiting angles of the detection ranges of these fisheye cameras 15i are indicated by dashed lines in Fig. 1. These opening areas overlap in an overlap area 19, which is indicated by hatching in Fig. 1. In particular, the object 2 is located in this overlap area 19.

The illumination light source 7 emits flash light of the illumination flashes along the forward direction in an illumination cone, the edge rays of which are illustrated in Fig. 1 at 8a. The object 2 and also the overlap area 19 lie within the illumination cone 8a of the flash light source 7.

The fisheye stereo camera 15 can optionally carry out a triangulatory object distance determination using at least two of the four fisheye cameras 15i, whereby it is possible to work with short baselines, distances of the fisheye cameras 151, 152 on the one hand and 15s, 154 on the other hand, or also with long baselines (distances of the fisheye cameras 151 and 153 or 15i and 154 or 152 and 153 or 152 and 154). This in turn can be used to carry out a redundant triangulatory object distance determination.

The use of such additional triangulation baselines is described, for example, in WO 2022/069424 Al and in WO 2022/179998 Al.

The device 1 also has a further fisheye stereo camera 20 with fisheye cameras 20i to 2Ü4, which correspond to the fisheye cameras 15i to 154 in terms of their structure, orientation and pairwise arrangement. The fisheye stereo camera 20 can be used redundantly to the fisheye stereo camera 15. What was stated above for the flash light sources 7 and 13 or for the stereo cameras 5 and 14 applies accordingly here.

The cameras 15i of the fisheye stereo camera 15 in turn have sensor arrays in the manner of the sensor arrays 6i described above, and can therefore in turn be used as part of a TOF detection, as already explained above.

The control device 10 is in signal connection with the stereo cameras 5, 14, 15 and 20 and with the flash light sources 7 and 13. Furthermore, the control device 10 can be in signal connection with components of the vehicle 3, for example with a drive and/or with a braking system and/or with vehicle display devices, for example in a dashboard of the vehicle 3.

In real-time operation, the device 1 can take an image recording of, for example, ten images per second. Fig. 3 shows a further embodiment of a device 21 which can be used instead of the device 1 for spatial image capture of an environment moving relative to the device. Components and functions of the device 21 which correspond to those of the device 1 have in particular the same reference numbers and are not discussed again in detail.

The device 21 is equipped with two stereo cameras 5, 14, which record the surroundings of the device 21, which is in turn mounted on a vehicle 3, in the forward direction 4. Three environmental objects 22, 23 and 24 are illustrated within this recorded environment.

The device 21 enables, in particular, a z-buffered operation in the forward direction 4, which is also referred to as the z-direction, in which a total distance range 25 along the forward direction 4, within which the objects 22 to 24 lie, is divided into a plurality of distance ranges 22i to 22 _max , i.e. into distance ranges 22i (i = 1 to max). The maximum number max of these distance ranges 22i can be in the range between 3 and 50, for example in the range of 10. These distance ranges 22i are also referred to as distance zones (gate zones) gz.

In the corresponding Z-buffering operation of the device 21, the image acquisition parameters illumination flash start (t = 0), illumination flash duration (Atfiash, gz-i), a detection start (t = tß) and a detection duration (At _exp , gz-i) are initially specified.

This image acquisition parameter specification is again carried out by means of the control device 10.

The distance range gz-1, for example the distance range 22 _max , is then recorded using the image acquisition parameters specified in this way.

Further image acquisition parameters are specified via the control device 10, namely a further flash start (t = tfi _as h), a further flash duration (Afi _as h, gz), a further detection start (t = tßi) and a detection duration (At _exp , gz). This specification is selected so that a Delay period (Ati _ag , gz) with these further specified image acquisition parameters is somewhat shorter than the initially specified delay period Atiag, gz-i. This means that when capturing with the last specified image acquisition parameters, a further detection range gz is captured, which is adjacent to the initially captured distance range gz-1, i.e. the distance range 22max, at shorter distances from the device 21.

The further distance range gz adjoins or partially overlaps the preceding distance range gz-1 along the forward direction 4 in a detection progress direction +z.

Object 24 is located in this now recorded detection area gz.

The last captured image in the distance range gz is now digitally filtered to filter out unstructured image areas so that the object 24 remains. The image captured in the detection area preceding the detection of the distance range gz (detection of the distance range gz-1) is then overwritten with the last captured image (distance range gz) in non-filtered image areas, i.e. in the area of the object 24, of this last captured image, so that an updated image is stored in the control device 10 which contains the information from the two distance ranges gz-1 and gz.

During operation, the control device 10 specifies further image acquisition parameters so that a further distance range gz+1 is acquired, which in turn adjoins the distance range gz or partially overlaps with it in the opposite direction to the forward direction 4.

Now the last described steps of capturing, filtering, overwriting and specifying image capture parameters are repeated for this distance range gz+1 as well as other distance ranges 224, 22s, 222 and 22i until the specified total distance range 25 is covered, which results from the superposition of the captured distance ranges 22i to 22 _max . In the course of these sequences, the image areas at the location of the other objects 23 (distance range 222) and 22 (distance range 22i) are overwritten in the image updated for the distance range 22i, so that in the last updated image all three objects 22 to 24 are shown with their object distances to the device 1 determined via the assignment to the respective distance range 22i.

The number i (i = 1 to max) of distance ranges 22i that are recorded during the operating procedure can be 50, for example. The operating procedure with scanning of all distance ranges 22max to 22i can take place in real time in 10 Hertz operation. The result of such a complete scanning process of the distance ranges _22max to 22i is a 2D image with the non-filtered out objects 22, 23 and 24 in all distance ranges 22i recorded within the total distance range 25. Overwritten information of the distance ranges _22max to 222 initially recorded within the total distance range 25 can also be retained in the last updated, output image after recording the distance range 22i for an independent spatial (3D) evaluation.

As Fig. 4 illustrates, successive detection steps for distance ranges gz-1 and gz can overlap in time. More than two such detection steps can also overlap in time, so that, for example, during a delay period when detecting a first distance range gz, a flash of at least one further detection of a further distance range gz+1, gz+2 begins. A detection period of a previous detection can also overlap, for example, with a flash period of a subsequent detection.

Distance extensions Az of the various distance ranges 22i, 22j can differ from one another. The detection times can therefore differ in the various detection steps of the operating method.

After detecting a respective distance range 22i, a triangulation determination of a distance of the objects 22 to 24 in the non-filtered image areas can additionally be carried out using the at least one stereo camera 5, 14. In this case, a stereo correspondence determination can again take place, as already explained above.

Before a respective acquisition step of the operating procedure, a region of interest (ROI) can be specified as part of an image field that can be captured via the respective camera 5i, 14i, For example, a road area identified by appropriate object filtering in the forward direction 4.

Fig. 5 shows a side view of a further embodiment of the image capture device 1. Components and functions which correspond to those already explained above with reference to Figures 1 to 4 have the same reference numerals and will not be discussed again in detail.

The use of the device 1 for determining and detecting a diffuse scattering medium along a flash light path 28 between the flash light source 7 and an exemplary environmental object 2 is explained with reference to Fig. 5. The camera 5i of the stereo camera 5 is shown as an example in Fig. 5, which enables a TOF distance determination in addition to the triangulatory distance determination.

In the detection example according to Fig. 5, the device 1 detects a total of three distance ranges 22i, 222, 22s by specifying corresponding TOF image detection parameters. The first, closest distance range 221 extends by a first distance di. The next, further distance range 222 extends by a second distance d2, for which the following applies: d2 = 2 x di. The next, third distance range 22a extends by a third distance da, for which the following applies: da = 3 x di.

In addition to specifying the TOF image acquisition parameters “start of illumination”, “duration of illumination”, “start of detection” and “duration of detection”, the specification of the image acquisition parameters for the three distance ranges 22i to 22a also includes specifying a flash intensity L for illuminating the respective distance range 22i. The flash intensity E for illuminating the distance range 222 is four times as large as the flash intensity Ii for illuminating the first, closest distance range 22i. The flash intensity I3 for illuminating the third distance range 22a, which is the furthest away in the example according to Fig. 5, is nine times as large as the flash intensity Ii. The flash intensity L is therefore quadratically dependent on a distance di within a total distance range 25 to be recorded.

Assuming a constant reflectivity or a constant scattering power of an ambient surface 29, this quadratic dependence results in the illumination intensity L on the distance di, that the sensor array 6i of the camera 5i experiences a constant intensity illumination when capturing images of the three distance ranges 22i, 222 and 22s. This constant intensity illumination of the sensor array 6i only occurs when there is no significant scattering of the illumination flash 8 by the diffuse scattering medium 27 along the flash light paths 311, 3h, 3h between the flash light source 7 and the surroundings in the distance ranges 22i to 223. In other words: inequalities in the intensity illumination of the sensor array 6i along the detection paths 30i with the given quadratic dependence of the illumination intensity L on the distance di within the distance range 22i to be detected are an indication of the presence of the diffuse scattering medium 27.

The following cases can be distinguished here: a) The diffuse scattering medium 27 is within the currently recorded distance range, in the case shown in Fig. 5 therefore within the distance range 222. In this case, the diffuse scattering medium 27 is directly illuminated along the illumination path 3h and a corresponding scattering of the illumination light 8 through the diffuse scattering medium 27 into the sensor array 6i is measured during the TOF detection, which leads to an increase in the intensity illumination along the detection path 3Ü2. b) The diffuse scattering medium 27 is located between the flash light source 7 and the sensor array 6i on the one hand and the currently recorded distance range, for example the distance range 22s, on the other. In this case, the illumination light is weakened along the flashlight path 3U when it passes through the diffuse scattering medium 27. The illumination light scattered back from the detection area 223 in the direction of the sensor array 6i along the detection path 3Ü3 is further weakened due to further scattering by the diffuse scattering medium 27. As a result, the intensity illumination of the sensor array 6i is weakened when detecting the distance range 223 due to the cloud of the diffuse scattering medium 27 present in the intermediate distance range 222. A combination of an increased intensity illumination of the sensor array 6i when detecting the distance range 222 compared to the expected constant intensity illumination and a reduction in the intensity illumination of the sensor array 6i when detecting the distance range 22s thus enables the identification of the diffuse scattering medium 27 within the distance range 222.

In the above-explained embodiments of the image capture device 1 or 21, a pixel sensitivity of sensor pixels of the various sensor arrays 6i, ... can be regulated as a function of an incident light intensity, i.e. as a function of an intensity illumination of the sensor array 6i, ... In particular, a y-correction can be carried out during image processing using the respective sensor array 6i, ... The exponent y used for the y-correction can vary in the range between 0.2 and 5, for example. Depending on the size of y, a pixel sensitivity can then be achieved to adapt a high dynamic range either with high or low intensity illumination of the sensor array.

A selection of object distance ranges 22i to be detected can be adapted to a movement mode of the device 1 or 21 relative to the environment in certain operating modes of the detection device 1 or 21. In particular, the total distance range 25 can be adapted to a relative speed of the device 1, 21 to the environment. Adaptation to a reversing or cornering mode is also possible.

Alternatively or in addition to the operating method explained above with the aid of Figures 3 and 4, the respective device 1 or 21 can also be operated as follows:

By means of a corresponding assignment, in particular of the flash illumination duration and the detection duration, a TOF distance of at least one object 2 or 22 to 24 is recorded using the TOF principle with the aid of the respective device 1, 21. In parallel or sequentially, a triangulatory distance of the respective object 2 or 22 to 24 is recorded using the triangulatory principle using the one stereo camera 5, . . . of the device 1, 21. The detection values "TOF distance" and "triangulatory distance" are then compared when the device 1, 21 is in operation. In particular, it is possible to use the determined TOF distance as an input variable for the triangulatory detection. This can particularly facilitate the correspondence analysis in the triangulatory detection. In particular, a disparity range to be covered in the correspondence analysis can be limited. The reverse case is also possible, i.e. that the triangulatory distance is used as an input variable for the distance detection using the TOF principle. This can be used to determine or divide a given total distance range into section-by-section distance ranges 22i and to specify the TOF image detection parameters based on the distance.

An illumination intensity by the flash light source 7 can be specified in particular such that a common and in particular scalar brightness control of all images of the Z-buffering of the first figures 3 and 4 is carried out with in particular constant intensity illumination of the respective sensor arrays 6i, ... This brightness control can be used in the form of a control loop in the device 1, 21.

A calibration of a dependence of an illumination intensity of the flash light source on the distance, i.e. an illumination intensity function, can be carried out using a calibration setup using a scattering medium with a homogeneous scattering particle distribution.

The pixel sensitivity as a function of the incident light intensity can be set in the form of a proportional function or in the form of a non-linear function.

During the operation of the device 1 or 21, selected objects 2 or 22 to 24 can be tracked. As soon as one of the objects 2 or 22 to 24 has been recorded as part of the triangulatory distance detection and/or as part of the TOF distance detection, in particular the TOF image detection parameters, but also the triangulatory image detection parameters (for example the length of the baseline of a respectively selected stereo camera, a lens focal length of the cameras of the respective stereo camera or a disparity range to be monitored) can be specified in such a way that the object once recorded can also be tracked in subsequent image captures. is again detected. In this case, a relative movement of the object detected in each case to the device 1 or 21 can be taken into account, so that, for example, if it is expected that the object and the device will approach each other, the distance of the respectively selected distance range 22i is reduced in the TOF detection during tracking. A light intensity of the flash light source can be adjusted to a distance range of the object to be tracked in each case and in particular can be regulated.

Depending on the operating method of the device 1, 21, exactly one object can be tracked or several objects can be tracked simultaneously.

If at least one object is tracked, a tracking detection of the at least one tracked object and a scan over the entire distance range 25 can be carried out intermittently, which increases the overall detection reliability with a reasonable detection effort.

The image acquisition parameters can be adjusted for the TOF acquisition of several distance ranges 22i, 22i+i so that detection via the respective camera takes place during exactly one detection period (same detection start, same detection duration) for these distance ranges 22i, 22i+i. In extreme cases, this can apply to all distance ranges 22i within the total distance range 25.

When tracking objects, the extent of the respective distance range 22i in which the object is expected can be determined by appropriately adjusting the flash illumination duration. For more precise tracking of objects, shorter flash illumination durations and shorter detection durations can be specified with a corresponding increase in the distance accuracy in the TOF distance determination.

The device 1 can contain a neural network or a Kl module, which can be used to train the respective operating procedure and to improve image recognition accordingly.

Claims

patent claims 1. Device (1; 21) for spatial image capture of an environment, - with at least one stereo camera (5, 14, 15, 20; 5, 14) with at least two cameras (5i, 52, 14i, 142, 15i, 152, 15s, 154, 20i, 2Ü2, 2O3, 2O4) spaced apart from one another for triangulatory object distance determination within the detected environment, - with at least one flash light source (7, 13) for ambient lighting with controlled Specification of a flash illumination start (t = 0, t = tfiash) and a flash illumination duration (Atfiash) of consecutive illumination flashes, - wherein the cameras (5i, 52, 14i, 142, 15i, 152, 15s, 154, 20i, 2O2, 2O3, 2O4) of the stereo camera (5, 14, 15, 20; 5, 14) each have a sensor array (6) with controlled specification of a detection start (t = tE, t = tEi) and a detection duration (At _exp ), - with a control device (10) which is connected to the cameras (5i, 52, 14i, M2, 15i, 152, 153, 154, 201, 2O2, 2O3, 2O4) and the flash light source (7, 13), for the time-synchronized specification of the following image acquisition parameters: the start of flash illumination (t = 0, t = tfiash) and the flash illumination duration (Atfiash) of the flash light source (7, 13) as well as the start of detection (t = tE, t = tEi) and the detection duration (At _exp ) of the cameras (5i, 5 ₂ , 14i, 14 ₂ , 15i, 15 ₂ , 15 ₃ , 15 ₄ , 20i, 20 ₂ , 20 ₃ , 20 ₄ ). 2. Device according to claim 1 - with at least one further flash light source (13) with controlled specification of a flash illumination start (t = 0, t = tfiash) and a flash illumination duration (Atfiash) of successive illumination flashes (8, 81) and/or - with at least one further stereo camera (14) with two further spaced-apart cameras (14i, 2) for triangulatory object distance determination within the enclosed environment. 3. Device according to claim 1 or 2, characterized by at least one fisheye stereo camera (15, 20) with at least two fisheye cameras (151 to 154, 20i to 204), each of the fisheye cameras (15i to 154, 20i to 204) for detecting the environment both in a main detection direction (4) and along this in a lateral direction (16) is aligned, wherein detection areas of the fisheye cameras (15 i, 20i) overlap in the main detection direction (4) in an overlap area (19). 4. Device according to one of claims 1 to 3, characterized in that the control device (10) has an image processor for real-time processing. 5. Method for operating a device (1; 21) according to one of claims 1 to 4, comprising the following steps: - Specifying the image acquisition parameters such that a first distance range (gz-1) is captured, - Capturing the distance range (gz-1) with the specified image acquisition parameters with the cameras of at least one stereo camera, - specifying the image acquisition parameters such that a further distance range (gz) is acquired which borders on or partially overlaps the previous distance range (gz-1) along the forward direction (4) in a detection progress direction (+z), - Capturing the further distance range (gz) with the last specified image acquisition parameters with the cameras of at least one stereo camera, - digital filtering of the last captured image to filter out structureless image areas, - Overwriting the data recorded in the previous step of the last recording image with the last captured image in non-filtered image areas of the last captured image to generate an updated image, - Specifying the image acquisition parameters in such a way that a wider distance range (gz+1) which borders on or partially overlaps the preceding distance range (gz) along the forward direction (4) in the detection progress direction (-z), - Repeat the last steps of acquisition, digital filtering, overwriting and specification for further distance ranges (224 to 22i) until a specified total distance range (25) is covered, which results from a superposition of the acquired distance ranges (22i to 22 max ), - Output the last updated image. 6. Method according to claim 5, characterized in that at least two of the successive detection steps overlap with each other in time. 7. Method according to claim 5 or 6, characterized in that distance extensions (Az) of the distance ranges (22i) differ from one another. 8. Method according to one of claims 5 to 7, characterized in that after the detection of the respective distance range (gz-1, gz, gz+1; 22i), a triangulation determination of a distance of objects (22 to 24) in non-filtered image areas is carried out by means of the at least one stereo camera (5, 14). 9. Method according to one of claims 5 to 8, characterized in that before the detection, a region of interest (ROI) is specified as part of an image field that can be detected by the stereo cameras (5, 14). 10. Method for operating a device (1; 21) according to one of claims 1 to 4, comprising the following steps: - detecting a time of flight (TOF) distance of at least one object (2; 22 to 24) via a corresponding assignment of the flash illumination duration and the detection duration using the TOF principle, - detecting a triangulatory distance of the object (2; 22 to 24) by means of the at least one stereo camera using the triangulatory principle, - Matching the TOF distance and the triangulation distance. 11. The method according to claim 10, characterized in that the TOF distance is used as an input variable for the triangular detection. 12. Method according to one of claims 5 to 11, characterized in that an illumination intensity of the flash light source (7, 13) is adjusted to a distance range (22i) to be detected. 13. Method according to claim 12, characterized in that the flash intensity depends quadratically on a distance (di) within the distance range (22i) to be detected. 14. Method according to one of claims 5 to 13, characterized in that a pixel sensitivity of a sensor pixel of the sensor array (6) is controlled as a function of an incident light intensity of the flash light source (7, 13). 15. Method according to one of claims 5 to 14, characterized in that a selection of object distance ranges (22i) to be detected is adapted to a movement mode of the device (1; 21) relative to the environment.