EP4706016A1

EP4706016A1 - A method for generating a bird's eye top view image, a computer program product, a computer-readable storage medium as well as an electronic computing device

Info

Publication number: EP4706016A1
Application number: EP24723715.9A
Authority: EP
Inventors: Antonyo Musabini; Rachid BENMOKHTAR; Amaury BREHERET
Original assignee: Valeo Schalter und Sensoren GmbH
Current assignee: Valeo Schalter und Sensoren GmbH
Priority date: 2023-05-04
Filing date: 2024-04-25
Publication date: 2026-03-11
Also published as: DE102023111525A1; CN121079725A; WO2024227677A1

Abstract

The invention relates to a method for generating a bird's eye top view image (9) for being presented on a display device (4) of an assistance system (2) of a motor vehicle (1) by an electronic computing device (3) of the assistance system (2). The method corrects the bird's eye top view by dynamically adjusting the calibration parameters of each capturing device per frame.

Description

A method for generating a bird’s eye top view image, a computer program product, a computer-readable storage medium as well as an electronic computing device

The invention relates to a method for generating a bird’s eye top view image for being presented on a display device of an assistance system of a motor vehicle by an electronic computing device of the assistance system. Furthermore, the invention relates to a computer program product, a computer-readable storage medium as well as an electronic computing device.

Inverse perspective mapping (IPM) is widely employed in self-driving applications. It takes the assumption of a plane world, in particular of a high-set equal zero. Then, with the calibration parameters of cameras, it maps all pixels from a given viewpoint onto this flat plane through homographic projection. Calibration parameters are known as the intrinsic parameters of cameras, which are related to the physical characteristics of it and the extrinsic parameters, which are their location and rotation compared to the motor vehicle.

The major limitation of a bird’s eye top view image obtained with the inverse perspective mapping is linked to the flat ground assumption. This assumption is not always true. For instance, if the motor vehicle is passing over a speed bump or driving over a pothole it may be false. Moreover, even if the motor vehicle is on a real flat surface, due to the vehicle movement such as accelerations and deceleration, the cameras might not be aligned as calibrated. When the cameras are not aligned or when the flat ground assumption is not respected, the projected image has a strangely moving effect called “wobbling”. While wobbling represented distances on the image are not consistent and the image is not pleasing to look at.

Therefore, according to the state of the art, these limitations are overcome by using a very narrow projection range around the motor vehicle and activating this view only when the motor vehicle is moving with low speed, for example, while executing a parking maneuver.

US 10 486 596 B2 proposes a stitching algorithm while projecting four cameras to a bird’s-eye-top view.

CN 103035005 B proposes an automatic calibration method for panorama view used by parking automated systems. It is an object of the invention to provide a method, a computer program product, a computer-readable storage medium as well as an electronic computing device, by which an improved bird’s eye top view image can be presented on a display device.

This object is solved by a method, a corresponding computer program product, a corresponding computer-readable storage medium as well as a corresponding electronic computing device according to the dependent claims. Advantageous forms of embodiments are presented in the dependent claims.

One aspect of the invention relates to a method for generating a bird’s eye top view image for being presented on a display device of an assistance system of a motor vehicle by an electronic computing device of the assistance system. At least a first image of the surroundings of the motor vehicle is captured in a first time step by a first capturing device of the assistance system. At least a second image of the surroundings of the motor vehicle is captured in a second time step by the first capturing device, wherein the second time step is following the first time step. At least the first image and the second image are transmitted to the electronic computing device. A movement of an object in the first image and the second image is determined by the electronic computing device. A height movement of the motor vehicle relative to the surroundings is determined depending on the determined movement of the object by the electronic computing device. A correction value for correcting the first image and the second image is generated by the electronic computing device. At least the first image, the second image, and the correction value is provided for generating the bird’s eye top view image depending on at least the first image, the second image, and the correction value by the electronic computing device.

Therefore, the provided method detects movements of the capturing device from its image space and dynamically apply, for example, an offset to its extrinsic parameters, for example, as translations and rotations, in particular in six degrees of freedom. Applying dynamically these offsets may compensate for the perturbations which are linked to for instance passing on speed bumper or the height variations caused from the movement of the motor vehicle, in particular during acceleration and deceleration. These computed offsets are not meant to replace the camera's existing calibrations, hence this is not an auto-calibration. These offsets are only used during the processing of images grabbed at the exact moment for creating a bird’s eye top view image. The resulting bird’s eye top view image hence is free from the wobbling effect described before. The method corrects a bird’s eye top view image by dynamically adjusting the calibration parameters of each capturing device per frame. This is the only method which proposes such a level of correction, in particular six degrees of freedom. Also, once the needed offsets are computed, the method does not modify the input images to create a correct top view, or does not modify the top view itself. Instead, it uses the raw images by dynamically adjusting the calibration parameters. Hence, the input images and the created top views are not processed again, which could be slower in terms of execution time, and it could bring visual artefacts to the image. Therefore, extending the limited range of a 360°-view may be proposed. For example, the motor vehicle may be able to create visual logical top views even when the motor vehicle moves faster than 15 kilometers per hour. Using the extended 360°-view for other use cases then parking, may be provided, for example, on a highway as a situation awareness or a lane change in an at least in part automated movement of the motor vehicle or on city roads for improving the driver’s vision.

According to another embodiment, the correction value is generated in an internal electronic computing device of the first capturing device. For example, the electronic computing device may be arranged at the capturing device. The correction value is then generated inside this electronic computing device. The first image, the second image and the correction value are then transmitted, for example, to a central electronic computing device, wherein the central electronic computing device may get further images from further capturing devices. Therefore, the correction value is generated inside the capturing device.

In another embodiment, the first image and the second image is transmitted to a central electronic computing device, and wherein the correction value is generated by the central electronic computing device. Therefore, the central electronic computing device may have a greater power than the capturing device, and therefore, the correction value can be generated in an improved manner.

In another embodiment, further images and further correction values are captured and generated by at least a second capturing device and the bird’s eye top view image is stitched together depending on at least the images and the correction values of the first capturing device and the further images and the further correction values of at least the second capturing device by a central electronic computing device. In particular, the method may be also computed for the second capturing device. The bird’s eye top view image may be stitched together by at least the images from the first capturing device and from the second capturing device. By correcting the images from the first capturing device as well as from the second capturing device, an improved bird’s eye top view image may be determined. In particular, the bird’s eye top view image may be then presented on a display device of the assistance system.

In another embodiment, a threshold for the height movement is provided, and if a value of the height movement is lower than the threshold, generating of the correction value is suppressed, and if the value of the height movement is higher than the threshold, the correction value is generated. In particular, if one of the remaining computed vertical movements exceeds the previously defined threshold, for example in the motor vehicle, wherein the motor vehicle may get into a speed bumper, the translation and rotation matrices to compensate for the vertical movements from a key point is computed. The minimization of vertical movements could be done with, for example, a solver. The resulting matrices are used as offset during the bird’s eye top view image generation.

According to another embodiment, depending on the correction value an extrinsic parameter of the first capturing device is adapted. In particular, an offset in the six degrees of freedom is provided depending on the correction value.

In another embodiment, the bird’s eye top view image is generated depending on captured images from at least four capturing devices. In particular, the four capturing devices may be arranged at the motor vehicle, for example one front camera, one tail camera, one left-side camera and one right-side camera. Therefore, a 360°-view as the bird’s eye top view image may be provided.

In another embodiment, the first capturing device is configured as a camera. In particular, cameras are already known in the state of the art and arranged at the motor vehicle. Therefore, the method can be performed with existing cameras of the motor vehicle.

In another embodiment, the object is determined by a computer vision algorithm. In particular, the camera movement detection is accomplished by comparing interest points into the images. These interest points could be detected with multiple ways, for instance, well-known computer vision algorithms. A neural network or any other method can be used. These key points are extracted from at least two successive images, in particular from the first image and the second image. The more the number of successive images to use is important, the more the system becomes stable but slow to react. The current speed of the motor vehicle can be extracted from, for example, a CAN-bus and can be used to vary the number of frames in real time. In particular, a slower motor vehicle might need a less reactive application so it might process more image frames. These key points from successive images are matched in order to know the location change of a single key point over successive image frames. The variation of location change of key points are used to define a movement vector per key point on horizontal and vertical axes. In order to eliminate the key point extracted from the other moving objects, for example such as other motor vehicles and pedestrians, the computed movement vectors are compared to the expected movement vector in terms of the ego-vehicle’s current speed. Key points moving more than the maximum threshold or less than a minimum threshold are eliminated.

In another embodiment, a velocity signal is transmitted to the electronic computing device and the velocity signal is taken into consideration by the electronic computing device. In particular, the velocity signal is taken into consideration for determining the movement of the object. Furthermore, the velocity signal is taken into consideration for determining an amount of images needed for determining the correction value.

In particular, the presented method is a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the preceding aspect.

A still further aspect of the invention relates to a computer-readable storage medium comprising at least the computer program product according to the preceding aspect.

Furthermore, the invention relates to an electronic computing device for an assistance system for generating a bird’s eye top view image for being presented on a display device of the assistance system of a motor vehicle, wherein the electronic computing device is configured for performing a method according to the preceding aspect. In particular, the method is performed by the electronic computing device.

The electronic computing device may comprise at least processors, circuits, in particular electronic circuits, and further electronic means for performing the method.

The electronic computing device can be an electronic computing device of a capturing device or a central electronic computing device.

Furthermore, the invention relates to an assistance system of a motor vehicle comprising at least the electronic computing device as well as at least one capturing device. Furthermore, the invention relates to a motor vehicle comprising at least the assistance system.

Advantageous embodiments of the method are to be regarded as advantageous embodiments of the computer program product, the computer-readable storage medium, the electronic computing device, the assistance system, as well as the motor vehicle. In particular the assistance system and the motor vehicle have device features for this purpose to be capable of performing corresponding method steps.

Further features of the invention are apparent from the claims, the figures and the figure description. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of figures and/or shown in the figures alone may comprise the invention not only in the respective combination stated, but also in other combinations, without leaving the scope of the invention. Thus, in particular, embodiments are to be regarded as comprised and disclosed by the invention, which are not explicitly shown and explained in the figures, however derive by separated feature combinations from the explained embodiments and can be generated therefrom. Also embodiments and combinations of features which thus do not have all features of an originally formulated independent claim are to be regarded as disclosed. Moreover, embodiments and feature combinations, in particular by the above explanations, are to be regarded as disclosed, which go beyond or deviate from the combinations of features set forth in the recitations of the claims may comprise the invention.

The drawings show in:

Fig. 1 a schematic top view according to an embodiment of a motor vehicle comprising an embodiment of an assistance system comprising of an embodiment of an electronic computing device;

Fig. 2 a schematic block diagram according to a bird’s eye top view image; and

Fig. 3 shows a schematic flow chart according to an embodiment of the invention. In the figures the same elements or elements having the same function are equipped with the same reference signs.

Fig. 1 shows a top view according to an embodiment of a motor vehicle 1 . The motor vehicle 1 comprises at least one assistance system 2. The assistance system 2 comprises at least an electronic computing device 3, a display device 4 as well as at least one capturing device 5, 6, 7, 8. In particular, according to the shown embodiment, the assistance system 2 comprises at least a first capturing device 5, a second capturing device 6, a third capturing device 7 as well as a fourth capturing device 8. The first capturing device 5 is arranged at the front of the motor vehicle 1 , the second capturing device 6 is arranged on a right side of the motor vehicle 1 , the third capturing device 7 is arranged at a tail of the motor vehicle 1 and the fourth capturing device 8 is arranged on a left side of the motor vehicle 1 . The assistance system 2 is configured for presenting a bird’s eye top view image 9 (Fig. 2) on the display device 4.

In particular, a x-direction, a y-direction, and a z-direction is shown. The x-direction and the y-direction are horizontal directions, wherein the z-direction is a vertical direction.

Fig. 2 shows a schematic block diagram according to an embodiment of the electronic computing device 3, in particular of the bird’s eye top view image 9.

In particular, a first area 10 of the bird’s eye top view image 9 is captured from the first capturing device 5. A second area 11 is captured by the second capturing device 6. A third area 12 is captured by the third capturing device 7, and a fourth area 13 is captured by the fourth capturing device 8. In particular, the bird’s eye top view image 9 is stitched together from the four presented areas 10, 11 , 12, 13.

According to an embodiment of the method for generating the bird’s eye top view image 9, at least a first image 14 of the surroundings 15 (Fig. 1 ) of the motor vehicle 1 in a first time step ti is performed by the first capturing device 5. At least a second image 16 of the surroundings 15 of the motor vehicle 1 is captured in a second time step t₂ by the first capturing device 5, wherein the second time step t₂ is following the first time step ti. At least the first image 14 and the second image 16 are transmitted to the electronic computing device 3. A movement of an object in the first image 14 and the second image is determined by the electronic computing device 3. A height movement, in particular in a z-direction of the motor vehicle 1 relatively to the surroundings 15, is determined depending on the determined movement of the object by the electronic computing device 3. In particular, the z-direction is a vertical direction.

In particular, a correction value 17 is generated for correcting the first image 14 and the second image 16 by the electronic computing device 3. At least the first image 14, the second image 16 and the correction value 17 for generating the bird’s eye top view image 9 is provided depending on at least the first image 14, the second image 16 and the correction value 17 by the electronic computing device 3. It should be noted that each part of the top view has its own correction.

According to an embodiment, the correction value 17 is generated in an internal electronic computing device 3 of the first capturing device 5. Alternatively, the first image 14 and the second image 16 are transmitted to a central electronic computing device 3, and wherein the correction value 17 is generated by the central electronic computing device 3.

As shown in Fig. 2, further images 18 and further correction values 19 are captured and generated by at least the second capturing device 6 and the bird’s eye top view image 9 is stitched together depending on at least the images 14, 16 and the correction values 17 of the first capturing device 5 and the further images 18 and the further correction values 19 of at least the second capturing device 6 by the central electronic computing device 9.

Furthermore, a threshold for the height movement is provided, and if a value of the height movement is lower than the threshold, generating of the correction value 17 is suppressed, and if the value of the height movement is higher than the threshold, the correction value 17 is generated.

Depending on the correction value 17, an extrinsic parameter of each of the capturing devices 5, 6, 7, 8 may be adapted, wherein the capturing devices 5, 6, 7, 8 may be, for example, a camera.

Furthermore, a velocity signal 20 of the motor vehicle 1 may be transmitted to the electronic computing device 3, and the velocity signal 20 may be taken into consideration by the electronic computing device 3. For example, the velocity signal 20 is taken into consideration for determining the movement of the object and/or the velocity signal 20 is taken into consideration for determining an amount of images 14, 16 needed for determining the correction value 17. In particular, a camera movement detection is accomplished by comparing interest points into the images 14, 16. The interest points may be also regarded as key points and may be regarded in this context as the object. These interest points could be detected with multiple ways, for instance, well-known computer vision algorithms. These key points are extracted from at least two successive images 14, 16. More the number of successive images 14, 16 to use is important, the more the assistance system 2 becomes stable but slow to react. The current speed of the motor vehicle 1 can be extracted from the CAN- bus and can be used to vary the number of frames in real time. In particular, a slower motor vehicle 1 might need a less reactive application so it might process more image frames. These key points from successive images are matched in order to know the location change of a single key point over successive image frames.

The variation of location change of key points is used to define a movement vector per key point on horizontal and vertical axes. In order to eliminate the key point extracted from the other moving objects, for example such as other motor vehicles and pedestrians, the computed movement vectors are compared to the expected movement vector in terms of the ego-vehicle’s current speed. Key points moving more than the maximum threshold or less than a minimum threshold are eliminated.

If one of the remaining computed vertical movements exceeds a previously defined threshold, for example the motor vehicle gets into a speed bumper, translation and rotation matrices to compensate for the vertical movement from a key point is computed. The minimization of vertical movements could be done with a solver.

The above computation is done for all exterior cameras of the motor vehicle 1 , individually.

Resulting matrices are used as offsets during the bird’s-eye-top-view image 9 generation.

Fig. 3 shows a schematic flow chart according to an embodiment of the invention. In a first step S1 , the method starts. In a second step S2, new frame-images 14, 16 are received from the capturing devices 5, 6, 7, 8. In a third step S3, the key points are determined. In a fourth step S4, the key points are stored in a memory. Coming from the first step S1 , a fifth step S5 is performed, wherein the velocity signal 20 is received from the CAN-bus. After the fourth step S4 and the fifth step S5, a sixth step S6 is performed, wherein the interest points or the key points are compared from last frames. A seventh step S7 is performed, wherein it is decided if an adjustment is needed. If no adjustment is needed, the method starts again from the first step S1 . If an adjustment is needed, an eighths step S8 is performed, wherein translation and rotation offset matrices are computed. From the eighths S8 a ninth step S9 is performed, wherein applying the offset while the bird’s-eye- top-view image projection is performed. From the ninth step S9, the first step S1 is performed again.

The method can be used to generate a bird’s eye top view image for presentation on a display device or as input for any other algorithm which uses top views as inputs.

Claims

1 . A method for generating a bird’s eye top view image (9) for being presented on a display device (4) of an assistance system (2) of a motor vehicle (1 ) by an electronic computing device (3) of the assistance system (2), comprising the steps of:

- capturing at least a first image (14) of the surroundings (15) of the motor vehicle (1) in a first time step (ti) by a first capturing device (5) of the assistance system (2);

- capturing at least a second image (16) of the surroundings (15) of the motor vehicle (1) in a second time step (t₂) by the first capturing device (5), wherein the second time step (t₂) is following the first time step (ti);

- transmitting at least the first image (14) and the second image (16) to the electronic computing device (3);

- determining a movement of an object in the first image (14) and the second image (16) by the electronic computing device (3);

- determining a height movement of the motor vehicle (1) relative to the surroundings (15) depending on the determined movement of the object by the electronic computing device (3);

- generating a correction value (17) for correcting the first image (14) and the second image (16) by the electronic computing device (3); and

- providing at least the first image (14), the second image (16), and the correction value (17) for generating the bird’s eye top view image (9) depending on at least the first image (14), the second image (16), and the correction value (17) by the electronic computing device (3) by dynamically adjusting the calibration parameters of each capturing device per frame.

2. A method according to claim 1 , wherein the correction value (17) is generated in an internal electronic computing device (3) of the first capturing device (5).

3. A method according to claim 1 , wherein the first image (14) and the second image (16) is transmitted to a central electronic computing device (3), and wherein the correction value (17) is generated by the central electronic computing device (3).

4. A method according to any one of claims 1 to 3, wherein further images (18) and further correction values (19) are captured and generated by at least a second capturing device (6) and the bird’s eye top view image (9) is stitched together depending on at least the images (14, 16) and the correction values (17) of the first capturing device (5) and the further images (18) and the further correction values (19) of at least the second capturing device (6) by a central electronic computing device (3).

5. A method according to any one of claims 1 to 4, wherein a threshold for the height movement is provided, and if a value of the height movement is lower than the threshold, generating of the correction value (17) is suppressed, and if the value of the height movement is higher than the threshold, the correction value (17) is generated.

6. A method according to any one of claims 1 to 5, wherein depending on the correction value (17) an extrinsic parameter of the first capturing device (5) is adapted.

7. A method according to any one of claims 1 to 6, wherein the bird’s eye top view image (9) is generated depending on captured images (14, 16, 18) from at least four capturing devices (5, 6, 7, 8).

8. A method according to any one of claims 1 to 7, wherein the first capturing device (5) is configured as a camera.

9. A method according to any one of claims 1 to 8, wherein the object is determined by a computer vision algorithm.

10. A method according to any one of claims 1 to 9, wherein a velocity signal (20) is transmitted to the electronic computing device (3) and the velocity signal (20) is taken into consideration by the electronic computing device (3).

11. A method according to claim 10, wherein the velocity signal (20) is taken into consideration for determining the movement of the object.

12. A method according to claim 10 or 11 , wherein the velocity signal (20) is taken into consideration for determining an amount of images (14, 16) needed for determining the correction value (17).

13. A computer program product comprising program code means for performing a method according to any one of claims 1 to 12.

14. A computer-readable storage medium comprising the computer program product according to claim 13.

15. An electronic computing device (3) for an assistance system (2) for generating a bird’s eye top view image (9) for being presented on a display device (4) of the assistance system (2) of a motor vehicle (1 ), wherein the electronic computing device (3) is configured for performing a method according to any one of claims 1 to