EP4705799A1

EP4705799A1 - Method and system to calibrate a vehicle sensing device

Info

Publication number: EP4705799A1
Application number: EP24723495.8A
Authority: EP
Inventors: Christoph Parl; Jonas KRAUSE; Josslyn Beltran-Madrigal; Thorsten BEUTH
Original assignee: Valeo Detection Systems GmbH
Current assignee: Valeo Schalter und Sensoren GmbH
Priority date: 2023-05-05
Filing date: 2024-04-30
Publication date: 2026-03-11
Also published as: WO2024231165A1; DE102023111738A1

Abstract

The present disclosure relates to a method and system to calibrate at least one frame-based vehicle sensing device, which may be a lidar forming a 2D or 3D map of a surrounding scene, wherein the at least one sensing device frame-wise captures the vehicle's (10) environment depending on a first set of parameters, wherein a processing device performs the calibration by adjusting the first set of parameters, wherein the at least one sensing device captures (A, C) at least one frame of the environment with the first set of parameters before (A) and after (C) capturing (B) at least one frame of the environment with a second set of parameters, wherein the processing device determines a third set of parameters depending on calibration data obtained by capturing the at least one frame of the environment with the second set of parameters, and compares (D) the frames with one another that have been captured (A, C) with the first set of parameters before (A) and after (C) capturing (B) the at least one frame of the environment with the second set of parameters, and wherein the processing device adjusts (E.1, E.2) the first set of parameters depending on the outcome of the comparison (D). Depending on the difference of scene in (A) compared to (C), it can be judged if the vehicle has moved and therefore is in a driving state which is unsuitable for calibration. The method may therefore serve to verify online calibration data.

Description

METHOD AND SYSTEM TO CALIBRATE A VEHICLE SENSING DEVICE

Field

The present disclosure relates to a method and system to calibrate at least one frame-based vehicle sensing device.

Background

Modern vehicles (cars, vans, trucks, motorcycles, etc.) may comprise different kinds of sensing devices, whose data are used for driver information and/or are used by driver assistance systems. The sensing devices may be used to capture the vehicle's surroundings including other road users. Based on the captured data, a model of the vehicle environment may be generated and a reaction to changes in this vehicle environment is possible, e. g. by the driver and/or by the driver assistance systems.

Sensing devices are constantly being developed for various functions, e. g. for the acquisition of environmental information in the near and far range of vehicles, such as passenger cars or commercial vehicles. Sensing devices can also serve as sensors for driver assistance systems, in particular for autonomous or semi-autonomous vehicle control. Sensing devices can for example be used to detect obstacles and/or other road users in the front, rear or blind spot areas of the vehicle. Sensing devices can be based on different sensor principles, such as radar, ultrasound, optics. An example for a sensing device based on optics is the capture of images by a camera.

In US10176596B1 a vehicle control system is described, wherein the relationship between a particular imaging device and a particular ranging device is calibrated to obtain conversion parameter values that may be used to translate ranging data from the ranging device to an image captured by the imaging device, and vice versa.

In US9734419B1 a system and method to validate the accuracy of a camera calibration is described. The camera calibration is validated by imaging a calibration object within the volume space of the camera arrangement. The resulting validation indicates if a recalibration is required. Summary

A frame-based vehicle sensing device is configured to frame-wise capture the vehicle's environment depending on a first set of parameters. A calibration of the vehicle sensing device is performed by adjusting the first set of parameters. A method to calibrate at least one frame-based vehicle sensing device comprises:

A) capturing at least one frame of the environment with the first set of parameters,

B) capturing at least one frame of the environment with a second set of parameters, wherein a third set of parameters is determined depending on calibration data obtained by capturing the at least one frame of the environment with the second set of parameters,

C) capturing at least one frame of the environment with the first set of parameters,

D) comparing the at least one frame captured in A) with the at least one frame captured in C) and

E) adjusting the first set of parameters depending on the outcome of D).

Frame-based sensing devices capture their surroundings within their sensing area at intervals, which may be fixed intervals. The data captured during one such interval is called a frame. Frame-based sensing devices may therefore produce a sequence of frames at a fixed rate. A lidar device may be such a frame-based sensing device. By scanning the laser along the horizontal and/or vertical directions, the lidar device may form a complete 2D or 3D map of the surrounding scene in the sensing area. Every such map may be called a frame.

The described method allows to perform a so-called online calibration of the at least one sensing device. An online calibration is a calibration performed during the operation phase of the sensing device, while being used in a vehicle. During the operation phase the sensing device may be mounted on a vehicle and may be outputting its sensor data to control units of the vehicle, which may use this data to control actors and/or other sensing devices of the vehicle.

The method allows to adjust different parameters during the regular operation of the vehicle. Visiting a certain pre-defined calibration environment may be avoided. Possible properties of the sensing device, that may be calibrated by adjusting parameters are e. g. signal gain, white balance, noise thresholds, deviations in optical path, readout channel, transmit power, etc. These properties and other properties may undergo change or degradation during use. Re-calibration may help to improve precision and/or reliability again. When calibrating them by adjusting parameters of the sensing device, the adjustments of the parameters may take into account the changes in the physical structure of the sensing devices.

The measurement data may be validated, because the method allows to judge if and how much the captured surroundings, also called the scene, has changed between the measurements in B) compared to A) and C). Depending on the difference of scene in A) compared to C), which is checked in D), it can for example be judged if the vehicle has moved and therefore is in a driving state which is unsuitable for calibration.

The method may serve to verify online calibration data, where first in A) at least one frame is captured with the first set of parameters. In B) at least one frame is captured with the second set of parameters to obtain the calibration data. After that at least one frame is captured with the first set of parameters again.

In an embodiment of the method, D) comprises detecting if the environment captured during A) is similar to the environment captured during C). With this embodiment it can be verified that the captured surroundings have not changed too much between the measurements in B) compared to A) and C). If the scene in A) is similar to the scene in C), it can be safely assumed that the scene has not changed in B) either and that the calibration data obtained in B) is based on similar conditions as the frame captured in A) and C).

In an embodiment of the method, A) and C) are performed by the at least one vehicle sensing device and E) comprises

E.l) updating the first set of parameters to be equal to the third set of parameters if the at least one frame captured in A) is similar to the at least one frame captured in C) or

E.2) keeping the first set of parameters if the at least one frame captured in A) is not similar to the at least one frame captured in C).

If the scene in A) is similar to the scene in C), it can be safely assumed that the scene has not changed in B) either and that the calibration data obtained in B) is based on similar conditions as the frame captured in A) and C). The calibration data obtained in B) can therefore be used to calibrate the device and to adjust the first set of parameters in E.l) depending on the calibration data. Otherwise, the first set of parameters can - in E.2) - be kept unchanged.

In an embodiment of the method, the sensing device is configured to provide sensor data depending on the vehicle's environment captured depending on the first set of parameters. In A), at least one frame is captured with the first set of parameters, which correspond to the set of parameters which correspond to the normal standard use of the sensing device in the vehicle to sense the surroundings of the vehicle. In B) at least one frame is captured with the second set of parameters to obtain the calibration data. After that at least one frame is captured with the first set of parameters again, which correspond to the set of parameters which correspond to the normal standard use of the sensing device in a vehicle to capture the surroundings of the vehicle.

In an embodiment, the method further comprises detecting a stationary condition of the vehicle and performing A) to E) during the stationary condition. The stationary condition of the vehicle may for example be obtained by another sensor device in the vehicle and/or a control unit of the vehicle, like for example a central control unit of the vehicle. The control unit is an electrical device that receives input signals, for example sensor signals and/or signals from other control units and/or input signals and generates a control signal as a function thereof. This control signal is then used, for example, to control a further control unit or, for example, an actuator, such as a brake, an airbag or an automated driving device. The control unit therefore has input and output interfaces.

In an embodiment of the method, the calibration data comprises reference data and/or tentative captured data. In B) at least one frame is captured with the second set of parameters to obtain the calibration data. The capture in B) serves to obtain reference data and/or check the last setting for deviations or optimize it. Reference data is data with which actual captured data of the sensing device is compared to adjust the parameters of the sensing device. Those settings may be considered tentative settings which may be confirmed by going through the method again to confirm them when going through B) again.

Method according to one of the preceding claims, wherein the at least one sensing device is at least one of a lidar device, a radar device, a camera device or an ultrasound device. The lidar technology (lidar: light detection and ranging) is an important optical sensor principle for environment detection, e. g. of vehicles. A lidar device comprises an optical transmission device and an optical reception device. The transmission device emits an optical signal, which can be continuous or pulsed. In addition, the optical signal may be modulated. For example, electromagnetic waves in the form of laser beams in the ultraviolet, visual or infrared range may be used as optical signals in a lidar device. The light is received by the reception device after reflection from an object in the sensing area in the range of the lidar device. The optical signal may for example be evaluated using a time-of-flight method and the spatial position and distance of the object on which the reflection occurred may be determined. In addition, it may be possible to determine a relative velocity between the object and the sensor device. Reflection or reflected light is understood to mean any light that is reflected back and should also include, in particular, light that is reflected back by scattering or absorption emission.

Another important sensor principle for capturing surroundings is the radar technology, which is based on electromagnetic waves in the radio frequency range, e. g. in the microwave range. The radar device sends and receives radar signals. The radar signals can be evaluated and information about an object on which a reflection of the radar signal occurred can be determined.

The lidar device has the advantage of very high angular and distance resolution. This means that lidar devices may scan objects with a high resolution and may be able to scan them more finely than e. g. radar devices. However, lidar devices may be more limited in fog or rain conditions. Radar devices, on the other hand, can detect vehicles even in difficult visibility and weather conditions. Cameras, for their part, are very well suited for recognizing traffic signs and traffic lights. However, cameras depend on sufficient light conditions for image capture.

Ultrasound sensing devices, also called ultrasonic devices or ultrasound devices, are surround sensors based on ultrasound waves. The ultrasonic devices send out short ultrasonic impulses which are reflected by obstacles. The echo signals are then received and processed. They may be used for range detection, e. g. the calculation of distances to obstacles, and to monitor the surroundings of a vehicle when parking and/or maneuvering.

In an embodiment of the method the calibration data is obtained in B) by capturing the at least one frame of the environment with the second set of parameters using a further sensing device. In this embodiment it is possible to use a further sensing device which is available to calibrate the at least one sensing device. For example, a lidar device may be calibrated using a camera device, a radar device or an ultrasound device.

In an embodiment of the method B) comprises obtaining calibration data by capturing multiple frames of the environment depending on multiple second sets of parameters. In this embodiment, multiple frames with multiple second sets of parameters may be used in B) to obtain the calibration data. This may further improve the calibration data.

In some embodiments, the multiple frames of the environment may be captured using multiple sensing devices. For example, several further sensing devices may be used in B) - with respective second sets of parameters - to obtain the calibration data. For example, a lidar device may be calibrated using a camera device and a radar device.

A system to calibrate at least one frame-based vehicle sensing device comprises the at least one sensing device configured to frame-wise capture the vehicle's environment depending on a first set of parameters. The system further comprises a processing device configured to perform the calibration by adjusting the first set of parameters. The system is suitable to perform the method to calibrate as described herein.

The at least one sensing device is configured to capture at least one frame of the environment with the first set of parameters before and after capturing at least one frame of the environment with a second set of parameters.

The processing device is configured to determine a third set of parameters depending on calibration data obtained by capturing the at least one frame of the environment with the second set of parameters, and to compare the frames with one another, that have been captured with the first set of parameters before and after capturing the at least one frame of the environment with the second set of parameters. The processing device is further configured to adjust the first set of parameters depending on the outcome of the comparison.

For example, first set of parameters may be updated to be equal to the third set of parameters if the compared frames are similar or the first set of parameters may be kept unchanged if the compared frames are not similar. The processing device may be comprised in the at least one sensing device. The processing device may be comprised in a central controller of the vehicle. The system as described herein may be comprised in a vehicle, like e. g. a passenger car, a commercial vehicle or other vehicles, e. g. two-wheeled vehicles.

The vehicle may comprise the system as described in this disclosure. The vehicle may further comprise the central controller configured to receive the sensor data from the at least one sensing device and/or further sensing devices and to further process the sensor data and to output data to other devices of the vehicle. The central controller may be configured to receive sensor data from a plurality of sensing devices and to exchange data with a plurality of actors and sensing devices of the vehicle to implement autonomous or semi-autonomous vehicle driving functions.

Brief description of the figures

Embodiments will now be described with reference to the attached drawing figures by way of example only. Like reference numerals are used to refer to like elements throughout. The illustrated structures and devices are not necessarily drawn to scale.

Fig. 1 schematically illustrates a method to calibrate a vehicle sensing device.

Fig. 2 schematically illustrates a vehicle comprising sensing devices and a central controller.

Detailed Description

Fig. 1 schematically illustrates a method to calibrate a vehicle sensing device 12, 14, 16, 18. The vehicle sensing device 12, 14, 16, 18 may be calibrated by adjusting a set of parameters. The method allows to determine calibration data and to validate the calibration data. The validation comprises the verification if the circumstances under which the calibration data have been determined are such that calibration data can be validly obtained. The method allows for online calibration during the operating phase of the sensing devices 12, 14, 16, 18. Re-calibration can therefore be applied to the sensing devices 12, 14, 16, 18 during their normal operation without having to seek a specific pre-defined environment. The validation of the calibration data is important, because during the operation phase of the sensing devices 12, 14, 16, 18, the environment can always change so that the calibration data may become invalid, because measurement data for comparison may be based on different conditions.

However, it often happens that several frames are recorded for the same scene, e. g. when the vehicle 10 is not moving. The method takes advantage of such situations, where at least one of the frame-based sensing devices 12, 14, 16, 18 captures frames of the same surroundings that do not change. Such a situation is useful for calibration, in particular online re-calibration.

Many sensing devices 12, 14, 16, 18 used in the automotive sector, e. g. camera device 16, lidar device 12, radar device 14, ultrasound device 18, have a framebased evaluation of data. Since the sensing devices 12, 14, 16, 18 on the vehicle 10 are exposed to frequently changing conditions and also to aging and to a wide range of temperatures, these may affect the performance of the sensing devices 12, 14, 16, 18. These effects may be predicted and compensated for by adjusting parameters of the sensing devices 12, 14, 16, 18. Additionally or alternatively, recalibration of the sensing devices 12, 14, 16, 18 is advantageous, because the prediction of aging processes may be complex. The use of vehicles 10 may also be highly variable, e. g. in terms of usage, ambient temperatures etc. Re-calibration may allow to take this into account, without knowing the exact circumstances of use of each particular vehicle 10 in advance. Re-calibration of properties like signal gain, white balance, noise values is therefore beneficial, especially for safety-relevant functions.

The method comprises:

A) comprises capturing at least one frame of the environment with a first set of parameters. The sensing device 12, 14, 16, 18 is in normal operation. It captures the sensing area 22 with its normal settings.

B) is a calibration mode with changed settings of the sensing device 12, 14, 16, 18. B) comprises capturing at least one frame of the environment with a second set of parameters. The changes from the first to the second set of parameters are done to capture calibration data. The calibration data may be reference data or data to check or optimize the former settings with the first set of parameters for deviations.

A third set of parameters is determined during B). The third set of parameters depends on the calibration data obtained by capturing the at least one frame of the environment with the second set of parameters. C) comprises capturing at least one frame of the environment with the first set of parameters. In C) the sensing device 12, 14, 16, 18 is back in normal operation and capturing with normal settings, which are the same as in A).

D) comprises comparing the at least one frame captured in A) with the at least one frame captured in C). The comparison in D) comprises that the measurement data of the captured data in A) and C) are compared with each other.

D) further comprises detecting if the environment captured during A) is similar to the environment captured during C). If the data captured in A) and C) are the same or have only very small deviations, it can be assumed that the environment has not changed during calibration operation.

E) comprises adjusting the first set of parameters depending on the outcome of

D).

E) comprises in particular:

E.l) updating the first set of parameters to be equal to the third set of parameters if the at least one frame captured in A) is similar to the at least one frame captured in C). This corresponds to the path marked with "+" in Fig. 1. or

E.2) keeping the first set of parameters if the at least one frame captured in A) is not similar to the at least one frame captured in C). This corresponds to the path marked with in Fig. 1. This means that the calibration data captured in B) are discarded.

After E) the normal operation will resumed with either the adjusted first set of parameters, which corresponds to optimized settings based on optimization from B), or the initial settings from A).

It is further possible to check if the vehicle is in standstill before starting the described method.

Fig. 2 schematically shows a vehicle 10, for example a passenger car. A plurality of sensing devices 12, 14, 15, 18 is arranged in a front area of the vehicle 10.

A lidar device 12 comprises an optical transmission device, an optical reception device and a processor. In the processor of the lidar device 12, transmitted and received optical signals Si may be evaluated e. g. using time-of-flight measurements. The evaluation may serve to detect objects 0 in the sensing area 22. The processor may also be used to monitor and control the transmitting process in the optical transmission device, the receiving process in the optical reception device.

In the example shown, the sensing area 22 of the lidar device 12 is located in front of the vehicle 10. Thus, in the example shown, an area in front of the vehicle 10 in the direction of travel can be monitored by the lidar device 12. There are also lidar devices possible for other parts of the vehicle 10, e. g. for surround-view functions such as at the sides and/or rear of the vehicle 10. It is also possible to arrange several lidar devices 12 on the vehicle 10, in particular also in corner areas of the vehicle 10.

A radar device 14 comprises a radar transmission device, a radar reception device and a processor. The processor of the radar device 14 is configured to evaluate the transmitted and received radar signals Si. From the evaluation, object O detection and ranging may be performed. The sensing area 22 of the radar device 14 may be similar to the sensing area 22 of the lidar device 12 or it may be different. In low-visibility conditions and for velocity estimation, the lidar device 12 may be enhanced by data provided by the radar device 14. Also, the calibration of the lidar device 12 may be supported by sensor data provided by the radar device 14 and vice versa.

A camera device 16 may also be located in the front area of the vehicle 10 to capture optical signals Si in the sensing area 22. The camera device 16 captures images of the sensing area 22. The sensing area 22 of the camera may be similar or different from the sensing area 22 of the lidar device 12 and/or the radar device 14. Also, the calibration of the camera 16 may be supported by sensor data provided by the lidar device 12 and/or the radar device 14 and vice versa.

The calibration of the devices 12, 14, 16 may be enhanced by data captured by an ultrasound device 18 and vice versa. The ultrasound device sends and receives ultrasound signals Si to obtain information in the surroundings of the vehicle 10.

The radar sensing device 14, the camera device 16 and/or the ultrasound device 18 may also be located in other areas of the vehicle 10. There may be more than one of the radar sensing device 14, the camera device 16 and/or the ultrasound device 18 located on the vehicle 10.

The sensing devices 12, 14, 16, 18 can be used to detect stationary or moving objects O in their respective sensing area 22. Objects O may be vehicles, persons, animals, plants, obstacles, roadway unevenness, like e. g. potholes or stones, roadway boundaries, traffic signs, open spaces, in particular parking spaces, precipitation, or the like.

Each of the sensing devices 12, 14, 18 may comprise a processor configured to evaluate the sent and received signals Si to perform the detection of objects 0 in the respective sensing area 22 and to perform monitoring of the respective sensing area 22. The camera device 16 may also comprise a processor configured to perform the image acquisition and/or to perform monitoring of the sensing area 22 of the camera 16.

The method to calibrate one or more frame-based vehicle sensing devices 12, 14, 16, 18 may be performed on the respective devices 12, 14, 16, 18 with their respective processor. The method may also be performed by one or more of the devices 12, 14, 16, 18 and a central controller 20. The central controller 22 may also be called central ECU (electronic control unit), zonal controller and/or domain controller. The central controller 20 may be configured to receive sensor data from the plurality of sensing devices 12, 14, 16, 18 and other sensors and to control actuators in the vehicle 10. The central controller 20 may in particular comprise program code which, when executed on the central controller 20, controls the vehicle 10 to perform autonomous or semi-autonomous driving. For such functions, calibration and in particular online calibration of the sensing devices 12, 14, 16, 18 to be able to rely on high quality sensor data.

Claims

1. Method to calibrate at least one frame-based vehicle sensing device (12, 14, 16, 18), wherein the at least one sensing device (12, 14, 16, 18) is configured to frame-wise capture the vehicle's (10) environment depending on a first set of parameters, wherein the calibration is performed by adjusting the first set of parameters, the method comprising

E) adjusting the first set of parameters depending on the outcome of D).

2. Method according to claim 1, wherein D) comprises detecting if the environment captured during A) is similar to the environment captured during C).

3. Method according to claim 1 or 2, wherein A) and C) are performed by the at least one vehicle sensing device (12, 14, 16, 18) and E) comprises

4. Method according to one of the preceding claims, wherein the sensing device is configured to provide sensor data depending on the vehicle's (10) environment captured depending on the first set of parameters.

5. Method according to one of the preceding claims, further comprising detecting a stationary condition of the vehicle (10) and performing A) to E) during the stationary condition.

6. Method according to one of the preceding claims, wherein the calibration data comprises reference data and/or tentative captured data.

7. Method according to one of the preceding claims, wherein the at least one sensing device (12, 14, 16, 18) is at least one of a lidar device (12), a radar device (14), a camera device (16) or an ultrasound device (18).

8. Method according to one of the preceding claims, wherein the calibration data is obtained by capturing the at least one frame of the environment with the second set of parameters using a further sensing device (12, 14, 16, 18).

9. Method according to one of the preceding claims, wherein B) comprises obtaining calibration data by capturing multiple frames of the environment depending on multiple second sets of parameters.

10. Method according to claim 9, wherein the multiple frames of the environment are captured using multiple sensing device (12, 14, 16, 18).

11. System to calibrate at least one frame-based vehicle sensing device, wherein the at least one sensing device (12, 14, 16, 18) is configured to frame-wise capture the vehicle's (10) environment depending on a first set of parameters, wherein a processing device is configured to perform the calibration by adjusting the first set of parameters, wherein the at least one sensing device is configured to capture at least one frame of the environment with the first set of parameters before and after capturing at least one frame of the environment with a second set of parameters, wherein the processing device is configured to determine a third set of parameters depending on calibration data obtained by capturing the at least one frame of the environment with the second set of parameters, and to compare the frames with one another, that have been captured with the first set of parameters before and after capturing the at least one frame of the environment with the second set of parameters, wherein the processing device is further configured to adjust the first set of parameters depending on the outcome of the comparison.

12. System according to claim 11, wherein the first set of parameters is updated to be equal to the third set of parameters if the compared frames are similar or wherein the first set of parameters is kept if the compared frames are not similar.

13. System according to claim 11 or 12, wherein the at least one sensing device (12, 14, 16, 18) is at least one of a lidar device (12), a radar device (14), a camera device (16) or an ultrasound device (18).

14. System according to one of claims 11 to 13, wherein the processing device is comprised in the at least one sensing device (12, 14, 16, 18) or wherein the processing device (12, 14, 16, 18) is comprised in a central controller (20) of the vehicle (10).

15. Vehicle comprising a system according to one of claims 11 to 14.