EP4634692A1 - Method to recalibrate a sensor of a device - Google Patents

Method to recalibrate a sensor of a device

Info

Publication number
EP4634692A1
EP4634692A1 EP23833332.2A EP23833332A EP4634692A1 EP 4634692 A1 EP4634692 A1 EP 4634692A1 EP 23833332 A EP23833332 A EP 23833332A EP 4634692 A1 EP4634692 A1 EP 4634692A1
Authority
EP
European Patent Office
Prior art keywords
sensor
function
recalibration
information
control unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP23833332.2A
Other languages
German (de)
French (fr)
Inventor
Filip Kozak
Mauricio Gruenwaldt
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Valeo Schalter und Sensoren GmbH
Original Assignee
Valeo Schalter und Sensoren GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Valeo Schalter und Sensoren GmbH filed Critical Valeo Schalter und Sensoren GmbH
Publication of EP4634692A1 publication Critical patent/EP4634692A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/497—Means for monitoring or calibrating
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/06—Systems determining position data of a target
    • G01S13/42—Simultaneous measurement of distance and other co-ordinates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/881—Radar or analogous systems specially adapted for specific applications for robotics
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/02—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems using reflection of acoustic waves
    • G01S15/06—Systems determining the position data of a target
    • G01S15/08—Systems for measuring distance only
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/86—Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/88—Sonar systems specially adapted for specific applications
    • G01S15/93—Sonar systems specially adapted for specific applications for anti-collision purposes
    • G01S15/931—Sonar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/42—Simultaneous measurement of distance and other co-ordinates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/003—Transmission of data between radar, sonar or lidar systems and remote stations
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/40—Means for monitoring or calibrating
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/52—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S15/00
    • G01S7/52004—Means for monitoring or calibrating
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/52—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S15/00
    • G01S7/52004—Means for monitoring or calibrating
    • G01S7/52006—Means for monitoring or calibrating with provision for compensating the effects of temperature
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/867—Combination of radar systems with cameras
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/497—Means for monitoring or calibrating
    • G01S2007/4975—Means for monitoring or calibrating of sensor obstruction by, e.g. dirt- or ice-coating, e.g. by reflection measurement on front-screen
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/497—Means for monitoring or calibrating
    • G01S2007/4975—Means for monitoring or calibrating of sensor obstruction by, e.g. dirt- or ice-coating, e.g. by reflection measurement on front-screen
    • G01S2007/4977—Means for monitoring or calibrating of sensor obstruction by, e.g. dirt- or ice-coating, e.g. by reflection measurement on front-screen including means to prevent or remove the obstruction
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G01S2013/9323—Alternative operation using light waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G01S2013/9324—Alternative operation using ultrasonic waves

Definitions

  • the invention relates to a method to calibrate a sensor of a device.
  • the device is configured to provide a function, in particular an at least semi-automatic function.
  • the invention furthermore relates to a device, a vehicle and a computer program product to perform such a method.
  • a device configured to perform a function such as an at least semi-automatic and, in particular, a fully automatic function can be organized according to a sense-plan-act model.
  • This model is typical for applications in robotics and can be applied, for example, to an at least semi-automatically operated vehicle and/or a manufacturing robot.
  • the device captures information about its environment.
  • a plan stage it determines a strategy to perform a given task, such as fully automatically operate the vehicle.
  • the device completes the task according to the defined strategy meaning that it takes actions such as fully automatically operating the vehicle according to the strategy.
  • the sense-plan-act model is, for example, described in document WO 2020/205648 A1 .
  • the device may face a situation in which recalibration of a sensor is necessary, wherein the sensor is required for the sense stage.
  • the recalibration of the sensor during performing the function may lead to inaccuracies in the plan stage, in particular if the plan stage requires continuously sensor information provided by the sensor. It is hence necessary to ensure that the recalibration of the sensor is performed at a reasonable time.
  • US 2018/0307238 A1 discloses a system and a method for calibration validation for autonomous vehicle operations.
  • a controller obtains an image during a stationary condition, identifies a reference position for reference objects during the stationary condition, identifies a reference object at a second position within the image and validates transformations associated with an imaging device based on a difference between the reference position and the second position.
  • US 2020/0353939 A1 provides a system and method for recalibration of an uncalibrated sensor.
  • the system comprises an error handling system configured to determine whether to perform a recalibration.
  • the error handling system comprises a recalibration engine configured to perform the recalibration. It is the object of the invention to provide a recalibration of a sensor at a reasonable point in time.
  • a first aspect of the invention relates to a method to recalibrate a sensor of a device.
  • the device is configured to provide a function.
  • the function is, for example, an at least semiautomatic and, in particular, a fully automatic function.
  • the device may be a vehicle, such as a motor vehicle.
  • the function may then be configured to perform steering, driving and/or braking of the vehicle.
  • the function may be a driver assistance function, such as a park assist or a lane assist.
  • the sensor is preferably a sensor configured to capture an environment of the device.
  • the sensor is hence, for example, a camera device comprising at least one camera, a radar device, a lidar device, an ultrasonic sensor and/or a laser distance sensor.
  • the senor was calibrated at least once before a current point in time. However, over time or due to external factors such as dirt, a temperature change and/or precipitation a recalibration of the sensor may be reasonable or necessary to maintain or regain a quality of sensor information captured by the sensor.
  • the method comprises providing a sensor information by means of the sensor of the device.
  • the sensor information describes the environment of the device.
  • the sensor information is provided to a control unit of the device.
  • the control unit is, for example, a computer.
  • the sensor information may be described by sensor data. If the sensor is, for example, the camera, it provides camera data as sensor information.
  • the environment of the device is spatially defined by a coverage area of the sensor.
  • the sensor captures the sensor information. It is possible that the sensor is not part of the device and hence, for example, an external sensor. In this case, the external sensor provides the sensor information to the control unit via a communication connection between the sensor and the device.
  • Providing the sensor information may represent a sense stage of a senseplan-act model.
  • the method comprises executing a function of the device based on the provided sensor information. This is done by means of the control unit.
  • the function is thus currently performed by the device.
  • the function is in particular the at least semi-automatic function.
  • the function may be the fully automatic function.
  • the function is based on the provided sensor information. Executing the function based on the sensor information may represent at least a plan stage and in particular as well the act stage of the sense-plan-act model.
  • the method comprises providing to the sensor a function information describing the executed function. This is done by means of the control unit. In other words, the control unit provides information about the function that is currently executed by the device to the sensor. This information is referred to as the function information.
  • the function information may describe the executed function in detail. It may hence comprise what kind of function is executed.
  • the method comprises determining a recalibration information by means of the sensor.
  • the recalibration information describes a request for recalibration of the sensor.
  • the recalibration information is determined based on the provided function information.
  • the recalibration information may describe, if recalibration is necessary or not. It is possible, that the recalibration information comprises more detail, for example, with regard to an urgency of the recalibration, a kind of recalibration that is requested and/or an approximated duration and/or energy requirement to perform the requested calibration.
  • the sensor then provides the determined recalibration information to the control unit. This means that the method comprises transmitting the determined recalibration information from the sensor to the control unit.
  • the sensor provides the sensor information and the recalibration information to the control unit and the control unit provides the function information to the sensor.
  • the described steps are performed one after another meaning that first the sensor information is provided, afterwards the function information is provided and after that the recalibration information is provided at a point in time, when recalibration is requested.
  • the sensor information and the function information are exchanged between the control unit and the sensor continuously. It is also possible that the control unit determines control commands to execute the function and provides these commands for an executing unit of the device that is configured to execute the function.
  • the communication connection or information exchange between the control unit and the sensor may be provided by a cable or as a wireless link, for example via a wireless local area network (WLAN), a Bluetooth link, and/or a mobile data network, for example based on the Long Term Evolution (LTE), Long Term Evolution Advanced (LTE-A), Fifth Generation (5G), or Sixth Generation (6G) cellular standard.
  • WLAN wireless local area network
  • LTE-A Long Term Evolution Advanced
  • 5G Fifth Generation
  • 6G Sixth Generation
  • the method comprises performing at least one preparation action to prepare for recalibration of the sensor according to the provided recalibration information. This is done by means of the control unit. After receiving the recalibration information, the control unit may take action and carries out preparatory measures to get the device ready for the expected recalibration of the sensor.
  • the preparation action may comprise, for example, activating a redundant sensor that may provide the sensor information required for executing the function during the recalibration of the sensor that requires recalibration.
  • the redundant sensor is configured to provide sensor information that may replace the sensor information provided by the sensor.
  • the preparation action may comprise transferring control of the device to a user. There are multiple possible preparation actions.
  • the preparation action may be device-dependent meaning that a vehicle as device may require another preparation action compared to, for example, a manufacturing robot as device.
  • the method comprises recalibrating the sensor.
  • the sensor preferably recalibrates itself.
  • an additional component of the device is provided and configured to perform the recalibration of the sensor. Recalibration may be done by performing a recalibration algorithm, for example, to auto-calibrate the sensor. In this way, recalibration of an ultrasonic sensor that depends on temperature is automatically possible. If, for example, recalibration is needed due to dirt on a lens of a camera or changing light conditions in the environment, an automatic lens cleaner may be activated or an exposure setting of the camera may be adapted.
  • feedback is provided from the plan stage to the sense stage.
  • This is achieved by providing the function information to the sensor.
  • the plan stage meaning the control unit, informs the sense stage, meaning the sensor, about a running function.
  • the information is provided in order to allow the sensor recalibration at the most suitable time. Therefore, the recalibration information is determined based on the provided function information.
  • the function information hence influences whether and when recalibration of the sensor is necessary. By doing so it is, for example, possible for the sensor to add an extra time window between receiving the function information and providing the recalibration information. Therefore, it is possible to have the recalibration dependent on the executed function and hence depending on the plan stage and not just on the needs of the sense stage. The method thus allows recalibration of the sensor at a reasonable point in time.
  • the method comprises revocation of the performed at least one preparation action by means of the control unit.
  • the recalibration there is no more need for the performed preparation action because the preparation action was only intended to bridge a time without provided sensor information during the recalibration of the sensor.
  • the user had control over the device because the preparation action comprised transferring control of the device to the user, this preparation action is determined and control over the device is transferred back to the control unit of the device. The function may hence take over control again.
  • the redundant sensor is activated during the recalibration due to preparation action, this preparation action is terminated by switching off or at least pausing the redundant sensor.
  • the method comprises returning the device to its original state before the preparation action had been performed.
  • revoking the preparation action preferably all preparation action is taken back.
  • the method is hence particularly comfortable for a user because after, for example, a short recalibration time of a few seconds or minutes the device is automatically reset to its original state.
  • a further embodiment comprises that the function information describes if the function is a safety-critical function or a non-safety-critical function of the device.
  • the safety-critical or non-safety-critical function may alternatively be referred to as safety relevant or safety irrelevant function, respectively.
  • Safety-critical may be a function that relates to driving, steering and/or braking the device.
  • a non-safety-critical function is, for example, a warning function that has no impact on a driving system, steering system and/or braking system of the device.
  • to assist with parking the vehicle as device may provide a distance warning system that typically outputs sounds depending on a distance between the vehicle and at least one object in the environment of the vehicle.
  • Such a function has no impact on the driving system, steering system and/or braking system of the vehicle but only informs a driver of the vehicle about a possible collision with the object.
  • the distance warning system is therefore a non-safety-critical function, because it does not contribute to an at least semi-automatic operation of the vehicle. Dividing possible functions in safety-critical or non-safety-critical function allows to quickly and reliably determine if recalibration of the sensor has an impact on the current operation of the device or not.
  • the method comprises that if the function information describes, that the function is the non-safety-critical function the at least one preparation action is postponed.
  • the function is not safety-critical and hence for example the above-described distance warning system the required recalibration is not performed immediately but performing the preparation action is on hold. It is hereby postponed by a predefined time window and/or until a current activity of the function is terminated.
  • the time window is a given time interval.
  • the time window may be 1 second, 3 seconds, 5 seconds, 10 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, 15 minutes, 30 minutes or in particular 1 hour long.
  • recalibration is deferred until the function has finished a current activity.
  • the sensor may wait until the vehicle has reached a parking position so that the distance warning system is no longer supporting the driver.
  • the parking position is reached, for example, as soon as a parking brake of the vehicle is activated, the preparation action is performed and recalibration is started because availability of the distance warning system is currently not required.
  • the benefit of this is that the driver of the vehicle can use the service of the distance warning system to finish his or her current driving task although the sensor providing the sensor information on which the function is based needs recalibration. This makes the method particularly comfortable because it does not interfere with ongoing activities and hence ongoing functions.
  • a further embodiment comprises that the recalibration information describes if the recalibration of the sensor is mandatory or optional.
  • the recalibration information thus at least differentiates between a recalibration that is needed immediately and therefore mandatory and a recalibration that could be postponed because it is not necessary at the moment and therefore an optional recalibration.
  • the recalibration information that describes the mandatory recalibration can be referred to as mandatory recalibration request.
  • the recalibration information that describes the optional recalibration may be referred to as initial recalibration request.
  • the recalibration is optional whenever a measurement accuracy of the sensor is still sufficiently good in quality to perform the function. However, recalibration would increase the quality of the sensor information.
  • the recalibration is mandatory whenever the sensor information is not sufficiently reliable anymore to perform the function of the device. It is, for example, not reasonable to postpone a mandatory recalibration, in particular, if the function is safety-critical according to the function information. In this case, there is an urgent need for recalibration according to the respective recalibration information. It is hence possible to easily determine an urgency of recalibration.
  • an embodiment comprises that if the function information describes that the function is a safety-critical function and if the recalibration information describes that recalibration of the sensor is optional, the at least one preparation action is at least temporarily suppressed. Due to the safety-critical function recalibration is of relatively high interest. However, if the sensor classifies the recalibration as optional and not mandatory, it is reasonable to wait for a moment that suits recalibration. Although the function is safety-critical, there are hence still cases in which postponing the recalibration by suppressing it are reasonable. Therefore, not all safety-critical functions result in immediate preparation action but it is distinguished whether the recalibration is mandatory or only optional. The method is hence particularly adapted to current circumstances.
  • a further embodiment comprises that the suppressed at least one preparation action is performed when and hence as soon as the function is terminated.
  • the optional recalibration information it is hence possible to wait until the current function is terminated, in particular, the action of the function. This contributes to finding an ideal point in time to perform the recalibration.
  • a further embodiment comprises that if the function information describes that the function is a safety-critical function and if the recalibration information describes that recalibration of the sensor is mandatory, the at least one preparation action is performed immediately. In case the mandatory recalibration request was received, the method hence prepares and executes recalibration of the sensor without delay because in this case recalibration should not wait. The method thus allows to easily determine the case when immediate recalibration is reasonable.
  • the preparation action comprises at least a transition of the device into a minimal risk condition.
  • the minimal risk condition may be a safe state.
  • the device is at standstill when the minimal risk condition is reached. Preferably, it is then also held at standstill.
  • automatic or manual stopping the vehicle may be performed as preparation action. This means that the vehicle may be decelerated until it stands and then the parking brake of the vehicle is activated. Due to the safety-critical function, it is reasonable to stop the device and to keep it stopped while recalibrating the sensor. It is particularly reasonable to perform a revocation of the preparation action after the recalibration, for example by restarting the device and in particular accelerating the vehicle.
  • a further embodiment comprises that if the device does not execute the function, the sensor recalibrates without performing the preparation action. If there is hence no function active, the recalibration can be triggered even, for example, if only it is only optional. The reason for this is that the recalibration has no significant impact on the operation of the device because no function that depends on sensor information provided by the sensor is active.
  • No function running means that the control unit (plan stage) sends no control command to the act stage that depends on the sensor (sense stage). The sense stage has hence no impact on the plan and act stage and hence recalibration can be performed at any time, in particular, immediately.
  • the method hence also considers situations in which no function is activated in the device.
  • the preparation action comprises at least one of the following actions: Transferring control of the device to a user of the device; providing the sensor information by means of a redundant sensor to the sensor and/or predicting the sensor information by applying a prediction algorithm on the provided sensor information.
  • Transferring control to the user of the device means that an at least semi-automatic mode of the device is switched to a manual mode in which the user at least partially, in particular, full control of the device.
  • the redundant sensor may be an alternative sensor that preferably provides equivalent sensor information compared to the sensor information provided by the sensor.
  • a radar device may, for example, be a redundant sensor for an ultrasonic sensor or vice versa.
  • the prediction algorithm is, for example, based on a mathematical model to estimate the sensor information during the recalibration when no or at least no reliable sensor information is provided by the sensor. This means that the sensor is for example switched off or at least paused during the recalibration but the sensor information to be expected from the sensor is calculated by applying the prediction algorithm.
  • the predicted sensor information is hence no real measurement data of the sensor but may imitate it. There are hence multiple and versatile preparation actions possible. This means that precautions are taken to compensate for the lack of sensor information when it is recalibrated.
  • the device comprises a control unit and a sensor.
  • the device is configured to perform or execute a method as described above.
  • the device performs the method.
  • a preferred embodiment of the device comprises that the device is a vehicle, in particular a motor vehicle.
  • the motor vehicle may be a passenger car, a truck, a bus and/or a motorcycle.
  • the device is a robot, in particular a manufacturing robot.
  • any device that may provide an, in particular, at least semi-automatic function for driving, steering and/or braking of the device depending on sensor information describing an environment of the device can be regarded as possible device to perform the described method.
  • a further aspect relates to a vehicle.
  • the vehicle comprises a control unit and a sensor.
  • the vehicle is preferably a motor vehicle as described above.
  • the vehicle is configured to perform the described method.
  • the vehicle performs the described method.
  • the sensor may comprise a computer to perform the steps of the method intended to be performed by the sensor
  • the control unit and the sensor are each configured to perform the steps for the sensor or the control unit described above. They perform the described steps, in particular, according to at least one of the embodiments or a combination of embodiments of the described method.
  • the control unit and the sensor comprise a processor device.
  • the processor device may comprise at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (field programmable gate array) and/or at least one DSP (digital signal processor).
  • the processor device may comprise program code, which may alternatively be referred to as a computer program product or computer program. The program code may be stored in a data memory of the processor device.
  • the invention furthermore relates to a computer program product comprising instructions which, when the program is executed by the control unit and the sensor, cause them to carry out the method as described above.
  • the computer program product can be referred to as computer program.
  • An information in the sense of the invention can be described by respective data.
  • the sensor information can be described by sensor data. This applies analogously to all information mentioned in the application.
  • Fig. 1 a schematic representation of a vehicle as a device
  • Fig. 2 a schematic representation of a robot as device
  • Fig. 3 a schematic representation of a method to recalibrate a sensor of a device
  • Fig. 4 a schematic representation of different scenarios of a method according to Fig. 3.
  • Fig. 1 shows a device 1 .
  • the device 1 is here a vehicle 2.
  • the device 1 comprises a control unit 3 and at least one sensor 4.
  • the sensor 4 is here a camera 5, in particular, a front camera 5, and/or an ultrasonic sensor 6.
  • the ultrasonic sensor 6 is here arranged in a door of the vehicle 2.
  • Other kinds of sensors 4 and/or sensor 4 at other positions in the vehicle 2 are possible.
  • the communication connections 7 may be wired or wireless.
  • the vehicle 2 comprises a steering system 8, a braking system 9 and a drive system 10.
  • the vehicle 2 is configures to be at least semi-automatically operated by a function 21 (see reference sign 21 in Fig. 3) of the vehicle 2.
  • the function may provide operating commands for the steering system 8, the braking system 9 and/or the drive system 10.
  • the operating commands may be provided by the control unit 3.
  • Fig. 2 shows a manufacturing robot 11 as device 1 .
  • the robot 11 may be configured to at least partially produce the vehicle 2.
  • the robot 1 1 comprises the control unit 3 as well as at least one sensor 4.
  • it comprises the camera 5 positioned on top of the robot 1 1 as well as the ultrasonic sensor 6.
  • Other and/or more sensors 4 are possible.
  • the robot 11 may comprise multiple movable components 12, which may be referred to as arms of the robot 1 1 .
  • the robot 11 furthermore comprises a gripper arm 13, which may be configured to grab or hold a component which is about to be produced by the robot 11 .
  • the gripper arm 13 may be configured to grab at least a component of the vehicle 2.
  • the vehicle 2 and the robot 11 are two possible examples for the device 1 which is configured to perform a method as sketched in Fig. 3.
  • Fig. 3 shows a method to recalibrate the sensor 4 of the device 1 .
  • the device 1 is configured to provide a function 21 , which may be an at least semi-automatic and, in particular, a fully automatic function.
  • a first step S1 is performed by the sensor 4. It comprises providing a sensor information 20 to the control unit 3 of the device 1 .
  • the sensor information 20 describes the environment of the device 1 . In case of the camera 5, the sensor information 20 may be a camera information captured by the camera 5. In case of the ultrasonic sensor 6, the sensor information 20 may be a distance information and/or a height information with regard to an object in the environment of the device 1 .
  • the sensor information 20 always describes the environment of the device 1 .
  • the sensor information 20 is hence no device-internal measurement data, for example, a temperature value determined by a temperature sensor located in the drive system 10 of the vehicle 2.
  • Step S1 is a sense stage according to a sense-plan-act model.
  • a step S2 is performed by the control unit 3. It comprises executing the function 21 of the device 1 based on the provided sensor information 20. Step S2 can at least be understood as a plan stage according to the sense-plan-act model. Step S2 also comprises providing a function information 22 to the sensor 4. The function information 22 describes the executed function 21 . The function information 22 may describe, if the function 21 is a safety-critical function 23 or a non-safety-critical function 24.
  • a step S3 is performed by the sensor 4. It comprises determining a recalibration information 25.
  • the recalibration information 25 describes a request for recalibration of the sensor 4.
  • the recalibration information 25 is determined based on the provided function information 22. If recalibration is needed or requested hence depends on the provided function information 22. Therefore, it may depend on whether the function 21 is a safety- critical function 23 or a non-safety-critical function 24.
  • the recalibration information 25 is provided to the control unit 3.
  • the recalibration information 25 comprises or describes, if the recalibration of the sensor 4 is mandatory 26 or optional 27.
  • a step S4 is performed by the control unit 3.
  • the preparation action 28 comprises performing at least one preparation action 28 to prepare for recalibration of the sensor 4 according to the provided recalibration information 25.
  • the preparation action 28 may comprise at least one of the following actions: Transferring control of the device 1 to a user 29 of the device 1 ; and/or providing the sensor information 20 by means of a redundant sensor 30, which is a redundant sensor 30 to the sensor 4; and/or predicting the sensor information 20 by applying a prediction algorithm 31 on the provided sensor information 20. There preparation action 28 hence ensures that the function 21 can be continued during the recalibration of the sensor 4.
  • step S5 recalibrating the sensor 4 occurs. This is done after performing the at least one preparation action 28 in step S4.
  • a step S6 it is possible that after recalibrating the sensor 4, the performed at least one preparation action 28 is revoked by means of the control unit 3. This means that the above-described preparation actions 28 are deactivated to transfer the device 1 in an original state it occupied previous to the preparation actions 28.
  • step S1 may be performed again.
  • Fig. 4 shows different combinations of the function information 22 and the recalibration information 25.
  • the function 21 is the nonsafety-critical function 24
  • a step S7 is performed.
  • the preparation action 28 is postponed by a predefined time window and/or until a current activity of the function 21 is terminated.
  • the function information 22 describes that the function 21 is the safety-critical function 23
  • the further steps depend on the recalibration information 25.
  • a step S8 is performed.
  • the step S8 comprises that the at least one preparation action 28 is at least temporarily suppressed.
  • the suppressed at least one preparation action 28 is performed as soon as the function 21 is terminated, more precisely when the current activity of the function is terminated. However, if the recalibration is mandatory 26, the step S4 is performed immediately. After step S7 or step S8, the method continues with step S4 as well.
  • the preparation action 28 comprises transition of the device 1 into a minimal risk condition.
  • the device 1 is preferably at standstill. It is hence possible to stop the vehicle 2 or the robot 11 as preparation action 28.
  • the minimal risk condition may alternatively be referred to as a safe state of the device 1 . If the device 1 does not execute the function 21 and this is determined in a step S9, the sensor 4 recalibrates without performing the preparation actions 28 meaning that step S5 is immediately performed for example after step S3 followed by step S9. In this scenario there is no active function 21 and hence there is no need for considering whether the function 21 is the safety-critical function 23 or the non-safety-critical function 24. Therefore, the recalibration information 25 may not even be determined.
  • an automotive sense-plan-act model for sensor recalibration in the context of autonomous driving level 2+ has been described.
  • the feedback from the plan stage to the sense stage is performed. Therefore, a handshake is made.
  • the plan stage informs about the running function 21 , in particular if it is the non-safety-critical function 24 or the safety- critical function 23, in order to allow the sensor recalibration at the most suitable time.
  • This mechanism allows to postpone or invoke the recalibration based on the needs of the function 21 and/or the sensor 4. If the non-safety-critical function 24 is running, it can take a short time and any recalibration can be postponed with using sensor quality management data.
  • recalibration can be triggered even by initial recalibration request (meaning recalibration is optional 27). If the safety-critical function 23 is running, the initial recalibration request (meaning recalibration is optional 27) can be suppressed, but the mandatory recalibration request (meaning recalibration is mandatory 26) shall execute recalibration of the sensor 4 and shall transition the vehicle 2 into the minimal risk condition.
  • the ultrasonic sensor 6 is dependent on temperature. If a big temperature deviation takes place, for example, due to a change in the weather, a request for the sensor recalibration is necessary or the sensor 4 will measure with less accuracy. If the temperature deviation is big enough, there is a need to recalibrate because the data (sensor information 20) are not reliable at all. The same can happen in a system based on cameras 5 (changing the exposure, lens cleaner, radar-based). For radar, an automatic self-calibration is performed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Acoustics & Sound (AREA)
  • Automation & Control Theory (AREA)
  • Robotics (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)

Abstract

The invention relates to a method to recalibrate a sensor (4) of a device (1). The method comprises: by means of the sensor (4) providing (S1) a sensor information (20) to a control unit (3) of the device (1); by means of the control unit (3) executing (S2) a function (21) of the device (1) based on the provided sensor information (20) and providing to the sensor (4) a function information (22) describing the executed function (21); by means of the sensor (4) determining (S3) a recalibration information (25) describing a request for recalibration of the sensor (4) based on the provided function information (22) and providing it to the control unit (3); by means of the control unit (3) performing (S4) a preparation action (28) to prepare for the recalibration according to the provided recalibration information (25); and after performing the preparation action (28) recalibrating (S5) the sensor (4).

Description

Method to recalibrate a sensor of a device
The invention relates to a method to calibrate a sensor of a device. The device is configured to provide a function, in particular an at least semi-automatic function. The invention furthermore relates to a device, a vehicle and a computer program product to perform such a method.
A device configured to perform a function such as an at least semi-automatic and, in particular, a fully automatic function, can be organized according to a sense-plan-act model. This model is typical for applications in robotics and can be applied, for example, to an at least semi-automatically operated vehicle and/or a manufacturing robot. In a sense stage of the model, the device captures information about its environment. In a plan stage, it determines a strategy to perform a given task, such as fully automatically operate the vehicle. In an act stage, the device completes the task according to the defined strategy meaning that it takes actions such as fully automatically operating the vehicle according to the strategy. The sense-plan-act model is, for example, described in document WO 2020/205648 A1 .
The device may face a situation in which recalibration of a sensor is necessary, wherein the sensor is required for the sense stage. However, the recalibration of the sensor during performing the function may lead to inaccuracies in the plan stage, in particular if the plan stage requires continuously sensor information provided by the sensor. It is hence necessary to ensure that the recalibration of the sensor is performed at a reasonable time.
Besides, methods for recalibration of sensors are known. US 2018/0307238 A1 discloses a system and a method for calibration validation for autonomous vehicle operations. A controller obtains an image during a stationary condition, identifies a reference position for reference objects during the stationary condition, identifies a reference object at a second position within the image and validates transformations associated with an imaging device based on a difference between the reference position and the second position.
US 2020/0353939 A1 provides a system and method for recalibration of an uncalibrated sensor. The system comprises an error handling system configured to determine whether to perform a recalibration. The error handling system comprises a recalibration engine configured to perform the recalibration. It is the object of the invention to provide a recalibration of a sensor at a reasonable point in time.
The independent claims solve the object.
A first aspect of the invention relates to a method to recalibrate a sensor of a device. The device is configured to provide a function. The function is, for example, an at least semiautomatic and, in particular, a fully automatic function. The device may be a vehicle, such as a motor vehicle. The function may then be configured to perform steering, driving and/or braking of the vehicle. The function may be a driver assistance function, such as a park assist or a lane assist. The sensor is preferably a sensor configured to capture an environment of the device. The sensor is hence, for example, a camera device comprising at least one camera, a radar device, a lidar device, an ultrasonic sensor and/or a laser distance sensor. It is assumed that the sensor was calibrated at least once before a current point in time. However, over time or due to external factors such as dirt, a temperature change and/or precipitation a recalibration of the sensor may be reasonable or necessary to maintain or regain a quality of sensor information captured by the sensor.
The method comprises providing a sensor information by means of the sensor of the device. The sensor information describes the environment of the device. The sensor information is provided to a control unit of the device. The control unit is, for example, a computer. The sensor information may be described by sensor data. If the sensor is, for example, the camera, it provides camera data as sensor information. The environment of the device is spatially defined by a coverage area of the sensor. Preferably, the sensor captures the sensor information. It is possible that the sensor is not part of the device and hence, for example, an external sensor. In this case, the external sensor provides the sensor information to the control unit via a communication connection between the sensor and the device. Providing the sensor information may represent a sense stage of a senseplan-act model.
The method comprises executing a function of the device based on the provided sensor information. This is done by means of the control unit. The function is thus currently performed by the device. The function is in particular the at least semi-automatic function. The function may be the fully automatic function. The function is based on the provided sensor information. Executing the function based on the sensor information may represent at least a plan stage and in particular as well the act stage of the sense-plan-act model. The method comprises providing to the sensor a function information describing the executed function. This is done by means of the control unit. In other words, the control unit provides information about the function that is currently executed by the device to the sensor. This information is referred to as the function information. The function information may describe the executed function in detail. It may hence comprise what kind of function is executed.
The method comprises determining a recalibration information by means of the sensor. The recalibration information describes a request for recalibration of the sensor. The recalibration information is determined based on the provided function information. The recalibration information may describe, if recalibration is necessary or not. It is possible, that the recalibration information comprises more detail, for example, with regard to an urgency of the recalibration, a kind of recalibration that is requested and/or an approximated duration and/or energy requirement to perform the requested calibration. The sensor then provides the determined recalibration information to the control unit. This means that the method comprises transmitting the determined recalibration information from the sensor to the control unit.
There is preferably a continuing information exchange between the sensor and the control unit. The sensor provides the sensor information and the recalibration information to the control unit and the control unit provides the function information to the sensor. Preferably, the described steps are performed one after another meaning that first the sensor information is provided, afterwards the function information is provided and after that the recalibration information is provided at a point in time, when recalibration is requested. Preferably, the sensor information and the function information are exchanged between the control unit and the sensor continuously. It is also possible that the control unit determines control commands to execute the function and provides these commands for an executing unit of the device that is configured to execute the function.
The communication connection or information exchange between the control unit and the sensor may be provided by a cable or as a wireless link, for example via a wireless local area network (WLAN), a Bluetooth link, and/or a mobile data network, for example based on the Long Term Evolution (LTE), Long Term Evolution Advanced (LTE-A), Fifth Generation (5G), or Sixth Generation (6G) cellular standard.
The method comprises performing at least one preparation action to prepare for recalibration of the sensor according to the provided recalibration information. This is done by means of the control unit. After receiving the recalibration information, the control unit may take action and carries out preparatory measures to get the device ready for the expected recalibration of the sensor. The preparation action may comprise, for example, activating a redundant sensor that may provide the sensor information required for executing the function during the recalibration of the sensor that requires recalibration. The redundant sensor is configured to provide sensor information that may replace the sensor information provided by the sensor. Alternatively or additionally, the preparation action may comprise transferring control of the device to a user. There are multiple possible preparation actions. The preparation action may be device-dependent meaning that a vehicle as device may require another preparation action compared to, for example, a manufacturing robot as device.
After performing the at least one preparation action, the method comprises recalibrating the sensor. The sensor preferably recalibrates itself. Alternatively, an additional component of the device is provided and configured to perform the recalibration of the sensor. Recalibration may be done by performing a recalibration algorithm, for example, to auto-calibrate the sensor. In this way, recalibration of an ultrasonic sensor that depends on temperature is automatically possible. If, for example, recalibration is needed due to dirt on a lens of a camera or changing light conditions in the environment, an automatic lens cleaner may be activated or an exposure setting of the camera may be adapted.
In summary, feedback is provided from the plan stage to the sense stage. This is achieved by providing the function information to the sensor. This means that the plan stage, meaning the control unit, informs the sense stage, meaning the sensor, about a running function. The information is provided in order to allow the sensor recalibration at the most suitable time. Therefore, the recalibration information is determined based on the provided function information. The function information hence influences whether and when recalibration of the sensor is necessary. By doing so it is, for example, possible for the sensor to add an extra time window between receiving the function information and providing the recalibration information. Therefore, it is possible to have the recalibration dependent on the executed function and hence depending on the plan stage and not just on the needs of the sense stage. The method thus allows recalibration of the sensor at a reasonable point in time.
According to an embodiment, after recalibrating the sensor the method comprises revocation of the performed at least one preparation action by means of the control unit. As soon as the recalibration is done, there is no more need for the performed preparation action because the preparation action was only intended to bridge a time without provided sensor information during the recalibration of the sensor. If, for example, during the recalibration the user had control over the device because the preparation action comprised transferring control of the device to the user, this preparation action is determined and control over the device is transferred back to the control unit of the device. The function may hence take over control again. If, for example, the redundant sensor is activated during the recalibration due to preparation action, this preparation action is terminated by switching off or at least pausing the redundant sensor. This means that after recalibration the method comprises returning the device to its original state before the preparation action had been performed. By revoking the preparation action, preferably all preparation action is taken back. The method is hence particularly comfortable for a user because after, for example, a short recalibration time of a few seconds or minutes the device is automatically reset to its original state.
A further embodiment comprises that the function information describes if the function is a safety-critical function or a non-safety-critical function of the device. The safety-critical or non-safety-critical function may alternatively be referred to as safety relevant or safety irrelevant function, respectively. Safety-critical may be a function that relates to driving, steering and/or braking the device. A non-safety-critical function is, for example, a warning function that has no impact on a driving system, steering system and/or braking system of the device. For example, to assist with parking the vehicle as device may provide a distance warning system that typically outputs sounds depending on a distance between the vehicle and at least one object in the environment of the vehicle. Such a function has no impact on the driving system, steering system and/or braking system of the vehicle but only informs a driver of the vehicle about a possible collision with the object. The distance warning system is therefore a non-safety-critical function, because it does not contribute to an at least semi-automatic operation of the vehicle. Dividing possible functions in safety-critical or non-safety-critical function allows to quickly and reliably determine if recalibration of the sensor has an impact on the current operation of the device or not.
According to another embodiment, the method comprises that if the function information describes, that the function is the non-safety-critical function the at least one preparation action is postponed. This means that if the function is not safety-critical and hence for example the above-described distance warning system the required recalibration is not performed immediately but performing the preparation action is on hold. It is hereby postponed by a predefined time window and/or until a current activity of the function is terminated. The time window is a given time interval. The time window may be 1 second, 3 seconds, 5 seconds, 10 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, 15 minutes, 30 minutes or in particular 1 hour long. Alternatively or additionally, recalibration is deferred until the function has finished a current activity. If the function is, for example, the distance warning system, the sensor may wait until the vehicle has reached a parking position so that the distance warning system is no longer supporting the driver. When the parking position is reached, for example, as soon as a parking brake of the vehicle is activated, the preparation action is performed and recalibration is started because availability of the distance warning system is currently not required. The benefit of this is that the driver of the vehicle can use the service of the distance warning system to finish his or her current driving task although the sensor providing the sensor information on which the function is based needs recalibration. This makes the method particularly comfortable because it does not interfere with ongoing activities and hence ongoing functions.
A further embodiment comprises that the recalibration information describes if the recalibration of the sensor is mandatory or optional. The recalibration information thus at least differentiates between a recalibration that is needed immediately and therefore mandatory and a recalibration that could be postponed because it is not necessary at the moment and therefore an optional recalibration. The recalibration information that describes the mandatory recalibration can be referred to as mandatory recalibration request. The recalibration information that describes the optional recalibration may be referred to as initial recalibration request. The recalibration is optional whenever a measurement accuracy of the sensor is still sufficiently good in quality to perform the function. However, recalibration would increase the quality of the sensor information. However, since it is not absolutely necessary to do the calibration right away, it is possible to postpone recalibration at least for the predefined time window and/or until the current activity or the function is terminated. The recalibration is mandatory whenever the sensor information is not sufficiently reliable anymore to perform the function of the device. It is, for example, not reasonable to postpone a mandatory recalibration, in particular, if the function is safety-critical according to the function information. In this case, there is an urgent need for recalibration according to the respective recalibration information. It is hence possible to easily determine an urgency of recalibration.
Moreover, an embodiment comprises that if the function information describes that the function is a safety-critical function and if the recalibration information describes that recalibration of the sensor is optional, the at least one preparation action is at least temporarily suppressed. Due to the safety-critical function recalibration is of relatively high interest. However, if the sensor classifies the recalibration as optional and not mandatory, it is reasonable to wait for a moment that suits recalibration. Although the function is safety-critical, there are hence still cases in which postponing the recalibration by suppressing it are reasonable. Therefore, not all safety-critical functions result in immediate preparation action but it is distinguished whether the recalibration is mandatory or only optional. The method is hence particularly adapted to current circumstances.
A further embodiment comprises that the suppressed at least one preparation action is performed when and hence as soon as the function is terminated. In the scenario with the safety-critical function but the optional recalibration information, it is hence possible to wait until the current function is terminated, in particular, the action of the function. This contributes to finding an ideal point in time to perform the recalibration.
A further embodiment comprises that if the function information describes that the function is a safety-critical function and if the recalibration information describes that recalibration of the sensor is mandatory, the at least one preparation action is performed immediately. In case the mandatory recalibration request was received, the method hence prepares and executes recalibration of the sensor without delay because in this case recalibration should not wait. The method thus allows to easily determine the case when immediate recalibration is reasonable.
In summary, different combinations of the function information and the recalibration information result in different cases or scenarios regarding how to proceed with performing the preparation action and the following recalibration.
Another embodiment comprises, that the preparation action comprises at least a transition of the device into a minimal risk condition. The minimal risk condition may be a safe state. In particular, the device is at standstill when the minimal risk condition is reached. Preferably, it is then also held at standstill. In case of the vehicle as device, automatic or manual stopping the vehicle may be performed as preparation action. This means that the vehicle may be decelerated until it stands and then the parking brake of the vehicle is activated. Due to the safety-critical function, it is reasonable to stop the device and to keep it stopped while recalibrating the sensor. It is particularly reasonable to perform a revocation of the preparation action after the recalibration, for example by restarting the device and in particular accelerating the vehicle. However, if the recalibration is mandatory extreme measures such as transition into the minimal risk condition are possible to provide sufficiently good conditions for recalibration of the sensor. A further embodiment comprises that if the device does not execute the function, the sensor recalibrates without performing the preparation action. If there is hence no function active, the recalibration can be triggered even, for example, if only it is only optional. The reason for this is that the recalibration has no significant impact on the operation of the device because no function that depends on sensor information provided by the sensor is active. No function running means that the control unit (plan stage) sends no control command to the act stage that depends on the sensor (sense stage). The sense stage has hence no impact on the plan and act stage and hence recalibration can be performed at any time, in particular, immediately. The method hence also considers situations in which no function is activated in the device.
According to another embodiment, the preparation action comprises at least one of the following actions: Transferring control of the device to a user of the device; providing the sensor information by means of a redundant sensor to the sensor and/or predicting the sensor information by applying a prediction algorithm on the provided sensor information. Transferring control to the user of the device means that an at least semi-automatic mode of the device is switched to a manual mode in which the user at least partially, in particular, full control of the device. The redundant sensor may be an alternative sensor that preferably provides equivalent sensor information compared to the sensor information provided by the sensor. A radar device may, for example, be a redundant sensor for an ultrasonic sensor or vice versa. The prediction algorithm is, for example, based on a mathematical model to estimate the sensor information during the recalibration when no or at least no reliable sensor information is provided by the sensor. This means that the sensor is for example switched off or at least paused during the recalibration but the sensor information to be expected from the sensor is calculated by applying the prediction algorithm. The predicted sensor information is hence no real measurement data of the sensor but may imitate it. There are hence multiple and versatile preparation actions possible. This means that precautions are taken to compensate for the lack of sensor information when it is recalibrated.
Another aspect of the invention relates to a device. The device comprises a control unit and a sensor. The device is configured to perform or execute a method as described above. The device performs the method.
A preferred embodiment of the device comprises that the device is a vehicle, in particular a motor vehicle. The motor vehicle may be a passenger car, a truck, a bus and/or a motorcycle. Alternatively or additionally the device is a robot, in particular a manufacturing robot. In general, any device that may provide an, in particular, at least semi-automatic function for driving, steering and/or braking of the device depending on sensor information describing an environment of the device can be regarded as possible device to perform the described method.
A further aspect relates to a vehicle. The vehicle comprises a control unit and a sensor. The vehicle is preferably a motor vehicle as described above. The vehicle is configured to perform the described method. The vehicle performs the described method.
The sensor may comprise a computer to perform the steps of the method intended to be performed by the sensor
The control unit and the sensor are each configured to perform the steps for the sensor or the control unit described above. They perform the described steps, in particular, according to at least one of the embodiments or a combination of embodiments of the described method. The control unit and the sensor comprise a processor device. The processor device may comprise at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (field programmable gate array) and/or at least one DSP (digital signal processor). Furthermore, the processor device may comprise program code, which may alternatively be referred to as a computer program product or computer program. The program code may be stored in a data memory of the processor device.
The invention furthermore relates to a computer program product comprising instructions which, when the program is executed by the control unit and the sensor, cause them to carry out the method as described above. The computer program product can be referred to as computer program.
An information in the sense of the invention can be described by respective data. For example, the sensor information can be described by sensor data. This applies analogously to all information mentioned in the application.
The embodiments described in connection with the method, individually as well as in combination with each other, apply accordingly, as far as applicable, to the inventive device, vehicle and/or computer program product. The invention includes combinations of the described embodiments. Thereby show:
Fig. 1 a schematic representation of a vehicle as a device,
Fig. 2 a schematic representation of a robot as device,
Fig. 3 a schematic representation of a method to recalibrate a sensor of a device and
Fig. 4 a schematic representation of different scenarios of a method according to Fig. 3.
Fig. 1 shows a device 1 . The device 1 is here a vehicle 2. The device 1 comprises a control unit 3 and at least one sensor 4. The sensor 4 is here a camera 5, in particular, a front camera 5, and/or an ultrasonic sensor 6. The ultrasonic sensor 6 is here arranged in a door of the vehicle 2. Other kinds of sensors 4 and/or sensor 4 at other positions in the vehicle 2 are possible. Between each of the sensors 4 and the control unit 3 there are communication connections 7 for information exchange between the respective sensor 4 and the control unit 3. The communication connections 7 may be wired or wireless.
The vehicle 2 comprises a steering system 8, a braking system 9 and a drive system 10. The vehicle 2 is configures to be at least semi-automatically operated by a function 21 (see reference sign 21 in Fig. 3) of the vehicle 2. The function may provide operating commands for the steering system 8, the braking system 9 and/or the drive system 10. The operating commands may be provided by the control unit 3.
Fig. 2 shows a manufacturing robot 11 as device 1 . The robot 11 may be configured to at least partially produce the vehicle 2. The robot 1 1 comprises the control unit 3 as well as at least one sensor 4. Here, it comprises the camera 5 positioned on top of the robot 1 1 as well as the ultrasonic sensor 6. Other and/or more sensors 4 are possible. The robot 11 may comprise multiple movable components 12, which may be referred to as arms of the robot 1 1 . The robot 11 furthermore comprises a gripper arm 13, which may be configured to grab or hold a component which is about to be produced by the robot 11 .
The gripper arm 13 may be configured to grab at least a component of the vehicle 2.
The vehicle 2 and the robot 11 are two possible examples for the device 1 which is configured to perform a method as sketched in Fig. 3.
Fig. 3 shows a method to recalibrate the sensor 4 of the device 1 . The device 1 is configured to provide a function 21 , which may be an at least semi-automatic and, in particular, a fully automatic function. A first step S1 is performed by the sensor 4. It comprises providing a sensor information 20 to the control unit 3 of the device 1 . The sensor information 20 describes the environment of the device 1 . In case of the camera 5, the sensor information 20 may be a camera information captured by the camera 5. In case of the ultrasonic sensor 6, the sensor information 20 may be a distance information and/or a height information with regard to an object in the environment of the device 1 . The sensor information 20 always describes the environment of the device 1 . The sensor information 20 is hence no device-internal measurement data, for example, a temperature value determined by a temperature sensor located in the drive system 10 of the vehicle 2. Step S1 is a sense stage according to a sense-plan-act model.
A step S2 is performed by the control unit 3. It comprises executing the function 21 of the device 1 based on the provided sensor information 20. Step S2 can at least be understood as a plan stage according to the sense-plan-act model. Step S2 also comprises providing a function information 22 to the sensor 4. The function information 22 describes the executed function 21 . The function information 22 may describe, if the function 21 is a safety-critical function 23 or a non-safety-critical function 24.
A step S3 is performed by the sensor 4. It comprises determining a recalibration information 25. The recalibration information 25 describes a request for recalibration of the sensor 4. The recalibration information 25 is determined based on the provided function information 22. If recalibration is needed or requested hence depends on the provided function information 22. Therefore, it may depend on whether the function 21 is a safety- critical function 23 or a non-safety-critical function 24. The recalibration information 25 is provided to the control unit 3. The recalibration information 25 comprises or describes, if the recalibration of the sensor 4 is mandatory 26 or optional 27. A step S4 is performed by the control unit 3. It comprises performing at least one preparation action 28 to prepare for recalibration of the sensor 4 according to the provided recalibration information 25. The preparation action 28 may comprise at least one of the following actions: Transferring control of the device 1 to a user 29 of the device 1 ; and/or providing the sensor information 20 by means of a redundant sensor 30, which is a redundant sensor 30 to the sensor 4; and/or predicting the sensor information 20 by applying a prediction algorithm 31 on the provided sensor information 20. There preparation action 28 hence ensures that the function 21 can be continued during the recalibration of the sensor 4.
In a step S5, recalibrating the sensor 4 occurs. This is done after performing the at least one preparation action 28 in step S4. In a step S6 it is possible that after recalibrating the sensor 4, the performed at least one preparation action 28 is revoked by means of the control unit 3. This means that the above-described preparation actions 28 are deactivated to transfer the device 1 in an original state it occupied previous to the preparation actions 28. After step S6, step S1 may be performed again.
Fig. 4 shows different combinations of the function information 22 and the recalibration information 25. If according to the function information 22, the function 21 is the nonsafety-critical function 24, a step S7 is performed. In step S7, the preparation action 28 is postponed by a predefined time window and/or until a current activity of the function 21 is terminated. If however the function information 22 describes that the function 21 is the safety-critical function 23, the further steps depend on the recalibration information 25. If according to the recalibration information 25 the recalibration is optional 27 a step S8 is performed. The step S8 comprises that the at least one preparation action 28 is at least temporarily suppressed. The suppressed at least one preparation action 28 is performed as soon as the function 21 is terminated, more precisely when the current activity of the function is terminated. However, if the recalibration is mandatory 26, the step S4 is performed immediately. After step S7 or step S8, the method continues with step S4 as well.
It is possible, that the preparation action 28 comprises transition of the device 1 into a minimal risk condition. In the minimal risk condition, the device 1 is preferably at standstill. It is hence possible to stop the vehicle 2 or the robot 11 as preparation action 28. The minimal risk condition may alternatively be referred to as a safe state of the device 1 . If the device 1 does not execute the function 21 and this is determined in a step S9, the sensor 4 recalibrates without performing the preparation actions 28 meaning that step S5 is immediately performed for example after step S3 followed by step S9. In this scenario there is no active function 21 and hence there is no need for considering whether the function 21 is the safety-critical function 23 or the non-safety-critical function 24. Therefore, the recalibration information 25 may not even be determined.
In summary, an automotive sense-plan-act model for sensor recalibration in the context of autonomous driving level 2+ has been described. The feedback from the plan stage to the sense stage is performed. Therefore, a handshake is made. The plan stage informs about the running function 21 , in particular if it is the non-safety-critical function 24 or the safety- critical function 23, in order to allow the sensor recalibration at the most suitable time. This mechanism allows to postpone or invoke the recalibration based on the needs of the function 21 and/or the sensor 4. If the non-safety-critical function 24 is running, it can take a short time and any recalibration can be postponed with using sensor quality management data. If no function 21 is running, the recalibration can be triggered even by initial recalibration request (meaning recalibration is optional 27). If the safety-critical function 23 is running, the initial recalibration request (meaning recalibration is optional 27) can be suppressed, but the mandatory recalibration request (meaning recalibration is mandatory 26) shall execute recalibration of the sensor 4 and shall transition the vehicle 2 into the minimal risk condition.
The ultrasonic sensor 6 is dependent on temperature. If a big temperature deviation takes place, for example, due to a change in the weather, a request for the sensor recalibration is necessary or the sensor 4 will measure with less accuracy. If the temperature deviation is big enough, there is a need to recalibrate because the data (sensor information 20) are not reliable at all. The same can happen in a system based on cameras 5 (changing the exposure, lens cleaner, radar-based). For radar, an automatic self-calibration is performed.
Compared to a normal sense-plan-act model diagram two additional diagram components are added: In the sense stage “recalibration control” is added. In the plan stage “used function status” is added. The used function status gives information on the function 21 to “recalibration control” and “recalibration control” provides it to a sensor measurement unit (meaning to the sensor 4). Therefore, information on the currently active function 21 are provided to the sensor 4 and the recalibration request can be determined under consideration of the currently running function 21 . This improves timing of sensor recalibration significantly.

Claims

Claims
1 . Method to recalibrate a sensor (4) of a device (1 ), wherein the method comprises:
- by means of the sensor (4) providing (S1) a sensor information (20) describing an environment of the device (1) to a control unit (3) of the device (1);
- by means of the control unit (3) executing (S2) a function (21 ) of the device (1 ) based on the provided sensor information (20) and providing to the sensor (4) a function information (22) describing the executed function (21);
- by means of the sensor (4) determining (S3) a recalibration information (25) describing a request for recalibration of the sensor (4) based on the provided function information (22) and providing the determined recalibration information (25) to the control unit (3);
- by means of the control unit (3) performing (S4) at least one preparation action (28) to prepare for recalibration of the sensor (4) according to the provided recalibration information (25); and
- after performing the at least one preparation action (28) recalibrating (S5) the sensor (4).
2. Method according to claim 1 , characterized in that after recalibrating the sensor (4) the method comprises revocation (S6) of the performed at least one preparation action (28) by means of the control unit (3).
3. Method according to any one of the preceding claims, characterized in that the function information (22) describes if the function (21 ) is a safety-critical function (23) or a non-safety-critical function (24) of the device (1).
4. Method according to claim 3, characterized in that if the function information (22) describes that the function (21) is the non-safety-critical function (24) performing the at least one preparation action (28) is postponed (S7) by a predefined time window and/or until a current activity of the function (21 ) is terminated.
5. Method according to any one of the preceding claims, characterized in that the recalibration information (25) describes if the recalibration of the sensor (4) is mandatory (26) or optional (27).
6. Method according to claim 3 or 4 and to claim 5, characterized in that if the function information (22) describes that the function (21) is the safety-critical function (23) and if the recalibration information (25) describes that recalibration of the sensor (4) is optional (27), the at least one preparation action (28) is at least temporarily suppressed (S8).
7. Method according to claim 6, characterized in that the suppressed at least one preparation action (28) is performed when the function (21) is terminated.
8. Method according to claim 3 or 4 and to any one of claims 5 to 7, characterized in that if the function information (22) describes that the function (21) is the safety- critical function (23) and if the recalibration information (25) describes that recalibration of the sensor (4) is mandatory (26), the at least one preparation action (28) is performed immediately (S4).
9. Method according to any one of the preceding claims, characterized in that the preparation action (28) comprises transition of the device (1) into a minimal risk condition, in which, in particular, the device (1) is at standstill.
10. Method according to any one of the preceding claims, characterized in that if the device (1) does not execute (S9) the function (21), the sensor (4) recalibrates without performing the preparation action (28).
11 . Method according to any one of the preceding claims, characterized in that the preparation action (28) comprises at least one of the following actions:
- transferring control of the device (1 ) to a user (29) of the device (1);
- providing the sensor information (20) by means of a redundant sensor (30) to the sensor (4); and/or
- predicting the sensor information (20) by applying a prediction algorithm (31) on the provided sensor information (20).
12. Device (1 ) with a control unit (3) and a sensor (4), characterized in that the device
(1) is configured to execute a method according to any one of the preceding claims.
13. Device (1 ) according to claim 12, characterized in that the device (1 ) is a vehicle (2), in particular a motor vehicle, and/or a robot (11), in particular a manufacturing robot.
14. Vehicle (2) with a control unit (3) and a sensor (4), characterized in that the vehicle
(2) is configured to perform a method according to any one of claims 1 to 11 .
15. Computer program product comprising instructions which, when the program is executed by a computer of a device (1 ), cause the computer to carry out a method according to any one of claims 1 to 11 .
EP23833332.2A 2022-12-15 2023-12-14 Method to recalibrate a sensor of a device Pending EP4634692A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102022133370.1A DE102022133370A1 (en) 2022-12-15 2022-12-15 Method for recalibrating a sensor of a device
PCT/EP2023/085734 WO2024126638A1 (en) 2022-12-15 2023-12-14 Method to recalibrate a sensor of a device

Publications (1)

Publication Number Publication Date
EP4634692A1 true EP4634692A1 (en) 2025-10-22

Family

ID=89430347

Family Applications (1)

Application Number Title Priority Date Filing Date
EP23833332.2A Pending EP4634692A1 (en) 2022-12-15 2023-12-14 Method to recalibrate a sensor of a device

Country Status (5)

Country Link
EP (1) EP4634692A1 (en)
KR (1) KR20250121115A (en)
CN (1) CN120283171A (en)
DE (1) DE102022133370A1 (en)
WO (1) WO2024126638A1 (en)

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9389295B2 (en) 2013-09-10 2016-07-12 Fisher Controls International Llc Hall effect sensor system with diagnostic capabilities
DE102016226312A1 (en) * 2016-12-29 2018-07-05 Robert Bosch Gmbh Method for operating a driver assistance system for motor vehicles
US10268203B2 (en) 2017-04-20 2019-04-23 GM Global Technology Operations LLC Calibration validation for autonomous vehicle operations
US11415683B2 (en) * 2017-12-28 2022-08-16 Lyft, Inc. Mobile sensor calibration
KR20210134634A (en) 2019-03-29 2021-11-10 인텔 코포레이션 autonomous vehicle system
US11117591B2 (en) 2019-05-08 2021-09-14 Pony Ai Inc. System and method for recalibration of an uncalibrated sensor
US11852730B2 (en) * 2019-12-09 2023-12-26 Plusai, Inc. System and method for collaborative calibration via landmark

Also Published As

Publication number Publication date
DE102022133370A1 (en) 2024-06-20
WO2024126638A1 (en) 2024-06-20
KR20250121115A (en) 2025-08-11
CN120283171A (en) 2025-07-08

Similar Documents

Publication Publication Date Title
JP6786013B2 (en) Update control device, update control system and update control method
US20200086852A1 (en) Vehicle remote parking assist systems and methods
CN108463845B (en) Method for providing outcome data relevant to a motor vehicle environment
JP7658416B2 (en) Control device, method and program
US11954476B2 (en) On-board update apparatus, update processing system, update processing method, and computer program
US20130103246A1 (en) Parking system having longitudinal and transverse guidance
KR102478809B1 (en) Method for invoking a teleoperated driving session, apparatus for performing the steps of the method, vehicle and computer program
US11524694B2 (en) Vehicle control apparatus, vehicle, vehicle control method, and non-transitory computer-readable storage medium
CN113859225A (en) Remote control parking method and device
CN112141086A (en) Parking path correction method, device and computer-readable storage medium
US20170076513A1 (en) Method and device for determining whether a motor vehicle is instantaneously driven manually or automatically
CN114746920B (en) Vehicle control device and vehicle control method
WO2024126638A1 (en) Method to recalibrate a sensor of a device
JP2020040517A (en) Vehicle control device and vehicle control method
CN110281940A (en) Driving assistance system and vehicle control method
CN117125064A (en) Control device, control system, control method, non-transitory storage medium and vehicle
US20250138552A1 (en) Control system for steering automated vehicles using heterogeneous redundancy checks
CN110942651B (en) A vehicle failure processing method, vehicle-mounted device and storage medium
CN109572812B (en) Apparatus and method for controlling angle covering operation of vehicle
US12428031B2 (en) Parking assist apparatus
CN113124144A (en) Signal processing method and device of vehicle control unit, storage medium, vehicle control unit and vehicle
EP4520634A1 (en) Safe orchestration of electromechanical actuators of a drive-by-wire system of an automated vehicle
US12472952B2 (en) Vehicle including a vehicle platform that transitions to a wake mode according to a wake command
CN112644474A (en) Parking assist apparatus
US20250065924A1 (en) Method for at Least Highly Automated Driving and for Infrastructure-based Assistance of a Motor Vehicle within a Parking Area

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20250602

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)