WO2023001539A1

WO2023001539A1 - Method for controlling a cleaning process for an optical path of a camera system of a mobile platform

Info

Publication number: WO2023001539A1
Application number: PCT/EP2022/068536
Authority: WO
Inventors: Mirko Thulke; Markus Klein; Michael Ostertag; Yao Li; Shishira KROLL; Andreas Kern; Niklas HEPPNER; Juergen PFADT
Original assignee: Robert Bosch Gmbh
Priority date: 2021-07-21
Filing date: 2022-07-05
Publication date: 2023-01-26
Also published as: DE102021207792A1

Abstract

The invention relates to a method for controlling a cleaning process for an optical path of a camera system of a mobile platform, comprising the steps of: providing a current status of cleaning resources; providing a current blindness status of the optical path of the camera system; determining a future driving scenario of the mobile platform; determining a cleaning scenario on the basis of the current status of the cleaning resources and/or on the current blindness status of the optical path of the camera system and/or on the future driving scenario of the mobile platform in order to enable the cleaning process.

Description

description

title

Method for controlling a cleaning process for an optical path of a camera system of a mobile platform

State of the art

The automation of driving goes hand in hand with equipping vehicles with increasingly extensive and powerful sensor systems for detecting the surroundings. When interpreting the sensor data, machine learning methods are used for classification and detection tasks.

An optical camera system can be degraded by a wide variety of disturbances in the recording path, e.g. by soiling of the camera lens or a windscreen in front of it, or damage to the camera or by other environmental influences such as rain,

Fog, snow or spray.

Sufficiently clear images from the optical sensors are particularly important for capturing environmental representations for at least partially automated driving using such optical sensors. Therefore, such camera systems or video sensors are provided with cleaning devices.

Disclosure of Invention

In the case of automatic cleaning requests, it should be taken into account that a cleaning process does not take place too frequently in order to avoid cleaning resources being used up unnecessarily, so that a cleaning functionality may no longer be implemented due to consumed resources, which functionalities of application processes that require an undisturbed optical require the path of the camera system, are restricted and/or are no longer available. This requires a regulation of the cleaning process that adapts the cleaning process to driving scenarios, depending on environmental and/or vehicle conditions, in order to optimize the consumption of cleaning resources.

For this purpose, the control can be based on a manual and/or automatic REAR and/or FRONT and/or side camera cleaning process and/or, d. H. based on a user input and/or an algorithmic blindness detection and/or a cleaning request of an application method of the camera system. The controller can output a software signal, for example to activate a cleaning pump as a cleaning process, which sprays cleaning liquid onto the optical path of the camera system in order to clean it.

According to aspects of the invention, a method for controlling a cleaning process for an optical path of a camera system of a mobile platform, a use of the method, a data processing device, a control device and a mobile platform are proposed according to the features of the independent claims. Advantageous configurations are the subject of the dependent claims and the following description.

Throughout this description of the invention, the sequence of method steps is presented in such a way that the method is easy to follow.

However, those skilled in the art will recognize that many of the method steps can also be carried out in a different order and lead to the same or a corresponding result. In this sense, the order of the method steps can be changed accordingly. Some features are numbered to improve readability or to clarify attribution, but this does not imply the presence of specific features.

According to one aspect of the invention, a method for controlling a cleaning process for an optical path of a camera system of a mobile platform is proposed, with a current status of cleaning resources being provided in one step. In a further step, a current blindness status of the optical path of the camera system is provided.

In a further step, a future driving scenario of the mobile platform is determined. In a further step, a cleaning scenario based on the current status of the cleaning resources and/or the current blindness status of the optical path of the camera system and/or the future driving scenario of the mobile platform in order to release the cleaning process.

In other words, a necessary cleaning can be identified with this method before a cleaning process is requested by an application method that uses an image of the camera system. i.e. a cleaning process of the optical path can already be performed when certain application processes that use an image of the camera system are not yet switched to operative, so that the cleaning process does not affect the function.

In addition, a cleaning process can be enabled and/or triggered during a specific driving condition in order to optimize a cleaning effect. For example, the cleaning process can be triggered at a speed at which a cleaning liquid used is removed from the optical path of the camera system by the relative wind, particularly in a speed range of 40 km/h to 60 km/h, after cleaning.

The driving scenario can be recognized, for example, by changing a driving cycle or by going through specific driving cycles from which a future driving scenario can be derived. Furthermore, for example, a specific driving scenario can be recognized by braking from a sustained high speed to a specific low speed range. This can, for example, be interpreted as leaving a freeway without further information.

The future driving scenario can be determined from a large number of possible driving scenarios by determining a probability for the occurrence of one of the large number of possible driving scenarios and can thus be determined as a probable future driving scenario.

The camera system can be set up both in the direction of travel and against the direction of travel as well as laterally to the direction of travel. i.e. the camera system can be a front camera system and/or a rear camera system and/or a side camera system.

A camera system can be an optical system for generating images of the surroundings of the mobile platform, such as an optical video system. In this context, regulating the cleaning process can be understood in the sense of controlling the cleaning process and/or regulating in the narrower sense.

A mobile platform can be understood to mean an at least partially automated system that is mobile and/or a driver assistance system of a vehicle. An example can be an at least partially automated vehicle or a vehicle with a driver assistance system. That is, in this context, an at least partially automated system includes a mobile platform in terms of at least partially automated functionality, but a mobile platform also includes vehicles and other mobile machines including driver assistance systems. Other examples of mobile platforms can be driver assistance systems with multiple sensors, mobile multi-sensor robots such as robotic vacuum cleaners or lawn mowers, a multi-sensor surveillance system, a manufacturing machine, a personal assistant or an access control system. Each of these systems can be a fully or partially automated system.

By using the cleaning scenario, the cleaning resources can be used in an optimized manner and adapted to the environmental and driving conditions. The cleaning scenario can also make it impossible for the driver of the mobile platform to manually trigger a cleaning process, for example in order not to disrupt the function of a current application method.

According to one aspect, it is proposed that a cleaning request from at least one application process for the mobile platform that uses an image of the camera system is provided and the cleaning process is triggered if the cleaning scenario has currently enabled the cleaning process and the cleaning request from at least one application process is active and/or a cleaning request was triggered by a driver.

The request for a cleaning process can originate from one or more of a large number of application processes and can be released or not released according to the process for controlling the cleaning process. Alternatively, a necessary cleaning process can be carried out using this method before the cleaning process is requested or after a cleaning process has been requested, in order to optimize the cleaning process and/or in order not to impair the function of the application process. For example, an internal parking space search algorithm can request a cleaning process independently of a driver request or prevent the cleaning process in order to provide functionality. If the future driving scenario determines a search for a parking space, the optical path of the camera system can be cleaned in a higher speed range, since the parking space search algorithm is only active up to 40 km/h.

Therefore, cleaning in a speed range above 40 km/h does not disturb the application of the parking space search algorithm.

Possible cleaning requirements of other application methods that are activated in a future driving scenario can be avoided with appropriate cleaning scenarios before the cleaning requirement of the respective application method in order to be able to provide the respective application method.

According to one aspect, it is proposed that, depending on the specific future driving scenario, a cleaning process is triggered if the cleaning scenario has enabled the cleaning process.

For example, when the mobile platform, such as a vehicle, exits a freeway in rainy conditions, a purge may be triggered. Furthermore, for example, a frequency of the cleaning process with a higher frequency can be enabled by the cleaning scenario during an off-road route.

The future driving scenario can be determined using a current driving scenario, for example by extrapolation from the current driving scenario. Additionally or alternatively, the future driving scenario can be determined based on a probability of occurrence from a large number of possible future driving scenarios.

For example, a current driving cycle and/or information about the surroundings of the mobile platform and/or vehicle status information can be taken into account to determine a future driving scenario.

According to one aspect, it is proposed that the future driving scenario of the mobile platform is determined using a trained machine learning system, and input variables for determining the future driving scenario of the mobile platform are provided to the trained machine learning system.

In other words, adaptive learning algorithms can be used to trigger the cleaning process using trained machine learning systems. Such a trained machine learning system can be, for example, a neural network that has been trained to determine future driving scenarios using the input variables. In this case, the input variables can relate to a driving cycle and/or to an environment of the mobile platform and/or to a vehicle state.

Advantageously, by determining the future driving scenario with a trained machine learning system, a prediction for a necessary cleaning process, depending on the future driving scenario, can be generated so that the cleaning process can be carried out proactively in order to optimize the cleaning process and/or the application program, using an image of the camera system to enable an optimized function.

Examples of machine learning systems are a neural network, in particular a neural convolutional network, possibly in combination with fully connected neural networks, possibly using classic regularization and stabilization layers such as batch normalization and training drop-outs various activation functions such as Sigmoid and ReLu, etc. Classic approaches such as support vector machines,

Boosting, decision trees and random forests can alternatively or additionally be used as machine learning systems for the method described.

In neural networks, the signal at a connection of artificial neurons can be a real number, and the output of an artificial neuron is calculated by a non-linear function of the sum of its inputs. The connections of the artificial neurons typically have a weight that adjusts as learning progresses. Weight increases or decreases the strength of the signal on a connection. Artificial neurons can have a threshold such that a signal is only output if the total signal exceeds this threshold.

Typically, a large number of artificial neurons are combined in layers. Different layers may perform different types of transformations on their inputs. Signals travel from the first layer, the input layer, to the last layer, the output layer; possibly after going through the shifts several times. Such neural networks must be trained for their specific task.

Each neuron of the corresponding architecture of the neural network receives z. B. a random starting weight. Then the input data is fed into the network, and each neuron can weight the input signals with its weight and passes the result on to the neurons of the next layer. The overall result is then provided at the output layer.

The size of the error can be calculated, as well as the contribution each neuron made to that error, and then change the weight of each neuron in the direction that minimizes the error. Then recursively runs, re-measures the error and adjusts the weights until an error criterion is met.

Such an error criterion can be, for example, the classification error on a test data set, or the current value of a loss function, for example on a training data set. Alternatively or additionally, the error criterion can relate to a termination criterion as a step in which overfitting would occur during training or the time available for training has expired.

In this method, the neural network can be trained with a large number of reference driving scenarios as input data to predict future driving scenarios, for example by using reference driving of the mobile platform to generate and store reference driving scenarios with different time intervals, with which the neural network can be trained , to predict driving scenarios at different time intervals based on a current driving scenario. A large number of corresponding input variables for the neural network can be detected and stored in the reference driving scenarios in order to use these input variables to identify and predict driving scenarios corresponding to training the neural network.

In this case, the reference driving scenarios can be driver scenarios that have been specifically recorded for training a machine learning system and, for example, have been selected and annotated manually, or have been generated synthetically and labeled accordingly. For example, such a labeling of the driving scenarios can be done manually.

According to one aspect, it is proposed that the input variables for the trained machine learning system include a transmission mode and/or an off-road mode and/or satellite navigation data and/or navigation data and/or a type of a driving cycle and/or detected traffic signs and/or objects in the vicinity of the mobile platform and/or vehicles in the vicinity of the mobile platform and/or detected subsurfaces of the mobile platform and/or a representation of the surroundings of the mobile platform.

Advantageously, the machine learning system, such as the neural network, can use these input variables to determine or predict future driving scenarios by taking into account as many relevant input variables as possible for determining the future driving scenario.

According to one aspect, it is proposed that the future driving scenario is determined using a current driving scenario and/or the future driving scenario is determined based on a probability of occurrence from a large number of possible future driving scenarios.

The future driving scenario can advantageously determine the future driving scenario by means of an extrapolation. Alternatively or additionally, the future driving scenario can be determined from a large number of scenarios by determining a probability of occurrence for a specific driving scenario.

According to one aspect, it is proposed that the future driving scenario has a speed range and/or an automatic parking process and/or a vehicle door status and/or a time of day and/or a lane change and/or a direction of travel and/or an off-road mode and/or a Type of driving cycle and/or detected mobile platform terrain includes.

These future driving scenarios are mentioned here as examples from a large number of possible driving scenarios.

According to one aspect, it is proposed that the cleaning process takes place by means of a wiping device and/or a washing device.

According to one aspect, it is proposed that the different application methods include an automatic parking process and/or an image-based application and/or an automatic emergency braking function and/or a calibrated camera system.

According to one aspect it is proposed that at least one application method is deactivated depending on the specific cleaning scenario and/or the provided blindness status and/or the cleaning resources. Advantageously, with this method for controlling the cleaning process, a possibility for activating the cleaning process can be determined in order to start a cleaning process if there is an at least temporarily limited possibility, to deactivate an application process at least temporarily in order to avoid a malfunction of the application process.

For example, a blindness of the optical path of the camera system can be determined, whereupon the vehicle is brought to a standstill, in particular by turning off the engine. When the vehicle is restarted, the optical path of the camera system is assumed to be sufficiently transparent, i. H. the camera system is assumed to be "not blind" in order to enable at least a partially automatic exit from a parking space.

According to one aspect, it is proposed that the camera system of the mobile platform includes a rear-facing camera system for reversing and/or a forward-facing camera system for forward driving and/or the cleaning scenario is dependent on the camera system used. Advantageously, due to the dependency of the cleaning scenario on the camera system used, i.e. whether the camera system used is directed forwards or backwards in relation to the mobile platform, it can be taken into account that different driving scenarios and in particular different future driving scenarios are possible when driving forwards or driving backwards . For example, a front automatic emergency brake system (AEB Front) is active during slow driving manoeuvres. Therefore, a possible cleaning request will be different than a rear AEB cleaning request, which is basically triggered when reversing is required. For example, the front AEB cleaning request may be triggered at the start of the drive cycle or when entering a low speed operating range.

According to one aspect it is proposed that the cleaning system is triggered when the requesting application method is not active and/or switched to a non-active state.

The regulation of the cleaning process for the optical path of the camera system can thus be specifically adapted to the corresponding application process, so that a cleaning process does not impair the function of the application process. In other words, according to an anticipated regulation, a cleaning process is then triggered in the corresponding driving cycle of a driving scenario if a function of an application method that uses an image from the camera system is outside of an active operating point, which means that a function is not blocked by the cleaning process.

A use of one of the methods described above for controlling the cleaning process for an optical path of a camera system of a mobile platform for controlling a cleaning process is proposed.

A device for data processing is proposed in order to regulate a cleaning process for an optical path of a camera system, the device having an input for providing a current status of cleaning resources and/or an input for providing a current blindness status of the optical path of the camera system and/or has an input for providing at least one cleaning request of at least one application method that uses at least one image of the camera system. Furthermore, the device for data processing has an arithmetic unit and/or a system-on-chip and an output for triggering a cleaning process, wherein the arithmetic unit and/or the system-on-chip is set up, one of the methods described above for controlling a cleaning process for an optical path of a camera system of a mobile platform.

Advantageously, with such a device, the method can easily be integrated into different systems.

A control unit for use in a vehicle, which has a data processing device as described above, is proposed.

Such a control device can also have additional functions.

A mobile platform, and in particular an at least partially automated vehicle, is proposed, which has a control unit as described above. Advantageously, such a mobile platform can realize all advantages of the method for controlling a cleaning process for an optical path of a camera system. exemplary embodiments

An exemplary embodiment of the invention is illustrated with reference to FIG. 1 and explained in more detail below. Show it:

FIG. 1 Flow chart of the method.

FIG. 1 schematically outlines a flowchart of a method for controlling a cleaning process for an optical path of a camera system on a mobile platform.

In a first step S1, a current status of cleaning resources is provided.

In a further step S2, a current blindness status of the optical path of the camera system is provided. In a further step S3, a future driving scenario of the mobile

platform determined.

In a further step S4, a cleaning scenario is determined based on the current status of the cleaning resources and/or the current blindness status of the optical path of the camera system and/or the future driving scenario of the mobile platform

release the cleaning process.

Claims

Expectations

Claims 1. A method for controlling a cleaning process for an optical path of a camera system of a mobile platform, comprising:

Providing a current status (Sl) of cleaning resources; Providing a current blindness status (S2) of the optical path of the camera system;

determining a future driving scenario (S3) of the mobile platform; determining a cleaning scenario (S4) based on the current status of the cleaning resources; and/or the current blindness status of the optical path of the camera system; and/or the future driving scenario of the mobile platform to enable the cleaning process.

2. The method of claim 1, wherein a cleaning request is provided to at least one application method for the mobile platform using an image of the camera system; and the cleaning process is triggered if the cleaning scenario has currently enabled the cleaning process; and the cleaning request of at least one application method is active.

3. The method according to any one of the preceding claims, wherein a cleaning process is triggered depending on the future driving scenario determined, provided that the cleaning scenario has enabled the cleaning process.

4. The method according to any one of the preceding claims, wherein the future driving scenario of the mobile platform is determined using a trained machine learning system, and the trained machine learning system is provided with input variables for determining the future driving scenario of the mobile platform.

5. The method according to claim 4, wherein the input variables for the trained machine learning system include a transmission mode and/or an off-road mode and/or satellite navigation data and/or navigation data and/or a type of driving cycle and/or detected traffic signs and/or objects in the vicinity of the mobile platform and/or vehicles in the vicinity of the mobile platform and/or detected surfaces of the mobile platform and/or a Include representation of the environment of the mobile platform.

6. The method according to any one of the preceding claims, wherein the future driving scenario is determined using a current driving scenario and / or the future driving scenario is determined based on a probability of occurrence from a variety of possible future driving scenarios.

7. The method according to any one of the preceding claims, wherein the future driving scenario includes a speed range and/or an automatic parking process and/or a vehicle door status and/or a time of day and/or a lane change and/or a direction of travel and/or an off-road mode and /or a type of driving cycle and/or detected surfaces of the mobile platform.

8. The method according to claim 1, wherein the different application methods include an automatic parking process and/or an image-based application and/or an automatic emergency braking function and/or a calibrated camera system and/or a manual request.

9. The method according to claim 1, wherein at least one application method is deactivated depending on the specific cleaning scenario and/or the provided blindness status and/or the cleaning resources.

10. The method according to any one of the preceding claims, wherein the camera system of the mobile platform comprises a rear-facing camera system for reversing and/or a forward-facing camera system for forward driving and/or a side camera system; and/or the cleaning scenario depends on the camera system used.

11. The method according to any one of the preceding claims, wherein the cleaning system is triggered when the requesting application method is not active and/or switched to a non-active state.

12. Use of the method according to any one of claims 1 to 11 for controlling a cleaning process.

13. Device for data processing to control a cleaning process for an optical path of a camera system, with an input for providing a current status of cleaning resources; and/or to provide a current blindness status of the optical path of the camera system; and/or for providing at least one cleaning requirement of at least one application method that uses at least one image of the camera system; and a computing unit and/or a system-on-chip; and an output for triggering a cleaning process; wherein the processing unit and/or the system-on-chip is set up to carry out a method according to one of Claims 1 to 12.

14. Control device for use in a vehicle, which has a device for data processing according to claim 13.

15. Mobile platform, in particular an at least partially automated vehicle, which has a control device according to claim 14.