DE102022117913A1 - Method and device for object recognition in a vehicle - Google Patents

Abstract

A device for detecting objects in the surroundings of a vehicle is described based on sensor data from one or more surroundings sensors of the vehicle. The device is set up to predict a traffic situation ahead in the direction of travel and to adapt an object recognition method depending on the predicted traffic situation ahead. The device is further set up to record sensor data in relation to the surroundings of the vehicle when driving through the traffic situation ahead using the one or more surroundings sensors of the vehicle, and one or more objects in the surroundings of the vehicle based on the sensor data and using the adapted object recognition method to detect.

Description

The invention relates to a method and a corresponding device which are designed to detect objects in the surroundings of a motor vehicle.

A vehicle can include one or more driving functions that enable at least partially automated longitudinal and/or lateral guidance of the vehicle. As part of a driving function, objects in the surroundings of the vehicle can be recognized, and the vehicle can be automatically guided longitudinally and/or laterally depending on the one or more detected objects. The quality of the driving function typically depends on the quality of the object recognition.

This document deals with the technical task of increasing the quality of object recognition in a vehicle in an efficient and reliable manner.

The task is solved by each of the independent claims. Advantageous embodiments are described, among other things, in the dependent claims. It should be noted that additional features of a patent claim dependent on an independent patent claim can form a separate invention independent of the combination of all the features of the independent patent claim, without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can be made the subject of an independent claim, a division application or a subsequent application. This applies equally to technical teachings described in the description, which may constitute an invention independent of the features of the independent patent claims.

According to one aspect, a device for detecting objects in the surroundings of a (motor) vehicle is described based on sensor data from one or more surroundings sensors of the vehicle. Examples of environmental sensors are one or more cameras, one or more radar sensors, one or more lidar sensors, etc.

The device is set up to predict a (safety-relevant) traffic situation ahead in the direction of travel. The device can be set up to receive situation information relating to the traffic situation ahead from a unit external to the vehicle (e.g. from a backend server) via a wireless communication connection.

The situational information may include, for example: a digital map relating to the road network traveled by the vehicle; a forecast of traffic and/or dangerous situations; Real Time Traffic Information; and/or a digital map relating to friction values of road surface coverings in the road network traveled by the vehicle.

The traffic situation ahead in the direction of travel can be predicted (with a relatively large prediction horizon) based on the situation information received. Alternatively or additionally, the device can be set up to predict the traffic situation ahead in the direction of travel (with a relatively high accuracy) based on sensor data from one or more environmental sensors.

Alternatively or additionally, the device can be set up to predict the traffic situation ahead in the direction of travel (with a relatively high accuracy) based on one or more data messages (in particular vehicle-to-X messages) that were received on the vehicle. The one or more data messages may have been sent from another vehicle and/or from an infrastructure unit (e.g. from a camera on a road). The one or more data messages may include sensor data from one or more environmental sensors of the other vehicle and/or the infrastructure unit.

The device can in particular be set up (based on the situation information and/or based on the sensor data) to determine the type of traffic situation ahead, in particular to select it from a large number of different, predefined situation types.

The plurality of situation types (of safety-relevant traffic situations) may include: at least one situation type for traffic accidents; at least one situation type for construction sites; at least one situation type for objects on the road traveled by the vehicle; at least one situation type for a traffic jam; at least one situation type for a pedestrian or an animal on the road traveled by the vehicle; and/or at least one situation type for an impaired surface covering (e.g. ice or oil) on the roadway traveled by the vehicle.

The device is also set up to adapt an object recognition method depending on the predicted (safety-relevant) traffic situation ahead. The object detection method can be particularly dependent be adapted to the determined type of traffic situation ahead.

The device can, for example, be set up to select the object recognition method depending on the selected type of traffic situation ahead from a variety of different, predefined, type-specific object recognition methods for the corresponding variety of different, predefined, situation types.

The different type-specific object recognition methods may have been trained (by machine) in advance using specific sensor data for the different types of situations.

The device is also set up to record sensor data relating to the surroundings of the vehicle using the one or more surroundings sensors of the vehicle when driving through the traffic situation ahead. One or more objects in the area surrounding the vehicle can then be detected based on the sensor data and using the adapted (in particular type-specific) object detection method.

The different (type-specific) object detection methods can be optimized with respect to different objects that can occur in the different situation types. For example, a first object detection method can be optimized for the detection of relatively small objects and a second object detection method for the detection of relatively large objects.

Furthermore, the device can be set up to guide the vehicle longitudinally and/or transversely at least partially automatically depending on the one or more detected objects when driving through the traffic situation ahead (safety-relevant and/or safety-critical).

A device is therefore described which is designed to adapt the object recognition method used to the current (safety-relevant) traffic situation in order to increase the quality of the object recognition and the quality of the automated longitudinal and/or lateral guidance of the vehicle.

The device can be set up to use and/or provide the sensor data recorded when driving through the traffic situation ahead or data derived therefrom for learning the type-specific object recognition method for the situation type of the predicted traffic situation ahead. In this way, the quality of object recognition can be further increased in an efficient and reliable manner.

According to a further aspect, a device for teaching an object recognition method for recognizing objects in the surroundings of a vehicle is described. The measures described in this document can be used individually and/or in combination for this device.

The device is set up to predict a traffic situation ahead in the direction of travel (based on the situation information and/or based on the sensor data from the one or more environmental sensors).

Furthermore, the device is set up to select the type of traffic situation ahead from a variety of different, predefined situation types (based on the situation information and/or based on the sensor data from the one or more environmental sensors).

The device is further set up to use one or more surrounding sensors of the vehicle to record sensor data in relation to the surroundings of the vehicle when driving through the traffic situation ahead, and the recorded sensor data or data derived therefrom for teaching a type-specific object recognition method for the To use and/or provide (to a vehicle-external unit) situation type of the predicted traffic situation ahead. This makes particularly precise object recognition possible.

According to a further aspect, a (road) motor vehicle (in particular a passenger car or a truck or a bus or a motorcycle) is described which comprises at least one of the devices described in this document.

According to a further aspect, a method for detecting objects in the surroundings of a (motor) vehicle is described based on sensor data from one or more surroundings sensors of the vehicle. The method includes predicting a traffic situation ahead in the direction of travel, as well as adapting a (machine-learned) object recognition method depending on the predicted traffic situation ahead.

The method further includes acquiring sensor data relating to the surroundings of the vehicle when driving through the traffic situation ahead based on one or more Surroundings sensors of the vehicle, as well as the detection of one or more objects in the surroundings of the vehicle based on the sensor data and using the adapted object detection method.

According to a further aspect, a method for learning a (machine-learned) object recognition method for recognizing objects in the surroundings of a vehicle is described. The aspects and/or measures described in this document are applicable individually or in combination for this procedure.

The method includes predicting a traffic situation ahead in the direction of travel, as well as selecting a type of traffic situation ahead from a variety of different, predefined situation types.

The method further includes acquiring sensor data relating to the surroundings of the vehicle when driving through the traffic situation ahead using one or more surrounding sensors of the vehicle, as well as using and/or providing the acquired sensor data or data derived therefrom for learning a type-specific Object detection method for the situation type of the predicted traffic situation ahead.

According to a further aspect, a software (SW) program is described. The SW program can be set up to run on a processor (e.g. on a vehicle control unit) and thereby carry out at least one of the methods described in this document.

According to a further aspect, a storage medium is described. The storage medium may comprise a SW program configured to be executed on a processor and thereby to carry out at least one of the methods described in this document.

It should be noted that the methods, devices and systems described in this document can be used both alone and in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, devices and systems described in this document can be combined with one another in a variety of ways. In particular, the features of the claims can be combined with one another in a variety of ways. Furthermore, features listed in brackets are to be understood as optional features.

• 1 beispielhafte Komponenten eines Fahrzeugs;
• 2 eine beispielhafte Fahrsituation eines Fahrzeugs; und
• 3 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Erkennung von Objekten im Umfeld eines Fahrzeugs.

The invention is further described in more detail using exemplary embodiments. Show it

• 1 exemplary components of a vehicle;
• 2 an exemplary driving situation of a vehicle; and
• 3 a flowchart of an exemplary method for detecting objects in the surroundings of a vehicle.

As explained at the beginning, the present document deals with increasing the quality of object recognition in a vehicle, in particular in order to increase the quality of one or more driving functions of the vehicle. In this context shows 1 an exemplary vehicle 100 that includes one or more environmental sensors 102, each of which is configured to capture sensor data related to the environment of the vehicle 100. The one or more environmental sensors 102 may include one or more cameras, one or more lidar sensors, one or more radar sensors, one or more ultrasonic sensors, etc.

A (control) device 101 of the vehicle 100 can be set up to recognize objects in the environment of the vehicle 100 based on the sensor data from the one or more environment sensors 102. An object recognition method can be used, which, for example, uses one or more machine-trained neural networks.

The device 101 can further be set up to operate one or more longitudinal and/or lateral guidance actuators 103 of the vehicle 100 depending on the one or more detected objects, e.g. to provide a driving function for automated longitudinal and/or lateral guidance of the vehicle 100. Example actuators 103 are a drive motor, a braking device and/or a steering device.

The vehicle 100 may include a communication unit 104, which is set up to communicate with a vehicle-external unit 110, for example with a backend server, via a (wireless) communication connection 111, for example via 3G, 4G or 5G.

• • ein Verkehrsunfall;
• • eine Baustelle;
• • ein Objekt auf der Fahrbahn 201;
• • ein Verkehrsstau;
• • ein Mensch oder ein Tier auf der Fahrbahn 201; und/oder
• • ein kritischer Zustand (z.B. Eis, Öl, Schlagloch, etc.) des Bodenbelags der Fahrbahn 201.

2 shows an exemplary driving situation in which the vehicle 100 is driving on a roadway 201 towards an area in which a safety-relevant traffic situation 200 exists. There may be different types of safety-relevant traffic situations 200, such as:

• • a traffic accident;
• • a construction site;
• • an object on roadway 201;
• • a traffic jam;
• • a person or animal on the roadway 201; and or
• • a critical condition (e.g. ice, oil, pothole, etc.) of the road surface 201.

• • eine digitale Karte in Bezug auf das von dem Fahrzeug 100 befahrene Fahrbahnnetz;
• • eine Vorhersage in Bezug auf die Verkehrsdichte;
• • Daten in Bezug auf den Reibwert des Bodenbelags der befahrenen Fahrbahn 201;
• • Real Time Traffic Information (RTTI), etwa über einen Radiosender; und/oder
• • eine Vorhersage über eine Gefahrensituation.

The device 101 can be set up to receive situation information from a vehicle-external unit 110 (via the (wireless) communication connection 111) in relation to a safety-relevant traffic situation 200 ahead in the direction of travel of the vehicle 100. For example, situation information can be provided:

• • a digital map relating to the road network traveled by the vehicle 100;
• • a forecast regarding traffic density;
• • Data relating to the coefficient of friction of the pavement of the road being traveled on 201;
• • Real Time Traffic Information (RTTI), for example via a radio station; and or
• • a prediction about a dangerous situation.

• • der Typ der Verkehrssituation (aus einer Liste von unterschiedlichen Typen von Verkehrssituationen); und/oder
• • eine Position der Verkehrssituation.

The situation information provided by a unit 110 external to the vehicle can, if necessary, be combined with the sensor data from the one or more surroundings sensors 102 of the vehicle 100. An upcoming safety-critical traffic situation 200 can then be predicted based on the situation information and/or based on the sensor data from the one or more environmental sensors 102. It can be predicted that

• • the type of traffic situation (from a list of different types of traffic situations); and or
• • a position of the traffic situation.

The device 101 can also be set up to adapt the object recognition method used for object recognition depending on the predicted traffic situation. For example, a corresponding variety of different object recognition methods can be provided for a variety of different types of traffic situations. The object recognition method that was trained for the type of predicted traffic situation can then be used (forward-looking) for the journey within the predicted traffic situation. By using a type-specific object detection method, the quality of object detection can be increased.

Furthermore, sensor data from one or more environmental sensors 102 can be recorded and stored when driving through the predicted traffic situation. This sensor data can be used to further optimize the type-specific object detection method. In this way, the quality of object recognition can be further improved.

3 shows a flowchart of a (possibly computer-implemented) method 300 for detecting objects in the environment of a (motor) vehicle 100 based on sensor data from one or more environment sensors 102 of the vehicle 100.

The method 300 includes predicting 301 a (safety-critical) traffic situation 200 ahead in the direction of travel. In particular, the position and/or the situation type of the traffic situation 200 can be predicted (e.g. based on situation information provided by a unit 110 external to the vehicle becomes).

The method 300 further includes adapting 302 an object recognition method depending on the predicted traffic situation 200 ahead. In particular, a type-specific object recognition method can be selected for the type of traffic situation 200 ahead.

Furthermore, the method 300 includes the acquisition 303 of sensor data relating to the surroundings of the vehicle 100 when driving through the traffic situation 200 ahead using the one or more surroundings sensors 102 of the vehicle 100. The method 300 also includes the detection 304 of one or more objects in the Surroundings of the vehicle 100 based on the sensor data and using the adapted object recognition method.

By adapting the object recognition method to the type of the respective traffic situation, the quality of the object recognition and the quality of a driving function of the vehicle 100 based on it can be increased.

Furthermore, this document describes a method for learning an object recognition method for recognizing objects in the surroundings of a (motor) vehicle 100. The measures described in this document can also be used individually or in combination for this procedure.

The method includes predicting a traffic situation 200 ahead in the direction of travel, as well as selecting the type of traffic situation 200 ahead from a variety of different, predefined situation types. This can be done based on situation information provided by a unit 110 external to the vehicle.

The method further includes acquiring sensor data relating to the surroundings of the vehicle 100 when driving through the traffic situation 200 ahead using one or more surroundings sensors 102 of the vehicle 100. The method further includes using and/or providing the captured sensor data or data derived therefrom Data for the (machine) training of a type-specific object recognition method for the situation type of the predicted traffic situation ahead 200. In this way, type-specific object recognition methods can be trained and provided in an efficient and precise manner.

Through the measures described in this document, the quality of object recognition in a vehicle 100 and thus the quality of one or more driving functions for automated driving can be increased in an efficient and reliable manner. This makes it possible, in particular, to increase the reliability and/or the maximum permissible driving speed of an autonomous driving function. Furthermore, the measures described can be used to specifically collect data for training type-specific object recognition methods, which can further increase the quality of object recognition.

The present invention is not limited to the exemplary embodiments shown. In particular, it should be noted that the description and the figures are only intended to illustrate the principle of the proposed methods, devices and systems by way of example.

Claims (12)

Device (101) for detecting objects in the surroundings of a vehicle (100) based on sensor data from one or more surroundings sensors (102) of the vehicle (100); wherein the device (101) is set up, - to predict a traffic situation (200) ahead in the direction of travel; - adapt an object recognition method depending on the predicted traffic situation (200) ahead; and - when driving through the traffic situation ahead (200) - to record sensor data relating to the surroundings of the vehicle (100) using the one or more surroundings sensors (102) of the vehicle (100); and - to detect one or more objects in the surroundings of the vehicle (100) based on the sensor data and using the adapted object detection method. Device (101) according to Claim 1 , wherein the device (101) is set up to - select a type of the traffic situation (200) ahead from a variety of different, predefined situation types; and - adapt the object recognition method depending on the selected type of traffic situation (200) ahead. Device (101) according to Claim 2 , wherein the plurality of situation types includes, - at least one situation type for traffic accidents; - at least one situation type for construction sites; - at least one situation type for objects on a roadway (201) traveled by the vehicle (100); - at least one situation type for a traffic jam; - at least one situation type for a pedestrian or for an animal on the roadway (201) traveled by the vehicle (100); and/or - at least one situation type for an impaired floor covering on the roadway (201) traveled by the vehicle (100). Device (101) according to one of Claims 2 until 3 , wherein the device (101) is set up to use the object recognition method depending on the selected type of traffic situation (200) ahead from a variety of different, predefined, type-specific object recognition methods for the corresponding variety of different, predefined situations -types to select. Device (101) according to one of the preceding claims, wherein the device (101) is set up - to receive situation information regarding the traffic situation ahead from a unit (110) external to the vehicle via a wireless communication connection (111); and - to predict the traffic situation (200) ahead in the direction of travel based on the received situation information. Device (101) according to Claim 5 , wherein the situation information comprises - a digital map relating to a road network traveled by the vehicle (100); - a forecast of traffic and/or dangerous situations; - Real Time Traffic Information; and/or - a digital map relating to the coefficients of friction of floor coverings of roadways (201) in the roadway network traveled by the vehicle (100). Device (101) according to one of the preceding claims, wherein the device (101) is set up to predict the traffic situation (200) ahead in the direction of travel based on sensor data from the one or more environment sensors (102). Device (101) according to one of the preceding claims, wherein the device (101) is set up to guide the vehicle (100) longitudinally and/or transversely at least partially automatically depending on the one or more detected objects when driving through the traffic situation (200) ahead . Device (101) according to one of the preceding claims, wherein the device (101) is set up to use the sensor data recorded when driving through the traffic situation (200) ahead or data derived therefrom for teaching a type-specific object recognition method for a situation type to use and/or provide the predicted traffic situation (200). Device (101) for teaching an object recognition method for recognizing objects in the surroundings of a vehicle (100); wherein the device (101) is set up, - to predict a traffic situation (200) ahead in the direction of travel; - select a type of the traffic situation (200) ahead from a variety of different, predefined situation types; - using one or more surroundings sensors (102) of the vehicle (100) to record sensor data relating to the surroundings of the vehicle (100) when driving through the traffic situation (200) ahead; and - to use and/or provide the recorded sensor data or data derived therefrom for training a type-specific object recognition method for the situation type of the predicted traffic situation (200) ahead. Method (300) for detecting objects in the surroundings of a vehicle (100) based on sensor data from one or more surroundings sensors (102) of the vehicle (100); wherein the method (300) comprises, - Predicting (301) a traffic situation (200) ahead in the direction of travel; - Adapting (302) an object recognition method depending on the predicted traffic situation ahead (200); - Acquiring (303) sensor data relating to the surroundings of the vehicle (100) when driving through the traffic situation (200) ahead using the one or more surroundings sensors (102) of the vehicle (100); and - Detecting (304) one or more objects in the surroundings of the vehicle (100) based on the sensor data and using the adapted object detection method. Method for learning an object recognition method for recognizing objects in the surroundings of a vehicle (100); wherein the method includes, - Predicting a traffic situation ahead in the direction of travel (200); - Selecting a type of the traffic situation ahead (200) from a variety of different, predefined situation types; - Acquiring sensor data relating to the surroundings of the vehicle (100) when driving through the traffic situation (200) ahead using one or more surroundings sensors (102) of the vehicle (100); and - Using and/or providing the recorded sensor data or data derived therefrom for training a type-specific object recognition method for the situation type of the predicted traffic situation ahead (200).

Images