DE102023202076A1 - Method and system for creating an extended semantic dynamic grid map with audio signals - Google Patents

Abstract

The invention relates to a method for generating an extended semantic dynamic grid map in an ego vehicle with the following steps:
- recording (S1) an environment of the ego vehicle by means of at least one environment detection sensor (2) and generating sensor data of the at least one environment detection sensor (2);
- Converting (S2) the sensor data from a sensor coordinate system to coordinates of the measuring grid to create a first measuring grid;
- recording (S3) auditory signals by means of at least one microphone array (3) of the ego vehicle;
- Generating (S4) an auditory heat map with event classes based on the recorded auditory signals;
- Converting (S5) the auditory heat map into a second semantic measurement grid and entering occupancy probabilities and event classes into corresponding grid cells;
- Generating (S6) an extended semantic dynamic grid map by fusion of the first measurement grid and the second semantic measurement grid.

Description

The invention relates to a method and a system for creating an extended semantic dynamic grid map (Dynamic Grid Map).

Seeing and hearing are among the sensory perceptions required when driving. However, current environment modeling for automated/autonomous driving is mostly based on cameras, radar, ultrasound and lidar. Target objects can only be detected if they are within the field of view of the sensors. These detected objects can then be entered, for example, into a measuring grid consisting of a large number of grid cells. However, the limitation of the field of view of the sensors used also limits the detection range and the possibility of detecting the environment. Furthermore, the sensors used can only detect certain features of the environment or objects, which can be used, for example, to classify the objects, which leads to a limited classification option.

It is therefore an object of the present invention to provide a method and a system by means of which an extended semantic dynamic grid map can be generated, which enables an improved understanding of the vehicle environment.

This object is solved by the subject matter of independent claims 1 and 6. Further advantageous embodiments are the subject matter of the subclaims.

Initial considerations were that in many situations objects can be heard much earlier than they can be seen. Even in the field of vision, the perception of the environment can be improved with the fusion of audio. It is therefore advantageous to use audio-visual environment modeling for automated/autonomous driving. There are increasing numbers of so-called Sound Event Localization and Detection (SELD) methods that can detect and localize several auditory events at the same time. However, few methods are currently known that use audio-visual fusion methods for environment modeling for automated/autonomous driving.

Most systems also lack microphones, which can act as additional sensors to improve the perception of the environment. There are hardly any audiovisual fusion processes for modeling the environment for automated/autonomous driving.

• - Aufzeichnen eines Umfelds des Ego-Fahrzeugs mittels zumindest eines Umfelderfassungssensors und Erzeugen von Sensordaten des zumindest eines Umfelderfassungssensors;
• - Konvertieren der Sensordaten aus einem Sensorkoordinatensystem zu Koordinaten des Messgitters zum Erstellen eines ersten Messgitters;
• - Aufzeichnen von auditorischen Signalen mittels zumindest eines Mikrofonarrays des Ego-Fahrzeugs;
• - Erzeugen einer auditorischen Heat-Map mit Ereignisklassen basierend auf den aufgezeichneten auditorischen Signalen;
• - Konvertieren der auditorischen Heat Map in ein zweites semantische Messgitter und Eintragen von Belegungswahrscheinlichkeiten und Ereignisklassen in entsprechende Gridzellen;
• - Erzeugen einer erweiterten semantischen dynamischen Gridkarte mittels Fusion des ersten Messgitters und des zweiten semantischen Messgitters.

According to the invention, a method for generating an extended semantic dynamic grid map in an ego vehicle is therefore proposed, comprising the following steps:

• - Recording an environment of the ego vehicle by means of at least one environment detection sensor and generating sensor data of the at least one environment detection sensor;
• - Converting the sensor data from a sensor coordinate system to measurement grid coordinates to create a first measurement grid;
• - Recording auditory signals using at least one microphone array of the ego vehicle;
• - Generating an auditory heat map with event classes based on the recorded auditory signals;
• - Converting the auditory heat map into a second semantic measurement grid and entering occupancy probabilities and event classes into corresponding grid cells;
• - Creating an extended semantic dynamic grid map by merging the first measurement grid and the second semantic measurement grid.

In the light of the invention, the step of recording the environment includes not only a one-time recording, but also a continuous recording of the environment during the movement of the ego vehicle. For each run of the first three steps, only one recording/frame or camera image is used. The at least one environment detection sensor can be, for example, a camera, a radar, lidar or ultrasound sensor. Advantageously, several sensors of the same and/or different types are used to enable better environment detection and more precise determination of detected objects, since each sensor type can detect certain features with varying degrees of success.

Depending on the environment detection sensor used, the first measurement grid can also contain semantic information. For example, when using a camera, the camera data can be classified using semantic segmentation. Using semantic segmentation, pixels of a camera image can be classified, for example, so that class information for each pixel can be determined. With a camera, semantic segmentation is first carried out at the image level. This means that each pixel is assigned to a class. The image is then converted to the measurement grid in world coordinates (2D Bird's Eye View). This means that each grid cell is assigned to a class. net. The sensor data can also be, for example, point clouds from a lidar sensor or reflections from a radar sensor. For the first measurement grid, information about the occupied and free masses is sufficient. Semantic information does not necessarily have to be assigned to these detections.

For the aforementioned semantic segmentation, several sets of sensor data, in the example of a camera, several camera images can be used as input for a neural network. The output of the neural network is then the semantic segmentation in a bird's eye view in a measurement grid. Another alternative would be to enter the class information of each pixel in grid cells of a measurement grid using a coordinate conversion from image coordinates to bird's eye view.

A measurement grid is a mostly two-dimensional, uniform grid-like division of space into individual cells. Depending on whether probability theory or evidence theory is used, each of these cells contains a probability value as to whether a cell is occupied by an obstacle, or two evidence values, one for the occupied state and one for the free state. To generate the dynamic grid map, measurement grids are merged. In addition, the cells occupied by moving objects are tracked and the evidence of the various statuses (statically occupied, dynamically occupied, free, invisible but dynamically occupied in the last frame) and the speed of dynamic cells are estimated. Such a method is described, for example, in the paper “ Grid-Based Environment Estimation Using Evidential Mapping and Particle Tracking“ by S. Steyer, et. (IEEE Transactions on Intelligent Vehicles, 2018 ) described.

Furthermore, auditory signals in the environment of the ego vehicle are preferably recorded at the same time as the aforementioned steps or subsequently using at least one microphone array. These auditory signals are entered into an auditory heat map with so-called event classes. The event classes are assigned to the various auditory signals using, for example, an appropriately trained neural network and the heat map is created based on this. The auditory signals of the microphone array can be filtered and pre-processed before they are transmitted to the neural network. The heat map also has cells in which the event classes of the auditory signals are entered. This heat map is then converted into a second measurement grid and the grid cells are divided into corresponding occupancy probabilities. The event classes are also entered into the grid cells. The heat map as well as the first and second measurement grids are available in a bird's eye view and accordingly describe a two-dimensional representation of the environment. The auditory heat map is created using, for example, an appropriately trained neural network.

The steps for creating the first and second measurement grids are preferably carried out at least twice or more in order to be able to create a semantic dynamic grid map from the first and second measurement grids. A different number of first and second semantic measurement grids can be created. For example, several first measurement grids can be created with the sensor data from the environment detection sensor and fused with a second semantic measurement grid. However, to create a first semantic dynamic grid map, the fusion of a single first measurement grid and a single second measurement grid is sufficient. To determine the evidence of the state of whether a cell is occupied by a dynamic object, dynamic cells in the grid map at time t _-1 are tracked with a particle filter to the grid map at time t ₀ .

The advantage here is that the auditory signals can expand the range of perception. The microphone array also increases the redundancy of the system. The microphone arrays provide good signals, for example, when cameras do not work well in poor lighting conditions. A semantic dynamic grid map based on a microphone array and, for example, a camera can model the environment better than a semantic dynamic grid map based purely on the camera, even if an object is in the camera's field of view.

In a preferred embodiment, the event classes include sirens from emergency vehicles, engine noises from motorized vehicles, vehicle horns, noises when starting an engine and noise emanating from other potential non-motorized road users. Engine noises from motorized vehicles include in particular engine noises from a moving car, truck, motorcycle or the like. The other potential non-motorized road users include, for example, noise emanating from bicycles, joggers, children playing and from the class "unknown" if the noise cannot be clearly assigned.

It is further preferred that a SELD (Sound Event Localization and Detection) method is used to generate the auditory heat map. It is particularly preferred for the SELD method A neural network is used. The SELD method is adapted so that the output is a heat map in a bird's eye view.

In a further preferred embodiment, an integration weight is each assigned to the first measurement grid and the second semantic measurement grid.

It is particularly preferred if the integration weight of the first and/or second measurement grid is adapted to the prevailing environmental conditions and driving scenarios. Accordingly, the integration weight could be adapted to the time of day, lighting conditions, speed of the ego vehicle and to the scenes. For example, on a quiet evening or in poor lighting conditions, the integration weight of the auditory measurement grid could be increased. The faster the ego vehicle is driving, the stronger the wind. With stronger wind, auditory reception becomes worse and the corresponding integration weight could be reduced. In addition, driving scenarios or scenes, e.g. highway, tunnel, country road, construction site, intersection, city center, etc., can be detected using their auditory and visual properties. The integration weight of the two measurement grids can be adapted to different driving scenarios. Based on the fused semantic dynamic grid map, both open space driven on can be extracted and static and dynamic objects can be recognized.

According to the invention, a system for generating an extended semantic dynamic grid map in an ego vehicle is further proposed, comprising at least one environment detection sensor for recording an environment of the ego vehicle and at least one microphone array for recording auditory signals and a computing device, wherein the computing device is designed to evaluate and process sensor data of the environment detection sensor and auditory signals of the microphone array and to create a first and a second semantic measurement grid and to generate an extended semantic dynamic grid map by merging the first measurement grid and the second semantic measurement grid.

The computing device can be designed as an ECU in the ego vehicle, for example. It would also be conceivable for the computing device to be arranged in one of the sensors. The microphone array, the at least one environment detection sensor and the computing device are communicatively connected to one another via a wired or wireless data connection. The microphone array could, for example, be arranged as a unit at a central location, such as in the middle of the car roof.

In a preferred embodiment of the system, the microphone array consists of several individual microphones. Several microphones are advantageous because one microphone can be attached to each of the different positions on the ego vehicle, which improves the detection accuracy and detection options of auditory signals.

Furthermore, the individual microphones are particularly preferably arranged on the outside of one side of the roof of the ego vehicle at four corner points. This type of arrangement can advantageously cover all directions from which auditory signals can occur. Furthermore, the influence of the vehicle's own engine noise and tire noise is minimal when arranged on the roof of the vehicle.

• 1: ein schematisches Ablaufdiagramm des Verfahrens gemäß einer Ausgestaltung der Erfindung;
• 2: eine schematische Darstellung eines Systems gemäß einer Ausführungsform der Erfindung.

Further advantageous embodiments and embodiments are the subject of the figures. Therein show:

• 1 : a schematic flow diagram of the method according to an embodiment of the invention;
• 2 : a schematic representation of a system according to an embodiment of the invention.

The 1 shows a schematic flow diagram of the method for generating an extended semantic dynamic grid map in an ego vehicle according to an embodiment of the invention. In step S1, an environment of the ego vehicle is recorded using at least one environment detection sensor and sensor data from the at least one environment detection sensor is generated. In step S2, the sensor data from a sensor coordinate system is converted to coordinates of the measurement grid in order to create a first measurement grid. In step S3, auditory signals are recorded using at least one microphone array of the ego vehicle. In step S4, an auditory heat map with event classes is generated based on the recorded auditory signals. In step S6, the auditory heat map is converted into a second semantic measurement grid and occupancy probabilities and event classes are entered into corresponding grid cells. In step S6, an extended semantic dynamic grid map is generated by merging the first measurement grid and the second semantic measurement grid. Steps S1 to S2 and S4 to S6 preferably run simultaneously.

2 shows a schematic representation of a system according to an embodiment of the invention. The system 1 has at least one environment detection sensor 2, a microphone array 3 and a computing device 4. The environment detection sensor 2 and the microphone array 3 are each connected to the computing device 4 via a data connection D. The data connection D is preferably designed as a wired connection. A wireless data connection D would also be conceivable. The computing device 4 is designed to evaluate and process sensor data from the environment detection sensor 2 and auditory signals from the microphone array 3, to create a first measurement grid and a second semantic measurement grid and to generate an extended semantic dynamic grid map by merging the first measurement grid and the second semantic measurement grid.

List of reference symbols

1

system

2

Environment detection sensor

3

Microphone array

4

Computing device

D

Data Connection

S1-S6

Process steps

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely to provide the reader with better information. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited non-patent literature

• Grid-Based Environment Estimation Using Evidential Mapping and Particle Tracking“ by S. Steyer, et. (IEEE Transactions on Intelligent Vehicles, 2018 [0011]

Claims (8)

Method for generating an extended semantic dynamic grid map in an ego vehicle with the following steps: - Recording (S1) an environment of the ego vehicle using at least one environment detection sensor (2) and generating sensor data from the at least one environment detection sensor (2); - Converting (S2) the sensor data from a sensor coordinate system to coordinates of the measurement grid to create a first measurement grid; - Recording (S3) auditory signals using at least one microphone array (3) of the ego vehicle; - Generating (S4) an auditory heat map with event classes based on the recorded auditory signals; - Converting (S5) the auditory heat map into a second semantic measurement grid and entering occupancy probabilities and event classes into corresponding grid cells; - Generating (S6) an extended semantic dynamic grid map by merging the first measurement grid and the second semantic measurement grid. Procedure according to Claim 1 , characterized in that the event classes include sirens from emergency vehicles, engine noises from motorised vehicles, horns from vehicles, noises when starting an engine and noise from other potential non-motorised road users. Procedure according to Claim 1 , characterized in that a SELD method is used to generate the auditory heat map. Procedure according to Claim 1 , characterized in that an integration weight is each assigned to the first measuring grid and the second semantic measuring grid. Procedure according to Claim 4 , characterized in that the integration weight of the first and/or second measuring grid is adapted to prevailing environmental conditions and driving scenarios. System (1) for generating an extended semantic dynamic grid map in an ego vehicle comprising at least one environment detection sensor (2) for recording an environment of the ego vehicle and at least one microphone array (3) for recording auditory signals and a computing device (4), wherein the computing device (4) is designed to evaluate and process sensor data of the environment detection sensor (2) and auditory signals of the microphone array (3) and to create at least a first measurement grid and a second semantic measurement grid and to generate a semantic dynamic grid map by merging the first measurement grid and the second semantic measurement grid. System according to Claim 6 , characterized in that the microphone array (4) consists of several individual microphones. System according to Claim 7 , characterized in that the individual microphones are arranged on the outside of a roof side of the ego vehicle at four corner points.

Images