DE102019201930A1 - Method for generating an environment model - Google Patents

Abstract

The invention relates to a method for generating an environment model, in particular a vehicle environment model, with the following steps: generating a grid map (1) of an area around a vehicle, the grid map (1) having a plurality of grid cells (2), estimating an occupancy status of each Grid cell (2), a distinction being made between free and occupied grid cells (2), determination of semantic class information of each free and occupied grid cell (2), assignment of at least one semantic class (V, P, B, F) with a certain probability for each free and occupied grid cell (2), the class being selected based on the semantic class information, - prediction of each occupied grid cell (2) with the aid of at least one movement model, - generation of a dynamic grid map (1) with semantic information.

Description

The invention relates to a method for generating an environment model, in particular a vehicle environment model.

Various vehicle environment models with different degrees of abstraction and detail are known from the prior art. Examples of such environment models include object lists, grid maps, parametric open space maps, elevation maps, voxel grids, multi-level surface maps or pixel worlds.

Grid maps are a widespread representation of the environment. They represent the driving environment through a number of discrete cells that surround your own vehicle. Grid cards are already used in a large number of automated driving systems and have proven themselves in practice.

A first generation of grid maps, which is still almost exclusively in use today, is only suitable for representing static driving environments. Each grid cell indicates the probability that there is a static obstacle or free space in it. Dynamic objects cause mapping errors in this first generation of grid maps.

Newer methods, so-called dynamic grid mapping methods, estimate not only a static but also a dynamic occupancy status of each grid cell, whereby a consistent representation of the static and dynamic environment can be achieved.

However, the known grid mapping methods and the resulting grid maps have some disadvantages.

It is accordingly an object of the invention to provide a method which improves the previous methods and generates a more precise environment model.

This object is achieved by the subject matter of claim 1. Further advantageous configurations and embodiments are the subject of the subclaims.

The first considerations were that the inherent disadvantage of current dynamic grid maps or grid mapping methods is that they do not contain any semantics. The semantics include e.g. the knowledge that there is a walking pedestrian or a moving vehicle in a cell. The current grid maps only contain the information that a cell is free or occupied and has a certain speed distribution.

However, the semantic information is not only of decisive importance for downstream extraction / cluster / detection approaches (e.g. for the extraction of vehicles, pedestrians or attributed free space boundaries), but also to improve the dynamic grid map creation itself, since a pedestrian e.g. moves differently than a cyclist. Since no information about the cell class (semantics) is available, current dynamic mapping approaches have to use very general cell movement models that are suitable for all classes and therefore suboptimal.

Furthermore, due to the lack of semantic information, it is currently not expedient to consider dynamic grid maps as the sole low-level intermediate representation within the environment model, since downstream driving functions require class information of objects that can no longer be derived from the limited knowledge base of current dynamic grid maps . Such a common intermediate representation, which makes this possible, is crucial, however, since inconsistencies in the generation of the final high-level environment model output representation can only be avoided by using a common internal knowledge base.

• - Erzeugen einer Gridkarte eines Umfelds eines Fahrzeugs, wobei die Gridkarte eine Vielzahl an Gridzellen aufweist,
• - Schätzen eines Belegungszustands jeder Gridzelle, wobei zwischen freien und belegten Gridzellen unterschieden wird,
• - Ermitteln von semantischen Klasseninformationen jeder freien und belegten Gridzelle,
• - Zuordnen zumindest einer semantischen Klasse mit einer bestimmten Wahrscheinlichkeit zu jeder freien und belegten Gridzelle, wobei die Klasse basierend auf den semantischen Klasseninformationen ausgewählt wird,
• - Prädizieren der belegten Gridzelle mit Hilfe mindestens eines Bewegungsmodells,
• - Erzeugen einer dynamischen Gridkarte mit semantischen Informationen.

According to the invention, a method for generating an environment model, in particular a vehicle environment model, is therefore proposed with the following steps:

• - Generating a grid map of the surroundings of a vehicle, the grid map having a plurality of grid cells,
• - Estimation of an occupancy status of each grid cell, a distinction being made between free and occupied grid cells,
• - Determination of semantic class information of each free and occupied grid cell,
• - Assignment of at least one semantic class with a certain probability to each free and occupied grid cell, the class being selected based on the semantic class information,
• - Predicting the occupied grid cell with the help of at least one movement model,
• - Generation of a dynamic grid map with semantic information.

The semantic class information contains information to specify the free and occupied grid cells in more detail and to subdivide them into semantic classes. The semantic classes In this context, can initially be subdivided into the semantic upper classes of free space upper class, including, for example, the free space sub-classes lane marking and lane, as well as upper obstacle class, comprising the obstacle sub-classes vehicle, cyclist, pedestrian, edging, lane delimitation and the like. This is advantageous because in current methods and grid maps a distinction is only made between the upper classes “obstacle”, i.e. occupied grid cell and “free space”, i.e. unoccupied grid cell, and no further subdivision takes place. However, this finer subdivision is helpful because, for example, the behavior of a pedestrian in the vehicle environment is fundamentally different from the behavior of a vehicle. It is also necessary to know where a lane is delimited by lane markings.

In a particularly preferred embodiment, the semantic class information of the free and occupied grid cells is determined by merging the data from at least two environment detection sensors. It is also conceivable to merge data from three or more sensors in order to enable a more specific subdivision. It is also advantageous to merge data from different sensors, since the different sensors can detect different attributes of objects and different features of open spaces.

For example, by merging the semantic information from different environment detection sensors, e.g. Radar, camera, lidar or ultrasound, an improved and more robust estimate of the cell classes at cell level can be achieved in every cell without having to go to object level. As a result, the main advantage of grid maps over object lists, which lies in the avoidance of error-prone object associations, is also retained in the semantic or class information fusion.

The at least one semantic class is particularly preferably represented via a probabilistic probability distribution or as a Dempster-Shafer belief mass function. This is to determine for each free and occupied grid cell with what probability the respective grid cell is a specific class.

In the case of the Dempster-Shafer Belief Mass Function, the decision frame (frame of discernment) corresponds to the chosen class hypotheses. The choice of Dampster Shafer belief mass functions allows the adequate handling of situations in which input sensors can only differentiate between upper classes and not discriminate between all subclasses and preferably enables the estimation of class hierarchies.

For example, a radar may only be able to differentiate between the upper classes "obstacle" and "free space", but not between the obstacle sub-classes "vehicle" and "pedestrian" or between the free space sub-classes "lane marking" and "lane" while a camera can break down the subclasses in more detail .

Each sensor is assigned a semantic sensor model that describes the statistical properties of the semantic input information according to its sensor characteristics and preprocessing; e.g. In cell 234, sensor 1 “measures” a pedestrian class with belief mass 0.7, a cyclist class with belief 0.2 etc. For example, sensor 2 “measures” a pedestrian class with belief 0.8 and a cyclist class with belief 0.1 for the same cell. These beliefs from the various sensors are fused with each other and result in a fused belief. In this way, a higher level of accuracy is achieved when estimating the respective semantic class.

In this step, the particle initialization required in dynamic mapping methods can already be optimized. According to the current state of the art, this is carried out identically for all object detections and is mostly modeled by an even distribution of the velocity vectors (in amplitude and direction) in the measured grid cell, since with a single sensor detection it is not yet known in which direction an object is moving which speed is moving and this information is only estimated over time by the dynamic grid mapping process as part of a cell tracking step. So that existing dynamic mapping approaches can also track fast-moving dynamic objects in a cell-based manner, the worst-case scenario must be assumed here during initialization (e.g. a cell object could move at maximum motorway speed), which negatively affects the convergence and the robustness of the estimate. If, as described in the previous embodiment, semantic information is already available for an individual measurement, the particles newly initialised by the measurement are adapted according to the cell class in the dynamic semantic grid mapping, thus improving the transient process and the robustness of the mapping.

In a further preferred embodiment, the probability distributions of non-semantic cell states are represented via particles or multi-modal Gaussian distributions.

Furthermore, the movement of each occupied grid cell is preferably predicted with the aid of a movement model as a function of a semantic class.

In a further preferred embodiment, the movement model contains a speed distribution and information about potential directions of movement and / or accelerations and / or yaw rates of the respective occupied grid cell.

If a semantic estimate is available for each grid cell, this can be used to improve the dynamic grid mapping itself. For example, the movement models can be adapted on the basis of the class information. A pedestrian can e.g. In the simplest case, they can be predicted with a movement model that has a lower maximum speed than that of a vehicle. In the general case, complex motion model predictions can be learned class-specifically.

In a further preferred embodiment, each occupied grid cell is predicted simultaneously with the aid of at least two different movement models based on the respective probability for a specific class.

Since each grid cell only belongs to a certain class with a certain probability, it is advantageous to carry out a multi-model prediction on cell level, analogous to known multi-model predictions on object level, e.g. known from interacting multiple model filters. Since current dynamic grid mapping methods are based almost exclusively on particle filter approaches, such an extension is also possible at cell level by predicting a certain proportion of particles within a cell with one movement model and another with another movement model. If the continuous cell states are not represented by particles, but alternatively by Gaussian mixed distributions (one mode per class), the prediction and the state updates can be carried out analogously to known interacting multiple model filters. In all cases, the number of movement models should ideally at least correspond to the number of potentially dynamic semantic classes. For example, in the potentially dynamic classes of vehicle, pedestrian and cyclist, three movement models should be used in parallel, which predict the particles from one cell to another. The class-specific movement models are weighted overlaid according to their class probability. If the Dempster Shafer theory is used for semantic cell class fusion, this class probability can e.g. can be derived from the merged Dempster Shafer mass functions by means of the pignistic transformation.

A dynamic 2D or 3D grid map is also preferably generated. Dynamic semantic grid maps are not limited to two spatial dimensions, but can also be extended to higher spatial dimensions in the same way as known voxel grid maps, e.g. due to the vertical height dimension as part of a 3D extension.

In a further preferred embodiment, an irregular dynamic grid map is generated. This means that the size of the individual grid cells of the grid map is not necessarily identical across the entire grid (regular grid), but quadtree (in 2D) or octree decompositions (in 3D) can also be used. This is advantageous in order to save storage space and computing time.

In a further preferred embodiment, the generated dynamic semantic grid map is used as an intermediate model of the environment.

Furthermore, the environment model initial representation is preferably derived from the environment model intermediate representation. In general, dynamic semantic grid maps represent the driving environment at a low level of fusion and can serve as consistent intermediate representations of the environment model, from which the final, more compact environment model output representation can be derived in a consistent and robust manner, since all downstream algorithms ( e.g. detectors and high-level object trackers of road users, static objects / open spaces as well as streets) on a common knowledge base - the dynamic semantic grid map.

The dynamic semantic grid map in turn represents a virtual 360 ° environment sensor with a greatly expanded measurement space (consisting of full speed measurement vectors and semantic classes) and thus an abstraction layer between the real sensor data and the final environment model output representation.

In general, an object-based environment model output representation (potentially consisting of several object classes) is derived from the grid-based environment model intermediate representation. For example, traffic participant lists, parametric open space maps and road models can be derived from dynamic semantic ground maps. In order to make this derivation as robust as possible, a sensor model is advantageously used for this purpose learned about this virtual sensor - analogous to sensor models of real sensors. Since the virtual dynamic semantic grid map sensor contains the merged information of all input sensors as well as extended estimated states such as the full speed vectors, detectors that work on its basis are advantageous compared to detectors that, for example, only on a single, individual camera, radar or lidar image work.

The environment model output representation derived from the environment model intermediate representation particularly preferably has a feedback to the environment model intermediate representation.

• 1: eine schematische Darstellung einer dynamischen semantischen Gridkarte gemäß einer Ausführungsform der Erfindung;
• 2: eine schematische Darstellung einer Mehrmodellprädiktion einer Gridzelle gemäß einer Ausführungsform der Erfindung;
• 3: eine schematische Darstellung einer dynamischen semantischen irregulären Gridkarte gemäß einer Ausführungsform der Erfindung;
• 4: eine schematische 3D-Darstellung einer dynamischen semantischen Gridkarte gemäß einer Ausführungsform der Erfindung;
• 5: eine beispielhafte Darstellung einer Dempster-Shafer Glaubensmassefunktion;
• 6: eine schematische Darstellung einer dynamischen semantischen Gridkarte als Umfeldmodell-Zwischenrepräsentation;
• 7: eine exemplarische Fusionsarchitektur;
• 8: ein schematisches Ablaufdiagramm eines Verfahrens gemäß einer Ausführungsform der Erfindung.

Further advantageous embodiments and refinements of the invention can be found in the drawings. Show in it:

• 1 : a schematic representation of a dynamic semantic grid map according to an embodiment of the invention;
• 2 : a schematic representation of a multi-model prediction of a grid cell according to an embodiment of the invention;
• 3 : a schematic representation of a dynamic semantic irregular grid map according to an embodiment of the invention;
• 4th : a schematic 3D representation of a dynamic semantic grid map according to an embodiment of the invention;
• 5 : an exemplary representation of a Dempster-Shafer belief mass function;
• 6th : a schematic representation of a dynamic semantic grid map as an intermediate model of the environment;
• 7th : an exemplary fusion architecture;
• 8th : a schematic flow diagram of a method according to an embodiment of the invention.

1 shows a schematic representation of a dynamic semantic grid map 1 according to one embodiment of the invention. The dynamic semantic grid map 1 comprises a large number of grid cells 2 . Various cell states consisting of speed and class distribution are also shown as examples. The reference number V describes a class that is assigned to a vehicle, for example, hereinafter referred to as a vehicle class V designated. Accordingly, all of the reference numbers V comprised grid cells 2 assigned to the vehicle class. Based on the class, vectors W. for the class V to be appreciated. The vectors W. are samples from the estimated speed distribution, for example realized by particle filter approaches, and represent the maximum speed by their length. Furthermore, the vectors W. different potential directions of movement are displayed. The reference number P describes a further class, which is assigned to a pedestrian, for example, hereinafter referred to as the pedestrian class P designated. Again, vectors VP based on the pedestrian class P estimated what both potential speeds and possible directions of movement of the pedestrian class P show schematically. The reference number B. describes a class that is assigned to a cyclist, for example, hereinafter referred to as a bicycle class B. designated. Even the bike class B. are vectors VB assigned.

Furthermore is a free space class F. (free space), which in this case marks a lane. The lane is shown here by the hatched area. The remaining grid cells 2 are not classified.

2 shows a schematic representation of a multi-model prediction of a grid cell 2 according to one embodiment of the invention. The starting point in this example is a grid cell 2 which is more likely to be a vehicle, less likely to be a cyclist, and even less likely to be a pedestrian. Thus, this grid cell 2 the vehicle class V , the bike class B. and the pedestrian class P assigned to each a certain probability. Those are predicted in the grid cell 2 particles with all three movement prediction models proportionally depending on the class probability. Thus, the corresponding vectors W. , VB and VP the grid cell 2 assigned.

The motion prediction models differ from one another in terms of their structure and / or parameterization. For example, unlike pedestrians, vehicles cannot change direction abruptly due to their non-holonomic constraint, but vehicles have a greater ability to accelerate. This is exemplified in particular by the different lengths and the different orientations of the respective vectors.

3 shows a schematic representation of a dynamic semantic irregular grid map 1 . In this illustration are the grid cells 2 In contrast to the previous illustrations, they are not all the same size, but there are grid cells of different sizes 2 . The other elements are the same as in 1 exemplary classes described V , P , B. as well as the associated vectors W. , VP and VB .

4th shows a schematic 3D representation of a dynamic semantic grid map 1 according to one embodiment of the invention. In this representation, the grid map 1 extended by further levels, for example to be able to display height information.

5 shows an exemplary representation of a Dempster-Shafer belief mass function. Such a belief mass fusion occurs in every grid cell 6th carried out. For the sake of simplicity it is assumed that the world consists only of vehicles (Vehicle V ), Pedestrians P ) and Freiraum (Free Space F. ) consists. As part of the Dempster Shafer Fusion, each element of the power set, which in this example case consists of 8 subsets “empty set”, V, P, F, {V, P}, {V, F}, {P, F} and {V , P, F} is initially assigned a mass function as part of a so-called Basic Belief Assignment (BBA). In this example it is assumed that a radar sensor can only differentiate between the upper classes “obstacle” and “free space” (in the diagram - here a 90% evidence for the obstacle class {V, P}, which is made up of vehicle V and pedestrians P composed) - the rest is assigned to the "unknown" mass {V, P, F}. The camera sensor, on the other hand, can discriminate between all subclasses, in this example, using upstream semantic segmentation processes (in the diagram - here a high level of evidence for the vehicle class V and a lower one for each class P and F. and unknown {V, P, F}). If these two BBAs are fused with one another in a cell, the fused belief mass results, which can be converted into a discrete probability distribution by means of the so-called pignistic transformation. In this example there is a high probability that there is a vehicle in the cell V is located. Are prior beliefs from a previous time step available (e.g. derived from multi-model particle predictions, see 2 , to the current measurement time) these are merged in analog form and the class distributions are updated recursively with the measured BBAs.

6th shows a schematic representation of a dynamic semantic grid map 1 as an intermediate model of the environment. Input data from various real environment sensors (camera, lidar, radar, sonar ...) are processed on the raw data level for the creation. If this raw data contains semantic information, for example from previous semantic segmentation, then this is taken into account in the dynamic semantic grid mapping using suitable sensor models. The dynamic semantic grid map (DSG) thus represents a virtual 360 ° environment sensor. The derivation of the environment model output representation, e.g. preferably consisting of a road user list, parametric open space map (PFS map) and road course model, is made on the basis of the common intermediate representation, which ensures consistency .

The central low-level fusion can thus be implemented using dynamic semantic grid mapping processes and fed with different raw sensor data. Downstream detector and high-level object trackers, e.g. for road users, open spaces, lanes, generate the final environment model output representation.

7th shows an exemplary fusion architecture. As shown in this figure, the high-level object tracking results are fed back to the low-level fusion. The feedback branch (dashed line) can contain, for example, ID information from tracked objects or higher states that are not generated in the context of object-free low-level fusion, for example yaw rates of tracked vehicles. The feedback can be used to further improve and support the low-level fusion.

In 8th a schematic flow diagram of a method according to an embodiment of the invention is shown. In step S1 becomes a grid map 1 of the surroundings of a vehicle, the grid map 1 a variety of grid cells 2 having. In a second step S2 becomes an occupancy status of each grid cell 2 estimated, with between free and occupied grid cells 2 A distinction is made at the same time as S2 in the step S3 semantic class information of each occupied grid cell 2 determined. These are in the following step S4 at least one semantic class, eg V, P, B, F with a certain probability for each occupied grid cell 2 assigned, the class being selected based on the semantic class information. Then step S5 each occupied grid cell with the help of one or more class-specific movement models 2 predicted for the next measurement time. Finally, in step S6 a dynamic grid map 1 with semantic information generated by a fusion step with the current measurement at cell level.

List of reference symbols

1

Grid map

2

Grid cell

B.

Bike class

F.

Free space class

P

Pedestrian class

V

Vehicle class

VB

Bicycle vector

VP

Pedestrian vector

VV

Vehicle vector

S1-S6

Procedural steps

Claims (12)

Method for generating an environment model, in particular a vehicle environment model, with the following steps: - Generating a grid map (1) of the surroundings of a vehicle, the grid map (1) having a plurality of grid cells (2), - Estimating an occupancy status of each grid cell (2), a distinction being made between free and occupied grid cells (2), - Determination of semantic class information of each free and occupied grid cell (2), - Assignment of at least one semantic class (V, P, B) with a certain probability to each free and occupied grid cell (2), the class being selected based on the semantic class information, - Predicting each occupied grid cell with the help of at least one movement model (2), - Generating a dynamic grid map (1) with semantic information. Procedure according to Claim 1 , characterized in that the semantic class information of the free and occupied grid cells (2) are determined by merging the data from at least two environment detection sensors. Procedure according to Claim 1 , characterized in that the at least one semantic class (V, P, B, F) is represented via a probabilistic probability distribution or as a Dempster-Shafer belief mass function. Procedure according to Claim 1 , characterized in that probability distributions of non-semantic cell states are represented via particles or multi-modal Gaussian distributions. Method according to one of the preceding claims, characterized in that the movement of each occupied grid cell (2) is predicted as a function of at least one class-specific movement model. Method according to one of the preceding claims, characterized in that the movement model contains a speed distribution and information about potential directions of movement and / or accelerations and / or yaw rates of the respective semantic class (V, P, B, F). Procedure according to Claim 1 , characterized in that each occupied grid cell (2) is predicted simultaneously with the aid of at least two different movement models based on the respective probability for a specific class (V, P, B). Procedure according to Claim 1 , characterized in that a dynamic 2D or 3D grid map (1) is generated. Method according to one of the preceding claims, characterized in that an irregular dynamic grid map (1) is generated. Method according to one of the preceding claims, characterized in that the generated dynamic semantic grid map (1) serves as an intermediate model of the environment. Procedure according to Claim 9 , characterized in that the environment model initial representation is derived from the environment model intermediate representation. Procedure according to Claim 11 characterized in that the environment model output representation derived from the environment model intermediate representation has a feedback to the environment model intermediate representation.

Images