DE102022207741A1 - Method and system for generating an environment model - Google Patents

Abstract

The invention relates to a method for generating an environment model in an ego vehicle (FE), comprising the following steps: - recording (S1) an environment of the ego vehicle (FE) using at least one environment detection sensor (2); - generating (S2) a first environment model (U1) based on the recording of the at least one environment detection sensor; - processing (S3) of the first environment model (U1) in a trained neural network; - generating (S4) a second environment model (U2) based on the processing of the first environment model ( U1).

Description

The invention relates to a method and a system for generating an environment model.

It is known from the prior art that different exteroceptive environment sensors such as cameras, lidar, radar or ultrasonic sensors are used in ADAS or AD systems in order to generate an adequate vehicle environment model. In experimental prototypes of automated vehicles, expensive, high-resolution 360° lidar sensors are used in particular, which, although they enable excellent environment model results, are still associated with high costs and are therefore replaced by more cost-effective solutions, especially for series vehicles.

Artificial intelligence (AI), in particular deep neural networks, are now widely used to outperform human results in specific applications, e.g. in the areas of image processing, language processing, etc. With this methodology, it is particularly possible to solve inverse problems are actually considered “impossible”, e.g. colorizing a black and white image or estimating a depth map purely based on monocular camera images.

It is therefore an object of the present invention to provide a method and a system by means of which an improved environment model can be provided cost-effectively.

This task is solved by the subject matter of claims 1 and 5. Further advantageous refinements and embodiments are the subject of the subclaims.

Initial considerations revealed that there are approaches in the field of computer graphics to optimize the image quality and calculation time of video games by first pre-calculating an image in low resolution, which is then scaled up to a higher resolution in a post-processing step. Nowadays, this is no longer done using fixed filter techniques, but rather using neural networks that have been pre-trained on the basis of many video games. It is now possible to produce a higher image quality than if the images were calculated directly in the higher, native image resolution. For example, in the US20210150669 an exemplary network for upscaling low-resolution images is shown.

This approach has now been applied at the environment model level.

• - Aufzeichnen eines Umfelds des Ego-Fahrzeugs mittels zumindest eines Umfelderfassungssensors;
• - Erzeugen eines ersten Umfeldmodells basierend auf der Aufzeichnung des zumindest einen Umfelderfassungssensors;
• - Verarbeiten des ersten Umfeldmodells in einem trainierten neuronalen Netz;
• - Erzeugen eines zweiten Umfeldmodells basierend auf der Verarbeitung des ersten Umfeldmodells.

Accordingly, according to the invention, a method for generating an environment model in an ego vehicle is proposed, comprising the following steps:

• - Recording an environment of the ego vehicle using at least one environment detection sensor;
• - Generating a first environment model based on the recording of the at least one environment detection sensor;
• - Processing the first environment model in a trained neural network;
• - Generating a second environment model based on the processing of the first environment model.

The at least one environment detection sensor can be, for example, a mono camera, a stereo camera, a radar sensor, a lidar sensor and/or an ultrasonic sensor. Preferably, not just a single sensor but several identical and/or different sensors are used in the ego vehicle to record the vehicle's surroundings.

A first environment model is now created from the data recorded by the environment detection sensor. For example, all detected objects/open spaces/lane markings/traffic signs can be entered into the environment model. Accordingly, the environment model would be created, for example, as a grid map with occupied and unoccupied grid cells (or other cell classes). Alternatively, it would also be conceivable to create an open space map, an object-based description or a combination of different models. When using several different sensors, the environment model would be created from the information from the different environment detection sensors.

The first environment model created is now processed in a trained neural network and a second environment model is created. Through processing in the neural network, the first environment model can be expanded or improved with information from the neural network in order to create an environment model that contains more precise information that is not detectable or not at a high level by the sensors available in the ego vehicle level of detail can be detected. The first and second environment models essentially (but not necessarily) have the same information, such as objects and open spaces etc., and differ mainly in the level of detail of the information representation.

The neural network is trained a priori, before use in the ego vehicle, with sensor data from high-precision environment sensors to create an AI-based virtual environment model supersensor that can improve the quality of environment models generated by simpler sensors. This can be seen in analogy to the aforementioned image upscaling networks in video games, which learn what an image should ideally look like based on high-quality image material (e.g. game scenes rendered in 16k resolution). Accordingly, to train the neural network, raw sensor data is generated in real vehicle environments using high-precision reference sensors from a vehicle fleet. An accurate, high-precision environment model is now created from these high-precision sensor raw data, possibly with the help of labeling, post-processing, etc. The neural network is then trained offline and typical environment model representations of driving environments are learned.

In a particularly preferred embodiment, the resolution of the first environment model generated is increased by the processing in the neural network. In the light of the invention, increasing the resolution of the first environment model means increasing the level of detail of the environment model generated with the help of the neural network. Accordingly, the first environment model is a low-resolution environment model and the second environment model after processing is a high-resolution environment model. This is advantageous because in this way no additional or more expensive sensors have to be used to create a highly detailed or highly accurate environment model.

In a further preferred embodiment, the neural network uses information from learned high-resolution semantic grid maps to process the first environment model. These are particularly preferably semantic, dynamic grid maps, which holistically represent static and dynamic driving environments in an object-free form. This is advantageous because such an intermediate representation within the environment model with a low level of abstraction contains more details than typical abstract environment model initial representations such as road user lists, road models, open space models, etc. In this way, the neural network learns what ideal, high-resolution semantic dynamic grid maps look like, which, as mentioned above, were generated based on highly accurate reference sensors. The neural network uses this knowledge, for example, to upscale and thus improve a poorly resolved grid map that is generated online in the ego vehicle in a post-processing step.

Furthermore, by increasing the resolution, a geometry improvement, a classification improvement, a completion of aspects that are not completely observed and/or an improvement of higher states are particularly preferably carried out on the first environment model. With the improvement in geometry, dimensions of other road users can, for example, be better determined. Furthermore, with more details, the classification of individual objects present in the environment model can be improved and a more precise distinction can be achieved, for example between different road users. The completion of incompletely observed aspects is advantageous because, with the knowledge from the high-resolution environment models, the neural network can, for example, complete lane markings, since the network knows what, for example, a solid road marking ideally looks like and at which position on the road such markings normally occur. For example, road markings that are incorrectly recognized as non-solid markings can be completed and corrected, which leads to an improved environment model and also increases safety in the ego vehicle, since driver assistance systems that use the environment model are provided with more precise and complete data. Improving higher states can be understood to mean, for example, improving a cell speed estimate. With a highly accurate environment model and a more precise determination of the other road users, their movement profiles with possible acceleration capabilities, the ability to change direction, etc. can be determined more precisely and, accordingly, it can be better predicted how the respective road user will move. For example, a cyclist has a lower acceleration capacity but a quicker ability to change direction than a car.

According to the invention, a system for generating an environment model is further proposed, comprising at least one environment detection sensor for detecting the environment of an ego vehicle, an evaluation unit for evaluating the data recorded by the environment detection sensor and a computing unit which is designed to generate a first environment model from the recording of the to create at least one environment detection sensor and to process the first environment model using a neural network and to generate a second environment model by processing the first environment model. The computing unit can be an ECU, ADCU or a computing unit integrated on the sensor side. A data connection is provided for data exchange between the environment detection sensor and the evaluation unit or between the evaluation unit and the computing unit seen. The data connection can be wired or wireless, for example as Bluetooth, mobile network or WiFi.

Recurrent neural networks, which can take temporal relationships into account, can preferably be used as neural networks. In particular, fully convolutional networks can be used for upscaling. In the case of an exemplary grid-based intermediate representation, the grid can be viewed as an image and similar network architectures can be used as when upscaling computer graphics.

Further advantageous refinements and embodiments can be seen from the drawings.

• 1: eine schematische Darstellung eines Ablaufdiagramms eines Verfahrens gemäß einer Ausführungsform der Erfindung;
• 2: eine schematische Darstellung eines Systems gemäß einer Ausführungsform der Erfindung;
• 3: eine schematische Darstellung eines Ablaufs zum Erzeugen eines Umfeldmodells.

Show in it:

• 1 : a schematic representation of a flowchart of a method according to an embodiment of the invention;
• 2 : a schematic representation of a system according to an embodiment of the invention;
• 3 : a schematic representation of a process for creating an environment model.

1 shows a schematic representation of a flowchart of a method according to an embodiment of the invention. In step S1, an environment of the ego vehicle FE is recorded using at least one environment detection sensor 2. In a further step S2, a first environment model U1 is generated based on the data from the recording of the at least one environment detection sensor. This first environment model U1 is processed in a trained neural network in step S3. Finally, in step S4, a second environment model U2 is generated based on the processing of the first environment model U1.

In 2 a schematic representation of a system according to an embodiment of the invention is shown. The system 1 has at least one environment detection sensor 2, an evaluation unit 3 and a computing unit 4. The at least one environment detection sensor 2, the evaluation unit 3 and the computing unit 4 are connected by means of a data connection D. The data connection D can be designed to be wired or wireless.

3 shows a schematic representation of a process for creating an environment model. As shown in the illustration, the neural network is trained with an environment model UF, which is generated by the sensor data of a vehicle fleet FF. The FF vehicle fleet is equipped with high-precision sensors. Accordingly, the UF environment model has highly accurate data. As shown, the environment model UF can be a semantic dynamic grid map in which, for example, dynamic and static objects as well as open spaces and lanes are entered (as well as potentially other information relevant to the driving task). This trained neural network is then used in the ego vehicle FE. The ego vehicle FE records the environment with at least one environment detection sensor and generates a first environment model U1. The ego vehicle FE does not have high-resolution sensors. Accordingly, the first environment model U1 is a low-resolution environment model U1. After processing the first environment model U1 in step S4 in the trained neural network, a second environment model U2 is generated in step S5. The environment model U1 was expanded and optimized in the neural network with the high-precision data and scaled up accordingly, so that the second environment model U2 is a high-resolution environment model.

Reference symbol list

1

system

2

Environment detection sensor

3

Evaluation unit

4

Computing unit

D

Data Connection

FE

Ego vehicle

FF

vehicle fleet

S1-S4

Procedural steps

UF

Vehicle fleet environment model

U1

first environment model

U2

second environment model

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 20210150669 [0006]

Claims (5)

Method for generating an environment model in an ego vehicle (FE) comprising the following steps: - Recording (S1) an environment of the ego vehicle (FE) using at least one environment detection sensor (2); - Generating (S2) a first environment model (U1) based on the recording of the at least one environment detection sensor; - Processing (S3) of the first environment model (U1) in a trained neural network; - Generating (S4) a second environment model (U2) based on the processing of the first environment model (U1). Procedure according to Claim 1 , characterized in that the resolution of the generated first environment model (U1) is increased by the processing in the neural network. Method according to one of the preceding claims, characterized in that the neural network uses information from learned high-resolution semantic grid maps (UF) to process the first environment model (U1). Method according to one of the preceding claims, characterized in that by increasing the resolution, a geometry improvement, a classification improvement, a completion of aspects that are not completely observed and/or an improvement of higher states are carried out on the first environment model (U1). System (1) for generating an environmental model comprising at least one environment detection sensor (2) for detecting an environment of an ego vehicle (FE), an evaluation unit (3) for evaluating the data recorded by the environment detection sensor (2) and a computing unit (4), which is designed to create a first environment model (U1) from the recording of the at least one environment detection sensor and to process the first environment model (U1) by means of a neural network and to generate a second environment model (U2) by processing the first environment model (U1).

Images