DE102020210379A1 - Computer-implemented method and computer program product for obtaining a representation of surrounding scenes for an automated driving system, computer-implemented method for learning a prediction of surrounding scenes for an automated driving system and control unit for an automated driving system - Google Patents

Abstract

Computer-implemented method for obtaining an environment scene representation (HSRV) for an automated driving system (R) comprising the steps: obtaining environment features from real and/or virtual environment sensor data of the driving system (R) (V1) and arranging the environment features in the scene representation ( HSRV) (V2) comprising a plurality of layers (AH) arranged in spatial relation, each comprising static (stat) or dynamic (dyn) environmental features, the static (stat) environmental features being regional information, position data of the driving system and/or the environmental features, traffic regulation information, traffic signs and anchor trajectories, the dynamic (dyn) environmental features include semantic information and movement information of the road users and the driving system (R) is regulated and/or controlled (V3) based on the scene representation (HSRV).

Description

The invention relates to a computer-implemented method and a computer program product for obtaining a representation of surrounding scenes for an automated driving system, a computer-implemented method for learning a prediction of surrounding scenes for an automated driving system, and a control unit for an automated driving system.

In the context of AD/ADAS applications, but also in the context of Industry 4.0 and collaborative human-robot interaction, purely sensory detection of the environment is not sufficient. Rather, the temporal prediction of the further development of the dynamic scene with all its independent interactors, e.g. people, vehicles, cyclists, is becoming increasingly important in order to be able to make intelligent decisions for automated vehicles, for example. Not only the interaction of all interactors, for example road users, is important here, but also the interaction of these with their direct environment, for example the traffic area and/or the infrastructure.

The environment is characterized by a large number of explicit, visible signs and markings, such as traffic signs, lane markings, curbs, roadsides, which are coupled with regionally different meanings, rules and real behavior and with a large number of underlying rules and standards, which do not visibly determine the behavior of the interactors in the environment, such as when an emergency vehicle approaches from behind, an emergency lane must be formed. On the one hand, these rules are applied very differently from region to region and, on the other hand, they depend on accompanying events, such as the approach of an emergency vehicle in an acute traffic jam situation in the previous example.

In order to ensure reliable and powerful scene prediction, all of these explicit, implicit, regional and event-driven rules/information must be considered and used for temporal prediction.

In the prior art, occupancy grids are known, a map-like representation of the static environment and road users located therein, see for example EP 2 771 873 B1 . Spatial dependencies can be detected by means of such grid representations. The disadvantage is that additional semantic information is usually not recorded or has to be managed separately.

The invention is based on the object of enabling improved movement planning for an intelligent agent including automated driving systems

The methods according to claims 1 and 8, the computer program product according to claim 7 and the control unit according to claim 12 each solve this task. The environment scene representation according to the invention represents a hybrid representation. The further processing based on this representation, in order to enable, for example, a chronological prediction of all road users over several time steps into the future, becomes faster, more efficient, more powerful, more precise, less error-prone, more robust and reliable. With the environment scene representation according to the invention, the advantages of the spatial and the semantic representation are brought into harmony with one another in an intelligent manner.

• • Erhalten von Umfeldmerkmalen aus realen und/oder virtuellen Umfeldsensordaten des Fahrsystems und
• • Anordnen der Umfeldmerkmale in der Szenen-Repräsentation umfassend mehrere in räumlicher Relation angeordnete Schichten umfassend jeweils statische oder dynamische Umfeldmerkmale.

One aspect of the invention relates to a computer-implemented method for obtaining a representation of an environment scene for an automated driving system, comprising the steps

• • Obtaining environmental features from real and/or virtual environmental sensor data of the driving system and
• • Arranging the environment features in the scene representation comprising several layers arranged in spatial relation, each comprising static or dynamic environment features.

The static environment features include regional information, position data of the driving system and/or the environment features, traffic regulation information, traffic signs and anchor trajectories. The dynamic environment features include semantic information and movement information of road users. The driving system is regulated and/or controlled based on the scene representation.

A further aspect of the invention relates to a computer program for obtaining a representation of an environment scene for an automated driving system. The computer program comprises instructions that cause a computer to carry out a method according to the invention when the program is run on the computer.

A further aspect of the invention relates to a computer-implemented method for learning a prediction of environmental scenes for an automated driving system. In this case, a machine learning algorithm receives the environmental scene representations obtained according to a method according to the invention together with the respective reference predictions as input data pairs. Based on these pairs of input data, the gradient-based prediction is learned from the surrounding scene representations.

A further aspect of the invention relates to a control unit for an automated driving system. The control unit includes first interfaces via which the control unit receives environmental sensor data from the driving system. Furthermore, the control unit includes a processing unit, which determines environmental features from the environmental sensor data, executes a machine learning algorithm learned according to a method according to the invention and receives predicted environmental scenes and, based on the predicted environmental scenes, determines regulation and/or control signals for automated operation of the driving system. In addition, the control unit includes second interfaces via which the control unit provides the control and/or control signals to actuators for longitudinal and/or lateral guidance of the driving system.

Advantageous refinements of the invention result from the definitions, the dependent claims, the drawings and the description of preferred exemplary embodiments.

Computer-implemented means that the steps of the method are executed by a data processing device, for example a computer, a computing system, a computer network, for example a cloud system, or parts thereof.

Automated driving systems include automated vehicles, road vehicles, people movers, robots and drones.

Environmental features include houses, streets, in particular street geometry and/or condition, signs, lane markings, vegetation, moving road users, vehicles, pedestrians, cyclists.

Surroundings sensor data includes raw data and/or data pre-processed, for example with filters, amplifiers, serializers, compression and/or conversion units, from cameras, radar sensors, lidar sensors, ultrasonic sensors, acoustic sensors, Car2X units and/or real-time/offline maps arranged on the driving system. According to one aspect of the invention, the surroundings sensor data are actually data entered with the driving system. According to a further aspect of the invention, the environmental sensor data includes virtually generated data, for example using software, hardware, model and/or vehicle-in-the-loop methods. According to a further aspect of the invention, the environmental sensor data is real data that has been virtually augmented and/or varied.

The environmental features are obtained from the environmental sensor data using object classifiers, for example artificial neural networks for semantic image segmentation.

The environment scene representation layers a scenario into several layers. A real scenario is presented as a hybrid of static and dynamic and thus semantic information. In this context, the environment scene representation according to the invention is also called Hybrid Scene Representation for Prediction, abbreviated HSRV. For example, the scenario is an image with i pixels in the x-direction and j pixels in the y-direction. The individual layers can also be represented as images and are arranged congruently with one another, for example the layers are spatially congruently one on top of the other. The environmental scene representation according to the invention can be imagined as a stack of digital photos lying one on top of the other, for example taken from a bird's-eye view of an intersection. On the other hand, this stack of images is combined with further layers of partly purely semantic information that is represented, for example, as pure feature vectors.

Static environmental characteristics are divided into two further categories. Elements that do not change or only change after a long period of time do not change their status in the short term and are rigidly labeled. Of course, HRSV also provides for an adaptation of these elements if, for example, there is a change in traffic routing. However, this aspect of adaptation takes place on a different time scale. Road markings are an example of this. In contrast, there are elements that can change state frequently and are therefore state-changing. Traffic lights or variable message signs, for example, are classified in the latter category.

The behavior of road users differs depending on the region. For example, in Germany traffic rules are relatively strictly observed, in Italy rather mildly, in Great Britain overtaking is done from the right, etc.

Position data of the driving system and/or the environmental features are recorded via map information. A map section is formed by assigning a value to each pixel of the map information corresponding layer of the environment scene representation. The values are based on discrete labels of the map, e.g. numeric codes for street, walkway, broken line, double line, etc.

In addition to the map, the right of way rules are shown via the traffic regulation information. A line is drawn in the middle of each lane. Additional lines are drawn at intersections, representing all permissible maneuvers. According to one aspect of the invention, implicitly regulated information such as "right before left" is overlaid with the signage. Any conflicting rule information is aggregated to form a consistent rule in this layer, so that the rules then in effect are treated as having priority.

Traffic advisors include state-changing and stateful traffic advisors. Status-changing traffic signs are usually used to summarize signals that are passed on to the driver visually and that can change their status several times in the course of a day. Examples of this category are traffic lights, variable message signs on motorways and entry signs at toll booths. These traffic signs are represented as a pixel value representing the current state in the spatial context of the local scene representation. For reasons of redundancy, such pixel regions are generally not limited to one pixel, but rather mapped to a larger number of pixels. The exact size of the expansion is mostly learned from data to an optimum.

The anchor trajectories combine information from the right of way rules and from the status-changing traffic signs. According to one aspect of the invention, the anchor trajectories determined in this way are brought into line with the rules of the status-changing traffic indicators and prioritized accordingly. According to one aspect of the invention, the layer of the anchor trajectories can supplement or replace the layers of traffic instructions and/or traffic regulation information, depending on the time required of the driving system.

The computer program instructions include software and/or hardware instructions. The computer program is loaded into a memory of the control device according to the invention, for example, or is already loaded into this memory.

According to a further aspect of the invention, the computer program according to the invention is executed on hardware and/or software of a cloud facility.

The computer program is loaded into the memory, for example, by a computer-readable data carrier or a data carrier signal. The invention is thus also implemented as an aftermarket solution.

The control unit, abbreviated as ECU, prepares input signals, processes them using an electronic circuit and provides logic and/or power levels as regulation and/or control signals. The control device according to the invention is scalable for assisted driving through to fully automated/autonomous/driverless driving.

According to one aspect of the invention, the control unit receives raw data from sensors and includes an evaluation unit that processes the raw data for HSRV. According to a further aspect of the invention, the control unit receives pre-processed raw data. According to a further aspect of the invention, the control unit includes an interface to an evaluation unit that processes the raw data for HSRV.

According to a further aspect of the invention, the control unit includes a software and/or hardware level for trajectory planning or high-level controlling. After this level, the signals are then sent to the actuators.

The processing unit includes, for example, a programmable electronic circuit. According to one aspect of the invention, the processing unit or the control device is designed as a system-on-chip.

• • eine erste Schicht umfassend die regionalen Informationen zum Verhalten der Verkehrsteilnehmer und/oder Wetterinformationen,
• • eine zweite Schicht umfassend Karteninformationen zur Bestimmung der Position des Fahrsystems,
• • eine dritte Schicht umfassend die Verkehrsregelinformationen,
• • eine vierte Schicht umfassend die Verkehrsweiser,
• • eine fünfte Schicht umfassend die Ankertrajektorien,
• • eine sechste Schicht umfassend semantisch-explizite Informationen,
• • eine siebte Schicht umfassend semantisch-latente Informationen und
• • eine achte Schicht umfassend die Bewegungsinformationen.

According to one aspect of the invention, the scene representation includes:

• • a first layer comprising the regional information on the behavior of road users and/or weather information,
• • a second layer containing map information for determining the position of the driving system,
• • a third layer comprising the traffic rule information,
• • a fourth layer comprising traffic signs,
• • a fifth layer comprising the anchor trajectories,
• • a sixth layer comprising semantically explicit information,
• • a seventh layer comprising semantic-latent information and
• • an eighth layer comprising the movement information.

• • die regionalen Informationen und/oder die Wetterinformationen in Form von Codes bereitgestellt oder ein Maschinenlernalgorithmus lernt über eine Eingabe von globalen Koordinaten und Fahrdaten des Fahrsystems einen Zusammenhang zwischen Region und Fahrverhalten,
• • die Position des Fahrsystems zu einem bestimmten Zeitpunkt aus einem Kartenausschnitt bestimmt und der Kartenausschnitt für jeden neuen Zeitschritt generiert oder der Kartenausschnitt nach einer vorgegebenen Anzahl von Zeitschritten aktualisiert, wobei jedem Pixel der zweiten Schicht ein Wert der Karte zugeordnet wird,
• • die Verkehrsregelinformationen mittels aus den Umfeldsensordaten erfassten Verkehrsschildern und/oder aus den regionalen Informationen abgeleiteten Verkehrsregeln bestimmt,
• • ein Zustand der Verkehrsweiser als Pixelwert in der vierten Schicht dargestellt,
• • die Ankertrajektorien, die nach einem Aspekt der Erfindung von einem Verkehrsteilnehmer erreichbaren Fahrbahnlinien umfassen, in Abhängigkeit der Verkehrsweiser priorisiert,
• • die semantischen Informationen in Form von Merkmalsvektoren dargestellt und/oder
• • die Bewegungsinformationen mittels eines Maschinenlernalgorithmus über Zeitschritte gelernt und bestimmt und räumlich dargestellt.

According to a further aspect of the invention

• • the regional information and/or the weather information is provided in the form of codes or a machine learning algorithm learns a connection between the region and driving behavior by entering global coordinates and driving data of the driving system,
• • the position of the driving system is determined from a map section at a specific point in time and the map section is generated for each new time step or the map section is updated after a specified number of time steps, with each pixel of the second layer being assigned a value on the map,
• • the traffic regulation information is determined by means of traffic signs recorded from the environmental sensor data and/or traffic regulations derived from the regional information,
• • a status of the traffic sign displayed as a pixel value in the fourth layer,
• • the anchor trajectories, which according to one aspect of the invention include lane lines that can be reached by a road user, are prioritized depending on the traffic signs,
• • the semantic information is presented in the form of feature vectors and/or
• • The movement information is learned and determined using a machine learning algorithm via time steps and displayed spatially.

Adding the regional information, for example in the form of a country code from a table, leads to an improvement in the prediction quality. Each region is represented by a specific country or region code. Equivalently, the current weather situation is processed via a weather code. This code can also be made available to the machine learning algorithm globally, i.e. not across a layer. The machine learning algorithm thus has the opportunity to learn the real connections between region and/or weather and actual driving behavior. Alternatively, to preserve the spatial aspect even with a convolutional network, the same regional value is assigned to each pixel in a layer. With regard to the regional information, one option is to learn a connection between the region and driving behavior directly via the global coordinates instead of a country code, and thus not having to carry out an expert-based delimitation of regions. Such country or weather codes usually remain the same for the same place over a longer period of time, so they are rigid. The weather changes, of course, and this change is taken into account, but usually the change is also somewhat slower than, for example, the changes in a traffic light and even slower than the movements of other road users.

For example, country codes are obtained from the following look-up table: Regional information description traffic rules 0 no - 1 Germany strong 2 Italy mild 3 Great Britain Vice versa, strong 4 USA Strong, different 5 Congo Unknown, different ... ... ...

For example, pixel values for traffic lights are taken from the following look-up table: traffic light phases description 0 green 1 yellow 2 Red 3 Yellow Red 4 Traffic light is flashing 5 Traffic light is completely off 6 Red Right Arrow 7 Red Left Arrow ... ...

According to a further aspect of the invention, street line types are taken from the following look-up table, for example: street lines description 0 no 1 Broken Lines 2 solid lines 3 double lines 4 bike lines ... ...

In addition to the dynamic behavior, a semantic description of the road users and their characteristics is also helpful for predicting the movement. For example, a truck has different dynamic behavior limitations than a cyclist. According to one aspect of the invention, semantic information is bundled into a feature vector. Examples of this type of information are the vehicle class, for example truck, car, motorcycle, bicycle, pedestrian, the height and width of the objects or states of the flashing lights, for example right, left, warning, off. Descriptors describe these properties, i.e. they generate the feature vectors for input into a machine learning algorithm. These descriptors are arranged in the same way as the dynamic information descriptors and form the semantically explicit information layer.

Optionally, additional semantic information of the road user is realized through the use of latent feature vectors. Latent means that the information cannot be directly interpreted by humans, but is in some way implicitly contained in the data. According to one aspect of the invention, these latent feature vectors are calculated using artificial deep neural networks accomplished. For example, object classifiers, which are upstream of the environmental scene representation according to the invention, are implemented as artificial deep neural networks. In general, such a latent feature vector is generated as an intermediate product during classification. These latent intermediate vectors of all road users are spatially arranged in the manner described above and form the layer of semantic-latent information. According to one aspect of the invention, the semantically explicit layer is supplemented with the semantically latent layer. An advantage of the semantically latent information is the robustness against noise signals of discrete classes. When the discrete classification varies between two classes, such as truck and passenger car, it is difficult to correctly interpret the class information. Because the latent feature vector is a vector of continuous numbers, fluctuations have little to no effect and allow for a more robust interpretation of the object's semantic information.

The dynamic part describes the moving road users in the scene. The coordinates of the road users are used over a certain period of time to generate a descriptor for this dynamic movement behavior. Driving behavior can also be contained latently. The calculation of this descriptor is learned on the one hand by means of an artificial deep neural network, for example a network comprising long-short-term memory layers, LSTM for short. With LSTMs, after a settling phase, an iterative adjustment of the descriptor is only possible by entering the coordinates of the next time step. According to one aspect of the invention, parameters of a vehicle dynamics or movement dynamics model are used here, for example by means of a Kalman filter. The descriptors of all road users are spatially arranged based on the last coordinate and form the layer of movement information.

According to a further aspect of the invention, the environmental features are represented in pixels of the layers and/or via feature vectors with spatial anchor points. The feature vectors have a predetermined spatial anchor point. According to a further aspect of the invention, the environmental features are interpreted as color values of the pixels.

According to a further aspect of the invention, a spatial position of the environment features is recorded in each layer via a corresponding position on a map.

This is advantageous for a spatially corresponding arrangement of the environmental features.

According to a further aspect of the invention, spatial coordinates of the driving system and/or the environmental features are represented in pixels, with one pixel in each of the layers corresponding to the same route length.

In one embodiment of the method according to the invention for learning a prediction, a number of environment scene representations are provided, which depict the static and dynamic environment features including the road users over a variable number of x time steps.

The machine learning algorithm is trained, validated and tested using these environment scene representations. During the validation, meta-parameters included in the learning process are adjusted appropriately. During the test phase, the prediction of the learned machine learning algorithm is evaluated.

According to the invention, the environment scene representation is coupled to the neural structures. The advantage of the environmental scene representation according to the invention is that a very large and very flexible amount of information is provided which the machine learning algorithm can access. Within the learning phase, in which the variable parameters/weights of the machine learning algorithm are adjusted, the use of the specific information that is best suited to perform the task of prediction then emerges.

• • die Umfeldszenen-Repräsentationen encodiert in ein Faltungsnetzwerk eingegeben werden,
• • das Faltungsnetzwerk lernt, Interaktionen zwischen den Schichten der Umfeldszenen-Repräsentation, Interaktionen zwischen Verkehrsteilnehmern und/oder Interaktionen zwischen Verkehrsteilnehmer und Umfeldmerkmalen darzustellen und in Form eines Ausgabevolumens, dessen Höhe und Breite gleich der Größe der Umfeldszenen-Repräsentation ist, auszugeben, wobei aus dem Ausgabevolumen für jeden Verkehrsteilnehmer eine Spalte basierend auf der Pixel-diskreten Position des Verkehrsteilnehmers ermittelt wird und die Spalte mit einem Vektor, der das dynamische Verhalten beschreibt, konkateniert wird,
• • aus dem Konkatenieren erhaltenen zusammengesetzten Merkmalsvektoren in prädizierte Trajektorien des Fahrsystems und/oder der Verkehrsteilnehmer decodiert werden.

In a further embodiment of the method according to the invention for learning a prediction of environmental scenes, the machine learning algorithm comprises an encoder-decoder structure,

• • the environmental scene representations are encoded and entered into a convolution network,
• • the convolution network learns to represent interactions between the layers of the environment scene representation, interactions between road users and/or interactions between road users and environment features and in the form of an output volume whose height and width are equal the size of the environment scene representation is to be output, with a column being determined from the output volume for each road user based on the pixel-discrete position of the road user and the column being concatenated with a vector that describes the dynamic behavior,
• • Composite feature vectors obtained from the concatenation are decoded into predicted trajectories of the driving system and/or the road users.

According to a further aspect of the invention, the encoders and/or decoders are based on long-short-term memory technology.

According to a further aspect of the invention, noise vectors are concatenated by generative adversarial learning and different trajectories in the future are generated by different noise vectors for identical trajectories in the past. This captures multimodal uncertainties of predictions.

In one embodiment of the invention, the machine learning algorithm is a multi-agent tensor fusion encoder-decoder. A multi-agent tensor fusion encoder-decoder for static environmental scenes is disclosed in arXiv: 1904.04776v2 [cs.CV]. The invention provides a multi-agent tensor fusion algorithm for the environment scene representation according to the invention, which also includes dynamic environment features in addition to static environment features. The multi-agent tensor fusion algorithm according to the invention does not receive static environmental scenes as input, but rather the HSRV containing dynamic environmental features.

An encoder-decoder LSTM network is particularly well suited to solving sequence-based problems.

According to one aspect of the invention, the noise vectors are generated by a generative adversarial network, abbreviated GAN, for example by the GAN disclosed in arXiv: 1904.04776v2 [cs.CV] under point 3.3.

• 1 eine Darstellung einer erfindungsgemäßen Umfeldszenen-Repräsentation,
• 2 eine Darstellung eines erfindungsgemäßen Maschinenlernverfahrens,
• 3 eine Darstellung eines erfindungsgemäßen Steuergeräts und
• 4 eine Darstellung des erfindungsgemäßen Verfahrens zum Erhalten der Umfeldszenen-Repräsentation aus 1.

The invention is illustrated in the following exemplary embodiments. Show it:

• 1 a representation of an environment scene representation according to the invention,
• 2 a representation of a machine learning method according to the invention,
• 3 a representation of a control unit according to the invention and
• 4 shows a representation of the method according to the invention for obtaining the environment scene representation 1 .

In the figures, the same reference symbols denote the same or functionally similar reference parts. For the sake of clarity, only the relevant reference parts are highlighted in the individual figures.

1 shows an example of a surrounding scene representation HSRV according to the invention. In the environment scene E shown, there is a car as an example of a driving system R at a junction. At the junction there is a pedestrian W. The right of way is controlled by a traffic light L. The traffic light circuit L shows the car R the green traffic light phase and the pedestrian W the red one. The various layers that are essential for the prediction of the trajectories of the road users are shown above the representation of this situation from a bird's eye view. Layer A shows the regional information. Layer B uses the map information, layer C the traffic regulation information. The stateful traffic signs and the anchor trajectories are contained in layer D and layer E. Layer F describes the semantic characteristics of the individual road users. Layer G and layer H contain latent information, with this information in layer G being based on properties that describe the road user and in layer H on the dynamic movement behavior.

Layers A to E are static layers and describe static environmental features stat of environmental scene E. Layers A to C describe rigid static environmental features stat_1 and layers D and E state-changing static environmental features stat_2.

The layers F to H are dynamic layers and describe dynamic environment features dyn of the environment scene E.

2 shows an exemplary architecture of an artificial deep neural network DNN that receives the environment scene representation HSRV as input. The environment scene representation HSRV is entered into the network DNN as a feature volume. For example, the network DNN includes a convolutional network-encoder-decoder structure, which uses multi-agent tensor fusion to control the interactions between the various layers AH and, due to its architecture based on filter masks, the interactions with elements of the environment scene representation located in the environment HSRV are modeled. A feature volume results from the network DNN, where height and width correspond to the input volume. The input volume is the environment scene representation HSRV. A column is now selected for each road user from the output volume V and concatenated with the vector that describes the dynamic behavior and a noise vector. The column is determined based on the quantized position of the road user. The assembled feature vectors are now each fed into an LSTM decoder. This decoder then generates the future trajectory for each road user. Since different noise vectors are concatenated in the training according to the GAN setup, different noise vectors for identical trajectories in the past can be used in the inference to generate different trajectories in the future.

This in 3 The control unit ECU shown receives environmental sensor data U via first interfaces INT 1 , for example from one or more cameras of the driving system R. A processing unit P, for example a CPU, GPU or FPGA, executes object classifiers and uses the environmental sensor data U to determine the static and/or or dynamic environmental characteristics stat and dyn. The processing unit P processes the environmental features using a machine learning algorithm learned according to the invention and obtains predicted environmental scenes. Based on the predicted environmental scenes, the processing unit P determines regulation and/or control signals for automated operation of the driving system R. The control unit ECU uses second interfaces INT 2 to provide the regulation and/or control signals to actuators for longitudinal and/or lateral guidance of the driving system R ready.

4 shows the steps of the method for obtaining a surround scene representation. In step V1, the environmental features stat and dyn are obtained. In step V2, the slices AH are generated with the respective environment features stat and dyn. In step V3, the driving system R is regulated and/or controlled based on the scene representation HSRV.

Reference List

HSRV

Environment Scene Representation

stat

static environment feature

stat_1

statically rigid environmental features

stat_2

static state-changing environmental features

dynamic

dynamic environment feature

AH

layers

E

environment scene

u

environmental sensor data

V

output volume

W

pedestrian

L

traffic light

DNN

artificial deep neural network

V1-V3

process steps

ECU

control unit

INT1

first interfaces

INT2

second interfaces

P

processing unit

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• EP 2771873 B1 [0005]

Claims (12)

Computer-implemented method for obtaining an environment scene representation (HSRV) for an automated driving system (R) comprising the steps • obtaining environment features from real and/or virtual environment sensor data (U) of the driving system (R) (V1) and • Arranging the environment features in the scene representation (HSRV) (V2) comprising a plurality of spatially arranged layers (A-H) each comprising static (stat) or dynamic (dyn) environment features, wherein • the static (stat) environment features include regional information, position data of the driving system and/or the environment features, traffic regulation information, traffic signs and anchor trajectories, • the dynamic (dyn) environment features include semantic information and movement information of road users and • the driving system (R) is regulated and/or controlled (V3) based on the scene representation (HSRV). procedure after claim 1 , wherein the scene representation (HSRV) comprises: • a first layer (A) comprising the regional information on the behavior of road users and/or weather information, • a second layer (B) comprising map information for determining the position of the driving system, • a third Layer (C) comprising the traffic rule information, • a fourth layer (D) comprising the traffic signs, • a fifth layer (E) comprising the anchor trajectories, • a sixth layer (F) comprising semantically explicit information, • a seventh layer (G) comprising semantic-latent information and • an eighth layer (H) comprising the movement information. procedure after claim 1 or 2 , where • the regional information and/or the weather information is provided in the form of codes or where a machine learning algorithm learns a connection between region and driving behavior by entering global coordinates and driving data of the driving system, • the position of the driving system at a specific point in time from a map section is determined and the map section is generated for each new time step or the map section is updated after a specified number of time steps, with each pixel of the second layer being assigned a value based on the class of the map segment, • the traffic regulation information using traffic signs recorded from the environmental sensor data and/or traffic rules derived from the regional information are determined, • a status of the traffic sign is displayed as a pixel value in the fourth layer (D), • the anchor trajectories are those that can be reached by a road user include lane lines and/or are prioritized depending on the traffic sign, • the semantic information is presented in the form of feature vectors and/or • the movement information is learned and determined using a machine learning algorithm over time steps and is presented spatially. Procedure according to one of Claims 1 until 3 , whereby the environmental features (stat, dyn) are represented in pixels of the layers (AH) and/or via feature vectors with spatial anchor points. Procedure according to one of Claims 1 until 4 , where in each layer (AH) a spatial position of the environmental features (stat, dyn) is recorded via a corresponding position on a map. Procedure according to one of Claims 1 until 5 , wherein spatial coordinates of the driving system (R) and/or the environmental features (stat, dyn) are represented in pixels, one pixel in each of the layers (AH) corresponding to the same route length. Computer program for obtaining an environment scene representation (HSRV) for an automated driving system (R) comprising instructions that cause a computer to perform one of the methods of Claims 1 until 6 executes when the program is run on the computer. Computer-implemented method for learning a prediction of environmental scenes (E) for an automated driving system (R), with a machine learning algorithm according to one of the methods of claims che 1 until 6 received environment scene representations (HSRV) together with respective reference predictions as input data pairs and based on these input data pairs gradient-based the prediction from the environment scene representations (HSRV) learns. procedure after claim 8 , where the machine learning algorithm comprises an encoder-decoder structure, where • the environment scene representations (HSRV) are encoded and input into a convolution network, • the convolution network learns interactions between layers (AH) of the environment scene representation (HSRV), interactions between road users and/or interactions between road users and environmental features (stat, dyn) and in the form of an output volume (V), whose height and width is equal to the size of the environment scene representation, to output, with the output volume (V) for each road user a column is determined based on the pixel-discrete position of the road user and the column is concatenated with a vector that describes the dynamic behavior, • composite feature vectors obtained from the concatenation are decoded into predicted trajectories of the driving system (R) and/or the road users. procedure after claim 9 , the encoders and/or decoders being based on long-short-term memory technology. procedure after claim 9 or 10 , where noise vectors are concatenated by generative adversarial learning and different trajectories in the future are generated by different noise vectors for identical trajectories in the past. Control unit (ECU) for an automated driving system (R) comprising • first interfaces (INT1) via which the control unit (ECU) receives environmental sensor data from the driving system (R), • a processing unit (P) which, from the environmental sensor data, generates environmental features (S, D ) determines a by one of the methods of Claims 7 until 10 executes the learned machine learning algorithm and receives predicted environmental scenes and, based on the predicted environmental scenes, determines regulation and/or control signals for automated operation of the driving system (R), and • second interfaces (INT2), via which the control unit (ECU) transmits the regulation and/or or provides control signals to actuators for longitudinal and/or lateral guidance of the driving system (R).

Images