DE102022004426A1 - Method for predicting movements of road users - Google Patents

Abstract

The invention relates to a method for predicting movements of road users, wherein a model (2) is created for the prediction, which is trained with at least one predetermined or dynamically created trajectory set (S, S1, S2). According to the invention, road users to be taken into account in the prediction are classified as agents according to their type, wherein the road users are each assigned to at least one of several agent classes (C1 to Cm) during the classification. The training is carried out individually for each agent class (C1 to Cm) using at least one trajectory set (S, S1, S2), wherein the respective trajectory set (S, S1, S2) is created dynamically specifically for the respective agent class (C1 to Cm) or is selected from a predetermined training data set.

Description

The invention relates to a method for predicting movements of road users according to the preamble of claim 1.

The prediction of movements of road users, such as vehicles, is an essential component in the operation of automated vehicles, such as highly automated or autonomous vehicles. Reliable and safe movement planning is only possible if the prediction is also of high quality.

Out of " Tung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijbom, Eric M. Wolff - CoverNet: Multimodal Behavior Prediction using Trajectory Sets; arXiv:1911.10298 “ is a method for multimodal, probabilistic trajectory prediction for urban traffic.

Furthermore, “ Yuning Chai, Benjamin Sapp, Mayank Bansal, Dragomir Anguelov - MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction; arXiv:1910.05449 “ a method for motion planning of an autonomously driving vehicle is known in which human behavior is predicted.

The invention is based on the object of specifying a novel method for predicting movements of road users.

The object is achieved according to the invention by a method which has the features specified in claim 1.

Advantageous embodiments of the invention are the subject of the subclaims.

In a method for predicting movements of road users, a model is created which is trained with at least one predefined or dynamically created set of trajectories.

According to the invention, road users to be taken into account in the prediction are classified as agents according to their type, with the road users being assigned to at least one of several agent classes during the classification. The training is carried out individually for each agent class using at least one trajectory set, with the respective trajectory set being dynamically created specifically for the respective agent class or selected from a predetermined training data set.

Using the method, a significant improvement in the prediction quality of set-based trajectory prediction approaches can be achieved by incorporating prior knowledge about the agent class. In particular, it is possible to predict trajectories that can be kinematically driven by the respective agent in the future. In contrast to the case where no agent-specific trajectory set is used, it can be avoided that a trajectory set contains movements that the respective road user cannot kinematically perform at all. If, for example, a single fixed trajectory set were used for several agent classes, there would be a risk that trajectories would be predicted for passenger cars, for example, that originally belonged to pedestrians, such as turning on the spot. This is avoided by the present method. If, for example, a single dynamic trajectory set with a kinematic bicycle model were used for all agent classes, there would be a risk that unrealistic trajectories would be predicted for trucks, for example, since their movement cannot typically be covered by a bicycle model. This is also avoided by the present method.

In a possible embodiment of the method, the dynamic creation of the at least one trajectory set is additionally carried out depending on an environmental state and/or visual features and/or a driving behavior of the road user. This can further improve the prediction quality of set-based trajectory prediction approaches.

In a further possible embodiment of the method, the selection of at least one trajectory set from the given training data set is additionally carried out depending on an environmental condition and/or visual features and/or driving behavior of the road user. This can further improve the prediction quality of set-based trajectory prediction approaches.

• - Wetterbedingungen und/oder
• - Umgebungslichtbedingungen, beispielsweise tiefstehende Sonne, und/oder
• - einer Fahrbahnbeschaffenheit, beispielsweise durch Erkennung von Niederschlägen, wie zum Beispiel Regen und/oder Schnee und/oder Eis, auf einer Fahrbahnoberfläche,

die visuellen Merkmale mittels

• - Lichtzeichen von Verkehrsanlagen, beispielsweise Verkehrsampeln, und/oder
• - Regelungsgehalten von Verkehrszeichen und/oder
• - Lichtzeichen von Verkehrsteilnehmern, beispielsweise mittels eines Fahrtrichtungsanzeigers und/oder eines Rücklichts und/oder eines Bremslichts und/oder eines Rückfahrscheinwerfers und/oder einer Nebelschlussleuchte und/oder Warnleuchten eines Einsatzfahrzeugs,

und das Fahrverhalten des Verkehrsteilnehmers mittels

• - eines Fahrzeugtyps und/oder
• - einer Geschwindigkeit des Verkehrsteilnehmers und/oder
• - einer Dynamik einer Bewegung des Verkehrsteilnehmers

definiert. Eine solche Definition ermöglicht eine besonders exakte Beschreibung des Umgebungszustands, der visuellen Merkmale und des Fahrverhaltens des Verkehrsteilnehmers, woraus wiederum eine besonders hohe Genauigkeit, Zuverlässigkeit und Güte des Verfahrens folgen.In a further possible embodiment of the procedure,
the environmental condition by means of

• - Weather conditions and/or
• - Ambient light conditions, such as low sun, and/or
• - a road surface condition, for example by detecting precipitation, such as rain and/or snow and/or ice, on a road surface,

the visual features using

• - Traffic light signals, such as traffic lights, and/or
• - Regulatory content of traffic signs and/or
• - light signals from road users, for example by means of a direction indicator and/or a tail light and/or a brake light and/or a reversing light and/or a rear fog light and/or warning lights of an emergency vehicle,

and the driving behaviour of the road user by means of

• - a vehicle type and/or
• - a speed of the road user and/or
• - a dynamic movement of the road user

Such a definition enables a particularly precise description of the environmental conditions, the visual characteristics and the driving behavior of the road user, which in turn results in a particularly high level of accuracy, reliability and quality of the procedure.

• - eine Personenkraftwagen-Klasse und/oder
• - eine Nutzfahrzeug-Klasse und/oder
• - eine Lastkraftwagen-Klasse und/oder
• - eine Bus-Klasse und/oder
• - eine Motorrad-Klasse und/oder
• - eine Fußgänger-Klasse und/oder
• - eine Fahrradfahrer-Klasse

verwendet. Eine derartige Unterteilung der Agentenklassen ermöglicht eine besonders exakte Beschreibung und Abbildung von möglichen zukünftigen Bewegungen des entsprechenden Verkehrsteilnehmers. Die Aufzählung der Agentenklassen ist dabei nicht abschließend.In another possible embodiment of the procedure, the agent classes

• - a passenger car class and/or
• - a commercial vehicle class and/or
• - a truck class and/or
• - a bus class and/or
• - a motorcycle class and/or
• - a pedestrian class and/or
• - a cycling class

used. Such a subdivision of the agent classes enables a particularly precise description and representation of possible future movements of the corresponding road user. The list of agent classes is not exhaustive.

In another possible embodiment of the method, the model is formed by a learning-based trajectory prediction model, which is given a respective agent class as input data when it is used and which learns in a learning process how individual trajectories of a trajectory set are arranged for each agent class. This enables a particularly reliable learning process of the model.

In another possible embodiment of the method, the selection of the respective trajectory set from the given training data set is learned by a neural network and carried out using this network. This means that manual selection is no longer necessary.

Embodiments of the invention are explained in more detail below with reference to drawings.

• 1 schematisch ein Blockschaltbild einer Vorrichtung zur Prädiktion von Bewegungen von Verkehrsteilnehmern,
• 2 schematisch ein Trajektorienset und
• 3 schematisch ein weiteres Trajektorienset.

Showing:

• 1 schematically a block diagram of a device for predicting movements of road users,
• 2 schematically a trajectory set and
• 3 schematically another set of trajectories.

Corresponding parts are provided with the same reference numerals in all figures.

In 1 a block diagram of a device 1 for predicting movements of road users is shown.

The device 1 comprises a model 2, which is designed as a learning and set-based trajectory prediction model and is designed to predict probabilities p of individual trajectories T1 to Tn of a trajectory set S. The device 1 also comprises a module 3, which is designed to select a trajectory set S from a predetermined training data set and/or to dynamically create a trajectory set S.

The prediction of movements of road users, such as vehicles, is an essential component in the operation of automated vehicles, such as highly automated or autonomous vehicles. Reliable and safe movement planning is only possible if the prediction is also of high quality.

Learning-based methods for trajectory prediction exist for such prediction. One type of such learning-based prediction approaches are set-based prediction approaches with a fixed or dynamic trajectory set S.

In classic regression approaches, a training data set for trajectory prediction is used to learn what a trajectory T1 to Tn of the future will look like. A direct output of the learning-based model 2 are coordinates x, y over several time steps, usually in a local coordinate system. In order to achieve multimodality, i.e. a road user, also In order for a model 2, referred to as an agent, to be able to predict with several possible futures, a number of prediction heads must be determined before training model 2.

The set-based prediction approaches mentioned above, however, have a different operating principle in which, for fixed trajectory sets S, a trajectory set S is extracted from the training data set using an algorithm before the learning-based model 2 is trained. The trajectory set S consists of a few trajectories T1 to Tn that may be driven in the future, selected using the algorithm. During the training of the learning-based model 2, the model 2 then learns which trajectory T1 to Tn of the trajectory set S most likely corresponds to the future of the agent to be predicted. The regression problem of entire trajectories T1 to Tn is thus converted into a classification problem. One advantage here is that a number of modes does not have to be specified during training. During execution, a dynamic decision can be made as to how many trajectories T1 to Tn are selected for further processing, because each trajectory T1 to Tn in the trajectory set S is assigned a probability p. During application, this means in concrete terms that trajectories T1 to Tn previously extracted from the training data set and forming the trajectory set S are "selected" for prediction. This selection is done via a classification of the trajectory set S by determining how likely it is that each individual trajectory T1 to Tn of the trajectory set S will occur in a current traffic scene.

In the case of dynamic trajectory sets S, however, no constant, i.e. fixed, trajectory set S is generated from the training data set, but rather a trajectory set S is generated during training and/or execution for the agent to be predicted, which is based on a state of the agent, for example a velocity vector and/or an angle. In this context, it is known to use a kinematic model. For simplicity, a kinematic bicycle model can always be used. The kinematic bicycle model is a classic state model. If this model is integrated with a given starting state over time and accelerations and steering angles are varied at different points in time, a dynamic trajectory set S is created.

The illustrated device 1 is designed to use different and thus specialized trajectory sets S for different agent classes C1 to Cm, whereby this is applied both for fixed and dynamic set-based prediction approaches.

Existing set-based prediction approaches can be extended with module 3, which selects a correct trajectory set S (= fixed trajectory set S) or generates one (= dynamic trajectory set S) to match the corresponding agent class C1 to Cm.

As shown, historical states of surrounding agents (= input data D1), information from a high-resolution environment map (= input data D2), sensor-related information (= input data D3), such as raw data or features of recognized objects, for example their color, a respective agent class C1 to Cm of an agent to be predicted (= input data D4) and a historical state of the agent to be predicted (= input data D5) are fed to model 2 as input data D1 to D5.

In the case of dynamic trajectory sets S, the historical state of the agent to be predicted (= input data D5) can also be fed to module 3 to set an initial state of the kinematic model.

• In [1] und [2] werden jeweils alle gefahrenen Trajektorien T1 bis Tn eines Trainingsdatensatzes unabhängig von ihrer Agentenklasse C1 bis Cm betrachtet. Im Trainingsdatensatz besteht hierbei eine Trajektorie T1 bis Tn aus vergangenheitsbezogenen Eingabedaten und zukünftigen, das heißt zu prädizierenden, Ausgabedaten. Für eine Auswahl der Trajektoriensets S werden immer nur zukünftige Ausschnitte betrachtet. Dabei wird beispielsweise in [1] immer der zukünftige Trajektorienausschnitt ausgewählt, der räumlich gesehen eine Maximalanzahl anderer zukünftiger Trajektorienausschnitte abdeckt. In [2] wird dies durch einen Clusteralgorithmus ähnlich gelöst. Somit wird in [1] und [2] die Generierung von fixen Trajektoriensets S als allgemeingültige Sets für alle Agentenklassen C1 bis Cm beschrieben.

Using module 3, a trajectory set S is generated for fixed trajectory sets S for each agent class C1 to Cm. For example, a trajectory set S defined in [1]: “ Tung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijbom, Eric M. Wolff - CoverNet: Multimodal Behavior Prediction using Trajectory Sets; arXiv: 1911.10298 “ and/or one in [2]: “ Yuning Chai, Benjamin Sapp, Mayank Bansal, Dragomir Anguelov - MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction; arXiv:1910.05449 “ described cluster algorithm can be used in a modified version:

• In [1] and [2], all trajectories T1 to Tn of a training data set are considered, regardless of their agent class C1 to Cm. In the training data set, a trajectory T1 to Tn consists of past-related input data and future, i.e., predictable, output data. For a selection of the trajectory sets S, only future sections are considered. In [1], for example, the future trajectory section is always selected that spatially covers a maximum number of other future trajectory sections. In [2], this is solved in a similar way using a cluster algorithm. Thus, in [1] and [2] the generation of fixed trajectory sets S is described as generally valid sets for all agent classes C1 to Cm.

To operate the device 1 shown, in one possible embodiment, the algorithms described in [1] and [2] are adapted in such a way that a generation of agent class-specific fixed trajectory sets S is possible. For this purpose, a respective agent class C1 to Cm of the agent to be predicted is used as additional information and a corresponding trajectory set S is generated for each agent class C1 to Cm. If, for example, a trajectory set S is to be generated for a cyclist, the entire training data set is initially filtered so that only trajectories T1 to Tn of cyclists are present. A set generation method from [1] or [2] can then be applied to this filtered training data set.

For example, pedestrians cover significantly shorter distances than vehicles due to lower speeds during the prediction horizon of, for example, 3 seconds to 6 seconds. The generation of agent class-specific fixed trajectory sets S makes it possible in this case to explicitly take such kinematic properties into account. An example of two different generated fixed trajectory sets S1, S2 for passenger cars, buses and motorcycles (= trajectory set S1) as well as pedestrians and cyclists (= trajectory set S2) can be found in the 2 and 3 These trajectory sets S1, S2 were generated from the same data set depending on the respective agent class C1 to Cm. The differences are particularly evident in the different scaling of the coordinates x, y.

During the training of model 2 and also during its subsequent application, module 3 selects the appropriate fixed trajectory set S, S1, S2 for the corresponding agent class C1 to Cm of the agent to be predicted.

For dynamic trajectory sets S, a kinematic model specifically designed for this purpose can be used for each agent class C1 to Cm. For example, pedestrians can be described by a so-called "constant velocity model" (CVM for short), cyclists by a kinematic bicycle model, two-lane vehicles by a kinematic two-lane model and trucks with trailers by a more complex model. Depending on the respective agent class C1 to Cm and a state of the agent to be predicted, an entire trajectory set S of agent class-specific trajectories T1 to Tn can then be generated at runtime by varying the input parameters, for example a steering angle and an acceleration. The provision of the agent class-specific kinematic model is also carried out by module 3.

This means that fixed and/or dynamic agent class-specific trajectory sets S are created and, using module 3, the trajectory set S that matches the agent class C1 to Cm to be predicted is selected during the application. In addition, the agent class C1 to Cm is made available to model 2, which is designed as a learning-based trajectory prediction model, as input data D4. In set-based approaches, a classifier is made available as input data D4. Model 2 therefore implicitly learns how the trajectories T1 to Tn are arranged in the trajectory set S for each agent class C1 to Cm.

In one possible embodiment, the selection of the respective trajectory set S is not only carried out depending on the agent class C1 to Cm, but additionally depending on an environmental state and/or visual features and/or a driving behavior of the road user.

• - Wetterbedingungen und/oder
• - Umgebungslichtbedingungen, beispielsweise tiefstehende Sonne, und/oder
• - einer Fahrbahnbeschaffenheit, beispielsweise durch Erkennung von Niederschlägen, wie zum Beispiel Regen und/oder Schnee und/oder Eis, auf einer Fahrbahnoberfläche,

die visuellen Merkmale mittels

• - Lichtzeichen von Verkehrsanlagen, beispielsweise Verkehrsampeln, und/oder
• - Regelungsgehalten von Verkehrszeichen und/oder
• - Lichtzeichen von Verkehrsteilnehmern, beispielsweise mittels eines Fahrtrichtungsanzeigers und/oder eines Rücklichts und/oder eines Bremslichts und/oder eines Rückfahrscheinwerfers und/oder einer Nebelschlussleuchte und/oder Warnleuchten eines Einsatzfahrzeugs,

und das Fahrverhalten des Verkehrsteilnehmers mittels

• - eines Fahrzeugtyps und/oder
• - einer Geschwindigkeit des Verkehrsteilnehmers und/oder
• - einer Dynamik einer Bewegung des Verkehrsteilnehmers

definiert werden.The environmental conditions can be determined by

• - Weather conditions and/or
• - Ambient light conditions, such as low sun, and/or
• - a road surface condition, for example by detecting precipitation, such as rain and/or snow and/or ice, on a road surface,

the visual features using

• - Traffic light signals, such as traffic lights, and/or
• - Regulatory content of traffic signs and/or
• - light signals from road users, for example by means of a direction indicator and/or a tail light and/or a brake light and/or a reversing light and/or a rear fog light and/or warning lights of an emergency vehicle,

and the driving behaviour of the road user by means of

• - a vehicle type and/or
• - a speed of the road user and/or
• - a dynamic movement of the road user

To be defined.

Furthermore, it is possible that the selection of the appropriate trajectory set S is not possible by manually created rules, but is learned by a neural network.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited non-patent literature

• Tung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijbom, Eric M. Wolff - CoverNet: Multimodal Behavior Prediction using Trajectory Sets; arXiv:1911.10298 [0003, 0031]
• Yuning Chai, Benjamin Sapp, Mayank Bansal, Dragomir Anguelov - MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction; arXiv:1910.05449 [0004, 0031]

Claims (7)

Method for predicting movements of road users, wherein a model (2) is created for the prediction, which is trained with at least one predetermined or dynamically created trajectory set (S, S1, S2), characterized in that - road users to be taken into account in the prediction are classified as agents according to their type, - the road users are each assigned to at least one of several agent classes (C1 to Cm) during the classification, - the training is carried out individually for each agent class (C1 to Cm) using at least one trajectory set (S, S1, S2), and - the respective trajectory set (S, S1, S2) is created dynamically specifically for the respective agent class (C1 to Cm) or is selected from a predetermined training data set. Procedure according to Claim 1 , characterized in that the dynamic creation of the at least one trajectory set (S, S1, S2) is additionally carried out as a function of - an environmental condition and/or - visual features and/or - a driving behavior of the road user. Procedure according to Claim 1 , characterized in that the selection of the at least one trajectory set (S, S1, S2) from the predetermined training data set is additionally carried out as a function of - an environmental condition and/or - visual features and/or - a driving behavior of the road user. Procedure according to Claim 2 or 3 , characterized in that the ambient state is defined by means of - weather conditions and/or - ambient light conditions and/or - a road surface condition, the visual features by means of - light signals of traffic facilities and/or - control contents of traffic signs and/or - light signals of road users, and the driving behavior of the road user is defined by means of - a vehicle type and/or - a speed of the road user and/or - a dynamics of a movement of the road user. Method according to one of the preceding claims, characterized in that - a passenger car class and/or - a commercial vehicle class and/or - a truck class and/or - a bus class and/or - a motorcycle class and/or - a pedestrian class and/or - a cyclist class and/or are used as agent classes (C1 to Cm). Method according to one of the preceding claims, characterized in that the model (2) is formed by a learning-based trajectory prediction model, which, when used, is informed of a respective agent class (C1 to Cm) as input data (D4) and which learns in a learning process how individual trajectories (T1 to Tn) of a trajectory set (S, S1, S2) are arranged for each agent class (C1 to Cm). Method according to one of the preceding claims, characterized in that the selection of the respective trajectory set (S, S1, S2) from the predetermined training data set is learned by a neural network and carried out by means of this.

Images