DE102022000238A1 - Method for training and testing an algorithm for predicting agents in a vehicle environment - Google Patents

Abstract

The invention relates to a method for training and testing an algorithm for predicting agents in a vehicle environment, which is carried out using a machine-trained trajectory autoencoder (1). After the training of the trajectory autoencoder (1), similar scenes (S1 to Sn) are found using the trajectory autoencoder (1) for an entire data set (D) to be searched for similar scenes (S1 to Sm) consisting of all scenes (S1 to Sn), latent representations (R1 to Rn) generated. Furthermore, a scene (S) to be searched for, for which similar scenes (S1 to Sm) are to be found in the data set (D), is encoded and a latent representation (R) is formed for the scene (S) to be searched for. Using a similarity metric (3), the latent representation (R) of the scene (S) to be searched for is compared with all other latent representations (R1 to Rn) in the dataset (D) and similar scenes (S1 to Sm) are searched for. An existing trajectory prediction algorithm is trained, retrained and/or tested by means of the trained trajectory autoencoder (1) and found similar scenes (S1 to Sm).

Description

The invention relates to a method for training and testing an algorithm for predicting agents in a vehicle environment according to the preamble of claim 1.

• - Empfangen eines oder mehrerer multimodaler Videoframes von mehreren Modalitäten einer Fahrzeugumgebung, die gegenwärtigen und vergangenen Zeitstempeln entsprechen;
• - Kodieren in eine latente Darstellung, wobei die Multimodalitäts-Videoframes durch ein räumlich-zeitliches Codierungs-Faltungs-Neuralnetzwerk codiert werden;
• - Kombinieren zu einer zusammengesetzten Darstellung, wobei die latente Darstellung mit codierten Konditionierungsparametern entsprechend Zeitstempeln am gewünschten zukünftigen Zeithorizont entspricht; und
• - Vorhersagen mehrerer zukünftiger multimodaler Videoframes entsprechend mehreren zukünftigen Modi eines multimodalen zukünftigen Lösungsraums mit zugeordneten Likelihood-Koeffizienten mittels eines generativen neuronalen Faltungsnetzwerks, welches zuvor in einem mittels eines generativen faltenden neuronalen Netzwerks durchgeführten Trainings trainiert wurde.

From the EP 3 663 965 A1 a computer-implemented method is known, comprising an operating phase and comprising the steps:

• - receiving one or more multi-modal video frames from multiple modalities of a vehicle environment corresponding to current and past time stamps;
• - encoding into a latent representation wherein the multimodality video frames are encoded by a spatio-temporal encoding convolutional neural network;
• - combining into a composite representation, the latent representation having encoded conditioning parameters corresponding to timestamps at the desired future time horizon; and
• - Predicting multiple future multimodal video frames corresponding to multiple future modes of a multimodal future solution space with associated likelihood coefficients using a generative convolutional neural network previously trained in a training performed using a generative convolutional neural network.

The invention is based on the object of specifying a novel method for predicting agents in a vehicle environment.

The object is achieved according to the invention by a method which has the features specified in claim 1.

Advantageous configurations of the invention are the subject matter of the dependent claims.

In a method for training and testing an algorithm for predicting agents in a vehicle environment, the prediction is performed using a machine-trained trajectory autoencoder.

According to the invention, the trajectory autoencoder learns in the training to reconstruct the trajectory of the relevant agent in a given scene whose unstructured dataset includes a trajectory of a relevant agent, trajectories of surrounding agents, interactions between these agents and semantic information. The trajectory autoencoder is trained on all available data. Furthermore, the trajectory autoencoder learns to encode the unstructured data volume of the scene into a structured and latent representation containing information. After the training of the trajectory autoencoder, latent representations are generated to find similar scenes by means of the trajectory autoencoder for an entire data set to be searched for similar scenes, consisting of all scenes. Furthermore, a scene to be searched for, for which similar scenes are to be found in the data set, is encoded and latent representation is formed for the scene to be searched for. Using a similarity metric, the latent representation of the scene to be searched for is compared with all other latent representations in the data set and similar scenes are searched for. As described below, finding similar scenes in a data set can be used advantageously for training, retraining and testing a trajectory prediction algorithm.

A prediction of surrounding agents participating in traffic is an essential prerequisite for carrying out an automated, in particular highly automated or autonomous driving operation of vehicles. In particular, algorithms based on machine learning methods have proven to perform well for this task, known as trajectory prediction.

A challenge of learning-based trajectory prediction algorithms is the coverage of so-called corner cases, i.e. scenes that have a specific and rare connection, for example when a vehicle is driving in the opposite direction of travel on a one-way street. If these corner cases are not available in sufficient quantity in the data set, there is a risk that a learned model generalizes to the rest of the data set and does not cover the corner cases satisfactorily.

In the case of any scene, for example a special corner case, it can therefore be useful to find similar scenes in one's dataset in order to train any trajectory prediction algorithms weighted on these similar scenes or to test them extensively.

However, the above also applies to general scenes such as right-turn operations.

Scenes are highly unstructured data, consisting of a trajectory of a relevant agent, i.e. the agent who, for example, triggers the corner case or turns right, trajectories of other surrounding agents, with wel chen the relevant agent interacts in a scene, and underlying semantic information, such as an arrangement of street topologies or a position of pedestrian crossings and traffic lights.

However, a similarity of scenes does not depend on a similarity of coordinates of the relevant agent, for example finding the same intersection with the same coordinates, but on an entire dataset of the scene. This includes the trajectory of the relevant agent, the trajectories of surrounding agents, interactions between these agents and the semantic information, which is given by a high-resolution map, for example.

Using the present method, it is possible in a particularly advantageous manner to find similar scenes, taking into account the aforementioned unstructured information. The process includes an entire scene with its highly unstructured data in a selection of similar scenes. This means that particularly large data sets for training trajectory prediction algorithms can also be searched for similar scenes. The method also enables post-training and extensive testing of trajectory prediction algorithms with similar and possibly critical scenes.

Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

• 1 schematisch einen Ablauf eines Verfahrens zur Prädiktion von Agenten in einer Fahrzeugumgebung,
• 2 schematisch ein Training eines Trajektorienautoencoders und
• 3 schematisch ein Auffinden ähnlicher Szenen.

show:

• 1 schematically shows a process for predicting agents in a vehicle environment,
• 2 schematically a training of a trajectory autoencoder and
• 3 schematically a finding of similar scenes.

Corresponding parts are provided with the same reference symbols in all figures.

In 1 a sequence of a possible exemplary embodiment of a method for predicting agents in a vehicle environment is shown.

In a first step V1 a training in the 1 and 2 Carried out trajectory autoencoders 1 shown in more detail.

In a second method step V2, depending on a given scene°S to be searched for, at least one also in 3 data set D shown in more detail is searched for similar scenes S1 to Sn. Scenes S1 to Sn are in 3 shown in more detail. Similar scenes°S1 to°Sm are identified via a similarity metric in a latent representation space

In a third method step V3, a trajectory prediction algorithm is trained, post-trained or tested using the identified scenes °S1 to °Sm.

2 shows a possible embodiment of a training of the trajectory autoencoder 1.

In this training, the trajectory autoencoder 1 learns, for a given scene S, from the set of all scenes S1 to Sn in the data set D, whose unstructured data set is a trajectory T1 of a relevant agent, for example a vehicle, trajectories T2 to Tx of surrounding agents, for example other vehicles , interactions between these agents and semantic information to reconstruct the trajectory T1 of the relevant agent. The semantic information is given by a high-resolution map 2 . The trajectory autoencoder 1 is trained on all available data. In the process, the trajectory autoencoder 1 learns to encode the unstructured data volume of the scene S, S1 to Sn into an information-containing structured and latent representation R, R1 to Rn that cannot be interpreted.

The data is fed to an encoder 1.1 of the trajectory autoencoder 1, which generates the latent representation R, R1 to Rn of the scene S, S1 to Sn for a reconstruction of the trajectory T of the relevant agent from the data. This representation R, R1 to Rn is supplied to a decoder 1.2 of the trajectory autoencoder 1, which generates a reconstructed trajectory T1' of the relevant agent from the representation R, R1 to Rn.

Any existing trajectory prediction algorithm can be used as the trajectory autoencoder 1 . This is adapted in such a way that it takes over the reconstruction task and no longer the prediction task of the relevant agent. Such a trajectory prediction algorithm, referred to as LaneCGN, is known, for example, from “Ming Liang, et. al: Learning Lane Graph Representations for Motion Forecasting; https://arxiv.org/abs/2007.13732". Another trajectory prediction algorithm called Trajectron++ is known, for example, from “Tim Salzmann, et. al: Trajectron++: Dynamically Feasible Trajectory Forecasting With Heterogeneous Data; https://arxiv.org/abs/2001.03093". Another trajectory prediction algorithm referred to as MTP is known, for example, from “Henggang Cui, et. al: Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks; https://arxiv.org/abs/1809.10732".

In 3 is a possible embodiment of finding similar scenes S1 to Sm using the according 2 trained trajectory autoencoders 1 shown.

After the training of the trajectory autoencoder 1, similar scenes S1 to Sm to a scene S to be searched for are found by means of the trajectory autoencoder 1 for the entire data set D to be searched for similar scenes S1 to Sm, consisting of the entirety of all scenes°S1°to° Sn, latent representations R1 through Rn generated. Decoder 1.2 is no longer required for this.

Furthermore, the scene S to be searched for, for which the similar scenes S1 to Sm are to be found in the data set D, is encoded and a latent representation R is formed for the scene S to be searched for. Decoder 1.2 is no longer required for this.

A similarity metric 3 is used to compare the latent representation R of the scene S to be searched for with all other latent representations R1 to Rn in the data set D, and similar scenes S1 to Sm are searched for. The comparison can be made, for example, via a so-called cosine similarity. On the basis of the similarity metric 3, similar scenes S1 to Sm can be found in the latent space.

An example is used below to explain the advantages of including the semantic information given by the high-resolution map 2 in the training and in the use of the trajectory autoencoder 1 .

For example, a critical scene S with a pedestrian crossing is characterized in that the relevant agent, in this case a vehicle, turns right. The trajectory autoencoder 1 was trained using such a scene S. Similar scenes S1 to Sm are to be found for this scene S. Known approaches would only fix a similarity to the trajectory T1 of the relevant agent and thus deliver similar right-turning processes as a result. However, the present method also takes into account the semantic information of the high-resolution map 2, so that scenes S1 to Sn with processes turning right and a pedestrian crossing are advantageously delivered as similar results. It is thus possible to find significantly more detailed and more suitable similar scenarios and use them for the subsequent training, retraining or testing of a trajectory prediction algorithm.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• EP 3663965 A1 [0002]

Claims (1)

Method for training and testing an algorithm for predicting agents in a vehicle environment, which is carried out by means of a machine-trained trajectory autoencoder (1), characterized in that in the training of the trajectory autoencoder (1) the trajectory autoencoder (1) - learns, at a given Scene (S, S1 to Sn) whose unstructured data set includes a trajectory (T1) of a relevant agent, trajectories (T2 to x) of surrounding agents, interactions between these agents and semantic information to reconstruct the trajectory (T1) of the relevant agent, - is trained on all available data and - learns to encode the unstructured data volume of the scene into an information-containing, structured and latent representation (R, R1 to Rn), and after training the trajectory autoencoder (1) to finding similar scenes (S1 to Sm) - by means of the trajectory autoencoder (1) for an entire similar hen scenes (S1 to Sm) to be searched data set (D) consisting of all scenes (S1 to Sn), latent representations (R1 to Rn) are generated, - a scene to be searched (S), for which similar scenes (S1 to Sm ) are to be found in the data set (D), is encoded and a latent representation (R) is formed for the scene (S) to be searched for, and - by means of a similarity metric (3), the latent representation (R) of the scene (S ) is compared with all other latent representations (R1 to Rn) in the dataset (D) and similar scenes (S1 to Sm) are sought, and a trajectory prediction algorithm is trained using the trained trajectory autoencoder (1) and found similar scenes (S1 to Sm), retrained and/or tested.

Images