EP4612609A1

EP4612609A1 - A computer-implemented method for predicting a joint set of trajectories for autonomous vehicles in a traffic environment

Info

Publication number: EP4612609A1
Application number: EP23789584.2A
Authority: EP
Inventors: Michael Colin HOY
Original assignee: Continental Automotive Technologies GmbH
Current assignee: Aumovio Germany GmbH
Priority date: 2022-11-03
Filing date: 2023-10-11
Publication date: 2025-09-10
Also published as: WO2024094394A1; GB202216330D0; GB2624154A

Abstract

In order to improve navigation of an autonomous vehicle through a traffic environment (10), a computer-implemented method for predicting a joint set of compatible trajectories (24) is proposed. After collating the relevant data, an input graph (58) is generated that represents the positions of the ego-vehicle (10) and objects (16) in the traffic environment (10). A graph neural network is iteratively executed on the input graph (58) thereby generating an output graph (80) that becomes the input for the next iteration. The edge weights are a measure for the compatibility of different trajectories, which are composed from the nodes of the last iteration and all the respective ancestor nodes. Based on the edge weights, a final joint set of compatible trajectories (24) is generated.

Description

TITLE

A computer-implemented method for predicting a joint set of trajectories for autonomous vehicles in a traffic environment

TECHNICAL FIELD

The invention relates to a computer-implemented method for predicting a joint set of trajectories for an ego-vehicle and objects in a traffic environment.

BACKGROUND

One known approach for trajectory prediction is to use a graph neural network (GNN) to aggregate information about different agents. Then a fully connected model or a sequence model, like long short-term memory (LSTM), is used to decode the individual trajectories. For further information, reference is made to the following:

[1] Wang, Chengxin, Shaofeng Cai, and Gary Tan. "Graphtcn: Spatio-temporal interaction modeling for human trajectory prediction." In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3450-3459. 2021.

[2] Zhang, Zhishuai, Jiyang Gao, Junhua Mao, Yukai Liu, Dragomir Anguelov, and Congcong Li. "Stinet: Spatio-temporal-interactive network for pedestrian detection and trajectory prediction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11346-11355. 2020.

[3] Zhang, Lidan, Qi She, and Ping Guo. "Stochastic trajectory prediction with social graph network." arXiv preprint arXiv:1907. 10233 (2019).

[4] Choi, Chiho, Joon Hee Choi, Jiachen Li, and Srikanth Malla. "Shared cross-modal trajectory prediction for autonomous driving." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 244-253. 2021 . [5] Zhang, Pu, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. "Sr- Istm: State refinement for Istm towards pedestrian trajectory prediction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12085-12094. 2019.

Other known ideas use a GNN to iteratively predict agent positions at successive timesteps. They may output an uncertainty ellipse for each prediction. These typically do not consider any type of branching process to represent uncertainty. For further information, reference is made to the following:

[6] Casas, S., Gulino, C., Liao, R., & Urtasun, R. (2019). Spatially aware graph neural networks for relational behavior forecasting from sensor data. arXiv preprint arXiv:1910.08233.

[7] Lee, Donsuk, Yiming Gu, derrick Hoang, and Micol Marchetti-Bowick. "Joint interaction and trajectory prediction for autonomous driving using graph neural networks." arXiv preprint arXiv:1912.07882 (2019).

It is possible to execute the entire joint prediction process multiple times with different random parameters, to obtain joint samples of all the traffic participants. For further information, reference is made to the following:

[8] Yeh, Raymond A., Alexander G. Schwing, Jonathan Huang, and Kevin Murphy. "Diverse generation for multi-agent sports games." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4610-4619. 2019.

[9] Casas, Sergio, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. "Implicit latent variable model for sceneconsistent motion forecasting." In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pp. 624-641. Springer International Publishing, 2020.

Another example for graph neural networks for trajectory prediction, with explicit modelling of the interaction between provisional trajectories is known from: [10] Jo, Eunsan, Myoungho Sunwoo, and Minchul Lee. "Vehicle Trajectory Prediction Using Hierarchical Graph Neural Network for Considering Interaction among Multimodal Maneuvers." Sensors 21 , no. 16 (2021 ):5354.

Some trajectory prediction methods with other mechanisms for handling trajectory interactions are also known. One approach is to first predict the possible destination points for each traffic participant, sample from these, and then perform path prediction. This may make multi-modal path prediction more stable:

[11] Mangalam, Karttikeya, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. "It is not the journey but the destination: Endpoint conditioned trajectory prediction." In European Conference on Computer Vision, pp. 759-776. Springer, Cham, 2020.

It is possible to model objects hierarchically - i.e. , objects are split into groups, predictions are made for each group, then correlated samples can be drawn for the individual objects comprising each group:

[12] Chen, Yuying, Congcong Liu, Bertram E. Shi, and Ming Liu. "HGCN-GJS: Hierarchical Graph Convolutional Network with Groupwise Joint Sampling for Trajectory Prediction." arXiv preprint arXiv:2009.07140 (2020).

[13] Sun, Jianhua, Qinhong Jiang, and Cewu Lu. "Recursive social behavior graph for trajectory prediction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 660-669. 2020.

Neural networks applied to problems that require branching predictions are known and may be used as an approach to play games, most famously AlphaGo achieving super-human performance in the boardgame Go:

[14] Silver, David, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, GeorgeVan Den Driessche, Julian Schrittwieser et al. "Mastering the game of Go with deep neural networks and tree search." nature 529, no. 7587 (2016): 484-489. Another approach may use a neural network to augment a tree search algorithm in two ways. A neural network may be used for evaluating a given board state (i.e. , to predict the probability of winning). A neural network may be used to suggest a small set of the most promising moves at each turn, which can then be further explored using tree search. Approaches combining tree search with neural networks have been used in combinatorial optimization problems beyond the original perfect information game application for solving Rubik’s cubes (see [15] Agostinelli, F., McAleer, S., Shmakov, A., & Baldi, P. (2019). Solving the Rubik’s cube with deep reinforcement learning and search. Nature Machine Intelligence, 1(8), 356-363.), or for solving imperfect information games, such as poker (see [16] Brown, Noam, Anton Bakhtin, Adam Lerer, and Qucheng Gong. "Combining deep reinforcement learning and search for imperfect-information games." arXiv preprint arXiv:2007. 13544 (2020)).

SUMMARY OF THE INVENTION

It is the object of the invention to improve trajectory determination in a traffic environment, preferably in a highly cluttered traffic environment.

The invention provides computer-implemented method for predicting a joint set of trajectories in a traffic environment, the method comprising: a) collating ego-vehicle data that are indicative of at least one motion or position related quantity of an ego-vehicle, and collating environment object data that are indicative of at least one motion or position related quantity of at least one object within the traffic environment; b) generating an input ego-vehicle node that has node feature data that are based on the ego-vehicle data and, for each object, generating an input object node that has node feature data that are based on the environment object data, and generating an input graph that includes the input ego-vehicle node and each input object node; c) executing a graph neural network with the input graph as input data, wherein the graph neural network determines at least one predicted ego-vehicle node and at least one predicted object node for each input object node, wherein each predicted ego-vehicle node is indicative of a future ego-vehicle position, wherein each predicted object node is indicative of a future object position, wherein the graph neural network determines a node weight for each predicted node, wherein the graph neural network determines a predicted edge weight for at least one graph edge that connects the ego-vehicle node with one of the object nodes; d) determining an ego-vehicle trajectory composed of each predicted ego-vehicle node and optionally of the input ego-vehicle node, and determining at least one object trajectory for each object, the object trajectory being composed of each predicted object node and optionally of the input object node; and e) determining a trajectory joint compatibility score based on each predicted edge weight for the trajectories determined in step d), wherein the trajectory joint compatibility score is indicative of whether the respective trajectories are considered in further processing, and determining the joint set of trajectories based on the trajectory joint compatibility score.

With this approach, two basic phases of the trajectory determination are performed. Initially participants, the ego-vehicle and other objects around it, are incorporated in a graph structure. The nodes represent locations of the respective participant. A collection of nodes forms a trajectory. Graph edge can represent the possiblity of interaction of the nodes. The edge weight associated with a graph edge is used as a measure of the likelihood of interaction between the connected nodes, and as a result the likelihood of interaction between the trajectories that include the particular nodes. In a second phase, possible trajectories for all participants are decoded from the graph structure. The trajectories are evaluated for their joint compatibility. The joint compatibility score measures to which degree the possible trajectories of the ego-vehicle and the other objects can be performed without causing problems, such as collisions, slowdowns or other issues. Compared to the usual approaches, the method considers all traffic participants allows modelling of the joint trajectories with greater accuracy and/or for a longer time scale.

Preferably, the environment object data include dynamic object data that are indicative of the motion related quantity and include static environment data that are indicative of environment features that do not change during execution of the method steps. Environments are complex and may change at a fast pace. With this approach dynamic objects, such as other vehicles, are distinguished from static objects, such as obstacles, traffic signs or other environmental features that while not changing over time themselves may change due to motion of the ego-vehicle. The accuracy of the joint trajectories may be improved by training the influence of both types of objects.

Preferably, in step b) the input ego-vehicle node and the input object node are generated by an input node encoder that includes a fully connected model and/or a sequence model. The use of fully connected models or sequence models is comparatively lightweight computationally. As a result, the model can be trained easier while retaining sufficient accuracy for improvement of the joint trajectory prediction.

Preferably, step c) is repeated a predetermined number of times depending on a prediction horizon, wherein an output graph of the previous iteration of the graph neural network becomes the input graph of the next iteration. Repeated execution of the graph model allows predicting further time steps.

Preferably, step c) comprises, for each node, pruning the respective node from the graph, if the node weight is below a predetermined threshold. With this approach nodes that represent implausible or unviable states of the corresponding participant are removed from further consideration, thereby allowing a faster execution of the method. Furthermore, trajectories that are incompatible can be removed at an earlier stage, thereby improving the overall results.

Preferably, in step c) a first and second predicted ego-vehicle node and/or a first and second predicted object node are determined. The use of two or more nodes allows branching of the candidate trajectories and allow exploration of more possible trajectories such that the expectation of an overall improved accuracy can be increased.

Preferably, in step c) a graph edge that connects the first and second predicted egovehicle node and/or the first and second predicted object node are removed from the graph. In this approach, branching has occurred and the ego-vehicle is represented by two or more predicted nodes. It is also possible that one of the objects is represented by two or more nodes. Removing the graph edge removes the possibility of interaction between nodes that represent the same ego-vehicle or object. With this the result is more accurate, since in general the ego-vehicle or the respective object cannot follow two trajectories simultaneously.

Preferably, in step d) the trajectories are determined by a trajectory decoder model that includes a sequence model. The graph neural network encodes the joint trajectories and their respective viability in the nodes, edges and associated weights of the graph neural network. The encoded information represents a sequence of nodes that together form a trajectory. A sequence model allows decoding the encoded information into the joint trajectories with higher accuracy.

The invention provides a computer-implemented method for determining a route through a traffic environment, the method comprising: a) carrying out a previously described method in order to determine at least one egovehicle trajectory; b) determining a route property that is influenced by the ego-vehicle trajectory, the route property preferably chosen from a group consisting of ego-vehicle trajectory length, time to travel along the ego-vehicle trajectory, number of crossings with the at least one object trajectory, route destination, and fuel consumption along the egovehicle trajectory. c) selecting the ego-vehicle trajectory in accordance with the property determined in step b) and generating a control signal that causes the ego-vehicle to follow the selected trajectory or displaying the route to the driver of the ego-vehicle.

Trajectories typically represent options for the ego-vehicle that can be followed. With this method a trajectory is selected as a path that is followed by the ego-vehicle based on the best joint trajectory available under at least one predetermined route property. Since the trajectories are modeled with higher accuracy, the overall path planning can be more reliable and accurate. The path also considers the movement of the other participants in the environment and can therefore be followed more safely. The invention provides a computer-implemented method for training a machine learning model for use in a previously described method, the training method using a loss function based on a deviation from a ground truth trajectory and an independent cross-entropy loss for each node weight and/or edge weight. Unsupervised learning with this approach allows a cost function that trains the model for different properties, such as safety, courtesy, advancement towards goals, braking/acceleration minimization, etc.

The invention provides a data processing apparatus comprising means for carrying out a previously described method.

The invention provides a vehicle comprising the data processing apparatus.

The invention provides a computer program comprising instructions which, when the program is executed by a data processing apparatus, cause data processing apparatus to carry out a previously described method.

The invention provides a computer-readable medium or a data carrier signal comprising the computer program.

Trajectory prediction is an important part of the (autonomous) vehicle perception stack. Examples of approaches include predicting a “heatmap” of possible future positions of each traffic participant; predicting the most likely trajectories for each traffic participant, together with an estimate of the uncertainty in each waypoint of the trajectory; predicting a set of possible trajectories independently for each participant; and predicting a joint set of trajectories for every participant.

A method for implementing such prediction algorithms is a graph neural network. The disclosed approach is a self-contained extension to the graph neural network approach, which allows better handling of interactions between multiple possible trajectories of multiple participants. It can be thought of as an improvement on the approach of predicting a joint set of trajectories for every participant. Trajectory prediction methods differ in how they model the ego-vehicle, the egovehicle typically referring to the vehicle that contains the sensors that perceive the environment around the vehicle.

One approach may assume a certain behavior of the ego-vehicle, with the predictions implicitly conditioned on this assumed behavior. Another approach may combine the trajectory planning for the ego-vehicle into the same neural network doing trajectory prediction for the other traffic participants. Yet another approach may entertain multiple possible trajectories for the ego-vehicle, and allow downstream modules to inspect the likely effect of different trajectory choices on the other traffic participants.

In some embodiments the second and third approaches are combined, i.e. , the neural network predicts multiple possible trajectories for the ego-vehicle, which can then be weighted using an additional cost function.

In some embodiments the trajectory prediction problem may be phrased or approached in game theoretic terms. There has been much research interest in applying neural network approaches to games like chess, Go, poker, Hanabi, Diplomacy, etc., however, these are discrete domains, and it is not obvious how to apply such approaches to continuous prediction problems like trajectory prediction.

There is a tradeoff between computational tractability and the ability to handle uncertainty and/or interactions between traffic participants. For example, typically a method which faithfully models the “joint trajectory sampling”, requires a number of samples which scale exponentially with regard to the number of traffic participants. In other words, it is rather difficult to use these methods in vehicles due to limited computing power.

Graph neural network approaches generally “internalize” all the analysis of interactions between participants, which may not generalize well to edge cases as a principled scenario based planning that limits the scope of the analysis that needs to be carried out by the GNN. In this disclosure a new method of formulating a graph neural network for the trajectory prediction task is proposed. In some embodiments multiple possible trajectories are predicted simultaneously. In some embodiments the interaction between these is modeled using the graph neural network. In some embodiments interactions/pairwise weightings between provisional trajectories are modeled, such that possible joint samples between trajectories may be decoded at the path planning stage. In some embodiments a backwards pruning algorithm is provided, that prunes unlikely samples, i.e. , samples that result in an unlikely configuration at some timestep in the future.

The method may invoke two phases. In a first phase each traffic participant may be represented as a node in a graph. A graph neural network may be executed. Then, a trajectory for each participant may be decoded. In some embodiments an initial GNN may be executed for planning provisional trajectories. The initial GNN may be followed in some embodiments by a larger GNN that may also consider interactions between the provisional trajectories of each participant.

In a second phase, a graph may be constructed where each node corresponds to a provisional trajectory of one of the traffic participants. The provisional trajectories may be refined and output as a set of refined trajectories. In particular, instead of each trajectory being represented as a node in the GNN, in a preferred approach each waypoint is represented by a node and predictions can be made iteratively across time.

In some embodiments of the second phase, each possible waypoint is treated as a node in the graph neural network. The graph neural network executes repeatedly at every time step of the prediction horizon.

In some embodiments, a mechanism to propagate trajectory feasibility information backwards through time (i.e., along the branches of the provisional trajectories) can be included.

In some embodiments the neural network can be provided with more comprehensive information across a wide range of time steps. In some embodiments the model may scale better in long horizon prediction problems, e.g., where the trajectory includes a series of distinct maneuvers.

One idea includes an extension of graph neural networks applied to trajectory prediction problems, which is able to explicitly handle multiple possible trajectories. The GNN may include a first graph convolution layer that has a plurality of in channels, e.g., 14, and a plurality of out channels, e.g., 256. The graph convolution layer may feed into a ReLLI layer. The ReLLI layer preferably feeds into a dropout layer. The output of the dropout layer is preferably fed into a second graph convolution layer that has a plurality of in channels, e.g., 256, and a plurality of out channels, e.g., 256. The second graph convolutional layer may feed into a ReLLI layer. The output of the ReLLI may be fed to a node prediction layer and/or an edge prediction layer. The node prediction layer may feed into a linear layer followed by a sigmoid layer that is only for node weights. The edge prediction layer may feed into a bilinear layer followed by a sigmoid layer.

In some embodiments information about the ego-vehicle and all other tracked objects in the surroundings can be collated. Each object is preferably represented as a node in a densely connected graph. Node features may include the position, velocity, and/or vehicle type. Edge features may be determined from covisibility (i.e. , whether the vehicles can see each other). Edge weights may be omitted at the graph preparation stage. In some embodiments an “edge weight” can be predicted. The edge weight is indicative of the possible interactions between traffic participants and can be more explicitly modelled during the subsequent path planning process. The edge weights are preferably determined during execution of the graph neural network. The edge weight may be calculated from the node weights of the respective nodes connected by the graph edge using a bilinear function of the node weights.

A high level trajectory is not required. Typically, an evaluation over all ego-vehicle trajectories that are potentially assumed by the other vehicles takes place. All these data are usually readily available in the vehicle computer. In some embodiments a graph neural network is executed with this graph as input. For each node possible future positions of the participant may be predicted. Preferably, one or two possible future positions are predicted. The positions may correspond to a non-branching and a branching trajectory, respectively. In some embodiments this can be expressed in the neural network output layer as two possible future positions together with a “node weight” that may give an overall appraisal of the viability of the trajectory giving an indication as to whether the trajectory is viable.

In some embodiments an “edge weight” for each other traffic participant graph node may be predicted. For example, once “branching” has occurred, each traffic participant may be represented by multiple graph nodes that correspond to multiple possible future positions of the other traffic participant. The “edge weights” may be later used to decode the likely interactions between the predicted trajectories of each traffic participant.

In some embodiments, the execution of the graph neural network may be repeated from the next predicted position of each traffic participant. Preferably, when a “branching event” has occurred, both of the possible predicted positions are included as nodes on the input of the graph neural network. In some embodiments all the pairwise connections between each pair of nodes in the graph neural network are maintained, except for the “branched” versions of the same traffic participant. These typically do not require graph edges connecting them. In some embodiments this process is repeated for a number of iterations according to some planning horizon, e.g., 10 iterations.

In some embodiments the neural network is trained. Suitable training data for the system are the waymo dataset, obtainable from https://waymo.com/open/. These data typically include a set of vehicle trajectories in different situations.

In some embodiments the true trajectory is matched to the nearest matching trajectory in the tree data structures. In some embodiments the positional component of the loss function is based on the physical distance between the trajectories. The loss function may include L2 loss (e.g., for distances and/or velocities), or a weighted cross entropy loss function (e.g., for nodes weights and affinity weights).

In some embodiments the “predicted node weight” is rewarded for the predicted trajectory. In some embodiments the “predicted node weight” is penalized for all other trajectories, i.e. , the trajectories that deviate from the predicted trajectory. In some embodiments the “edge weight” is rewarded for pairs of waypoints in the predicted trajectory, and penalized for all other pair of waypoints. In some embodiments node weights and edge weights associated with the nearest matching prediction are reinforced. In some embodiments the other node weights and edge weights, i.e., non-associated weights, are anti-reinforced.

In some embodiments, for handling the ego-vehicle, an actual trajectory of the egovehicle in the training data as the ground truth may be used. Alternately or additionally, iterations of supervised learning on the training data may be “interspersed” with iterations of self-supervised/reinforcement learning inside a simulation engine.

In some embodiments, when running the network, a path planning step is included that makes use of the predictions of the network. Assuming, for example, that the path planning for the ego-vehicle has been included in GNN outputs, the process may be described as follows. Preferably, for each possible trajectory, for the egovehicle, all trajectories for all agents according to the predicted edge weights are pruned/reweighted. Preferably, with the reweighted trajectories, some cost function related to safety, courtesy, advancement towards goals, braking/acceleration minimization, etc. is evaluated.

This disclosure proposes a new method for handling interactions in graph neural network based path prediction. The method allows modeling of the joint distribution of trajectories over long time scales more accurately.

The method may be applied for identifying routes through a cluttered environment (i.e., perform the tree prediction without modelling other traffic participants) and/or realistic simulation of autonomous vehicles.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings. Fig. 1 depicts an exemplary traffic environment;

Fig. 2 depicts an embodiment of a machine learning model for predicting trajectories; Fig. 3 depicts an embodiment of a graph neural network;

Fig. 4 to Fig. 9 depict an embodiment of a method for predicting trajectories.

DETAILED DESCRIPTION OF EMBODIMENT

Referring to Fig. 1 , a traffic environment 10 is depicted. The traffic environment 10 includes a plurality of participants 12. One of the participants 12 is an ego-vehicle 14, whereas the other participants 12 are objects 16, preferably other vehicles. The participants 12 are preferably autonomous participants 18 that may navigate the traffic environment 10 in an autonomous manner.

The ego-vehicle 14 may follow an ego-vehicle trajectory 20. The objects 16 may follow object trajectories 22, which are however unknown in advance. It is the goal to determine a joint set of trajectories 24 that are jointly compatible to each other. In other words, the ego-vehicle 14 aims to estimate one or more trajectories for each participant 12, i.e. , including itself, that avoid collisions and may also fulfill other constraints.

Referring to Fig. 2, a machine learning model 26 is depicted in more detail. The machine learning model 26 is configured to determine the joint set of trajectories 24.

The machine learning model 26 comprises an encoder layer 28, a graph neural network layer 30 and a decoder layer 32. The encoder layer 28 generates an input graph and feeds the input graph to the graph neural network layer 30. The graph neural network layer 30 may be executed more than once in an iterative manner. The output of the last iteration is fed to the decoder layer 32 which is configured to generate the joint set of trajectories 24.

The encoder layer 28 is fed with ego-vehicle data and environment object data. The ego-vehicle data are indicative of a motion or position related quantity of the ego- vehicle 14, such as position, velocity, and/or acceleration. The ego-vehicle data may also include a vehicle-type, that is preferably encoded using a hot-one encoding.

The environment object data may include dynamic object data. The dynamic object data are indicative of a motion or position related quantity of the object 16, such as position, velocity, and/or acceleration. The dynamic object data may also include an object type, specifically a vehicle type, that is preferably encoded using a hot-one encoding.

The environment object data may include static environment data. The static environment data are indicative of non-moving portions of the traffic environment 10 that in general change slowly. The static environment data may include but are not limited to, road condition, weather condition, traffic signs, blocked lanes, etc.

The encoder layer 28 may include a fully connected model for generating the input graph. The fully connected model preferably includes at least one fully connected layer. Alternatively, or additionally, the encoder layer 28 may include a sequence model for generating the input graph. The sequence model preferably includes at least one long short-term memory (LSTM) layer. Other sequence models are also possible.

Referring to Fig. 3, the graph neural network layer 30 is described in more detail. The graph neural network layer 30 includes a plurality of common layers 34, a node weight prediction portion 36, and an edge weight prediction portion 38.

Subsequently, the common layers 34 are described in more detail. An input layer 40 receives the input graph. The input layer 40 may leave the input graph unchanged or perform some preprocessing tasks on the input graph that are known per se. The input layer 40 feeds forward to a first graph convolution layer 42 which performs a graph convolution on the (possibly preprocessed) input graph. The first graph convolution layer 42 feeds forward to a first ReLLI layer 44 which in turn feeds into a dropout layer 46. The dropout layer 46 randomly prevents data items from further processing with a predetermined probability. The dropout layer 46 feeds into a second graph convolution layer 48 that in turn feeds into a second ReLLI layer 50. The second ReLLI layer 50 feeds its output to each of the node weight prediction portion 36 and the edge weight prediction portion 38.

The node weight prediction portion 36 includes a linear layer 52. The linear layer 52 is preferably configured to reduce the dimension of its input.

The edge weight prediction portion 38 includes a bilinear layer 54 that feeds into an activation layer 56. The bilinear layer 54 preferably reduces the dimension of its input down to one. The activation layer 56 may use a sigmoid activation function.

Referring to Fig. 4 through Fig. 9, an embodiment of a method for predicting the joint set of trajectories 24 for a predetermined prediction horizon is described in more detail.

As shown in Fig. 4, at time t + 0 an input graph 58 is prepared. The input graph 58 includes an input ego-vehicle node 60. The input ego-vehicle node 60 is generated by the encoder layer 28 and comprises a node weight that is based on the egovehicle data.

The input graph 58 further comprises at least one input object node 62 that is based on one of the objects 16. These kind of object nodes may also be designated as dynamic object nodes 64. Similarly, each input object node 62 is generated by the encoder layer 28.

The input graph 58 may also comprise input object nodes 62 that are based on the static environment data. These kind of object nodes may also be designated as static environment nodes 66.

The input graph 58 may include input edge weights (not shown) that connect the ego-vehicle node 60 with each dynamic object node 64. Each edge weight is a measure for a certain proximity between the ego-vehicle node 60 and the connected dynamic object node 64. The input edge weight may be determined by some heuristic model of proximity. Alternatively, or additionally, the input edge weight may be determined from covisibility between the ego-vehicle 14 and the connected object 16. This may be done using sensor data and/or vehicle to anything communication.

The graph neural network layer 30 is executed on the input graph 58 in order to advance from time t + 0 to time t + 1 .

The graph neural network layer 30 generates a first predicted ego-vehicle node 68 and a second predicted ego-vehicle node 70. The node weight of each predicted ego-vehicle node 68, 70 is obtained from the node weight prediction portion 36.

The graph neural network layer 30 also generates a first predicted object node 72 and a second predicted object node 74 for each input object node 62. The node weight of each predicted object node 72, 74 is obtained from the node weight prediction portion 36.

It should be noted that in this example, the graph neural network layer 30 generates two predicted nodes for each input node. This is designated as a branching factor of two. It is possible to have a higher branching factor than two.

The node weight for each predicted ego-vehicle node 68, 70 and each predicted object node 72, 74 is determined. The node weight is indicative of the viability of the respective node. If the node weight does not exceed a predetermined threshold, the corresponding node, e.g., the second predicted ego-vehicle node 70 and the first predicted object node 72 are pruned (see Fig. 5).

The graph neural network layer 30 may also determine predicted static environment nodes 76.

The graph neural network layer 30 may also determine a predicted edge weight 78 for each edge that connects the input ego-vehicle node 60 with a dynamic object node 64. The graph neural network layer 30 may determine a predicted edge weight 78 for each edge that connects two dynamic object nodes 64, provided said nodes represent different objects 16. The execution of the graph neural network layer 30 results in an output graph 80 that includes all calculated and unpruned predicted nodes, here, the first ego-vehicle node 68, the second object node 74, and the static environment nodes 76.

Referring to Fig. 6, the graph neural network layer 30 is executed with the output graph 80 as the input graph 58 in order to iteratively advance from time t + 1 to t + 2. As an example, in this iteration, none of the predicted ego-vehicle nodes 68, 70 and of the predicted object nodes 72, 74 are pruned. It should be noted that as a result the ego-vehicle 14 and the object 16 are represented by more than one node.

Referring to Fig. 7, the graph neural network layer 30 is executed again on the result of the previous iteration as the input graph 58 in order to iteratively advance from time t + 2 to t + 3.

As shown, the graph neural network layer 30 determines two predicted nodes 82 for each node in the input graph 58. The graph neural network layer 30 determines a node weight for each predicted node 82 and a predicted edge weight 78 as previously described.

As shown in Fig. 8, when the prediction horizon is reached, e.g., three steps in this example, some nodes may be pruned from the output graph 80 in accordance with their node weight.

Referring to Fig. 9, the decoder layer 32 iterates over all nodes of the final output graph 84 at time t + 3 and determines a plurality of trajectories 86 by adding to the respective trajectory 86 the currently iterated node and all its ancestors down to time t + 0.

As shown in Fig. 9, in total five trajectories 86 are determined, wherein three relate to the ego-vehicle 14 and two relate to the object 16.

For each of those trajectories 86 an overall edge weight 88 is determined by iterating over all unique pairs of trajectories 86, wherein the overall edge weight 88 is determined as the product of predicted edge weights 78 for each pair of nodes of the respective unique pair. The overall edge weight 88 measures to which degree the trajectories 86 can be used without causing problems and is designated as the trajectory joint compatibility score. The trajectories 86 form the joint set of trajectories 24.

Subsequently, a method for path planning of a path of the ego-vehicle 14 through the traffic environment 10 is described.

The trajectories 86 that relate to the ego-vehicle 14 can be pruned or reweighted in accordance with the predicted edge weights 78 and/or the overall edge weight 88. The trajectories 86 that remain, may be evaluated according to some cost function that may be related to courtesy, advancement towards goals, braking/acceleration minimization, etc. The trajectory 86 that has the best evaluation is then used as a path that the ego-vehicle 14 will follow through the traffic environment 10.

The training of the machine learning model 26 is described in more detail.

A true trajectory is matched to the nearest matching trajectory in the graph data structures. The positional component of the loss function is based on the physical distance between the trajectories. The predicted node weight is rewarded for the predicted trajectory, and penalized for all other trajectories. The edge weight is rewarded for pairs of waypoints in the predicted trajectory, and penalized for all other pairs of waypoints. All node weights and edge weights associated with the nearest matching prediction are reinforced. All other node weights and edge weights are antireinforced. For handling the ego-vehicle 14, the actual trajectory of the ego-vehicle 14 in the training data may be used as the ground truth. Alternately, iterations of supervised learning on the training data may be interspersed with iterations of self- supervised/reinforcement learning inside a simulation engine.

The previously described methods are further illustrated using the following pseudocode based on Python. The pseudocode includes the predicition method and the training method.

The input data is preferably https://waymo.com/open/ The model used includes hUj-^Z/d' J_ill.pub/2021/gnn-intro/ and https://github.com/tkipf/pygcn/blob/master/pygcn/models.py

Common layers:

* GraphConv(in_channels=14, out_channels=256)

* Relu

* Dropout(p=0.2)

* GraphConv(in_channels=256, out_channels=256)

* Relu

Node prediction layers:

* Linear(256, 10)

Edge prediction layer:

* Bilinear(left_in_channels=256, right_in_channels=256, out_channels=1 ) // i.e. using features from the two nodes related to the edge

* Sigmoid struct Objectstate: object d: ID

// assume a predetermined number of possible object types with a one hot encoding, e.g., 10 object types vehicle_type: float[10] position: float[2] velocity: float[2] struct BranchPrediction: state: Objectstate node_weight: float (use sigmoid) struct NodePrediction:

// use a branching factor of two branch_predictions: [BranchPrediction, BranchPrediction] struct EdgePrediction: edge_weight: float func execute_counterfactual_graph_neural_network( network: GraphNN, detected_vehicles: ObjectState[n_vehicle], environmental_features: ObjectState[n_environmental_features], is_training: bool ground_truth_trajectory:

Optional[ObjectState[n_vehicle][n_horizon]]) ->

(Optional[Graph[node_type=[ObjectState[n_horizon]]]], Optional[float]): graph[0] = construct_graph( nodes="detected_vehicles and environmental_features", edges- 'determine edges based on covisibility/proximity heuristics")

11 a typical value of n_horizon might be 10 for i_horizon = 1 ,.n_horizon: node_predictions[i_horizon], edge_predictions[i_horizon] = network.forward(input_graph) new_nodes = [] for node_prediction in node_predictions[i_horizon]: if "node_prediction is from environmental_features": continue //skip environmental nodes for branch_prediction in node_prediction.branch_predictions: if not is_training and branch_prediction.node_weight

< 0.5 and "vehicle can be better represented by another node": continue new_nodes.append(branch_prediction. state) graph[i_horizon] = construct_graph( nodes = "new_nodes and environmental_features", edges = ("determine edges based on covisibility/proximityheuristics”, "but disallow edges between nodes correspondingto the same vehicle")

) if is_training: loss = 0.0

// determine the match with the actual trajectory best_distance = inf best_node = null for node in graph[n_horizon], nodes: if "node_prediction is from environmental_features": continue distance = "overall distance from all ancestor nodes to ground_truth_trajectory" if distance < best_distance: best_distance = distance best_node = node

// same as a 12 loss function loss += best_distance

// independent cross entropy loss for every weight loss -= "sum of log(node.branch_predictions[].node_weight) for all ancestors of best_node ignoring, environmental_features" loss -= 1 e-5 * "sum of log( 1 - node.branch_predictions[].node_weight) for all nodes at all levels, ignoring environmental_features" loss -= ("sum of log(edge_predictions[].edge_weight), for all ancestors of best_node, if both nodes for the edge" "are included in the ancestors of best_node, ignoring environmental_features") loss -= 1e-5 * "sum of log(edge_predicions[].edge_weight) for all nodes at all levels, ignoring environmental_features" return (None, loss) else: trajectories = [] for node in graph[n_horizon], nodes: if "node_prediction is from environmental_features": continue trajectory = "trajectory from all ancestors of node" trajectories. append(trajectory) edges = [] for traj_a, traj_b = "iterate all unique pairs of trajectories": if traj_a.id == traj_b.id: continue overall_edge_weight = "product of edge weights for each pair of waypoints of traj_a and traj_b" if overall_edge_weight > 0.2 or "there is no better edge connecting a pair of vehicles”: edges. append((traj_a, traj_b)) predictions = construct_graph( nodes=trajectories, edges=edges) return (predictions, None) func train_network(training_corpus:

ObjectState[n_training_sample][n_object][n_horizon]]): network = GraphNNQ

// n_train_iter may be set to 10^A6 for i_train_iter = O..n_train_iter: objects, label = random_sample(training_corpus) trajectory_predictions, loss = execute_counterfactual_graph_neural_network( network, detected_vehicles=objects["objects which are vehicles"][O], environmental_features=objects["objects which are environmental features"][O], is_training=True, ground_truth_trajectory=objects["objects which are vehicles"][:])

// ADAM learning rate may be set to 10^A-4, other ADAM paramters may be set to default values adam_optim(network, loss)

REFERENCE SIGNS

10 traffic environment

12 participant

14 ego-vehicle

16 objects

18 autonomous participants

20 ego-vehicle trajectory

22 object trajectories

24 joint set of trajectories

26 machine learning model

28 encoder layer

30 graph neural network layer

32 decoder layer

34 common layer

36 node weight prediction portion

38 edge weight prediction portion

40 input layer

42 first graph convolution layer

44 first ReLLI layer

46 dropout layer

48 second graph convolution layer

50 second ReLLI layer

52 linear layer

54 bilinear layer

56 activation layer

58 input graph

60 input ego-vehicle node

62 input object node

64 dynamic object node

66 static environment node

68 first predicted ego-vehicle node

70 second predicted ego-vehicle node

72 first predicted object node

74 second predicted object node predicted static environment node predicted edge weight output graph predicted node final output graph trajectory overall edge weight

Claims

1 . A computer-implemented method for predicting a joint set of trajectories (24) in a traffic environment (10), the method comprising: a) collating ego-vehicle data that are indicative of at least one motion or position related quantity of an ego-vehicle (14), and collating environment object data that are indicative of at least one motion or position related quantity of at least one object (16) within the traffic environment (10); b) generating an input ego-vehicle node (60) that has node feature data that are based on the ego-vehicle data and, for each object (16), generating an input object node (62) that has node feature data that are based on the environment object data, and generating an input graph (58) that includes the input ego-vehicle node (60) and each input object node (62); c) executing a graph neural network with the input graph (58) as input data, wherein the graph neural network determines at least one predicted ego-vehicle node (68, 70) and at least one predicted object node (72, 74) for each input object node (62), wherein each predicted ego-vehicle node (68, 70) is indicative of a future ego-vehicle position, wherein each predicted object node (72, 74) is indicative of a future object position, wherein the graph neural network determines a node weight for each predicted node (82), wherein the graph neural network determines a predicted edge weight (78) for at least one graph edge that connects the ego-vehicle node (68, 709 with one of the object nodes (72, 74); d) determining an ego-vehicle trajectory composed of at least one predicted egovehicle node (68, 70) and optionally of the input ego-vehicle node (60), and determining at least one object trajectory for each object (16), the object trajectory being composed of at least one predicted object node (72, 74) and optionally of the input object node (62); and e) determining a trajectory joint compatibility score based on each predicted edge weight (78) for the trajectories (86) determined in step d), wherein the trajectory joint compatibility score is indicative of whether the respective trajectories (86) are considered in further processing, and determining the joint set of trajectories (24) based on the trajectory joint compatibility score.

2. The method according to claim 1 , wherein the environment object data include dynamic object data that are indicative of the motion related quantity and include static environment data that are indicative of environment features that do not change during execution of the method steps.

3. The method according to any of the preceding claims, wherein in step b) the input ego-vehicle node (60) and the input object node (62) are generated by an input node encoder that includes a fully connected model and/or a sequence model.

4. The method according to any of the preceding claims, wherein step c) is repeated a predetermined number of times depending on a prediction horizon, wherein an output graph (80) of the previous iteration of the graph neural network becomes the input graph (58) of the next iteration.

5. The method according to any of the preceding claims, wherein step c) comprises, for each node, pruning the respective node from the graph, if the node weight is below a predetermined threshold.

6. The method according to any of the preceding claims, wherein in step c) a first and second predicted ego-vehicle node (68, 70) and/or a first and second predicted object node (72, 74) are determined.

7. The method according to claim 6, wherein in step c) a graph edge that connects the first and second predicted ego-vehicle node (68, 70) and/or the first and second predicted object node (72, 74) are removed from the graph.

8. The method according to any of the preceding claims, wherein in step d) the trajectories (86) are determined by a trajectory decoder model that includes a sequence model.

9. A computer-implemented method for determining a route through a traffic environment (10), the method comprising: a) carrying out a method according to any of the preceding claims in order to determine at least one ego-vehicle trajectory; b) determining a route property that is influenced by the ego-vehicle trajectory, the route property preferably chosen from a group consisting of ego-vehicle trajectory length, time to travel along the ego-vehicle trajectory, number of crossings with the at least one object trajectory, route destination, and fuel consumption along the egovehicle trajectory. c) selecting the ego-vehicle trajectory in accordance with the property determined in step b) and generating a control signal that causes the ego-vehicle to follow the selected trajectory or displaying the route to the driver of the ego-vehicle.

10. A computer-implemented method for training a machine learning model for use in a method according any of the preceding claims, the training method using a loss function based on a deviation from a ground truth trajectory and an independent cross-entropy loss for each node weight and/or edge weight.

11 . A data processing apparatus comprising means for carrying out a method according to any of the preceding claims.

12. A vehicle comprising a data processing apparatus according to claim 11 .

13. A method for operating a vehicle according to claim 12, wherein the data processing apparatus carries out a method according to claim 9, so as to obtain an ego-vehicle trajectory for the vehicle and the data processing apparatus processes the control signal and controls the vehicle such that the vehicle follows the egovehicle trajectory.

14. A computer program comprising instructions which, when the program is executed by a data processing apparatus, cause data processing apparatus to carry out a method according to any of the claims 1 to 10.

15. A computer-readable medium or a data carrier signal comprising a computer program according to claim 14.