CN114997067A - Trajectory prediction method based on space-time diagram and space-domain aggregation Transformer network - Google Patents

Abstract

The invention discloses a trajectory prediction method based on a space-time diagram and airspace aggregation Transformer network, which solves the problem of insufficient extraction of interactive features in the existing pedestrian trajectory prediction output, uses a space-time diagram convolution neural network and a time sequence feature transformation network to operate and finish effective and accurate extraction of pedestrian trajectory features in a scene, meanwhile, a brand-new airspace aggregation Transformer framework is designed to perform pedestrian time sequence characteristic transformation, efficient aggregation and utilization of airspace pedestrian characteristics are completed, output of predicted tracks of pedestrians is finally completed in a probability distribution mode, the purposes of reasonably avoiding sudden conditions and keeping movement consistency of group pedestrians are achieved, relevant indexes show that the framework breaks through in the aspect of predicting pedestrian endpoints, the purpose of more accurate and efficient prediction of pedestrian track distribution is completed, and important help is provided for development in the fields of automatic driving, intelligent traffic and the like.

Description

A Trajectory Prediction Method Based on Spatio-temporal Graph and Spatial Domain Aggregated Transformer Network

technical field

The invention relates to a trajectory prediction method based on a spatiotemporal map and a spatial domain aggregation Transformer network, and belongs to the field of artificial intelligence and automatic driving.

Background technique

Pedestrian trajectory prediction technology has profound theoretical background and practical application value. In fields such as unmanned driving and intelligent monitoring, pedestrian trajectory recognition and prediction technology has always occupied a relatively important position. In recent years, due to the advancement of artificial intelligence and deep learning technology, the implementation and application of intelligent algorithms related to pedestrian trajectory prediction has gradually attracted attention and heated discussions.

For a long time, it has always been the pedestrian trajectory to enable the agent to better understand and judge the behavior characteristics of the traffic participants in the scene, establish a pedestrian trajectory prediction model with spatial interactive feature information and make relevant predictions, and then make accurate, fast and reasonable relevant decisions. Predict what the problem is trying to achieve. However, the high complexity and uncertainty of the pedestrian trajectory prediction problem determines that it has the following difficulties: the complex scene feature information makes the pedestrian's future trajectory not only affected by its own historical trajectory and the established trajectory route, but also by the obstacles in the scene. Multiple effects with other traffic participants in the spatiotemporal dimension. Therefore, whether a reasonable and accurate model can be established and rapid prediction output and decision-making can be performed is the key to the application of pedestrian trajectory prediction in practical scenarios.

Thanks to the development of machine learning in the field of artificial intelligence, trajectory prediction methods based on LSTM and CNN algorithms have been the mainstream prediction methods for a long time. This type of prediction method has a simple model, and can achieve fairly good prediction results with fewer parameters and a more basic model architecture. These architectures also provide ideas and basic module frameworks for subsequent in-depth algorithm research, which is of pioneering significance.

Since graphs and their network architectures have natural advantages in the representation of data and information for pedestrian trajectory prediction problems, graph-based pedestrian trajectory prediction research has become a hot research direction in recent years. Mohamed A et al. used the method of spatiotemporal graph neural network in the literature of 2020 (Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction [C]), and conducted two different temporal and spatial domains respectively. The convolution operation is used to obtain the trajectory feature information and predict the output at the same time. Similarly, the model considers the randomness and uncertainty of pedestrian trajectories in space, that is, the model does not know the predetermined trajectory and end point of each pedestrian in advance when predicting. Therefore, a reasonable research method is to assume that the pedestrian predicts the trajectory. The abscissa and ordinate of are conformed to a two-dimensional Gaussian distribution, and the trajectory is output by sampling in the process of testing the prediction. The model also completed predictions based on such assumptions, and achieved relatively good results. However, in such pedestrian trajectory prediction model, there is still no further processing of pedestrian interaction feature information, resulting in insufficient spatial interaction ability, resulting in a large trajectory inertia, and it cannot be closely related according to the movement patterns between groups of pedestrians. Motion prediction.

In recent years, many scholars have studied pedestrian trajectory prediction based on graph representation, combined with many other different algorithm tools and research methods, and have made progress in many aspects. In the literature (Spatial-TemporalBlock and LSTM Network for Pedestrian Trajectories Prediction[J]), Dan X et al. proposed a pedestrian trajectory prediction model architecture based on spatiotemporal modules and LSTM. The relationship feature vector between individual pedestrian nodes and their neighbors is input into the spatiotemporal graph pedestrian interaction feature vector encoded in LSTM, and then related predictions are made, and good prediction results are obtained; Rainbow B A et al. in the literature (Semantics-STGCNN) In :A semantics-guided spatial-temporal graph convolutionalnetwork for multi-class trajectory prediction[C]), a semantic-based spatial-temporal graph pedestrian trajectory prediction model Semantics-STGCNN is proposed. The model starts from the scene semantic understanding and combines the pedestrian object's trajectory. The category label is embedded in the label adjacency matrix, combined with the velocity adjacency matrix, the semantic adjacency matrix is output, the modeling of semantic information is completed, and the prediction result is finally output; Yu C et al. in the literature (Spatio-temporal graph transformer networks for pedestrian trajectory prediction[C ]), a Transformer-based network model is used, which uses Transformer's excellent performance in other fields to design and directly splicing multiple Transformer basic frameworks to extract the spatiotemporal feature information of pedestrians in the scene and complete related predictions.

In the present invention, in view of the problems and deficiencies existing in the existing trajectory prediction methods in the extraction and prediction of pedestrian spatial interaction features, a new network architecture using spatiotemporal map and spatial aggregation Transformer for pedestrian trajectory prediction is proposed. Carry out proper graph representation and preprocessing, use spatiotemporal graph convolutional neural network and time series feature transformation network to extract original pedestrian trajectory feature information, and introduce a spatial Transformer network architecture to fully extract and aggregate deep spatial feature information to ensure that Model effectiveness and accuracy in spatial pedestrian interaction features. The invention pays attention to the rationality of the model prediction results in the aspect of space interaction, ensures the pedestrian space walking characteristics while taking into account the interaction influence, and especially achieves a breakthrough in the prediction of the end point of the pedestrian trajectory, and is used for modeling the pedestrian trajectory interaction in complex scenes and predicting the pedestrian trajectory. It plays a positive role, and helps and inspires research and exploration in areas such as unmanned driving and artificial intelligence.

SUMMARY OF THE INVENTION

The invention discloses a trajectory prediction method based on a spatiotemporal map and a spatial domain aggregated Transformer network. The method aims at the insufficient extraction of spatial pedestrian trajectory information in the existing pedestrian trajectory prediction method, which results in that the relative position relationship of pedestrians is not clear enough when walking, and cannot be used for collisions and the like. To solve problems such as large-scale rotation, consider establishing a new trajectory prediction model architecture in the form of a graph, and complete the feature extraction of pedestrians in the scene through operations such as spatiotemporal graph convolutional neural networks and time series feature transformation networks, and design a new The spatial aggregation Transformer architecture performs the transformation and utilization of pedestrian time series features, and finally completes the output of pedestrian predicted trajectories in the form of probability distribution to achieve the purpose of reasonable prediction.

In terms of spatiotemporal graph convolutional neural network, the pedestrian trajectory feature information in the scene is represented and preprocessed in the form of a graph, and a graph convolutional neural network is constructed to complete the preliminary extraction of pedestrian trajectory feature information in space as the subsequent network input.

In the time series feature transformation network, the extraction of time series feature information and the transformation of feature dimension are completed through a convolution extending into the network, and the network is reasonably designed to simplify the model parameters and improve the model performance.

In the spatial aggregation Transformer network, the features previously obtained from the spatiotemporal graph convolutional neural network and the temporal feature transformation network are further processed. In order to further mine and model the interaction of pedestrian features in the spatial scene, the model in the present invention uses the time series feature vector of each pedestrian as an input vector, and inputs it into the spatial aggregation Transformer network to fully extract and aggregate the pedestrian spatial trajectory features. , while completing the task of trajectory prediction output.

The present invention mainly comprises the following steps:

Step (1): Use the characteristics of the graph to represent and preprocess the pedestrian trajectory feature information in the scene from the input original data, select the appropriate kernel function to complete the construction of the adjacency matrix, and provide accurate and efficient input for the subsequent network architecture. information of pedestrians in the scene;

Step (2): Establish a spatiotemporal graph convolutional neural network module, construct a graph convolutional neural network, and complete the preliminary extraction of pedestrian trajectory feature information in space by selecting the number of graph convolutions for pedestrian trajectory features to ensure accurate and accurate extraction of features. efficient;

Step (3): establish a time series feature transformation network module, and complete the extraction of time series features and the transformation of feature dimensions by designing a convolutional neural network;

Step (4): Establish an airspace aggregation Transformer network, use the time series feature vector of each pedestrian in the scene as an input vector, and input the Transformer network to further aggregate airspace features, and complete the output of the pedestrian trajectory prediction sequence.

Further, in the step (1), a spatiotemporal graph is introduced to represent the input original pedestrian trajectory data, and an appropriate kernel function is selected from a variety of kernel functions to construct an adjacency matrix in the sense of graph, so as to complete an efficient pedestrian in the scene. Feature construction and selection provide accurate and efficient information for subsequent modeling.

每个

的特征信息使用观测的相对坐标变化

来进行刻画，即：Further, the introduction of a space-time map to graphically represent the input original pedestrian trajectory data is specifically: for each time t, a space map G _t is introduced to represent the interactive feature relationship between pedestrians at each time point; G _t Defined as G _t =(V _t , E _t ), where V _t specifically represents the coordinate information of pedestrians in the scene at time t, namely

each

The feature information of using the observed relative coordinate changes

to characterize, that is:

Among them, i=1,...,N, t=2,...,T _obs , for the initial moment, the relative position offset is defined as 0, that is,

的取值由如下方式给出：E _t represents the side information of the spatial graph G _t , which is a matrix with a dimension of n×n; it is defined as

The value of is given by:

与节点

相连，那么

反之，如果节点

与节点

不相连，那么

if node

with node

connected, then

Conversely, if the node

with node

not connected, then

Further, the adjacency matrix in the sense of selecting a suitable kernel function from a variety of kernel functions to construct a graph is specifically:

The weighted adjacency matrix A _t is introduced to represent the node information of the pedestrian spatial graph by weight, and the magnitude of the mutual influence between pedestrians is obtained through the kernel function transformation and stored in the weighted adjacency matrix A _t ;

The inverse of the distance between two nodes in Euclidean space is used as the kernel function, and in order to avoid the function divergence problem caused by the two being too close, a tiny constant ε is added to accelerate the model convergence, the expression is as follows:

In the time dimension, the spatial graph G _t at each moment is stacked, that is, the pedestrian trajectory prediction spatio-temporal graph sequence G={G ₁ ,...,G _T } under the graph representation is obtained.

Further, the step (2) is specifically:

For the input feature map time series, the output is obtained through the established spatiotemporal map convolutional neural network:

e _t =GNN(G _t ) (1.6)

Among them, GNN represents the constructed spatiotemporal graph convolutional neural network, which obtains the output result by iterative multi-layer graph convolution; e _t represents the spatiotemporal feature information initially extracted from the spatial dimension through the graph neural network;

For the output of each moment, there is such an operation; and the output obtained by the actual graph convolutional neural network is a stack of such time series:

e _g = Stack(e _t ) (1.7)

Among them, Stack( ) represents the superposition of the input on the extended dimension, and e _g represents the output of the graph convolution; in the actual processing process, multiple extended dimensions are simultaneously sent to the graph neural network for processing in parallel;

Then, through a fully connected layer FC, the features are appropriately dimensionally transformed:

V _GNN = FC(e _g ) (1.8)

Thereby, the preliminary extraction output of the feature information of the spatiotemporal graph convolutional neural network is obtained.

Further, in the step (3), the output of the spatiotemporal graph convolutional neural network is subjected to dimensional transformation, and a CNN-based time series feature transformation network module is used and the number of convolutions is designed to complete the extraction of the pedestrian's own historical trajectory feature information;

Further, the step (3) is specifically:

After obtaining the feature extraction information of the spatiotemporal graph convolutional neural network, it is sent to a time series feature transformation network to extract the time series features; since the dimension features have been appropriately transformed through a fully connected layer in step 2, the The network module directly utilizes the obtained feature information; in the present invention, the multi-layer CNN convolutional neural network is selected to process the time dimension feature information, which can be expressed as:

_ec = CNN(V _GNN ) (1.9)

Among them, V _GNN represents the feature information extracted from the graph convolutional neural network, and e _c represents the output of the time series feature transformation network; then a multi-layer perceptron MLP is used to increase the expressive ability of the network:

V _CNN = MLP( _ec ) (1.10)

The feature transformation and processing are performed through the above network, that is, the output _VCNN of the time series feature transformation network is obtained.

Further, the main construction and calculation content of step 4 includes: in order to increase the connection between pedestrian features in the airspace, an airspace Transformer network is designed to further spatially aggregate the above extracted feature information. In particular, the feature vector of the same pedestrian in the time series is input as the input vector, and the extracted features of different pedestrians are input in sequence.

For the spatial aggregation Transformer network, the encoder layer of the Transformer architecture is selected, and position encoding is first added to the input:

V _in = V _CNN +PE _pos,i (V _CNN ) (1.11)

where pos represents the relative position of the input feature and i represents the dimension of the input feature. Then introduce a multi-head attention layer, use the three attention layers obtained by matrix transformation from the input layer to input Query(Q), Key(K), Value (V), divide the input features according to the set number of heads, and calculate The attention score, expressed as follows:

head _i =Attention(Q _i ,K _i ,V _i ) (1.13)

Among them, i=1,...,nhead, nhead represents the number of long heads. The final multi-head output completes feature extraction by splicing, and the expression is as follows:

V _Multi = ConCat(head ₁ ,...,head _h )W _o (1.14)

Among them, _ConCat represents the concatenation operation, and Wo represents the parameter matrix output by the attention layer.

Then, the final output of the spatial Transformer is completed through the feedforward neural network and layer normalization, which is expressed as:

V _out = LN(Feedback(V _Multi )) (1.15)

Through this architectural method, the aggregation of pedestrian spatial interaction features through the initially extracted spatiotemporal features can be better completed, and the goal of better outputting pedestrian trajectories that conform to the pedestrian association and interaction in the scene is achieved.

In terms of loss function, the sum of negative log-likelihoods of each point on the pedestrian predicted trajectory is selected as the loss function. The loss function of the i-th pedestrian is expressed as follows:

是待预测的未知的行人轨迹特征参数，T_obs，T_pred分别表示观测和预测终点时刻；而所有行人的损失函数之和即为最终的损失函数：in,

is the unknown pedestrian trajectory feature parameter to be predicted, T _obs , T _pred represent the end point of observation and prediction respectively; and the sum of the loss functions of all pedestrians is the final loss function:

By performing forward loss function calculation and reverse parameter update on the above-mentioned model architecture proposed by the present invention, the training of the model can be completed, and a reasonable pedestrian predicted trajectory output can be obtained.

beneficial effect

In order to solve the problem of insufficient interactive feature extraction in the existing pedestrian trajectory prediction output, which leads to the inconspicuous pedestrian spatial characteristics, on the one hand, the pedestrian prediction trajectory has a large inertia, and it cannot be used for high-speed, sudden and other situations. Avoidance, on the other hand, is manifested in the problem that the movement consistency of pedestrian group behaviors is not maintained enough, resulting in the problem that the closely related people in the space cannot maintain the same movement trend for a period of time. The present invention proposes a new network model architecture. , using spatiotemporal graph convolutional neural network and time series feature transformation network and other related transformation operations to complete the effective and accurate extraction of pedestrian features in the scene, and design a new spatial aggregation Transformer architecture to transform and utilize pedestrian time series features, and finally use probability distribution It completes the output of pedestrian predicted trajectories in the form of a reasonable avoidance of emergencies, maintains the consistency of group pedestrian movements, completes more accurate and reasonable prediction of pedestrian space interaction, and conducts further in-depth research on pedestrian trajectory prediction. It provides new ideas for more accurate and timely prediction and application in actual scenarios, and provides help for the development of autonomous driving, intelligent transportation and other fields.

Description of drawings

Fig. 1 is the overall schematic diagram of the Transformer network framework based on the spatiotemporal map and the spatial domain aggregation in the specific embodiment of the present invention;

FIG. 2 is a schematic diagram of using time series transformation features to input spatial aggregation Transformer network to perform trajectory prediction in the present invention.

Detailed ways

The present invention relates to a pedestrian trajectory prediction method based on a spatiotemporal map and a spatial domain aggregation Transformer network. The specific implementation mainly includes the following steps:

表示。有了如上定义，那么本问题的一般表述为，对每一组已知的给定观测行人轨迹序列：For the pedestrian trajectory prediction problem in a given scene, it consists of the coordinates of N pedestrians in the scene at each observation moment. For the coordinate information of the i-th pedestrian at the t-th time, use

express. With the above definition, the general formulation of this problem is, for each set of known sequences of observed pedestrian trajectories:

By constructing a network framework, the pedestrian trajectory characteristics are extracted and modeled through the input data to obtain appropriate trajectory characteristic information, and a reasonable trajectory prediction output in the scene is given:

表示模型给出的行人轨迹预测值。where T _obs and T _pred represent the pedestrian observation time span and prediction time span, respectively, ( ) represents the true value of pedestrian trajectory prediction,

Represents the predicted pedestrian trajectory given by the model.

The overall schematic diagram of the Transformer network framework based on the spatiotemporal map and the spatial domain aggregation in the specific embodiment of the present invention is shown in FIG. 1 .

Step 1: Appropriate graph representation and preprocessing of the data to provide accurate and efficient pedestrian information in the scene

In the present invention, an appropriate graph representation method is used to first perform correlation graph transformation and preprocessing on the input original pedestrian trajectory data, so as to facilitate subsequent extraction and efficient use of input feature information.

对于本模型，每个

的特征信息使用观测的相对坐标变化

来进行刻画，即：For each time t, a spatial graph G _t is introduced to represent the interactive feature relationship between pedestrians at each time point. G _t is defined as G _t =(V _t , E _t ), where V _t represents the node information of the spatial graph G _t . In this model, V _t specifically represents the coordinate information of pedestrians in the scene at time t, that is,

For this model, each

The feature information of using the observed relative coordinate changes

to characterize, that is:

Among them, i=1,...,N, t=2,...,T _obs , for the initial moment, the relative position offset is defined as 0, that is,

的取值由如下方式给出：如果节点

与节点

相连，那么

反之，如果节点

与节点

不相连，那么

E _t represents the side information of the spatial graph G _t , which is a matrix with a dimension of n×n. It is usually defined as

The value of is given by: if the node

with node

connected, then

Conversely, if the node

with node

not connected, then

For this prediction task, it is not only hoped to obtain whether the pedestrians are related, but also to measure the relative size of the mutual influence between pedestrians in the space. Therefore, a weighted adjacency matrix A _t is introduced to weight the node information of the pedestrian space graph, and transform it through a kernel function. The size of the mutual influence between pedestrians is obtained and stored in the weighted adjacency matrix A _t . In the present invention, the inverse of the distance between the two nodes in the Euclidean space is selected as the kernel function, and in order to avoid the function divergence problem caused by the two being too close, A tiny constant ε is added to speed up the model convergence, and the expression is as follows:

In the time dimension, the spatial graph G _t at each moment is stacked, that is, the pedestrian trajectory prediction spatio-temporal graph sequence G={G ₁ ,...,G _T } under the graph representation is obtained. Through this definition and transformation, the graph representation and preprocessing of the data in the pedestrian trajectory prediction problem are completed.

Step 2: Establish a spatiotemporal graph convolutional neural network to initially extract feature information

In the present invention, the feature information is preliminarily extracted by using a spatiotemporal graph convolutional neural network for the data after the original data is represented graphically in step 1.

In this model architecture, the graph convolutional neural network is used to determine the appropriate number of convolutional layers and perform appropriate feature iterations to achieve the purpose of better extraction of trajectory features in space.

For the input feature map time series, the output is obtained through the established spatiotemporal map convolutional neural network:

e _t =GNN(G _t ) (1.6)

Among them, GNN represents the constructed spatiotemporal graph convolutional neural network, which is iterated by multiple layers of graph convolution to obtain the output result; e _t represents the spatiotemporal feature information initially extracted from the spatial dimension through the graph neural network.

For the output at each moment, there is such an operation. The output obtained by the actual graph convolutional neural network is a stack of such time series:

e _g = Stack(e _t ) (1.7)

Among them, Stack( ) represents the superposition of the input in the extended dimension, and e _g represents the output of the graph convolution. In the actual processing process, multiple extended dimensions are simultaneously sent to the graph neural network for processing in parallel.

Then, through a fully connected layer FC, the features are appropriately dimensionally transformed:

V _GNN = FC(e _g ) (1.8)

Thereby, the preliminary extraction output of the feature information of the spatiotemporal graph convolutional neural network is obtained.

Step 3: Establish a time series feature transformation network, and complete the time series feature extraction and feature dimension transformation by designing a convolutional neural network;

After obtaining the feature extraction information of the spatiotemporal graph convolutional neural network, it is sent to a time series feature transformation network to extract the time series features. Since the dimension feature has been appropriately transformed by a fully connected layer in step 2, the network module in this step directly utilizes the obtained feature information. In the present invention, the multi-layer CNN convolutional neural network is selected to process the time dimension feature information, which can be expressed as:

_ec = CNN(V _GNN ) (1.9)

Among them, V _GNN represents the feature information extracted from the graph convolutional neural network, and _ec represents the output of the time series feature transformation network. Then pass a multi-layer perceptron MLP to increase the expressive ability of the network:

V _CNN = MLP( _ec ) (1.10)

The feature transformation and processing are performed through the above network, that is, the output _VCNN of the time series feature transformation network is obtained.

Step 4: Establish an airspace aggregation Transformer network for further aggregation of airspace features, and complete the output of the pedestrian trajectory prediction sequence

In order to solve the problem of insufficient interactive feature extraction in the existing pedestrian trajectory prediction output, which leads to the inconspicuous pedestrian spatial characteristics, on the one hand, the pedestrian prediction trajectory has a large inertia, and it cannot be used for high-speed, sudden and other situations. Avoidance, on the other hand, is manifested in that the movement consistency of pedestrian group behavior is not maintained enough, resulting in the inability of closely related groups in space to maintain the same movement trend for a period of time.

In the present invention, in order to increase the connection between pedestrian features in the airspace, an airspace Transformer network is designed to further spatially aggregate the above extracted feature information. In particular, the feature vector of the same pedestrian in the time series is input as the input vector, and the extracted features of different pedestrians are input in sequence.

For the spatial aggregation Transformer network, the encoder layer of the Transformer architecture is selected, and position encoding is first added to the input:

V _in = V _CNN +PE _pos,i (V _CNN ) (1.11)

where pos represents the relative position of the input feature and i represents the dimension of the input feature. Then introduce a multi-head attention layer, use the three attention layers obtained by matrix transformation from the input layer to input Query(Q), Key(K), Value (V), divide the input features according to the set number of heads, and calculate The attention score, expressed as follows:

head _i =Attention(Q _i ,K _i ,V _i ) (1.13)

Among them, i=1,...,nhead, nhead represents the number of long heads. The final multi-head output completes feature extraction by splicing, and the expression is as follows:

V _Multi = ConCat(head ₁ ,...,head _h )W _o (1.14)

Among them, _ConCat represents the concatenation operation, and Wo represents the parameter matrix output by the attention layer.

Then, the final output of the spatial Transformer is completed through the feedforward neural network and layer normalization, which is expressed as:

V _out = LN(Feedback(V _Multi )) (1.15)

Through this architectural method, the aggregation of pedestrian spatial interaction features through the initially extracted spatiotemporal features can be better completed, and the goal of better outputting pedestrian trajectories that conform to the pedestrian association and interaction in the scene is achieved.

In terms of loss function, the sum of negative log-likelihoods of each point on the pedestrian predicted trajectory is selected as the loss function. The loss function of the i-th pedestrian is expressed as follows:

是待预测的未知的行人轨迹特征参数，T_obs，T_pred分别表示观测和预测终点时刻；而所有行人的损失函数之和即为最终的损失函数：in,

is the unknown pedestrian trajectory feature parameter to be predicted, T _obs , T _pred represent the end point of observation and prediction respectively; and the sum of the loss functions of all pedestrians is the final loss function:

By performing forward loss function calculation and reverse parameter update on the above-mentioned model architecture proposed by the present invention, the training of the model can be completed, and a reasonable pedestrian predicted trajectory output can be obtained.

In the process of evaluating the accuracy and validity of the model, similar to the commonly used trajectory prediction evaluation methods, the average offset error (Average Differential Error, ADE) and the end point offset error (Final Differential Error, FDE) are used as evaluation indicators to describe the prediction. accuracy of the trajectory. The average displacement error refers to the average value of the L2 norm of the predicted displacement and the actual displacement error of each pedestrian in the scene at each moment, and the end point displacement error refers to the difference between the predicted displacement and the actual displacement error of each pedestrian in the scene at the end point. The average value of the L2 norm, which is expressed as:

表示预测行人待预测的轨迹真值，

表示该模型的输出的行人预测轨迹； T_pred表示预测终点时刻，T_p表示预测时间范围，对于FDE指标，仅对于场景内每个行人终点坐标的误差取平均而对于行走路线的选择没有较高要求，而对于ADE指标，则需要对每个时刻点的坐标误差求和取平均。对两种指标而言，值越小表明与实际轨迹越接近，预测性能也越好。in,

represents the true value of the predicted pedestrian trajectory to be predicted,

Represents the pedestrian predicted trajectory of the output of the model; T _pred represents the predicted end point time, T _p represents the prediction time range, for the FDE indicator, only the error of the end point coordinates of each pedestrian in the scene is averaged and the selection of the walking route is not high requirements, and for the ADE index, the sum of the coordinate errors at each time point needs to be averaged. For both metrics, the smaller the value, the closer it is to the actual trajectory and the better the prediction performance.

Since the actual output is the probability distribution of the trajectory in the two-dimensional plane, when evaluating the actual trajectory prediction performance, in order to ensure the trajectory diversity and generalization ability, multiple sampling predictions (such as 20 times) are often used, and the closest to the true trajectory is used for prediction. The predicted trajectories of the value trajectories are used as output trajectories to calculate ADE/FDE and evaluate the model. Specifically, for the five datasets on ETH and UCY, the pedestrian trajectory data is generated by sampling every 0.4 seconds, and every 20 frames is used as a data sample, by giving the pedestrian trajectory of the past 8 frames for a total of 3.2 seconds. The data is used as input, and the model is trained and verified by predicting pedestrian trajectories in the next 12 frames for a total of 4.8s. The performance of the model in the present invention is compared with other two algorithms that also use the graph network model, and the obtained comparison results are shown in Table 1, and the best performance is marked in red:

Table 1 Comparison of prediction results between this model and mainstream models of graph networks

It can be seen from Table 1 that the framework proposed by the present invention has made a great breakthrough in the end point prediction problem, and the FDE index is the best in almost all data sets, and the average ADE and FDE indicators are also the best performance. Compared with the optimal performance of the two graph network algorithms, this model improves the FDE by 17%, 21%, 5%, and 12% on ETH, UNIV, ZARA1, and ZARA2 respectively, and the average FDE index improves by 16%. It can be seen from the above data that this model uses a spatial aggregation Transformer architecture to input the pedestrian time series feature vector, focusing on the utilization of the extracted features from the spatiotemporal graph neural network and the time series feature transformation network, and completes the spatial pedestrian interaction feature. Good aggregation, achieves better prediction effect, and has made great breakthroughs in FDE, and has stronger perception and expression of pedestrian interaction characteristics in space.

Claims (8)

1. a trajectory prediction method based on spatiotemporal map and spatial domain aggregation Transformer network, is characterized in that, comprises the following steps: (1) Use the characteristics of the graph to represent and preprocess the pedestrian trajectory feature information in the scene from the input original data, select the appropriate kernel function to complete the construction of the adjacency matrix, and provide an accurate and efficient scene for the subsequent network architecture input Insider trajectory feature information; (2) Establish a spatiotemporal graph convolutional neural network module, and construct a graph convolutional neural network. By selecting the number of graph convolutions for pedestrian trajectory features, the preliminary representation of the graph in step (1) and the preprocessed pedestrian trajectory feature information are completed. Extraction to ensure that the extracted features are accurate and effective; (3) Establish a time series feature transformation network module, and complete the extraction of time series feature information and the transformation of feature dimensions by designing a convolutional neural network; (4) Establish a spatial aggregation Transformer network, use the time series feature vector of each pedestrian in the scene as the input vector, and input the Transformer network to further aggregate spatial features, and complete the output of the pedestrian trajectory prediction sequence. 2. a kind of trajectory prediction method based on spatiotemporal map and spatial domain aggregation Transformer network as claimed in claim 1, it is characterized in that, in described step (1), introduce spatiotemporal map to the original pedestrian trajectory data of input to carry out graph representation, Select the appropriate kernel function from a variety of kernel functions to construct an adjacency matrix in the sense of graph, complete the efficient construction and selection of pedestrian features in the scene, and provide accurate and efficient information for subsequent modeling. 3. a kind of trajectory prediction method based on spatiotemporal map and spatial domain aggregation Transformer network as claimed in claim 2, it is characterized in that, described introducing spatiotemporal map to the original pedestrian trajectory data of input carries out graph representation and is specifically: for each moment t, a spatial graph G _t is introduced to represent the interactive feature relationship between pedestrians at each time point; G _t is defined as G _t =(V _t , E _t ), where V _t specifically represents the time t of pedestrians in the scene. coordinate information, i.e.

each

The feature information of using the observed relative coordinate changes

to characterize, that is:

Among them, i=1,...,N, t=2,...,T _obs , for the initial moment, the relative position offset is defined as 0, that is,

E _t represents the side information of the spatial graph G _t , which is a matrix with a dimension of n×n; it is defined as

The value of is given by: if node

with node

connected, then

Conversely, if the node

with node

not connected, then

4. a kind of trajectory prediction method based on spatiotemporal graph and space domain aggregation Transformer network as claimed in claim 2, it is characterized in that, the described adjacency matrix under the meaning of selecting suitable kernel function construction graph from multiple kernel functions is specifically: : The weighted adjacency matrix A _t is introduced to represent the node information of the pedestrian spatial graph by weight, and the magnitude of the mutual influence between pedestrians is obtained through the kernel function transformation and stored in the weighted adjacency matrix A _t ; The inverse of the distance between two nodes in Euclidean space is used as the kernel function, and in order to avoid the function divergence problem caused by the two being too close, a tiny constant ε is added to accelerate the model convergence, the expression is as follows:

In the time dimension, the spatial graph G _t at each moment is stacked, that is, the pedestrian trajectory prediction spatio-temporal graph sequence G={G ₁ ,...,G _T } under the graph representation is obtained. 5. a kind of trajectory prediction method based on space-time map and space domain aggregation Transformer network as claimed in claim 4, is characterized in that, described step (2) is specifically: For the input feature map time series, the output is obtained through the established spatiotemporal map convolutional neural network: e _t =GNN(G _t ) (1.6) Among them, GNN represents the constructed spatiotemporal graph convolutional neural network, which obtains the output result by iterative multi-layer graph convolution; e _t represents the spatiotemporal feature information initially extracted from the spatial dimension through the graph neural network; For the output of each moment, there is such an operation; and the output obtained by the actual graph convolutional neural network is a stack of such time series: e _g = Stack(e _t ) (1.7) Among them, Stack( ) represents the superposition of the input on the extended dimension, and e _g represents the output of the graph convolution; in the actual processing process, multiple extended dimensions are simultaneously sent to the graph neural network for processing in parallel; Then, through a fully connected layer FC, the features are appropriately dimensionally transformed: V _GNN = FC(e _g ) (1.8) Thereby, the preliminary extraction output of the feature information of the spatiotemporal graph convolutional neural network is obtained. 6. a kind of trajectory prediction method based on spatiotemporal graph and space domain aggregation Transformer network as claimed in claim 1, is characterized in that, in described step (3), the output of spatiotemporal graph convolutional neural network is transformed through appropriate dimension, Using a CNN-based time series feature transformation network module and designing the number of convolutions to complete the extraction of the pedestrian's own historical trajectory feature information. 7. a kind of trajectory prediction method based on spatiotemporal map and space domain aggregation Transformer network as claimed in claim 6, is characterized in that, described step (3) is specifically: After obtaining the feature extraction information of the spatiotemporal graph convolutional neural network, it is sent to a time series feature transformation network to extract the time series features; since the dimension features have been appropriately transformed through a fully connected layer in step 2, the The network module directly utilizes the obtained feature information; in the present invention, the multi-layer CNN convolutional neural network is selected to process the time dimension feature information, which can be expressed as: _ec = CNN(V _GNN ) (1.9) Among them, V _GNN represents the feature information extracted from the graph convolutional neural network, and e _c represents the output of the time series feature transformation network; then a multi-layer perceptron MLP is used to increase the expressive ability of the network: V _CNN = MLP( _ec ) (1.10) The feature transformation and processing are performed through the above network, that is, the output _VCNN of the time series feature transformation network is obtained. 8. a kind of trajectory prediction method based on spatiotemporal map and spatial domain aggregation Transformer network as claimed in claim 1, is characterized in that, described step (4) is specifically: The feature vector of the same pedestrian in the time series is input as the input vector, and the extracted features of different pedestrians are input in turn; For the spatial aggregation Transformer network, the encoder layer of the Transformer architecture is selected, and position encoding is first added to the input: V _in = V _CNN +PE _pos,i (V _CNN ) (1.11) Among them, pos represents the relative position of the input feature, and i represents the dimension of the input feature; then a multi-head attention layer is introduced, and the three attention layers obtained by matrix transformation from the input layer are used to input Query, Key, and Value, according to the set number of heads. Divide the input features and calculate the attention score, the expression is as follows:

head _i =Attention(Q _i ,K _i ,V _i ) (1.13) Among them, i=1,...,nhead, nhead represents the number of heads; and the final output of multiple heads completes feature extraction by splicing, and the expression is as follows: V _Multi = ConCat(head ₁ ,...,head _h )W _o (1.14) Among them, _ConCat represents the concatenation operation, and Wo represents the parameter matrix output by the attention layer. Then, the final output of the spatial Transformer is completed through the feedforward neural network and layer normalization, which is expressed as: V _out = LN(Feedback(V _Multi )) (1.15) In terms of loss function, the sum of the negative log-likelihood of each point on the pedestrian predicted trajectory is selected as the loss function; the loss function of the i-th pedestrian is expressed as follows:

in,

is the unknown pedestrian trajectory feature parameter to be predicted, T _obs , T _pred represent the end point of observation and prediction respectively; and the sum of the loss functions of all pedestrians is the final loss function:

By performing forward loss function calculation and reverse parameter update on the above model architecture, the training of the model is completed, and a reasonable pedestrian predicted trajectory output is obtained.

Images