CN116523002A - Method and system for predicting dynamic graph generation countermeasure network track of multi-source heterogeneous data - Google Patents
Method and system for predicting dynamic graph generation countermeasure network track of multi-source heterogeneous data Download PDFInfo
- Publication number
- CN116523002A CN116523002A CN202310577728.XA CN202310577728A CN116523002A CN 116523002 A CN116523002 A CN 116523002A CN 202310577728 A CN202310577728 A CN 202310577728A CN 116523002 A CN116523002 A CN 116523002A
- Authority
- CN
- China
- Prior art keywords
- track
- data
- network
- graph
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000012549 training Methods 0.000 claims abstract description 51
- 238000013528 artificial neural network Methods 0.000 claims abstract description 20
- 230000003993 interaction Effects 0.000 claims abstract description 19
- 239000013598 vector Substances 0.000 claims description 51
- 230000006870 function Effects 0.000 claims description 25
- 230000004913 activation Effects 0.000 claims description 17
- 238000012360 testing method Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 15
- 238000010586 diagram Methods 0.000 claims description 13
- 238000009826 distribution Methods 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000006073 displacement reaction Methods 0.000 claims description 8
- 238000007781 pre-processing Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 6
- 238000013459 approach Methods 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 5
- 238000009827 uniform distribution Methods 0.000 claims description 5
- 239000000203 mixture Substances 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 239000002131 composite material Substances 0.000 claims description 3
- 230000007774 longterm Effects 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 238000003786 synthesis reaction Methods 0.000 claims description 2
- 230000003213 activating effect Effects 0.000 claims 1
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 230000006403 short-term memory Effects 0.000 abstract description 2
- 230000002452 interceptive effect Effects 0.000 abstract 1
- 230000007787 long-term memory Effects 0.000 abstract 1
- 238000012545 processing Methods 0.000 description 11
- 230000006399 behavior Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 241000282414 Homo sapiens Species 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000008485 antagonism Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 208000001613 Gambling Diseases 0.000 description 1
- 206010039203 Road traffic accident Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003042 antagnostic effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a method and a system for predicting a track of a dynamic graph generation countermeasure network of multi-source heterogeneous data, wherein a track prediction generator and a semantic type generator are used for generating future tracks and semantic types of a moving target, the graph sequence data with random weights are used for establishing an interactive relation between the moving targets, and the evolution modes of node attributes and edge attributes are directly learned through a graph neural network and a long-term and short-term memory network so as to improve the generation capability of the generator through countermeasure training. The method can autonomously capture potential interaction information of all graph nodes and semantic information of graph nodes with different semantic types, and improves accuracy and robustness of track prediction. The application of the method is beneficial to solving the problem of predicting the track of the moving target in the fields of traffic management, intelligent traffic, robot navigation and the like, and has wide application prospect and beneficial social benefit.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a method and a system for predicting a dynamic graph generation countermeasure network track of multi-source heterogeneous data.
Background
The birth of the automobile has very important significance in the history of human development, not only expands the range of human activities, brings great convenience to the life of human beings, but also drives the vigorous development of most industries of society. Since the advent of automobiles, the inventors began to study automatic driving automobiles, and particularly in recent years, with the development of science and technology, artificial neural networks, more and more companies began to actively explore automatic driving technologies, and automobiles are developing to intelligentization and networking.
The track prediction is one of important technologies for the intelligent development of the automatic driving automobile, and the automatic driving automobile can make a decision in advance to plan the self-driving path by making accurate track prediction on moving targets with different semantic types, so that traffic accidents are avoided.
Therefore, a plurality of methods for track prediction are proposed at home and abroad, and are mainly divided into two main categories: a data-based driven approach and a behavior-based driven approach. The data driving-based method is to mine the behavior characteristics of the target through massive historical data, and predict the movement trend of the target by combining the current position of the target. Such as probability statistical methods: bayesian networks, kalman filters, gaussian mixture models, etc.; the neural network method comprises the following steps: long Short Term Memory (LSTM), graph neural network (GNN, graph Neural Network), graph roll-up neural network (GCN, graph Convolutional Network), generation countermeasure network (GAN, generative Adversarial Network), and some other hybrid models, etc. The behavior-driven method is used for judging the intention of the target through the current state of the target and the environment of the target, so as to predict the future track of the target, such as a dynamics model. Although there are many track prediction methods, most methods only predict the track of a single moving target, such as pedestrian track prediction, taxi track prediction, private car track prediction, etc., based on the assumption that the targets are cooperative, and a real person may prefer to optimize the personal target rather than the joint strategy. The subsequent research simulates the interaction behavior of different traffic participants by modeling the distance between the different traffic participants on the basis of the previous research, and further improves the accuracy of model prediction. In real life, however, the most common scenario is interaction among a plurality of moving targets, the methods ignore interaction among moving targets with different semantic types, have no universality, some methods for predicting the trajectories of the moving targets begin to appear in recent years, deep neural networks based on graph structures are used for establishing interaction relations among the moving targets, and the evolution modes of node attributes and side attributes are directly learned by constructing a space-time diagram, but deeper semantic features of the moving targets with different semantics are ignored.
Disclosure of Invention
The invention aims at overcoming the defects of the prior art and provides a method and a system for predicting a dynamic diagram generation countermeasure network track of multi-source heterogeneous data.
The aim of the invention is realized by the following technical scheme: a method for predicting the track of a dynamic graph generation countermeasure network of multi-source heterogeneous data is characterized in that the dynamic graph generation countermeasure network consists of a comprehensive generator and a discriminator D; the integrated generator is composed of a track prediction generator G T And semantic type generator G C The track prediction method comprises the following steps:
(1) Acquiring tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, respectively preprocessing the tracks of the moving targets in each scene, acquiring space-time track data and the semantic types of the moving targets, and constructing a continuous time stamp topological graph of the moving targets with different semantic types; taking the space-time track data and the semantic type of the moving target as data sets, and dividing the data sets into training sets and test sets;
(2) Inputting the time stamp topological graph obtained in the step (1) into a graph annotation force network GAT, obtaining the neighborhood characteristics of each node through the GAT, distributing different weights for different nodes in the neighborhood, and capturing topological information hidden in each graph to obtain graph node characteristic vectors;
(3) Inputting the characteristic vector of the graph node obtained in the step (2) into a long-short-term memory network LSTM to capture the evolution mode of a weighted dynamic network so as to output a final hidden vector, and inputting the final hidden vector into a feedforward neural network to predict and generate trajectories of moving targets with different semantic types;
(4) Inputting the final hidden vector output in the step (3) to G C Capturing track distribution characteristics of different moving targets to generate semantic types of the moving targets;
(5) Splicing the tracks of the moving targets with different semantic types generated in the step (3) and the semantic types of the moving targets generated in the step (4) to obtain final predicted track information, inputting the final predicted track information and the real track information in the training set into D together, and distinguishing real data samples in the training set from predicted data samples through D;
(6) The maximum and minimum games are utilized to carry out joint optimization on the comprehensive generator and the discriminator D so as to achieve the stable situation of the games, namely Nash equilibrium;
(7) Inputting test set into trained G T Future track information is predicted and generated.
Further, the trajectory prediction generator G T The system consists of a schematic network GAT and a long-short-period memory network LSTM; the semantic type generator G C The method comprises the steps of forming a multi-layer sensor with a sigmoid activation function; the discriminator D consists of a multi-layer perceptron with a ReLU activation function.
Further, the step (1) specifically comprises the following steps:
collecting tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, converting the tracks of the moving targets in each scene into a unified data format through preprocessing, removing redundant data, filling missing values by adopting a linear interpolation method, and then respectively standardizing track coordinates by adopting a Z-Score method and adopting one-hot coding for different semantic types; constructing a set of successive time stamp topology graphs g= { G for all moving objects within T time with moving objects of different semantic types 1 ,G 2 ,L,G T T epsilon {1, L, T }, representing moving targets by nodes of the graph, and representing interactions of the moving targets by edges between the nodes; for each G t All have G t ＝(V t ,A t ) Wherein V is t Node set representing graph, A t Representing the edge set of the graph, using w t,i,j Representing interaction between node i and node j at time t, where w t,i,j ∈A t ；w t,i,j A value of 1 indicates that there is an interaction between nodes i, j, otherwise, there is no interaction between nodes i, j.
Further, the step (2) is specifically:
let t moment node set V t The number of the middle nodes is N, and then the node set V t Feature vector set h t Is represented as followsWherein (1)>Is the characteristic vector of the node i at the moment t, comprises track coordinates and one-hot codes thereof, has the dimension of F, and is formed by the way of +.>And->The characteristic vector of (2) is subjected to linear transformation W, the characteristic dimension of each characteristic vector is changed from F to F ', the characteristic vector and the characteristic vector are spliced to obtain a vector with the dimension of 2F', and then a feedforward neural network and a LeakyReLU activation function are used for obtaining the weight e of a node j to a node i at the moment t t,i,j Finally, normalizing the ownership weights by using a softmax function to obtain a final attention coefficient alpha t,i,j ＝sofxmax(e t,i,j ) The method comprises the steps of carrying out a first treatment on the surface of the At G T In, introducing random noise Z t In order to disturb the adjacency matrix A t Wherein Z is t Representing random generation of noise matrix obeying uniform distribution at t moment, let w t,i,j ∈A t When w is t,i,j When=1, a new attention coefficient α' t,i,j ＝z t,i,j ,z t,i,j ∈Z t When w is t,i,j When=0, α' t,i,j = - ≡; then use alpha' t,i,j The characteristics of all neighbor nodes j of the node i at the moment t are linearly weighted to obtain the final output characteristic vector +.>Wherein W is a weight matrix of the corresponding input linear transformation, and sigma (°) is a nonlinear activation function; the feature set of the output node obtained at time t is +.>Wherein h' t As the node feature vector set with the dimension of F ', feature aggregation H= { H ' in the T time is obtained after the node feature vector set passes through a GAT network ' 1 ,h' 2 ,L,h' T },t∈{1,L,T}。
Further, the step (3) is specifically:
LSTM consists of input gate i t Forgetting to open the door f t Output gate o t A memory cell c t The composition is used for learning the long-term dependence of the sequence data, taking the graph node characteristic vector output by the GAT as the input of the LSTM so as to capture the evolution mode of the weighted dynamic network; given the GAT output sequence h= { H' 1 ,h' 2 ,L,h' T T epsilon {1, L, T }, obtaining the final hidden vector sequence via LSTMInputting to a feedforward neural network and outputting track information T for predicting and generating moving targets of different semantic types pr 。
Further, the step (4) specifically comprises:
G C receiving final hidden vector of LSTM outputAnd a noise vector z as input, wherein z is uniformly distributed, and then passes through an intermediate plurality of hidden layers with fully connected feedforward neural network of ReLU activation function, and passes through the hidden feature vectorMulti-layer perceptron with sigmoid activation function, mapping it to 0-1 value, taking the type corresponding to the maximum probability as final semantic type C pr And outputting.
Further, the step (5) specifically comprises:
the track prediction generator G T And semantic type generator G C Output result T of (2) pr And C pr Splicing to obtain final predicted track information X pr The method comprises the steps of carrying out a first treatment on the surface of the Wherein T is pr Representing the true trajectory during time T, C pr Representing semantic types of corresponding moving targets; during training, D alternately predicts information X pr And ground truth value X tr As input, where ground truth value X tr True track information is in the training set.
Further, the step (6) specifically comprises:
the arbiter D attempts to correlate the real data in the training set with G T And G C The generated data are distinguished, G T And G C Attempting to fool D and generate samples that approach real data; to process continuous and discrete data simultaneously, a penalty function is calculated using the Wasserstein distance with gradient penalty;
G T and G C Are trained to minimize loss, while the discriminant is trained to maximize loss; the training efficiency is improved by using small batches, each iteration is performed by training the discriminators multiple times before training the comprehensive generator, and then performing countermeasure training on the comprehensive generator and the discriminators until a Nash equilibrium state is reached.
Further, the dynamic graph generation countermeasure network is optimized by calculating the mean absolute error MAE, the mean displacement error ADE, the final displacement error FDE, and the KL divergence.
A dynamic graph generation countermeasure network trajectory prediction system for multi-source heterogeneous data, comprising:
the data set acquisition module is used for acquiring tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, and preprocessing the tracks to acquire space-time track data and the semantic types of the moving targets; taking the space-time track data and the semantic type of the moving target as data sets, and dividing the data sets into training sets and test sets;
a model training module comprising a trajectory prediction generator G T And semantic type generator G C Generating an countermeasure network by a dynamic diagram composed of a comprehensive generator and a discriminator D; the training set and the test set are input into the dynamic diagram generation countermeasure network to train, the accuracy of the model is calculated for multiple times to evaluate the average accuracy degree of the model, and training is stopped when the discriminant and the comprehensive generator reach Nash equilibrium.
A model prediction module comprising a trajectory prediction generator G T And semantic type generator G C A composite generator is formed; by inputting spatiotemporal trace data in test sets to G T Future track information is predicted and generated.
A system for predicting the track of a countermeasure network generated by a dynamic graph of multi-source heterogeneous data comprises one or more processors, and the system is used for realizing the method for predicting the track of the countermeasure network generated by the dynamic graph of the multi-source heterogeneous data.
A computer readable storage medium having stored thereon a program which, when executed by a processor, is adapted to carry out a method of dynamic graph generation of antagonistic network trajectories for multi-source heterogeneous data as described above.
The beneficial effects of the invention are as follows:
1. the invention expands the application of GAN to graph sequence data with random weight, utilizes the learning ability of a neural network to autonomously capture potential interaction information of all graph nodes and semantic information of graph nodes with different semantic types, and improves the generation ability of a generator through countermeasure training.
2. According to the method, only the space-time track data of the moving target history is required to be input into the trained generator, and the generator predicts and generates the future track of the moving target and the semantic type thereof so as to better assist the path planning and decision of the intelligent agent in the downstream task in the complex scene.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a flow chart of the model training module of the present invention;
FIG. 3 is a diagram of a method for generating a countermeasure network trajectory prediction from a dynamic map of multi-source heterogeneous data according to an embodiment of the present invention;
fig. 4 is a hardware configuration diagram of the present invention.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of systems and methods that are consistent with aspects of the invention as detailed in the accompanying claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the invention. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
The present invention will be described in detail with reference to the accompanying drawings. The features of the examples and embodiments described below may be combined with each other without conflict.
As shown in FIG. 3, the method for generating the frame diagram of the countermeasure network track prediction method based on the dynamic diagram of the multi-source heterogeneous data comprises a track prediction generator G T And semantic type generator G C A composite generator and a discriminator D. Trajectory prediction generator G T Semantic type generator G for predicting trajectories of moving objects over a period of time in the future C For predicting semantic types of moving objects over a period of time in the future. The arbiter D attempts to correlate the real data in the training set with G T And G C The predictively generated data are differentiated, G T And G C The combined generator is constructed to try to fool D and predict the generation of high quality samples, both of which are continuously gambling, until the combined generator and arbiter eventually reaches nash equilibrium.
In one aspect of the invention, a method for predicting a trajectory of an countermeasure network generated based on a dynamic graph of heterogeneous multi-source data includes the steps of:
(1) The method is characterized in that tracks of moving targets with different semantic types at fixed time intervals under a plurality of scenes are acquired from different channels (such as GPS data, pictures or videos, and the like), and because the original track data are influenced by sampling frequency, sampling precision and sampling modes and cannot be directly applied to various mining algorithms, firstly, the multi-source heterogeneous data are respectively preprocessed according to different scenes of the multi-source heterogeneous data to acquire space-time track data and the semantic types of the moving targets, and the space-time track data and the semantic types of the moving targets are converted into a unified data format. For example: for video picture data, converting the video into pictures with fixed frame intervals, and obtaining track coordinates and semantic types of different moving targets by coding the pictures and combining with a semantic map; track coordinates and semantic types of different moving targets at fixed time intervals are extracted for the GPS data. Then carrying out one-hot coding on semantic types of the moving targets according to track information of the moving targets in different scenes, and finally constructing a continuous time stamp topological graph of the moving targets with different semantic types by splicing with track coordinates, wherein the moving targets form a node set of the graph, the track information of the moving targets form characteristic information of the nodes of the graph, and interactions among the moving targets form an edge set of the graph;
(2) Inputting the time stamp topological graph obtained in the step (1) into a graph annotation force network GAT, obtaining the neighborhood characteristics of each node through the GAT, distributing different weights for different nodes in the neighborhood, and capturing topological structure information hidden in each graph to obtain graph node characteristic vectors;
(3) Inputting the characteristic vector of the graph node obtained in the step (2) into a long-short-term memory network LSTM to capture the evolution mode of a weighted dynamic network so as to output a final hidden vector, and inputting the final hidden vector into a feedforward neural network to predict and generate trajectories of moving targets with different semantic types;
(4) Inputting the final hidden vector output in the step 3 to G C Capturing track distribution characteristics of different moving targets to generate semantic types of the moving targets;
(5) Splicing the tracks of the moving targets with different semantic types generated in the step (3) and the semantic types of the moving targets generated in the step (4) to obtain final generated track information, then inputting the final generated track information and the real track information in the training set into a D of a multi-layer perceptron with a sigmoid activation function, and distinguishing real data samples in the training set from predicted generated data samples through the D;
(6) And carrying out joint optimization on the comprehensive generator and the arbiter by utilizing the maximum and minimum games to achieve the stable situation of the games, namely Nash equilibrium.
(7) Inputting the test set into a trained comprehensive generator G T Future track information is predicted and generated.
The step (1) specifically comprises the following steps: a space-time trajectory is a sequence of time-stamped (x, y) coordinate points, each point having aA time stamp. Due to the reasons of too low hardware configuration of the acquisition equipment and the like, some errors inevitably occur, firstly, trace data are required to be cleaned, redundant data and abnormal data are removed, and a linear interpolation method is adopted to fill up missing values. After data are cleaned, the track coordinates are standardized by adopting a Z-Score method, and one-hot codes are adopted for different semantic types. Constructing a set of successive time stamp topology graphs g= { G for all moving objects within T time with moving objects of different semantic types 1 ,G 2 ,L,G T T e {1, l, t }, the nodes of the graph represent moving objects, and the edges between the nodes represent interactions of the moving objects. For each G t All have G t ＝(V t ,A t ) Wherein V is t Node set representing graph, A t Represents the edge set of the graph, A t The definition is as follows:
wherein w is t,i,j And the interaction between the node i and the node j at the moment t is represented, the value of 1 is represented as that the interaction exists between the nodes i and j, otherwise, the interaction does not exist between the nodes i and j.
Different semantic types are converted into one-hot (one-hot) codes as shown in table 1:
table 1: semantic type one-hot encoding
The step (2) specifically comprises the following steps:
let t moment node set V t The number of the middle nodes is N, and then the node set V t Feature vector set h t The expression is as follows:
wherein,,is the characteristic vector of the node i at the moment t, comprises track coordinates and one-hot codes thereof, has the dimension of F, and calculates the weight e of the node j at the moment t to the node i t,i,j The method comprises the following steps:
wherein a represents a shared attention mechanism, the correlation degree calculated by the target node and all neighbors is subjected to unified normalization processing, and the normalization processing is performed by using a softmax function:
wherein N is t,i Representing the set of neighbor nodes of the node i in t time, and the complete weight coefficient calculation formula is as follows:
where I represents the stitching operation, leakyReLU is a nonlinear activation function with a negative half-axis slope of 0.2,sharing parameters of the attention mechanism a for the feedforward neural network, · T Representing the transpose operation.
At G T In, introducing random noise Z t In order to disturb the adjacency matrix A t The method comprises the following steps:
A' t ＝A t ·Z t
wherein A 'is' t Is an adjacency matrix with noise vectors, Z t Representing random generation of noise matrix obeying uniform distribution at t moment and z t,i,j ∈Z t ，z t,i,j Represents the ith row and j columns at t timeThe weight generated by the machine is filtered according to the following formula to obtain a new attention coefficient alpha' t，i，j The method comprises the following steps:
wherein w is t,i,j ∈A t ，w t,i,j The weight of the edge of the node i and the node j at the moment t is represented, the linear combination of the corresponding characteristics is calculated, and after the nonlinear activation function is passed, the final output characteristic vector of each node i is obtained by the following formula:
where W is the weight matrix of the corresponding input linear transformation and σ (·) is the nonlinear activation function.
The output node characteristics obtained for time t are:
wherein h' t And obtaining feature aggregation H in the T time after passing through the GAT network for the node feature vector set with the dimension of F':
H＝{h' 1 ,h' 2 ,L,h' T },t∈{1,L,T}
the step (3) specifically comprises the following steps:
LSTM consists of input gate i t Forgetting to open the door f t Output gate o t And a memory unit c t The composition has strong capability to learn the long-term dependence of the sequence data, and takes the graph node feature vector output by GAT as the input of LSTM to capture the evolution mode of the weighted dynamic network.
Given the GAT output sequence h= { H' 1 ,h' 2 ,L,h' T },t∈{1,L,T}，
An input door:
forgetting to gate:
output door:
memory cell candidate:
memory cell state update:
c t ＝i t e u t +f t e c t-1
and (3) outputting:
wherein h' t Representing the input of the current time step t,representing the output of the last step LSTM. The last sequence of hidden vectors of LSTM +.>Inputting a feedforward neural network and outputting a predicted track T for generating moving targets with different semantic types pr 。
The step (4) specifically comprises the following steps:
semantic type generator G C Receiving final hidden vector of LSTM outputAnd a noise vector z as input, where z follows a uniform distribution and then passes through the intermediate plurality of hidden layers (with ReLU activation functionFull-connection feedforward neural network), then passing the hidden feature vector through a multi-layer perceptron with a sigmoid activation function to map the hidden feature vector into 0-1 value, and taking the type corresponding to the maximum probability as the final semantic type C pr And outputting.
The step (5) specifically comprises the following steps:
the track prediction generator G T And semantic type generator G C Output result T of (2) pr And C pr Splicing to obtain final predicted track information X pr ：
X pr ＝Contat(T pr ,C pr )
During the training process, X is alternately changed pr And ground truth value X tr As input. Wherein X is tr ＝Contat(T tr ,C tr )，T tr Representing the true trajectory during time T, C tr Representing the semantic type of the corresponding moving object.
The step (6) specifically comprises the following steps:
the arbiter D attempts to correlate the real data in the training set with G T And G C The generated data are distinguished, G T And G C Attempts to fool D and generate samples that approach real data.
Due to semantic type generator G C Discrete in the real dataset, trajectory prediction generator G T Is continuous in the real dataset. To process both continuous and discrete data, a gradient-penalized wasperstein distance was used to calculate the loss function, as follows:
wherein D (X) pr ) Representing the discrimination result of the sample generated by the synthesis generator, p g Representing the distribution of the generated samples, D (X tr ) Representing the discrimination result of a real sample, p r Representing the distribution of real samples, data is sampled uniformly along a straight line between pairs of objects sampled from real data and generated data, D (data) represents the discrimination result of data, p data For its distribution. G T And G C Trained to minimize loss, while the discriminant is trained to maximize loss. The training efficiency is improved by using small batches, and each iteration is performed by training the discriminators for a plurality of times before training the comprehensive generator, and then performing countermeasure training on the comprehensive generator and the discriminators until a Nash equilibrium state is reached.
Wherein the dynamic graph generation countermeasure network is optimized by calculating the average absolute error MAE, the average displacement error ADE, the final displacement error FDE, and the KL divergence.
The mean absolute error MAE calculation formula is:
wherein X is pr Representing track information predicted and generated by a comprehensive generator, X tr Representing real track information;
the average displacement error ADE is the overall track deviation average value of each moving object/all the moving objects, and the specific formula is as follows:
wherein,,representing the track coordinates of the predictive generation, +.>Representing the real track coordinates, T p Representing the predicted time range.
The final displacement error FDE is the track end point deviation of each moving object/all moving objects:
wherein,,representing final track coordinates of the predictive generation, +.>Representing the actual final trajectory target.
KL divergence is used to characterize the degree to which probability distribution q fits probability distribution p. In generating a antagonism network, p is the probability distribution of the real data, q is the probability distribution of the random noise generation data, and the aim of antagonism is to make q fit to p fully. If q is insufficient to fit p, information loss is generated, and the whole information loss is the KL divergence of p and g, and the specific formula is as follows:
where P (X), Q (X) are two probability distributions over the random variable X.
In another aspect of the present invention, a system for track prediction for a multi-source heterogeneous data dynamic graph generation countermeasure network includes:
the data set acquisition module is used for acquiring tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, and preprocessing the tracks to acquire space-time track data and the semantic types of the moving targets; taking the space-time track data as a data set, dividing the data set into n parts, wherein one part is taken as a test set, and the other n-1 parts are taken as training sets; n is greater than or equal to 2;
a model training module comprising a trajectory prediction generator G T And semantic type generator G C Generating an countermeasure network by a dynamic diagram composed of a comprehensive generator and a discriminator D; the training set and the test set are input into the dynamic diagram generation countermeasure network to train, the accuracy of the model is calculated for multiple times to evaluate the average accuracy degree of the model, and training is stopped when the discriminant and the comprehensive generator reach Nash equilibrium.
A model prediction module comprising a trajectory prediction generator G T And semantic type generator G C The method comprises the steps of carrying out a first treatment on the surface of the By integrating test-focused space-time railsTrace data is input to G T Future track information is predicted and generated.
Fig. 1 shows a flowchart of a method and a system for predicting a track of an countermeasure network based on a dynamic graph of multi-source heterogeneous data, which comprises three modules, namely a data set acquisition module, a model training module and a model prediction module. The data set acquisition module is used for preprocessing the historical track information of the moving targets with different semantic types serving as a data set and comprising GPS data and picture or video data with different lengths under different scenes to obtain initial data sets with track coordinates and semantic types of the different moving targets at fixed time intervals, and splitting the initial data sets into a training set and a testing set by a cross validation method, namely, all observation samples except one observation sample are used as part of the training set, and the rest is used as the testing set. This is repeated for all data sets. As shown in fig. 2, the model training module performs training, testing and evaluation in the countermeasure network trajectory prediction method by inputting the training set and the test set into the dynamic graph of the constructed multi-source heterogeneous data. The track prediction method for generating the countermeasure network based on the dynamic graph is composed of a comprehensive generator (track prediction generator G respectively) T And semantic type generator G C ) And a discriminator D. Trajectory prediction generator G T Receiving a noise vector z and a timestamp topological graph as inputs, and outputting to generate a predicted track; semantic type generator G C Received noise vector z and trajectory prediction generator G T The final hidden vector is taken as input, the generated semantic type is output, G is taken as T And G C The output splice of (2) is used as the final output generated track information; training a discriminator D with real track information, inputting the generated track information into the discriminator D, and determining a G pair according to the result of the discriminator D T 、G C And optimizing parameters of the discriminator; continuously iterating until the discriminator, G T And G C Nash equilibrium is achieved. The model prediction module predicts and generates future track information by inputting track information of moving targets with different semantic types into a track prediction method of a dynamic graph generation countermeasure network.
Integrating the modules as software to be deployed on a cloud computing platform; the module is integrated and deployed into a chip in a hardware form to be made into an SOC chip; the modules are integrated and deployed to a terminal, so that the application of the edge side is realized; the system is built by combining the software and the hardware.
Corresponding to the embodiment of the method for predicting the dynamic graph generation countermeasure network track of the multi-source heterogeneous data, the invention also provides an embodiment of a system for predicting the dynamic graph generation countermeasure network track of the multi-source heterogeneous data.
Referring to fig. 4, a system for generating a dynamic graph of heterogeneous data according to an embodiment of the present invention includes one or more processors configured to implement a method for generating a dynamic graph of heterogeneous data according to the above embodiment.
The embodiment of the system for predicting the network track by generating the dynamic graph of the multi-source heterogeneous data can be applied to any device with data processing capability, such as a computer or the like. The system embodiment may be implemented by software, or may be implemented by hardware or a combination of hardware and software. Taking software implementation as an example, the system in a logic sense is formed by reading corresponding computer program instructions in a nonvolatile memory into a memory through a processor of any device with data processing capability. In terms of hardware, as shown in fig. 4, a hardware structure diagram of an apparatus with data processing capability, where the system is located, is generated for a dynamic diagram of multi-source heterogeneous data according to the present invention, except for the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 4, where the apparatus with data processing capability in the embodiment is located, generally, according to the actual function of the apparatus with data processing capability, other hardware may be further included, which is not described herein.
The implementation process of the functions and roles of each unit in the above system is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.
For system embodiments, reference is made to the description of method embodiments for the relevant points, since they essentially correspond to the method embodiments. The system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present invention. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The embodiment of the invention also provides a computer readable storage medium, on which a program is stored, which when executed by a processor, implements a method for generating a countermeasure network trajectory prediction by using a dynamic graph of multi-source heterogeneous data in the above embodiment.
The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any of the data processing enabled devices described in any of the previous embodiments. The computer readable storage medium may be any device having data processing capability, for example, a plug-in hard disk, a Smart Media Card (SMC), an SD Card, a Flash memory Card (Flash Card), or the like, which are provided on the device. Further, the computer readable storage medium may include both internal storage units and external storage devices of any data processing device. The computer readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing apparatus, and may also be used for temporarily storing data that has been output or is to be output.
The above embodiments are merely for illustrating the design concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement the same, the scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications according to the principles and design ideas of the present invention are within the scope of the present invention.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. The specification and examples are to be regarded in an illustrative manner only.
It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof.
Claims (12)
1. A method for predicting the track of a dynamic graph generation countermeasure network of multi-source heterogeneous data is characterized in that the dynamic graph generation countermeasure network consists of a comprehensive generator and a discriminator D; the integrated generator is composed of a track prediction generatorAnd semantic type generator->The track prediction method comprises the following steps:
(1) Acquiring tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, respectively preprocessing the tracks of the moving targets in each scene, acquiring space-time track data and the semantic types of the moving targets, and constructing a continuous time stamp topological graph of the moving targets with different semantic types; taking the space-time track data and the semantic type of the moving target as data sets, and dividing the data sets into training sets and test sets;
(2) Inputting the time stamp topological graph obtained in the step (1)Meaning of the drawingIn the force network GAT, the neighborhood characteristics of each node are obtained through the GAT, different weights are distributed for different nodes in the neighborhood, and topology information hidden in each graph is captured, so that graph node characteristic vectors are obtained;
(3) Inputting the characteristic vector of the graph node obtained in the step (2) into a long-short-term memory network LSTM to capture the evolution mode of a weighted dynamic network so as to output a final hidden vector, and inputting the final hidden vector into a feedforward neural network to predict and generate trajectories of moving targets with different semantic types;
(4) Inputting the final hidden vector output in the step (3) toCapturing track distribution characteristics of different moving targets to generate semantic types of the moving targets;
(5) Splicing the tracks of the moving targets with different semantic types generated in the step (3) and the semantic types of the moving targets generated in the step (4) to obtain final predicted track information, inputting the final predicted track information and the real track information in the training set into D together, and distinguishing real data samples in the training set from predicted data samples through D;
(6) The maximum and minimum games are utilized to carry out joint optimization on the comprehensive generator and the discriminator D so as to achieve the stable situation of the games, namely Nash equilibrium;
(7) Inputting test sets into trainedFuture trajectory information is predicted to be generated.
2. The method for generating a countermeasure network trajectory prediction from a dynamic map of heterogeneous data of claim 1, wherein said trajectory prediction generatorThe system consists of a schematic network GAT and a long-short-period memory network LSTM; said semantic type generator->The method comprises the steps of forming a multi-layer sensor with a sigmoid activation function; the discriminator D consists of a multi-layer perceptron with a ReLU activation function.
3. The method for predicting the trajectory of a countermeasure network by generating a dynamic map of heterogeneous data according to claim 1, wherein the step (1) specifically comprises:
collecting tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, converting the tracks of the moving targets in each scene into a unified data format through preprocessing, removing redundant data, filling missing values by adopting a linear interpolation method, and then respectively standardizing track coordinates by adopting a Z-Score method and adopting one-hot coding for different semantic types; for a pair ofAll moving objects within a time construct a set of successive time stamped topology graphs of moving objects with different semantic typesRepresenting the moving object by nodes of the graph, and representing interaction of the moving object by edges between the nodes; for each +.>All have->Wherein->Node set representing graph, ++>Representing edge sets of a graph, usingRepresenting the interaction between node i and node j at time t, wherein +.>；/>A value of 1 indicates that there is an interaction between nodes i, j, otherwise, there is no interaction between nodes i, j.
4. The method for predicting the trajectory of a countermeasure network by generating a dynamic map of heterogeneous data according to claim 1, wherein the step (2) is specifically:
let t moment node setThe number of the middle nodes is N, and the node set is +.>Feature vector set +.>Is represented as followsWherein->Is the characteristic vector of the node i at the moment t, comprises track coordinates and one-hot codes thereof, and has the dimension of +.>By ∈two nodes>And->Is subjected to a linear transformation W, each of which is uniqueSyndrome dimension from->Become->The two are spliced to obtain the product with one dimension of +.>Is then passed through a feed-forward neural network and a +.>Activating the function to obtain the weight of the node j to the node i at the moment t +.>Finally, normalizing the ownership weights by using a softmax function to obtain a final attention coefficient +.>The method comprises the steps of carrying out a first treatment on the surface of the At->In (1) introducing random noise->With perturbation adjacency matrix->Wherein->Representing random generation of noise matrix obeying uniform distribution at t moment, letWhen->New attention factor->When->When (I)>The method comprises the steps of carrying out a first treatment on the surface of the Then use +.>The characteristics of all neighbor nodes j of the node i at the moment t are linearly weighted to obtain the final output characteristic vector +.>Wherein W is the weight matrix of the corresponding input linear transformation, < >>Is a nonlinear activation function; the feature set of the output node obtained at time t is +.>Wherein->For dimension +.>Is subjected to GAT network to obtain +.>Feature aggregation over time。
5. The method for predicting the trajectory of a countermeasure network by generating a dynamic map of heterogeneous data according to claim 1, wherein the step (3) is specifically:
LSTM by input gateForgetting to leave the door->Output door->And a memory cell->The composition is used for learning the long-term dependence of the sequence data, taking the graph node characteristic vector output by the GAT as the input of the LSTM so as to capture the evolution mode of the weighted dynamic network; given GAT output sequence->Obtaining the final hidden vector sequence by LSTM>Inputting to a feedforward neural network and outputting track information for predicting and generating moving targets with different semantic types>。
6. The method for predicting the trajectory of a countermeasure network for generating a dynamic map of heterogeneous data according to claim 1, wherein the step (4) specifically comprises:
final hidden vector for receiving LSTM output +.>And a noise vector z as inputWherein z is subjected to uniform distribution, then through a plurality of hidden layers in the middle, the hidden layers are provided with a fully-connected feedforward neural network of a ReLU activation function, and then the hidden feature vector is mapped into 0-1 value through a multi-layer perceptron with a sigmoid activation function, and the type corresponding to the maximum probability is taken as the final semantic type->And outputting.
7. The method for predicting the trajectory of a countermeasure network for generating a dynamic map of heterogeneous data according to claim 1, wherein the step (5) specifically comprises:
track prediction generatorAnd semantic type generator->Output result of +.>And->Splicing to obtain final predicted track information>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Is indicated at->Real track in time, +.>Representing semantic types of corresponding moving targets; during training, D alternately predicts information +.>And ground truth value->As input, ground truth value ++>True track information is in the training set.
8. The method for predicting the trajectory of a countermeasure network for generating a dynamic map of heterogeneous data according to claim 1, wherein the step (6) specifically comprises:
the arbiter D attempts to compare the real data in the training setAnd->The data generated are distinguished, and +.>And->Attempting to fool D and generate samples that approach real data; to process continuous and discrete data simultaneously, a penalty function is calculated using the Wasserstein distance with gradient penalty;
and->Are trained to minimize loss, while the discriminant is trained to maximize loss; training efficiency is improved by using small batches, each iteration training the arbiter multiple times before training the synthesis generatorTraining is then performed to counter-train the integrated generator and the arbiter until a Nash equilibrium state is reached.
9. The method for predicting the trajectory of a dynamic map generated countermeasure network for heterogeneous data according to claim 1, wherein the dynamic map generated countermeasure network is optimized by calculating average absolute error MAE, average displacement error ADE, final displacement error FDE, and KL divergence.
10. A system for dynamic graph generation of multisource heterogeneous data against network trajectory prediction, comprising:
the data set acquisition module is used for acquiring tracks of moving targets of different semantic types at fixed time intervals in a plurality of scenes, and preprocessing the tracks to acquire space-time track data and the semantic types of the moving targets; taking the space-time track data and the semantic type of the moving target as data sets, and dividing the data sets into training sets and test sets;
a model training module comprising a trajectory prediction generatorAnd semantic type generator->Generating an countermeasure network by a dynamic diagram composed of a comprehensive generator and a discriminator D; training is carried out by inputting a training set and a testing set into a dynamic graph generation countermeasure network, the accuracy of the model is calculated for multiple times to evaluate the average accuracy of the model, and training is stopped when a discriminant and a comprehensive generator reach Nash equilibrium;
a model prediction module comprising a trajectory prediction generatorAnd semantic type generator->Composite generatorThe method comprises the steps of carrying out a first treatment on the surface of the By inputting spatiotemporal trajectory data in the test set +.>Future track information is predicted and generated.
11. A system for generating a dynamic graph of heterogeneous data against a network trajectory prediction, comprising one or more processors configured to implement the method for generating a dynamic graph of heterogeneous data against a network trajectory prediction of any one of claims 1-9.
12. A computer readable storage medium having stored thereon a program which, when executed by a processor, is adapted to carry out a method of dynamic graph generation of multisource heterogeneous data according to any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310577728.XA CN116523002A (en) | 2023-05-22 | 2023-05-22 | Method and system for predicting dynamic graph generation countermeasure network track of multi-source heterogeneous data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310577728.XA CN116523002A (en) | 2023-05-22 | 2023-05-22 | Method and system for predicting dynamic graph generation countermeasure network track of multi-source heterogeneous data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116523002A true CN116523002A (en) | 2023-08-01 |
Family
ID=87395857
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310577728.XA Pending CN116523002A (en) | 2023-05-22 | 2023-05-22 | Method and system for predicting dynamic graph generation countermeasure network track of multi-source heterogeneous data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116523002A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117975696A (en) * | 2024-03-28 | 2024-05-03 | 南京邦固消防科技有限公司 | Linkage type fire alarm control system and method |
-
2023
- 2023-05-22 CN CN202310577728.XA patent/CN116523002A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117975696A (en) * | 2024-03-28 | 2024-05-03 | 南京邦固消防科技有限公司 | Linkage type fire alarm control system and method |
| CN117975696B (en) * | 2024-03-28 | 2024-07-05 | 南京邦固消防科技有限公司 | Linkage type fire alarm control system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Khan et al. | Batteries state of health estimation via efficient neural networks with multiple channel charging profiles | |
| CN110110707A (en) | Artificial intelligence CNN, LSTM neural network dynamic identifying system | |
| Saxena et al. | D-GAN: Deep generative adversarial nets for spatio-temporal prediction | |
| CN114220271A (en) | Traffic flow prediction method, equipment and storage medium based on dynamic space-time graph convolution cycle network | |
| CN106909938B (en) | Visual angle independence behavior identification method based on deep learning network | |
| Yuan et al. | STGAN: Spatio-temporal generative adversarial network for traffic data imputation | |
| CN114155270A (en) | Pedestrian trajectory prediction method, device, equipment and storage medium | |
| Dai et al. | Spatio-temporal deep learning framework for traffic speed forecasting in IoT | |
| CN114550053A (en) | Traffic accident responsibility determination method, device, computer equipment and storage medium | |
| Dai et al. | Hybrid deep model for human behavior understanding on industrial internet of video things | |
| CN115690153A (en) | Intelligent agent track prediction method and system | |
| Chen et al. | Dynamic facial expression recognition model based on BiLSTM-Attention | |
| He et al. | IRLSOT: Inverse reinforcement learning for scene‐oriented trajectory prediction | |
| CN116523002A (en) | Method and system for predicting dynamic graph generation countermeasure network track of multi-source heterogeneous data | |
| CN112241802A (en) | Interval prediction method for wind power | |
| Hu et al. | Building occupancy detection and localization using CCTV camera and deep learning | |
| CN116052254A (en) | Visual continuous emotion recognition method based on extended Kalman filtering neural network | |
| Qin et al. | A typhoon trajectory prediction model based on multimodal and multitask learning | |
| Shao et al. | Failure detection for motion prediction of autonomous driving: An uncertainty perspective | |
| CN116338571A (en) | RSSI fingerprint positioning method based on self-encoder and attention mechanism | |
| Gammelli et al. | Recurrent flow networks: A recurrent latent variable model for density estimation of urban mobility | |
| Zhang et al. | C 3-GAN: Complex-Condition-Controlled Urban Traffic Estimation through Generative Adversarial Networks | |
| Xu et al. | Residual autoencoder-LSTM for city region vehicle emission pollution prediction | |
| Saxena et al. | Multimodal spatio-temporal prediction with stochastic adversarial networks | |
| Zhang et al. | Obstacle‐transformer: A trajectory prediction network based on surrounding trajectories |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |