CN117252905A - Pedestrian track prediction method and system based on neural differential equation - Google Patents
Pedestrian track prediction method and system based on neural differential equation Download PDFInfo
- Publication number
- CN117252905A CN117252905A CN202311539965.3A CN202311539965A CN117252905A CN 117252905 A CN117252905 A CN 117252905A CN 202311539965 A CN202311539965 A CN 202311539965A CN 117252905 A CN117252905 A CN 117252905A
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- data
- track
- neural
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000001537 neural effect Effects 0.000 title claims abstract description 45
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000004088 simulation Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000012549 training Methods 0.000 claims abstract description 13
- 238000004519 manufacturing process Methods 0.000 claims abstract description 12
- 238000013528 artificial neural network Methods 0.000 claims description 78
- 230000011218 segmentation Effects 0.000 claims description 24
- 238000001514 detection method Methods 0.000 claims description 12
- 238000006073 displacement reaction Methods 0.000 claims description 6
- 230000007613 environmental effect Effects 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000012800 visualization Methods 0.000 claims description 3
- 239000002245 particle Substances 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000007476 Maximum Likelihood Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems
- G06F17/13—Differential equations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Data Mining & Analysis (AREA)
- Pure & Applied Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Medical Informatics (AREA)
- Operations Research (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Algebra (AREA)
- Human Computer Interaction (AREA)
- Molecular Biology (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Computer Hardware Design (AREA)
- Geometry (AREA)
- Traffic Control Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a pedestrian track prediction method and a pedestrian track prediction system based on a neural differential equation, wherein the pedestrian track prediction method comprises the following steps: s1, acquiring real scene data of a target area; s2, analyzing and processing the real scene data, and storing the processed real scene data into a database; s3, constructing a simulation data production model, producing diversified simulation track data according to priori and hypothesis data, and storing the simulation track data into a database; s4, constructing a track prediction model, and performing incremental learning and training on the track prediction model; s5, acquiring scene data to be predicted in real time, inputting the scene data into a trained track prediction model, and outputting a prediction result of a future motion track of the pedestrian; and S6, visually displaying the predicted result of the future motion trail of the pedestrian. The invention not only has cold start capability, can provide service in the beginning stage, saves time and labor, but also has dynamic update capability, can adapt to various changes, and has strong robustness of the prediction result.
Description
Technical Field
The invention relates to the technical field of pedestrian track prediction, in particular to a pedestrian track prediction method and system based on a neural differential equation.
Background
The pedestrian track prediction refers to a method for reasonably predicting the future moving track of the current pedestrian by modeling the environment information, surrounding moving targets and the current and historical states of the pedestrian.
Currently, methods related to pedestrian trajectory prediction are mainly classified into three types: firstly, a rule-driven method is needed, and the method is based on strong priori and assumption, such as dynamics, particle System (Particle System), social force model (Social Force Model) and the like; meanwhile, the method can use specific mathematical expression or specific physical model, so that the method has good interpretability, but is not fit to a real scene, namely a predicted result is not accurate enough. In the patent application CN109300144a (a pedestrian track prediction method integrating a social force model and a kalman filter), the parameters of the social force model are calibrated by using an adaptive variant particle swarm algorithm, and then the predicted track is corrected by using the kalman filter.
Secondly, based on a data driving method, the method can solve the problem of under fitting to a real scene based on a rule driving method by counting a machine learning optimization method and real scene data, but the method has limited fitting capacity under the condition of a large amount of data. In the patent application CN115424236a (a pedestrian crossing track prediction method combining pedestrian intention and social force model), pedestrian intention recognition is added on the basis of the traditional social force model, but parameters of the social force model are still set according to experience, so that the pedestrian track prediction effect in an actual scene is poor and the method cannot be applied to a continuously-changing complex scene.
And thirdly, a deep learning-based method is used, wherein the method is based on a deep neural network to directly fit data of a real scene, the neural network comprises a convolutional neural network, a cyclic neural network, a graph neural network and the like, but the deep neural network is a black box model and has unexplainability. In the patent application of CN114462667A (a method for predicting the trajectory of a pedestrian crossing based on an SFM-LSTM neural network model), firstly, predicting the trajectory of the pedestrian crossing by using the LSTM model, and then, calibrating the parameters of a social force model by using a maximum likelihood function; however, when the LSTM model is used for predicting the pedestrian track, influences caused by surrounding environments, such as obstacles, are not considered, and the final prediction result is not accurate enough; meanwhile, the prediction speed of the LSTM model is slower than that of a convolution network; furthermore, the social force model parameters are calibrated using maximum likelihood functions, meaning that the social force model parameters of all pedestrians are shared, but the reality should be that every person should have self-existing parameters in the social force model.
In view of this, how to dynamically update the prediction model in real time can be better suitable for the pedestrian track prediction of various complex actual scenes is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the invention provides a pedestrian track prediction method and system based on a neural differential equation, which has cold starting capability, can dynamically update a prediction model in real time, and can be better suitable for pedestrian track prediction of various complex actual scenes.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a pedestrian track prediction method based on a neural differential equation comprises the following steps:
s1, acquiring real scene data of a target area;
s2, analyzing and processing the real scene data, and storing the processed real scene data into a database;
s3, constructing a simulation data production model, producing diversified simulation track data according to priori and hypothesis data, and storing the simulation track data into a database;
s4, constructing a track prediction model, and performing incremental learning and training on the track prediction model;
s5, acquiring scene data to be predicted in real time, inputting the scene data into a trained track prediction model, and outputting a prediction result of a future motion track of the pedestrian;
and S6, visually displaying the predicted result of the future motion trail of the pedestrian.
Further, the real scene data in step S1 includes pedestrian data, field data, and pedestrian surrounding moving object data.
Further, the analyzing and processing the real scene data in step S2 includes:
performing target detection and tracking analysis on the pedestrian data and the pedestrian surrounding moving targets to obtain a pedestrian historical motion track and a pedestrian surrounding moving target track;
and carrying out semantic segmentation on the field data to generate a field segmentation map.
Further, the track prediction model constructed in the step S4 includes two neural networks, namely a target point prediction neural network and a neural differential neural network;
the target point prediction neural network is used for predicting a target point to be reached by a pedestrian in the future;
the neural differential neural network is used for predicting the complete motion trail of the pedestrian in the future according to the target point predicted by the target point prediction neural network.
Further, the neural differential neural network combines a differential equation with the neural network, and the state of the pedestrian is represented by the following differential equation:
wherein,representing the overall modeling of pedestrian status, surrounding moving targets, environmental information;
representing parameters that the neural network needs to learn;
representing the status of the pedestrian, including displacement and velocity;
representing a target point that a pedestrian will reach in the future;
representing the influence of the surrounding moving targets of the pedestrians on the current pedestrians;
representing the impact of the surrounding environment on the current pedestrian.
A pedestrian trajectory prediction system based on a neural differential equation, comprising:
the data acquisition module is used for acquiring real scene data of the target area;
the data processing module is used for analyzing and processing the real scene data and storing the processed real scene data into a database;
the simulation data production module is used for producing diversified simulation track data according to priori and hypothesis data and storing the simulation track data into a database;
the incremental learning module is used for performing incremental learning and training on the track prediction model;
the track prediction module is used for predicting the future motion track of the pedestrian;
and the visualization module is used for carrying out visual display on the predicted result of the future motion trail of the pedestrian.
Further, the real scene data includes pedestrian data, field data, and pedestrian surrounding moving target data.
Further, the data processing module comprises a target detection and tracking module and a semantic segmentation module;
the target detection and tracking module is used for carrying out target detection and tracking analysis on the pedestrian data and the pedestrian surrounding moving targets to obtain a pedestrian historical motion track and a pedestrian surrounding moving target track;
the semantic segmentation module is used for carrying out semantic segmentation on the field data to generate a field segmentation map.
Further, the track prediction module comprises a target point prediction neural network and a neural differential neural network;
the target point prediction neural network is used for predicting a target point to be reached by a pedestrian in the future;
the neural differential neural network is used for predicting the complete motion trail of the pedestrian in the future according to the target point predicted by the target point prediction neural network.
Further, the neural differential neural network combines a differential equation with the neural network, and the state of the pedestrian can be represented by the following differential equation:
wherein,representing the overall modeling of pedestrian status, surrounding moving targets, environmental information;
representing parameters that the neural network needs to learn;
representing the status of the pedestrian, including displacement and velocity;
representing a target point that a pedestrian will reach in the future;
representing the influence of the surrounding moving targets of the pedestrians on the current pedestrians;
representing the impact of the surrounding environment on the current pedestrian.
Compared with the prior art, the invention has the following advantages:
1. the invention has cold start capability, when no enough real scene data is accumulated in the beginning stage, the simulation data production model is constructed, diversified and generalized simulation track data is produced according to priori and hypothesis data, and the produced simulation data is used for training the track prediction model, so that corresponding service is conveniently provided in the beginning stage; meanwhile, the data produced by the capability does not need to be manually marked, so that a large amount of manpower, material resources and financial resources can be saved;
2. the target point prediction network can effectively improve the accuracy of target point prediction by modeling a real scene, a pedestrian history track and surrounding moving targets;
3. the neural differential network solves the corresponding differential equation by using the neural network, and the predicted motion trail of the pedestrians can fully reflect the respective motion characteristics of the pedestrians, namely each pedestrian has a model, so that the neural differential network has the interpretable capacity of the traditional model and the strong fitting capacity of the neural network;
4. the invention has the capability of dynamically updating the model, and when the real scene information or crowd motion distribution is changed, the track prediction model can be updated in time through the real-time semantic segmentation module and the incremental learning module, so that the final prediction result is adapted to the changes.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
Fig. 1 shows specific implementation steps of the pedestrian track prediction method of the present invention.
Fig. 2 is a structural diagram of a pedestrian trajectory prediction system of the present invention.
Fig. 3 is a diagram showing the complete movement track result of the pedestrian in the region to be tested.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention discloses a pedestrian track prediction method based on a neural differential equation, which is shown by referring to FIG. 1 and specifically comprises the following steps of:
s1, acquiring real scene data of a target area;
s2, analyzing and processing the real scene data, and storing the processed real scene data into a database;
s3, constructing a simulation data production model, producing diversified simulation track data according to priori and hypothesis data, and storing the simulation track data into a database;
s4, constructing a track prediction model, and performing incremental learning and training on the track prediction model;
s5, acquiring scene data to be predicted in real time, inputting the scene data into a trained track prediction model, and outputting a prediction result of a future motion track of the pedestrian;
and S6, visually displaying the predicted result of the future motion trail of the pedestrian.
In the present embodiment, the real scene data in step S1 includes pedestrian data, field data, and pedestrian surrounding moving target data. The pedestrian surrounding target data is not limited to a person, and may be a vehicle, an animal, or the like.
In this embodiment, the analyzing and processing the real scene data in step S2 includes:
performing target detection and tracking analysis on pedestrian data and pedestrian surrounding moving targets to obtain a pedestrian historical motion track and a pedestrian surrounding moving target track;
and carrying out semantic segmentation on the field data to generate a field segmentation map.
In this embodiment, the simulation data production model constructed in step S3 includes, but is not limited to, a social force model, a particle model, a dynamics model, and the like. Because the invention can not acquire enough real scene data for track prediction model training in the beginning stage, corresponding simulation data can be acquired through a simulation data production model in the cold start stage; in the process, according to the random initialization of the related priori knowledge and some fixed strategies, the produced simulation data are sufficiently diversified; the batch of data is utilized to train the track prediction model, so that generalization of the prediction result can be improved to a certain extent.
In this embodiment, the track prediction model constructed in step S4 includes two neural networks, i.e., a target point prediction neural network and a neural differential neural network;
the target point prediction neural network is used for predicting a target point to be reached by a pedestrian in the future;
specifically, the target point prediction neural network predicts the future predicted track of the pedestrian by modeling the historical track of the pedestrian, the site segmentation map and the track of the moving target around the pedestrian, then performing incremental learning and training, predicting the final time step by using the trained target point prediction neural network, and finally obtaining the complete moving track of the pedestrian by taking the predicted result of the final time step as the destination of the pedestrian movement.
And the neural differential neural network is used for predicting the complete motion trail of the pedestrian in the future according to the target point predicted by the target point prediction neural network.
In this embodiment, the neural differential neural network combines a differential equation with the neural network, and the state of the pedestrian is represented by the following differential equation:
wherein,representing the overall modeling of pedestrian status, surrounding moving targets, environmental information;
representing parameters that the neural network needs to learn;
representing the status of the pedestrian, including displacement and velocity;
representing a target point that a pedestrian will reach in the future;
representing the influence of the surrounding moving targets of the pedestrians on the current pedestrians;
representing the impact of the surrounding environment on the current pedestrian.
Specifically, the neural differential neural network solves the differential equation by using the neural network, so that the neural differential neural network has the explanatory capacity of a traditional mode and the strong fitting capacity of the neural network. The motion characteristics of each pedestrian are different, and the predicted track of the network reflects the motion characteristics of each pedestrian, which is equivalent to that each pedestrian has a model. Notably, the neural network herein is not limited to convolutional neural networks, recurrent neural networks, or graph roll-up neural networks; the differential equation may be any method of modeling pedestrians, moving objects around pedestrians, and on-site environments, such as a social force model, a particle model, a kinetic model, etc., without specific limitation.
It is worth noting that the neural differential network can be absent under the conditions of simple scene, no need of considering environment information and moving targets around pedestrians, namely, the prediction target point network can be directly used for predicting the complete motion trail of the pedestrians.
In this embodiment, the method may be developed into an online service platform, for example, deployed in a single computer, or may be deployed in a system formed by a computer and a server. Real scene data of a target area are acquired in real time by using a camera, target detection and tracking analysis are carried out on the acquired real scene data, corresponding track data are obtained, the track data obtained in real time are input into an online service platform, the future motion track of the pedestrian predicted by the track prediction model can be output, and finally the historical motion track of the pedestrian and the future motion track of the pedestrian are displayed on a visual screen in real time.
Another embodiment of the present invention discloses a pedestrian trajectory prediction system based on a neural differential equation, referring to fig. 2, comprising:
the data acquisition module 1 is used for acquiring real scene data of a target area;
the data processing module 2 is used for analyzing and processing the real scene data and storing the processed real scene data into a database;
the simulation data production module 3 is used for producing diversified simulation track data according to priori and hypothesis data and storing the simulation track data into a database;
the incremental learning module 4 is used for performing incremental learning and training on the track prediction model;
the track prediction module 5 is used for predicting the future motion track of the pedestrian;
and the visualization module 6 is used for visually displaying the predicted result of the future motion trail of the pedestrian.
In this embodiment, the real scene data collected by the data collection module 1 includes pedestrian data, field data, and moving object data around pedestrians. The pedestrian surrounding target data is not limited to a person, and may be a vehicle, an animal, or the like.
In this embodiment, the data processing module 2 includes a target detecting and tracking module 21 and a semantic segmentation module 22;
the target detection and tracking module 21 is used for performing target detection and tracking analysis on pedestrian data and pedestrian surrounding moving targets to obtain a pedestrian historical motion track and a pedestrian surrounding moving target track;
the semantic segmentation module 22 is configured to perform semantic segmentation on the field data, and generate a field segmentation map.
In the present embodiment, the simulation data production module 3 includes, but is not limited to, a social force model, a particle model, a kinetic model, and the like. Since the present invention cannot acquire enough real scene data for the track prediction model training at the beginning stage, corresponding simulation data is acquired through the simulation data production model 3 at the cold start stage; in the process, according to the random initialization of the related priori knowledge and some fixed strategies, the produced simulation data are sufficiently diversified; the batch of data is utilized to train the track prediction model, so that generalization of the prediction result can be improved to a certain extent.
In this embodiment, the incremental learning module 4 trains the track prediction model periodically by using the latest real scene data, that is, the incremental learning module 4 has dynamic updating property, so that the change of the field can be perceived in time. For example, when the actual scene or the pedestrian motion distribution changes, like a moving functional area, the latest field segmentation map can be obtained after the processing of the semantic segmentation module 22, and then the latest field segmentation map is used for training the track prediction model, so that the track prediction model can fully adapt to the changes.
In this embodiment, the track prediction module 5 includes two neural networks, i.e., a target point prediction neural network 51 and a neural differential neural network 52;
the target point prediction neural network 51 is used for predicting a target point that a pedestrian will reach in the future;
specifically, the target point prediction neural network predicts the future predicted track of the pedestrian by modeling the historical track of the pedestrian, the site segmentation map and the track of the moving target around the pedestrian, then performing incremental learning and training, predicting the final time step by using the trained target point prediction neural network, and finally obtaining the complete moving track of the pedestrian by taking the predicted result of the final time step as the destination of the pedestrian movement.
The neural differential neural network 52 is configured to predict a future complete motion trail of the pedestrian based on the target point predicted by the target point prediction neural network, and visually display the complete motion trail of the pedestrian in the target area, as shown in fig. 3, to represent the complete motion trail of the pedestrian in the area to be detected, where the dot represents the historical motion trail of the pedestrian, and the star represents the predicted future motion trail of the pedestrian.
In the present embodiment, the neural differential neural network 52 combines a differential equation with the neural network, and the state of the pedestrian can be expressed by the following differential equation:
wherein,representing the overall modeling of pedestrian status, surrounding moving targets, environmental information;
representing parameters that the neural network needs to learn;
representing the status of the pedestrian, including displacement and velocity;
representing a target point that a pedestrian will reach in the future;
representing the influence of the surrounding moving targets of the pedestrians on the current pedestrians;
representing the impact of the surrounding environment on the current pedestrian.
Specifically, the neural differential neural network solves the differential equation by using the neural network, so that the neural differential neural network has the explanatory capacity of a traditional mode and the strong fitting capacity of the neural network. The motion characteristics of each pedestrian are different, and the predicted track of the network reflects the motion characteristics of each pedestrian, which is equivalent to that each pedestrian has a model. Notably, the neural network herein is not limited to convolutional neural networks, recurrent neural networks, or graph roll-up neural networks; the differential equation may be any method of modeling pedestrians, moving objects around pedestrians, and on-site environments, such as a social force model, a particle model, a kinetic model, etc., without specific limitation.
It is worth noting that the neural differential network can be absent under the conditions of simple scene, no need of considering environment information and moving targets around pedestrians, namely, the prediction target point network can be directly used for predicting the complete motion trail of the pedestrians.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. The pedestrian track prediction method based on the neural differential equation is characterized by comprising the following steps of:
s1, acquiring real scene data of a target area;
s2, analyzing and processing the real scene data, and storing the processed real scene data into a database;
s3, constructing a simulation data production model, producing diversified simulation track data according to priori and hypothesis data, and storing the simulation track data into a database;
s4, constructing a track prediction model, and performing incremental learning and training on the track prediction model;
s5, acquiring scene data to be predicted in real time, inputting the scene data into a trained track prediction model, and outputting a prediction result of a future motion track of the pedestrian;
and S6, visually displaying the predicted result of the future motion trail of the pedestrian.
2. The pedestrian trajectory prediction method based on the neural differential equation according to claim 1, wherein the real scene data in step S1 includes pedestrian data, field data, and pedestrian surrounding moving target data.
3. The pedestrian trajectory prediction method based on the neural differential equation according to claim 2, wherein analyzing and processing the real scene data in step S2 includes:
performing target detection and tracking analysis on the pedestrian data and the pedestrian surrounding moving targets to obtain a pedestrian historical motion track and a pedestrian surrounding moving target track;
and carrying out semantic segmentation on the field data to generate a field segmentation map.
4. The pedestrian trajectory prediction method based on the neural differential equation according to claim 1, wherein the trajectory prediction model constructed in step S4 includes two neural networks, namely a target point prediction neural network and a neural differential neural network;
the target point prediction neural network is used for predicting a target point to be reached by a pedestrian in the future;
the neural differential neural network is used for predicting the complete motion trail of the pedestrian in the future according to the target point predicted by the target point prediction neural network.
5. The pedestrian trajectory prediction method based on the neural differential equation according to claim 4, wherein the neural differential neural network combines the differential equation with a neural network, and the state of the pedestrian is represented by the following differential equation:
；
wherein,representing the overall modeling of pedestrian status, surrounding moving targets, environmental information;
indicating that the neural network needs to learnParameters;
representing the status of the pedestrian, including displacement and velocity;
representing a target point that a pedestrian will reach in the future;
representing the influence of the surrounding moving targets of the pedestrians on the current pedestrians;
representing the impact of the surrounding environment on the current pedestrian.
6. A pedestrian trajectory prediction system based on a neural differential equation, comprising:
the data acquisition module is used for acquiring real scene data of the target area;
the data processing module is used for analyzing and processing the real scene data and storing the processed real scene data into a database;
the simulation data production module is used for producing diversified simulation track data according to priori and hypothesis data and storing the simulation track data into a database;
the incremental learning module is used for performing incremental learning and training on the track prediction model;
the track prediction module is used for predicting the future motion track of the pedestrian;
and the visualization module is used for carrying out visual display on the predicted result of the future motion trail of the pedestrian.
7. The pedestrian trajectory prediction system based on a neural differential equation according to claim 6, wherein the real scene data includes pedestrian data, field data, and pedestrian surrounding moving target data.
8. The pedestrian trajectory prediction system based on a neural differential equation according to claim 6, wherein the data processing module comprises a target detection and tracking module and a semantic segmentation module;
the target detection and tracking module is used for carrying out target detection and tracking analysis on pedestrian data and pedestrian surrounding moving targets to obtain a pedestrian historical motion track and a pedestrian surrounding moving target track;
the semantic segmentation module is used for carrying out semantic segmentation on the field data to generate a field segmentation map.
9. The pedestrian trajectory prediction system based on a neural differential equation according to claim 6, wherein the trajectory prediction module comprises two neural networks, namely a target point prediction neural network and a neural differential neural network;
the target point prediction neural network is used for predicting a target point to be reached by a pedestrian in the future;
the neural differential neural network is used for predicting the complete motion trail of the pedestrian in the future according to the target point predicted by the target point prediction neural network.
10. The pedestrian trajectory prediction system based on a neural differential equation according to claim 9, wherein the neural differential neural network combines the differential equation with a neural network, and the state of the pedestrian is represented by the differential equation:
；
wherein,representing the entirety of pedestrian status, surrounding moving objects, and environmental informationModeling a body;
representing parameters that the neural network needs to learn;
representing the status of the pedestrian, including displacement and velocity;
representing a target point that a pedestrian will reach in the future;
representing the influence of the surrounding moving targets of the pedestrians on the current pedestrians;
representing the impact of the surrounding environment on the current pedestrian.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311539965.3A CN117252905B (en) | 2023-11-20 | 2023-11-20 | Pedestrian track prediction method and system based on neural differential equation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311539965.3A CN117252905B (en) | 2023-11-20 | 2023-11-20 | Pedestrian track prediction method and system based on neural differential equation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117252905A true CN117252905A (en) | 2023-12-19 |
| CN117252905B CN117252905B (en) | 2024-03-19 |
Family
ID=89126834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311539965.3A Active CN117252905B (en) | 2023-11-20 | 2023-11-20 | Pedestrian track prediction method and system based on neural differential equation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117252905B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111222438A (en) * | 2019-12-31 | 2020-06-02 | 的卢技术有限公司 | Pedestrian trajectory prediction method and system based on deep learning |
| CN116880184A (en) * | 2023-07-12 | 2023-10-13 | 浙江大学海南研究院 | Unmanned ship track tracking prediction control method, unmanned ship track tracking prediction control system and storage medium |
| WO2023207437A1 (en) * | 2022-04-28 | 2023-11-02 | 长安大学 | Scene flow digital twin method and system based on dynamic trajectory flow |
| CN117077727A (en) * | 2023-09-18 | 2023-11-17 | 华东师范大学 | Track prediction method based on space-time attention mechanism and neural ordinary differential equation |
-
2023
- 2023-11-20 CN CN202311539965.3A patent/CN117252905B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111222438A (en) * | 2019-12-31 | 2020-06-02 | 的卢技术有限公司 | Pedestrian trajectory prediction method and system based on deep learning |
| WO2023207437A1 (en) * | 2022-04-28 | 2023-11-02 | 长安大学 | Scene flow digital twin method and system based on dynamic trajectory flow |
| CN116880184A (en) * | 2023-07-12 | 2023-10-13 | 浙江大学海南研究院 | Unmanned ship track tracking prediction control method, unmanned ship track tracking prediction control system and storage medium |
| CN117077727A (en) * | 2023-09-18 | 2023-11-17 | 华东师范大学 | Track prediction method based on space-time attention mechanism and neural ordinary differential equation |
Non-Patent Citations (1)
| Title |
|---|
| JIANGBEI YUE ET AL: "Human Trajectory Prediction via Neural Social Physics", 《ARXIV:2207.10435V2》, pages 1 - 26 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117252905B (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108803617B (en) | Trajectory prediction method and apparatus | |
| WO2023207437A1 (en) | Scene flow digital twin method and system based on dynamic trajectory flow | |
| CN113221677B (en) | Track abnormality detection method and device, road side equipment and cloud control platform | |
| CN113705636B (en) | Method and device for predicting track of automatic driving vehicle and electronic equipment | |
| CN110378191B (en) | Pedestrian and vehicle classification method based on millimeter wave sensor | |
| WO2021071772A1 (en) | Multi-agent simulations | |
| CN107689052A (en) | Visual target tracking method based on multi-model fusion and structuring depth characteristic | |
| CN113298053B (en) | Multi-target unmanned aerial vehicle tracking identification method and device, electronic equipment and storage medium | |
| Dimitriou et al. | A deep learning framework for simulation and defect prediction applied in microelectronics | |
| Sharma et al. | Pedestrian intention prediction for autonomous vehicles: A comprehensive survey | |
| CN112487954B (en) | Pedestrian crossing behavior prediction method for plane intersection | |
| CN112016499A (en) | Traffic scene risk assessment method and system based on multi-branch convolutional neural network | |
| CN115238958A (en) | Dangerous event chain extraction method and system based on complex traffic scene | |
| CN115690153A (en) | Intelligent agent track prediction method and system | |
| CN111695627A (en) | Road condition detection method and device, electronic equipment and readable storage medium | |
| CN109215311B (en) | Dynamic parameter weight determination method of public event early warning model based on reinforcement learning | |
| Lin et al. | Optimal CNN-based semantic segmentation model of cutting slope images | |
| CN117252905B (en) | Pedestrian track prediction method and system based on neural differential equation | |
| CN112633074A (en) | Pedestrian information detection method and device, storage medium and electronic equipment | |
| Han et al. | A novel end-to-end model for steering behavior prediction of autonomous ego-vehicles using spatial and temporal attention mechanism | |
| CN116166642A (en) | Spatio-temporal data filling method, system, equipment and medium based on guide information | |
| CN114187759B (en) | Road side unit driving assistance method and device based on data driving model | |
| US20220237897A1 (en) | Computer-implemented method for analyzing relevance of visual parameters for training a computer vision model | |
| CN115564800A (en) | Action track prediction method and device | |
| CN112733728A (en) | Visibility edge calculation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |