CN110516613A - A Pedestrian Trajectory Prediction Method from the First View - Google Patents

Abstract

The invention discloses a method for predicting pedestrian trajectories from a first perspective, which adopts a codec structure combined with a circular convolution network to predict pedestrian trajectories from a first perspective. The original image is encoded to obtain the feature vector of pedestrian trajectory information, and then the feature vector is decoded to predict the future trajectory information of pedestrians. In the public data set and the data set collected by itself, the present invention will accurately predict the trajectory information of multiple pedestrians in the next 10 frames, and the L2 distance error between the final predicted trajectory and the final actual trajectory is increased to 40, which is higher than the current There are ways to improve the 30 pixel accuracy. The present invention proposes a time-space convolutional loop network method for predicting pedestrian trajectories, uses one-dimensional convolution for encoding and decoding processing, and predicts through the space-time convolution network. In the current related methods, the implementation is relatively simple, and the data acquisition and processing are clear and concise. , Strong practicability.

Description

A Pedestrian Trajectory Prediction Method from the First View

technical field

The invention relates to a method for predicting pedestrian trajectories, in particular to a method for predicting pedestrian trajectories from a first perspective.

Background technique

Today, with the rapid development of autonomous driving and robot technology, it is a very important task to obtain the surrounding environment information of the vehicle through the on-board camera, predict the trajectory information of pedestrians in the video, control the driving behavior of the vehicle, and make a more reasonable path planning to avoid obstacles and pedestrians. important task.

Non-first perspective, such as the trajectory prediction of pedestrians under the surveillance camera does not need to consider the impact of the camera’s own motion on the pedestrian trajectory prediction. For example, the pedestrian detection frame in the surveillance camera is getting larger and larger in the video, indicating that pedestrians are walking towards the camera. However, the first-person perspective distinguishes fixed-shot videos such as surveillance videos, and the movement of robots or the photographer itself directly affects the acquisition and prediction of pedestrian information in the video. The first perspective is a movement perspective, and the photographer himself is also moving. This movement will affect the judgment of the future behavior of the pedestrian. In close proximity to pedestrians, pedestrian trajectory prediction is also inaccurate.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, the present invention proposes a pedestrian trajectory prediction method in the first perspective that can improve the trajectory prediction accuracy of pedestrians in the first perspective.

The idea of the present invention is: based on the codec structure model, by introducing pedestrian position information, self-motion history information and circular convolution network to predict the future trajectory of pedestrians, and by adding the future information of self-motion to effectively improve the future of pedestrians in the video Accuracy of trajectory prediction.

In order to achieve the above object, the technical solution of the present invention is as follows: a pedestrian trajectory prediction method under the first viewing angle, comprising the following steps:

A. Network encoder encoding to obtain trajectory features

A1. Pedestrians wear or hold motion cameras on their heads to obtain recorded videos from the first perspective in real time;

A2. Divide the video into several images at a frame rate of k frames per second, where k ranges from 5 to 20;

A3, the image divided in step A2 is processed by the processor, and the pedestrian position feature vector is obtained through the following steps:

A31. Use the marking tool to mark the pedestrians in the image, and mark the pedestrian detection frame;

A32. Correct the pedestrian detection frame marked in step A31 through the time window sampling algorithm. Since the coordinate origin in the image space is in the upper left corner of the image, the x value of the horizontal axis coordinate increases from left to right, and the y value of the vertical axis coordinate increases from top to bottom, so the position information of the upper left corner of the pedestrian detection frame (x _i ^min , y _i ^min ) ^T and the position information of the lower right corner ( _xi ^max , y _i ^max ) ^T are used as pedestrian trajectory data. Take the trajectory sequences of all pedestrians included in n consecutive frames as a set of training samples, the range of n is 10-20, and the training samples of each pedestrian are denoted as L _in :

Wherein, l _i =( _xi ^min , y _i ^min , _xi ^max , y _i ^max )∈R ⁴ , and the value range of i is t _current -T _his ~t _current . t _current is the current moment, T _his represents the historical frame range, and the value of T _his is 5-20.

A33. Construct a pedestrian position feature extraction convolutional network, and process the pedestrian position and the size of the detection frame to obtain the pedestrian position feature vector L _in ^F :

Lin ^F = ( _{lf 1} , .., lf _m ),

Among them, lf _i represents the i-th eigenvalue of the pedestrian position feature.

The pedestrian position feature extraction convolutional network structure adopts a 4-layer structure, and the input data Lin is first input to the first layer of _Conv1d one-dimensional convolution layer, and the output result of the first layer of Conv1d one-dimensional convolution layer is input to the second layer of Convld one-dimensional Convolution layer, the output result of the second layer Conv1d one-dimensional convolution layer is input to the third layer Convld one-dimensional convolution layer, the output result of the third layer Conv1d one-dimensional convolution layer is input to the fourth layer Conv1d one-dimensional convolution layer, the fourth layer The output result of the one-dimensional convolution layer of layer Conv1d obtains the feature vector L _in ^F ; the output result of each layer is processed by BN batch normalization and activated by the Relu activation function.

A4. Obtain the camera self-motion history feature vector;

A41. Obtain the camera self-motion information of the current frame image relative to the previous frame image through the Structure From Motion algorithm. The camera self-motion information includes the Euler angle r _t ∈ R ³ of the camera self-rotation information and the velocity information v _t ∈ R ³ . The Euler angles include yaw angle ψ, roll angle Φ, and pitch angle θ, and the speed information includes projections v _x , v _y , v _z of the instant speed of the camera on three-dimensional coordinate axes. The camera ego-motion history feature vector is denoted as E _H ;

Wherein, e _t =(r _t ^T , v _t ^T ) ^T ∈ R ⁶ , and the value range of t is t _current −T _his ~t _current .

A42. Construct a 4-layer camera self-motion history feature extraction convolutional network, extract the features of the camera self-motion history information EH, and obtain the camera self-motion history feature vector E _H ^F ;

E _H ^F = (ef ₁ , . . . , ef _n )

Among them, ef _i represents the i-th eigenvalue of the camera ego-motion history feature.

The convolutional network structure for extracting self-motion history features of the camera adopts the same structure as the pedestrian position feature extraction convolutional network.

A5. Connect the two vectors L _in ^F and E _H ^F end to end to get the feature vector LE ^F :

LE ^F = (lf ₁ , .., lf _m , ef ₁ , . . . , ef _n )

A6. Obtain the future feature vector of the camera's self-motion;

A61, adopt the same method as step A41, obtain the T _future frame motion information of the self-moving future of the motion camera through the Structure From Motion algorithm, and the range is 5～20, which indicates that the motion camera is going to where in the future, and is recorded as E _Fur ;

A62. Construct the camera self-motion future feature extraction convolutional network CNN, extract the features of the camera's future self-motion information E _Fur , and obtain the camera self-motion future feature vector E _Fur ^F :

E _Fur ^F = (eff ₁ ,..., eff _n )

Among them, eff _i represents the i-th eigenvalue of the camera ego-motion future feature.

The structure of the convolutional network for extracting the camera's future ego-motion features is the same as the convolutional network structure for extracting the camera's ego-motion history features in step A42.

B. The network decoder decodes and predicts the future trajectory of pedestrians

In the network decoder structure, the pedestrian position information features and self-motion information features output by the network encoder are decoded, and the future trajectory of pedestrians is obtained through the deconvolution network. In order to improve the prediction accuracy, the future motion information of the motion camera itself that can represent the future trend is added. The specific steps are as follows:

B1. Construct a standard recurrent neural network RNN with n units;

B2. The feature vectors LE ^F and E _Fur ^F are used as two inputs of the recurrent neural network RNN, and the output of the network is obtained as a prediction sequence L _out .

B3. Construct a deconvolution network for decoding trajectory sequences;

The deconvolution network structure of the decoding trajectory sequence adopts a 4-layer structure. First, the prediction sequence L _out is input into the first layer of Convld one-dimensional convolutional layer, and the output result of the first layer of Conv1d one-dimensional convolutional layer is input into the second layer of Conv1d one-dimensional convolutional layer. layer, the output of the second layer Conv1d one-dimensional convolutional layer is input to the third layer Conv1d one-dimensional convolutional layer, and the output results of the first three layers are first processed by BN batch normalization and activated by the Relu activation function. Finally, the output result of the third layer Convld one-dimensional convolutional layer is input into the fourth layer Conv1d one-dimensional convolutional layer.

B4. Input the predicted sequence L _out into the deconvolution network constructed in step B3 to obtain the pedestrian's future detection frame size information and trajectory information L _pre :

Among them, l _i =( _xi ^min , y _i ^min , _xi ^max , y _i ^max )∈R ⁴ , the value range of i is t _current +1～t _current +T _future , t _current is the current moment, T _future is the number of predicted future frames, ranging from 5 to 20.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention uses a codec structure combined with a circular convolutional network to predict pedestrian trajectory strategies in the first perspective. The eigenvectors of pedestrian trajectory information obtained by encoding the original image in step A are decoded in step B to predict the trajectory information of future pedestrians. In the public data set and the data set collected by itself, the present invention will accurately predict the trajectory information of multiple pedestrians in the next 10 frames, and the L2 distance error between the final predicted trajectory and the final actual trajectory is increased to 40, which is higher than the current There are ways to improve the 30 pixel accuracy.

2. The present invention proposes a space-time convolutional network method for predicting pedestrian trajectories, using one-dimensional convolution for encoding and decoding processing, and predicting through the space-time convolutional network. In the current related methods, the implementation is relatively simple, and the data acquisition and processing are clear , Simple and practical.

Description of drawings

The present invention has 6 accompanying drawings, wherein:

Figure 1 is the image after marking with the marking tool.

Figure 2 marks the historical and future positions and detection frame information of pedestrians at time t ₀ .

Figure 3 shows the historical and future positions and detection frame information of pedestrians at time t ₀ +10.

Fig. 4 is a flowchart of the present invention.

FIG. 5 is a structural diagram of a convolutional network used in the present invention.

Fig. 6 is a structure diagram of a deconvolution network used in the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. Calculate the first-view image according to the process shown in Figure 4. First, use the moving camera to capture the camera image in motion, and use n frames of pictures as the original image for the first-view pedestrian prediction. According to step A31 of the present invention, the original image is processed to obtain the marked image, as shown in FIG. 1 . Here, the marking result needs to be corrected according to the precision of the marking tool.

According to steps A and B of the present invention, trajectory prediction results are obtained. In order to visually display the prediction effect, the predicted trajectory, real trajectory and historical trajectory are all marked on the image. Assuming that Figure 2 is the image at time t ₀ , in Figure 2, mark the trajectory prediction result of the pedestrian 10 seconds after time t ₀ with a triangle mark, mark the pedestrian’s real trajectory 10 seconds after time t 0 with a four-pointed star mark, and mark the pedestrian’s real trajectory 10 seconds after time t ₀ with a diamond mark Mark the real historical track 10 seconds before time t ₀ , as shown in Figure 2. Figure 3 is the image at time t ₀ +10. By comparing Figure 2 and Figure 3, it can be seen that the predicted pedestrian trajectory at time t ₀ represented by the triangle mark in Figure 2 is consistent with the future real pedestrian trajectory represented by the diamond mark, and the deviation of the coordinate points of the two trajectories is very small. In Figure 3, the center of the box is the center point of the real position of the pedestrian at time t ₀ +10, which has been predicted in the predicted trajectory at time t ₀ in Figure 2, that is, the leftmost triangle point of the triangle mark trajectory in Figure 2 . Analyzing the prediction results, it can be seen that the method according to the present invention can accurately predict the future trajectory of pedestrians.

The present invention is not limited to this embodiment, and any equivalent ideas or changes within the technical scope disclosed in the present invention are listed in the protection scope of the present invention.

Claims (1)

1. A pedestrian trajectory prediction method under a first viewing angle, characterized in that: comprising the following steps: A. Network encoder encoding to obtain trajectory features A1. Pedestrians wear or hold motion cameras on their heads to obtain recorded videos from the first perspective in real time; A2. Divide the video into several images at a frame rate of k frames per second, where k ranges from 5 to 20; A3, the image divided in step A2 is processed by the processor, and the pedestrian position feature vector is obtained through the following steps: A31. Use the marking tool to mark the pedestrians in the image, and mark the pedestrian detection frame; A32. Correct the pedestrian detection frame marked in step A31 through the time window sampling algorithm; since the coordinate origin in the image space is in the upper left corner of the image, the x value of the horizontal axis increases from left to right, and the y value of the vertical axis increases from top to bottom Increments downward, so the position information ( _xi ^min , y _i ^min ) ^T of the upper left corner of the pedestrian detection frame and the position information ( _xi ^max , y _i ^max ) ^T of the lower right corner of the pedestrian detection frame are taken as the pedestrian trajectory data; The trajectory sequence of pedestrians is used as a set of training samples, n ranges from 10 to 20, and the training samples of each pedestrian are recorded as L _in : Among them, l _i =( _xi ^min , y _i ^min , _xi ^max , y _i ^max )∈R ⁴ , the value range of i is t _current -T _his ～t _current ; t _current is the current moment, and _This represents History frame range, _The value of This is 5 to 20; A33. Construct a pedestrian position feature extraction convolutional network, and process the pedestrian position and the size of the detection frame to obtain the pedestrian position feature vector L _in ^F : Lin ^F = ( _{lf 1} , . . . , lf _m ), Among them, lf _i represents the i-th eigenvalue of the pedestrian position feature; The pedestrian position feature extraction convolutional network structure adopts a 4-layer structure, and the input data Lin is first input to the first layer of _Convld one-dimensional convolution layer, and the output result of the first layer of Convld one-dimensional convolution layer is input to the second layer of Convld one-dimensional Convolution layer, the output result of the second layer Convld one-dimensional convolution layer is input to the third layer Convld one-dimensional convolution layer, the output result of the third layer Convld one-dimensional convolution layer is input to the fourth layer Convld one-dimensional convolution layer, the fourth layer The output result of the one-dimensional convolutional layer of the layer Convld obtains the feature vector L _in ^F ; the output result of each layer is processed by BN batch normalization and activated by the Relu activation function; A4. Obtain the camera self-motion history feature vector; A41. Obtain the camera self-motion information of the current frame image relative to the previous frame image through the Structure From Motion algorithm; the camera self-motion information includes the Euler angle r _t ∈ R ³ of the camera self-rotation information and the velocity information v _t ∈ R ³ ; the Euler angle includes yaw angle ψ, roll angle φ and pitch angle θ, and the speed information includes the projection v _x of the instant speed of the camera on the three-dimensional coordinate axis, v _y , v _z ; the self-motion history characteristics of the camera The vector is denoted as E _H ; Among them, e _t = (r _t ^T , v _t ^T ) ^T ∈ R ⁶ , the value range of t is t _current - T _his ~ t _current ; A42. Construct a 4-layer camera self-motion history feature extraction convolutional network, extract the features of the camera self-motion history information _EH , and obtain the camera self-motion history feature ^{vector EHF} _; E _H ^F = (ef ₁ , . . . , ef _n ) Among them, ef _i represents the i-th eigenvalue of the camera's ego-motion history feature; The camera self-motion history feature extraction convolutional network structure adopts the same structure as the pedestrian position feature extraction convolutional network; A5. Connect the two vectors L _in ^F and E _H ^F end to end to get the feature vector LE ^F : LE ^F = (lf ₁ , . . . , lf _m , ef ₁ , . . . , ef _n ) A6. Obtain the future feature vector of the camera's self-motion; A61, adopt the same method as step A41, obtain the T _future frame motion information of the self-motion future of the motion camera through the Structure From Motion algorithm, represent the future where the motion camera intends to go, and be denoted as E _Fur ; A62. Construct the camera self-motion future feature extraction convolutional network CNN, extract the features of the camera's future self-motion information E _Fur , and obtain the camera self-motion future feature vector E _Fur ^F : E _Fur ^F = (eff ₁ ,..., eff _n ) Among them, eff _i represents the i-th eigenvalue of the camera's self-motion future feature; The convolutional network structure of the camera's future self-motion feature extraction is the same as that of the camera's self-motion history feature extraction convolutional network structure in step A42; B. The network decoder decodes and predicts the future trajectory of pedestrians In the network decoder structure, the pedestrian position information features and self-motion information features output by the network encoder are decoded, and the future trajectory of pedestrians is obtained through the deconvolution network; in order to improve the prediction accuracy, the future of the motion camera itself that can represent the future trend is added Sports information, the specific steps are as follows: B1. Construct a standard recurrent neural network RNN with n units; B2, using the feature vectors LE ^F and E _Fur ^F as two inputs of the recurrent neural network RNN, and obtaining the output of the network as the prediction sequence L _out ; B3. Construct a deconvolution network for decoding trajectory sequences; The deconvolution network structure of the decoding trajectory sequence adopts a 4-layer structure. First, the prediction sequence L _out is input into the first layer of Convld one-dimensional convolution layer, and the output result of the first layer of Convld one-dimensional convolution layer is input into the second layer of Convld one-dimensional convolution Layer, the output of the second layer Convld one-dimensional convolutional layer is input to the third layer Convld one-dimensional convolutional layer, the output results of the first three layers are first processed by BN batch normalization and activated by the Relu activation function; finally, the second layer The output result of the three-layer Convld one-dimensional convolutional layer is input to the fourth layer Convld one-dimensional convolutional layer; B4. Input the predicted sequence L _out into the deconvolution network constructed in step B3 to obtain the pedestrian's future detection frame size information and trajectory information L _pre : Among them, l _i =( _xi ^min , y _i ^min , _xi ^max , y _i ^max )∈R ⁴ , the value range of i is t _current +1～t _current +T _future , t _current is the current moment, T _future is the number of predicted future frames, ranging from 5 to 20.