WO2022110611A1 - 一种面向平面交叉口的行人过街行为预测方法 - Google Patents

一种面向平面交叉口的行人过街行为预测方法 Download PDF

Info

Publication number
WO2022110611A1
WO2022110611A1 PCT/CN2021/086572 CN2021086572W WO2022110611A1 WO 2022110611 A1 WO2022110611 A1 WO 2022110611A1 CN 2021086572 W CN2021086572 W CN 2021086572W WO 2022110611 A1 WO2022110611 A1 WO 2022110611A1
Authority
WO
WIPO (PCT)
Prior art keywords
behavior
layer
pedestrian
convolution
feature map
Prior art date
Application number
PCT/CN2021/086572
Other languages
English (en)
French (fr)
Inventor
李旭
胡锦超
徐启敏
胡玮明
Original Assignee
东南大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 东南大学 filed Critical 东南大学
Priority to US18/249,715 priority Critical patent/US20230410659A1/en
Publication of WO2022110611A1 publication Critical patent/WO2022110611A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G06V40/25—Recognition of walking or running movements, e.g. gait recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/052—Detecting movement of traffic to be counted or controlled with provision for determining speed or overspeed
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the invention relates to a prediction method, in particular to a pedestrian crossing behavior prediction method facing a plane intersection, and belongs to the technical field of behavior modeling and prediction of traffic participants.
  • Pedestrians are the main participants in road traffic, and their behavior is an important factor affecting traffic safety, especially at level intersections with a large number of crossing behaviors, such as school entrances and exits, unsignaled access points, etc.
  • the total number of traffic accidents is nearly 70%. Therefore, the identification and prediction of pedestrian crossing behaviors at level intersections, especially the real-time prediction of pedestrian crossing behaviors when dangers such as collisions and rubbing are induced, and danger warnings for pedestrians and vehicles crossing the street is an important way to build an intelligent road.
  • the basic requirements of the side system are also conducive to reducing the incidence of traffic accidents in key road sections such as level intersections and ensuring the safety of pedestrians in the traffic environment.
  • a class of model-based methods such as social force models, energy function or potential energy field models, and Markov models, which convert the personality characteristics of pedestrian movements, interactions between pedestrians and other traffic participants into social forces, Concepts such as potential energy field, use social force or mathematical analysis of potential energy field to build models, and then infer pedestrian movement behavior; the other is based on data-driven deep learning methods.
  • recurrent neural network recurrent neural network
  • LSTM long short-term memory
  • S-LSTM social long short-term memory
  • GAN generative adversarial network
  • GAN generativeadversarial network
  • GAT graph attention
  • GTT graph attention
  • the social long short-term memory (S-LSTM, social long-short tern memory) network model considers the interdependence of pedestrians and surrounding pedestrians, and uses the different characteristics of surrounding pedestrians to predict pedestrian movement trajectories.
  • the model based on generative adversarial network (GAN, generative adversarial network) can generate multiple acceptable pedestrian motion trajectories.
  • GAN generative adversarial network
  • the graph attention (GAT, graph attention) network model enhances the reasoning ability of pedestrian trajectory prediction by using the graph attention mechanism.
  • both current methods need to establish a mathematical model of pedestrian motion in advance or construct a large number of labeled datasets.
  • the behavior of pedestrians crossing the street is not only interdependent, but also affected by factors such as age, gender, psychology, education level, etc., there are individual differences when pedestrians cross the street.
  • Running quickly and other behaviors with a certain randomness For model-based methods, it is impossible to construct an explicit mathematical model to describe pedestrian crossing behavior at level intersections.
  • For data-driven deep learning methods it is difficult to obtain massive labeled datasets to extract the interdependence and randomness of pedestrian crossing behavior.
  • the present invention is aimed at the problems existing in the prior art, and provides a pedestrian crossing behavior prediction method oriented to a plane intersection.
  • the technical solution does not require the establishment of a complex pedestrian motion model or the preparation of massive labeled data sets, and realizes self-learning.
  • the behavior characteristics of pedestrians crossing the street at level intersections and predicting their behaviors such as walking, stopping, and fast running, especially the real-time prediction of pedestrian crossing behaviors when dangers such as collisions and rubbing are induced, and danger warnings for pedestrians and vehicles crossing the street. It is conducive to reducing the incidence of traffic accidents in key road sections such as level intersections and ensuring the safety of pedestrians in the traffic environment.
  • the technical solution adopted in the present invention is: the roadside equipment for data collection selects millimeter wave radar and visual camera.
  • the improved MTTC of the time to collision is used as the immediate reward of the state;
  • a fully convolutional neural network-long short-term memory network (FCN-LSTM) model is established to extract the interdependence and randomness characteristics of pedestrian crossing behavior, and predict the action reward function Secondly, the fully convolutional neural network-long short-term memory network (FCN-LSTM) model is trained based on reinforcement learning; finally, the behaviors of pedestrians walking, running, and stopping when crossing the street are predicted, and danger warning is given to pedestrians and vehicles crossing the street.
  • the method of the present invention specifically comprises the following steps:
  • step 1 design an instant reward function
  • the immediate reward r t of the state is the time to collision MTTC currently detected by the roadside millimeter wave radar.
  • the TTC definition of vehicle conflict only considers the speed of the following vehicle faster than the preceding vehicle, ignoring many conflicts caused by differences in acceleration or deceleration. Especially when the vehicle encounters a pedestrian crossing the street at a level intersection, the vehicle brakes to slow down or accelerates to pass, which may cause danger at this time. Therefore, define an improved time-to-collision MTTC that considers the relative position, relative velocity, and relative acceleration between the vehicle and the pedestrian:
  • ⁇ X t represents the relative position
  • ⁇ V t represents the relative velocity
  • ⁇ A t represents the relative acceleration
  • Step 2 Establish a fully convolutional neural network-long short-term memory network (FCN-LSTM) model to predict the action reward function;
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • the full convolutional neural network (FCN) is used to achieve semantic segmentation, and the pedestrians in the input image are separated from the background. Preserve the spatial information of pedestrians in the input image.
  • the long short-term memory network LSTM is used to utilize the pedestrian's forward behavior information.
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • Standard convolutional layer 1_1 The input of the first layer is the original image, the pixels are Q ⁇ Q', the number of channels is 3, and 96 11 ⁇ 11 convolution kernels are used to convolve with the original input image, and the step size is 4, do not expand the edge.
  • the linear rectification unit (Rectified Linear Units, ReLU) is used as the activation function of the neuron. After ReLU activation, the output dimension is The feature map of ;
  • Local response normalization layer 1_2 In order to prevent data overfitting after the standard convolution layer 1_1 is activated by ReLU, local response normalization is performed.
  • Maximum pooling layer 1_3 Then connect the pooling layer, and perform maximum pooling on the output image after convolution.
  • the size of the pooling layer is 3 ⁇ 3 and the stride is 2.
  • the output feature map of the first layer is obtained, and its dimensions are:
  • Local response normalization layer 2_1 In order to prevent data overfitting after the standard convolution layer 2_1 is activated by ReLU, local response normalization is performed.
  • Maximum pooling layer 2_3 Then connect the pooling layer, and perform maximum pooling on the output image after convolution.
  • the size of the pooling layer is 3 ⁇ 3, and the stride is 2.
  • the output feature map of the second layer is obtained, and its dimensions are:
  • Maximum pooling layer 5_2 Then connect the pooling layer to perform maximum pooling, the size of the pooling layer is 3 ⁇ 3, and the step size is 2, and the output feature map of the fifth layer is obtained, and its dimensions are:
  • Fully convolutional layer 6_1 to allow the input picture to be any size beyond a certain size.
  • the input of the full convolution layer 6 is the output feature map of the convolution layer 5, the number of convolution kernels is 4096, the size of the convolution kernel is: 1 ⁇ 1, no edge expansion, the stride is 1, and the dimension of the output feature map is for:
  • Full convolution layer 7_1 The input of full convolution layer 7 is the output feature map of convolution layer 6, the number of convolution kernels is 4096, the size of convolution kernel is: 1 ⁇ 1, no edge expansion, step size is 1, and the dimension of the output feature map is:
  • Upsampling and skip-level structure processing Upsampling the output feature map of the full convolutional layer 7_1 by 32 times, and upsampling the output feature map of the standard convolutional layer 4_1 by 16 times to obtain the same size as the original input image. image. Since the output feature map of the fully convolutional layer 7_1 is too small and the details are lost too much, in order to make the output image of the fully convolutional layer 7_1 have richer global information and more local details, a skip-level structure is adopted.
  • the 32 times upsampling image of the output feature map of the full convolution layer 7_1 is added and fused with the 16 times upsampling image of the output feature map of the standard convolution layer 4_1, so as to realize the global prediction of the image and predict the details of the local image, and output Q
  • the ⁇ Q' segmented image serves as the input to the next layer.
  • LSTM layer Considering that the pedestrian crossing behavior has a certain continuity, in order to utilize the temporal continuity of pedestrian behavior, the LSTM layer is selected, the input dimension is Q, the time step is Q', and the output dimension is 3.
  • Output Output the reward function values corresponding to the three behaviors of walking, running, and stopping in this state.
  • q(s, walk), q(s, run fast), q(s, stop), S represents the current state of the pedestrian.
  • Step 3 Train a fully convolutional neural network-long short-term memory network (FCN-LSTM) model based on reinforcement learning;
  • the FCN-LSTM model established in the second step of training based on reinforcement learning idea. Considering that the behavior of pedestrians has a certain randomness when crossing the street, in the iterative training process, pedestrians walking, stopping and running quickly are randomly selected with the probability of ⁇ . The pedestrian behavior is greedily selected with a probability of 1- ⁇ , that is, the behavior corresponding to the maximum value of the behavior reward function output in step 2 10) is selected.
  • the FCN-LSTM model can learn that the pedestrian crossing behavior has a certain purpose, and at the same time, different pedestrians have a certain randomness.
  • the specific training steps are as follows:
  • q(s, a) represents the action value function value of the pedestrian, where s represents the current state, a represents the current behavior, and a ⁇ walk, stop, run ⁇ . Initialize the current state s.
  • Sub-step 2 Execute one-step behavior. Randomly generate a random number random of [0, 1]. If random ⁇ ⁇ , then randomly select pedestrian behavior, that is, randomly select a behavior from the pedestrian behavior action set ⁇ walk, stop, run quickly ⁇ ; random ⁇ , ⁇ is assumed to be If the value is 0, 1, the greedy strategy is used to select the pedestrian behavior, that is, the behavior that maximizes the value of q(s, a) from the pedestrian behavior action set ⁇ walk, stop, run ⁇ .
  • Sub-step 3 Update the state and reward function values. After the pedestrian performs one-step behavior, it enters a new state s', uses the immediate reward function r t designed in step 1, and updates the reward function value according to formula (2).
  • q(s t , a t ) represents the action reward function value in the current state
  • t represents the time step
  • max a q(s t+1 , a) represents the maximum action reward function value in the next state
  • ⁇ represents the exploration rate
  • ⁇ is assumed to be 0.1
  • r t is the immediate reward value of the current state
  • ⁇ is the reward decay factor, that is, the influence of subsequent states on the current action decreases step by step
  • ⁇ is assumed to be 0.95.
  • the FCN-LSTM model established in step 2 is trained based on the gradient descent method, and the weight parameters of the FCN-LSTM model are optimized.
  • Sub-step 5 Repeat sub-step 2, sub-step 3, and sub-step 4 until s is terminated, that is, the pedestrian completes the behavior of crossing the street.
  • Step 4 Predict pedestrian crossing behavior and danger warning
  • step 3 to complete multiple rounds of training of the FCN-LSTM model.
  • the present invention has the following advantages: 1) The technical solution does not need to establish a mathematical model of pedestrians crossing the street at a plane intersection in advance, and does not need to prepare a massive data set with labels in advance, and the present invention realizes autonomous learning of pedestrians at plane intersections. Interdependence and randomness characteristics when crossing the street; 2) The technical solution predicts the behaviors of pedestrians walking, stopping and running fast when crossing the street at the level intersection, and warns pedestrians and passing vehicles when it is dangerous.
  • Fig. 1 is the overall scheme schematic diagram of the present invention
  • FIG. 2 is a test scene diagram of a specific embodiment, wherein P represents pedestrians crossing the street at a level intersection, and C represents vehicles;
  • FIG. 3 is a graph showing the accuracy of pedestrian crossing behavior prediction and danger early warning during testing according to an embodiment of the present invention.
  • Example 1 Referring to Figure 1, pedestrians are the main participants in road traffic, and their behavior is an important factor affecting traffic safety, especially at level intersections with a large number of street-crossing behaviors such as school entrances and exits, unsignaled access points, etc., when pedestrians cross the street.
  • the number of traffic accidents that occurred accounted for nearly 70% of the total number of pedestrian traffic accidents. Therefore, the identification and prediction of pedestrian crossing behaviors at level intersections, especially the real-time prediction of pedestrian crossing behaviors when dangers such as collisions and rubbing are induced, and danger warnings for pedestrians and vehicles crossing the street is an important way to build an intelligent road.
  • the basic requirements of the side system are also conducive to reducing the incidence of traffic accidents in key road sections such as level intersections and ensuring the safety of pedestrians in the traffic environment.
  • a class of model-based methods such as social force models, energy function or potential energy field models, and Markov models, which convert the personality characteristics of pedestrian movements, interactions between pedestrians and other traffic participants into social forces, Concepts such as potential energy field, use social force or mathematical analysis of potential energy field to build models, and then infer pedestrian movement behavior; the other is based on data-driven deep learning methods.
  • recurrent neural network recurrent neural network
  • LSTM long short-term memory
  • S-LSTM social long short-term memory
  • GAN generative adversarial network
  • GAN generativeadversarial network
  • GAT graph attention
  • GTT graph attention
  • the social long short-term memory (S-LSTM, social long-short tern memory) network model considers the interdependence of pedestrians and surrounding pedestrians, and uses the different characteristics of surrounding pedestrians to predict pedestrian movement trajectories.
  • the model based on generative adversarial network (GAN, generative adversarial network) can generate multiple acceptable pedestrian motion trajectories.
  • GAN generative adversarial network
  • the graph attention (GAT, graph attention) network model enhances the reasoning ability of pedestrian trajectory prediction by using the graph attention mechanism.
  • both current methods need to establish a mathematical model of pedestrian motion in advance or construct a large number of labeled datasets.
  • the behavior of pedestrians crossing the street is not only interdependent, but also affected by factors such as age, gender, psychology, education level, etc., there are individual differences when pedestrians cross the street.
  • Running quickly and other behaviors with a certain randomness For model-based methods, it is impossible to construct an explicit mathematical model to describe pedestrian crossing behavior at level intersections.
  • For data-driven deep learning methods it is difficult to obtain massive labeled datasets to extract the interdependence and randomness of pedestrian crossing behavior.
  • a method for predicting pedestrian crossing behavior based on deep reinforcement learning is invented.
  • the roadside equipment for data collection in the present invention selects millimeter wave radar and visual camera.
  • the improved time to collision MTTC is used as the immediate reward of the state;
  • a fully convolutional neural network-long short-term memory network (FCN-LSTM) model is established to extract the interdependence and randomness characteristics of pedestrian crossing behavior, and predict the action reward function value;
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • the method of the present invention does not need to establish a complex pedestrian motion model, and does not need to prepare a large number of labeled data sets, so as to realize self-learning behavior characteristics of pedestrians crossing the street at plane intersections and predict their behaviors such as walking, stopping, fast running, etc., especially for inducing pedestrians and vehicles.
  • Real-time prediction of pedestrian crossing behavior in danger of collision, rubbing, etc., and danger warning for pedestrians and passing vehicles which is conducive to reducing the incidence of traffic accidents in key road sections such as level intersections and ensuring the safety of pedestrians in the traffic environment.
  • the method of the present invention specifically comprises the following steps:
  • Step 1 Design an instant reward function
  • the immediate reward r t of the state is the time to collision MTTC that is currently detected by the roadside millimeter wave radar.
  • the TTC definition of vehicle conflict only considers the speed of the following vehicle faster than the preceding vehicle, ignoring many conflicts caused by differences in acceleration or deceleration. Especially when a vehicle encounters a pedestrian crossing the street at a level intersection, the vehicle brakes to slow down or accelerates to pass, which may cause danger at this time. Therefore, define an improved time-to-collision MTTC that considers the relative position, relative velocity, and relative acceleration between the vehicle and the pedestrian:
  • ⁇ X t represents the relative position
  • ⁇ V t represents the relative velocity
  • ⁇ A t represents the relative acceleration
  • Step 2 Establish a fully convolutional neural network-long short-term memory network (FCN-LSTM) model to predict the action reward function;
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • the full convolutional neural network (FCN) is used to achieve semantic segmentation, and the pedestrians in the input image are separated from the background. Preserve the spatial information of pedestrians in the input image.
  • the long short-term memory network LSTM is used to utilize the pedestrian's forward behavior information.
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • Standard convolutional layer 1_1 The input of the first layer is the original image, the pixels are Q ⁇ Q', the number of channels is 3, and 96 11 ⁇ 11 convolution kernels are used to convolve with the original input image, and the step size is 4, do not expand the edge.
  • the linear rectification unit (Rectified Linear Units, ReLU) is used as the activation function of the neuron. After ReLU activation, the output dimension is The feature map of ;
  • Local response normalization layer 1_2 In order to prevent data overfitting after the standard convolution layer 1_1 is activated by ReLU, local response normalization is performed.
  • Maximum pooling layer 1_3 Then connect the pooling layer, and perform maximum pooling on the output image after convolution.
  • the size of the pooling layer is 3 ⁇ 3 and the stride is 2.
  • the output feature map of the first layer is obtained, and its dimensions are:
  • Local response normalization layer 2_1 In order to prevent data overfitting after the standard convolution layer 2_1 is activated by ReLU, local response normalization is performed.
  • Maximum pooling layer 2_3 Then connect the pooling layer, and perform maximum pooling on the output image after convolution.
  • the size of the pooling layer is 3 ⁇ 3, and the stride is 2.
  • the output feature map of the second layer is obtained, and its dimensions are:
  • Maximum pooling layer 5_2 Then connect the pooling layer to perform maximum pooling, the size of the pooling layer is 3 ⁇ 3, and the step size is 2, and the output feature map of the fifth layer is obtained, and its dimensions are:
  • Fully convolutional layer 6_1 to allow the input picture to be any size beyond a certain size.
  • the input of the full convolution layer 6 is the output feature map of the convolution layer 5, the number of convolution kernels is 4096, the size of the convolution kernel is: 1 ⁇ 1, no edge expansion, the stride is 1, and the dimension of the output feature map is for:
  • Full convolution layer 7_1 The input of full convolution layer 7 is the output feature map of convolution layer 6, the number of convolution kernels is 4096, the size of convolution kernel is: 1 ⁇ 1, no edge expansion, step size is 1, and the dimension of the output feature map is:
  • Upsampling and skip-level structure processing Upsampling the output feature map of the full convolutional layer 7_1 by 32 times, and upsampling the output feature map of the standard convolutional layer 4_1 by 16 times to obtain the same size as the original input image. image. Since the output feature map of the fully convolutional layer 7_1 is too small and the details are lost too much, in order to make the output image of the fully convolutional layer 7_1 have richer global information and more local details, a skip-level structure is adopted.
  • the 32 times upsampling image of the output feature map of the full convolution layer 7_1 is added and fused with the 16 times upsampling image of the output feature map of the standard convolution layer 4_1, so as to realize the global prediction of the image and predict the details of the local image, and output Q
  • the ⁇ Q' segmented image serves as the input to the next layer.
  • LSTM layer Considering that the pedestrian crossing behavior has a certain continuity, in order to utilize the temporal continuity of pedestrian behavior, the LSTM layer is selected, the input dimension is Q, the time step is Q', and the output dimension is 3.
  • Output Output the reward function values corresponding to the three behaviors of walking, running, and stopping in this state.
  • q(s, walk), q(s, run fast), q(s, stop), S represents the current state of the pedestrian.
  • Step 3 Train a fully convolutional neural network-long short-term memory network (FCN-LSTM) model based on reinforcement learning;
  • the FCN-LSTM model established in the second step of training based on reinforcement learning idea. Considering that the behavior of pedestrians has a certain randomness when crossing the street, in the iterative training process, pedestrians walking, stopping and running quickly are randomly selected with the probability of ⁇ . The pedestrian behavior is greedily selected with a probability of 1- ⁇ , that is, the behavior corresponding to the maximum value of the behavior reward function output in step 2 10) is selected.
  • the FCN-LSTM model can learn that the pedestrian crossing behavior has a certain purpose, and at the same time, different pedestrians have a certain randomness.
  • the specific training steps are as follows:
  • q(s, a) represents the action value function value of the pedestrian, where s represents the current state, a represents the current behavior, and a ⁇ walk, stop, run ⁇ . Initialize the current state s.
  • Sub-step 2 Execute one-step behavior. Randomly generate a random number random of [0, 1]. If random ⁇ ⁇ , then randomly select pedestrian behavior, that is, randomly select a behavior from the pedestrian behavior action set ⁇ walk, stop, run quickly ⁇ ; random ⁇ , ⁇ is assumed to be If the value is 0, 1, the greedy strategy is used to select the pedestrian behavior, that is, the behavior that maximizes the value of q(s, a) from the pedestrian behavior action set ⁇ walk, stop, run ⁇ .
  • Sub-step 3 Update the state and reward function values. After the pedestrian performs one-step behavior, it enters a new state s', uses the immediate reward function r t designed in step 1, and updates the reward function value according to formula (2).
  • q(s t , a t ) represents the action reward function value in the current state
  • t represents the time step
  • max a q(s t+1 , a) represents the maximum action reward function value in the next state
  • ⁇ represents the exploration rate
  • ⁇ is assumed to be 0.1
  • r t is the immediate reward value of the current state
  • ⁇ is the reward decay factor, that is, the influence of subsequent states on the current action decreases step by step
  • ⁇ is assumed to be 0.95.
  • the FCN-LSTM model established in step 2 is trained based on the gradient descent method, and the weight parameters of the FCN-LSTM model are optimized.
  • Sub-step 5 Repeat sub-step 2, sub-step 3, and sub-step 4 until s terminates, that is, pedestrians complete the behavior of crossing the street.
  • Step 4 Predict pedestrian crossing behavior and danger warning
  • step 3 to complete multiple rounds of training of the FCN-LSTM model.
  • the intelligent vehicle and intelligent traffic simulation test platform prescanv8.5 and the matlab/simulink 2020a co-simulation platform are used to construct the plane intersection scene shown in Figure 2, and the roadside equipment for data collection selects millimeters Wave radar and vision cameras.
  • FCN-LSTM fully convolutional neural network-long short-term memory network
  • the pedestrian crossing scene at the plane intersection is randomly changed, and the test is repeated 20 times.
  • the accuracy rate of pedestrian crossing behavior prediction and danger warning is shown in Figure 3. . It can be seen that the method of the present invention can accurately predict the pedestrian crossing behavior at the level intersection.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Human Computer Interaction (AREA)
  • Computational Linguistics (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)

Abstract

一种面向平面交叉口的行人过街行为预测方法，包括以下步骤：步骤一：设计即时奖励函数；步骤二：建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测动作奖励函数；步骤三：基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型；步骤四：预测行人过街行为及危险预警；该技术方案无需建立复杂的行人运动模型、无需准备海量的带标签数据集，实现自主学习平面交叉口处行人过街行为特征并预测其行走、驻足、快跑等行为，特别是对诱发人车碰撞、擦蹭等危险时行人过街行为的实时预测，对过街行人和来往车辆进行危险预警，有利于减少平面交叉口等重点路段交通事故发生率，保障交通环境中行人的安全。

Description

一种面向平面交叉口的行人过街行为预测方法 技术领域
本发明涉及一种预测方法，具体涉及一种面向平面交叉口的行人过街行为预测方法，属于交通参与者行为建模及预测技术领域。
背景技术
行人作为道路交通的主要参与者，其行为是影响交通安全的重要因素，尤其是在学校出入口、无信号接入口等存在大量过街行为的平面交叉口，行人过街时发生的交通事故数占行人发生交通事故总数近七成。因此，针对平面交叉口处行人过街行为的识别和预测，特别是对诱发人车碰撞、擦蹭等危险时行人过街行为的实时预测，并对过街行人和来往车辆进行危险预警，是构建智能路侧系统的基本要求，也有利于减少平面交叉口等重点路段交通事故发生率，保障交通环境中行人的安全。
目前，主要有两类方法实现对行人过街行为的预测。一类基于模型的方法，如社会力模型、能量函数或者势能场模型、马尔科夫模型，该类模型将行人运动的个性特征、行人与其他交通参与者之间的相互作用转换为社会力、势能场等概念，利用社会力或者势能场的数学解析式构建模型，进而推断行人运动行为；另一类是基于数据驱动的深度学习的方法。如循环神经网络(RNN,recurrent neural network)以及长短期记忆(LSTM,long-short term memory)网络、社会长短期记忆(S-LSTM,social long-short tern memory)网络、生成对 抗网络(GAN,generativeadversarial network)、图注意力(GAT,graph attention)网络等。其中循环神经网络(RNN,recurrent neural network)以及长短期记忆(LSTM,long-short term memory)将行人的连续行为看作时间序列，实现了行人行为的序列化预测。在此基础上，社会长短期记忆(S-LSTM,social long-short tern memory)网络模型考虑行人与周围行人的相互依赖性，利用周围行人的不同特征预测行人运动轨迹。基于生成对抗网络(GAN,generativeadversarial network)的模型可生成多条可接受的行人运动轨迹。图注意力(GAT,graph attention)网络模型通过使用图注意力机制增强了行人运动轨迹预测的推理能力。
虽然目前方法在预测行人简单行为及行人间相互影响方面取得很好的效果，但是目前两类方法都需要事先建立行人运动的数学模型或构建大量带标签的数据集。对于平面交叉口这类行人共享空间的环境，行人过街行为既是相互依赖的，同时，受年龄、性别、心理、受教育程度等因素影响，行人过街时个体又存在差异性，存在行走、驻足、快跑等具有一定随机性的行为。对基于模型的方法而言，无法构建明确的数学模型描述平面交叉口行人过街行为。对基于数据驱动的深度学习方法而言，难以获取海量的带标签数据集以提取行人过街行为的相互依赖性及随机性特征。针对目前的基于模型的方法和基于数据驱动的深度学习方法在预测平面交叉口行为过街行为时所存在的难点，需要发明一种平面交叉口行人过街行为预测方法，该方法无需事先建立复杂的行人运动模型、无需准备海量的带标签数据集，该方 法能够实现自主学习平面交叉口处行人过街行为特征并预测其行走、驻足、快跑等行为。
发明内容
本发明正是针对现有技术中存在的问题，提供一种面向平面交叉口的行人过街行为预测方法，该技术方案无需建立复杂的行人运动模型、无需准备海量的带标签数据集，实现自主学习平面交叉口处行人过街行为特征并预测其行走、驻足、快跑等行为，特别是对诱发人车碰撞、擦蹭等危险时行人过街行为的实时预测，对过街行人和来往车辆进行危险预警，有利于减少平面交叉口等重点路段交通事故发生率，保障交通环境中行人的安全。
为实现本发明的目的，本发明所采用的技术方案是：数据采集的路侧设备选用毫米波雷达和视觉相机。首先，以改进的将要碰撞时间MTTC作为状态的即时奖励；其次建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型提取行人过街行为的相互依赖及随机性特征，并预测动作奖励函数值；再其次基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型；最后预测行人过街时行走、快跑、驻足等行为，并对过街行人和来往车辆进行危险预警。本发明的方法具体包括以下步骤：
一种面向平面交叉口的行人过街行为预测方法，步骤一：设计即时奖励函数；
以路侧毫米波雷达当前检测出的改进将要碰撞时间MTTC作为状 态的即时奖励r t。TTC定义车辆冲突仅仅考虑后车比前车的速度快，忽略了很多因加速度或减速度的差异造成的冲突。特别是车辆在平面交叉口遇到过街行人时，车辆刹车减速或者加速通过，此时可能会造成危险。因此，定义一种考虑车辆与行人间的相对位置、相对速度、相对加速度的改进将要碰撞时间MTTC：
Figure PCTCN2021086572-appb-000001
其中，ΔX t表示相对位置，ΔV t表示相对速度，ΔA t表示相对加速度，根据ΔX t、ΔV t、ΔA t的正负情况，且使MTTC t≥0，取公式(1)中±。
如果平面交叉口某一状态内检测到多位行人或者多辆车，则依据公式(1)计算每一位行人与所有车辆间的MTTC，取最小的MTTC作为该行人当前状态的即时奖励r t。
步骤二：建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测动作奖励函数；
考虑到行人行为间的相互依赖关系，这种依赖关系则表现为行人在空间上相互依赖、相互约束，故利用全卷积神经网络FCN实现语义分割，将输入图像中行人从背景中分离出来，保留输入图像中行人的空间信息。同时，考虑行人行为在时间上具有连续性，故利用长短期记忆网络LSTM对行人前向行为信息加以利用。建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测行人行为奖励函数值，即向FCN-LSTM模型输入路侧相机拍摄的平面交叉口行人过街图像，FCN-LSTM模型输出行走、快跑、驻足三种离散行为对应的奖励函数值。FCN-LSTM模型具体结构如下：
1)标准卷积层1_1：第一层的输入是原始图像，像素为Q×Q’，通道数为3，用96个11×11的卷积核与原始输入图像做卷积，步长为4，不扩充边缘。将线性整流单元(Rectified Linear Units,ReLU)作为神经元的激活函数，经过ReLU激活，输出维度为
Figure PCTCN2021086572-appb-000002
的特征图；
局部响应规范化层1_2：为防止标准卷积层1_1经ReLU激活之后数据过拟合，进行局部响应归一化。
最大池化层1_3：接着连接池化层，对卷积后输出图像进行最大池化，池化层大小为3×3，步长为2，得到第一层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000003
2)标准卷积层2_1：卷积层2的输入为卷积层1的输出特征图，卷积核的个数为256，卷积核大小为：5×5，以padding＝2进行边缘扩充，步长为1。经过ReLU激活，输出维度为
Figure PCTCN2021086572-appb-000004
的特征图。
局部响应规范化层2_1：为防止标准卷积层2_1经ReLU激活之后数据过拟合，进行局部响应归一化。
最大池化层2_3：接着连接池化层，对卷积后输出图像进行最大池化，池化层大小为3×3，步长为2，得到第二层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000005
3)标准卷积层3_1：卷积层3的输入为卷积层2的输出特征图，卷积核的个数为384，卷积核大小为：3×3，以padding＝1进行边缘 扩充，步长为1，经过ReLU激活，得到第三层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000006
4)标准卷积层4_1：卷积层4的输入为卷积层3的输出特征图，卷积核的个数为384，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1，经过ReLU激活，得到第四层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000007
5)标准卷积层5_1：卷积层5的输入为卷积层4的输出特征图，卷积核的个数为256，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1。过ReLU激活，输出维度为维度为
Figure PCTCN2021086572-appb-000008
的特征图。
最大池化层5_2：接着连接池化层，进行最大池化，池化层大小为3×3，步长为2，得到第五层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000009
6)全卷积层6_1：为允许输入的图片为超过某一尺寸的任意大小。全卷积层6的输入为卷积层5的输出特征图，卷积核的个数为4096，卷积核大小为：1×1，无边缘扩充，步长为1，输出特征图的维度为：
Figure PCTCN2021086572-appb-000010
7)全卷积层7_1：全卷积层7的输入为卷积层6的输出特征图，卷积核的个数为4096，卷积核大小为：1×1，无边缘扩充，步长为1，输出特征图的维度为：
Figure PCTCN2021086572-appb-000011
8)上采样与跳级结构处理：将全卷积层7_1的输出特征图进行32倍上采样、将标准卷积层4_1的输出特征图进行16倍的上采样，获得与原始输入图像相同尺寸的图像。由于全卷积层7_1的输出特征图过小，细节损失过多，为使全卷积层7_1的输出图像有更丰富的全局信息和更多的局部细节，采用跳级结构。即将全卷积层7_1输出特征图的32倍上采样图像与标准卷积层4_1输出特征图的16倍上采样图像进行相加融合，实现图像全局预测的同时进行局部图像细节的预测，输出Q×Q'分割图像作为下一层的输入。
9)LSTM层：考虑到行人过街行为具有一定的连续性，为利用行人行为在时间上的连续性，故选用LSTM层，输入维度为Q，时间步为Q'，输出维度为3。
10)输出：输出行人在该状态下行走、快跑、驻足三种行为对应的奖励函数值。q(s,行走),q(s,快跑),q(s,驻足)，S表示行人当前状态。由q(s,行走),q(s,快跑),q(s,驻足)组成行为奖励函数预测值q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}。
步骤三：基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型；
基于强化学习思想训练步骤二建立的FCN-LSTM模型。考虑行人过街时行为具有一定的随机性，在迭代训练过程中，以ξ的概率随机选择行人行走、驻足、快跑行为。以1-ξ的概率贪婪的选择行人行为，即选择步骤二10)输出的行为奖励函数最大值所对应的行为。使得FCN-LSTM模型能够学习到行人过街行为既具有一定目的性，同时不 同行人个体又具有一定的随机性的特点。其训练具体步骤如下：
子步骤1：初始化q(s,a)＝0。q(s,a)表示行人的动作价值函数值，其中s表示当前状态，a表示当前的行为，a∈{行走、驻足、快跑}。初始化当前状态s。
子步骤2：执行一步行为。随机生成[0,1]的随机数random，若random＜ξ，则随机选择行人行为，即从行人行为动作集{行走,驻足,快跑}中随机选择一行为；random≥ξ，ξ拟取值0,1，则使用贪婪策略选择行人行为，即从行人行为动作集{行走,驻足,快跑}中使q(s,a)值最大的行为。
子步骤3：更新状态和奖励函数值。行人执行一步行为后，进入新的状态s’，利用步骤一设计的即时奖励函数r t，并根据式(2)更新奖励函数值。
q(s t,a t)←q(s t,a t)+α(r t+γmax aq(s t+1,a)-q(s t,a t))      (2)
其中，q(s t,a t)表示当前状态下动作奖励函数值，t表示时间步，max aq(s t+1,a)表示下一状态最大的动作奖励函数值，α表示探索率，α拟取0.1，r t表示当前状态的即时奖励值，γ表示奖励衰减因子，即后续状态对当前动作的影响逐级递减，γ∈[0,1]，γ拟取值0.95。
子步骤4：取式(2)更新后的q(s t,a t)作为真值，取步骤二中FCN-LSTM模型输出的q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}的最大值作为预测值。基于梯度下降方法训练步骤二中建立的FCN-LSTM模型，优化FCN-LSTM模型的权重参数。
子步骤5：重复执行子步骤2、子步骤3、子步骤4，直到s终止， 即行人完成过街行为。
步骤四：预测行人过街行为及危险预警；
重复执行步骤三，完成FCN-LSTM模型的多轮训练。向训练完成的FCN-LSTM模型输入部署在平面交叉口路侧的相机图像，FCN-LSTM模型输出q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}，取max{q(s,行走),q(s,驻足),q(s,快跑)}所对应的行为即为本发明预测的平面交叉口行人过街的行为。若根据当前状态，预测行为是行走或快跑时，则向平面交叉口处过街行人发出预警信号，提醒其注意可能发生的危险。
相对于现有技术，本发明具有如下优点，1)该技术方案无需事先建立平面交叉口处行人过街的数学模型、无需事先准备带标签的海量数据集，本发明实现自主学习平面交叉口处行人过街时相互依赖性与随机性特征；2)该技术方案预测出平面交叉口处行人过街时行走、驻足、快跑等行为，并在危险时对过街行人和来往车辆进行预警。
附图说明
图1是本发明整体方案示意图；
图2是具体实施例的测试场景图，其中P表示平面交叉口过街行人，C表示车辆；
图3是本发明实施例测试时行人过街行为预测及危险预警准确率结果图。
具体实施方式
为了加深对本发明的理解，下面结合附图对本实施例做详细的说明。
实施例1：参见图1，行人作为道路交通的主要参与者，其行为是影响交通安全的重要因素，尤其是在学校出入口、无信号接入口等存在大量过街行为的平面交叉口，行人过街时发生的交通事故数占行人发生交通事故总数近七成。因此，针对平面交叉口处行人过街行为的识别和预测，特别是对诱发人车碰撞、擦蹭等危险时行人过街行为的实时预测，并对过街行人和来往车辆进行危险预警，是构建智能路侧系统的基本要求，也有利于减少平面交叉口等重点路段交通事故发生率，保障交通环境中行人的安全。
目前，主要有两类方法实现对行人过街行为的预测。一类基于模型的方法，如社会力模型、能量函数或者势能场模型、马尔科夫模型，该类模型将行人运动的个性特征、行人与其他交通参与者之间的相互作用转换为社会力、势能场等概念，利用社会力或者势能场的数学解析式构建模型，进而推断行人运动行为；另一类是基于数据驱动的深度学习的方法。如循环神经网络(RNN,recurrent neural network)以及长短期记忆(LSTM,long-short term memory)网络、社会长短期记忆(S-LSTM,social long-short tern memory)网络、生成对抗网络(GAN,generativeadversarial network)、图注意力(GAT,graph attention)网络等。其中循环神经网络(RNN,recurrent neural network)以及长短期记忆(LSTM,long-short term memory)将行人的连续行为看作时间序列，实现了行人行为的序列化预测。在此基础上，社会长短期记忆(S-LSTM,social long-short tern  memory)网络模型考虑行人与周围行人的相互依赖性，利用周围行人的不同特征预测行人运动轨迹。基于生成对抗网络(GAN,generativeadversarial network)的模型可生成多条可接受的行人运动轨迹。图注意力(GAT,graph attention)网络模型通过使用图注意力机制增强了行人运动轨迹预测的推理能力。
虽然目前方法在预测行人简单行为及行人间相互影响方面取得很好的效果，但是目前两类方法都需要事先建立行人运动的数学模型或构建大量带标签的数据集。对于平面交叉口这类行人共享空间的环境，行人过街行为既是相互依赖的，同时，受年龄、性别、心理、受教育程度等因素影响，行人过街时个体又存在差异性，存在行走、驻足、快跑等具有一定随机性的行为。对基于模型的方法而言，无法构建明确的数学模型描述平面交叉口行人过街行为。对基于数据驱动的深度学习方法而言，难以获取海量的带标签数据集以提取行人过街行为的相互依赖性及随机性特征。
针对目前的基于模型的方法和基于数据驱动的深度学习方法在预测平面交叉口行为过街行为时所存在的难点，需要发明一种平面交叉口行人过街行为预测方法，该方法无需事先建立复杂的行人运动模型、无需准备海量的带标签数据集，该方法能够实现自主学习平面交叉口处行人过街行为特征并预测其行走、驻足、快跑等行为。
为实现本发明的目的，发明了一种基于深度强化学习的行人过街行为预测方法。本发明数据采集的路侧设备选用毫米波雷达和视觉相机。首先，以改进的将要碰撞时间MTTC作为状态的即时奖励；其次， 建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型提取行人过街行为的相互依赖及随机性特征，并预测动作奖励函数值；再其次，基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型；最后预测行人过街时行走、快跑、驻足等行为，并对过街行人和来往车辆进行危险预警。本发明的方法无需建立复杂的行人运动模型、无需准备海量的带标签数据集，实现自主学习平面交叉口处行人过街行为特征并预测其行走、驻足、快跑等行为，特别是对诱发人车碰撞、擦蹭等危险时行人过街行为的实时预测，对过街行人和来往车辆进行危险预警，有利于减少平面交叉口等重点路段交通事故发生率，保障交通环境中行人的安全。
本发明的方法具体包括以下步骤：
步骤一：设计即时奖励函数；
以路侧毫米波雷达当前检测出的改进将要碰撞时间MTTC作为状态的即时奖励r t。TTC定义车辆冲突仅仅考虑后车比前车的速度快，忽略了很多因加速度或减速度的差异造成的冲突。特别是车辆在平面交叉口遇到过街行人时，车辆刹车减速或者加速通过，此时可能会造成危险。因此，定义一种考虑车辆与行人间的相对位置、相对速度、相对加速度的改进将要碰撞时间MTTC：
Figure PCTCN2021086572-appb-000012
其中，ΔX t表示相对位置，ΔV t表示相对速度，ΔA t表示相对加速度，根据ΔX t、ΔV t、ΔA t的正负情况，且使MTTC t≥0，取公式(1)中±。
如果平面交叉口某一状态内检测到多位行人或者多辆车，则依据 公式(1)计算每一位行人与所有车辆间的MTTC，取最小的MTTC作为该行人当前状态的即时奖励r t。
步骤二：建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测动作奖励函数；
考虑到行人行为间的相互依赖关系，这种依赖关系则表现为行人在空间上相互依赖、相互约束，故利用全卷积神经网络FCN实现语义分割，将输入图像中行人从背景中分离出来，保留输入图像中行人的空间信息。同时，考虑行人行为在时间上具有连续性，故利用长短期记忆网络LSTM对行人前向行为信息加以利用。建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测行人行为奖励函数值，即向FCN-LSTM模型输入路侧相机拍摄的平面交叉口行人过街图像，FCN-LSTM模型输出行走、快跑、驻足三种离散行为对应的奖励函数值。FCN-LSTM模型具体结构如下：
1)标准卷积层1_1：第一层的输入是原始图像，像素为Q×Q’，通道数为3，用96个11×11的卷积核与原始输入图像做卷积，步长为4，不扩充边缘。将线性整流单元(Rectified Linear Units,ReLU)作为神经元的激活函数，经过ReLU激活，输出维度为
Figure PCTCN2021086572-appb-000013
的特征图；
局部响应规范化层1_2：为防止标准卷积层1_1经ReLU激活之后数据过拟合，进行局部响应归一化。
最大池化层1_3：接着连接池化层，对卷积后输出图像进行最大池化，池化层大小为3×3，步长为2，得到第一层的输出特征图，其 维度为：
Figure PCTCN2021086572-appb-000014
2)标准卷积层2_1：卷积层2的输入为卷积层1的输出特征图，卷积核的个数为256，卷积核大小为：5×5，以padding＝2进行边缘扩充，步长为1。经过ReLU激活，输出维度为
Figure PCTCN2021086572-appb-000015
的特征图。
局部响应规范化层2_1：为防止标准卷积层2_1经ReLU激活之后数据过拟合，进行局部响应归一化。
最大池化层2_3：接着连接池化层，对卷积后输出图像进行最大池化，池化层大小为3×3，步长为2，得到第二层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000016
3)标准卷积层3_1：卷积层3的输入为卷积层2的输出特征图，卷积核的个数为384，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1，经过ReLU激活，得到第三层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000017
4)标准卷积层4_1：卷积层4的输入为卷积层3的输出特征图，卷积核的个数为384，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1，经过ReLU激活，得到第四层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000018
5)标准卷积层5_1：卷积层5的输入为卷积层4的输出特征图，卷积核的个数为256，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1。过ReLU激活，输出维度为维度为
Figure PCTCN2021086572-appb-000019
的特征 图。
最大池化层5_2：接着连接池化层，进行最大池化，池化层大小为3×3，步长为2，得到第五层的输出特征图，其维度为：
Figure PCTCN2021086572-appb-000020
6)全卷积层6_1：为允许输入的图片为超过某一尺寸的任意大小。全卷积层6的输入为卷积层5的输出特征图，卷积核的个数为4096，卷积核大小为：1×1，无边缘扩充，步长为1，输出特征图的维度为：
Figure PCTCN2021086572-appb-000021
7)全卷积层7_1：全卷积层7的输入为卷积层6的输出特征图，卷积核的个数为4096，卷积核大小为：1×1，无边缘扩充，步长为1，输出特征图的维度为：
Figure PCTCN2021086572-appb-000022
8)上采样与跳级结构处理：将全卷积层7_1的输出特征图进行32倍上采样、将标准卷积层4_1的输出特征图进行16倍的上采样，获得与原始输入图像相同尺寸的图像。由于全卷积层7_1的输出特征图过小，细节损失过多，为使全卷积层7_1的输出图像有更丰富的全局信息和更多的局部细节，采用跳级结构。即将全卷积层7_1输出特征图的32倍上采样图像与标准卷积层4_1输出特征图的16倍上采样图像进行相加融合，实现图像全局预测的同时进行局部图像细节的预测，输出Q×Q'分割图像作为下一层的输入。
9)LSTM层：考虑到行人过街行为具有一定的连续性，为利用行人行为在时间上的连续性，故选用LSTM层，输入维度为Q，时间步为Q'，输出维度为3。
10)输出：输出行人在该状态下行走、快跑、驻足三种行为对应的奖励函数值。q(s,行走),q(s,快跑),q(s,驻足)，S表示行人当前状态。由q(s,行走),q(s,快跑),q(s,驻足)组成行为奖励函数预测值q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}。
步骤三：基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型；
基于强化学习思想训练步骤二建立的FCN-LSTM模型。考虑行人过街时行为具有一定的随机性，在迭代训练过程中，以ξ的概率随机选择行人行走、驻足、快跑行为。以1-ξ的概率贪婪的选择行人行为，即选择步骤二10)输出的行为奖励函数最大值所对应的行为。使得FCN-LSTM模型能够学习到行人过街行为既具有一定目的性，同时不同行人个体又具有一定的随机性的特点。其训练具体步骤如下：
子步骤1：初始化q(s,a)＝0。q(s,a)表示行人的动作价值函数值，其中s表示当前状态，a表示当前的行为，a∈{行走、驻足、快跑}。初始化当前状态s。
子步骤2：执行一步行为。随机生成[0,1]的随机数random，若random＜ξ，则随机选择行人行为，即从行人行为动作集{行走,驻足,快跑}中随机选择一行为；random≥ξ，ξ拟取值0,1，则使用贪婪策略选择行人行为，即从行人行为动作集{行走,驻足,快跑}中使q(s,a)值最大的行为。
子步骤3：更新状态和奖励函数值。行人执行一步行为后，进入新的状态s’，利用步骤一设计的即时奖励函数r t，并根据式(2)更新奖 励函数值。
q(s t,a t)←q(s t,a t)+α(r t+γmax aq(s t+1,a)-q(s t,a t))      (2)
其中，q(s t,a t)表示当前状态下动作奖励函数值，t表示时间步，max aq(s t+1,a)表示下一状态最大的动作奖励函数值，α表示探索率，α拟取0.1，r t表示当前状态的即时奖励值，γ表示奖励衰减因子，即后续状态对当前动作的影响逐级递减，γ∈[0,1]，γ拟取值0.95。
子步骤4：取式(2)更新后的q(s t,a t)作为真值，取步骤二中FCN-LSTM模型输出的q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}的最大值作为预测值。基于梯度下降方法训练步骤二中建立的FCN-LSTM模型，优化FCN-LSTM模型的权重参数。
子步骤5：重复执行子步骤2、子步骤3、子步骤4，直到s终止，即行人完成过街行为。
步骤四：预测行人过街行为及危险预警；
重复执行步骤三，完成FCN-LSTM模型的多轮训练。向训练完成的FCN-LSTM模型输入部署在平面交叉口路侧的相机图像，FCN-LSTM模型输出q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}，取max{q(s,行走),q(s,驻足),q(s,快跑)}所对应的行为即为本发明预测的平面交叉口行人过街的行为。若根据当前状态，预测行为是行走或快跑时，则向平面交叉口处过街行人发出预警信号，提醒其注意可能发生的危险。
为进一步验证本发明的效果，利用智能车与智能交通仿真测试平台prescanv8.5和matlab/simulink 2020a联合仿真平台，构建如附 图2所示的平面交叉口场景，数据采集的路侧设备选用毫米波雷达和视觉相机。全卷积神经网络-长短期记忆网络(FCN-LSTM)模型训练完成后，随机改变平面交叉口的行人过街场景，重复测试20次，行人过街行为预测及危险预警准确率如附图3所示。可以看出：本发明方法能够准确地预测出平面交叉口处行人的过街行为。
需要说明的是上述实施例，并非用来限定本发明的保护范围，在上述技术方案的基础上所作出的等同变换或替代均落入本发明权利要求所保护的范围。

Claims (5)

  1. 一种面向平面交叉口的行人过街行为预测方法，其特征在于，所述方法包括以下步骤：
    步骤一：设计即时奖励函数；
    步骤二：建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测动作奖励函数；
    步骤三：基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型；
    步骤四：预测行人过街行为及危险预警。
  2. 根据权利要求1所述的面向平面交叉口的行人过街行为预测方法，其特征在于，所述步骤一：设计即时奖励函数，具体如下：
    以路侧毫米波雷达当前检测出的改进将要碰撞时间MTTC作为状态的即时奖励r t，TTC定义车辆冲突仅仅考虑后车比前车的速度快，定义一种考虑车辆与行人间的相对位置、相对速度、相对加速度的改进将要碰撞时间MTTC：
    Figure PCTCN2021086572-appb-100001
    其中，ΔX t表示相对位置，ΔV t表示相对速度，ΔA t表示相对加速度，根据ΔX t、ΔV t、ΔA t的正负情况，且使MTTC t≥0，取公式(1)中±；
    如果平面交叉口某一状态内检测到多位行人或者多辆车，则依据公式(1)计算每一位行人与所有车辆间的MTTC，取最小的MTTC作为该行人当前状态的即时奖励r t。
  3. 根据权利要求2所述的面向平面交叉口的行人过街行为预测方法， 其特征在于，所述步骤二：建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测动作奖励函数，具体如下：
    考虑到行人行为间的相互依赖关系，这种依赖关系则表现为行人在空间上相互依赖、相互约束，故利用全卷积神经网络FCN实现语义分割，将输入图像中行人从背景中分离出来，保留输入图像中行人的空间信息，同时，考虑行人行为在时间上具有连续性，故利用长短期记忆网络LSTM对行人前向行为信息加以利用，建立全卷积神经网络-长短期记忆网络(FCN-LSTM)模型预测行人行为奖励函数值，即向FCN-LSTM模型输入路侧相机拍摄的平面交叉口行人过街图像，FCN-LSTM模型输出行走、快跑、驻足三种离散行为对应的奖励函数值，FCN-LSTM模型具体结构如下：
    1)标准卷积层1_1：第一层的输入是原始图像，像素为Q×Q’，通道数为3，用96个11×11的卷积核与原始输入图像做卷积，步长为4，不扩充边缘。将线性整流单元(RectifiedLinear Units,ReLU)作为神经元的激活函数，经过ReLU激活，输出维度为
    Figure PCTCN2021086572-appb-100002
    的特征图；
    局部响应规范化层1_2：为防止标准卷积层1_1经ReLU激活之后数据过拟合，进行局部响应归一化；
    最大池化层1_3：接着连接池化层，对卷积后输出图像进行最大池化，池化层大小为3×3，步长为2，得到第一层的输出特征图，其维度为：
    Figure PCTCN2021086572-appb-100003
    2)标准卷积层2_1：卷积层2的输入为卷积层1的输出特征图，卷积核的个数为256，卷积核大小为：5×5，以padding＝2进行边缘扩充，步长为1。经过ReLU激活，输出维度为
    Figure PCTCN2021086572-appb-100004
    的特征图；
    局部响应规范化层2_1：为防止标准卷积层2_1经ReLU激活之后数据过拟合，进行局部响应归一化；
    最大池化层2_3：接着连接池化层，对卷积后输出图像进行最大池化，池化层大小为3×3，步长为2，得到第二层的输出特征图，其维度为：
    Figure PCTCN2021086572-appb-100005
    3)标准卷积层3_1：卷积层3的输入为卷积层2的输出特征图，卷积核的个数为384，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1，经过ReLU激活，得到第三层的输出特征图，其维度为：
    Figure PCTCN2021086572-appb-100006
    4)标准卷积层4_1：卷积层4的输入为卷积层3的输出特征图，卷积核的个数为384，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1，经过ReLU激活，得到第四层的输出特征图，其维度为：
    Figure PCTCN2021086572-appb-100007
    5)标准卷积层5_1：卷积层5的输入为卷积层4的输出特征图，卷积核的个数为256，卷积核大小为：3×3，以padding＝1进行边缘扩充，步长为1。过ReLU激活，输出维度为维度为
    Figure PCTCN2021086572-appb-100008
    的特征图；
    最大池化层5_2：接着连接池化层，进行最大池化，池化层大小为3×3，步长为2，得到第五层的输出特征图，其维度为：
    Figure PCTCN2021086572-appb-100009
    6)全卷积层6_1：为允许输入的图片为超过某一尺寸的任意大小，全卷积层6的输入为卷积层5的输出特征图，卷积核的个数为4096，卷积核大小为：1×1，无边缘扩充，步长为1，输出特征图的维度为：
    Figure PCTCN2021086572-appb-100010
    7)全卷积层7_1：全卷积层7的输入为卷积层6的输出特征图，卷积核的个数为4096，卷积核大小为：1×1，无边缘扩充，步长为1，输出特征图的维度为：
    Figure PCTCN2021086572-appb-100011
    8)上采样与跳级结构处理：将全卷积层7_1的输出特征图进行32倍上采样、将标准卷积层4_1的输出特征图进行16倍的上采样，获得与原始输入图像相同尺寸的图像。由于全卷积层7_1的输出特征图过小，细节损失过多，为使全卷积层7_1的输出图像有更丰富的全局信息和更多的局部细节，采用跳级结构，即将全卷积层7_1输出特征图的32倍上采样图像与标准卷积层4_1输出特征图的16倍上采样图像进行相加融合，实现图像全局预测的同时进行局部图像细节的预测，输出Q×Q'分割图像作为下一层的输入；
    9)LSTM层：考虑到行人过街行为具有一定的连续性，为利用行人行为在时间上的连续性，故选用LSTM层，输入维度为Q，时间步为Q'，输出维度为3；
    10)输出：输出行人在该状态下行走、快跑、驻足三种行为对应 的奖励函数值；q(s,行走),q(s,快跑),q(s,驻足)，S表示行人当前状态，由q(s,行走),q(s,快跑),q(s,驻足)组成行为奖励函数预测值q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}。
  4. 根据权利要求3所述的面向平面交叉口的行人过街行为预测方法，其特征在于，步骤三：基于强化学习训练全卷积神经网络-长短期记忆网络(FCN-LSTM)模型，具体如下：
    基于强化学习思想训练步骤二建立的FCN-LSTM模型，考虑行人过街时行为具有一定的随机性，在迭代训练过程中，以ξ的概率随机选择行人行走、驻足、快跑行为，以1-ξ的概率贪婪的选择行人行为，即选择步骤二10)输出的行为奖励函数最大值所对应的行为，使得FCN-LSTM模型能够学习到行人过街行为既具有一定目的性，同时不同行人个体又具有一定的随机性的特点，其训练具体步骤如下：
    子步骤1：初始化q(s,a)＝0。q(s,a)表示行人的动作价值函数值，其中s表示当前状态，a表示当前的行为，a∈{行走、驻足、快跑}。初始化当前状态s；
    子步骤2：执行一步行为，随机生成[0,1]的随机数random，若random＜ξ，则随机选择行人行为，即从行人行为动作集{行走,驻足,快跑}中随机选择一行为；random≥ξ，ξ拟取值0,1，则使用贪婪策略选择行人行为，即从行人行为动作集{行走,驻足,快跑}中使q(s,a)值最大的行为；
    子步骤3：更新状态和奖励函数值。行人执行一步行为后，进入新的状态s’，利用步骤一设计的即时奖励函数r t，并根据式(2)更新奖 励函数值；
    q(s t,a t)←q(s t,a t)+α(r t+γmax aq(s t+1,a)-q(s t,a t))    (2)
    其中，q(s t,a t)表示当前状态下动作奖励函数值，t表示时间步，max aq(s t+1,a)表示下一状态最大的动作奖励函数值，α表示探索率，α拟取0.1，r t表示当前状态的即时奖励值，γ表示奖励衰减因子，即后续状态对当前动作的影响逐级递减，γ∈[0,1]，γ拟取值0.95；
    子步骤4：取式(2)更新后的q(s t,a t)作为真值，取步骤二中FCN-LSTM模型输出的q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}的最大值作为预测值，基于梯度下降方法训练步骤二中建立的FCN-LSTM模型，优化FCN-LSTM模型的权重参数；
    子步骤5：重复执行子步骤2、子步骤3、子步骤4，直到s终止，即行人完成过街行为。
  5. 根据权利要求3或4所述的面向平面交叉口的行人过街行为预测方法，其特征在于，步骤四：预测行人过街行为及危险预警，重复执行步骤三，完成FCN-LSTM模型的多轮训练，向训练完成的FCN-LSTM模型输入部署在平面交叉口路侧的相机图像，FCN-LSTM模型输出q_value＝{q(s,行走),q(s,驻足),q(s,快跑)}，取max{q(s,行走),q(s,驻足),q(s,快跑)}所对应的行为即为本发明预测的平面交叉口行人过街的行为，若根据当前状态，预测行为是行走或快跑时，则向平面交叉口处过街行人发出预警信号，提醒其注意可能发生的危险。
PCT/CN2021/086572 2020-11-26 2021-04-12 一种面向平面交叉口的行人过街行为预测方法 WO2022110611A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/249,715 US20230410659A1 (en) 2020-11-26 2021-04-12 Method for predicting pedestrian crossing behavior for intersection

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011357565.7A CN112487954B (zh) 2020-11-26 2020-11-26 一种面向平面交叉口的行人过街行为预测方法
CN202011357565.7 2020-11-26

Publications (1)

Publication Number Publication Date
WO2022110611A1 true WO2022110611A1 (zh) 2022-06-02

Family

ID=74936212

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/086572 WO2022110611A1 (zh) 2020-11-26 2021-04-12 一种面向平面交叉口的行人过街行为预测方法

Country Status (3)

Country Link
US (1) US20230410659A1 (zh)
CN (1) CN112487954B (zh)
WO (1) WO2022110611A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115294661A (zh) * 2022-10-10 2022-11-04 青岛浩海网络科技股份有限公司 一种基于深度学习的行人危险行为识别方法
CN115402350A (zh) * 2022-08-01 2022-11-29 宁波大学科学技术学院 异质性人车交互行为虚拟仿真方法
CN116612636A (zh) * 2023-05-22 2023-08-18 暨南大学 基于多智能体强化学习及多模态信号感知的信号灯协同控制方法

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112487954B (zh) * 2020-11-26 2023-12-05 东南大学 一种面向平面交叉口的行人过街行为预测方法
CN113793497B (zh) * 2021-08-04 2022-09-20 东南大学 一种考虑多因素的行人过街行为预测方法
CN117057656B (zh) * 2023-08-17 2024-05-31 广东飞翔云计算有限公司 基于数字孪生的智慧城市管理方法及系统
CN117436937B (zh) * 2023-12-21 2024-07-05 暗物智能科技(广州)有限公司 一种考虑行人画像的路径预测方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105678034A (zh) * 2016-03-14 2016-06-15 东南大学 行人过街时间模型及信号交叉口人行横道宽度优化方法
US20200293064A1 (en) * 2019-03-15 2020-09-17 Nvidia Corporation Temporal information prediction in autonomous machine applications
CN111860269A (zh) * 2020-07-13 2020-10-30 南京航空航天大学 一种多特征融合的串联rnn结构及行人预测方法
CN112487954A (zh) * 2020-11-26 2021-03-12 东南大学 一种面向平面交叉口的行人过街行为预测方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109614907B (zh) * 2018-11-28 2022-04-19 安徽大学 基于特征强化引导卷积神经网络的行人再识别方法及装置
CN109684933A (zh) * 2018-11-30 2019-04-26 广州大学 一种前方行人窜出马路的预警方法
US11242050B2 (en) * 2019-01-31 2022-02-08 Honda Motor Co., Ltd. Reinforcement learning with scene decomposition for navigating complex environments
CN111178584B (zh) * 2019-12-04 2021-12-07 常熟理工学院 一种基于双层融合模型的无人驾驶行为预测方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105678034A (zh) * 2016-03-14 2016-06-15 东南大学 行人过街时间模型及信号交叉口人行横道宽度优化方法
US20200293064A1 (en) * 2019-03-15 2020-09-17 Nvidia Corporation Temporal information prediction in autonomous machine applications
CN111860269A (zh) * 2020-07-13 2020-10-30 南京航空航天大学 一种多特征融合的串联rnn结构及行人预测方法
CN112487954A (zh) * 2020-11-26 2021-03-12 东南大学 一种面向平面交叉口的行人过街行为预测方法

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115402350A (zh) * 2022-08-01 2022-11-29 宁波大学科学技术学院 异质性人车交互行为虚拟仿真方法
CN115402350B (zh) * 2022-08-01 2023-11-07 宁波大学科学技术学院 异质性人车交互行为虚拟仿真方法
CN115294661A (zh) * 2022-10-10 2022-11-04 青岛浩海网络科技股份有限公司 一种基于深度学习的行人危险行为识别方法
CN116612636A (zh) * 2023-05-22 2023-08-18 暨南大学 基于多智能体强化学习及多模态信号感知的信号灯协同控制方法
CN116612636B (zh) * 2023-05-22 2024-01-23 暨南大学 基于多智能体强化学习的信号灯协同控制方法

Also Published As

Publication number Publication date
CN112487954B (zh) 2023-12-05
CN112487954A (zh) 2021-03-12
US20230410659A1 (en) 2023-12-21

Similar Documents

Publication Publication Date Title
WO2022110611A1 (zh) 一种面向平面交叉口的行人过街行为预测方法
Al-qaness et al. An improved YOLO-based road traffic monitoring system
WO2021203972A1 (zh) 基于碰撞模型的风险预防方法、装置、设备及存储介质
Zhang et al. Prediction of pedestrian-vehicle conflicts at signalized intersections based on long short-term memory neural network
CN107730903A (zh) 基于深度卷积神经网络的违章停车和抛锚车视觉检测系统
Zhu et al. Deep learning for autonomous vehicle and pedestrian interaction safety
Hu et al. Efficient mapping of crash risk at intersections with connected vehicle data and deep learning models
Chen et al. Applying Artificial Intelligence and Deep Belief Network to predict traffic congestion evacuation performance in smart cities
Takimoto et al. Predicting traffic accidents with event recorder data
Jia et al. Lane‐Changing Behavior Prediction Based on Game Theory and Deep Learning
Saravanarajan et al. Car crash detection using ensemble deep learning
Hua et al. Modeling and simulation of approaching behaviors to signalized intersections based on risk quantification
Gupta et al. Real-time traffic control and monitoring
Wu et al. Lane-GNN: Integrating GNN for Predicting Drivers' Lane Change Intention
Ottom et al. An adaptive traffic lights system using machine learning.
CN115981302A (zh) 车辆跟驰换道行为决策方法、装置及电子设备
Aysal et al. Real-Time Application of Traffic Sign Recognition Algorithm with Deep Learning
Jalali Moghaddam et al. Fuzzy Q-learning traffic light control based on traffic flow and pedestrian number estimated from visual information
Tumuluru et al. SMS: SIGNS MAY SAVE–Traffic Sign Recognition and Detection using CNN
Desai et al. Vehicle-Pedestrian Detection Methods for Urban Traffic Control System: A Survey
Zhao et al. Efficient textual explanations for complex road and traffic scenarios based on semantic segmentation
Gu et al. Research on fast recognition of vulnerable traffic participants in intelligent connected vehicles on edge computing
Lv et al. A Lane‐Changing Decision‐Making Model of Bus Entering considering Bus Priority Based on GRU Neural Network
Ismail et al. Vehicle Detection, Counting, and Classification System based on Video using Deep learning Models
Manguri et al. Emergency vehicles classification for traffic signal system using optimized transfer DenseNet201 model

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21896127

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21896127

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 27/11/2023)