WO2022126669A1

WO2022126669A1 - Subway pedestrian flow network fusion method based on video pedestrian recognition, and pedestrian flow prediction method

Info

Publication number: WO2022126669A1
Application number: PCT/CN2020/137804
Authority: WO
Inventors: 徐超; 高思斌; 李少利; 李永强; 戴李杰
Original assignee: 中电海康集团有限公司
Priority date: 2020-12-16
Filing date: 2020-12-19
Publication date: 2022-06-23
Also published as: CN112541440A; CN112541440B

Abstract

A subway pedestrian flow network fusion method based on video pedestrian recognition and a pedestrian flow prediction method. Video data of subway stations is used for performing statistical analysis on specific directions of the pedestrian flows entering and exiting the subway stations, the movement of crowds is performed in a huge network by means of fusing a subway network and a ground traffic network, each subway station is a node in the network, and each subway line and each on-ground road traffic line are sides of the network. A graph neural network is used for deducing the changes of the pedestrian flow of the whole traffic network so as to analyze and predict the pedestrian flow amount and the direction of the pedestrian flows at each station. Deeper pedestrian flow analysis for entrances of each subway station is achieved, the resource scheduling of each station and each entrance is facilitated, and the mutual influence of the on-ground pedestrian flow and the underground pedestrian flow can be pre-determined in a timely manner, so that traffic early warning can be performed on the ground or underground in a timely manner to avoid traffic congestion, deploy station security measures in advance, etc.

Description

A subway pedestrian flow network fusion method and pedestrian flow prediction method based on video pedestrian recognition

technical field

The invention belongs to the technical field of smart cities, and in particular relates to a subway pedestrian flow network fusion method and a pedestrian flow prediction method based on video pedestrian recognition.

Background technique

The application of video surveillance is getting more and more applications in the field of digital security, and people counting through video is becoming more and more important. For example, in stations, tourist attractions, exhibition areas, commercial streets, etc. Staff mobility, resource allocation and better security.

The existing subway people flow prediction is generally based on the import and export card swiping data of each station, and the obtained results can only obtain the prediction of the people flow in and out of the station. Generally, by analyzing the historical credit card data and road network map of the subway station, a passenger flow prediction model of the subway station is constructed to predict the future passenger flow change of the station. Stations and departures. In addition, the crowd statistics scheme mainly uses infrared, cameras, communication data, etc., to monitor the crowd density in subway cars in real time. For example, the image-based method generally uses the different degrees of shading of the light source of the image collected by the camera through different crowd densities to obtain crowd distribution images under different densities. Image analysis and processing are carried out on this and before and after comparison, and finally the subway is comprehensively obtained. The density of people in the carriage.

However, the existing schemes all have the following problems: each station generally has many entrances and exits, and there are various situations in the way of each entrance and exit. As a result, the above-ground and underground transportation networks cannot be deeply integrated, so it is impossible to reasonably allocate the resources of each entrance and exit of the site; at the same time, the impact of the flow of people entering and leaving the site on the above-ground traffic cannot be obtained.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a subway pedestrian flow network fusion method and pedestrian flow prediction method based on video pedestrian recognition, connect the above-ground and underground traffic routes and subway routes, refine the outbound or inbound direction of the pedestrian flow at each station, and improve the Traffic prediction accuracy.

To achieve the above object, the technical scheme adopted by the present invention is:

A subway pedestrian flow network fusion method based on video pedestrian recognition is used to realize the fusion statistics of subway and ground pedestrian flow to assist traffic early warning. The subway pedestrian flow network fusion method based on video pedestrian recognition includes:

Step 1. Receive monitoring images of various entrances and exits of the subway station, and the monitoring images are acquired by image acquisition equipment disposed at each entrance and exit;

Step 2, extracting pedestrian target coordinate frame information and pedestrian target feature information in the monitoring image, where the pedestrian target feature information includes pedestrian features, pedestrian entry and exit status, and pedestrian exit or entry direction;

Step 3. Based on the monitoring image of the same image acquisition device, the similarity calculation is performed according to the pedestrian target coordinate frame information and the pedestrian target feature information, and the pedestrian trajectory for the same pedestrian is obtained;

Step 4. Perform similarity matching of pedestrian target feature information based on the pedestrian trajectories corresponding to different image acquisition devices, and combine the successfully matched pedestrian trajectories to update the pedestrian trajectories of the corresponding pedestrians;

Step 5. Obtain the subway routes, subway stations, entrances and exits of each station in the designated area, and the traffic routes on the ground corresponding to the entrances and exits, and integrate and construct a subway traffic network map in the designated area;

Step 6. According to the latest pedestrian trajectories in the preset time period, count the total inbound and outbound pedestrian flow of each station, and the inbound and outbound pedestrian flow on the traffic routes corresponding to each entrance and exit in each station;

Step 7. On the basis of the subway traffic network map, superimpose the total inbound and outbound traffic of each station of the subway station and the inbound and outbound traffic on the traffic routes corresponding to each entrance and exit in each station to obtain the subway and ground traffic Converged people flow mobile network diagram.

Several optional methods are also provided below, which are not intended to be additional limitations on the above-mentioned overall solution, but are merely further additions or optimizations. On the premise of no technical or logical contradiction, each optional method can be independently implemented for the above-mentioned overall solution. The combination can also be a combination between multiple optional ways.

Preferably, based on the monitoring image of the same image acquisition device, the similarity calculation is performed according to the pedestrian target coordinate frame information and the pedestrian target feature information to obtain the pedestrian trajectory for the same pedestrian, including:

Step 3.1, obtain the pedestrian target coordinate frame information and the pedestrian target feature information in the current monitoring image of the current image acquisition device;

Step 3.2: Determine whether the tracking track set corresponding to the image acquisition device is empty. The tracking track set is used to save the pedestrian track of the pedestrian. If the tracking track set is not empty, perform step 3.3; otherwise, directly use the pedestrian obtained this time. The target coordinate frame information and the pedestrian target feature information are added to the tracking trajectory set and ended;

Step 3.3, using unscented Kalman filtering to obtain the estimated target coordinate frame information based on the pedestrian trajectory in the tracking trajectory set;

Step 3.4. According to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculate the coordinate frame similarity between the current pedestrian target and the saved pedestrian target one by one, and the pedestrian target feature information based on the pedestrian trajectory and the current monitoring image. The feature information of the pedestrian target in the calculation is to calculate the feature similarity between the current pedestrian target and the saved pedestrian target one by one, and the current pedestrian target and the saved pedestrian target are obtained based on the weighted summation of the coordinate frame similarity and the feature similarity. similarity between the two;

Step 3.5. Based on the similarity between the current pedestrian target and the saved pedestrian target, the Hungarian matching algorithm is used to match the pedestrian trajectory in the tracking trajectory set and the pedestrian target obtained this time;

Step 3.6. If there is a pedestrian target that has not been successfully matched this time, directly add the pedestrian target coordinate frame information and pedestrian target feature information corresponding to the pedestrian target to the tracking trajectory set, and mark it as a new trajectory; if the pedestrian trajectory and the pedestrian target are successfully matched. If the pedestrian target is a pedestrian target, the pedestrian trajectory of the pedestrian is updated according to the pedestrian target coordinate frame information corresponding to the pedestrian target and the pedestrian target feature information; If there are pedestrian trajectories that have not been successfully matched for multiple consecutive frames in the tracking trajectory set, it is considered that the pedestrian target has left the monitoring range of the current image acquisition device, and the pedestrian trajectory is marked as leaving trajectory; if marked as leaving If the pedestrian trajectory of the trajectory is not successfully matched within the specified time threshold, the pedestrian trajectory is considered to be complete, and the pedestrian trajectory is deleted from the tracking trajectory set.

Preferably, the similarity matching of pedestrian target feature information is performed based on the pedestrian trajectories corresponding to different image acquisition devices, and the successfully matched pedestrian trajectories are combined to update the pedestrian trajectories of the corresponding pedestrians, including:

Step 4.1. Take a set of tracking trajectories corresponding to an image acquisition device, and compare the pedestrian trajectories marked as new trajectories in the set of tracking trajectories with the pedestrian trajectories marked as departure trajectories in the tracking trajectories set corresponding to the rest of the image acquisition devices one by one. calculate;

Step 4.2. If the similarity is greater than the preset threshold, it means that the two pedestrian trajectories are successfully matched;

Step 4.3: Combine the two successfully matched pedestrian trajectories to obtain a new pedestrian trajectory of the pedestrian, and use the new pedestrian trajectory to replace the corresponding pedestrian trajectory in the tracking trajectory set where the new trajectory is located.

The present invention also provides a people flow prediction method for predicting people flow based on the fusion of subway and ground people flow to assist traffic early warning. The people flow prediction method includes:

Using the subway pedestrian flow network fusion method based on video pedestrian recognition to obtain the pedestrian flow network map in the specified time period;

Based on the people flow network graph, the graph neural network is used to predict the total inbound and outbound traffic of each site in a specified time period in the future;

Based on the inbound and outbound traffic on the traffic route corresponding to each entrance and exit of each site in the pedestrian flow network diagram, the mean value of the inbound and outbound proportions of the traffic route corresponding to each entry and exit of each site is obtained;

According to the mean value of inbound and outbound proportions, the total inbound and outbound predicted traffic of each station is allocated to obtain the inbound and outbound predicted traffic on the traffic route corresponding to each entry and exit of each station.

Preferably, the use of the subway pedestrian flow network fusion method based on video pedestrian recognition to obtain the people flow mobile network map within a specified time period, including:

Preferably, the predicted traffic of people entering and exiting each site within a specified time period in the future is predicted by using a graph neural network based on the people flow mobile network graph, including:

The people flow network diagram takes subway stations as vertices and the traffic routes corresponding to the entrances and exits of the stations as edges, each vertex has a feature vector including the total inbound and outbound people flow, and the model for constructing the people flow mobile network diagram is as follows :

G _t =(V _t , ε, W)

In the formula, G _t is the people flow network graph at time t, V _t is the vector composed of the feature vectors of all vertices, ε represents the edge set between vertices, W is the weight of the adjacency matrix, and t is the current moment;

When predicting the flow of people at each vertex, based on the feature vector of the vertex in the historical time period t-M+1 to t, predict the feature vector of the specified time period t+1 to t+H in the future, where M and H are preset coefficients, and the target model for predicting the flow of people is constructed as follows:

In the formula,

is the predicted feature vector from time t+1 to time t+H, v _t-M+1 ,...,v _t represents the input feature vector from time -M+1 to time t;

Based on the constructed people flow prediction target model, a graph neural network is used to solve the people flow prediction target model, and the total inbound and outbound predicted people flow of each site in a specified time period in the future is obtained.

The present invention provides a subway pedestrian flow network fusion method and a pedestrian flow prediction method based on video pedestrian recognition. The video data of the subway station is used to perform statistical analysis on the specific direction of people entering and exiting the subway station, and the aboveground and underground road traffic lines and subway lines are opened up to form a Measurable and complete traffic network of people flow on the ground and underground; the integration of subway network and ground transportation network will make the movement of people in a huge network, and each subway station will become a node in this network, Every subway line and overground road traffic line will be an edge of this network. The graph neural network is used to deduce the change of people flow in the entire transportation network, so as to analyze and predict each station, the number of people flow and the direction of people flow. Realizing a more in-depth analysis of the flow of people at the entrances and exits of each subway station not only facilitates the resource scheduling of each station and the entrance and exit, but also can predict the mutual influence of the above-ground and underground people in a timely manner, so as to timely provide traffic warnings on the ground or underground, avoid traffic jams, and Deploy site security measures, etc.

Description of drawings

Fig. 1 is the flow chart of the subway pedestrian flow network fusion method based on video pedestrian recognition of the present invention;

Fig. 2 is the training schematic diagram of SSD target detection network of the present invention;

Fig. 3 is the training schematic diagram of MobileNet neural network of the present invention;

4 is a schematic diagram of the present invention performing a distillation operation on a network neural network;

Fig. 5 is the flow chart of the pedestrian trajectory tracking of the present invention;

6 is a flowchart of a multi-factor fusion pedestrian target tracking method of the present invention;

7 is a flowchart of a target tracking method based on pedestrian image features of the present invention;

8 is a schematic diagram of an embodiment of a subway traffic network diagram of the present invention;

9 is a schematic diagram of the present invention performing problem modeling based on a spatiotemporal sequence;

FIG. 10 is a schematic structural diagram of the framework of the STGCN of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention.

In one of the embodiments, a method for integrating subway pedestrian flow networks based on video pedestrian recognition is provided, which establishes the connection between the traffic routes on the ground and the subway routes on the ground, and analyzes the inbound and outbound flow of people at each entrance and exit of each subway station. whereabouts. The correlation between the above-ground (that is, the ground) and underground (that is, the subway) people flow overcomes the fact that the existing above-ground or underground people flow statistics only consider the above-ground or underground single-level people flow, and do not further consider the factors that affect the above-ground and underground people flow. Insufficient accuracy of statistics or predictions. The people flow statistics method based on the integration of the above-ground and underground networks of the present invention can not only utilize the resource scheduling of each entrance and exit of each subway station, but also can combine the above-ground transportation network for the early warning of the people flow of the subway stations, and at the same time, it can combine the underground subway network. , improve the forward-looking and timeliness of traffic control.

As shown in FIG. 1, the present embodiment of the subway pedestrian flow network fusion method based on video pedestrian recognition includes the following steps:

Step 1. Receive monitoring images of various entrances and exits of the subway station, where the monitoring images are acquired by image acquisition equipment disposed at each entrance and exit.

Due to the need to correlate the above-ground and underground pedestrian flow data, the image acquisition device should be deployed so that its monitoring range includes the entire entrance and exit and the above-ground traffic route corresponding to the entrance and exit, which lays the foundation for identifying pedestrian entry and exit status, exit or entry direction.

The image capture device in this embodiment may be an optical camera, a binocular camera, a TOF camera, etc., and each image capture device has a unique device id so as to distinguish each image capture device. Therefore, while receiving the monitoring image, the device id of the image acquisition device corresponding to the monitoring image and the corresponding time stamp are obtained.

It should be noted that generally, an image acquisition device is installed at each entrance and exit of a subway station to meet the application requirements of the present invention, but the present invention is not limited to installing only one image acquisition device at each entrance and exit. In actual monitoring needs or monitoring accuracy requirements In the case of a subway station, multiple image capture devices can be installed at one entrance and exit to obtain video image information more comprehensively, and image capture devices can be installed inside subway stations and along the traffic routes corresponding to the entrances and exits to expand the range of video image capture. Get more comprehensive and complete people flow statistics or pedestrian trajectories.

Step 2: Extract the pedestrian target coordinate frame information and pedestrian target feature information in the monitoring image, where the pedestrian target feature information includes pedestrian features, pedestrian entry and exit status, and pedestrian exit or entry direction.

The pedestrian target coordinate frame information and the pedestrian target feature information are the basic information for identifying and locating the pedestrian target. In this embodiment, two parts of information are extracted based on a neural network. There are many target recognition neural networks and feature recognition neural networks currently used, and this embodiment does not limit the neural networks used. In order to facilitate understanding, this embodiment is described by taking the SSD target detection network extracting pedestrian target coordinate frame information and the MobileNet neural network extracting pedestrian target feature information as examples.

As shown in Figure 2, the SSD target detection network is used as the training and application process of the target recognition algorithm. The specific steps are as follows:

1. Construct a pedestrian target dataset. The dataset includes image data and annotation data. The annotation data indicates the area of the pedestrian target in the specified image.

2. Count the aspect ratio of the pedestrian target in the dataset, and use the clustering method to cluster the aspect ratio data to obtain n cluster centers, that is, n aspect ratios. The ratio of the anchor frame in the target recognition network is the n aspect ratio.

3. Enhance the image data to obtain training data, such as color change, random cropping, image enlargement and reduction, rotation, etc.

4. The training data is input into the SSD target detection network for target recognition, and the neural network outputs the coordinate frame information of the pedestrian in the image.

5. For the coordinate frame output by the SSD target detection network, the non-maximum suppression method (NMS) is used to remove the duplicate frame, and the final output coordinate frame is obtained.

6. In the output coordinate frame, select the frame whose intersection ratio with the labeled coordinate frame is greater than a larger threshold as a positive sample, and select a frame whose intersection ratio is less than a smaller threshold as a negative sample. And randomly select a certain proportion and number of samples from the positive and negative samples as training samples to train the neural network.

7. Calculate the loss function according to the final training sample coordinate frame and the labeled data, and adjust the neural network parameters using the back-propagation method.

8. After a certain training, the final SSD target detection network is obtained.

In the application of SSD target detection network, the trained SSD target detection network accepts the image input, outputs the pedestrian coordinate frame, performs non-maximum suppression operation on the coordinate frame, deletes the duplicate coordinate frame, and finally sets a certain threshold, when the coordinate frame Pedestrian target coordinate frame information as output when the reliability of is greater than the threshold.

As shown in Figure 3, the MobileNet neural network is used as the training application process of the pedestrian re-identification algorithm. The specific steps are as follows:

1. Build a pedestrian re-identification data set, which includes image data and id data of pedestrians corresponding to the image.

2. Three images are selected for one training, two different images of pedestrian A and one image of other pedestrians. After image enhancement, the three images are input into the MobileNet neural network respectively, and the pedestrian features are output.

3. Calculate the ternary loss function according to the output of the neural network. This loss function can make the pedestrian image features of the same person more similar and the image features of different pedestrians less similar, so it can be used to train the pedestrian re-identification algorithm. After calculating the loss function, backpropagation is used to train the neural network.

When deploying the target detection network, in order to run the network faster, it is necessary to perform a distillation operation on the network neural network, as shown in Figure 4. The teacher model is generally a relatively large trained neural network model. This model generally has high accuracy, but the network has many parameters and slow running time. The student model is generally a model with a small number of parameters. If this model is directly trained with labeled data, it is often difficult to train. The neural network distillation allows the student model to learn from the labeled data and the teacher model at the same time, which can often achieve better results.

In the MobileNet neural network application, the final MobileNet neural network obtained by training receives image data and pedestrian id data, and outputs the pedestrian target feature information corresponding to each pedestrian.

The target recognition neural network, the feature recognition neural network and the corresponding training application method provided by this embodiment can completely, comprehensively and accurately extract the corresponding data in the monitoring image, and improve high-quality basic information for the statistics of people flow.

Since the real trajectories of pedestrians cannot be accurately identified only based on the pedestrian's state in a monitoring image, it is necessary to track the trajectories of pedestrians. As shown in FIG. 5 , the multi-target tracking method in this embodiment is divided into two parts, one of which is a multi-factor fusion pedestrian target tracking method under the monitoring screen of the same image acquisition device, and this method generates the image acquisition device under the monitoring screen. trajectories of pedestrians. The other is a pedestrian feature-based target tracking method across image acquisition devices, which is used to match the pedestrian trajectories of the same pedestrian under different image acquisition devices. The pedestrian target tracking method across image acquisition devices directly uses the pedestrian target feature information to calculate the similarity, and if the similarity is greater than a certain threshold, it is judged as the same pedestrian, and related trajectories are associated. Combining the two methods can obtain cross-regional pedestrian trajectory data, so as to perform complete pedestrian trajectory tracking and improve the accuracy of pedestrian flow statistics.

Step 3: Based on the monitoring image of the same image acquisition device, the similarity calculation is performed according to the pedestrian target coordinate frame information and the pedestrian target feature information, and the pedestrian trajectory for the same pedestrian is obtained. That is, the multi-factor fusion pedestrian target tracking method, as shown in Figure 6, as follows:

Step 3.1, obtain the pedestrian target coordinate frame information and the pedestrian target feature information in the current monitoring image of the current image acquisition device.

Step 3.2: Determine whether the tracking track set corresponding to the image acquisition device is empty. The tracking track set is used to save the pedestrian track of the pedestrian. If the tracking track set is not empty, perform step 3.3; otherwise, directly use the pedestrian obtained this time. The target coordinate frame information and the pedestrian target feature information are added to the tracking track set and ended.

Step 3.3, using unscented Kalman filtering to obtain the estimated target coordinate frame information based on the pedestrian trajectory in the tracking trajectory set.

Unscented Kalman filter is developed on the basis of Kalman filter and transformation. It uses lossless transformation to apply Kalman filter under linear assumption to nonlinear systems. This method can be used in the case of many overlapping occlusions. Better tracking of pedestrians. The unscented Kalman filter is used to estimate the position of each existing pedestrian trajectory at the current moment.

Step 3.4. According to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculate the coordinate frame similarity between the current pedestrian target and the saved pedestrian target one by one, and the pedestrian target feature information based on the pedestrian trajectory and the current monitoring image. Pedestrian target feature information in , calculate the feature similarity between the current pedestrian target (that is, the pedestrian target obtained this time) and the saved pedestrian target one by one, based on the weighted summation of the coordinate frame similarity and the feature similarity to get The similarity between the current pedestrian target and the saved pedestrian target.

Based on the pedestrian target estimation frame in the trajectory set predicted by the unscented Kalman filter and the pedestrian target frame currently detected, the IOU of the pedestrian target, the center point distance of the pedestrian target, and the size gap of the pedestrian target are calculated. The feature similarity of the information (such as the cosin similarity of the feature, etc.) is weighted and summed to construct the similarity between the existing trajectory and the pedestrian to be matched.

The similarity between the current pedestrian target and the saved pedestrian target finally obtained in this embodiment combines the coordinate frame similarity and the feature similarity, and matches the pedestrian target from multiple directions to significantly improve the accuracy of the pedestrian trajectory. It should be noted that the calculation of the similarity of the coordinate frame and the similarity of the features are relatively mature technologies in the field of pedestrian trajectory tracking, which will not be repeated in this embodiment. The weights of the weighted summation can be set according to the focus of actual use.

And in order to intuitively express the similarity between pedestrian targets, a similarity matrix can be used for storage. The vertical direction in the similarity matrix is the saved pedestrian target, the horizontal direction is the current pedestrian target, and the corresponding value in the matrix is the corresponding value. The similarity between the saved pedestrian target and the current pedestrian target.

Step 3.5: Based on the similarity between the current pedestrian target and the saved pedestrian target, the Hungarian matching algorithm is used to match the pedestrian trajectory in the tracking trajectory set and the pedestrian target obtained this time.

In this embodiment, when a pedestrian trajectory under the same image acquisition device is generated, the pedestrian trajectory of each pedestrian is updated in real time, and when a new pedestrian is added within the monitoring range, the newly added pedestrian is confirmed through successive successful matches to avoid false detection; And after the pedestrian trajectory is not successfully matched for many times in a row, the pedestrian trajectory is deleted to reduce the storage pressure and matching pressure, and improve the matching speed.

Step 4. Perform similarity matching of pedestrian target feature information based on pedestrian trajectories corresponding to different image acquisition devices, and combine the successfully matched pedestrian trajectories to update the pedestrian trajectories of the corresponding pedestrians. That is, the target tracking method based on pedestrian image features, as shown in Figure 7, is as follows:

Step 4.1. Take a set of tracking trajectories corresponding to an image acquisition device, and perform similarity between the pedestrian trajectories marked as new trajectories in the set of tracking trajectories and the pedestrian trajectories marked as departure trajectories in the set of tracking trajectories corresponding to the rest of the image acquisition devices one by one. calculate.

Since pedestrians do not appear on the two monitoring images at the same time under normal circumstances, in this embodiment, only the departure trajectories corresponding to other image acquisition devices are used for matching, so as to ensure that the matching results conform to normal pedestrian movement behaviors, and at the same time reduce the feature matching pressure. , Improve the speed of cross-region matching.

Step 4.2. If the similarity is greater than the preset threshold, it means that the two pedestrian trajectories are successfully matched. In this embodiment, the similarity is calculated based on the pedestrian target feature information carried by the two pedestrian trajectories, and the pedestrian target feature information obtained from the calculation may be the pedestrian target feature information in the latest monitoring image in the pedestrian trajectory, or the latest pedestrian target feature information. The average value of pedestrian target feature information in several frames of surveillance images. The similarity can be cosine similarity and is matched by the Hungarian algorithm.

In this embodiment, two segments of pedestrian trajectories that have been successfully matched are combined, and the combination is preferably by splicing two segments of pedestrian trajectories in chronological order to obtain a pedestrian trajectory that conforms to the pedestrian's real moving path, and the combined cross-regional pedestrian trajectory is moved to In the tracking track set where the new track is located, the departure track is also moved from the original tracking track set to the tracking track set where the new track is located, realizing joint management of pedestrian tracks.

Step 5: Obtain the subway routes, subway stations, entrances and exits of each station in the designated area, and the traffic routes on the ground corresponding to the entrances and exits, and integrate and construct a subway traffic network map in the designated area.

In this embodiment, the traffic route on the ground corresponding to each entrance and exit should be understood as the ground road where the entrance and exit are located. Of course, taking out the ground road where the entrance is located is the basic operation for the integration of the above-ground and underground transportation networks of the present invention. It is the same as the installation of the image acquisition equipment in step 1. If the image acquisition equipment is not only installed at the entrance and exit, but also extended to the preset range of the subway station. For all ground roads, the corresponding ground traffic routes may also be other ground roads extended by the ground road where the entrance and exit are located where the image acquisition device is located. The final constructed subway traffic network diagram is shown in Figure 8. The dots represent subway stations, the solid lines represent ground roads, and the dotted lines represent subway routes. The connection point of the subway station indicates the entrance and exit. Of course, since the present invention mainly counts the flow of people at the entrance and exit of the subway station and the flow of people on the ground road, the formed subway traffic network diagram may also only include the network diagram with the subway station as the point and the ground road as the edge.

Step 6: According to the latest pedestrian trajectories in the preset time period, count the total inbound and outbound pedestrian flow of each site, and the inbound and outbound pedestrian flow on the traffic route corresponding to each entrance and exit in each site.

The pedestrian trajectory includes at least the pedestrian moving path in the same image acquisition device. The moving path has a direction. Therefore, according to the pedestrian trajectory, the pedestrian can be identified as entering or exiting the station, and the corresponding statistics can be obtained. Outbound foot traffic.

And since the monitoring range of the image acquisition device includes the above-ground traffic route, the pedestrian trajectory in the monitoring screen includes the process of entering the screen from a certain direction of the traffic route to enter the station, or exiting the station and moving out of the screen in a certain direction of the traffic route, that is, the corresponding The inbound and outbound pedestrian flow on the traffic route can be obtained, and the inbound and outbound pedestrian flow on the traffic route includes the direction of pedestrians leaving the station or entering the station (taking the exit and the exit only include two directions of left turn and right turn as For example, the outbound traffic on the traffic route includes the traffic of people who turn left and enter the traffic route after exiting the station and the traffic volume of people who turn right and enter the traffic route after exiting the station).

The mobile network diagram of the flow of people finally obtained by the present invention shows the flow of people entering and leaving the station at each entrance and exit of each subway station. The corresponding traffic route is north-south extension, then the flow of people going south after exiting the station, and the flow of people going north after exiting the station), the direction of the incoming traffic from the traffic route (for example, the traffic route corresponding to the entrance and exit is north-south). If the route is extended, the inbound traffic flows from the route to the south, and the inbound traffic flows from the north).

The above-mentioned pedestrian flow direction or direction can be obtained by superimposing multiple monitoring images in a time series according to the pedestrian's entry and exit status and the pedestrian's exit or entry direction in each frame of the monitoring image, that is, the pedestrian trajectory is generated. It is convenient to analyze the possible traffic congestion and other situations of the above-ground traffic routes based on the subway pedestrian flow, so as to take timely measures such as evacuation and early warning.

In another embodiment, a people flow prediction method is also provided, which is based on the fusion of subway and ground people flow to predict the flow of people to assist traffic early warning, and the people flow prediction method includes:

Using the subway pedestrian network fusion method based on video pedestrian recognition, the pedestrian flow network map in the specified time period is obtained.

Based on the human flow mobile network graph, the graph neural network is used to predict the total inbound and outbound traffic of each site within a specified time period in the future.

Based on the inbound and outbound flow of people on the traffic routes corresponding to the entrances and exits of each site in the pedestrian flow network diagram, the mean value of the inbound and outbound ratios of the traffic routes corresponding to each entry and exit of each site is obtained.

According to the mean value of inbound and outbound proportions, the total inbound and outbound predicted people flow of each station is allocated to obtain the inbound and outbound predicted people flow on the traffic route corresponding to each entrance and exit of each station.

In another embodiment, the use of a subway pedestrian flow network fusion method based on video pedestrian recognition to obtain a people flow mobile network map within a specified time period includes:

Step 1, receive the monitoring image of each entrance and exit of subway station, and described monitoring image is acquired by the image acquisition equipment that is arranged at each entrance and exit;

In another embodiment, based on the monitoring image of the same image acquisition device, the similarity calculation is performed according to the pedestrian target coordinate frame information and the pedestrian target feature information to obtain the pedestrian trajectory for the same pedestrian, including:

Step 3.4. According to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculate the coordinate frame similarity between the current pedestrian target and the saved pedestrian target one by one, and the pedestrian target feature information based on the pedestrian trajectory and the current monitoring image. The feature information of the pedestrian target in the calculation is to calculate the feature similarity between the current pedestrian target and the saved pedestrian target one by one, and the current pedestrian target and the saved pedestrian target are obtained based on the weighted summation of the coordinate frame similarity and the feature similarity. similarity between the two.

Step 3.5, based on the similarity between the current pedestrian target and the saved pedestrian target, adopt the Hungarian matching algorithm to match the pedestrian trajectory in the tracking trajectory set and the pedestrian target obtained this time;

In another embodiment, the similarity matching of pedestrian target feature information is performed based on pedestrian trajectories corresponding to different image acquisition devices, and the successfully matched pedestrian trajectories are combined to update the pedestrian trajectories of the corresponding pedestrians, including:

Step 4.1. Take a set of tracking trajectories corresponding to an image acquisition device, and perform similarity between the pedestrian trajectories marked as new trajectories in the set of tracking trajectories and the pedestrian trajectories marked as departure trajectories in the set of tracking trajectories corresponding to the rest of the image acquisition devices one by one. calculate;

For the specific limitation of using the subway pedestrian flow network fusion method based on video pedestrian recognition in this embodiment to obtain the pedestrian flow network map within a specified time period, please refer to the above-mentioned specific restrictions on the subway pedestrian flow network fusion method based on video pedestrian recognition, which will not be discussed here. Let's go into details. The construction of the underground and above-ground people flow network graph in this embodiment is to represent the subway network of the entire city as a graph, in which the subway station is the vertex, and the subway line and the traffic routes connecting the entrances and exits of the subway station on the ground are the edges (or only take The traffic route connecting the entrances and exits of the subway station on the ground is an edge. If the edge contains a subway line, the flow of people on the subway line is no data, or on the basis of the present invention, the subway line is obtained by using the existing subway station swiping card information and other methods. internal movement of people). Each vertex has a feature vector composed of video statistics of people flow, and an adjacency matrix can be defined to encode pairwise dependencies between vertices. Therefore, the subway network does not need to use grids to represent subway stations, nor does it need to use CNN to capture features, but can be described by a general network graph, and the use of graph neural network (GCN) can effectively capture the subway network level and Not irregular spatiotemporal dependencies at the grid level.

Among them, the problem modeling is the mathematical modeling of the application of underground and above-ground people flow statistics network in the prediction of people flow in all sides. It uses the historical people flow value of the network to predict the people flow of the network in the next few days. Specifically, the problem can be modeled with the spatiotemporal sequence shown in Figure 9, where we define a city-wide subway network on a graph and focus on the structured time-series passenger flow. The model for constructing the flow of people mobile network graph is as follows:

G _t =(V _t , ε, W)

G _t is the flow of people moving network graph at time t, is a graph composed of multiple nodes, V _t is a limited set of nodes, representing the vertices in the graph, used to monitor the flow of people at each node, namely V _t is a vector composed of the traffic flow of all nodes, ε represents the set of edges between vertices, and W showing the connectivity between vertices is the weight of the adjacency matrix.

When predicting the flow of people at each vertex, the historical data of the previous t moments are given to predict one or more moments in the future. This article predicts the flow of people. Given the historical data from time t-M+1 to t, to predict the flow of people from t+1 to t+H, the target model for predicting the flow of people is constructed as follows:

In the formula,

is the predicted feature vector from time t+1 to time t+H (total inbound/outbound traffic), v _t-M+1 ,...,v _t represents the input features from time -M+1 to time t vector (total in/out foot traffic). It should be noted that v _t+1 ,...,v _t+H is also the predicted feature vector from time t+1 to time t+H,

A mathematical expression used to distinguish predictors.

It is easy to understand that if the input data is the total inbound traffic of a certain station from t-M+1 to t, the predicted feature vector obtained is also the total inbound traffic in a specified time period in the future; the same is true. If the input data is the total outbound traffic of a site from time t-M+1 to t, the predicted feature vector obtained is also the total outbound traffic in a specified time period in the future.

For the graph neural network model, this paper directly uses the graph structure data to extract high-order features in the spatial domain, using the Chebyshev polynomial approximation. The Chebyshev graph convolution formula is as follows:

The model framework used is the STGCN framework, which consists of multiple spatiotemporal convolution modules, each of which is structured like a sandwich (as shown in Figure 10), with two gated sequence convolution layers and a spatial graph convolution in the middle. module. Temporal Gated-Conv is used to capture temporal correlation, which consists of a 1-D Conv and a gated linear unit GLU; Spatial Graph-Conv is used to capture spatial correlation, mainly composed of the above-mentioned Chebyshev graph convolution module.

After obtaining the total inbound and outbound predicted traffic of each site, the total predicted traffic is allocated according to the proportional mean value corresponding to each entry and exit of the site. The mean value of the proportion of inbound traffic at each entrance and exit of a station is calculated based on the inbound traffic at each entrance and exit in the corresponding time period; similarly, the average proportion of the outbound traffic at each entrance and exit of a site is calculated according to the corresponding time period at each entrance and exit. The outbound traffic within the . When assigning the total predicted flow of people, it is also based on the mean of the inbound proportion (ie, the mean of the proportion of the inbound traffic), and based on the mean of the outgoing proportion (that is, the mean of the proportion of the outbound traffic), so as to obtain traceability. The prediction results are guaranteed to have practical application value.

The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction between the combinations of these technical features, All should be regarded as the scope described in this specification.

The above-mentioned embodiments only represent several embodiments of the present invention, and the descriptions thereof are more specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can also be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

A subway pedestrian flow network fusion method based on video pedestrian recognition is used to realize the fusion statistics of subway and ground pedestrian flow to assist traffic early warning. It is characterized in that, the subway pedestrian flow network fusion method based on video pedestrian recognition includes:

Step 1. Receive monitoring images of various entrances and exits of the subway station, and the monitoring images are acquired by image acquisition equipment disposed at each entrance and exit;

Step 2, extracting pedestrian target coordinate frame information and pedestrian target feature information in the monitoring image, where the pedestrian target feature information includes pedestrian features, pedestrian entry and exit status, and pedestrian exit or entry direction;

Step 3. Based on the monitoring image of the same image acquisition device, the similarity calculation is performed according to the pedestrian target coordinate frame information and the pedestrian target feature information, and the pedestrian trajectory for the same pedestrian is obtained;

Step 4. Perform similarity matching of pedestrian target feature information based on the pedestrian trajectories corresponding to different image acquisition devices, and combine the successfully matched pedestrian trajectories to update the pedestrian trajectories of the corresponding pedestrians;

Step 5. Obtain the subway routes, subway stations, entrances and exits of each station in the designated area, and the traffic routes on the ground corresponding to the entrances and exits, and integrate and construct a subway traffic network map in the designated area;

Step 6. According to the latest pedestrian trajectories in the preset time period, count the total inbound and outbound pedestrian flow of each station, and the inbound and outbound pedestrian flow on the traffic routes corresponding to each entrance and exit in each station;

Step 7. On the basis of the subway traffic network map, superimpose the total inbound and outbound traffic of each station of the subway station and the inbound and outbound traffic on the traffic routes corresponding to each entrance and exit in each station to obtain the subway and ground traffic Converged people flow mobile network diagram.
The subway pedestrian flow network fusion method based on video pedestrian recognition according to claim 1, characterized in that, the similarity calculation is performed based on the monitoring image of the same image acquisition device according to the pedestrian target coordinate frame information and the pedestrian target feature information to obtain Pedestrian trajectories for the same pedestrian, including:

Step 3.1, obtain the pedestrian target coordinate frame information and the pedestrian target feature information in the current monitoring image of the current image acquisition device;

Step 3.2: Determine whether the tracking track set corresponding to the image acquisition device is empty. The tracking track set is used to save the pedestrian track of the pedestrian. If the tracking track set is not empty, perform step 3.3; otherwise, directly use the pedestrian obtained this time. The target coordinate frame information and the pedestrian target feature information are added to the tracking trajectory set and ended;

Step 3.3, using unscented Kalman filtering to obtain the estimated target coordinate frame information based on the pedestrian trajectory in the tracking trajectory set;

Step 3.4. According to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculate the coordinate frame similarity between the current pedestrian target and the saved pedestrian target one by one, and the pedestrian target feature information based on the pedestrian trajectory and the current monitoring image. The feature information of the pedestrian target in the calculation is to calculate the feature similarity between the current pedestrian target and the saved pedestrian target one by one, and the current pedestrian target and the saved pedestrian target are obtained based on the weighted summation of the coordinate frame similarity and the feature similarity. similarity between the two;

Step 3.5. Based on the similarity between the current pedestrian target and the saved pedestrian target, the Hungarian matching algorithm is used to match the pedestrian trajectory in the tracking trajectory set and the pedestrian target obtained this time;

Step 3.6. If there is a pedestrian target that has not been successfully matched this time, directly add the pedestrian target coordinate frame information and pedestrian target feature information corresponding to the pedestrian target to the tracking trajectory set, and mark it as a new trajectory; if the pedestrian trajectory and the pedestrian target are successfully matched. If the pedestrian target is a pedestrian target, the pedestrian trajectory of the pedestrian is updated according to the pedestrian target coordinate frame information corresponding to the pedestrian target and the pedestrian target feature information; If there are pedestrian trajectories that have not been successfully matched for multiple consecutive frames in the tracking trajectory set, it is considered that the pedestrian target has left the monitoring range of the current image acquisition device, and the pedestrian trajectory is marked as leaving trajectory; if marked as leaving If the pedestrian trajectory of the trajectory is not successfully matched within the specified time threshold, the pedestrian trajectory is considered to be complete, and the pedestrian trajectory is deleted from the tracking trajectory set.
The subway pedestrian flow network fusion method based on video pedestrian recognition according to claim 2, wherein the similarity matching of pedestrian target feature information is performed based on pedestrian trajectories corresponding to different image acquisition devices, and the pedestrian trajectories that are successfully matched are compared. Jointly, update the pedestrian trajectory of the corresponding pedestrian, including:

Step 4.1. Take a set of tracking trajectories corresponding to an image acquisition device, and compare the pedestrian trajectories marked as new trajectories in the set of tracking trajectories with the pedestrian trajectories marked as departure trajectories in the tracking trajectories set corresponding to the rest of the image acquisition devices one by one. calculate;

Step 4.2. If the similarity is greater than the preset threshold, it means that the two pedestrian trajectories are successfully matched;

Step 4.3: Combine the two successfully matched pedestrian trajectories to obtain a new pedestrian trajectory of the pedestrian, and use the new pedestrian trajectory to replace the corresponding pedestrian trajectory in the tracking trajectory set where the new trajectory is located.
A people flow prediction method for predicting people flow based on the fusion of subway and ground people flow to assist traffic early warning, characterized in that the people flow prediction method comprises:

Using the subway pedestrian flow network fusion method based on video pedestrian recognition to obtain the pedestrian flow network map in the specified time period;

Using graph neural network to predict the total inbound and outbound traffic of each site within a specified time period in the future based on the people flow mobile network map;

Based on the inbound and outbound traffic on the traffic route corresponding to each entrance and exit of each site in the pedestrian flow network diagram, the mean value of the inbound and outbound proportions of the traffic route corresponding to each entry and exit of each site is obtained;

According to the mean value of inbound and outbound proportions, the total inbound and outbound predicted people flow of each station is allocated to obtain the inbound and outbound predicted people flow on the traffic route corresponding to each entrance and exit of each station.
The method for predicting people flow as claimed in claim 4, wherein the use of the subway people flow network fusion method based on video pedestrian recognition to obtain the people flow mobile network map in a specified time period, comprising:

Step 1. Receive monitoring images of various entrances and exits of the subway station, and the monitoring images are acquired by image acquisition equipment disposed at each entrance and exit;

Step 2, extracting pedestrian target coordinate frame information and pedestrian target feature information in the monitoring image, where the pedestrian target feature information includes pedestrian features, pedestrian entry and exit status, and pedestrian exit or entry direction;

Step 3, based on the monitoring image of the same image acquisition device, perform similarity calculation according to the pedestrian target coordinate frame information and the pedestrian target feature information, and obtain the pedestrian trajectory for the same pedestrian;

Step 4. Perform similarity matching of pedestrian target feature information based on the pedestrian trajectories corresponding to different image acquisition devices, and combine the successfully matched pedestrian trajectories to update the pedestrian trajectories of the corresponding pedestrians;

Step 5. Obtain the subway routes, subway stations, entrances and exits of each station in the designated area, and the traffic routes on the ground corresponding to the entrances and exits, and integrate and construct a subway traffic network map in the designated area;

Step 6. According to the latest pedestrian trajectories in the preset time period, count the total inbound and outbound pedestrian flow of each station, and the inbound and outbound pedestrian flow on the traffic routes corresponding to each entrance and exit in each station;

Step 7. On the basis of the subway traffic network map, superimpose the total inbound and outbound traffic of each station of the subway station and the inbound and outbound traffic on the traffic routes corresponding to each entrance and exit in each station to obtain the subway and ground traffic Converged people flow mobile network diagram.
The method for predicting pedestrian flow according to claim 5, characterized in that, based on the monitoring image of the same image acquisition device, similarity calculation is performed according to the pedestrian target coordinate frame information and the pedestrian target feature information to obtain the pedestrian trajectory for the same pedestrian, include:

Step 3.1, obtain the pedestrian target coordinate frame information and the pedestrian target feature information in the current monitoring image of the current image acquisition device;

Step 3.2: Determine whether the tracking track set corresponding to the image acquisition device is empty. The tracking track set is used to save the pedestrian track of the pedestrian. If the tracking track set is not empty, perform step 3.3; otherwise, directly use the pedestrian obtained this time. The target coordinate frame information and the pedestrian target feature information are added to the tracking trajectory set and ended;

Step 3.3, using unscented Kalman filtering to obtain the estimated target coordinate frame information based on the pedestrian trajectory in the tracking trajectory set;

Step 3.4. According to the pedestrian target coordinate frame information and the estimated target coordinate frame information, calculate the coordinate frame similarity between the current pedestrian target and the saved pedestrian target one by one, and the pedestrian target feature information based on the pedestrian trajectory and the current monitoring image. The feature information of the pedestrian target in the calculation is to calculate the feature similarity between the current pedestrian target and the saved pedestrian target one by one, and the current pedestrian target and the saved pedestrian target are obtained based on the weighted summation of the coordinate frame similarity and the feature similarity. similarity between the two;

Step 3.5. Based on the similarity between the current pedestrian target and the saved pedestrian target, the Hungarian matching algorithm is used to match the pedestrian trajectory in the tracking trajectory set and the pedestrian target obtained this time;

Step 3.6. If there is a pedestrian target that has not been successfully matched this time, directly add the pedestrian target coordinate frame information and pedestrian target feature information corresponding to the pedestrian target to the tracking trajectory set, and mark it as a new trajectory; if the pedestrian trajectory and the pedestrian target are successfully matched. If the pedestrian target is a pedestrian target, the pedestrian trajectory of the pedestrian is updated according to the pedestrian target coordinate frame information corresponding to the pedestrian target and the pedestrian target feature information; If there are pedestrian trajectories that have not been successfully matched for multiple consecutive frames in the tracking trajectory set, it is considered that the pedestrian target has left the monitoring range of the current image acquisition device, and the pedestrian trajectory is marked as leaving trajectory; if marked as leaving If the pedestrian trajectory of the trajectory is not successfully matched within the specified time threshold, the pedestrian trajectory is considered to be complete, and the pedestrian trajectory is deleted from the tracking trajectory set.
The pedestrian flow prediction method according to claim 6, wherein the similarity matching of pedestrian target feature information is performed based on pedestrian trajectories corresponding to different image acquisition devices, and the successfully matched pedestrian trajectories are combined to update the pedestrian corresponding to the pedestrian. trajectories, including:

Step 4.1. Take a set of tracking trajectories corresponding to an image acquisition device, and perform similarity between the pedestrian trajectories marked as new trajectories in the set of tracking trajectories and the pedestrian trajectories marked as departure trajectories in the set of tracking trajectories corresponding to the rest of the image acquisition devices one by one. calculate;

Step 4.2. If the similarity is greater than the preset threshold, it means that the two pedestrian trajectories are successfully matched;

Step 4.3: Combine the two successfully matched pedestrian trajectories to obtain a new pedestrian trajectory of the pedestrian, and use the new pedestrian trajectory to replace the corresponding pedestrian trajectory in the tracking trajectory set where the new trajectory is located.
The method for predicting people flow according to claim 4, characterized in that, using a graph neural network to predict the total inbound and outbound people flow of each site within a specified time period in the future based on the people flow mobile network graph, comprising:

The people flow network diagram takes subway stations as vertices and the traffic routes corresponding to the entrances and exits of the stations as edges, each vertex has a feature vector including the total inbound and outbound people flow, and the model for constructing the people flow mobile network diagram is as follows :

G t =(V t , ε, W)

In the formula, G t is the people flow network graph at time t, V t is the vector composed of the feature vectors of all vertices, ε represents the edge set between vertices, W is the weight of the adjacency matrix, and t is the current moment;

When predicting the flow of people at each vertex, based on the feature vector of the vertex in the historical time period t-M+1 to t, predict the feature vector of the specified time period t+1 to t+H in the future, where M and H are preset coefficients, and the target model for predicting the flow of people is constructed as follows:

In the formula,
is the predicted feature vector from time t+1 to time t+H, v t-M+1 ,...,v t represents the input feature vector from time -M+1 to time t;

Based on the constructed people flow prediction target model, a graph neural network is used to solve the people flow prediction target model, and the total inbound and outbound predicted people flow of each site in a specified time period in the future is obtained.