WO2022126668A1 - Method for pedestrian identification in public places and human flow statistics system - Google Patents

Abstract

A method for pedestrian identification in public places and a human flow statistics system. The method comprises: acquiring an optical image, detecting a pedestrian in the optical image, and outputting a three-dimensional bounding box and a corresponding time stamp for the pedestrian; acquiring pedestrian features on the basis of the optical image and the three-dimensional bounding box; performing pedestrian identification on the basis of the pedestrian features in a historical feature library; according to a matching result at said time and a historical matching result, labeling the pedestrian status of the pedestrian that corresponds to the three-dimensional bounding box as an initial matching success, failure, re-matching success after failure, continuous matching success, or moved out of a camera range. By taking into consideration the apparent features, three-dimensional motion features and motion mode of a pedestrian, the method accurately identifies the pedestrian and acquires the time and position at which the pedestrian enters or exits a statistics range and the movement trajectory within the statistics range. On the basis of the method, a human flow statistics system is provided to accurately perform statistics on the people flow amount exiting or entering the statistics range within a unit of time.

Description

A method for identifying pedestrians in public places and a people flow statistics system technical field

The invention belongs to the technical field of computer vision, and in particular relates to a pedestrian identification method and a people flow statistics system in a public place.

Background technique

People flow statistics involve pedestrian identification, pedestrian stay time in the statistical area, and entry and exit trajectories. Currently, the commonly used statistical methods include base station-based statistical methods, such as Bluetooth base stations, 4G base stations, etc., but the positioning accuracy of the above methods is not accurate enough; The statistical methods of optical imaging equipment, such as the statistical methods of infrared arrays and millimeter-wave radars, the above methods have relatively high positioning accuracy, but cannot accurately identify pedestrians, which is likely to cause repeated statistics; there are statistical methods based on optical imaging equipment, such as cameras, positioning It has high accuracy and can accurately identify pedestrians, but there is the problem of pedestrians being occluded. Some of them are based on the statistical method of pedestrian re-identification, and there are also repeated statistical problems caused by the discrepancy between the pedestrian movement pattern and the filtering predicted trajectory.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a pedestrian identification method and a people flow statistics system in a public place, which can accurately identify pedestrians and have a high accuracy rate of people flow statistics.

To achieve the above object, the technical scheme adopted by the present invention is:

A pedestrian identification method in a public place, the method for pedestrian identification in a public place, comprising:

Step 1. Obtain an optical image, detect pedestrians in the optical image, and output the three-dimensional bounding box of the pedestrian and the corresponding timestamp;

Step 2. Obtain pedestrian features based on optical images and 3D bounding boxes, including:

Step 2.1, extract the human body shape and features of the pedestrians in the optical image as the pedestrian appearance features of each pedestrian, and save them in the historical feature database;

Step 2.2, based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to the time series in the historical feature library, extract the three-dimensional motion feature of each pedestrian, and save it in the historical feature library;

Step 2.3, based on the pedestrian 3D motion feature and the pedestrian 3D motion feature within the specified time in the historical feature library, predict the pedestrian 3D motion feature at the next moment, and save it in the historical feature library;

Step 3. Perform pedestrian recognition based on the pedestrian features in the historical feature library, including:

Step 3.1. Calculate the apparent feature distance one by one based on the current pedestrian apparent features and the historical pedestrian apparent features of each pedestrian in the historical feature database. If the apparent feature distance is greater than the apparent threshold, determine the current pedestrian apparent feature and history The pedestrian apparent features in the feature library belong to the same pedestrian, and the current apparent feature distance is determined as the pedestrian's apparent feature distance;

Step 3.2. Calculate the spatial feature distances one by one based on the current three-dimensional motion features of pedestrians and the three-dimensional motion features of pedestrians at the next moment predicted from the previous moment of each pedestrian in the historical feature database. If the spatial feature distance is greater than the spatial threshold, determine the current The three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted in the historical feature database belong to the same pedestrian, and the current spatial feature distance is determined as the spatial feature distance of the pedestrian;

Step 3.3. Based on the current three-dimensional motion features of pedestrians, apparent feature distances, spatial feature distances, and the three-dimensional motion features of each pedestrian history in the historical feature database, determine whether it conforms to the motion pattern of the same pedestrian, and output the motion pattern matching degree as the Pedestrian motion pattern matching degree;

Step 3.4: Perform weighted calculation on the apparent feature distance, spatial feature distance, and motion pattern matching degree belonging to the same pedestrian, and obtain the matching result between the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical feature database, and the matching result includes a successful matching or a matching result. If the matching fails, the pedestrian information obtained by the matching is also included when the matching is successful;

Step 4: Mark the pedestrian status of the pedestrian corresponding to the three-dimensional bounding box according to the current matching results and the historical matching results as initial matching successful, lost, successful re-matching after loss, successful continuous matching or out of the camera range.

Several optional methods are also provided below, which are not intended to be additional limitations on the above-mentioned overall solution, but are merely further additions or optimizations. On the premise of no technical or logical contradiction, each optional method can be independently implemented for the above-mentioned overall solution. The combination can also be a combination between multiple optional ways.

Preferably, the pedestrian in the optical image is detected, and a three-dimensional bounding box of the pedestrian is output, including:

The camera used to obtain the optical image is calibrated, and the mapping relationship between the pixel in the optical image and the distance of the camera is obtained;

Detect pedestrians in optical images, and obtain two-dimensional bounding boxes of pedestrians in optical images;

Based on the 2D bounding box and the mapping relationship, the 3D bounding box of the pedestrian is obtained.

Preferably, the three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to the time series in the historical feature library, including:

Step 2.2.1. Extraction of direction vector: extract the movement direction of pedestrians in the horizontal direction and the movement direction in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

Step 2.2.2. Movement speed extraction: extract the movement speed of the person in the horizontal direction and the movement speed in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

Step 2.2.3. Relative position extraction: According to the mapping relationship obtained after the camera is calibrated, the coordinates of the pedestrian in the three-dimensional coordinate system centered on the camera are output based on the current three-dimensional bounding box and the historical three-dimensional bounding box;

Step 2.2.4. Use the direction vector, motion speed and relative position extracted in steps 2.2.1 to 2.2.3 as the three-dimensional motion feature of the pedestrian.

Preferably, the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box is marked according to the current matching result and the historical matching result as initial matching success, loss, rematching success after loss, continuous matching success or out of the camera range, including:

If the pedestrian feature is successfully extracted, but the matching result is that the matching fails, the status of the current pedestrian is marked as the initial matching success;

If the same pedestrian in the historical matching results is not matched for M consecutive times, the status of the pedestrian is marked as lost;

If the pedestrian marked as lost is successfully re-matched in the current matching result, the status of the pedestrian is updated to be lost and the re-match is successful;

If the same pedestrian in the historical matching result is matched for L consecutive times, the status of the pedestrian is updated to indicate that the continuous matching is successful;

If the same pedestrian in the historical matching result is not matched for N consecutive times, the state of the pedestrian is marked as walking out of the camera range, and M<N.

Preferably, if the status of the current pedestrian is marked as successful for the first time, new pedestrian information is allocated to the pedestrian in the historical feature database, and the pedestrian characteristic of the pedestrian is associated with the newly allocated pedestrian information.

The present invention also provides a people flow statistics system, and the people flow statistics system includes:

The pedestrian detection module is used to obtain optical images, detect pedestrians in the optical images, and output the three-dimensional bounding boxes of pedestrians and corresponding timestamps;

The feature extraction module is used to obtain pedestrian features based on optical images and 3D bounding boxes. The specific steps are as follows:

a. Extract the human body shape and features of the pedestrians in the optical image as the pedestrian appearance features of each pedestrian, and save them in the historical feature database;

b. Based on the current 3D bounding box of the pedestrian and the 3D bounding box distributed according to the time series in the historical feature library, extract the three-dimensional motion feature of each pedestrian and save it in the historical feature library;

c. Predict the 3D motion feature of the pedestrian at the next moment based on the 3D motion feature of the pedestrian and the 3D motion feature of the pedestrian within the specified time in the historical feature library, and save it in the historical feature library;

The pedestrian recognition module is used for pedestrian recognition based on the pedestrian features in the historical feature database, and the specific steps are as follows:

a. Calculate the apparent feature distance one by one based on the current pedestrian apparent feature and the historical feature of each pedestrian in the historical feature database. If the apparent feature distance is greater than the apparent threshold, then judge the current pedestrian apparent feature and historical feature The pedestrian apparent features in the library belong to the same pedestrian, and the current apparent feature distance is determined as the pedestrian's apparent feature distance;

b. Calculate the spatial feature distances one by one based on the current three-dimensional motion features of pedestrians and the three-dimensional motion features of pedestrians at the next moment predicted from the previous moment of each pedestrian in the historical feature database. If the spatial feature distance is greater than the spatial threshold, the current pedestrian is judged The three-dimensional motion feature and the three-dimensional motion feature of the pedestrian at the next moment predicted at the previous moment in the historical feature library belong to the same pedestrian, and the current spatial feature distance is determined as the spatial feature distance of the pedestrian;

c. Judging whether it conforms to the motion pattern of the same pedestrian based on the current three-dimensional motion features, apparent feature distance, spatial feature distance, and the three-dimensional motion features of each pedestrian in the historical feature database, and output the motion pattern matching degree as the pedestrian Motion pattern matching degree;

d. Perform weighted calculation on the apparent feature distance, spatial feature distance, and motion pattern matching degree belonging to the same pedestrian, and obtain the matching result between the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical feature database, and the matching result includes matching success or matching. If the match fails, the pedestrian information obtained by the match is also included when the match is successful;

The pedestrian marking module is used to mark the pedestrian status of the pedestrian corresponding to the three-dimensional bounding box according to the current matching results and the historical matching results as initial matching success, loss, rematching success after loss, continuous matching success or out of the camera range;

The people flow statistics module is used to count the flow of people within the statistical range corresponding to the optical image within the preset time according to the pedestrian state.

Preferably, the pedestrian in the optical image is detected, and the three-dimensional bounding box of the pedestrian is output, and the following operations are performed:

The camera used to obtain the optical image is calibrated, and the mapping relationship between the pixel in the optical image and the distance of the camera is obtained;

Detect pedestrians in optical images, and obtain two-dimensional bounding boxes of pedestrians in optical images;

Based on the 2D bounding box and the mapping relationship, the 3D bounding box of the pedestrian is obtained.

As preferably, the three-dimensional bounding box based on the current three-dimensional bounding box of pedestrians and the three-dimensional bounding box distributed according to time series in the historical feature library, extract the three-dimensional motion feature of pedestrians of each pedestrian, and perform the following operations:

Direction vector extraction: Through the current 3D bounding box and the historical 3D bounding box, the pedestrian's movement direction in the horizontal direction and the movement direction in the vertical direction are extracted;

Movement speed extraction: extract the movement speed of the person in the horizontal direction and the movement speed in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

Relative position extraction: According to the mapping relationship obtained after the camera is calibrated, the coordinates of the pedestrian in the three-dimensional coordinate system centered on the camera are output based on the current three-dimensional bounding box and the historical three-dimensional bounding box;

Feature integration: The extracted direction vector, motion speed and relative position are used as pedestrian 3D motion features.

Preferably, according to the current matching results and the historical matching results, the pedestrian states of the pedestrians corresponding to the three-dimensional bounding box are marked as successful initial matching, lost, successful re-matching after loss, successful continuous matching, or out of the camera range, and the following operations are performed. :

If the pedestrian feature is successfully extracted, but the matching result is that the matching fails, the status of the current pedestrian is marked as the initial matching success;

If the same pedestrian in the historical matching results is not matched for M consecutive times, the status of the pedestrian is marked as lost;

If the pedestrian marked as lost is successfully re-matched in the current matching result, the status of the pedestrian is updated to be lost and the re-match is successful;

If the same pedestrian in the historical matching result is matched for L consecutive times, the status of the pedestrian is updated to indicate that the continuous matching is successful;

If the same pedestrian in the historical matching result is not matched for N consecutive times, the state of the pedestrian is marked as walking out of the camera range, and M<N.

Preferably, if the status of the current pedestrian is marked as successful for the first time, new pedestrian information is allocated to the pedestrian in the historical feature database, and the pedestrian characteristic of the pedestrian is associated with the newly allocated pedestrian information.

The present invention provides a pedestrian identification method in a public place, which comprehensively considers pedestrian appearance features, three-dimensional motion features and motion patterns, accurately identifies pedestrians, obtains the time and position of pedestrians entering and exiting the statistical range, and the movement trajectory within the statistical range, and based on the This method proposes a people flow statistics system, which can accurately count the flow of people entering and leaving the statistical range in unit time.

Description of drawings

Fig. 1 is the flow chart of the pedestrian identification method in public places of the present invention;

Fig. 2 is the flow chart of the present invention outputting the three-dimensional bounding box of pedestrian;

3 is a flow chart of the present invention for acquiring pedestrian features based on an optical image and a three-dimensional bounding box;

4 is a schematic diagram of the present invention extracting motion features by human body structure in a right-handed coordinate system;

Fig. 5 is the flow chart that the present invention carries out pedestrian recognition based on the pedestrian characteristic in the historical characteristic database;

6 is a flow chart of the present invention for marking pedestrian status;

FIG. 7 is a structural diagram of the people flow statistics system of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention.

In one embodiment, a pedestrian identification method in a public place is provided, which can accurately identify pedestrians and can be used for urban planning based on pedestrian identification statistics, business strategy adjustment based on pedestrian flow statistics in shopping malls, and subway shift adjustment based on subway station traffic statistics.

As shown in Figure 1, the method for identifying pedestrians in public places in this embodiment includes the following steps:

Step 1. Obtain an optical image, detect pedestrians in the optical image, and output a three-dimensional bounding box of the pedestrian and a corresponding timestamp.

In this embodiment, the optical image is acquired by the camera, and the obtained time stamp is the time when the optical image is captured by the camera. It is easy to understand that the acquisition of the optical image may be based on any image acquisition device, and this embodiment takes a camera as an example for description.

Since the 3D bounding box has depth information, and the optical image does not contain depth information, this embodiment includes the following steps when forming the 3D bounding box, as shown in Figure 2:

The camera used to obtain the optical image is calibrated, and the mapping relationship between the pixel and the camera distance in the optical image is obtained; the pedestrian in the optical image is detected, and the two-dimensional bounding box (BBox) of the pedestrian in the optical image is obtained; based on the two-dimensional The bounding box and the mapping relationship get the pedestrian's three-dimensional bounding box (3D Bounding Box, 3D BBox).

In this embodiment, the mapping relationship between the pedestrian pixel and the camera distance in the optical image is obtained by calibrating the camera, and the corresponding depth information is obtained based on the mapping relationship. The depth information reflects the actual distance between the pedestrian and the camera, including the movement of the pedestrian. It is convenient to extract the three-dimensional motion features of pedestrians based on the depth information.

In this embodiment, a monocular fixed-focus camera is used to shoot a video, and a cube with a side length of 1 meter is used to calibrate the mapping relationship. Each face of the cube is evenly divided into 100 black and white grids, and the camera shooting range is Statistical range.

It should be noted that the calibration is a conventional technical means used by the camera, and the specific steps of the calibration are not limited in this embodiment, and the depth information is obtained based on the mapping relationship of the camera calibration, which is the preferred method provided in this embodiment, but is not limited to the only means. , for example, the depth information can be superimposed by the combination of the camera and the depth camera.

In this embodiment, pedestrians in the optical image are identified based on the pedestrian detection method and a two-dimensional bounding box is output. The pedestrian detection method used is a conventional method in image recognition, for example, the recognition based on Yolo extension obtained by training on the pedestrian dataset is used. network. When the 3D bounding box is obtained, the 2D bounding box and the mapping relationship are input into the 3D estimation method, and the 3D bounding box is output.

The three-dimensional estimation method used in this embodiment is a monocular depth estimation method based on optical flow, which can output the inverse depth, and the depth information can be obtained by calculating the inverse depth. The depth information has different error coefficients in different ranges from the camera. In this embodiment, an error matrix is used.

Pedestrian recognition based on the 3D bounding box can effectively overcome the pedestrian occlusion problem. Since the human body structure conforms to the geometric constraints, the partially occluded 2D bounding box can be restored to a complete 3D bounding box, and its spatial error is within the allowable range.

Step 2. Obtain pedestrian features based on the optical image and the 3D bounding box, as shown in Figure 3, including:

Step 2.1, extract the human body shape and features of the pedestrians in the optical image as the pedestrian appearance features of each pedestrian, and save them in the historical feature database.

Since the shape and features of the human body are important features to distinguish different pedestrians, this embodiment adopts the pedestrian apparent feature extraction method, mainly extracting the human body traits and features that can be observed visually, and marking the pedestrian apparent feature as F for the convenience of distinction _appearance .

In this embodiment, the embedding structure (embedding, embedding) of the recognition network based on Yolo extension is used as the method for extracting the apparent feature of pedestrians.

Step 2.2, based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to the time series in the historical feature library, extract the three-dimensional motion feature of each pedestrian, and save it in the historical feature library.

The three-dimensional motion feature of pedestrians is the feature of the position change of pedestrians in three-dimensional space. For the convenience of distinction, it is marked as F _displacement , which is an important feature for data association in time series.

As shown in Figure 4, in this embodiment, a right-handed coordinate system is established, and the three-dimensional bounding box is divided into three parts: head, upper body, and lower body according to the human body structure, and the three-dimensional bounding box is input to the pedestrian three-dimensional motion feature extraction method, which mainly extracts people in three-dimensional space. The position change feature in the output is the three-dimensional motion feature of the pedestrian.

In this embodiment, the three-dimensional motion feature extraction method consists of the following parts:

Step 2.2.1. Extraction of direction vector: Extract the moving direction of pedestrians in the horizontal direction and the moving direction in the vertical direction through the current three-dimensional bounding box and the historical three-dimensional bounding box.

Since the 3D bounding box has a corresponding timestamp, the direction vector of the pedestrian can be obtained based on the position change of the 3D bounding box based on the time distribution. For multiple 3D bounding boxes, only the first two bounding boxes can be used to determine the direction vector, or multiple pairs of 3D bounding boxes can be used, and the mean, median or other values of the multiple direction vectors can be taken as the final direction vector.

Step 2.2.2. Movement speed extraction: extract the movement speed of the person in the horizontal direction and the movement speed in the vertical direction through the current 3D bounding box and the historical 3D bounding box.

Similar to the direction vector extraction, the 3D bounding box based on the time distribution obtains the movement speed of the pedestrian according to the time difference and the position difference of the corresponding 3D bounding box, and multiple 3D bounding boxes can only use the first two bounding boxes to calculate the movement speed, It is also possible to use multiple pairs of three-dimensional bounding boxes, and take the mean, median or other values of multiple motion velocities as the final motion velocity.

Step 2.2.3. Relative position extraction: According to the mapping relationship obtained after the camera is calibrated, the coordinates of the pedestrian in the three-dimensional coordinate system centered on the camera are output based on the current three-dimensional bounding box and the historical three-dimensional bounding box.

When determining the coordinates, each three-dimensional bounding box is equivalent to a point, and the coordinates of the point are obtained as the coordinates of the pedestrian. The point can be the center point of the three-dimensional bounding box, a vertex or any point.

Step 2.2.4. Use the direction vector, motion speed and relative position extracted in steps 2.2.1 to 2.2.3 as the three-dimensional motion feature of the pedestrian.

Since there are usually multiple 3D bounding boxes of pedestrians in the historical feature database, when extracting the 3D motion features of pedestrians, first perform feature matching on the current and historical 3D bounding boxes (for example, using the Hungarian matching algorithm), and take the unused and The historical 3D bounding box with the highest matching degree is used to extract the 3D motion features of pedestrians.

If the current 3D bounding box is the 3D bounding box of a new pedestrian entering the statistical range, and the above matching fails to obtain the corresponding historical 3D bounding box, set the direction vector and movement speed of the new pedestrian as default values (for example, The direction vector is none and the transport speed is 0), and the coordinates are the coordinates of the current three-dimensional bounding box as the three-dimensional motion feature of the pedestrian.

Step 2.3: Predict the three-dimensional motion feature of the pedestrian at the next moment based on the three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian in the historical feature library at a specified time, and save the three-dimensional motion feature of the pedestrian in the historical feature library.

The three-dimensional motion feature of the pedestrian at the next moment is marked as F _predicted , which is predicted by the trajectory prediction algorithm and represents the movement trend of the pedestrian in space and time. Using this feature can further solve the problem of target loss caused by the pedestrian being occluded; in this embodiment Use Kalman filtering to predict the three-dimensional motion characteristics of pedestrians at the next moment.

Similar to the extraction of pedestrian 3D motion features, since there are usually multiple pedestrian 3D motion features in the historical feature database, when predicting pedestrian 3D motion features, feature matching is first performed on the current and historical pedestrian 3D motion features (for example, using Hungarian matching algorithm), take the historical pedestrian 3D motion features that are not used and have the highest matching degree to predict pedestrian 3D motion features.

If the current pedestrian 3D motion feature is the pedestrian 3D motion feature of a new pedestrian entering the statistical range, and the above matching fails to obtain the corresponding historical pedestrian 3D motion feature, the prediction is made directly based on the current pedestrian 3D motion feature. Step 3. Perform pedestrian recognition based on the pedestrian features in the historical feature database, as shown in Figure 5, including:

Step 3.1. Calculate the apparent feature distance one by one based on the current pedestrian apparent features and the historical pedestrian apparent features of each pedestrian in the historical feature database. If the apparent feature distance is greater than the apparent threshold, determine the current pedestrian apparent feature and history The pedestrian apparent features in the feature library belong to the same pedestrian, and the apparent feature distance is determined as the pedestrian's apparent feature distance.

In this embodiment, the pedestrian apparent feature matching method is used to calculate the current pedestrian apparent feature

Pedestrian appearance features in historical feature database

To judge whether they belong to the same pedestrian, for example, using the weighted Mahalanobis distance and cosine distance to calculate the apparent feature distance, the coefficients are 0.02 and 0.98 respectively. The historical pedestrian appearance feature in this embodiment mainly adopts the pedestrian appearance feature of the previous moment. In order to improve the matching result, in another embodiment, if the apparent characteristics of pedestrians belonging to the same pedestrian are determined, a list of apparently similar pedestrians can be established, each pedestrian corresponds to a list of apparently similar pedestrians, and the table is further distinguished based on the list. View feature distance.

For example, the current moment has the apparent characteristics of two persons, A and B, where A and A{t-1} (the pedestrian apparent characteristics of the previous moment), B and A{t-1} are also very similar, even The similarity of multiple time periods in the apparently similar pedestrian lists of B and A is high, but the similarity of multiple time periods in the apparently similar pedestrian lists of B and B is higher, then it can be judged that B is B. However, since this search method is time-consuming, it is generally used in scenarios with high requirements for pedestrian recognition.

Step 3.2. Calculate the spatial feature distances one by one based on the current three-dimensional motion features of pedestrians and the three-dimensional motion features of pedestrians at the next moment predicted from the previous moment of each pedestrian in the historical feature database. If the spatial feature distance is greater than the spatial threshold, determine the current The three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted in the historical feature database belong to the same pedestrian, and the current spatial feature distance is determined as the spatial feature distance of the pedestrian.

The three-dimensional motion feature of the pedestrian at the next moment predicted at the previous moment is the predicted three-dimensional motion feature of the current pedestrian. Matching the predicted and the actual current three-dimensional motion feature of the pedestrian can be used as the basis for judging whether they belong to the same pedestrian. One, because the three-dimensional motion characteristics of the same pedestrian will not change too much, so the matching is for reference. In this embodiment, the Hungarian algorithm is used as the pedestrian three-dimensional motion feature matching method for calculation to determine whether they belong to the same pedestrian.

Similar to the list of apparently similar pedestrians, in another embodiment, if three-dimensional motion characteristics of pedestrians belonging to the same pedestrian are determined, a list of spatially similar pedestrians can be established.

Step 3.3. Based on the current three-dimensional motion features of pedestrians, apparent feature distances, spatial feature distances, and the three-dimensional motion features of each pedestrian history in the historical feature database, determine whether it conforms to the motion pattern of the same pedestrian, and output the motion pattern matching degree as the Pedestrian motion pattern matching degree.

For the movement pattern of pedestrians, this embodiment focuses on the change speed of pedestrians in time series and the movement logic in spatial position. The movement logic includes but is not limited to common behaviors such as turning back, staying in place, trotting, and squatting. At the same time, considering the regular movement of pedestrians, in this embodiment, the change speed of the target within 3 seconds and the movement logic of the target in the shooting space are mainly investigated. Since the cameras acquire optical images based on preset intervals, pedestrians with reasonable speed changes can be judged as the same pedestrian.

Input the current pedestrian 3D motion feature and the 3D motion feature, apparent feature distance, and spatial feature distance in the historical feature database into the motion pattern matching method to determine whether the pedestrian behavior conforms to the common motion patterns of pedestrians in public places. If the motion pattern matching degree is less than the motion threshold It is considered that they belong to the same pedestrian, and a list of pedestrians with similar movement patterns is established.

To calculate the matching degree of the pedestrian's motion pattern, the matching degree can be directly output based on the pre-trained neural network, or it can be directly judged by predicting the preset matching rules. The former is relatively flexible in judgment, but needs to be trained based on a large number of samples. The latter can be directly generated and used, and is easy to add, delete and modify, but the flexibility is relatively low, and an appropriate method can be selected according to actual needs.

In one embodiment, based on actual observations and statistics, a matching rule (specific probability values omitted) is established as shown in Table 1, which represents the probability of converting the behavior pattern of the previous stage to the corresponding behavior pattern of the current stage.

Table 1 Probability of transition from the behavior pattern of the previous stage to the corresponding behavior pattern of the current stage

When the motion pattern matching degree is performed based on Table 1, the historical three-dimensional motion features of pedestrians corresponding to the apparent feature distances are taken, and the three-dimensional historical three-dimensional motion features of pedestrians corresponding to the spatial feature distances are taken. If the pedestrian corresponding to the motion feature is not the same pedestrian, this match will be discarded. If it is the same pedestrian, the behavior pattern and current behavior pattern of the pedestrian in the previous stage will be judged according to the obtained historical pedestrian 3D motion feature and the current pedestrian 3D motion feature. The behavior pattern of the stage can be looked up in the table to obtain the probability value as the motion pattern matching degree.

It should be noted that the behavior pattern of a stage is determined by at least two pedestrian three-dimensional motion features. Since a pedestrian three-dimensional motion feature has a direction vector, motion speed and coordinates, the current stage can be determined by the change of the two direction vectors. Going forward, turning back or turning, combined with the coordinate change can further distinguish whether it is going forward or staying, and further distinguish whether it is going forward or accelerating forward combined with the movement speed.

Of course, the above table is a preferred matching rule adopted in this embodiment, which can be further optimized in actual use, for example, the turning is refined into a left turn or a right turn, etc., and the probability values in the table can also be calculated according to the actual use. The probability is updated to improve the pedestrian recognition rate.

Step 3.4. Perform weighted calculation on the apparent feature distance, spatial feature distance, and motion pattern matching degree belonging to the same pedestrian (ie, input it into the weighting calculator), and obtain the matching result between the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical feature database. , the matching result includes matching success or matching failure, and when the matching is successful, it also includes the pedestrian information obtained by the matching.

In this embodiment, the weights of apparent feature distance, spatial feature distance, and motion pattern matching degree are 0.6, 0.2, and 0.2, respectively. Since apparent feature is the most intuitive feature to distinguish different pedestrians, the apparent feature distance is set in this embodiment. has the highest weight. Of course, in actual use, the weight can be adjusted, such as increasing the weight of the motion pattern matching degree, so as to avoid misjudgment caused by two people whose apparent characteristics are too similar.

The matching failure in the final matching result indicates that there is no historical record for the characteristics of the current pedestrian, that is, the pedestrian is a pedestrian who has newly entered the statistical range; and the matching success indicates that the current pedestrian's characteristics have historical records, so the matching pedestrian information is output to correlate. New and historical features of the same pedestrian. Pedestrian information can be a unique identifier (eg, ID value), spatial location, time, and the like.

Step 4: Mark the pedestrian status of the pedestrian corresponding to the three-dimensional bounding box according to the current matching results and the historical matching results as initial matching successful, lost, successful re-matching after loss, successful continuous matching or out of the camera range.

As shown in FIG. 6, a specific matching method provided in this embodiment may be:

If the pedestrian feature is successfully extracted (that is, the recognition is successful at the current moment), but the matching result is a matching failure (that is, there is no historical record), the status of the current pedestrian is marked as the initial matching success;

If M consecutive times (for example, 50 consecutive times (10 seconds*5 times/sec)) do not match the same pedestrian in the historical matching results (for example, the current recognition fails and the previous moment also fails, or the current recognition fails and the continuous recognition fails The number of times is not greater than the threshold), the status of the pedestrian is marked as lost;

If the pedestrian marked as lost is successfully re-matched in the current matching result (for example, the current moment is successfully identified and there is a historical record, but the previous match failed), then the status of the pedestrian is updated to be lost and the re-match is successful;

If the same pedestrian in the historical matching result is matched for L consecutive times (for example, 50 consecutive times (10 seconds*5 times/second)) (for example, the current moment is successfully recognized, there is a historical record, and the previous moment was also successfully matched), update The status of the pedestrian is continuous matching success;

If the same pedestrian in the historical matching results is not matched for N consecutive times (for example, 150 consecutive times (10 seconds*15 times/sec)) (for example, the current recognition fails and the number of consecutive recognition failures is greater than the threshold), mark the status of the pedestrian is out of the camera range, and M<N.

If the status of the current pedestrian is marked as successful for the first time, assign new pedestrian information to the pedestrian in the historical feature database, and associate the pedestrian characteristics of the pedestrian with the newly allocated pedestrian information, which can be used as historical data at the next moment. to identify and track the pedestrian.

As shown in Figure 7, in another implementation, a people flow statistics system is provided, comprising:

The pedestrian detection module is used to obtain optical images, detect pedestrians in the optical images, and output the three-dimensional bounding boxes of pedestrians and corresponding timestamps;

The feature extraction module is used to obtain pedestrian features based on optical images and 3D bounding boxes. The specific steps are as follows:

a. Extract the human body shape and features of the pedestrians in the optical image as the pedestrian appearance features of each pedestrian, and save them in the historical feature database;

b. Based on the current 3D bounding box of the pedestrian and the 3D bounding box distributed according to the time series in the historical feature library, extract the three-dimensional motion feature of each pedestrian and save it in the historical feature library;

c. Predict the 3D motion feature of the pedestrian at the next moment based on the 3D motion feature of the pedestrian and the 3D motion feature of the pedestrian within the specified time in the historical feature library, and save it in the historical feature library;

The pedestrian recognition module is used for pedestrian recognition based on the pedestrian features in the historical feature database, and the specific steps are as follows:

a. Calculate the apparent feature distance one by one based on the current pedestrian apparent feature and the historical feature of each pedestrian in the historical feature database. If the apparent feature distance is greater than the apparent threshold, then judge the current pedestrian apparent feature and historical feature The pedestrian apparent features in the library belong to the same pedestrian, and the current apparent feature distance is determined as the pedestrian's apparent feature distance;

b. Calculate the spatial feature distances one by one based on the current three-dimensional motion features of pedestrians and the three-dimensional motion features of pedestrians at the next moment predicted from the previous moment of each pedestrian in the historical feature database. If the spatial feature distance is greater than the spatial threshold, the current pedestrian is judged The three-dimensional motion feature and the three-dimensional motion feature of the pedestrian at the next moment predicted at the previous moment in the historical feature library belong to the same pedestrian, and the current spatial feature distance is determined as the spatial feature distance of the pedestrian;

c. Judging whether it conforms to the motion pattern of the same pedestrian based on the current three-dimensional motion features, apparent feature distance, spatial feature distance, and the three-dimensional motion features of each pedestrian in the historical feature database, and output the motion pattern matching degree as the pedestrian Motion pattern matching degree;

d. Perform weighted calculation on the apparent feature distance, spatial feature distance, and motion pattern matching degree belonging to the same pedestrian, and obtain the matching result between the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical feature database, and the matching result includes matching success or matching. If the match fails, the pedestrian information obtained by the match is also included when the match is successful;

The pedestrian marking module is used to mark the pedestrian status of the pedestrian corresponding to the three-dimensional bounding box according to the current matching results and the historical matching results as initial matching success, loss, rematching success after loss, continuous matching success or out of the camera range;

The people flow statistics module is used to count the flow of people within the statistical range corresponding to the optical image within the preset time according to the pedestrian state.

For the specific limitations in the people flow statistics system, please refer to the above-mentioned specific limitations on the pedestrian identification method in public places, which will not be repeated here.

In a preferred embodiment, the detection of pedestrians in the optical image, outputting a three-dimensional bounding box of the pedestrians, performs the following operations:

The camera used to obtain the optical image is calibrated, and the mapping relationship between the pixel in the optical image and the distance of the camera is obtained;

Detect pedestrians in optical images, and obtain two-dimensional bounding boxes of pedestrians in optical images;

Based on the 2D bounding box and the mapping relationship, the 3D bounding box of the pedestrian is obtained.

In this embodiment, the pedestrian detection module has a camera calibration function and a parameter management function. In other embodiments, the camera calibration device can be used as a part independent of the people flow statistics system in this embodiment, and the calibrated external parameters of the equipment and The mapping relationship may be sent to the parameter management module of the people flow statistics system in this embodiment.

It should be noted that this embodiment is based on optical image process people flow statistics, that is, the people flow statistics system further includes a video acquisition module, the video acquisition module is connected with the video acquisition device of the peripheral equipment, and after acquiring the real-time video within the statistical range, each The optical picture of the frame is sent to the pedestrian detection module.

In another embodiment, the three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to the time series in the historical feature library, and the following operations are performed:

Direction vector extraction: Through the current 3D bounding box and the historical 3D bounding box, the pedestrian's movement direction in the horizontal direction and the movement direction in the vertical direction are extracted;

Movement speed extraction: Extract the movement speed of the person in the horizontal direction and the movement speed in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

Relative position extraction: According to the mapping relationship obtained after the camera is calibrated, the coordinates of the pedestrian in the three-dimensional coordinate system centered on the camera are output based on the current three-dimensional bounding box and the historical three-dimensional bounding box;

Feature integration: The extracted direction vector, motion speed and relative position are used as pedestrian 3D motion features.

In another embodiment, the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box is marked according to the current matching result and the historical matching result as successful initial matching, lost, successful re-matching after loss, successful continuous matching, or out of camera range , do the following:

If the pedestrian feature is successfully extracted, but the matching result is that the matching fails, the status of the current pedestrian is marked as the initial matching success;

If the same pedestrian in the historical matching results is not matched for M consecutive times, the status of the pedestrian is marked as lost;

If the pedestrian marked as lost is successfully re-matched in the current matching result, the status of the pedestrian is updated to be lost and the re-match is successful;

If the same pedestrian in the historical matching result is matched for L consecutive times, the status of the pedestrian is updated to indicate that the continuous matching is successful;

If the same pedestrian in the historical matching result is not matched for N consecutive times, the state of the pedestrian is marked as walking out of the camera range, and M<N.

In another embodiment, if the status of the current pedestrian is marked as successful for the first match, new pedestrian information is allocated to the pedestrian in the historical feature database, and the pedestrian characteristic of the pedestrian is associated with the newly allocated pedestrian information.

The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction between the combinations of these technical features, All should be regarded as the scope described in this specification.

The above-mentioned embodiments only represent several embodiments of the present invention, and the descriptions thereof are more specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can also be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims (10)

1. A method for pedestrian identification in a public place, characterized in that the method for pedestrian identification in a public place comprises:

   Step 1. Obtain an optical image, detect pedestrians in the optical image, and output the three-dimensional bounding box of the pedestrian and the corresponding timestamp;

   Step 2. Obtain pedestrian features based on optical images and 3D bounding boxes, including:

   Step 2.1, extract the human body shape and features of the pedestrians in the optical image as the pedestrian appearance features of each pedestrian, and save them in the historical feature database;

   Step 2.2, based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to the time series in the historical feature library, extract the three-dimensional motion feature of each pedestrian, and save it in the historical feature library;

   Step 2.3, based on the pedestrian 3D motion feature and the pedestrian 3D motion feature within the specified time in the historical feature library, predict the pedestrian 3D motion feature at the next moment, and save it in the historical feature library;

   Step 3. Perform pedestrian recognition based on the pedestrian features in the historical feature library, including:

   Step 3.1. Calculate the apparent feature distance one by one based on the current pedestrian apparent features and the historical pedestrian apparent features of each pedestrian in the historical feature database. If the apparent feature distance is greater than the apparent threshold, determine the current pedestrian apparent feature and history The pedestrian apparent features in the feature library belong to the same pedestrian, and the current apparent feature distance is determined as the pedestrian's apparent feature distance;

   Step 3.2. Calculate the spatial feature distances one by one based on the current three-dimensional motion features of pedestrians and the three-dimensional motion features of pedestrians at the next moment predicted from the previous moment of each pedestrian in the historical feature database. If the spatial feature distance is greater than the spatial threshold, determine the current The three-dimensional motion feature of the pedestrian and the three-dimensional motion feature of the pedestrian at the next moment predicted in the historical feature database belong to the same pedestrian, and the current spatial feature distance is determined as the spatial feature distance of the pedestrian;

   Step 3.3. Based on the current three-dimensional motion features of pedestrians, apparent feature distances, spatial feature distances, and the three-dimensional motion features of each pedestrian history in the historical feature database, determine whether it conforms to the motion pattern of the same pedestrian, and output the motion pattern matching degree as the Pedestrian motion pattern matching degree;

   Step 3.4: Perform weighted calculation on the apparent feature distance, spatial feature distance, and motion pattern matching degree belonging to the same pedestrian, and obtain the matching result between the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical feature database, and the matching result includes a successful matching or a matching result. If the matching fails, the pedestrian information obtained by the matching is also included when the matching is successful;

   Step 4. Mark the pedestrian status of the pedestrian corresponding to the three-dimensional bounding box according to the current matching results and the historical matching results as initial matching success, loss, re-matching success after loss, continuous matching success or out of the camera range.
2. The method for pedestrian identification in a public place according to claim 1, wherein the detecting a pedestrian in an optical image and outputting a three-dimensional bounding box of the pedestrian comprises:

   The camera used to obtain the optical image is calibrated, and the mapping relationship between the pixel in the optical image and the distance of the camera is obtained;

   Detect pedestrians in optical images, and obtain two-dimensional bounding boxes of pedestrians in optical images;

   Based on the 2D bounding box and the mapping relationship, the 3D bounding box of the pedestrian is obtained.
3. The method for pedestrian identification in a public place according to claim 2, wherein the three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to time series in the historical feature library, include:

   Step 2.2.1. Extraction of direction vector: extract the movement direction of pedestrians in the horizontal direction and the movement direction in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

   Step 2.2.2. Movement speed extraction: extract the movement speed of the person in the horizontal direction and the movement speed in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

   Step 2.2.3. Relative position extraction: According to the mapping relationship obtained after the camera is calibrated, the coordinates of the pedestrian in the three-dimensional coordinate system centered on the camera are output based on the current three-dimensional bounding box and the historical three-dimensional bounding box;

   Step 2.2.4. Use the direction vector, motion speed and relative position extracted in steps 2.2.1 to 2.2.3 as the three-dimensional motion feature of the pedestrian.
4. The method for pedestrian identification in a public place according to claim 1, wherein the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box is marked according to the current matching result and the historical matching results as successful initial matching, loss, and re-setting after loss. Successful match, continuous match, or out of camera range, including:

   If the pedestrian feature is successfully extracted, but the matching result is that the matching fails, the status of the current pedestrian is marked as the initial matching success;

   If the same pedestrian in the historical matching results is not matched for M consecutive times, the status of the pedestrian is marked as lost;

   If the pedestrian marked as lost is successfully re-matched in the current matching result, the status of the pedestrian is updated to be lost and the re-match is successful;

   If the same pedestrian in the historical matching result is matched for L consecutive times, the status of the pedestrian is updated to indicate that the continuous matching is successful;

   If the same pedestrian in the historical matching result is not matched for N consecutive times, the state of the pedestrian is marked as walking out of the camera range, and M<N.
5. The pedestrian identification method in a public place according to claim 1, wherein if the status of the current pedestrian is marked as successful for the first time, new pedestrian information is allocated to the pedestrian in the historical feature database, and the pedestrian feature of the pedestrian is assigned to the pedestrian. Associated with the newly assigned pedestrian information.
6. A people flow statistics system, characterized in that the people flow statistics system includes:

   The pedestrian detection module is used to obtain optical images, detect pedestrians in the optical images, and output the three-dimensional bounding boxes of pedestrians and corresponding timestamps;

   The feature extraction module is used to obtain pedestrian features based on optical images and 3D bounding boxes. The specific steps are as follows:

   a. Extract the human body shape and features of the pedestrians in the optical image as the pedestrian appearance features of each pedestrian, and save them in the historical feature database;

   b. Based on the current 3D bounding box of the pedestrian and the 3D bounding box distributed according to the time series in the historical feature library, extract the three-dimensional motion feature of each pedestrian and save it in the historical feature library;

   c. Predict the 3D motion feature of the pedestrian at the next moment based on the 3D motion feature of the pedestrian and the 3D motion feature of the pedestrian within the specified time in the historical feature library, and save it in the historical feature library;

   The pedestrian recognition module is used for pedestrian recognition based on the pedestrian features in the historical feature database, and the specific steps are as follows:

   a. Calculate the apparent feature distance one by one based on the current pedestrian apparent feature and the historical feature of each pedestrian in the historical feature database. If the apparent feature distance is greater than the apparent threshold, then judge the current pedestrian apparent feature and historical feature The pedestrian apparent features in the library belong to the same pedestrian, and the current apparent feature distance is determined as the pedestrian's apparent feature distance;

   b. Calculate the spatial feature distances one by one based on the current three-dimensional motion features of pedestrians and the three-dimensional motion features of pedestrians at the next moment predicted from the previous moment of each pedestrian in the historical feature database. If the spatial feature distance is greater than the spatial threshold, the current pedestrian is judged The three-dimensional motion feature and the three-dimensional motion feature of the pedestrian at the next moment predicted at the previous moment in the historical feature library belong to the same pedestrian, and the current spatial feature distance is determined as the spatial feature distance of the pedestrian;

   c. Judging whether it conforms to the motion pattern of the same pedestrian based on the current three-dimensional motion features, apparent feature distance, spatial feature distance, and the three-dimensional motion features of each pedestrian in the historical feature database, and output the motion pattern matching degree as the pedestrian Motion pattern matching degree;

   d. Perform weighted calculation on the apparent feature distance, spatial feature distance, and motion pattern matching degree belonging to the same pedestrian, and obtain the matching result between the pedestrian in the current three-dimensional bounding box and the pedestrian in the historical feature database, and the matching result includes matching success or matching. If the match fails, the pedestrian information obtained by the match is also included when the match is successful;

   The pedestrian marking module is used to mark the pedestrian status of the pedestrian corresponding to the three-dimensional bounding box according to the current matching results and the historical matching results as initial matching success, loss, rematching success after loss, continuous matching success or out of the camera range;

   The people flow statistics module is used to count the flow of people within the statistical range corresponding to the optical image within the preset time according to the pedestrian state.
7. The people flow statistics system according to claim 6, characterized in that, the pedestrian in the optical image is detected, the three-dimensional bounding box of the pedestrian is output, and the following operations are performed:

   The camera used to obtain the optical image is calibrated, and the mapping relationship between the pixel in the optical image and the distance of the camera is obtained;

   Detect pedestrians in optical images, and obtain two-dimensional bounding boxes of pedestrians in optical images;

   Based on the 2D bounding box and the mapping relationship, the 3D bounding box of the pedestrian is obtained.
8. The people flow statistics system according to claim 7, wherein the three-dimensional motion feature of each pedestrian is extracted based on the current three-dimensional bounding box of the pedestrian and the three-dimensional bounding box distributed according to the time series in the historical feature library, and the execution is as follows operate:

   Direction vector extraction: Through the current 3D bounding box and the historical 3D bounding box, the pedestrian's movement direction in the horizontal direction and the movement direction in the vertical direction are extracted;

   Movement speed extraction: Extract the movement speed of the person in the horizontal direction and the movement speed in the vertical direction through the current 3D bounding box and the historical 3D bounding box;

   Relative position extraction: According to the mapping relationship obtained after the camera is calibrated, the coordinates of the pedestrian in the three-dimensional coordinate system centered on the camera are output based on the current three-dimensional bounding box and the historical three-dimensional bounding box;

   Feature integration: The extracted direction vector, motion speed and relative position are used as pedestrian 3D motion features.
9. The people flow statistics system according to claim 6, wherein the pedestrian state of the pedestrian corresponding to the three-dimensional bounding box is marked according to the current matching result and the historical matching result as successful initial matching, lost, and successful re-matching after loss , continuous matching successfully or out of the camera range, perform the following operations:

   If the pedestrian feature is successfully extracted, but the matching result is that the matching fails, the status of the current pedestrian is marked as the initial matching success;

   If the same pedestrian in the historical matching result is not matched for M consecutive times, the state of the pedestrian is marked as lost;

   If the pedestrian marked as lost is successfully re-matched in the current matching result, the status of the pedestrian is updated to be lost and the re-match is successful;

   If the same pedestrian in the historical matching result is matched for L consecutive times, the status of the pedestrian is updated to indicate that the continuous matching is successful;

   If the same pedestrian in the historical matching result is not matched for N consecutive times, the state of the pedestrian is marked as walking out of the camera range, and M<N.
10. The people flow statistics system according to claim 6, wherein if the current state of the pedestrian is marked as successful for the first time, new pedestrian information is allocated to the pedestrian in the historical feature database, and the pedestrian feature of the pedestrian is compared with the new pedestrian information. Assigned pedestrian information association.

Images