CN118155293B - A method and system for identifying dangerous pedestrian crossing behaviors based on small target tracking - Google Patents

Abstract

The present invention discloses a method and system for distinguishing dangerous behaviors of pedestrians crossing the street based on small target tracking, and the method comprises: step 1, using a camera to collect a video stream under a bird's-eye view in a designated crossing area; step 2, reading the collected video images frame by frame, performing frame-by-frame target detection on the images, and extracting the target trajectories of pedestrians and other traffic participants; step 3, using the pedestrian trajectory obtained in step 2, performing first-order time difference to obtain the instantaneous speed of the pedestrian, performing first-order time difference on the instantaneous speed curve to obtain the acceleration change curve, and using the power spectrum density to calculate the walking frequency of the pedestrian; step 4, calculating the conflict index between the pedestrian and other traffic participants; step 5, judging whether the pedestrian behavior is dangerous crossing behavior by using the pedestrian crossing gait parameters calculated in step 3 and the conflict index between the pedestrian and other traffic participants calculated in step 4. The method of the present invention can improve the discrimination accuracy of dangerous pedestrian crossing behavior.

Description

A method and system for identifying dangerous pedestrian crossing behaviors based on small target tracking

Technical Field

The invention relates to a method for distinguishing dangerous behaviors of pedestrians crossing the street, and belongs to the technical field of image recognition and traffic safety management.

Background technique

Walking is an important way for residents to travel short distances and occupies an important position in urban transportation. However, in recent years, the personal safety and property losses caused by pedestrian safety accidents have become increasingly serious. Pedestrian movement is highly random and obviously periodic. Pedestrian walking characteristic parameters such as cadence, stride length, and speed play an important role in analyzing pedestrian behavior. The transportation field pays more and more attention to the research on pedestrian safety. However, the current pedestrian dangerous behavior identification methods have low accuracy and reliability. Existing methods include recognition based on the degree of overlap between the pedestrian's position and the calibrated dangerous area, recognition based on vehicle driving images, and recognition of pedestrian dangerous features based on a single image.

Although the judgment based on the location and the marked danger zone can reflect the degree of danger of pedestrians to a certain extent, the judgment is too broad and it is easy to misjudge pedestrians who are not actually dangerous. It is also impossible to accurately identify pedestrians who are in danger in non-dangerous areas. The recognition method based on vehicle driving images is suitable for the field of intelligent driving, but it cannot play an effective role in regional traffic behavior analysis and key area supervision. The recognition of pedestrian danger features based on a single image ignores the spatiotemporal relationship of dangerous behaviors and the interaction with other road participants, and its accuracy and precision are not high.

Summary of the invention

The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a method for distinguishing dangerous pedestrian crossing behaviors, which can accurately assess whether dangerous pedestrian crossing behaviors have occurred.

In order to solve the above technical problems, the present invention adopts the following technical solutions:

A method for identifying dangerous pedestrian crossing behaviors based on small target tracking includes the following steps:

Step 1: Using a camera to collect a video stream of the designated crossing area from a bird's-eye view in the designated crossing area;

Step 2: Read the video images acquired in step 1 frame by frame, perform frame-by-frame target detection on the images, and extract the target trajectories of pedestrians and other traffic participants, including the following steps:

Step 21) Calculate the ratio of the target real frame area to the small target area parameter, and calculate the target anchor frame matching intersection over union ratio (IoU) threshold and the loss expansion coefficient for the small target object through the ratio;

Step 22) After obtaining the positions of pedestrians and other types of traffic participants in the video image frame by frame, the DeepSORT target association tracking algorithm is used to associate the targets of the previous and next frames, and the EfficientDet convolutional target detection network is used to extract the motion trajectories of pedestrians and other types of traffic participants in the video; the other types of traffic participants include cars, trucks, bicycles, and motorcycles;

Step 23) Select a checkerboard calibration plate of known size and shape, fix the camera, take several images containing the calibration plate, use OpenCV image processing algorithm to extract the coordinates of the corner points on the calibration plate in each image, correspond the corner point coordinates to the real world coordinates, and establish a mapping relationship between pixel coordinates and real coordinates; use Zhang Zhengyou camera calibration algorithm to calculate the internal and external parameters of the camera, and obtain the transformation matrix between the pixel coordinate system and the real coordinate system;

By using the correspondence between coordinate systems, the trajectory coordinates of pedestrians and other types of traffic participants are converted into trajectory coordinates in the world coordinate system. The vehicle trajectories are uniformly sampled at n trajectory points per second, and the trajectory position sequence coordinates of each vehicle are denoised and smoothed by Kalman filtering to obtain the smooth trajectories of pedestrians and other types of traffic participants in the real world.

Step 3: Using the pedestrian trajectory obtained in step 2, the instantaneous speed of the pedestrian is obtained by first-order time difference, the acceleration change curve is obtained by first-order time difference of the instantaneous speed curve, and the walking frequency of the pedestrian is calculated by using the power spectrum density;

Step 4: Calculate the conflict index TTC between pedestrians and other traffic participants;

Step 5: Determine whether the pedestrian's behavior is dangerous crossing behavior by using the pedestrian crossing gait parameters calculated in step 3 and the conflict index TTC between pedestrians and other traffic participants calculated in step 4.

In the aforementioned pedestrian crossing dangerous behavior identification method based on small target tracking, in step 2, the pre-trained EfficientDet convolutional target detection network with feature pyramid is used to perform frame-by-frame target detection on the image to extract the target trajectories of pedestrians and other traffic participants.

In the aforementioned pedestrian crossing dangerous behavior identification method based on small target tracking, in step 21), the specific method is:

Project the ratio of the target real box area to the small target area parameter into the interval formed by the given basic anchor box threshold and the small target anchor box threshold, calculate the target anchor box matching intersection over union ratio IoU threshold, project the inverse of the ratio of the target real box area to the small target area parameter into the interval of 0-1 and multiply it by a coefficient as the loss expansion coefficient for the small target object:

,

,

in, is the calculated target anchor box matching intersection over union (IoU) threshold, is the IoU threshold for matching target anchor boxes of normal-sized objects. The lower limit of the IoU threshold for matching the small target anchor box is set. is the real frame area, is the set small target box area parameter, a is the magnification parameter, is the loss expansion coefficient, b is the expansion parameter, c is the amplification parameter, and sigmoid is a mathematical function used to map any real number to between (0, 1).

In the aforementioned pedestrian crossing dangerous behavior identification method based on small target tracking, in step 23), the relationship between the pixel coordinate system and the real coordinate system is as follows:

,

in is the scale factor, and are the horizontal and vertical coordinates of the pixel coordinate system, , , are the horizontal abscissa, horizontal ordinate and vertical coordinate of the world coordinate system respectively. and are the focal lengths of the camera in the x and y directions, respectively. is the non-orthogonal factor between pixels, where They represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, is the rotation matrix, is the translation vector; Represents an all-0 matrix of size 1*3. The 3 in the formula represents 3 rows and T represents transpose.

In the aforementioned pedestrian crossing dangerous behavior identification method based on small target tracking, in step 3, the formula for calculating the pedestrian's walking frequency is as follows:

,

in, For each frequency The power spectral density estimate of is the sampling frequency, is the discrete time series of the pedestrian walking process, is the instantaneous speed of the pedestrian , the double vertical lines indicate the magnitude of the calculated vector, i.e. the magnitude of the instantaneous velocity; is the length of the signal, which refers to the length of the pedestrian walking time series;

The frequency with the maximum power density is selected to represent the step frequency, and the pedestrian's walking stride is obtained by dividing the pedestrian's walking speed by the step frequency. Take out in seconds The pedestrian gait parameters within seconds and the corresponding pedestrian trajectory and traffic participant trajectory are used for the calculation in step 4.

In the aforementioned pedestrian crossing dangerous behavior identification method based on small target tracking, in step 4, In the second trajectory segment, the conflict index TTC between the pedestrian and the N closest other traffic participants is calculated, and the smallest conflict index TTC is used as the conflict index TTC used in step 5. The calculation method of the conflict index TTC is as follows:

,

in, is the TTC between pedestrian i and traffic participant j , is the relative distance between pedestrians and other traffic participants, is the projection of the relative speed of pedestrians and other traffic participants on the position line, and the relative distance Obtained through the real world coordinates of the trajectory points, It is to take the first-order difference of time of the trajectories of pedestrians and other traffic participants, calculate the speed of pedestrians and other traffic participants, perform vector synthesis of the speed, and project it onto the line connecting pedestrians and other traffic participants.

In the aforementioned pedestrian crossing dangerous behavior identification method based on small target tracking, in step 5, The gait parameters of pedestrians crossing the street within seconds are input into the pre-trained Transformer temporal attention neural network classifier to determine whether the change pattern of the gait parameters of pedestrians crossing the street conforms to the characteristics of dangerous behavior, forming an evaluation index R1; in addition, a statistical analysis is performed on the gait parameters of pedestrians crossing the street to determine whether there is a situation where the step frequency, stride or speed is greater than the threshold or the change amplitude within the set time is greater than the threshold, forming an evaluation index R2; the evaluation index R1, the evaluation index R2 and the conflict index TTC are used to construct a judgment table, and the judgment table is used to determine whether a dangerous pedestrian crossing behavior occurs.

In the aforementioned method for identifying dangerous pedestrian crossing behaviors based on small target tracking, in step 5, when it is determined that a dangerous behavior occurs, the video of a set time period before and after the behavior occurs is captured as a criterion, and corresponding indicators are output, including evaluation indicator 1 R1, evaluation indicator 2 R2 and conflict indicator TTC.

A computer system comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

The beneficial effects achieved by the present invention are as follows: The method for distinguishing dangerous pedestrian crossing behaviors based on small target tracking of the present invention performs target detection through a convolutional target detection network, and uses a multi-scale learning method to assign higher weights to the shallower features of FPN to enhance the retention of small target features, and is supplemented by a smaller anchor frame and a higher small target sample loss weight to improve the detection of small pedestrian targets. It can effectively identify pedestrians in the picture, and extract trajectories by tracking pedestrians and other traffic participants. The gait characteristics of pedestrian movement are extracted in combination with the acceleration change law of pedestrian targets, including the step frequency, stride length and speed, and the traffic conflict index is calculated using the trajectory interaction relationship between pedestrians and other traffic participants. The dangerousness of the behavior can be judged by combining the pedestrian's own behavior and the interactive behavior with traffic participants, which can improve the accuracy of distinguishing dangerous pedestrian crossing behaviors and has practical engineering value in the field of traffic safety management and control.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for identifying dangerous pedestrian crossing behaviors based on small target tracking in Embodiment 1 of the present invention.

Detailed ways

The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

Example 1

As shown in FIG1 , this embodiment provides a method for identifying dangerous pedestrian crossing behaviors based on small target tracking, comprising the following steps:

Step 1: Use a camera to collect a video stream of the designated crossing area from a bird's-eye view in the designated crossing area, and input the collected video stream into the following steps to perform operations;

The video format and frame number captured by the camera are shown in Table 1 below;

Table 1 Camera video stream collection table

Roadside cameras can be used to continuously acquire video at a moderate height, so pedestrians can be captured stably, making it easy to pre-train on existing network weights.

Step 2: Read the video images collected in step 1 frame by frame, use the pre-trained EfficientDet convolutional target detection network with feature pyramid (FPN) to perform frame-by-frame target detection on the image, and extract the trajectories of pedestrians and other traffic participants.

In the pre-trained dataset, a large number of anchor box samples are annotated for pedestrians in a bird's-eye view. At the same time, in order to ensure the recognition rate of small pedestrian targets, the fusion weight of FPN shallow features is restricted in the design of the EfficientDet convolutional target detection network, so that the semantic features of small targets will not be lost excessively. At the same time, the network detection anchor box is specially designed, using a smaller anchor box with a lower anchor box matching intersection over union (IoU) threshold for small target objects, and setting a higher loss for small targets to further ensure the detection accuracy of small targets. The following steps are included:

Step 21) Calculate the ratio of the target real frame area to the small target area parameter, and calculate the target anchor frame matching intersection over union ratio (IoU) threshold and the loss expansion coefficient for the small target object by the ratio. The specific method is as follows:

Project the ratio of the target real box area to the small target area parameter into the interval formed by the given basic anchor box threshold and the small target anchor box threshold, calculate the target anchor box matching intersection over union ratio IoU threshold, project the inverse of the ratio of the target real box area to the small target area parameter into the interval of 0-1 and multiply it by a coefficient as the loss expansion coefficient for the small target object:

in, is the calculated target anchor box matching intersection over union (IoU) threshold, is the IoU threshold for matching target anchor boxes of normal-sized objects. The lower limit of the IoU threshold for matching the small target anchor box is set. is the real frame area, is the set small target box area parameter, a is the magnification parameter, is the loss expansion coefficient, b is the expansion parameter, c is the magnification parameter, sigmoid is a mathematical function used to map any real number to between (0, 1), and the image presents an "S"curve;

There are two key steps in the training process of the EfficientDet convolutional target detection network, namely detection target matching and network weight updating.

A key parameter in the target matching process is the IoU threshold (intersection over union threshold). The match is considered successful only when the IoU of the detection box and the target box is greater than the threshold. For small targets, a smaller IoU threshold is used than for normal targets, so that the network can better match small targets.

The basis for updating the network weights is the loss function, which assigns higher losses to small targets so that the network receives more penalties after failing to detect small targets. Small targets are detected as much as possible to optimize the network parameters.

Step 22) After obtaining the positions of pedestrians and other types of traffic participants in the video image frame by frame, the DeepSORT target association tracking algorithm is used to associate the targets of the previous and next frames, and the EfficientDet convolutional target detection network is used to extract the motion trajectories of pedestrians and other types of traffic participants in the video; the other types of traffic participants include cars, trucks, bicycles, motorcycles, etc.;

Step 23) Select a checkerboard calibration plate of known size and shape, fix the camera, take several images containing the calibration plate, use OpenCV image processing algorithm to extract the coordinates of the corner points on the calibration plate of each image, correspond the corner point coordinates to the real world coordinates, and establish the mapping relationship between pixel coordinates and real coordinates; use Zhang Zhengyou camera calibration algorithm to calculate the internal and external parameters of the camera, and obtain the transformation matrix between the pixel coordinate system and the real coordinate system. The pixel coordinate system and the real coordinate system have the following relationship:

in is the scale factor, and are the horizontal and vertical coordinates of the pixel coordinate system, , , are the horizontal abscissa, horizontal ordinate and vertical coordinate of the world coordinate system respectively. and are the focal lengths of the camera in the x and y directions, respectively. is the non-orthogonal factor between pixels, where They represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, is the rotation matrix, is the translation vector; Represents an all-0 matrix of size 1*3. The 3 in the formula represents 3 rows and T represents transpose.

For a fixed camera, the calibration transformation matrix includes the camera's intrinsic parameter matrix M1 and extrinsic parameter matrix M2:

By utilizing the correspondence between coordinate systems, the trajectory point coordinates of pedestrians and other types of traffic participants are converted into trajectory point coordinates in the world coordinate system. The vehicle trajectories are uniformly sampled at n trajectory points per second, and the trajectory position sequence coordinates of each vehicle are denoised and smoothed by Kalman filtering to obtain the smooth trajectories of pedestrians and other types of traffic participants in the real world.

If the transformation matrix is not used to establish the mapping relationship between the pixel coordinate system and the real coordinate system, the coordinates obtained after running the tracking algorithm will be pixel coordinates, which are different from their actual meanings and have a certain impact on the accuracy of subsequent calculations. Therefore, after extracting the motion trajectory of the traffic participant, its trajectory is projected into the real-world coordinate system through the transformation matrix to ensure the accuracy of subsequent calculations.

The extracted trajectories of pedestrians and other traffic participants are shown in Table 2:

Table 2 Trajectory table of pedestrians and other traffic participants

The obtained trajectory table is shown in Table 3:

Table 3 Trajectory table

Step 3: Calculate the pedestrian gait parameters. Pedestrian movement is periodic, and the movement acceleration changes periodically according to the walking steps. Using the pedestrian trajectory obtained in step 2, the instantaneous speed of the pedestrian is obtained by the first-order time difference. The acceleration change curve can be obtained by taking the first-order time difference of the instantaneous speed curve. The pedestrian walking frequency is calculated using the power spectrum density. The formula is as follows:

in, For each frequency The power spectral density estimate of is the sampling frequency, is the discrete time series of the pedestrian walking process, is the instantaneous speed of the pedestrian , the double vertical lines indicate the magnitude of the calculated vector, i.e. the magnitude of the instantaneous velocity; is the length of the signal, which refers to the length of the pedestrian walking time series;

The frequency with the maximum power density is selected to represent the cadence, and the pedestrian's walking stride is obtained by dividing the pedestrian's walking speed by the cadence. The pedestrian's gait parameters within t2 seconds and the corresponding pedestrian trajectory and traffic participant trajectory are taken out every t1 seconds for the calculation of step 4. In this embodiment, t1 is 2 seconds and t2 is 4 seconds.

In this embodiment, the power spectrum density method is used to calculate the cadence, and then the stride is calculated using the relationship between the cadence, stride length and speed. The power spectrum density method utilizes the wave nature of the pedestrian's acceleration change. It is more accurate to obtain the cadence through the power wave, and at the same time, it can filter out the influence of some noise.

The obtained pedestrian gait parameter table is shown in Table 4.

Table 4 Pedestrian gait parameters

In the table, Indicates the step frequency, represents the step length, Indicates pace.

Step 4: Calculate the conflict index TTC between pedestrians and other traffic participants;

In the extracted t2-second trajectory segment, the conflict index TTC between the pedestrian and the N closest other traffic participants is calculated, and the smallest conflict index TTC is used as the conflict index TTC used in step 5. The calculation method of the conflict index TTC is as follows:

in, is the TTC (Time to Time To Collision) between pedestrian i and traffic participant j , is the relative distance between pedestrians and other traffic participants, is the projection of the relative speed of pedestrians and other traffic participants on the position line, and the relative distance It can be obtained by the real world coordinates of the trajectory points. It is to take the first-order difference of time of the trajectories of pedestrians and other traffic participants, calculate the speed of pedestrians and other traffic participants, perform vector synthesis of the speed, and project it onto the line connecting pedestrians and other traffic participants.

The N≤3, the larger the number, the higher the computing power requirement, because pedestrians in the picture may have interactive relationships with many traffic participants. By filtering out units with closer distances for calculation, the computing efficiency can be improved.

Step 5: using the pedestrian crossing gait parameters calculated in step 3 and the conflict index TTC between pedestrians and other traffic participants calculated in step 4, determining whether the pedestrian's behavior is a dangerous crossing behavior, including the following steps:

The pedestrian crossing gait parameters within t2 seconds are input into the pre-trained Transformer temporal attention neural network classifier to determine whether the change pattern of the pedestrian crossing gait parameters conforms to the characteristics of dangerous behavior, forming an evaluation index R1. In addition, the pedestrian crossing gait parameters are statistically analyzed to determine whether there is a situation where the step frequency, stride or speed is greater than the threshold or the change amplitude within a set time (such as 1s) is greater than the threshold, forming an evaluation index R2; the evaluation index R1, the evaluation index R2 and the conflict index TTC are used to construct a judgment table, and the judgment table is used to determine whether a pedestrian crossing dangerous behavior occurs. The judgment table is shown in Table 5:

Table 5 Dangerous behavior identification table

In the table, Thred_R_1, Thred_R_2, , They respectively represent evaluation index threshold 1, evaluation index threshold 2, conflict index TTC threshold 1, and conflict index TTC threshold 2.

When a dangerous behavior is determined to have occurred, the video of a set time period (such as 10 seconds) before and after the behavior is captured as a criterion, and the corresponding indicators are output, including evaluation indicator 1 R1, evaluation indicator 2 R2 and conflict indicator TTC.

In this embodiment, the Transformer temporal attention neural network classifier is used to determine whether the gait parameters of pedestrians crossing the street meet the characteristics of dangerous behavior. The Transformer network can adaptively learn rules from data, with high robustness and accuracy. At the same time, it combines conflict substitution indicators and objective gait rules to make judgments, with strong comprehensiveness, high robustness, and little impact from the environment.

Evaluation index 1 R1, evaluation index 2 R2 and conflict index TTC can be disassembled and judged according to the actual project. All three indicators can be used to judge whether it is dangerous separately. The judgment accuracy can be further improved by combining the three. Some other networks can also be used for judgment, such as using LSTM to replace Transformer, and PET can be used instead of TTC in terms of indicators.

Example 2

This embodiment provides a method for identifying dangerous pedestrian crossing behaviors based on small target tracking, including the following steps:

Step 1: Using a camera to collect a video stream of the designated crossing area from a bird's-eye view in the designated crossing area;

Step 2: Read the video images collected in step 1 frame by frame, use the pre-trained EfficientDet convolutional target detection network with feature pyramid (FPN) to perform frame-by-frame target detection on the images, and extract the trajectories of pedestrian targets and other traffic participants;

Step 21) Calculate the ratio of the target real frame area to the small target area parameter, and calculate the target anchor frame matching intersection over union ratio (IoU) threshold and the loss expansion coefficient for the small target object through the ratio;

Step 22) After obtaining the positions of pedestrians and other types of traffic participants in the video image frame by frame, the nearest neighbor algorithm, probabilistic data association algorithm or SORT algorithm are used to associate the targets of the previous and next frames, and the motion trajectories of pedestrians and other types of traffic participants in the video are extracted using the EfficientDet convolutional target detection network;

Step 23) Select a checkerboard calibration plate of known size and shape, fix the camera, take several images containing the calibration plate, use OpenCV image processing algorithm to extract the coordinates of the corner points on the calibration plate in each image, correspond the corner point coordinates to the real world coordinates, and establish the mapping relationship between pixel coordinates and real coordinates; use Zhang Zhengyou camera calibration algorithm to calculate the internal and external parameters of the camera, and obtain the transformation matrix between the pixel coordinate system and the real coordinate system.

Step 3: Using the pedestrian trajectory obtained in step 2, the instantaneous speed of the pedestrian is obtained by first-order time difference. The acceleration change curve can be obtained by first-order time difference of the instantaneous speed curve, and the walking frequency of the pedestrian is calculated using the power spectrum density;

Step 4: Calculate the conflict index TTC between pedestrians and other traffic participants;

Step 5: Determine whether the pedestrian's behavior is dangerous crossing behavior by using the pedestrian crossing gait parameters calculated in step 3 and the conflict index TTC between pedestrians and other traffic participants calculated in step 4.

Example 3

This embodiment provides a method for identifying dangerous pedestrian crossing behaviors based on small target tracking, including the following steps:

Step 1: Using a camera to collect a video stream of the designated crossing area from a bird's-eye view in the designated crossing area;

Step 2: Read the video image acquired in step 1 frame by frame, use yolov1, yolov2 or fastrcnn method to perform frame-by-frame target detection on the image, and extract the target trajectories of pedestrians and other traffic participants;

Step 3: Using the pedestrian trajectory obtained in step 2, the instantaneous speed of the pedestrian is obtained by first-order time difference. The acceleration change curve can be obtained by first-order time difference of the instantaneous speed curve, and the walking frequency of the pedestrian is calculated using the power spectrum density;

Step 4: Calculate the conflict index TTC between pedestrians and other traffic participants;

Step 5: Determine whether the pedestrian's behavior is dangerous crossing behavior by using the pedestrian crossing gait parameters calculated in step 3 and the conflict index TTC between pedestrians and other traffic participants calculated in step 4.

Example 4

A computer system comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

Example 5

A computer-readable storage medium having a computer program/instruction stored thereon, characterized in that the computer program/instruction implements the steps of the above method when executed by a processor.

Example 6

A computer program product comprises a computer program/instruction, wherein the computer program/instruction implements the steps of the above method when executed by a processor.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present application is described with reference to the flowcharts and/or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims (9)

1. A method for identifying dangerous pedestrian crossing behaviors based on small target tracking, characterized in that it comprises the following steps: Step 1: Using a camera to collect a video stream of the designated crossing area from a bird's-eye view in the designated crossing area; Step 2: Read the video images acquired in step 1 frame by frame, perform frame-by-frame target detection on the images, and extract the target trajectories of pedestrians and other traffic participants, including the following steps: Step 21) Calculate the ratio of the target real frame area to the small target area parameter, and calculate the target anchor frame matching intersection over union ratio (IoU) threshold and the loss expansion coefficient for the small target object through the ratio; Step 22) After obtaining the positions of pedestrians and other types of traffic participants in the video image frame by frame, the DeepSORT target association tracking algorithm is used to associate the targets of the previous and next frames, and the EfficientDet convolutional target detection network is used to extract the motion trajectories of pedestrians and other types of traffic participants in the video; the other types of traffic participants include cars, trucks, bicycles, and motorcycles; Step 23) Select a checkerboard calibration plate of known size and shape, fix the camera, take several images containing the calibration plate, use OpenCV image processing algorithm to extract the coordinates of the corner points on the calibration plate in each image, correspond the corner point coordinates to the real world coordinates, and establish a mapping relationship between pixel coordinates and real coordinates; use Zhang Zhengyou camera calibration algorithm to calculate the internal and external parameters of the camera, and obtain the transformation matrix between the pixel coordinate system and the real coordinate system; By using the correspondence between coordinate systems, the trajectory coordinates of pedestrians and other types of traffic participants are converted into trajectory coordinates in the world coordinate system. The vehicle trajectories are uniformly sampled at n trajectory points per second, and the trajectory position sequence coordinates of each vehicle are denoised and smoothed by Kalman filtering to obtain the smooth trajectories of pedestrians and other types of traffic participants in the real world. Step 3: Using the pedestrian trajectory obtained in step 2, the instantaneous speed of the pedestrian is obtained by first-order time difference, the acceleration change curve is obtained by first-order time difference of the instantaneous speed curve, and the walking frequency of the pedestrian is calculated by using the power spectrum density; Step 4: Calculate the conflict index TTC between pedestrians and other traffic participants; Step 5: Determine whether the pedestrian's behavior is dangerous crossing behavior by using the pedestrian crossing gait parameters calculated in step 3 and the conflict index TTC between pedestrians and other traffic participants calculated in step 4. 2. According to the method for identifying dangerous pedestrian crossing behaviors based on small target tracking in claim 1, in step 2, a pre-trained EfficientDet convolutional target detection network with a feature pyramid is used to perform frame-by-frame target detection on the image to extract the target trajectories of pedestrians and other traffic participants. 3. The method for identifying dangerous pedestrian crossing behaviors based on small target tracking according to claim 1 is characterized in that, in step 21), the specific method is: Project the ratio of the target real box area to the small target area parameter into the interval formed by the given basic anchor box threshold and the small target anchor box threshold, calculate the target anchor box matching intersection over union ratio IoU threshold, project the inverse of the ratio of the target real box area to the small target area parameter into the interval of 0-1 and multiply it by a coefficient as the loss expansion coefficient for the small target object: , , in, is the calculated target anchor box matching intersection over union (IoU) threshold, is the IoU threshold for matching target anchor boxes of normal-sized objects. The lower limit of the IoU threshold for matching the small target anchor box is set. is the real frame area, is the set small target box area parameter, a is the magnification parameter, is the loss expansion coefficient, b is the expansion parameter, c is the amplification parameter, and sigmoid is a mathematical function used to map any real number to between (0, 1). 4. The method for identifying dangerous pedestrian crossing behaviors based on small target tracking according to claim 1 is characterized in that, in step 23), the pixel coordinate system and the real coordinate system have the following relationship: , in, is the scale factor, and are the horizontal and vertical coordinates of the pixel coordinate system, , , are the horizontal abscissa, horizontal ordinate and vertical coordinate of the world coordinate system respectively. and are the focal lengths of the camera in the x and y directions, respectively. is the non-orthogonal factor between pixels, They represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, is the rotation matrix, is the translation vector; Represents an all-0 matrix of size 1*3. The 3 in the formula represents 3 rows and T represents transpose. 5. The method for identifying dangerous pedestrian crossing behaviors based on small target tracking according to claim 1 is characterized in that, in step 3, the formula for calculating the pedestrian's walking frequency is as follows: , in, For each frequency The power spectral density estimate of is the sampling frequency, is the discrete time series of the pedestrian walking process, is the instantaneous speed of the pedestrian , the double vertical lines indicate the magnitude of the calculated vector, i.e. the magnitude of the instantaneous velocity; is the length of the signal, which refers to the length of the pedestrian walking time series; The frequency with the maximum power density is selected to represent the step frequency, and the pedestrian's walking stride is obtained by dividing the pedestrian's walking speed by the step frequency. Take out in seconds The pedestrian gait parameters within seconds and the corresponding pedestrian trajectory and traffic participant trajectory are used for the calculation in step 4. 6. The method for identifying dangerous pedestrian crossing behaviors based on small target tracking according to claim 1 is characterized in that in step 4, In the second trajectory segment, the conflict index TTC between the pedestrian and the N closest other traffic participants is calculated, and the smallest conflict index TTC is used as the conflict index TTC used in step 5. The calculation method of the conflict index TTC is as follows: , in, is the TTC between pedestrian i and traffic participant j , is the relative distance between pedestrians and other traffic participants, is the projection of the relative speed of pedestrians and other traffic participants on the position line, and the relative distance Obtained through the real world coordinates of the trajectory points, It is to take the first-order difference of time of the trajectories of pedestrians and other traffic participants, calculate the speed of pedestrians and other traffic participants, perform vector synthesis of the speed, and project it onto the line connecting pedestrians and other traffic participants. 7. The method for identifying dangerous pedestrian crossing behaviors based on small target tracking according to claim 1 is characterized in that, in step 5, The gait parameters of pedestrians crossing the street within seconds are input into the pre-trained Transformer temporal attention neural network classifier to determine whether the change pattern of the gait parameters of pedestrians crossing the street conforms to the characteristics of dangerous behavior, forming an evaluation index R1; in addition, a statistical analysis is performed on the gait parameters of pedestrians crossing the street to determine whether there is a situation where the step frequency, stride or speed is greater than the threshold or the change amplitude within the set time is greater than the threshold, forming an evaluation index R2; the evaluation index R1, the evaluation index R2 and the conflict index TTC are used to construct a judgment table, and the judgment table is used to determine whether a dangerous pedestrian crossing behavior occurs. 8. According to the method for distinguishing dangerous pedestrian crossing behaviors based on small target tracking in claim 7, it is characterized in that, in step 5, when it is determined that a dangerous behavior occurs, a video of a set time period before and after the behavior occurs is captured as a criterion, and corresponding indicators are output, including evaluation indicator 1 R1, evaluation indicator 2 R2 and conflict indicator TTC. 9. A computer system comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.