CN118155293B - Pedestrian crossing dangerous behavior discrimination method and system based on small target tracking - Google Patents
Pedestrian crossing dangerous behavior discrimination method and system based on small target tracking Download PDFInfo
- Publication number
- CN118155293B CN118155293B CN202410567563.2A CN202410567563A CN118155293B CN 118155293 B CN118155293 B CN 118155293B CN 202410567563 A CN202410567563 A CN 202410567563A CN 118155293 B CN118155293 B CN 118155293B
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- frame
- traffic participants
- target
- small target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012850 discrimination method Methods 0.000 title claims description 9
- 238000000034 method Methods 0.000 claims abstract description 52
- 230000005021 gait Effects 0.000 claims abstract description 25
- 230000008859 change Effects 0.000 claims abstract description 10
- 230000001133 acceleration Effects 0.000 claims abstract description 9
- 238000001228 spectrum Methods 0.000 claims abstract description 8
- 230000006399 behavior Effects 0.000 claims description 59
- 238000001514 detection method Methods 0.000 claims description 22
- 238000011156 evaluation Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 16
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 8
- 230000003321 amplification Effects 0.000 claims description 6
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 230000015572 biosynthetic process Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000007620 mathematical function Methods 0.000 claims description 3
- 230000003595 spectral effect Effects 0.000 claims description 3
- 238000007619 statistical method Methods 0.000 claims description 3
- 238000003786 synthesis reaction Methods 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a pedestrian crossing dangerous behavior judging method and system based on small target tracking, wherein the method comprises the following steps: step 1, acquiring video streams in a overlooking view angle by using a camera in a designated street crossing area; step 2, reading the acquired video images frame by frame, detecting the targets of the images frame by frame, and extracting target tracks of pedestrian targets and other traffic participants; step 3, obtaining the instantaneous speed of the pedestrian by utilizing the pedestrian track obtained in the step 2 and the time first-order difference, obtaining an acceleration change curve by performing the time first-order difference on the instantaneous speed curve, and calculating the walking frequency of the pedestrian by utilizing the power spectrum density; step 4, calculating collision indexes of pedestrians and other traffic participants; and 5, judging whether the pedestrian behavior is dangerous street crossing behavior or not through the pedestrian crossing gait parameters calculated in the step 3 and the conflict indexes of the pedestrians and other traffic participants calculated in the step 4. The method can improve the discrimination precision of the dangerous pedestrian crossing behavior.
Description
Technical Field
The invention relates to a pedestrian crossing dangerous behavior judging method, and belongs to the technical field of image recognition and traffic safety management.
Background
Walking is an important way for residents to travel in short distances and plays an important role in urban traffic. However, personal safety and property loss caused by pedestrian safety accidents are more serious in recent years. The pedestrian movement has strong randomness and obvious periodicity, the pedestrian walking characteristic parameters such as walking frequency, stride, pace and the like have important roles in analyzing pedestrian behaviors, the research on the aspect of pedestrian safety in the traffic field is more and more important, however, the existing pedestrian dangerous behavior judging method is low in accuracy and reliability, and the existing method comprises recognition based on the coincidence degree of the position of the pedestrian and the calibrated dangerous area, recognition based on a vehicle driving image and recognition based on the dangerous characteristics of the pedestrian of a single image picture.
Although the judgment based on the position and the calibrated dangerous area can reflect the dangerous degree of pedestrians to a certain extent, the judgment is too wide, false judgment is easy to occur to pedestrians which are not dangerous actually, and pedestrians which are dangerous in non-dangerous areas cannot be accurately identified. The identification method based on the vehicle driving image is suitable for the intelligent driving field, and cannot play an effective role in regional traffic behavior analysis and key region supervision. Based on the single image picture, the pedestrian dangerous feature recognition ignores the space-time relationship of dangerous behaviors and the interaction with other road participants, and has low accuracy and precision.
Disclosure of Invention
The invention aims to solve the technical problem of overcoming the defects of the prior art and providing a pedestrian crossing dangerous behavior judging method which can accurately evaluate whether the pedestrian crossing dangerous behavior occurs.
In order to solve the technical problems, the invention adopts the following technical scheme:
a pedestrian crossing dangerous behavior distinguishing method based on small target tracking comprises the following steps:
step 1, acquiring a video stream of a specified street crossing area under a overlooking view angle by using a camera in the specified street crossing area;
Step 2, reading the video image acquired in the step 1 frame by frame, detecting the frame by frame target of the image, and extracting target tracks of pedestrian targets and other traffic participants, wherein the method comprises the following steps:
Step 21), calculating the ratio of the target real frame area to the small target area parameter, and calculating the target anchor frame matching intersection ratio IoU threshold and the loss expansion coefficient aiming at the small target object according to the ratio;
Step 22), after positions of pedestrians and other various traffic participants in the video image are acquired frame by frame, correlating targets of front and rear frames by utilizing DeepSORT target correlation tracking algorithm, and extracting motion tracks of the pedestrians and the other various traffic participants in the video by utilizing EFFICIENTDET convolution target detection network; the other various traffic participants comprise automobiles, trucks, bicycles and motorcycles;
step 23) selecting a checkerboard calibration plate with known size and shape, fixing a camera, shooting a plurality of images containing the calibration plate, extracting angular point coordinates on each image calibration plate by using an OpenCV image processing algorithm, corresponding the angular point coordinates with real world coordinates, and establishing a mapping relation between pixel coordinates and real coordinates; calculating internal and external parameters of the camera by using Zhang Zhengyou camera calibration algorithm to obtain a conversion matrix between a pixel coordinate system and a real coordinate system;
Converting the track point coordinates of pedestrians and other various traffic participants into track point coordinates under a world coordinate system by utilizing the corresponding relation among the coordinate systems, uniformly sampling the running track of the vehicle according to n track points per second, and carrying out Kalman filtering denoising smoothing on the track position sequence coordinates of each vehicle to obtain smooth tracks of the pedestrians and other various traffic participants in the real world;
Step 3, obtaining the instantaneous speed of the pedestrian by utilizing the pedestrian track obtained in the step2 and the time first-order difference, obtaining an acceleration change curve by performing the time first-order difference on the instantaneous speed curve, and calculating the walking frequency of the pedestrian by utilizing the power spectrum density;
Step 4, calculating collision indexes TTC of pedestrians and other traffic participants;
And 5, judging whether the pedestrian behavior is dangerous street crossing behavior or not through the pedestrian crossing gait parameters calculated in the step 3 and the conflict indexes TTC of the pedestrians and other traffic participants calculated in the step 4.
In the method for judging the pedestrian crossing dangerous behavior based on small target tracking, in the step 2, the pre-trained EFFICIENTDET convolution target detection network with the characteristic pyramid is used for carrying out frame-by-frame target detection on the image, and the target track of the pedestrian target and other traffic participants is extracted.
In the foregoing method for determining pedestrian crossing dangerous behavior based on small target tracking, in step 21), the specific method is as follows:
Projecting the ratio of the target real frame area to the small target area parameter into a section formed by a given basic anchor frame threshold value and a small target anchor frame threshold value, calculating a target anchor frame matching intersection ratio IoU threshold value, projecting the inverse of the ratio of the target real frame area to the small target area parameter into a 0-1 section, multiplying the inverse by a coefficient, and then taking the inverse as a loss expansion coefficient for a small target object:
，
，
Wherein, For the calculated target anchor frame matching cross ratio IoU threshold,For a target anchor frame matching cross ratio IoU threshold for a normal size object,For a set small target anchor frame matching cross ratio IoU threshold lower limit,For the real frame area to be a real frame area,For the set small target frame area parameter, a is the amplification parameter,To lose the expansion coefficient, b is the expansion parameter, c is the amplification parameter, sigmoid is a mathematical function for mapping any real number between (0, 1).
In the foregoing pedestrian crossing dangerous behavior determination method based on small target tracking, in step 23), the pixel coordinate system and the real coordinate system have the following relationship:
，
Wherein the method comprises the steps of As a scale factor, the number of the elements is,AndRespectively the abscissa and the ordinate of the pixel coordinate system,，，Respectively a horizontal abscissa, a horizontal ordinate and a vertical coordinate of a world coordinate system,AndThe focal lengths of the cameras in the x and y directions respectively,Is a non-orthogonality factor between pixels, whereRespectively representing the coordinates of the center of the camera photosensitive plate under a pixel coordinate system,In order to rotate the matrix is rotated,Is a translation vector; representing an all 0 matrix of size 1*3, 3 in the formula representing 3 rows and T representing the transpose.
In the foregoing pedestrian crossing dangerous behavior judging method based on small target tracking, in step 3, the formula for calculating the walking frequency of the pedestrian is as follows:
，
Wherein, For each frequencyIs used for the power spectral density estimation of (a),For the sampling frequency to be the same,As a discrete time sequence of the pedestrian walking process,For pedestrian instantaneous speed valuesDouble vertical lines represent the magnitude of the calculated vector, i.e., the magnitude value of the instantaneous speed; the length of the signal refers to the length of the pedestrian walking time sequence;
Selecting a frequency representative step frequency with the maximum power density, dividing the step frequency by the pedestrian step speed to obtain the pedestrian walking steps, wherein each step is Second take outThe pedestrian gait parameters in seconds and the corresponding pedestrian trajectory and traffic participant trajectory are used for the calculation of step 4.
In the above-mentioned pedestrian crossing dangerous behavior determination method based on small target tracking, in step 4, the pedestrian crossing dangerous behavior is taken outAnd 5, in the second track segment, calculating conflict indexes TTC between the pedestrian and N other traffic participants closest to the pedestrian, wherein the minimum conflict index TTC is used as the conflict index TTC used in the step 5, and the calculation method of the conflict index TTC is as follows:
，
Wherein, Is the TTC between pedestrian i and traffic participant j,Is the relative distance between the pedestrian and the other traffic participants,Is the projection of the relative speed of the pedestrian and other traffic participants on the position connecting line, and the relative distanceObtained by real world coordinates of the track points,The method is that the first-order difference of time is carried out on the tracks of pedestrians and other traffic participants, the speeds of the pedestrians and the other traffic participants are obtained, vector synthesis is carried out on the speeds, and then the speeds are projected on the connecting line of the pedestrians and the other traffic participants.
In the method for judging the pedestrian crossing dangerous behavior based on small target tracking, in step 5, the pedestrian crossing dangerous behavior is judgedInputting pedestrian crossing gait parameters in seconds into a pre-trained transducer time sequence attention neural network classifier, judging whether the variation rule of the pedestrian crossing gait parameters accords with dangerous behavior characteristics, and forming an evaluation index R1; in addition, statistical analysis is carried out on pedestrian crossing gait parameters, whether the step frequency, the step length or the step speed is larger than a threshold value or the variation amplitude in set time is larger than the threshold value is judged, and an evaluation index II R2 is formed; and constructing a judging table by using the first evaluation index R1, the second evaluation index R2 and the conflict index TTC, and judging whether the pedestrian crossing dangerous behavior occurs or not through the judging table.
In the foregoing method for determining dangerous behavior of pedestrian crossing based on small target tracking, in step 5, when it is determined that dangerous behavior occurs, video of a set period of time before and after the occurrence of behavior is intercepted is left as a criterion, and corresponding indexes including an evaluation index one R1, an evaluation index two R2 and a collision index TTC are output.
A computer system comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to carry out the steps of the above method.
The invention has the beneficial effects that: according to the pedestrian crossing dangerous behavior judging method based on small target tracking, target detection is carried out through the convolution target detection network, the characteristics of the shallower layer of the FPN are given higher weight by utilizing the multi-scale learning method to enhance the retention of the characteristics of the small target, and the detection of the small target of the pedestrian is promoted by assisting with a smaller anchor frame and a higher sample loss weight of the small target, so that the pedestrian in a picture can be effectively identified, and the extraction track of the pedestrian and other traffic participants is tracked. The gait characteristics of pedestrian movement are extracted by combining the acceleration change rule of the pedestrian target, the gait characteristics comprise the step frequency stride and the step speed, and meanwhile, the traffic conflict index is calculated by utilizing the track interaction relation between the pedestrian and other traffic participants. The risk of the pedestrian's own behavior and the interactive behavior with the traffic participants is combined to judge the behavior, so that the judgment precision of the pedestrian's dangerous street crossing behavior can be improved, and the method has practical engineering value in the traffic safety management control field.
Drawings
Fig. 1 is a flowchart of a pedestrian crossing dangerous behavior discrimination method based on small target tracking in embodiment 1 of the present invention.
Detailed Description
The technical scheme of the invention is further described below with reference to the accompanying drawings and examples.
Example 1
As shown in fig. 1, the embodiment provides a pedestrian crossing dangerous behavior discrimination method based on small target tracking, which includes the following steps:
Step 1, acquiring video streams of a designated street crossing area under a overlooking view angle by using a camera in the designated street crossing area, and transmitting the acquired video streams to the following steps to execute operation;
the video format and frame number acquired by the camera are shown in the following table 1;
Table 1 video camera video stream acquisition table
The video can be continuously acquired by utilizing the road side camera, the height is moderate, pedestrians can be stably shot, and pre-training is easy to be carried out on the existing network weight.
And 2, reading the video image acquired in the step 1 frame by frame, performing frame by frame target detection on the image by using a pre-trained EFFICIENTDET convolution target detection network with a characteristic pyramid (FPN), and extracting target tracks of pedestrians and other traffic participants.
In a pre-trained data set, labeling a large number of anchor frame samples for pedestrians from a overlooking view, limiting fusion weights of FPN shallow features in EFFICIENTDET convolution target detection network design to ensure that the small target semantic features cannot be excessively lost, specially designing a network detection anchor frame, matching a lower anchor frame matching intersection ratio (IoU) threshold value for a small target object with a smaller anchor frame, setting higher loss for the small target, and further ensuring detection accuracy of the small target, wherein the method comprises the following steps of:
Step 21) calculating the ratio of the target real frame area to the small target area parameter, and calculating the target anchor frame matching intersection ratio IoU threshold and the loss expansion coefficient aiming at the small target object according to the ratio, wherein the specific method comprises the following steps:
Projecting the ratio of the target real frame area to the small target area parameter into a section formed by a given basic anchor frame threshold value and a small target anchor frame threshold value, calculating a target anchor frame matching intersection ratio IoU threshold value, projecting the inverse of the ratio of the target real frame area to the small target area parameter into a 0-1 section, multiplying the inverse by a coefficient, and then taking the inverse as a loss expansion coefficient for a small target object:
Wherein, For the calculated target anchor frame matching cross ratio IoU threshold,For a target anchor frame matching cross ratio IoU threshold for a normal size object,For a set small target anchor frame matching cross ratio IoU threshold lower limit,For the real frame area to be a real frame area,For the set small target frame area parameter, a is the amplification parameter,For loss of expansion coefficient, b is expansion parameter, c is amplification parameter, sigmoid is mathematical function for mapping arbitrary real number between (0, 1), and the image presents "S" type curve;
two key steps in the training process of EFFICIENTDET convolution target detection network are detection target matching and network weight updating respectively.
One key parameter in the process of detecting target matching is IoU threshold (cross-over threshold), and matching is considered successful only when the cross-over ratio of the detection frame and the target frame is larger than the threshold, and a smaller cross-over threshold is used for a small target compared with a normal target, so that the network can be better matched with the small target.
The basis of the network weight update is a loss function, and higher loss is given to the small targets, so that the network gets more punishment after the failure of detecting the small targets, and the small targets are detected as far as possible so as to optimize network parameters.
Step 22), after positions of pedestrians and other various traffic participants in the video image are acquired frame by frame, correlating targets of front and rear frames by utilizing DeepSORT target correlation tracking algorithm, and extracting motion tracks of the pedestrians and the other various traffic participants in the video by utilizing EFFICIENTDET convolution target detection network; the other various traffic participants comprise automobiles, trucks, bicycles, motorcycles and the like;
Step 23) selecting a checkerboard calibration plate with known size and shape, fixing a camera, shooting a plurality of images containing the calibration plate, extracting angular point coordinates on each image calibration plate by using an OpenCV image processing algorithm, corresponding the angular point coordinates with real world coordinates, and establishing a mapping relation between pixel coordinates and real coordinates; calculating internal and external parameters of a camera by using Zhang Zhengyou camera calibration algorithm to obtain a conversion matrix between a pixel coordinate system and a real coordinate system, wherein the pixel coordinate system and the real coordinate system have the following relationship:
Wherein the method comprises the steps of As a scale factor, the number of the elements is,AndRespectively the abscissa and the ordinate of the pixel coordinate system,，，Respectively a horizontal abscissa, a horizontal ordinate and a vertical coordinate of a world coordinate system,AndThe focal lengths of the cameras in the x and y directions respectively,Is a non-orthogonality factor between pixels, whereRespectively representing the coordinates of the center of the camera photosensitive plate under a pixel coordinate system,In order to rotate the matrix is rotated,Is a translation vector; representing an all 0 matrix of size 1*3, 3 in the formula representing 3 rows and T representing the transpose.
Calibrating a conversion matrix for a fixed camera, wherein the conversion matrix comprises an internal reference matrix M1 and an external reference matrix M2 of the camera:
And converting the track point coordinates of pedestrians and other various traffic participants into track point coordinates under a world coordinate system by utilizing the corresponding relation among the coordinate systems, uniformly sampling the running track of the vehicle according to n track points per second, and carrying out Kalman filtering denoising smoothing on the track position sequence coordinates of each vehicle to obtain smooth tracks of the pedestrians and other various traffic participants in the real world.
If the mapping relation between the pixel coordinate system and the real coordinate system is established without using a conversion matrix, the coordinate obtained after the tracking algorithm is operated is the pixel coordinate, and the coordinate is different from the real meaning of the pixel coordinate, and has a certain influence on the subsequent operation precision, so that the movement track of the traffic participant is projected into the real world coordinate system through the conversion matrix after the movement track of the traffic participant is extracted, and the subsequent operation precision is ensured.
The extracted trajectories of pedestrians and other traffic participants are shown in table 2:
TABLE 2 pedestrian and other traffic participant trajectory tables
The resulting track table is shown in table 3:
TABLE 3 track table
And step 3, calculating gait parameters of pedestrians. The pedestrian motion has periodicity, the motion acceleration is periodically changed according to the walking steps, the pedestrian track obtained in the step 2 is utilized to obtain the pedestrian instantaneous speed, the instantaneous speed curve is subjected to the time first-order difference to obtain the acceleration change curve, the walking step frequency is calculated by utilizing the power spectrum density, and the formula is as follows:
Wherein, For each frequencyIs used for the power spectral density estimation of (a),For the sampling frequency to be the same,As a discrete time sequence of the pedestrian walking process,For pedestrian instantaneous speed valuesDouble vertical lines represent the magnitude of the calculated vector, i.e., the magnitude value of the instantaneous speed; the length of the signal refers to the length of the pedestrian walking time sequence;
the frequency representative step frequency with the maximum power density is selected, the walking step speed of the pedestrian is obtained by dividing the step frequency, the gait parameters of the pedestrian within t2 seconds and the corresponding trajectories of the pedestrian and the traffic participant are taken out every t1 seconds for calculating in the step 4, and in the embodiment, t1 is 2 seconds and t2 is 4 seconds.
In the embodiment, the step frequency is calculated by using a power spectrum density method, and the step frequency and step speed relationship is used for calculating the step, and the power spectrum density method utilizes the wave property of the pedestrian acceleration change, so that the step frequency is obtained accurately by using the power wave, and meanwhile, the influence of some noise can be filtered.
The obtained pedestrian gait parameters are shown in table 4.
Table 4 pedestrian gait parameter table
In the table of the present invention,The step frequency is indicated as being a function of the step frequency,The step size is indicated as such,Indicating the pace.
Step 4, calculating collision indexes TTC of pedestrians and other traffic participants;
And 5, in the extracted t2 second track segment, calculating conflict indexes TTC between the pedestrian and N other traffic participants closest to the pedestrian, wherein the minimum conflict index TTC is used as the conflict index TTC used in the step 5, and the calculation method of the conflict index TTC is as follows:
Wherein, Is the TTC (Time to Time To Collision, time to collision) between pedestrian i and traffic participant j,Is the relative distance between the pedestrian and the other traffic participants,Is the projection of the relative speed of the pedestrian and other traffic participants on the position connecting line, and the relative distanceCan be found from the real world coordinates of the track points,The method is that the first-order difference of time is carried out on the tracks of pedestrians and other traffic participants, the speeds of the pedestrians and the other traffic participants are obtained, vector synthesis is carried out on the speeds, and then the speeds are projected on the connecting line of the pedestrians and the other traffic participants.
The larger the number is, the higher the calculation force requirement is, because pedestrians and a plurality of traffic participants in a picture can form an interactive relation, and the calculation is performed by screening units with closer distances, so that the calculation efficiency can be improved.
And 5, judging whether the pedestrian behavior is dangerous street crossing behavior or not through the pedestrian street crossing gait parameters calculated in the step 3 and the conflict indexes TTC of the pedestrians and other traffic participants calculated in the step 4, wherein the method comprises the following steps of:
Inputting pedestrian crossing gait parameters within t2 seconds into a pre-trained transducer time sequence attention neural network classifier, judging whether the change rule of the pedestrian crossing gait parameters accords with dangerous behavior characteristics or not, forming an evaluation index I R1, and carrying out statistical analysis on the pedestrian crossing gait parameters to judge whether the situation that the step frequency, the step length or the step speed is larger than a threshold value or the change amplitude is larger than the threshold value within a set time (such as 1 s) exists or not, and forming an evaluation index II R2; constructing a judging table by using the first evaluation index R1, the second evaluation index R2 and the conflict index TTC, judging whether the pedestrian crossing dangerous behavior occurs or not through the judging table, wherein the judging table is shown in the table 5:
TABLE 5 dangerous behavior discriminant
In the tables Thred _R_1, thred _R_2,、The first evaluation index threshold value, the second evaluation index threshold value, the first conflict index TTC threshold value and the second conflict index TTC threshold value are respectively represented.
When the dangerous behavior is judged to occur, video of a set time period (such as 10 s) before and after the dangerous behavior is intercepted is reserved as a criterion, and corresponding indexes including an evaluation index I R1, an evaluation index II R2 and a conflict index TTC are output.
In the embodiment, whether the pedestrian crossing gait parameters accord with dangerous behavior characteristics is judged by using the transducer time sequence attention neural network classifier, the transducer network can adaptively learn rules from data, the robustness and the accuracy are high, meanwhile, the judgment is carried out by combining conflict substitution indexes and gait objective rules, the comprehensiveness is strong, the robustness is high, and the influence on the environment is small.
The three indexes of the first evaluation index R1, the second evaluation index R2 and the conflict index TTC can be disassembled and judged according to engineering practice, the three indexes can be used for independently judging whether dangerous or not, the combination judgment accuracy of the three indexes is further improved, other networks can be used for judging, for example, LSTM is used for replacing a transducer, and PET can be used for replacing TTC on the indexes.
Example 2
The embodiment provides a pedestrian crossing dangerous behavior judging method based on small target tracking, which comprises the following steps:
step 1, acquiring a video stream of a specified street crossing area under a overlooking view angle by using a camera in the specified street crossing area;
Step 2, reading the video image acquired in the step 1 frame by frame, performing frame by frame target detection on the image by using a pre-trained EFFICIENTDET convolution target detection network with a characteristic pyramid (FPN), and extracting target tracks of pedestrians and other traffic participants;
Step 21), calculating the ratio of the target real frame area to the small target area parameter, and calculating the target anchor frame matching intersection ratio IoU threshold and the loss expansion coefficient aiming at the small target object according to the ratio;
step 22), after the positions of pedestrians and other various traffic participants in the video image are acquired frame by frame, the targets of the front and rear frames are associated by utilizing a nearest neighbor algorithm, a probability data association algorithm or a SORT algorithm, and the motion trail of the pedestrians and other various traffic participants in the video is extracted by utilizing a EFFICIENTDET convolution target detection network;
Step 23) selecting a checkerboard calibration plate with known size and shape, fixing a camera, shooting a plurality of images containing the calibration plate, extracting angular point coordinates on each image calibration plate by using an OpenCV image processing algorithm, corresponding the angular point coordinates with real world coordinates, and establishing a mapping relation between pixel coordinates and real coordinates; calculating internal and external parameters of the camera by using Zhang Zhengyou camera calibration algorithm to obtain a conversion matrix between the pixel coordinate system and the real coordinate system,
Step 3, obtaining the instantaneous speed of the pedestrian by utilizing the pedestrian track obtained in the step 2 and performing time first-order difference on the instantaneous speed curve to obtain an acceleration change curve, and calculating the walking frequency of the pedestrian by utilizing the power spectrum density;
Step 4, calculating collision indexes TTC of pedestrians and other traffic participants;
And 5, judging whether the pedestrian behavior is dangerous street crossing behavior or not through the pedestrian crossing gait parameters calculated in the step 3 and the conflict indexes TTC of the pedestrians and other traffic participants calculated in the step 4.
Example 3
The embodiment provides a pedestrian crossing dangerous behavior judging method based on small target tracking, which comprises the following steps:
step 1, acquiring a video stream of a specified street crossing area under a overlooking view angle by using a camera in the specified street crossing area;
Step 2, reading the video image acquired in the step 1 frame by frame, performing frame by frame target detection on the image by using a yolov, yolov or fastrcnn method, and extracting target tracks of pedestrians and other traffic participants;
Step 3, obtaining the instantaneous speed of the pedestrian by utilizing the pedestrian track obtained in the step 2 and performing time first-order difference on the instantaneous speed curve to obtain an acceleration change curve, and calculating the walking frequency of the pedestrian by utilizing the power spectrum density;
Step 4, calculating collision indexes TTC of pedestrians and other traffic participants;
And 5, judging whether the pedestrian behavior is dangerous street crossing behavior or not through the pedestrian crossing gait parameters calculated in the step 3 and the conflict indexes TTC of the pedestrians and other traffic participants calculated in the step 4.
Example 4
A computer system comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to carry out the steps of the above method.
Example 5
A computer readable storage medium having stored thereon a computer program/instruction, characterized in that the computer program/instruction, when executed by a processor, implements the steps of the above method.
Example 6
A computer program product comprising computer programs/instructions which when executed by a processor implement the steps of the above method.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (9)
1. The pedestrian crossing dangerous behavior distinguishing method based on small target tracking is characterized by comprising the following steps of:
step 1, acquiring a video stream of a specified street crossing area under a overlooking view angle by using a camera in the specified street crossing area;
Step 2, reading the video image acquired in the step 1 frame by frame, detecting the frame by frame target of the image, and extracting target tracks of pedestrian targets and other traffic participants, wherein the method comprises the following steps:
Step 21), calculating the ratio of the target real frame area to the small target area parameter, and calculating the target anchor frame matching intersection ratio IoU threshold and the loss expansion coefficient aiming at the small target object according to the ratio;
Step 22), after positions of pedestrians and other various traffic participants in the video image are acquired frame by frame, correlating targets of front and rear frames by utilizing DeepSORT target correlation tracking algorithm, and extracting motion tracks of the pedestrians and the other various traffic participants in the video by utilizing EFFICIENTDET convolution target detection network; the other various traffic participants comprise automobiles, trucks, bicycles and motorcycles;
step 23) selecting a checkerboard calibration plate with known size and shape, fixing a camera, shooting a plurality of images containing the calibration plate, extracting angular point coordinates on each image calibration plate by using an OpenCV image processing algorithm, corresponding the angular point coordinates with real world coordinates, and establishing a mapping relation between pixel coordinates and real coordinates; calculating internal and external parameters of the camera by using Zhang Zhengyou camera calibration algorithm to obtain a conversion matrix between a pixel coordinate system and a real coordinate system;
Converting the track point coordinates of pedestrians and other various traffic participants into track point coordinates under a world coordinate system by utilizing the corresponding relation among the coordinate systems, uniformly sampling the running track of the vehicle according to n track points per second, and carrying out Kalman filtering denoising smoothing on the track position sequence coordinates of each vehicle to obtain smooth tracks of the pedestrians and other various traffic participants in the real world;
Step 3, obtaining the instantaneous speed of the pedestrian by utilizing the pedestrian track obtained in the step2 and the time first-order difference, obtaining an acceleration change curve by performing the time first-order difference on the instantaneous speed curve, and calculating the walking frequency of the pedestrian by utilizing the power spectrum density;
Step 4, calculating collision indexes TTC of pedestrians and other traffic participants;
And 5, judging whether the pedestrian behavior is dangerous street crossing behavior or not through the pedestrian crossing gait parameters calculated in the step 3 and the conflict indexes TTC of the pedestrians and other traffic participants calculated in the step 4.
2. The pedestrian crossing dangerous behavior judging method based on small target tracking according to claim 1, wherein in step 2, a pre-trained EFFICIENTDET convolution target detection network with a characteristic pyramid is used for carrying out frame-by-frame target detection on images, and pedestrian targets and other traffic participant target tracks are extracted.
3. The pedestrian crossing dangerous behavior discrimination method based on small target tracking according to claim 1, wherein in step 21), the specific method is as follows:
Projecting the ratio of the target real frame area to the small target area parameter into a section formed by a given basic anchor frame threshold value and a small target anchor frame threshold value, calculating a target anchor frame matching intersection ratio IoU threshold value, projecting the inverse of the ratio of the target real frame area to the small target area parameter into a 0-1 section, multiplying the inverse by a coefficient, and then taking the inverse as a loss expansion coefficient for a small target object:
，
，
Wherein, For the calculated target anchor frame matching cross ratio IoU threshold,For a target anchor frame matching cross ratio IoU threshold for a normal size object,For a set small target anchor frame matching cross ratio IoU threshold lower limit,For the real frame area to be a real frame area,For the set small target frame area parameter, a is the amplification parameter,To lose the expansion coefficient, b is the expansion parameter, c is the amplification parameter, sigmoid is a mathematical function for mapping any real number between (0, 1).
4. The pedestrian crossing dangerous behavior discrimination method based on small target tracking according to claim 1, wherein in step 23), there is a relationship between a pixel coordinate system and a real coordinate system as follows:
，
Wherein, As a scale factor, the number of the elements is,AndRespectively the abscissa and the ordinate of the pixel coordinate system,，，Respectively a horizontal abscissa, a horizontal ordinate and a vertical coordinate of a world coordinate system,AndThe focal lengths of the cameras in the x and y directions respectively,Is a non-orthogonal factor between pixels,Respectively representing the coordinates of the center of the camera photosensitive plate under a pixel coordinate system,In order to rotate the matrix is rotated,Is a translation vector; representing an all 0 matrix of size 1*3, 3 in the formula representing 3 rows and T representing the transpose.
5. The pedestrian crossing dangerous behavior discrimination method based on small target tracking according to claim 1, wherein in step 3, a formula for calculating a pedestrian walking frequency is as follows:
，
Wherein, For each frequencyIs used for the power spectral density estimation of (a),For the sampling frequency to be the same,As a discrete time sequence of the pedestrian walking process,For pedestrian instantaneous speed valuesDouble vertical lines represent the magnitude of the calculated vector, i.e., the magnitude value of the instantaneous speed; the length of the signal refers to the length of the pedestrian walking time sequence;
Selecting a frequency representative step frequency with the maximum power density, dividing the step frequency by the pedestrian step speed to obtain the pedestrian walking steps, wherein each step is Second take outThe pedestrian gait parameters in seconds and the corresponding pedestrian trajectory and traffic participant trajectory are used for the calculation of step 4.
6. The pedestrian crossing dangerous behavior discrimination method based on small target tracking according to claim 1, wherein in step 4, in the extractedAnd 5, in the second track segment, calculating conflict indexes TTC between the pedestrian and N other traffic participants closest to the pedestrian, wherein the minimum conflict index TTC is used as the conflict index TTC used in the step 5, and the calculation method of the conflict index TTC is as follows:
，
Wherein, Is the TTC between pedestrian i and traffic participant j,Is the relative distance between the pedestrian and the other traffic participants,Is the projection of the relative speed of the pedestrian and other traffic participants on the position connecting line, and the relative distanceObtained by real world coordinates of the track points,The method is that the first-order difference of time is carried out on the tracks of pedestrians and other traffic participants, the speeds of the pedestrians and the other traffic participants are obtained, vector synthesis is carried out on the speeds, and then the speeds are projected on the connecting line of the pedestrians and the other traffic participants.
7. The pedestrian crossing dangerous behavior discrimination method based on small target tracking according to claim 1, wherein in step 5, the following is performedInputting pedestrian crossing gait parameters in seconds into a pre-trained transducer time sequence attention neural network classifier, judging whether the variation rule of the pedestrian crossing gait parameters accords with dangerous behavior characteristics, and forming an evaluation index R1; in addition, statistical analysis is carried out on pedestrian crossing gait parameters, whether the step frequency, the step width or the step speed is larger than a threshold value or the variation amplitude in set time is larger than the threshold value is judged, and an evaluation index II R2 is formed; and constructing a judging table by using the first evaluation index R1, the second evaluation index R2 and the conflict index TTC, and judging whether the pedestrian crossing dangerous behavior occurs or not through the judging table.
8. The pedestrian crossing dangerous behavior judging method based on small target tracking according to claim 7, wherein in step 5, when judging that dangerous behavior occurs, capturing videos of a set period of time before and after the occurrence of behavior is left as a criterion, and outputting corresponding indexes including an evaluation index one R1, an evaluation index two R2 and a collision index TTC.
9. A computer system comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to carry out the steps of the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410567563.2A CN118155293B (en) | 2024-05-09 | 2024-05-09 | Pedestrian crossing dangerous behavior discrimination method and system based on small target tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410567563.2A CN118155293B (en) | 2024-05-09 | 2024-05-09 | Pedestrian crossing dangerous behavior discrimination method and system based on small target tracking |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118155293A CN118155293A (en) | 2024-06-07 |
| CN118155293B true CN118155293B (en) | 2024-07-05 |
Family
ID=91295141
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410567563.2A Active CN118155293B (en) | 2024-05-09 | 2024-05-09 | Pedestrian crossing dangerous behavior discrimination method and system based on small target tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118155293B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109334563A (en) * | 2018-08-31 | 2019-02-15 | 江苏大学 | A kind of anticollision method for early warning based on road ahead pedestrian and bicyclist |
| CN115760921A (en) * | 2022-11-28 | 2023-03-07 | 昆明理工大学 | Pedestrian trajectory prediction method and system based on multi-target tracking |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10522040B2 (en) * | 2017-03-03 | 2019-12-31 | Kennesaw State University Research And Service Foundation, Inc. | Real-time video analytics for traffic conflict detection and quantification |
| CN109101859A (en) * | 2017-06-21 | 2018-12-28 | 北京大学深圳研究生院 | The method for punishing pedestrian in detection image using Gauss |
| CN117994987B (en) * | 2024-04-07 | 2024-06-11 | 东南大学 | Traffic parameter extraction method and related device based on target detection technology |
-
2024
- 2024-05-09 CN CN202410567563.2A patent/CN118155293B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109334563A (en) * | 2018-08-31 | 2019-02-15 | 江苏大学 | A kind of anticollision method for early warning based on road ahead pedestrian and bicyclist |
| CN115760921A (en) * | 2022-11-28 | 2023-03-07 | 昆明理工大学 | Pedestrian trajectory prediction method and system based on multi-target tracking |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118155293A (en) | 2024-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108062349B (en) | Video monitoring method and system based on video structured data and deep learning | |
| CN112669349B (en) | Passenger flow statistics method, electronic equipment and storage medium | |
| CN102073851A (en) | Method and system for automatically identifying urban traffic accident | |
| CN110633643A (en) | Abnormal behavior detection method and system for smart community | |
| CN108564598B (en) | Improved online Boosting target tracking method | |
| CN114267082B (en) | Bridge side falling behavior identification method based on depth understanding | |
| Jadhav et al. | Aerial multi-object tracking by detection using deep association networks | |
| CN112183355A (en) | Effluent height detection system and method based on binocular vision and deep learning | |
| CN115861383B (en) | Pedestrian track prediction device and method for multi-information fusion in crowded space | |
| CN117994987B (en) | Traffic parameter extraction method and related device based on target detection technology | |
| CN112232240A (en) | Road sprinkled object detection and identification method based on optimized intersection-to-parallel ratio function | |
| CN104778699A (en) | Adaptive object feature tracking method | |
| CN116311166A (en) | Traffic obstacle recognition method and device and electronic equipment | |
| CN110533692B (en) | Automatic tracking method for moving target in aerial video of unmanned aerial vehicle | |
| CN118155293B (en) | Pedestrian crossing dangerous behavior discrimination method and system based on small target tracking | |
| CN117334040A (en) | Cross-domain road side perception multi-vehicle association method and system | |
| CN115100249B (en) | Intelligent factory monitoring system based on target tracking algorithm | |
| CN104537690B (en) | Moving point target detection method based on maximum value-time index combination | |
| CN114820723B (en) | Online multi-target tracking method based on joint detection and association | |
| CN110210324A (en) | A kind of road target quickly detects method for early warning and system | |
| CN115188081A (en) | Complex scene-oriented detection and tracking integrated method | |
| Lin et al. | Incorporating appearance and edge features for vehicle detection in the blind-spot area | |
| CN114677647A (en) | Safety monitoring system and method for road running state | |
| Mishra et al. | Occlusion handling strategies for multiple moving object classification | |
| CN112164097A (en) | Ship video detection sample acquisition method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |