CN113011331B - Method and device for detecting whether motor vehicle gives way to pedestrians, electronic equipment and medium - Google Patents

Abstract

The invention is applicable to the technical field of intelligent transportation, and provides a method, device, electronic device and storage medium for detecting whether a motor vehicle is courteous to pedestrians. Element identification, obtain the target element set, use the preset target tracking algorithm to track the motion track of the target element in the target element set, based on the identified main crosswalk line and the tracked motion track of the target element, use The preset illegal behavior detection algorithm detects whether there is a violation of the vehicle inappropriately yielding to the pedestrian in the current video frame, thereby improving the detection accuracy of the motor vehicle inappropriately yielding to the pedestrian.

Description

Method, device, electronic device and medium for detecting whether a motor vehicle yields to pedestrians

technical field

The invention belongs to the technical field of intelligent transportation, and in particular relates to a method, device, electronic device and storage medium for detecting whether a motor vehicle is courteous to a pedestrian.

Background technique

Smart transportation is an inevitable product of the development of electronics, computers, automation and other technologies to a certain extent. In recent years, my country has intensively issued a number of policies to support the development of smart transportation. Developed countries in the world such as the United States, Europe and other regions have generally applied smart transportation to their transportation construction. As the number of motor vehicles in my country tends to be saturated, the use of intelligent transportation systems to improve the close cooperation between people, vehicles and roads to improve transportation efficiency, ease traffic congestion, improve road network passing capacity, and reduce traffic accidents has become my country's traffic management department. Urgent problems. The traditional traffic violation detection mainly confirms whether the vehicle behavior violates the traffic rules through multiple rounds of manual review and verification of the vehicle information captured by the camera. This method consumes a lot of time and labor costs, and the manual review method is mixed with many subjective factors such as auditor fatigue and emotion, which makes the review efficiency low, and the results lack fairness and accuracy.

Even if many cities are promoting intelligent traffic at high speed, and the detection and supervision of simple violations such as running red lights, pressing lines, and occupying non-motorized vehicle lanes can be automated, how to realize complex comity pedestrian behaviors under the premise of saving labor, equipment, and technology costs The efficient, fast, accurate and fair automated detection of

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method, device, electronic device and storage medium for detecting whether a motor vehicle is courteous to a pedestrian, aiming to solve the problem that the detection accuracy of whether a motor vehicle is courteous to a pedestrian is not high enough in the prior art.

In one aspect, the present invention provides a method for detecting whether a motor vehicle yields to a pedestrian, the method comprising the following steps:

Use the trained target recognition network to perform target element recognition on the collected video frames to obtain a target element set;

Use a preset target tracking algorithm to track the motion trajectory of the target element in the target element set;

Based on the identified main crosswalk line and the tracked movement trajectory of the target element, a preset violation detection algorithm is used to detect whether there is a violation of a motor vehicle disobeying a pedestrian in the current video frame.

Preferably, the target recognition network is an SSD MobileNet-v2 network.

Preferably, before the step of using the trained target recognition network to perform target element recognition on the collected video frames, the method further includes:

If the preset crosswalk line recognition conditions are met, the main crosswalk line is recognized, wherein the crosswalk line recognition conditions include that the current video frame is the first frame of the video, or the current video frame is a video corresponding to a preset crosswalk line recognition period frame, and/or the frame interval between the current video frame and the previous video frame in which the crosswalk line is not identified is equal to the preset first interval threshold.

Preferably, the first interval threshold is 10.

Preferably, the step of identifying the main crosswalk line further includes:

Use the trained segmentation network to segment the crosswalk line instance on the current video frame to obtain at least one coarsely positioned crosswalk detection frame;

The main crosswalk line is finely positioned based on all of the coarsely positioned crosswalk detection frames.

Preferably, the segmentation network is a Mask R-CNN network, and the step of training the Mask R-CNN network includes:

Get the lane line dataset;

If the lane markings in the lane marking data set only include the side contours of the lane markings, the lane markings are preprocessed to obtain a preprocessing lane marking data set, wherein the lane markings include crosswalk markings, The preprocessed lane markings contain segmentation masks;

The preprocessed lane line data set is input into the Mask R-CNN network for training to obtain a trained Mask R-CNN network.

Preferably, the step of preprocessing the lane lines in the lane line data set further includes:

If the marking of the lane line contains only one line, take the line as the axis and spread the preset pixel width on both sides;

If the marking of the lane line includes two straight lines, the head ends and the ends of the two lines are respectively connected to form a quadrilateral closed area, and the quadrilateral closed area is filled according to the category of the lane line.

Preferably, the pixel width is 5 pixels wide.

Preferably, the step of finely locating the main crosswalk line based on all the roughly positioned crosswalk detection frames includes:

Based on the confidence of each of the coarsely positioned crosswalk detection frames, a preset conditional scoring algorithm is used to screen the main crosswalk line from all of the coarsely positioned crosswalk detection frames, wherein the conditional scoring algorithm combines the Coarsely locate the position of the pedestrian crossing detection frame in the video frame and/or the ratio of the coarsely positioned pedestrian crossing detection frame relative to the video frame.

Preferably, the step of screening the main crosswalk line from all the roughly positioned crosswalk detection frames using a preset conditional scoring algorithm includes:

Use a preset conditional scoring algorithm to score each of the roughly positioned pedestrian crossing detection frames to obtain a conditional total score of each of the roughly positioned pedestrian crossing detection frames;

The coarse-positioned crosswalk detection frame with the highest total condition score is obtained, and the coarse-positioned crosswalk detection frame with the highest condition total score is used as the main crosswalk line detection frame.

Preferably, the formula used by the conditional scoring algorithm is as follows:

表示第i个所述粗定位人行横道检测框的条件总分，

分别表示第i个所述粗定位人行横道检测框的左上角相对当前视频帧左上角在水平方向和垂直方向的距离，

分别表示第i个所述粗定位人行横道检测框的宽度和高度，height_IMG表示视频帧的高度，

表示第i个所述粗定位人行横道检测框的置信度，I()为指示函数，(R1_t/h，R2_t/h)表示第一预设区间，(R1_w/h，R2_w/h)表示第二预设区间，s1表示第一预设固定值，s2表示第二预设固定值。in,

represents the conditional total score of the i-th coarsely positioned pedestrian crossing detection frame,

respectively represent the distance between the upper left corner of the i-th coarsely positioned crosswalk detection frame relative to the upper left corner of the current video frame in the horizontal direction and the vertical direction,

respectively represent the width and height of the i-th coarsely positioned crosswalk detection frame, height _IMG represents the height of the video frame,

Represents the confidence of the i-th coarsely positioned pedestrian crossing detection frame, I() is the indicator function, (R1 _t/h , R2 _t/h ) represents the first preset interval, (R1 _w/h , R2 _{w/h )} ) represents the second preset interval, s1 represents the first preset fixed value, and s2 represents the second preset fixed value.

Preferably, the first preset interval is (0.4, 0.6), the first preset fixed value is 0.4, the second preset interval is (0.07, 0.14), and the second preset fixed value is 0.14.

Preferably, after the step of obtaining the coarsely positioned pedestrian crossing detection frame with the highest condition total score, the method further includes:

Determine whether the width ratio of the coarse-positioned crosswalk detection frame with the highest conditional total score to the video frame is less than a preset width ratio threshold;

If it is less than, expand the coarse-positioned crosswalk detection frame with the highest conditional total score to both sides at equal distances, so that the width ratio of the coarse-positioned crosswalk detection frame with the highest conditional total score to the video frame reaches the width ratio threshold , taking the expanded crosswalk detection frame with the highest conditional total score as the main crosswalk line detection frame;

If it is not less than, the roughly positioned pedestrian crossing detection frame with the highest total condition score is used as the main pedestrian crossing line detection frame.

Preferably, the width ratio threshold is 0.85.

Preferably, the target element set includes a category and a detection frame of each target element, the target tracking algorithm is a maximum intersection ratio screening method, and the target in the target element set is tracked by using a preset target tracking algorithm The steps of the motion trajectory of the element, including:

For each target element in the target element set, a first sequence box list set is obtained according to all current tracking path sequence box lists, wherein the first sequence box list set includes all the target elements related to the target element. A list of tracking path sequence boxes of the same category, each of the tracking path sequence box lists contains the category of the tracked target element and the detection boxes of the target element arranged in chronological order, each of the tracking path sequence box lists Corresponding to the motion trajectory of a target element;

Calculate the intersection ratio of the detection frame of the target element and the last detection frame of each tracking path sequence frame list in the first sequence frame list set;

Judging whether there is an intersection and ratio target frame, the intersection and ratio target frame is the final detection frame whose intersection ratio is greater than a preset intersection ratio threshold;

If it exists, then determine the motion track tracked to the target element, and add the detection frame of the target element to the list of tracking path sequence frames where the last detection frame with the largest intersection ratio is located;

If it does not exist, it is determined that the target element is a newly appeared element, and a tracking path sequence frame list is created according to the detection frame and category of the target element.

Preferably, the cross-union ratio threshold is 0.75.

Preferably, before the step of obtaining the first sequence box list set according to all the current tracking path sequence box lists, the method further includes:

Obtain the second frame interval of the current video frame and the video frame corresponding to the last detection frame in each of the current tracking path sequence frame lists;

The list of tracking path sequence boxes whose second frame interval is not less than the preset second interval threshold is discarded.

Preferably, the second interval threshold is five.

Preferably, after the step of judging whether there is a final detection frame with an intersection ratio greater than a preset intersection ratio threshold, the method further includes:

If there is the cross-union ratio target frame, the largest cross-union ratio and the second largest cross-union ratio are subjected to a logarithmic test;

If the logarithm test is passed, then determine the movement track that is tracked to the target element;

If the logarithm test is not passed, the preset cosine distance comparison method is used to track the motion trajectory of the target element;

The use formula of the logarithmic test is:

(log ₂ IoU _max -log ₂ IoU _{second_max} )＜ε

Among them, IoU _max represents the maximum cross-union ratio, IoU _{second_max} represents the second largest cross-union ratio, and ε is a constant, which is expressed as a preset logarithmic test threshold.

Preferably, the step of using a preset cosine distance comparison method to track the motion trajectory of the target element includes:

Obtain the latest feature map of the target element in the current video frame, and the first feature map and the first feature map corresponding to the last detection frame and the second last detection frame of each of the tracking path sequence frame lists in the second sequence frame list set respectively. Two feature maps, wherein the second sequence box list set includes all the tracking path sequence box lists where the intersection and ratio target boxes are located;

Normalizing the latest feature map and all the first feature maps and the second feature maps to obtain feature maps of uniform size;

Calculate the first cosine distance and the second cosine distance of the latest normalized feature map and each of the first feature maps and each of the second feature maps respectively;

For each of the first cosine distances and each of the second cosine distances, take the logarithm to obtain the corresponding first feature similarity factor and second feature similarity factor;

Perform linear weighting calculation on each of the first feature similarity factors and the second feature similarity factors corresponding to each of the first feature similarity factors to obtain a minimum weighted feature similarity factor;

Taking the motion trajectory corresponding to the tracking path sequence box list corresponding to the minimum weighted feature similarity factor as the motion trajectory of the target element, and adding the detection frame of the target element to the minimum weighted feature similarity factor corresponding to the in the Trace Path sequence box list.

Preferably, the preset illegal behavior detection algorithm is a violation warning point algorithm, and the step of using the preset illegal behavior detection algorithm to detect whether there is an illegal behavior of a motor vehicle disrespecting pedestrians in the current video frame includes:

Predict the violation warning point in the current video frame according to the motion trajectory of each pedestrian on the main crosswalk;

Determine whether the violation warning point is within the upper half area of the last detection frame of any identified motor vehicle;

If so, it is determined that the violation is detected in the current video frame.

Preferably, the calculation method of the violation warning point is:

分别表示行人的跟踪路径序列框列表中第m个检测框中心点的坐标，c表示行人的跟踪路径序列框列表中检测框的个数。Wherein, x _wp and y _wp respectively represent the coordinates of the violation warning point,

respectively represent the coordinates of the center point of the mth detection box in the pedestrian tracking path sequence box list, and c represents the number of detection boxes in the pedestrian tracking path sequence box list.

Preferably, before the step of predicting the violation warning point in the current video frame according to the movement trajectory of each pedestrian on the main crosswalk, the method includes:

Determine whether each pedestrian in the current video frame is on the main crosswalk line;

The step of judging whether each pedestrian in the current video frame is on the main crosswalk line includes:

Determine whether the center point of the lower edge of the pedestrian detection frame of the pedestrian is within the pedestrian crossing detection frame;

If it is within the pedestrian crossing detection frame, it is determined that the pedestrian is on the pedestrian crossing line.

Preferably, after the step of using a preset illegal behavior detection algorithm to detect whether there is an illegal behavior of the motor vehicle disobeying pedestrians in the current video frame, the method further includes:

If the violation is detected, use a preset license plate recognition algorithm to identify the license plate of the motor vehicle where the violation occurs;

A visualization shows the license plate, the violation, and/or the time when the violation occurred.

In another aspect, the present invention provides a device for detecting whether a motor vehicle yields to a pedestrian, the device comprising:

The target element recognition unit is used to use the trained target recognition network to perform target element recognition on the collected video frames to obtain a target element set;

a motion trajectory tracking unit, used for tracking the motion trajectory of the target element in the target element set using a preset target tracking algorithm; and

The illegal behavior identification unit is used to detect, based on the identified main crosswalk line and the tracked motion trajectory of the target element, whether there is an illegal behavior of a motor vehicle rudely yielding to pedestrians in the current video frame by using a preset illegal behavior detection algorithm .

In another aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor is implemented when the processor executes the computer program The steps of the method as described above.

In another aspect, the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the steps of the above-mentioned method.

The present invention uses a trained target recognition network to perform target element recognition on the collected video frames to obtain a target element set, and uses a preset target tracking algorithm to track the motion trajectory of the target element in the target element set. The main pedestrian crossing line and the tracked motion trajectory of the target element are used to detect whether there is a violation of the vehicle's rude pedestrians in the current video frame by using the preset violation detection algorithm, thereby improving the detection accuracy of the vehicle's rude pedestrians. .

Description of drawings

1A is a flow chart of the realization of a method for detecting whether a motor vehicle yields to a pedestrian according to Embodiment 1 of the present invention;

FIG. 1B is an exemplary rendering of the target element identification of the collected video frame provided by Embodiment 1 of the present invention;

Fig. 2 is the realization flow chart of identifying the main crosswalk line provided by the second embodiment of the present invention;

Fig. 3 is the realization flow chart of training Mask R-CNN network provided by the third embodiment of the present invention;

Fig. 4 is the realization flow chart of using the maximum intersection ratio screening method to track the motion trajectory of the target element provided by the fourth embodiment of the present invention; And

FIG. 5 is a flow chart of the implementation of tracking the motion trajectory of the target element by using the cosine distance comparison method according to Embodiment 5 of the present invention.

6A is a flowchart of an implementation of using the violation warning point algorithm to detect the violation behavior of a motor vehicle in disrespect to pedestrians provided by Embodiment 6 of the present invention;

FIG. 6B is an exemplary diagram of using the violation warning point algorithm to detect whether there is a violation of a motor vehicle disobeying pedestrians in a current video frame according to Embodiment 6 of the present invention;

7 is a schematic structural diagram of a device for detecting whether a motor vehicle yields to a pedestrian according to Embodiment 7 of the present invention; and

FIG. 8 is a schematic structural diagram of an electronic device according to Embodiment 5 of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

Example 1:

1A shows the implementation process of the method for detecting whether a motor vehicle yields to pedestrians provided by the first embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown, and the details are as follows:

In step S101, target element recognition is performed on the collected video frame by using the trained target recognition network to obtain a target element set.

The embodiments of the present invention are applicable to electronic devices, specifically, to surveillance cameras, hard disk video recorders connected to surveillance cameras, computers, servers, and other electronic devices. In the embodiment of the present invention, the video at the intersection is collected by a camera disposed at the intersection, the target elements may include motor vehicles, pedestrians, and traffic lights, and the target element set may include the category and detection frame of each identified target element, etc. .

The target recognition network can be a network such as R-CNN, Fast R-CNN, Faster R-CNN, Yolo or SSD, preferably, the target recognition network SSD MobileNet-v2 network to improve the recognition speed of target elements. When using the trained target recognition network to perform target element recognition on the collected video frames to obtain the target element set, specifically, the trained target recognition network can be used to forward the collected video frames, and in each The feature map with the highest confidence is selected from the anchor feature map group that is not a background category, and the category and detection frame of the target element corresponding to the selected feature map are obtained, and the target element is formed according to the category and detection frame of the target element. gather.

Before using the trained target recognition network to perform target element recognition on the collected video frames, the target recognition data set can be used to train the target recognition network to obtain a trained target recognition network. The target objects in the target recognition data set may include pedestrians, traffic lights, and motor vehicles, and the motor vehicles may further include categories such as motor vehicles, pedestrians, signal lights, trucks, buses, or motorcycles.

Before using the trained target recognition network to identify the target elements of the collected video frames, each video frame can be detected by crosswalk line. Usually responsible for one area, so the main crosswalk line of each video frame can be identified. Based on the position of the crosswalk line in each video frame, the area does not always change, and the viewing angle of the camera is usually fixed. Preferably, if the preset crosswalk is satisfied If the line identification conditions are met, the main crosswalk line is identified, so as to correct the position change of the main crosswalk line caused by external factors such as camera position and road line update in time without wasting computing resources. Wherein, the crosswalk line recognition conditions include that the current video frame is the first frame of the video, or the current video frame is a video frame corresponding to a preset crosswalk line recognition period, and/or the current video frame is the same as the previous one that was recognized but did not recognize the crosswalk line. The frame interval of the video frame is equal to the preset first interval threshold. Further preferably, the first interval threshold is 10, so as to meet the actual detection needs on the premise of reducing computing resources.

Specifically, a pedestrian crossing line recognition period can be preset, and when the current video frame is the first frame of the video, the main pedestrian crossing line recognition is performed, and then the main pedestrian crossing line recognition is periodically performed according to the set pedestrian crossing line recognition period. During the process, if the frame interval between the current video frame and the previous video frame for which the crosswalk line is not recognized is equal to the first interval threshold (for example, 10), the crosswalk line identification is performed on the current video frame. Among them, the previous video frame that should be recognized but not recognized the crosswalk line can be understood as a video frame that needs to perform the crosswalk line but has not recognized the crosswalk line, and the crosswalk line recognition cycle can be set by the user, for example, once every 10 seconds Crosswalk line recognition, or crosswalk line recognition every 30 frames, of course, the crosswalk line recognition period can be comprehensively determined by the acquired current camera type, cruise path and/or cruise period. For example, for a camera without a gimbal such as a fixed gun, based on the fact that the shooting area of the camera is usually unchanged, a relatively large pedestrian crossing line recognition cycle can be set; The camera can be rotated to change the shooting area. If the camera starts automatic cruise, the pedestrian crossing line recognition cycle can be comprehensively determined according to the cruise time and the cruise path.

FIG. 1B is an example diagram of target element recognition on the collected video frame. The target elements identified in FIG. 1B include the detection frame and confidence of the identified pedestrian, the detection frame and confidence of the identified motor vehicle, and the detected frame and confidence of the identified motor vehicle. The identified main crosswalk detection box.

In step S102, a preset target tracking algorithm is used to track the motion trajectory of the target element in the target element set.

In this embodiment of the present invention, the target tracking algorithm may use a related algorithm for pedestrian and vehicle re-identification, such as ReID algorithm, SOTA algorithm, PROVID algorithm, etc., for each target element in the identified current video frame, preferably, use The preset maximum intersection ratio screening method tracks the motion trajectories of the target elements in the target element set. Through the calculation amount of linear time complexity, the huge amount of calculation required for the forward propagation of the relocation algorithm is eliminated, and the detection is significantly improved. efficiency, and further improve the detection accuracy. For the specific implementation of tracking the motion trajectory of the target element in each target element set using the preset maximum intersection ratio screening method, reference may be made to the relevant description of the fourth embodiment.

In step S103 , based on the identified main pedestrian crossing line and the tracked movement trajectory of the target element, a preset illegal behavior detection algorithm is used to detect whether there is a violation of the motor vehicle disobeying pedestrians in the current video frame.

In the embodiment of the present invention, the illegal behavior detection algorithm may be a related algorithm by judging whether the paths of the pedestrian and the motor vehicle intersect, and the illegal behavior detection algorithm algorithm is used to detect whether there is a violation of the motor vehicle disrespecting the pedestrian in the current video frame. Behavior, preferably, the violation behavior detection algorithm is a violation warning point algorithm, so as to detect whether there is a violation behavior of a motor vehicle disrespecting pedestrians in the current video frame through the violation warning point. The specific implementation manner of using the violation warning point algorithm to detect the violation behavior of the motor vehicle disobeying pedestrians can refer to the relevant description of the sixth embodiment, and will not be repeated here.

After detecting whether there is a violation of the motor vehicle disrespecting pedestrians in the current video frame, preferably, if the violation is detected, a preset license plate recognition algorithm is used to identify the license plate of the motor vehicle in which the violation occurs, and the violation is displayed visually. The license plate, the violation, and/or the time the violation occurred, to visualize the violation. The preset license plate recognition algorithm may use a license plate recognition algorithm such as HyperLPR.

In the embodiment of the present invention, a trained target recognition network is used to perform target element recognition on the collected video frames to obtain a target element set, and a preset target tracking algorithm is used to track the motion trajectory of the target element in the target element set, Based on the identified main crosswalk line and the tracked motion trajectory of the target element, detect whether there is a violation of the vehicle in the current video frame that is not polite to pedestrians,

Embodiment 2:

FIG. 2 shows the implementation process of identifying the main crosswalk line provided by the second embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown, and the details are as follows:

In step S201, use the trained segmentation network to segment the current video frame as a crosswalk line instance to obtain at least one coarsely positioned crosswalk detection frame.

In the embodiment of the present invention, a trained segmentation network is used to segment the current video frame as a crosswalk line instance to obtain at least one coarsely positioned crosswalk detection frame. The above-mentioned segmentation network can be a network such as R-CNN, Fast R-CNN or Faster R-CNN. Split result. The specific implementation of training the Mask R-CNN network may correspond to the relevant description in the third embodiment.

Based on the prior knowledge that the camera angle of view is fixed and the road line is relatively stable, it runs once when the Mask R-CNN network is initialized. In the running result, all coarsely positioned pedestrian crossing detection frames are screened and saved. Further, the confidence level of each roughly positioned pedestrian crossing detection frame can also be obtained. An example of the data format for each coarsely positioned pedestrian crossing detection frame is as follows:

crossdata=[{"roi":{"rois":[400,300,240,50],"class_ids":1},"score":0.85,"index":0},{"roi":{"rois":[ 800, 100, 1000, 150], "class_ids": 1}, "score": 0.65, "index": 1}]

In step S202, the main crosswalk line is precisely positioned based on all the coarsely positioned crosswalk detection frames.

In this embodiment of the present invention, all coarsely positioned pedestrian crossing detection frames may be sorted according to the confidence of each coarsely positioned pedestrian crossing detection frame, and the pedestrian crossing detection frame with the highest confidence may be used as the main pedestrian crossing detection frame. Coarsely locate the confidence level of the crosswalk detection frame, and use a preset conditional scoring algorithm to screen the main crosswalk line from all the coarsely positioned crosswalk detection frames. The conditional scoring algorithm combines the position of the coarsely positioned crosswalk detection frame in the video frame and/or Or roughly locate the ratio of the crosswalk detection frame relative to the video frame, so as to more accurately screen out the main crosswalk line through the conditional scoring algorithm.

When using the preset conditional scoring algorithm to screen the main crosswalk lines from all the coarsely positioned crosswalk detection frames, preferably, the preset conditional scoring algorithm is used to score each coarsely positioned crosswalk detection frame to obtain each coarsely positioned crosswalk The conditional total score of the detection frame is obtained, and the coarsely positioned pedestrian crossing detection frame with the highest conditional total score is obtained, and the coarsely positioned pedestrian crossing detection frame with the highest conditional total score is used as the main pedestrian crossing line detection frame. The formula used by the conditional scoring algorithm is as follows:

表示第i个粗定位人行横道检测框的条件总分，

分别表示第i个粗定位人行横道检测框的左上角相对当前视频帧左上角在水平方向和垂直方向的距离，

分别表示第i个粗定位人行横道检测框的宽度和高度，width_IMG、height_IMG分别表示视频帧的宽度和高度，

表示第i个粗定位人行横道检测框的置信度，I()为指示函数，(R1_t/h，R2_t/h)表示第一预设区间，(R1_w/h，R2_w/h)表示第二预设区间，s1表示第一预设固定值，s2表示第二预设固定值。in,

represents the conditional total score of the i-th coarsely positioned pedestrian crossing detection frame,

respectively represent the distance between the upper left corner of the i-th coarsely positioned crosswalk detection frame relative to the upper left corner of the current video frame in the horizontal and vertical directions,

respectively represent the width and height of the i-th coarse positioning crosswalk detection frame, width _IMG and height _IMG respectively represent the width and height of the video frame,

Represents the confidence of the i-th coarsely positioned pedestrian crossing detection frame, I() is the indicator function, (R1 _t/h , R2 _t/h ) represents the first preset interval, (R1 _w/h , R2 _w/h ) represents In the second preset interval, s1 represents the first preset fixed value, and s2 represents the second preset fixed value.

Specifically, if the ratio of the distance from the upper edge of the coarsely positioned crosswalk detection frame to the upper edge of the video frame to the height of the video frame is within the first preset interval, then the condition of the coarsely positioned crosswalk detection frame is divided into the confidence level of the coarsely positioned crosswalk detection frame. plus the first preset fixed value, if the ratio of the height of the coarsely positioned crosswalk detection frame to the height of the video frame is within the second preset interval, then the condition of the coarsely positioned crosswalk line boundary box is divided into the coarsely positioned crosswalk line boundary The confidence level of the box plus the second preset fixed value, if the above two conditions are met at the same time, the conditions for roughly positioning the crosswalk line bounding box are the total confidence level of the roughly positioning crosswalk line bounding box plus the first preset fixed value. and a second preset fixed value.

Further preferably, the first preset interval is (0.4, 0.6), the first preset fixed value is 0.4, the second preset interval is (0.07, 0.14), and the second preset fixed value is 0.14, In order to set relevant values according to the experimental results, the accuracy of the main pedestrian crossing positioning is further improved.

After obtaining the coarse-positioned crosswalk detection frame with the highest total condition score, preferably, it is determined whether the width ratio of the coarse-positioned crosswalk detection frame with the highest conditional total score to the video frame is less than a preset width ratio threshold; The coarse-positioned crosswalk detection frame with the highest total score is extended equidistantly to both sides, so that the width ratio of the coarse-positioned crosswalk detection frame with the highest conditional total score to the video frame reaches the width ratio threshold, and the expanded coarsely positioned with the highest conditional total score The crosswalk detection frame is used as the main crosswalk line detection frame. If it is not less than, the coarsely positioned crosswalk detection frame with the highest conditional total score will be used as the main crosswalk line detection frame. Pedestrian prediction violation warning points. Further preferably, the width ratio threshold is 0.85, so as to be set according to the experimental results, thereby further improving the prediction accuracy of violation warning points.

In the embodiment of the present invention, a trained segmentation network is used to segment the current video frame for crosswalk line instances, to obtain at least one coarsely positioned crosswalk detection frame, and based on all the coarsely positioned crosswalk detection frames, the main crosswalk line is precisely positioned. When crossing a crosswalk, sort all the coarsely positioned crosswalk detection frames according to the confidence of each coarsely positioned crosswalk detection frame, and use a preset conditional scoring algorithm to screen the main crosswalk line from all the coarsely positioned crosswalk detection frames, so as to pass the condition The scoring algorithm more accurately screened out the main crosswalk lines.

Embodiment three:

FIG. 3 shows the implementation process of training the Mask R-CNN network provided by the third embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown, and the details are as follows:

In step S301, a data set of lane lines is acquired.

In the embodiment of the present invention, the lane lines in the lane line data set include pedestrian crossing lines, and the lane line data set may adopt a lane line data set such as BDD100K.

In step S302, if the lane markings in the lane marking data set only include the side contours of the lane markings, the lane markings are preprocessed to obtain a preprocessing lane marking data set, wherein the preprocessing lane markings are The markup contains the segmentation mask.

In the embodiment of the present invention, the marking mask based on Mask R-CNN training needs to include pixels inside the object. Therefore, when training the Mask R-CNN network, if the lane markings in the lane marking dataset only include lanes The side contour of the line, then the lane line is preprocessed to obtain the preprocessed lane line data set, and the preprocessed lane line data set is input into the Mask R-CNN network for training, and the trained Mask R is obtained. -CNN network to meet the training requirements of Mask R-CNN network.

When preprocessing the lane lines in the lane line data set, preferably, if the mark of the lane line contains only one line, take the line as the axis and spread the preset pixel width on both sides. The mark contains two straight lines, then connect the head and end of the two lines to form a quadrilateral closed area, and fill the quadrilateral closed area according to the category of the lane line, so that the preprocessed lane line The marker mask contains the pixels inside the object.

Further preferably, the pixel width is 5 pixels width, so as to set the pixel width according to the actual test result.

In step S103, the preprocessed lane line data set is input into the Mask R-CNN network for training, and the trained Mask R-CNN network is obtained.

Embodiment 4:

FIG. 4 shows the implementation process of tracking the motion trajectory of the target element using the maximum intersection ratio screening method provided by the fourth embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown, including:

In step S401, a first sequence box list set is obtained according to all current tracking path sequence box lists.

In this embodiment of the present invention, the first sequence box list set includes all tracking path sequence box lists that have the same category as the target element, and each tracking path sequence box list includes the category of the tracked target element and the time For the detection frames of the target element arranged in sequence, each tracking path sequence box list corresponds to a motion trajectory of a target element. In other words, all detection frames in each tracking path sequence box list constitute a motion trajectory of a target element.

Because in the target detection algorithm, a certain target may be missed in a certain frame, so that no new frame will be added to the bounding box sequence corresponding to the object at the corresponding moment of the frame, but if it is detected again in subsequent frames This object will cause the tracking of the same target to be interrupted, and it is also possible that the disappearance of the target will lead to the loss of valid frames. Therefore, preferably, before obtaining the first sequence box list set according to all the current tracking path sequence box lists, obtain the current video frame and the current frame. The second frame interval of the video frame corresponding to the last detection frame in each tracking path sequence frame list is discarded, and the tracking path sequence frame list whose second frame interval is not less than the preset second interval threshold is discarded. Specifically, the frame number of the corresponding video frame when the last detection frame is detected can be additionally recorded for each tracking path sequence frame list, and this frame is called the last valid frame, and there will be no missed detection and successful detection of the target. The video frame of the element is called a valid frame, and a second interval threshold is set to calculate the frame number difference between the current video frame and the last valid frame of each tracking path sequence box list. If the frame number difference is within the frame threshold, the tracking path sequence box list is still reserved, otherwise it is considered that the target no longer exists in the video, and the tracking path sequence box list is discarded.

Preferably, the second interval threshold is 5, so as to set the second interval threshold according to the experimental result, thereby improving the detection accuracy.

In step S402, the intersection ratio of the detection frame of the target element and the last detection frame of each tracking path sequence frame list in the first sequence frame list set is calculated.

In the embodiment of the present invention, the calculation formula of the intersection ratio is:

Among them, box _cur represents the detection frame of the target element, q represents the total number of the tracking path sequence box list in the first sequence box list, box _j represents the last detection frame of the jth tracking path sequence box list in the first sequence box list , Area represents the area of the detection frame.

In step S403, it is judged whether there is an intersection ratio target frame, if so, step S404 is performed, otherwise, step S405 is performed.

In this embodiment of the present invention, the cross-unit ratio target frame is the final detection frame whose cross-unit ratio is greater than a preset cross-unit ratio threshold. Preferably, the cross-unit ratio threshold is 0.75.

For two or more objects of the same category moving in the same direction or similar directions, the latest frame of one object is too close to the last valid frame of other objects, and the maximum intersection ratio cannot effectively combine these two or more objects. When the trajectory of the target is distinguished, after judging whether there is a final detection frame whose intersection ratio is greater than the preset intersection ratio threshold, preferably, if there is this intersection ratio target frame, then the maximum intersection ratio and the second The logarithm test is performed on the large intersection and union ratio. If the logarithm test is passed, step S404 is performed; otherwise, other algorithms such as similarity can be used to further track the motion trajectory of the target element.

The formula used to take the logarithmic test is:

(log ₂ IoU _max -log ₂ IoU _{second_max} )＜ε

Among them, IoU _max represents the maximum cross-union ratio, IoU _{second_max} represents the second largest cross-union ratio, and ε is a constant, representing the preset logarithmic test threshold.

Preferably, a preset cosine distance comparison method is used to track the movement trajectory of the target element, so as to track the movement trajectory of the target element through the cosine distance, and a specific implementation of the preset cosine distance comparison method to track the movement trajectory of the target element For the method, reference may be made to the relevant description of the fifth embodiment.

In step S404, determine the motion track tracked to the target element, and add the detection frame of the target element to the tracking path sequence frame list where the last detection frame with the largest intersection ratio is located.

In step S405, it is determined that the target element is a newly appearing element, and a tracking path sequence frame list is created according to the detection frame and category of the target element.

Embodiment 5:

The embodiment of the present invention is based on the fourth embodiment. FIG. 5 shows the implementation process of tracking the motion trajectory of the target element by using the cosine distance comparison method provided by the fifth embodiment of the present invention. Relevant sections, which include:

In step S501, obtain the latest feature map of the target element in the current video frame, and the first feature corresponding to the last detection frame and the second last detection frame of each tracking path sequence frame list in the second sequence frame list set respectively and the second feature map.

In the embodiment of the present invention, the feature map of the currently tracked target element in the current video frame is obtained, and for the convenience of description, the latest feature map is used to represent the feature map of the target element (target element to be tracked) in the current video frame, and Obtain the feature maps corresponding to the last detection frame and the second-end detection frame of each tracking path sequence frame list in the second sequence frame list set. For the convenience of description, the first feature map and the second feature map are used to represent each The feature maps corresponding to the last detection frame and the second last detection frame of the tracking path sequence frame list, the first feature map and the second feature map refer to the feature maps in the corresponding valid frames, wherein the second sequence frame list The set contains all the tracking path sequence box lists where the target frame of the intersection ratio is located. In other words, the second sequence box list set contains all the tracking paths where the last detection frame whose intersection ratio is greater than the preset intersection ratio threshold is located. Path sequence box list.

In step S502, normalize the latest feature map and all the first and second feature maps to obtain feature maps of uniform size.

In step S503, the first cosine distance and the second cosine distance between the latest normalized feature map and each first feature map and each second feature map are calculated respectively.

In the embodiment of the present invention, the cosine distance calculation formula is:

m＝1或2，k∈[1，t]

m=1 or 2, k∈[1,t]

Among them, t represents the number of tracking path sequence box lists in the third sequence box list, A represents the latest feature map after normalization processing; B _k1 represents the kth tracking path sequence box list in the third sequence box list at the end of the list The normalized first feature map corresponding to the detection frame, that is, B _k1 represents the k-th normalized first feature map; B _k2 represents the k-th tracking path sequence frame in the third sequence frame list The normalized second feature map corresponding to the detection frame at the end of the list, that is, B _k2 represents the k-th normalized second feature map, cos(θ) _k1 represents the latest feature map and the k-th feature map The cosine distance of the first normalized feature map, cos(θ) _k2 represents the cosine distance between the latest feature map and the kth normalized second feature map.

In step S504, logarithms are taken for each first cosine distance and each second cosine distance, respectively, to obtain the corresponding first feature similarity factor and second feature similarity factor.

In the embodiment of the present invention, the calculation formula of the feature similarity factor is:

表示第k个第一特征相似因子，

表示第k个第二特征相似因子。in,

represents the kth first feature similarity factor,

represents the kth second feature similarity factor.

In step S505, linearly weighted calculation is performed on each first feature similarity factor and the second feature similarity factor corresponding to each first feature similarity factor to obtain a minimum weighted feature similarity factor.

In the embodiment of the present invention, the calculation formula of the weighted minimum feature similarity factor is:

Among them, a represents the weight of the first feature similarity factor, b represents the weight of the second feature similarity factor, and the weight of the first feature similarity factor is greater than the weight of the second feature similarity factor, that is, a>b.

In step S506, the motion track corresponding to the tracking path sequence frame list corresponding to the minimum weighted feature similarity factor is taken as the motion track of the target element, and the detection frame of the target element is added to the minimum weighted feature similarity factor The corresponding trace path sequence box list.

In the embodiment of the present invention, the motion trajectory formed by the tracking path sequence box list corresponding to the minimum weighted feature similarity factor is the motion trajectory of the target element.

Embodiment 6:

FIG. 6A shows the implementation process of using the violation warning point algorithm to detect the violation behavior of the motor vehicle in disobeying pedestrians provided by the sixth embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown, including:

In step S601, the violation warning point in the current video frame is predicted according to the movement trajectory of each pedestrian on the main crosswalk.

In the embodiment of the present invention, the position of each pedestrian in the next video frame can be predicted based on the motion trajectory of each pedestrian, so as to determine the violation warning point according to the predicted position of the pedestrian. Predict the position where a frame appears, that is, the closer it is to the current frame, the greater the role it plays in the prediction. Further, each predictor can be amplified by taking the logarithm, so that the time factor can be The effect of the prediction result is more obvious, and finally the calculation result point (x _wp , y _wp ) is set as the violation warning point. Therefore, preferably, the calculation method of the violation warning point is:

分别表示行人的跟踪路径序列框列表中第m个检测框中心点的坐标，c表示行人的跟踪路径序列框列表中检测框的个数。Among them, x _wp and y _wp respectively represent the coordinates of the violation warning point,

respectively represent the coordinates of the center point of the mth detection box in the pedestrian tracking path sequence box list, and c represents the number of detection boxes in the pedestrian tracking path sequence box list.

In a specific implementation, one violation warning point can be predicted according to the motion trajectory of each pedestrian, that is, there may be multiple violation warning points in the current video frame.

Before predicting the violation warning point in the current video frame according to the movement trajectory of each pedestrian on the main crosswalk, preferably, it is determined whether each pedestrian in the current video frame is on the main crosswalk. When each pedestrian is on the main crosswalk line, it can be calculated according to the intersection ratio between the pedestrian detection frame and the crosswalk detection frame. If the intersection ratio is greater than zero, it is determined that the pedestrian who appears for the first time is on the crosswalk line. Preferably, determine Whether the center point of the lower edge of the pedestrian detection frame is within the pedestrian crossing detection frame, so as to further reduce the amount of computation in the judgment process. The formula used to judge whether the center point of the lower edge of the pedestrian detection frame is within the detection frame of the pedestrian crossing line is:

Among them, (x _p , y _p ) represent the coordinates of the upper left corner of the pedestrian detection frame, width _p , height _p represent the width and height of the pedestrian detection frame, respectively, left _pc , top _pc represent the upper left corner of the main crosswalk detection frame relative to the current The distance between the upper left corner of the video frame in the horizontal and vertical directions, width _pc and height _pc represent the width and height of the main crosswalk detection frame respectively, F(x) represents the prediction result, and the value of F(x) is 0 or 1, when F When (x)=0, it means that the center point of the lower edge of the pedestrian detection frame is within the crosswalk line detection frame. When F(x)=1, it means that the center point of the lower edge of the pedestrian detection frame is not within the crosswalk line detection frame.

In step S602, it is judged whether the violation warning point is within the upper half area of the last detection frame of any identified motor vehicle, if so, execute step S603, if not, execute step S604.

In step S603, it is determined that a violation is detected in the current video frame.

In step S604, it is determined that no violation behavior is detected in the current video frame.

FIG. 6B shows an example diagram of using the violation warning point algorithm to detect whether there is a violation behavior of a motor vehicle disobeying pedestrians in the current video frame. In FIG. 6B , the violation warning point is not located in the upper half area of the last detection frame of the motor vehicle , that is, there is no violation of vehicle disobedience to pedestrians in Figure 6B.

In the embodiment of the present invention, the predicted violation warning point in the current video frame is used to detect whether there is a violation of the motor vehicle in the current video frame. , the method can also be applied to the situation where the pedestrian is stationary and the pedestrian has passed, thereby further improving the recognition accuracy of illegal behavior and the scene generalization ability.

Embodiment 7:

Fig. 7 shows the structure of the device for detecting whether a motor vehicle yields to pedestrians provided by the seventh embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown, including:

The target element identification unit 71 is used for using the trained target identification network to carry out target element identification on the collected video frame to obtain a target element set;

A motion track tracking unit 72, configured to use a preset target tracking algorithm to track the motion track of the target element in the target element set; and

The illegal behavior identification unit 73 is configured to use a preset illegal behavior detection algorithm to detect whether there is an illegal behavior of a motor vehicle disobeying pedestrians in the current video frame based on the identified main pedestrian crossing line and the tracked movement trajectory of the target element.

Preferably, the device includes:

The crosswalk recognition unit is used to identify the main crosswalk line if the preset crosswalk line recognition conditions are met, wherein the crosswalk line recognition conditions include that the current video frame is the first frame of the video, or the current video frame corresponds to the preset crosswalk line recognition cycle and/or the frame interval between the current video frame and the previous video frame that is identified but not identified as the crosswalk line is equal to the preset first interval threshold.

Preferably, the pedestrian crossing identification unit further includes:

a crosswalk segmentation unit, used to segment the current video frame by crosswalk line instances using the trained segmentation network to obtain at least one coarsely positioned crosswalk detection frame; and

The fine positioning unit is used to finely locate the main crosswalk line based on all the coarsely positioned crosswalk detection frames.

Preferably, the precise positioning unit further includes:

The fine positioning sub-unit is used to screen the main crosswalk line from all the coarse positioning crosswalk detection frames by using a preset conditional scoring algorithm based on the confidence of each coarsely positioned crosswalk detection frame. The conditional scoring algorithm combines the coarse positioning The position of the crosswalk detection frame in the video frame and/or the ratio of the coarsely located crosswalk detection frame relative to the video frame.

Preferably, the precise positioning subunit also includes:

a conditional total score calculation unit, configured to use a preset conditional scoring algorithm to score each roughly positioned pedestrian crossing detection frame to obtain a conditional total score of each roughly positioned pedestrian crossing detection frame; and

The main crosswalk determination unit is used to obtain the coarsely positioned pedestrian crossing detection frame with the highest total condition score, and use the coarsely positioned pedestrian crossing detection frame with the highest conditional total score as the main pedestrian crossing line detection frame.

Preferably, the main cross lane determination unit further includes:

A width ratio judgment unit, used for judging whether the width ratio of the coarse positioning pedestrian crossing detection frame with the highest total score of the condition and the video frame is less than a preset width ratio threshold;

The first determination subunit is used to expand the coarse-positioned crosswalk detection frame with the highest conditional total score to both sides at equal distances if it is less than the width ratio threshold, so as to make the coarse-positioned crosswalk detection frame with the highest conditional total score and the width of the video frame When the ratio reaches the width ratio threshold, the coarsely positioned crosswalk detection frame with the highest extended conditional total score is used as the main crosswalk line detection frame; and

The second determination subunit is configured to use the roughly positioned pedestrian crossing detection frame with the highest total condition score as the main pedestrian crossing line detection frame if it is not less than the width ratio threshold.

Preferably, the width ratio threshold is 0.85.

Preferably, the target element set includes the category and detection frame of each target element, the target tracking algorithm is a maximum intersection ratio screening method, and the motion track tracking unit also includes:

The first set obtaining unit is configured to, for each target element in the target element set, obtain a first sequence box list set according to all current tracking path sequence box lists, wherein the first sequence box list set includes all the A list of tracking path sequence boxes with the same category of the target element, each tracking path sequence box list contains the category of the tracked target element and the detection boxes of the target element in chronological order, each tracking path sequence box list corresponds to The trajectory of a target element;

an intersection ratio calculation unit, used to calculate the intersection ratio between the detection frame of the target element and the last detection frame of each tracking path sequence frame list in the first sequence frame list set;

a first judging unit, used for judging whether there is an intersection and ratio target frame, and the intersection and ratio target frame is the last detection frame whose intersection ratio is greater than a preset intersection ratio threshold;

The trajectory determination unit is used to determine the motion trajectory tracked to the target element if there is an intersection and ratio target frame, and add the detection frame of the target element to the tracking path sequence frame where the last detection frame with the largest intersection ratio is located List;

The new element discovery unit is used to determine that the target element is a newly appeared element if there is no intersection and ratio target frame, and create a new tracking path sequence frame list according to the detection frame and category of the target element.

Preferably, the cross-union ratio threshold is 0.75.

Preferably, the motion track tracking unit further includes:

The second frame interval obtaining unit is used to obtain the second frame interval of the current video frame and the video frame corresponding to the last detection frame in each current tracking path sequence frame list;

A list discarding unit, configured to discard the list of tracking path sequence boxes whose second frame interval is not less than a preset second interval threshold.

Preferably, the second interval threshold is five.

Preferably, the motion track tracking unit further includes:

The cross-union ratio test unit is used for taking the logarithm test of the largest cross-union ratio and the second largest cross-union ratio if there is the cross-union ratio target frame;

a test result determination unit, configured to determine the motion track tracked to the target element if the logarithm test is passed; and

The tracking subunit is used for tracking the movement track of the target element by adopting the preset cosine distance comparison method if the logarithm test is not passed.

Preferably, the tracking subunit further includes:

The feature map acquisition unit is used to obtain the latest feature map of the target element in the current video frame, and the first detection frame corresponding to the last detection frame and the second last detection frame of each tracking path sequence frame list in the second sequence frame list set respectively. A feature map and a second feature map, wherein the second sequence box list set contains all the tracking path sequence box lists where the intersection and ratio target boxes are located;

a normalization processing unit for normalizing the latest feature map and all the first feature maps and the second feature maps to obtain feature maps of uniform size;

a cosine distance calculation unit, used for calculating the first cosine distance and the second cosine distance of the latest normalized feature map and each first feature map and each second feature map respectively;

a similarity factor calculation unit, which is used to obtain the corresponding first feature similarity factor and second feature similarity factor after taking the logarithm of each first cosine distance and each second cosine distance respectively;

A weighted calculation unit, configured to perform linear weighted calculation on each of the first feature similarity factors and the second feature similarity factors corresponding to each of the first feature similarity factors to obtain a minimum weighted feature similarity factor; and

The trajectory determination subunit is used for taking the motion trajectory corresponding to the tracking path sequence frame list corresponding to the minimum weighted feature similarity factor as the motion trajectory of the target element, and adding the detection frame of the target element to the minimum weighted feature Similar factors corresponding to the tracking path sequence box list.

Preferably, the preset illegal behavior detection algorithm is a violation warning point algorithm, and the illegal behavior identification unit further includes:

The warning point prediction unit is used to predict the violation warning point in the current video frame according to the movement trajectory of each pedestrian on the main crosswalk;

a positional relationship judgment unit for judging whether the violation warning point is within the upper half area of the last detection frame of any identified motor vehicle;

A violation behavior determination unit, configured to determine that a violation behavior is detected in the current video frame if the violation warning point is within the upper half area of any motor vehicle detection frame in the current video frame.

Preferably, the violation identification unit further includes:

a pedestrian position determination unit, used to determine whether each pedestrian in the current video frame is on the main crosswalk;

The pedestrian position determination unit further includes:

Determine whether the center point of the lower edge of the pedestrian detection frame of the pedestrian is within the pedestrian crossing detection frame; and

The pedestrian position determination unit is configured to determine that the pedestrian is on the line of the pedestrian crossing if it is within the detection frame of the pedestrian crossing.

In the embodiment of the present invention, each unit of the device for detecting whether the motor vehicle yields to pedestrians may be implemented by corresponding hardware or software units. Each unit may be an independent software or hardware unit, or may be integrated into a software and hardware unit. Here It is not intended to limit the present invention. For the specific implementation of each unit of the device for detecting whether a motor vehicle yields to a pedestrian, reference may be made to the description of the foregoing method embodiments, which will not be repeated here.

Embodiment 8:

FIG. 8 shows the structure of the electronic device provided by the fifth embodiment of the present invention. For convenience of description, only the part related to the embodiment of the present invention is shown.

The electronic device 8 of the embodiment of the present invention includes a processor 80 , a memory 81 , and a computer program 82 stored in the memory 81 and executable on the processor 80 . When the processor 80 executes the computer program 82 , the steps in the above-mentioned method embodiments are implemented, for example, steps S101 to S103 shown in FIG. 1 . Alternatively, when the processor 80 executes the computer program 82, the functions of the units in the above-mentioned apparatus embodiments, for example, the functions of the units 71 to 73 shown in FIG. 7 are realized.

Embodiment 9:

In an embodiment of the present invention, a computer-readable storage medium is provided, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented, for example, as shown in FIG. 1 . Steps S101 to S103 shown. Alternatively, when the computer program is executed by the processor, the functions of the units in the above-mentioned apparatus embodiments, for example, the functions of the units 71 to 73 shown in FIG. 7 , are implemented.

The computer-readable storage medium of the embodiments of the present invention may include any entity or device capable of carrying computer program codes, recording medium, for example, memory such as ROM/RAM, magnetic disk, optical disk, flash memory, and the like.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. a detection method of whether a motor vehicle yields to pedestrians, it is characterized in that, described method comprises the steps: Use the trained target recognition network to perform target element recognition on the collected video frames to obtain a target element set; Use a preset target tracking algorithm to track the motion trajectory of the target element in the target element set; Based on the identified main crosswalk line and the tracked motion trajectory of the target element, use a preset illegal behavior detection algorithm to detect whether there is a violation of a motor vehicle in the current video frame that is not courteous to a pedestrian; The target element set includes the category and detection frame of each target element, the target tracking algorithm is a maximum intersection ratio screening method, and the preset target tracking algorithm is used to track the movement of the target elements in the target element set. The steps of the trajectory, including: For each target element in the target element set, a first sequence box list set is obtained according to all current tracking path sequence box lists, wherein the first sequence box list set includes all the target elements related to the target element. A list of tracking path sequence boxes of the same category, each of the tracking path sequence box lists contains the category of the tracked target element and the detection boxes of the target element arranged in chronological order, each of the tracking path sequence box lists Corresponding to the motion trajectory of a target element; Calculate the intersection ratio of the detection frame of the target element and the last detection frame of each tracking path sequence frame list in the first sequence frame list set; Judging whether there is an intersection and ratio target frame, the intersection and ratio target frame is the final detection frame whose intersection ratio is greater than a preset intersection ratio threshold; If the intersection and ratio target frame does not exist, then determine that the target element is a newly appeared element, and create a new tracking path sequence frame list according to the detection frame and category of the target element; If there is the cross-union ratio target frame, the largest cross-union ratio and the second largest cross-union ratio are subjected to a logarithmic test; If the logarithm test is passed, then determine the motion track of the target element to be tracked, and add the detection frame of the target element to the list of tracking path sequence frames where the last detection frame with the largest intersection ratio is located; If the logarithm check is not passed, obtain the latest feature map of the target element in the current video frame, and the last detection frame and the last detection frame of each of the tracking path sequence frame lists in the second sequence frame list set The first feature map and the second feature map corresponding to the detection frame respectively, wherein the second sequence frame list set includes all the tracking path sequence frame lists where the intersection and ratio target frame is located; Normalizing the latest feature map and all the first feature maps and the second feature maps to obtain feature maps of uniform size; Calculate the first cosine distance and the second cosine distance of the latest normalized feature map and each of the first feature maps and each of the second feature maps respectively; For each of the first cosine distances and each of the second cosine distances, take the logarithm to obtain the corresponding first feature similarity factor and second feature similarity factor; Perform linear weighting calculation on each of the first feature similarity factors and the second feature similarity factors corresponding to each of the first feature similarity factors to obtain a minimum weighted feature similarity factor; Taking the motion trajectory corresponding to the tracking path sequence box list corresponding to the minimum weighted feature similarity factor as the motion trajectory of the target element, and adding the detection frame of the target element to the minimum weighted feature similarity factor corresponding to the in the trace path sequence box list. 2. method as claimed in claim 1, is characterized in that, before the step of described using trained target recognition network to carry out target element identification to the video frame collected, also comprises: If the preset crosswalk line recognition conditions are met, the main crosswalk line is recognized, wherein the crosswalk line recognition conditions include that the current video frame is the first frame of the video, or the current video frame is a video corresponding to a preset crosswalk line recognition period frame, and/or the frame interval between the current video frame and the previous video frame that has been identified without identifying the crosswalk line is equal to the preset first interval threshold; The step of identifying the main crosswalk line also includes: Use the trained segmentation network to segment the crosswalk line instance on the current video frame to obtain at least one coarsely positioned crosswalk detection frame; Finely locate the main crosswalk line based on all the coarsely positioned crosswalk detection frames; The segmentation network is a Mask R-CNN network, and the steps of training the Mask R-CNN network include: Get the lane line dataset; If the lane markings in the lane marking data set only include the side contours of the lane markings, the lane markings are preprocessed to obtain a preprocessing lane marking data set, wherein the lane markings include crosswalk markings, The preprocessed lane markings contain segmentation masks; Inputting the preprocessed lane line data set into the Mask R-CNN network for training to obtain a trained Mask R-CNN network; The step of preprocessing the lane lines in the lane line dataset further includes: If the marking of the lane line contains only one line, then take the line as the axis and spread the preset pixel width on both sides, wherein the pixel width is 5 pixel width; If the marking of the lane line includes two straight lines, the head ends and the ends of the two straight lines are respectively connected to form a quadrilateral closed area, and the quadrilateral closed area is filled according to the category of the lane line. 3. The method according to claim 2, wherein the step of finely positioning the main crosswalk line based on all of the rough positioning crosswalk detection frames comprises: Based on the confidence of each of the coarsely positioned crosswalk detection frames, a preset conditional scoring algorithm is used to screen the main crosswalk line from all of the coarsely positioned crosswalk detection frames, wherein the conditional scoring algorithm combines the Coarsely locate the position of the crosswalk detection frame in the video frame and/or the ratio of the roughly positioned pedestrian crossing detection frame to the video frame; The step of screening the main crosswalk line from all the roughly positioned crosswalk detection frames using a preset conditional scoring algorithm, including: Use a preset conditional scoring algorithm to score each of the roughly positioned pedestrian crossing detection frames to obtain a conditional total score of each of the roughly positioned pedestrian crossing detection frames; Obtain the coarse-positioned crosswalk detection frame with the highest total condition score, and use the coarse-positioned crosswalk detection frame with the highest conditional total score as the main crosswalk line detection frame; The formula used by the conditional scoring algorithm is as follows:

in,

represents the conditional total score of the i-th coarsely positioned pedestrian crossing detection frame,

respectively represent the distance between the upper left corner of the i-th coarsely positioned crosswalk detection frame relative to the upper left corner of the current video frame in the horizontal direction and the vertical direction,

respectively represent the width and height of the i-th coarsely positioned crosswalk detection frame, height _IMG represents the height of the video frame,

Represents the confidence of the i-th coarsely positioned pedestrian crossing detection frame, I() is the indicator function, (R1 _t/h , R2 _t/h ) represents the first preset interval, (R1 _w/h , R2 _{w/h )} ) represents the second preset interval, s1 represents the first preset fixed value, and s2 represents the second preset fixed value; After the step of obtaining the coarsely positioned pedestrian crossing detection frame with the highest total score of the condition, it also includes: Determine whether the width ratio of the coarse-positioned crosswalk detection frame with the highest conditional total score to the video frame is less than a preset width ratio threshold; If it is less than, expand the coarse-positioned crosswalk detection frame with the highest conditional total score to both sides at equal distances, so that the width ratio of the coarse-positioned crosswalk detection frame with the highest conditional total score to the video frame reaches the width ratio threshold , taking the expanded crosswalk detection frame with the highest conditional total score as the main crosswalk line detection frame; If it is not less than, the roughly positioned pedestrian crossing detection frame with the highest total condition score is used as the main pedestrian crossing line detection frame. 4. The method of claim 3, wherein the target recognition network is an SSD MobileNet-v2 network, the first interval threshold is 10, and the first preset interval is (0.4, 0.6), The first preset fixed value is 0.4, the second preset interval is (0.07, 0.14), the second preset fixed value is 0.14, and the width ratio threshold is 0.85. 5. The method of claim 1, wherein the cross-union ratio threshold is 0.75, Before the step of obtaining the first sequence box list set according to all the current tracking path sequence box lists, the method further includes: Obtain the second frame interval of the current video frame and the video frame corresponding to the last detection frame in each of the current tracking path sequence frame lists; Discard the list of tracking path sequence boxes whose second frame interval is not less than a preset second interval threshold, where the second interval threshold is 5. 6. The method of claim 1, wherein The use formula of the logarithmic test is: (log ₂ IoU _max -log ₂ IoU _{second_max} )＜ε Among them, IoU _max represents the maximum cross-union ratio, IoU _{second_max} represents the second largest cross-union ratio, and ε is a constant, representing the preset logarithmic test threshold. 7 . The method of claim 1 , wherein the preset illegal behavior detection algorithm is a violation warning point algorithm, and the preset illegal behavior detection algorithm is used to detect whether there is a motor vehicle disobeying pedestrians in the current video frame. 8 . The steps for the violation include: Predict the violation warning point in the current video frame according to the motion trajectory of each pedestrian on the main crosswalk; Determine whether the violation warning point is within the upper half area of the last detection frame of any identified motor vehicle; If it is, it is determined that the violation is detected in the current video frame; The calculation method of the violation warning point is:

Wherein, x _wp and y _wp respectively represent the coordinates of the violation warning point,

respectively represent the coordinates of the center point of the mth detection box in the pedestrian tracking path sequence box list, and c represents the number of detection boxes in the pedestrian tracking path sequence box list; Before the step of predicting the violation warning point in the current video frame according to the motion trajectory of each pedestrian on the main crosswalk, the steps include: Determine whether each pedestrian in the current video frame is on the main crosswalk line; The step of judging whether each pedestrian in the current video frame is on the main crosswalk line includes: Determine whether the center point of the lower edge of the pedestrian detection frame of the pedestrian is within the pedestrian crossing detection frame; If it is within the pedestrian crossing detection frame, it is determined that the pedestrian is on the pedestrian crossing line; After the step of using the preset illegal behavior detection algorithm to detect whether there is an illegal behavior of the motor vehicle in disrespect to pedestrians in the current video frame, it also includes: If the violation is detected, use a preset license plate recognition algorithm to identify the license plate of the motor vehicle where the violation occurs; A visualization shows the license plate, the violation, and/or the time when the violation occurred. 8. A detection device for whether a motor vehicle yields to a pedestrian, wherein the device comprises: The target element recognition unit is used to use the trained target recognition network to perform target element recognition on the collected video frames to obtain a target element set; a motion trajectory tracking unit, used for tracking the motion trajectory of the target element in the target element set using a preset target tracking algorithm; and The illegal behavior identification unit is used to detect, based on the identified main crosswalk line and the tracked motion trajectory of the target element, whether there is an illegal behavior of a motor vehicle rudely yielding to pedestrians in the current video frame by using a preset illegal behavior detection algorithm ; The target element set includes the category and detection frame of each target element, the target tracking algorithm is a maximum intersection ratio screening method, and the motion track tracking unit also includes: a first set obtaining unit, configured to obtain, for each target element in the target element set, a first sequence box list set according to all current tracking path sequence box lists, wherein the first sequence box list set contains all the tracking path sequence box lists with the same category as the target element, and each tracking path sequence box list contains the category of the tracked target element and the detection frame of the target element arranged in chronological order, each Each of the tracking path sequence box lists corresponds to a motion trajectory of a target element; an intersection ratio calculation unit, configured to calculate the intersection ratio between the detection frame of the target element and the last detection frame of each tracking path sequence frame list in the first sequence frame list set; a first judging unit for judging whether there is a cross-union ratio target frame, the cross-union ratio target frame is the last detection frame whose cross-union ratio is greater than a preset cross-union ratio threshold; A new element discovery unit, configured to determine that the target element is a newly-appeared element if the intersection and ratio target frame does not exist, and create a new tracking path sequence frame list according to the detection frame and category of the target element; The trajectory determination unit is used to determine the motion trajectory tracked to the target element if there is the intersection and ratio target frame, and add the detection frame of the target element to the location where the final detection frame with the largest intersection ratio is located. Trace path sequence box list; The motion track tracking unit also includes: a cross-union ratio test unit, configured to perform a logarithmic test on the largest cross-union ratio and the next largest cross-and-merge ratio if the cross-union ratio target frame exists; a test result determination unit, configured to determine the motion track tracked to the target element if the logarithm test is passed; A tracking subunit, used for tracking the motion trajectory of the target element by adopting a preset cosine distance comparison method if the logarithm test is not passed; The tracking subunit also includes: The feature map acquisition unit is used to acquire the latest feature map of the target element in the current video frame, and the last detection frame and the second last detection frame of each of the tracking path sequence frame lists in the second sequence frame list set correspond respectively to The first feature map and the second feature map of , wherein, the second sequence box list set contains all the tracking path sequence box lists where the intersection and ratio target boxes are located; a normalization processing unit, configured to perform normalization processing on the latest feature map and all the first feature maps and the second feature maps to obtain feature maps of uniform size; A cosine distance calculation unit, used for calculating the first cosine distance and the second cosine distance of the latest normalized feature map and each of the first feature maps and each of the second feature maps respectively; a similarity factor calculation unit, used to obtain the corresponding first feature similarity factor and second feature similarity factor after taking the logarithm of each of the first cosine distances and each of the second cosine distances respectively; a weighted calculation unit, configured to perform linear weighted calculation on each of the first feature similarity factors and the second feature similarity factors corresponding to each of the first feature similarity factors to obtain a minimum weighted feature similarity factor; and The trajectory determination subunit is used to take the motion trajectory corresponding to the tracking path sequence frame list corresponding to the minimum weighted feature similarity factor as the motion trajectory of the target element, and add the detection frame of the target element to the target element. in the list of the tracking path sequence box corresponding to the minimum weighted feature similarity factor. 9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the computer program as claimed in the claims The steps of any one of 1 to 7 of the method. 10. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented .

Images