CN108764148A - Multizone real-time action detection method based on monitor video - Google Patents

Abstract

The multizone real-time action detection method based on monitor video that the invention discloses a kind of having following steps：Model training stage and test phase, wherein model training stage is to obtain training data：The database of the specific action marked；The dense optical flow of the video sequence in training data is calculated, the light stream sequence of the video sequence in training data is obtained, and the light stream image in light stream sequence is labeled；Using in training data video sequence and light stream sequence target detection model yolo v3 are respectively trained, respectively obtain RGB yolo v3 models and light stream yolo v3 models.The present invention can not only realize the space-time position detection to specific action in monitor video, and can realize the real-time processing to monitoring.

Description

Multi-region real-time action detection method based on surveillance video

technical field

The invention belongs to the field of computer vision, and in particular relates to a human motion detection system in a monitoring video scene.

Background technique

As the application of monitoring facilities becomes more and more popular, more and more monitoring-based technologies are applied. Action recognition, as one of the most valuable technologies, is mainly used in the interaction of human-machine equipment in indoor and factory environments, as well as in public environments. The security field is used for the detection and identification of specific dangerous actions.

Most of the action recognition methods based on surveillance video mainly focus on the action recognition and classification tasks of the entire scene. Such videos are generally artificially processed video clips, and the video clips generally only contain one type of action, but this kind of video and Natural video clips are very different, and some scholars put the research task on detecting the beginning and receiving position of the action on the entire time axis, but in real applications, the beginning and end of the action in the video and the action in the space The scope of occurrence is very useful. In addition, although existing motion detection methods have achieved good detection results in existing databases and competitions, these methods generally divide the entire video into many small blocks or Process the entire video, and then output the spatio-temporal position of the action in this video. To achieve real-time action detection, it is necessary to achieve video frame-level processing, so this method cannot be deployed in the monitoring system.

With the popularization of monitoring equipment, the detection of human motion in surveillance video has gradually become a popular research field, Wang L., Qiao Y., Tang X. "Action recognition with trajectory-pooled deep convolutional descriptors." (in 2015IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015).) In the method, the video features are extracted by integrating the deep neural network and the features obtained by using the dense tracking algorithm. To achieve action recognition for the entire video, D.Tran, L.Bourdev, R.Fergus, L.Torresani, and M.Paluri. "Learning spatiotemporal features with 3dconvolutional networks." (at 2015IEEE International Conference on ComputerVision (ICCV) (2015)) proposed to use 3D convolution and 3D pooling to form a C3D framework to extract human action features in videos, Simonyan K, Zisserman A. "Two-Stream Convolutional Networks for Action Recognition in Videos" (in Computational Linguistics, 2014 ) by extracting the optical flow sequence from the RGB image sequence, training it with a convolutional neural network and fusing the features obtained by the two networks to achieve the recognition effect of the action. Although the above models have achieved good results, this method can only realize the recognition of the entire video, and cannot locate the spatio-temporal position of the action.

G.Gkioxari and J.Malik. "Finding action tubes" (in IEEE Int.Conf.onComputer Vision and Pattern Recognition, 2015.) mainly detects the action proposals of each frame and then connects the action proposals of each frame to form Action sequence, J.Lu, r.Xu, and J.J.Corso's "Human action segmentation with hierarchical supervoxel consistency" (in IEEEInt.Conf.on Computer Vision and Pattern Recognition, June 2015) proposed a hierarchical MRF model , to connect low-level video clips with high-level human motion and appearance to achieve segmentation of actions in videos, these methods mainly achieve spatial segmentation of actions in videos, and these algorithms require a large number of frame levels The region proposals require a lot of calculations.

In Yuan J, Ni B, Yang X's "Temporal Action Localization with Pyramid of Score Distribution Features" (in IEEE: Computer Vision and Pattern Recognition.2016), a score distribution pyramid feature (Pyramid of Score Distribution Feature, Pyramid of Score Distribution Feature, PSDF), and then use the LSTM network to process the PSDF feature sequence, and process the prediction of the behavior segment according to the output frame-level behavior category confidence score. Shou Z, WangD, Chang S F. "Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs" (in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)) first used the sliding window method to generate multiple The size of the video segment (segment), and then use a multi-stage network (Segment-CNN) to process, and finally use non-maximization suppression to remove overlapping segments and complete the prediction. Shou Z, Chan J, Zareian A, "CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos" (in 2017IEEEConference on Computer Vision and Pattern Recognition (CVPR) (2017)) based on C3D (3D CNN network ) designed a Convolutional Deconvolutional Network (CDC) that inputs a short video and outputs frame-level action category probabilities. The network is mainly used to fine-tune the action boundary in temporal action detection to make the action boundary more accurate. Although the above framework can achieve real-time effects, the above algorithm is mainly to achieve accurate detection of actions in the time dimension, and cannot Realize the spatio-temporal detection of actions.

J.C. van Gemert, M. Jain, E. Gati, and C.G. Snoek. "APT: Action localization proposals from dense trajectories" (in BMVC, volume 2, page 4, 2015) uses unsupervised clustering to generate a set of bounding boxes Proposals for space-time actions. Since the method is based on dense trajectory features, this method cannot detect actions characterized by small movements. "Learning to track for spatio-temporal action localization" by P. Weinzaepfel, Z. Harchaoui, and C. Schmid. (IEEE Computer Vision and Pattern Recognition, 2015.) performs actions by combining frame-level EdgeBoxes region proposals with a track detection framework space-time detection. However, the detection of the temporal dimension of actions is still achieved by a multi-scale sliding window on each track, making this method inefficient for longer video sequences.

Contents of the invention

Aiming at some problems existing in the existing motion detection, the present invention proposes a monitoring video-based multi-region real-time motion detection method. The technical means adopted in the present invention are as follows:

A multi-regional real-time action detection method based on surveillance video, characterized in that it has the following steps:

Model training phase:

A1. Obtain training data: a database of marked specific actions;

A2. Calculate the dense optical flow of the video sequence in the training data, obtain the optical flow sequence of the video sequence in the training data, and mark the optical flow image in the optical flow sequence;

A3. Use the video sequence and optical flow sequence in the training data to train the target detection model yolo v3 respectively, and obtain the RGB yolo v3 model and the optical flow yolo v3 model respectively;

Testing phase:

B1. Extract the sparse optical flow image sequence of the video through the pyramidal Lucas-Kanande optical flow method, and then send the RGB image sequence and sparse optical flow image sequence of the video into the RGB yolo v3 model and the optical flow yolo v3 model respectively, and the RGByolo v3 model outputs A sequence of detection boxes for Extract the top n detection boxes for all action categories using the non-maximum suppression method i=1...n, each detection box has a label of an action category and a probability score belonging to that action A series of detection frames output by the optical flow yolo v3 model uses the non-maximum value suppression method to extract the first n detection frames of all action categories k = 1...n, each detection box has a label of an action category and a probability score s that belongs to _the action Traverse the detection frame output by the RGB yolo v3 model and the optical flow yolo v3 model respectively, and the detection frame output by each RGB yolo v3 model The detection frame of the same action category output by the optical flow yolo v3 model Do the intersection and comparison, and set the detection frame of the same action category output by the optical flow yolo v3 model corresponding to the largest intersection ratio to If the maximum intersection and union ratio is greater than the threshold K, then the probability scores of the detection frames output by the corresponding two RGB yolo v3 models and the optical flow yolov3 model are fused as As the detection box output by this RGB yolo v3 model confidence level, satisfy the following formula:

in, express and The intersection ratio, for with The same action category with the largest cross-merge ratio probability score;

B2. According to the confidence score of each action category of the detection frame output by each RGB yolo v3 model obtained through fusion, connect the detection frames between the RGB image sequences of the video to form a tube:

Initialize the tube and use the detection frame of the first frame image in the RGB image sequence of the video to initialize the tube. For example, if the first frame image in the RGB image sequence of the video generates n detection frames, then the initial n tubes, video The number of tubes of a certain action category in the first frame image in the RGB image sequence is:

_ncategory (1) = n;

Do the following for all action categories separately:

S1. Match the detection frame generated by each tube and t frame, first traverse the tubes belonging to the same action category, if there are n tubes in the action category, calculate the average of the confidence of each frame of the tube for each tube, as the The value of the tube, and the values of the n tubes of the action category are arranged in descending order to form a list list _category . When determining the action category of each tube, a list I={l _t-k+1 ... l _t } is defined To determine the action category of the tube, the list I={l _t-k+1 ... l _t } is used to store the action category of the last k frames of the tube;

S2, traverse the list list _category and the t frame i=1...n, choose the ones that meet the following conditions Add to tube:

Traverse the tubes in the list _category , and select the same action category as the tube in the t frame to match, if the If the intersection ratio with the detection frame in the last frame image of the tube is greater than the threshold d, then the Add to the queue H_list _category ;

if Then select the one with the highest confidence in the H_list _category Add to the tube, and traverse the t frame again When i=1...n, remove the one with the highest confidence

if then the tube does not add any And remain unchanged, if no new tube is added for continuous k frames then terminate the tube;

If t frame has unmatched recorded as Then traverse all the tubes to find And the intersection and union ratio of the last frame of all tubes, and select the tube with the intersection ratio greater than the threshold k and the largest intersection ratio, denoted as T ^* , and put Added to the tube, T ^* satisfies the following formula:

if but if but T ⁱ is the i-th tube, T ⁱ (t-1) is the t-1th frame of the i-th tube;

If there is still an unmatched detection frame in the tth frame, use the detection frame as the starting point to generate a new tube, and use the detection frame as the first frame image of the tube to initialize the tube;

S3, all tubes are matched Afterwards, update the action category list I={l _t-k+1 ... l _t } of the last k frames of each tube, wherein l _t is the action category of the tth frame of the tube, update the action category L of each tube, Count the action categories I={l _t-k+1 ...l _t } of the last k frames of each tube, and the action category with the largest number is taken as the action category L of the tube, which satisfies the following formula:

If l _i =c, then g(l _i ,c)=1; if l _i ≠c, then g(l _i ,c)=0, c is a certain action category, that is, statistics I={l _{t-k +1} … l _t } in the action category, the action category with the largest number is the action category of the tube.

In the step A1, the marked specific action database is the Action Detection data set of UCF101.

In the step A2, the dense optical flow of the video sequence in the training data is calculated using the calcOpticalFlowFarneback function in the OpenCV library.

Compared with the prior art, the present invention can not only realize the time-space position detection of the specific action in the surveillance video, but also realize the real-time processing of the surveillance.

Based on the above reasons, the present invention can be widely promoted in fields such as computer vision.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1 is a schematic diagram of cross-union ratio calculation in a specific embodiment of the present invention.

FIG. 2 is an overall schematic diagram of a multi-region real-time motion detection method based on surveillance video in a specific embodiment of the present invention.

Fig. 3 is a program flow chart of a multi-region real-time motion detection method based on surveillance video in a specific embodiment of the present invention.

Fig. 4 is a schematic diagram of a processing process of a frame of image in a specific embodiment of the present invention.

Fig. 5 is a schematic diagram of a processing process of a continuous image sequence in a specific embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figures 1-5, a multi-region real-time motion detection method based on surveillance video has the following steps:

Model training phase:

A1. Obtain training data: a database of marked specific actions;

A2. Calculate the dense optical flow of the video sequence in the training data, obtain the optical flow sequence of the video sequence in the training data, and mark the optical flow image in the optical flow sequence;

A3. Use the video sequence and optical flow sequence in the training data to train the target detection model yolo v3 respectively, and obtain the RGB yolo v3 model and the optical flow yolo v3 model respectively;

Testing phase:

B1. Extract the sparse optical flow image sequence of the video through the pyramidal Lucas-Kanande optical flow method, and then send the RGB image sequence and sparse optical flow image sequence of the video into the RGB yolo v3 model and the optical flow yolo v3 model respectively, and the RGByolo v3 model outputs A sequence of detection boxes for Extract the top n detection boxes for all action categories using the non-maximum suppression method i=1...n, each detection box has a label of an action category and a probability score belonging to that action A series of detection frames output by the optical flow yolo v3 model uses the non-maximum value suppression method to extract the first n detection frames of all action categories k = 1...n, each detection box has a label of an action category and a probability score s that belongs to _the action Traverse the detection frame output by the RGB yolo v3 model and the optical flow yolo v3 model respectively, and the detection frame output by each RGB yolo v3 model The detection frame of the same action category output by the optical flow yolo v3 model Do the intersection and comparison, and set the detection frame of the same action category output by the optical flow yolo v3 model corresponding to the largest intersection ratio to If the maximum intersection and union ratio is greater than the threshold K, then the probability scores of the detection frames output by the corresponding two RGB yolo v3 models and the optical flow yolov3 model are fused as As the detection box output by this RGB yolo v3 model confidence level, satisfy the following formula:

in, express and The intersection ratio, for with The same action category with the largest cross-merge ratio probability score, Indicates the probability score, such as, the intersection and union ratio IOU(A,B) of images A and B can be shown in Figure 1,

Where area(A) represents the area of image A, and area(A)∩area(B) is the area where the images intersect.

B2. According to the confidence score of each action category of the detection frame output by each RGB yolo v3 model obtained through fusion, connect the detection frames between the RGB image sequences of the video to form a tube:

Initialize the tube and use the detection frame of the first frame image in the RGB image sequence of the video to initialize the tube. For example, if the first frame image in the RGB image sequence of the video generates n detection frames, then the initial n tubes, video The number of tubes of a certain action category in the first frame image in the RGB image sequence is:

_ncategory (1) = n;

Do the following for all action categories separately:

S1. Match the detection frame generated by each tube and t frame, first traverse the tubes belonging to the same action category, if there are n tubes in the action category, calculate the average of the confidence of each frame of the tube for each tube, as the The value of the tube, and the values of the n tubes of the action category are sorted in descending order to form a list _category , When determining the action category of each tube, a list I={l _t-k+1 ...l _t } is defined to determine the action category of the tube, and the list I={l _t-k+1 ...l _t } is used to Store the action category of the last k frames of the tube;

S2, traverse the list list _category and the t frame i=1...n, choose the ones that meet the following conditions Add to tube:

Traverse the tubes in the list _category , and select the same action category as the tube in the t frame to match, if the If the intersection ratio with the detection frame in the last frame image of the tube is greater than the threshold d, then the Add to the queue H_list _category ;

if Then select the one with the highest confidence in the H_list _category Add to the tube, and traverse the t frame again When i=1...n, remove the one with the highest confidence

if then the tube does not add any And remain unchanged, if no new tube is added for continuous k frames then terminate the tube;

If t frame has unmatched recorded as Then traverse all the tubes to find And the intersection and union ratio of the last frame of all tubes, and select the tube with the intersection ratio greater than the threshold k and the largest intersection ratio, denoted as T ^* , and put Added to the tube, T ^* satisfies the following formula:

if but if but T ⁱ is the i-th tube, T ⁱ (t-1) is the t-1th frame of the i-th tube;

If there is still an unmatched detection frame in the tth frame, use the detection frame as the starting point to generate a new tube, and use the detection frame as the first frame image of the tube to initialize the tube;

S3, all tubes are matched Afterwards, update the action category list I={l _t-k+1 ... l _t } of the last k frames of each tube, wherein l _t is the action category of the tth frame of the tube, update the action category L of each tube, Count the action categories I={l _t-k+1 ...l _t } of the last k frames of each tube, and the action category with the largest number is taken as the action category L of the tube, which satisfies the following formula:

If l _i =c, then g(l _i ,c)=1; if l _i ≠c, then g(l _i ,c)=0, c is a certain action category, that is, statistics I={l _{t-k +1} … l _t } in the action category, the action category with the largest number is the action category of the tube.

In Figure 2, (a) shows the RGB image sequence of the video; (b) shows that the optical flow algorithm uses the pyramidal Lucas-Kanande optical flow method in OpenCV to extract sparse optical flow images in the testing stage, and extracts dense optical flow images in the training stage; (c ) is the obtained sparse optical flow image; (d) is the action detection model, one is the RGB yolov3 model trained using the RGB image sequence of the video, and the other is the optical flow yolo v3 model trained with the optical flow sequence; (e) represents The detection result output by the RGB yolo v3 model; (f) represents the detection result of the optical flow yolo v3 model; (g) represents the result of fusing the output of the two models to obtain features with better robustness; (h) represents the use of fusion The resulting features connect the detection boxes between the RGB image sequences of the video as tubes.

Figure 4(a) is the image in the RGB image sequence of the video; (b) represents the optical flow image corresponding to the image in the RGB image sequence of the video; (c) represents the image in the RGB image sequence of the video through the RGB yolo v3 model The detection result output after processing; (d) represents the detection result output after the optical flow image is processed by the optical flow yolo v3 model;

The continuous image sequence in the video in Figure 5; (a) represents the image in the RGB image sequence of the video at equal intervals; (b) represents the optical flow sequence corresponding to the image in the RGB image sequence of the video; (c) represents the RGB image sequence of the video The output detection results of the images in the image sequence processed by the RGB yolo v3 model; (d) indicates the output detection results of the optical flow sequence processed by the optical flow yolo v3 model; (e) indicates the fusion of (c) and (d) The tube obtained from the test results;

In the step A1, the marked specific action database is the Action Detection data set of UCF101.

In the step A2, the dense optical flow of the video sequence in the training data is calculated using the calcOpticalFlowFarneback function in the OpenCV library.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (3)

1. A multi-region real-time action detection method based on a surveillance video is characterized by comprising the following steps:

a model training stage:

a1, acquiring training data: a database of labeled specific actions;

a2, calculating dense optical flows of video sequences in training data, acquiring optical flow sequences of the video sequences in the training data, and labeling optical flow images in the optical flow sequences;

a3, respectively training a target detection model yolo v3 by utilizing a video sequence and an optical flow sequence in training data to respectively obtain an RGB yolo v3 model and an optical flow yolo v3 model;

and (3) a testing stage:

b1, extracting a sparse optical flow image sequence of the video by a pyramid Lucas-Kanande optical flow method, then respectively sending the RGB image sequence and the sparse optical flow image sequence of the video into an RGB yolo v3 model and an optical flow yolo v3 model, and extracting the first n detection frames of all action categories by using a non-maximum suppression method through a series of detection frames output by the RGByolo v3 modelEach detection box has a label of an action category and a probability score belonging to the actionA series of detection frames output by the optical flow yolo v3 model uses a non-maximum suppression method to extract the first n detection frames of all action classes Each detection box has a label of an action category and a probability score belonging to the action Respectively traversing the detection frames output by the RGB yolo v3 model and the optical flow yolo v3 model, and the detection frame output by each RGB yolo v3 modelDetection frame of the same action category as that output by optical flow yolo v3 modelMaking a cross-over ratio, and setting a detection frame of the same action type output by the optical flow yolo v3 model corresponding to the maximum cross-over ratio as a detection frame of the same action typeIf the maximum intersection ratio is larger than the threshold value K, fusing the probability scores of the detection frames output by the corresponding two RGB yolo v3 models and the optical flow yolo v3 models intoDetection frame output as the RGB yolo v3 modelThe degree of confidence of (a) is,the following formula is satisfied:

wherein,to representAndthe cross-over-cross-over ratio of (c),is prepared by reacting withOf the same action class with the largest cross-over ratioA probability score;

b2, connecting the detection frames between the RGB image sequences of the video to form a tube according to the confidence score of each action type of the detection frame output by each RGB yolo v3 model obtained by fusion:

initializing a tube, and initializing the tube by using a detection frame of a first frame image in an RGB image sequence of a video;

the following operations are performed for all action categories, respectively:

s1, matching each tube and the detection frame generated by t frame, firstly traversing tube belonging to the same action type, if there are n tubes in the action type, calculating the average value of confidence of each frame of the tube for each tube as the value of the tube, and arranging the values of the n tubes in descending order to form list_CategoriesWhen determining the action category of each tube, a list I ═ l is defined_t-k+1…l_tUsed to determine the action category of tube, list I ═ l_t-k+1…l_tThe action category of the k frame after tube is stored;

s2, traversing list_CategoriesAnd in t framesFrom which one satisfying the following conditions is selectedAddition to tube:

traverse list_CategoriesAnd selects t frames and tube of the same action categoryPerforming a match ifIf the intersection ratio with the detection frame in the last frame image of tube is larger than the threshold value d, the detection frame is processedAdd to queue H _ list_CategoriesPerforming the following steps;

if it is notPick H _ list_CategoriesWith highest confidence level in the middleAdded to the tube and traversed the t frame againThen, the one with the highest confidence coefficient is eliminated

If it is notThen the tube does not add anythingAnd remains unchanged if no new frame tube is added for consecutive k framesTerminating the tube;

if t frames have not been matchedIs marked asThen go through all tube to find outAnd the cross-over ratio of the last frame of all tube is selectedTube greater than threshold k and having the largest cross-over ratio is denoted as T^*Handle barAdded to the tube, T^*The following formula is satisfied:

if it is notThen If it is notThenTⁱIs the ith tube, Tⁱ(t-1) the t-1 th frame of the ith tube;

if the t-th frame still has the detection frame which is not matched, generating a new tube by taking the detection frame as a starting point, and initializing the tube by taking the detection frame as a first frame image of the tube;

s3, matching all tubeThen, the action category list I of the k-frame after each tube is updated to { l ═ l_t-k+1…l_tIn which l_tFor the action type of t-th frame of tube, update action type L of each tube, and count action type I of k-th frame of each tube as { L }_t-k+1…l_tAmong them, the most motion class is the motion class L of the tube, and the following formula is satisfiedFormula (II):

if l is_iC, then g (l)_iC) 1; if l is_iNot equal to c, then g (l)_iC) is 0, c is a certain action category, i.e. the statistic I is { l ═_t-k+1…l_tThe action type with the largest number is the action type of the tube.

2. The multi-region real-time action detection method based on surveillance video according to claim 1, characterized in that: in step a1, the database of the labeled specific Action is the Action Detection data set of UCF 101.

3. The multi-region real-time action detection method based on surveillance video according to claim 1, characterized in that: in the step a2, a dense optical flow of the video sequence in the training data is calculated by using a calcoptical flow farneback function in the OpenCV library.