CN112381072B - A human abnormal behavior detection method based on spatiotemporal information and human-object interaction - Google Patents

Abstract

The invention discloses a human abnormal behavior detection method based on space-time information and human-object interaction. The steps are as follows: S1, data collection and labeling; S2, extracting position information of people and objects; S3, extracting motion information of people and objects ; S4, interactive relationship modeling of human and object features; S5, behavior classification and fusion; S6, optimization of detection results. For the detection of abnormal movements of falling to the ground, climbing and physical conflicts, and persistent abnormal states, the method of human interaction is used to assist in judging abnormal behaviors, and the persistent state of abnormal behaviors is detected by combining the changes of the center of gravity. At the same time, in addition to detecting abnormal behaviors In addition, the present invention can also detect normal actions such as walking, standing and sitting.

Description

A human abnormal behavior detection method based on spatiotemporal information and human-object interaction

technical field

The invention relates to the technical fields of computer vision and deep learning, in particular to a method for detecting abnormal human behavior based on space-time information and human-object interaction.

Background technique

Human abnormal behavior detection has important applications in the field of security and intelligent monitoring, which greatly relieves the pressure of manual monitoring and improves the detection efficiency. Some of the existing solutions use manual feature extraction motion features for discrimination, which have low accuracy in actual real-world applications; while some current deep learning-based methods can only detect a certain abnormal behavior, and cannot adapt to many real conditions. Automatic determination of abnormal behavior. However, abnormal actions such as climbing and falling to the ground have certain particularities. It is not only necessary to detect the abnormal actions being performed by the actor in real time, but also to be able to continuously discriminate the state of these abnormal actions. For example, after falling to the ground, you may continue to lie there still, and after climbing, you may continue to walk on a table or other auxiliary objects, all of which bring challenges to the existing detection technology. The current method cannot detect the persistence of abnormal movements. state, so new technical methods are needed to solve it.

SUMMARY OF THE INVENTION

In order to solve the problems existing in the prior art, the present invention provides a method for detecting abnormal human behavior based on space-time information and human-object interaction, which solves the problems mentioned in the above background art.

In order to achieve the above purpose, the present invention provides the following technical solutions: a method for detecting abnormal human behavior based on space-time information and human-object interaction, the steps are as follows: S1, data collection and labeling; S2, extracting the position information of people and objects; S3 , extracting the motion information of people and objects; S4, modeling the interaction relationship between people and objects; S5, behavior classification and fusion; S6, optimization of detection results; the abnormal behavior refers to behavior beyond the normal range, with a certain scene Relevance, indicating unacceptable behavior in the context.

Preferably, the data collection and labeling in step S1 includes: collecting normal actions and abnormal actions in video surveillance, cropping video data, generating initial spatial positions of people and objects through the SSD target detection network, and finally using simple labeling The tool manually corrects the generated position information, corrects inaccurate object positions, and obtains accurate position information; the normal actions refer to actions that are acceptable in the monitoring scene, and normal actions include walking, sitting or standing. ; and abnormal actions represent unacceptable actions in the scene, including falling to the ground, climbing, or physical conflict.

Preferably, the simple labeling tool is used to correct the position information of the frame, read and display the picture and its corresponding person and object frame, can judge whether the position of the display frame is accurate, and redraw a new frame through the mouse, New data overwrites old data.

Preferably, the extraction of the position information of people and objects in the step S2 includes: by fine-tuning the SSD target detection network pre-trained on the MS COCO data set on the collected data set to accurately detect the positions of people and objects.

Preferably, the fine-tuning refers to retraining only the last two layers of the network for the training data on the basis of the model pre-trained in the MS COCO dataset, and the parameters of the remaining layers remain unchanged.

Preferably, the motion information of extracting people and objects in the step S3 includes: adopting the 3D-ShuffleNet network as the backbone network of the spatiotemporal motion information, taking the current frame and the previous 15 frames of data to form an input segment together as input data, to The input 16 frames of data are subjected to feature extraction, and finally the spatiotemporal information feature map of a single frame is obtained.

，其中,

表示 第i个人的时空特征与各个物体特征总体的相关性,

表示第i个人的时空运动特征；

表 示第j个物体的特征；

表示当前帧物体特征集合。

表示人和物体关系模型，

表示整合多 个人物关系模型的结果。 Preferably, the modeling of the interaction relationship between people and objects features in the step S4 includes: applying the position information of the people and objects obtained in step S2 to the feature map extracted in step S3 to obtain spatiotemporal feature information; The features of the object are individually cut out and interactively modeled. The formula is as follows:

,in,

Represents the correlation between the spatiotemporal characteristics of the ith person and the overall characteristics of each object,

represents the spatiotemporal motion characteristics of the i-th person;

Represents the feature of the jth object;

Represents the feature set of objects in the current frame.

Represents a model of the relationship between people and objects,

Represents the result of integrating multiple person relationship models.

，其中，C表示将

和

的分类得分融合得到的动作分类结 果，

表示人体运动信息得到的分类结果得分，

表示人和物体交互关系建模得到的分类 结果得分，

为可学习的超参数，表示结果的重要程度，如果

＜0.5，则说明行为与物体的 相关性小，模型更关注人体运动信息的分类结果，反之，模型则更关注人和物体交互关系建 模的分类结果。 Preferably, the behavior classification and fusion in the step S5 includes: classifying the behavior of the human body motion information and the human-object interaction model, respectively, and merging the two classification results to obtain a preliminary detection result. The fusion formula is as follows:

, where C means that the

and

The action classification results obtained by the fusion of the classification scores,

represents the classification result score obtained from human motion information,

Represents the classification result score obtained by modeling the interaction between people and objects,

is a learnable hyperparameter, indicating the importance of the result, if

<0.5, it means that the correlation between behavior and objects is small, and the model pays more attention to the classification results of human motion information. On the contrary, the model pays more attention to the classification results of modeling the interaction between people and objects.

，

表示相邻帧的速度变化信息，将

与阈值

作比较，如果小于阈 值

，则表明仍然处于倒地状态，并将该结果覆盖检测出的结果；如果大于等于阈值

，则表 明已经不再处于倒地的状态，采用模型检测到的结果作为最终的结果并输出行为类别。 Preferably, the optimization of the detection results in the step S6 includes: judging whether the action of falling to the ground is detected by the preliminary detection result of the previous frame, and if the action of falling to the ground is not detected, then the preliminary detection result of the previous frame is used. As the final result, the behavior category is output; if the falling motion is detected, the center of gravity of the human body is calculated through the position box, and the speed change information of adjacent frames is calculated to obtain

,

Indicates the speed change information of adjacent frames, the

with threshold

for comparison, if less than the threshold

, it indicates that it is still in the downed state, and the result is covered by the detected result; if it is greater than or equal to the threshold

, it indicates that it is no longer in the state of falling to the ground, and the result detected by the model is used as the final result and the behavior category is output.

The beneficial effects of the present invention are: through the method of the present invention, the target detection module can accurately locate the specific spatial position of the actor and the object, and the model will finally give the behavior category of the actor. Finally, the human frame and behavior category are drawn onto the original image (without the object frame), and the abnormal behavior category is recorded. The present invention mainly uses human-object interaction modeling analysis, behavior classification and fusion, and optimization of results based on the center of gravity velocity model, adopts the method of human interaction to assist in judging abnormal behavior, and combines the change of center of gravity to detect the persistent state of abnormal behavior . Meanwhile, in addition to detecting abnormal behaviors, the present invention can also detect normal actions such as walking, standing and sitting.

Description of drawings

Fig. 1 is the network model diagram of the present invention;

Fig. 2 is the flow chart of data collection and labeling of the present invention;

Fig. 3 is the flow chart of detection result optimization of the present invention;

Fig. 4 is a flow chart of model training and operation of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to Figures 1-4. The model training and running process is shown in Figure 3. The present invention provides a technical solution: a method for detecting abnormal human behavior based on space-time information and human-object interaction. The steps are as follows: (1) Data Collection and labeling; (2) Extracting the position information of people and objects; (3) Extracting motion information of people and objects; (4) Modeling the interaction relationship between people and objects; (5) Behavior classification and fusion; (6) Detection Results are optimized.

(1) Data collection and labeling

The present invention collects normal actions and abnormal actions in the real video surveillance scene. In order to facilitate the data annotation, we cut the video data of the real scene, and then use the SSD target detection network to generate the initial spatial positions of people and objects. The network model diagram is shown in Figure 1. Finally, the generated position information is manually corrected using the invented simple annotation tool, and the inaccurate object position is corrected. The specific process is shown in Figure 2.

Simple annotation tool description: This tool is mainly used to correct the position information of the frame. It can read and display the picture and its corresponding person and object frame. The user can judge whether the position of the display frame is accurate, and redraw the new frame through the mouse. box, the new data will overwrite the old data.

(2) Extract the location information of people and objects

The invention fine-tunes the pre-trained SSD (Single Shot MultiBox Detector) target detection network on the MS COCO data set on the collected data set, so as to adapt to the target features in the monitoring scene and accurately detect the positions of people and objects.

Fine-tuning method: Based on the pre-trained model of the MS COCO dataset, only the last two layers of the network are retrained for the training data, and the parameters of the remaining layers remain unchanged.

(3) Extract the motion information of people and objects

In order to take into account the running speed and detection accuracy, the present invention proposes to use 3D-ShuffleNet as the backbone network of spatiotemporal motion information. The specific process is as follows:

1) Data sampling, the present invention uses 16 frames of data as input, and the specific sampling process is: take the current frame and the previous 15 frames of data to form an input segment as input data;

2) Use spatiotemporal downsampling to extract features from the input 16 frames of data, and finally obtain the spatiotemporal information feature map of a single frame by downsampling the features, which can be easily combined with the target detection module.

(4) Modeling the interaction relationship between human and object features

The main process of this module includes the following steps:

1) Apply the position information obtained in step (2) to the feature map extracted in step (3) to obtain spatiotemporal feature information;

2) Cut out the features of people and objects separately, and conduct interactive modeling analysis. The formula is as follows:

表示第i个人的时空运动特征；

表示第j个物体的特征；

表示当前帧物体 特征集合。

表示人和物体关系模型，

表示整合多个人物关系模型的结果，这两者都是通 过卷积神经网络实现。 in

represents the spatiotemporal motion characteristics of the i-th person;

Represents the feature of the jth object;

Represents the feature set of objects in the current frame.

Represents a model of the relationship between people and objects,

Represents the result of integrating multiple person relationship models, both of which are implemented via convolutional neural networks.

(5) Behavior classification and fusion

This module mainly consists of three steps:

1) Behavior classification of the human motion information obtained in (3);

2) Classify the behavior of the relational model established in (4);

3) For the fusion of the two classification results, the formula used is as follows:

表示1）得到的分类结果得分，

表示2）中关系建模得到的分类结果得分，

为可学习的超参数，表示结果的重要程度，如果行为与物体的关系较小，则

较小，模型更 关注1）中的分类结果，否则，2）中的分类结果更重要。 in,

Represents 1) the obtained classification result score,

Represents the classification result score obtained by relational modeling in 2),

is a learnable hyperparameter, indicating the importance of the result, if the relationship between the behavior and the object is small, then

Smaller, the model pays more attention to the classification result in 1), otherwise, the classification result in 2) is more important.

(4) Optimization of test results

This step is mainly used to optimize the detection results of abnormal behavior of falling to the ground. After the movement of the human body, the change of the center of gravity of the human body is calculated to assist in judging whether it is still in the state of falling to the ground. The optimization process is shown in Figure 4.

The optimization process of the current detection results is as follows:

1) According to the preliminary detection result of the previous frame, determine whether the falling motion is detected. If the falling motion is not detected, the preliminary detection result of the previous frame is used as the final result and the behavior category is output; if the falling motion is detected The action of the ground is the second step;

；3）将

与阈值

作比较，如果小于阈值

，则表明仍然处于倒地状态，并将该结果覆盖检测出的结 果；如果大于等于阈值

则表明已经不再处于倒地的状态（比如从倒地站立起来），此时采 用模型检测到的结果作为最终的行为类别。 2) Calculate the center of gravity of the human body through the position box, and calculate the speed change information of adjacent frames to get

; 3) will

with threshold

for comparison, if less than the threshold

, it indicates that it is still in the downed state, and the result is covered by the detected result; if it is greater than or equal to the threshold

It indicates that it is no longer in the state of falling to the ground (such as standing up from the ground), and at this time, the result detected by the model is used as the final behavior category.

Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the foregoing embodiments, or to perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (6)

1. A method for detecting abnormal human behavior based on space-time information and interaction between people and objects, characterized in that the steps are as follows: S1, data collection and labeling; S2, extract the position information of people and objects; S3, extract the position information of people and objects Motion information; S4, interactive relationship modeling of human and object features; S5, behavior classification and fusion; S6, optimization of detection results; The data collection and labeling in the step S1 includes: collecting normal actions and abnormal actions in video surveillance, cropping the video data, generating the initial spatial positions of people and objects through the SSD target detection network, and finally using a simple labeling tool to generate data. The position information is manually corrected to correct the inaccurate object position and obtain accurate position information; The modeling of the interaction relationship between people and objects features in the step S4 includes: applying the position information of the people and objects obtained in step S2 to the feature map extracted in step S3 to obtain spatiotemporal feature information; Cut them out individually and perform interactive modeling. The formula is as follows: R(P _i )=F _α {G _β (P _i ,O _j ),O _j ∈O}, where R(P _i ) represents the space-time of the ith person The correlation between the feature and the overall feature of each object, Pi represents the spatiotemporal motion feature of the ith person; _Oj represents the feature of the _jth object; O represents the feature set of objects in the current frame; G _β represents the relationship model between people and objects, F _α Represents the result of integrating multiple character relationship models; The optimization of the detection results in the step S6 includes: through the preliminary detection results of the previous frame, judging whether the action of falling to the ground is detected, if the action of falling to the ground is not detected, then the preliminary detection result of the previous frame is used as the final result. And output the behavior category; if the action of falling to the ground is detected, the center of gravity of the human body is calculated through the position box, and the speed change information of adjacent frames is calculated to obtain V _i , and V _i is compared with the threshold μ, if it is smaller than the threshold μ, It indicates that it is still in the downed state, and the result is covered by the detected result; if it is greater than or equal to the threshold μ, it indicates that it is no longer in the downed state, and the result detected by the model is used as the final result and the behavior category is output. 2. the human body abnormal behavior detection method based on time-space information and human-object interaction according to claim 1, is characterized in that: described simple and convenient labeling tool is used to correct the position information of the frame, read and display the picture and it. The corresponding person and object frame can judge whether the position of the display frame is accurate, and redraw a new frame through the mouse, and the new data will overwrite the old data. 3. the human body abnormal behavior detection method based on time-space information and human, object interaction according to claim 1, is characterized in that: the position information of extracting people and objects in the described step S2 comprises: by the MS COCO data set on the The pre-trained SSD object detection network is fine-tuned on the collected dataset to accurately detect the location of people and objects. 4. the human abnormal behavior detection method based on space-time information and human-object interaction according to claim 3, it is characterized in that: described fine-tuning refers to on the basis of the model pre-trained in MS COCO data set, only for training data. The last two layers of the network are retrained, and the parameters of the remaining layers remain unchanged. 5. the human body abnormal behavior detection method based on space-time information and human, thing interaction according to claim 1, is characterized in that: the motion information of extracting people and objects in described step S3 comprises: adopt 3D-ShuffleNet network as space-time The backbone network of motion information takes the current frame and the previous 15 frames of data to form an input segment as input data, and performs feature extraction on the input 16 frames of data, and finally obtains the spatiotemporal information feature map of a single frame. 6. The method for detecting abnormal human behavior based on space-time information and human-object interaction according to claim 1, wherein the behavior classification and fusion in the step S5 include: human motion information and human-object interaction respectively The relational model performs behavior classification, and fuses the two classification results to obtain preliminary detection results. The fusion formula is as follows: C=(1-θ)*S ₁ +θ*S ₂ , where C represents the combination of S ₁ and S ₂ The action classification result obtained by the fusion of classification scores, S ₁ represents the classification result score obtained from human motion information, S ₂ represents the classification result score obtained by modeling the interaction between people and objects, θ is a learnable hyperparameter, indicating the importance of the result. , if θ < 0.5, it means that the correlation between behavior and objects is small, and the model pays more attention to the classification results of human motion information. On the contrary, the model pays more attention to the classification results of human-object interaction modeling.

Images