CN111523421A - Multi-person behavior detection method and system based on deep learning fusion of various interactive information - Google Patents

Abstract

A behavior detection network is trained by constructing a video library with labels as a sample set, the trained network processes a video to be detected, and detection of object behaviors in an area is realized according to a final output vector. The invention fully considers the complexity of human behaviors, integrates the interactive relation between the human behaviors and other people, objects and long-term memory information while considering the self motion of the human, and effectively improves the precision of video behavior detection.

Description

Multi-person behavior detection method and system based on deep learning fusion of various interactive information

technical field

The invention relates to a technology in the field of artificial intelligence video recognition, in particular to a multi-person behavior detection method and system integrating various interactive information based on deep learning.

Background technique

The goal of computer vision is to use computer programs to handle various visual tasks, often involving multimedia such as images and videos. Convolutional neural network is a deep learning technique widely used in computer vision tasks. It obtains more general deep robust representations by training filter parameters in image convolution operations. These representations are in the form of high-dimensional vectors. Or matrix, which can be used for behavior detection or classification, that is, to detect the positions of people appearing in the video, and to judge their respective behaviors.

The existing behavior detection technology generally detects the human bounding box, extracts the representation of the video through a three-dimensional convolutional neural network, and extracts the regional representation of the person from the video representation according to the human bounding box by linear interpolation. The regional representation of the person makes the final judgment. Its defect is that it only considers the movement changes of a single person inside its bounding box, does not utilize the interaction information between people and other people or objects, and cannot accurately detect more complex interactive behaviors, such as opening the door, watching TV, and interacting with others. Conversation with others, etc.

SUMMARY OF THE INVENTION

Aiming at the above-mentioned shortcomings of the prior art, the present invention proposes a multi-person behavior detection method and system based on deep learning fusion of various interactive information. It improves the accuracy of multi-person behavior detection and has good feasibility and robustness.

The present invention is achieved through the following technical solutions:

The invention relates to a multi-person behavior detection method based on deep learning and fusion of various interactive information. The behavior detection network is trained by constructing a video library with labels as a sample set, and the trained network processes the video to be tested, according to the The final output vector enables detection of object behavior in the region.

The labeled video library is obtained by the following methods: after the videos in the sample set are labeled at equal intervals, the video size is normalized, and each labeled frame is cut into several segments, for example: for each labeled frame. Each marked frame is taken as the middle frame, and the content of 32 frames is taken before and after the frame to obtain a corresponding segment of 64 frames.

The content of the equally spaced annotation includes: the bounding box of each person in the frame and the respective behaviors of each person in a time interval of 1.5 seconds before and after the frame.

The bounding box is obtained by adopting but not limited to a series of relatively mature image object detection algorithms such as Faster-R convolutional neural network and YOLO. Bounding boxes and categories.

The behavior detection network includes: a three-dimensional convolutional neural network for extracting video representations, a representation extraction module with a memory pool, a multi-interaction relationship modeling fusion network, three fully connected layers and a sigmoid regression layer, Among them: the three-dimensional convolutional neural network extracts the video representation according to the input video clip and outputs it to the representation extraction module. The representation extraction module uses RoIAlign to linearly interpolate each bounding box area on the video representation and obtain the area of people and objects after pooling. The simultaneous representation extraction module obtains the memory representation through the memory pool, and the multi-interaction relationship modeling and fusion network models and fuses the regional representation of people and objects and the memory representation to obtain robust behavior representations. The fully connected layer and sigmoid regression layer are used to obtain each The predicted probability of the class.

The three-dimensional convolutional neural network adopts but is not limited to commonly used video representation extraction networks such as I3D network, SlowFast network, and C3D network.

According to the different contents in the bounding box area, the representation extraction module can obtain the regional representation of the person and the regional representation of the object.

The memory pool obtains the memory representation by splicing the region representation of the person in the historical segment of the current segment according to the region representation of the person in the region representation of the person and object in each video segment.

The multi-interaction relationship modeling fusion network includes: two human-human interaction modeling modules for receiving regional representations of people, and two human-human interaction modeling modules for respectively receiving regional representations of people and objects. , two human memory modeling interaction modules for receiving human regional representation and memory representation respectively, wherein: the first human interaction modeling module, the first character interaction modeling module, the first human memory modeling interaction module, The second human-human interaction modeling module, the second character interaction modeling module, and the second human memory modeling interaction module are connected in turn and transmit the sequentially enhanced regional representations of people. Modeling an interaction relationship between interaction and human memory interaction, and fused with the regional representation of the human, and then transferred to the next module. , which is the robust behavioral representation of the final output.

The human-human interaction refers to the interaction between different actors in the same video clip.

The character interaction refers to the interaction between an actor and an object in the same video clip.

The human memory interaction refers to the interaction between the actor in the current segment and the actor in the adjacent segment with a longer history.

其中：Q，K分别为输入的两种表征，W_Q，W_K1，W_K2，W_O是全连接层的权重，d是KW_K1的维度。By modeling is meant:

Among them: Q, K are the two representations of the input, W _Q , W _K1 , W _K2 , W _O are the weights of the fully connected layer, and d is the dimension of KW _K1 .

According to the different input representation K, the module processes different interaction relationships: the value of K includes the regional representation of people, the regional representation of objects, and the memory representation, and the corresponding modeling modules deal with human-human interaction, character interaction and memory interaction in turn. The output fuses the corresponding representations of this type of interaction information; when the six modules are connected in series, the output of the previous modeling module is further input as the next Q, and finally a variety of different interaction relationships are fused.

The three fully connected layers include two hidden layers and one output layer.

The sigmoid regression layer includes a sigmoid function and a cross-entropy loss function, the output vector of the output layer can obtain the prediction probability of each category through the sigmoid layer, and the cross-entropy loss function is used to train the entire network.

The training refers to: taking the samples in the sample set and the corresponding object bounding boxes and the regional representations of people in the adjacent video clips in the memory pool set in the representation extraction module as the input of the behavior detection network, using the cross-entropy loss function. , combined with the back-propagation BP algorithm to adjust the network parameters, and at the same time update the regional representation of the person in the video clip into the memory pool.

The processing of the video to be tested refers to inputting the video to be detected into the object detection algorithm and the trained behavior detection network, and using the sigmoid regression layer to obtain the final prediction probability of each behavior.

technical effect

The present invention solves the technical problem of detecting the behavior of each person appearing in the long video as a whole, that is: for a person appearing in a certain frame in the video, it is necessary to give the bounding box of each person, and each person in the frame Compared with the prior art, the present invention fully considers the complexity of human behavior, considers the human's own movement, and synthesizes its relationship with other people, objects and long-term memory information. The interaction relationship can effectively improve the accuracy of video behavior detection.

Description of drawings

Fig. 1 is the network training flow chart of the present invention;

Fig. 2 is the test video flow chart of the present invention to be tested;

Fig. 3 is the schematic diagram of interactive modeling module in the present invention;

In the figure: N represents the number of people in the video clip, and N' represents the number of interactive objects in the video clip, that is, the number of regional representations of people or the number of regional representations of objects or the number of all people in the memory representation;

4 is a schematic diagram of a multi-interaction relationship modeling fusion network in the present invention;

Each small rectangle in the figure represents an interaction modeling module. The input on the left is Q, and the input on the bottom is K. According to different K, different interactions are modeled.

Detailed ways

This embodiment relates to a multi-person behavior detection system based on deep learning that integrates various interactive information, including: a training sample acquisition module, an object detection module, and a behavior detection network module that integrates multiple interactions, wherein: the samples of the training sample acquisition module And the object detection frame of the object detection module is used as the input of the behavior detection network module. After the behavior detection network is trained, it uses the bounding box area of people and objects to obtain the regional representation of people and objects and the model of memory representation, and further carry out this representation. Multi-classification judgment, the object detection module detects people and objects in the video to be tested, and the behavior detection network module further tests and infers the judgment of each person's behavior in the video according to the detection results of the object detection module.

The behavior detection network includes: a three-dimensional convolutional neural network for extracting video representations, a representation extraction module, a multi-interaction relationship modeling fusion network, three fully connected layers and a sigmoid regression layer, wherein: three-dimensional convolutional The neural network extracts the video representation according to the input video clip and outputs it to the representation extraction module. The representation extraction module uses RoIAlign to linearly interpolate each bounding box area on the video representation and pools to obtain the regional representation of people and objects, and the representation extraction The memory pool in the module obtains the memory representation, and the multi-interaction relationship modeling fusion network performs corresponding modeling and fusion according to the regional representation and memory representation of people and objects to obtain a robust behavior representation. Predict the probability.

As shown in Figure 1, the above behavior detection network is trained through the following steps:

Step 1, initialize the 3D convolutional neural network using weights pretrained on other video behavior classification datasets.

In this embodiment, the SlowFast network is used as the specific structure of the three-dimensional convolutional neural network. The network is firstly pre-trained on the Kinetics behavior classification data set, and the pre-trained weights are used to initialize the structure of the three-dimensional convolutional network. For other parts of the parameters in the behavior detection network, use some different small random numbers for initialization.

In other cases, some behavior classification datasets can be used for pre-training, such as Kinetics, UCF, HMDB, etc.

Step 2: Initialize the memory pool set in the representation extraction module for providing a long-term memory representation of a video segment: at the beginning of training, use an all-zero vector for initialization.

Step 3, data processing and reading:

Step 3.1: Collect long videos of different scenes, and label the long videos at one-second intervals. That is, after labeling a frame, label it again on the next frame that is one second apart from the frame, and so on. The annotated content includes the bounding box of everyone on this frame of image, as well as the type of behavior that each person took 1.5 seconds before and after the frame. In this embodiment, AVA (Atomic Visual Action) data set is used as the data set for verifying the effectiveness of the method of the present invention.

Step 3.2: For each frame with an annotation, run the object detection algorithm on the frame to detect the common object categories that appear in the frame, and the object category should not include people. In this embodiment, the Faster R-convolutional neural network algorithm is used as the object detection module in this embodiment.

Step 3.3: For each frame with an annotation, extract a total of 64 video clips before and after the frame, and normalize it to 256 × 464 (height × width), and the video clip input into the behavior detection network is 64 × A 256x464x3 tensor, where 3 are RGB color channels.

Step 3.4: Randomly shuffle all the video clips in the sample set to increase the randomness during training. The sample set contains multiple long videos, so the video clips used in training may come from different long videos. In each iteration, a video clip is randomly selected from the sample set for training.

Step 3.5: Input the bounding box of the person on the video clip and its corresponding intermediate frame, the behavior category of the person, and the bounding box of the object detected by the detection algorithm into the behavior detection network.

Step 4, training iteration:

Step 4.1: Input the randomly selected video clips from the sample set into the 3D convolutional network, and obtain the representation of the entire video clip as a 16 × 29 × 2304 (height × width × depth) tensor; The bounding box is interpolated on the representation of the video clip to obtain a 7×7×2304 tensor, and further pooled to obtain the region representation of each person or object as a 2304-dimensional vector.

Step 4.2: Update the regional representation of the person in the current segment obtained in step 4.1 into the memory pool, and judge: when there is no representation of the segment in the memory pool, just store it directly, otherwise delete the representation of the segment in the memory pool , updated to the representation extracted in this iteration.

Step 4.3: Read the representations of historical segments from the memory pool to form memory representations: There are regional representations of people in video segments from different long videos in the memory pool, read from the same long video as the current segment, and the time Representation of human regions in 30 video clips within 30 seconds before the current clip; splicing all representations to form the memory representation of this clip, when there are 5 regional representations of people in each video clip, 30×5=150, then memory The dimension of the representation is a representation tensor of 150×2304.

其中：Q，K分别为输入的两种表征，W_Q，W_K1，W_K2，W_O是全连接层的权重，维度均为1024×1024，d是KW_K1的维度，即1024。图4为多交互关系建模融合网络的整体结构示意图，上一个模块的输出作为下一个模块的输入Q，而K采用不同的输入表征：人的区域表征、物体的区域表征和记忆表征，以融合个中不同的交互关系，得到更为鲁棒的行为表征。输入到该结构中的各类表征被全连接层降维到1024维，最终该多交互关系建模融合网络得到多个人的行为表征，即N×1024，N代表该片段中人的数量。Step 4.4: Input the regional representation of the person, the regional representation of the object, and the memory representation into the multi-interaction relationship modeling fusion network. Each module in the multi-interaction relationship modeling fusion network is used to model different interaction relationships, such as As shown in Figure 3, the details are:

Among them: Q and K are the two representations of the input, W _Q , W _K1 , W _K2 , and W _O are the weights of the fully connected layer, and the dimensions are 1024×1024, and d is the dimension of KW _K1 , which is 1024. Figure 4 is a schematic diagram of the overall structure of the multi-interaction relationship modeling fusion network. The output of the previous module is used as the input Q of the next module, and K uses different input representations: the regional representation of people, the regional representation of objects, and the memory representation. Integrate different interactions among them to get a more robust behavioral representation. The various representations input into the structure are reduced to 1024 dimensions by the fully connected layer, and finally the multi-interaction relationship modeling fusion network obtains the behavior representations of multiple people, namely N×1024, where N represents the number of people in the segment.

Step 4.5: Input the behavior representation of multiple people obtained in step 4.4 into three fully connected layers, namely two hidden layers and one output layer, and obtain the value of the loss function through the sigmoid regression layer, where the weights of the two hidden layers are The size is 1024×1024, and the scale of the output layer weights is 1024×C, where C is the total number of action categories involved, which is 80 in the AVA dataset. According to the loss function, the BP algorithm is used to optimize the entire behavior detection network.

The optimized parameter part includes: parameters in the three-dimensional convolutional neural network, parameters in each interaction modeling module in the multi-interaction relationship modeling fusion network, and parameters in the three fully connected layers.

Step 5: When the optimization in step 4.5 reaches the maximum number of times, terminate the training, otherwise return to step 4.1 to continue the training iteration.

As shown in Figure 2, the described test inference includes the following steps:

Step i: Obtain the video to be detected.

Step ii: Segment and normalize the video to be tested: continuously extract video clips containing 64 frames from the video to be tested, each clip is normalized to 256×464 (height×width), and the next clip is the same as the previous one. The segment start time interval is one second. The video clips need to be input into the behavior detection network trained in step 5 in chronological order.

Step iii, perform test inference on the video to be tested, including:

Step a: Read the processed video clip from the data input module, run the object detection algorithm on the intermediate frame of the clip, and detect people and common objects appearing in the frame. In this embodiment, the Faster R-convolutional neural network algorithm is used as the object detection module in this embodiment.

Step b: Input the video clip into a 3D convolutional network to obtain the representation of the video clip, which is a tensor of 16×29×2304 (height×width×depth). Using the representation extraction module, based on the bounding boxes of people and objects, RoIAlign is used to interpolate the representation of the video clip to obtain a tensor of 7×7×2304, and further pool to obtain the regional representation of the person and the object. The regional representation is 2304-dimensional. vector.

Step c: Save the regional representation of the person in the segment into the memory pool.

Step d: Read the regional representations of people in the first 30 tested video clips of the clip from the memory pool, and splicing together the memory representations of the current clip. If the representation of a tested video segment does not exist, or does not belong to the same video to be tested as the current segment, it is supplemented with a zero vector. Assuming that there are regional representations of 5 people in each video segment, 30×5=150, the dimension of the memory representation is a representation tensor of 150×2304.

Step e: Input the regional representation of the person, the regional representation of the object, and the memory representation into the multi-interaction relationship modeling fusion network. The output is the behavioral representation of each person in the segment, with a dimension of N×1024, where N represents the number of people in the segment.

Step f: Behavior representation After three fully connected layers and sigmoid function layers, the probability of each person having different behaviors is obtained, a matrix of dimension N×C, where: N is the number of people in the segment, and C is the behavior category. number, which is 80 in the AVA dataset. Each value in the matrix is a decimal from 0 to 1 and represents the probability that a person will perform a certain action in that segment. When the probability is greater than the threshold, it is determined that the behavior occurs, otherwise the behavior does not occur. There are situations where more than one behavior occurs at the same time.

Step iv: when the processing of the last segment of the video to be tested is completed, end or process other videos to be tested, otherwise return to step 3 to continue processing the next video segment.

In this embodiment, the verification is performed on the verification video of the AVA data set. According to the different selection of the threshold in the test step f, the performance evaluation data of this method is shown in Table 1. The standard for this performance evaluation is that when a detection result is consistent with a label behavior category, and the IoU (Intersection over Union) of the two bounding boxes is greater than 0.5, the detection result is considered correct. The calculation method of the IoU is the ratio of the area of the union area of the areas occupied by the two frames to the area of the intersection area of the areas occupied by the two frames.

Table 1 Detection performance evaluation data

threshold 0.3 0.4 0.5 0.6 recall 42.45% 33.85% 26.76% 20.82% Accuracy 31.61% 42.94% 56.63% 72.84%

精确度

其中TP为真阳性的数量，FN为假阴性的数量，即预测判定为发生某类行为的样本在实际发生这类行为的样本总数中所占比例，FP为假阳性的数量，即预测判定为发生某类行为的样本中实际发生这类行为的样本的所占比例。The threshold mentioned in Table 1 is that when the predicted probability of a certain behavior is greater than this value when detecting the video to be tested, it is determined that the behavior occurs, and the effect of the detection task is judged by the recall rate and the precision rate: recall rate

Accuracy

Among them, TP is the number of true positives, FN is the number of false negatives, that is, the proportion of samples that are predicted to have a certain type of behavior in the total number of samples that actually occur such behaviors, and FP is the number of false positives, that is, the prediction is determined as The proportion of samples that actually engage in a certain type of behavior among the samples that engage in that behavior.

On the basis of jointly considering the recall rate and the precision rate, and comprehensively analyzing the visualization results, the selected threshold value in this embodiment is 0.5. Under this threshold, the detection performance data on all validation videos of 80 categories in the AVA dataset are: recall 26.76% and precision 42.94%. The detection performance evaluation data on the most common 10 categories of verification videos in the AVA data set are: recall rate of 63.64% and accuracy of 76.51%, that is, among the 10,000 people to be tested who have behaviors, 6,364 of them can be correctly detected. Individual bounding boxes and behaviors; 7651 of the 10,000 people detected have correct bounding boxes and behaviors.

The above-mentioned specific implementation can be partially adjusted by those skilled in the art in different ways without departing from the principle and purpose of the present invention. The protection scope of the present invention is subject to the claims and is not limited by the above-mentioned specific implementation. Each implementation within the scope is bound by the present invention.

Claims (11)

1. A multi-user behavior detection method based on deep learning and fusion of various interactive information is characterized in that a behavior detection network is trained by constructing a video library with labels as a sample set, the trained network processes a video to be detected, and the detection of object behaviors in an area is realized according to a final output vector;

the behavior detection network comprises: the video representation extraction system comprises a three-dimensional convolutional neural network for extracting video representations, a representation extraction module with a memory pool, a multi-interaction relationship modeling fusion network, three full-connection layers and a sigmoid regression layer, wherein: the three-dimensional convolutional neural network extracts video representations according to input video clips and outputs the video representations to the representation extraction module, the representation extraction module utilizes RoIAlign to perform linear interpolation on each bounding box area on the video representations and obtains the area representations of people and objects through pooling, meanwhile, the representation extraction module obtains memory representations through a memory pool, the multi-interaction relation modeling fusion network performs modeling fusion on the area representations of people and objects and the memory representations to obtain robust behavior representations, and prediction probabilities of all categories are obtained through a full connection layer and a sigmoid regression layer.

2. The method of claim 1, wherein the tagged video library is obtained by: and after the videos in the sample set are labeled at equal intervals, normalizing the sizes of the videos, and cutting the videos into a plurality of segments according to each labeled frame.

3. The method of claim 1, wherein said equally spaced annotations comprise: the bounding box for each person in the frame and the behavior that each person has individually occurred during the 1.5 second time interval before and after the frame.

4. The method of claim 1, wherein the three-dimensional convolutional neural network is selected from the group consisting of an I3D network, a SlowFast network, and a C3D network.

5. The method of claim 1, wherein the memory pool is based on the person region representation in the region representations of the person and the object in each video segment, and the memory representations are obtained by concatenating the person region representations in the historical segment of the current segment.

6. The method of claim 1, wherein the multiple interaction relationship modeling fusion network comprises: two human-human interaction modeling modules for receiving the region representation of the person, two human interaction modeling modules for respectively receiving the region representation of the person and the region representation of the object, and two human memory modeling interaction modules for respectively receiving the region representation and the memory representation of the person, wherein: the system comprises a first human-human interaction modeling module, a first human memory modeling interaction module, a second human-human interaction modeling module, a second human character interaction modeling module and a second human memory modeling interaction module, wherein the first human-human interaction modeling module, the first human memory modeling interaction module, the second human memory modeling interaction module and the second human memory modeling interaction module are sequentially connected and transmit region representations of human which are sequentially enhanced, each interaction modeling module models one interaction relationship among human interaction, human character interaction and human memory interaction, the interaction relationships are fused with the region representations of the human and then transmitted to the next module, and the finally output region representations of the human comprehensively fuse human interaction, human character interaction and human memory interaction relationships, namely the finally output robust behavior representations;

the human-human interaction is as follows: interaction between different agents in the same video segment;

the character interaction is as follows: interaction between an agent and an object in the same video clip;

the human memory interaction is as follows: interaction between an agent in the current segment and an agent in an adjacent segment of longer history.

7. The method of claim 6, wherein said modeling is by:

wherein: q and K are respectively two representations of input, W_Q，W_K1，W_K2，W_OIs the weight of the fully connected layer, d is KW_K1Dimension (d);

according to the difference of input representation K, the module processes different interaction relations: the value of K comprises a human region representation, an object region representation and a memory representation, and the corresponding modeling module sequentially processes human-human interaction, human interaction and memory interaction correspondingly and outputs corresponding representations fused with the type interaction information; when the six modules are connected in series, the output of the last modeling module is further used as the input of the next Q, and finally, a plurality of different interaction relations are fused.

8. The method of claim 1, wherein the three fully-connected layers include two hidden layers and an output layer.

9. The method as claimed in claim 1, wherein the sigmoid regression layer comprises a sigmoid function and a cross entropy loss function, the output vector of the output layer can obtain the prediction probability of each category through the sigmoid layer, and the cross entropy loss function is used for training the whole network.

10. The method of claim 1, wherein the training is: taking the samples in the sample set, the corresponding object boundary frames and the human region representation of the adjacent video segments in the memory pool arranged in the representation extraction module as the input of the behavior detection network, adjusting network parameters by adopting a cross entropy loss function and combining a back propagation BP algorithm, and updating the human region representation in the video segments into the memory pool.

11. A multi-person behavior detection system according to the method of any of claims 1 to 10, comprising: training sample acquisition module, object detection module, the action detection network module that fuses multiple interaction, wherein: the method comprises the steps that a sample of a training sample acquisition module and an object detection frame of an object detection module are used as input of a behavior detection network module, the behavior detection network obtains models of area characteristics and memory characteristics of people and objects by utilizing boundary frame areas of the people and the objects after training, multi-classification judgment is further carried out on the characteristics, the object detection module detects the people and the objects in a video to be detected, and the behavior detection network module further tests and deduces to obtain judgment on behaviors of each person in the video according to a detection result of the object detection module.

Images