CN112766177A - Behavior identification method based on feature mapping and multi-layer time interaction attention - Google Patents

Abstract

The invention discloses a behavior recognition method based on feature mapping and multi-layer temporal interactive attention, which solves the problem of insufficient modeling of temporal dynamic information in the prior art, ignoring the interdependence between different frames, and thus leading to inaccurate behaviors. The problem of insufficient recognition ability. The implementation steps of the present invention are: (1) generating a training set; (2) acquiring a depth feature map; (3) constructing a feature mapping matrix; (4) generating a temporal interactive attention matrix; (5) generating a temporal interactive attention weighted feature (6) Generate a multi-layer temporal interactive attention weighted feature matrix; (7) Obtain the feature vector of the video; (8) Perform behavior recognition on the video. Since the present invention constructs a feature map matrix and proposes multi-layer temporal interactive attention, the present invention can improve the accuracy of behavior recognition in video.

Description

Action Recognition Method Based on Feature Mapping and Multilayer Temporal Interactive Attention

technical field

The invention belongs to the technical field of video processing, and further relates to a behavior recognition method based on feature mapping and multi-layer temporal interactive attention in the technical field of computer vision. The present invention can be used for human action recognition in video.

Background technique

Video-based human behavior recognition tasks occupy an important position in the field of computer vision and have broad application prospects. The goal of human behavior recognition is to determine the category of human behavior in a video, which is essentially a classification problem. In recent years, with the development of deep learning, behavior recognition methods based on deep learning have been widely studied.

South China University of Technology disclosed a human action recognition method in its patent document "Human Action Recognition Method Based on Time Attention Mechanism and LSTM" (Patent Application No.: CN201910271178.2, Application Publication No. CN110135249A). The main implementation steps of the method are: 1. Obtain the video data of the RGB monocular vision sensor; 2. Extract the 2D skeleton joint point data; 3. Extract the joint structural features of the joint points; 4. Construct the LSTM long short-term memory network; A time attention mechanism is added to the LSTM network; 6. Human action recognition is performed using the softmax classifier. The temporal attention mechanism proposed by this method independently explores the importance of each frame in the video, and assigns a large weight to the features of important frames. However, the method still has the disadvantage that it ignores the difference between different frames in the video. The interdependence between them, thus losing part of the global information, leading to errors in behavior recognition.

Limin Wang et al. disclosed an action recognition method in their paper "Temporal segment networks for actionrecognition in videos" (IEEE transactions on pattern analysis and machineintelligence, 2018, 2740-2755). The main implementation steps of this method are: 1. Divide the video into 7 video segments evenly; 2. Randomly sample a frame of RGB image in each video segment to obtain 7 frames of RGB image; 3. Divide each frame of RGB image obtained Input into the convolutional neural network to get the classification score of each frame of RGB image; 4. Use the segment consensus function and the prediction function to combine the classification scores of the 7 frames of RGB images to obtain the behavior recognition result of the video. The disadvantage of this method is that for a long video, only sampling 7 frames of RGB images will lead to the loss of information in the video, and it is impossible to model more complete temporal dynamic information, resulting in a low accuracy of behavior recognition.

SUMMARY OF THE INVENTION

The purpose of the present invention is to address the above-mentioned shortcomings of the prior art, and propose a behavior recognition method based on feature mapping and multi-layer temporal interactive attention, which is used to solve the problem that the prior art does not adequately model temporal dynamic information, ignores different The interdependence between frames leads to the problem of poor behavior recognition ability.

In order to achieve the above purpose, the idea of the present invention is to construct a feature mapping matrix and embed the temporal and spatial information in the video; obtain temporal interactive attention by exploring the mutual influence between different frames in the video; use multi-layer temporal interactive attention Powerful mining of complex temporal dynamic information in videos.

For achieving the above object, the concrete steps of the realization of the present invention are as follows:

(1) Generate a training set:

(1a) Select the RGB videos containing N behavior categories in the video data set to form a sample set, each category contains at least 100 videos, and each video has a definite behavior category, where N>50;

(1b) Preprocess each video in the sample set to obtain the RGB image corresponding to the video, and compose the RGB image of all the preprocessed videos into a training set;

(2) Generate a deep feature map:

Input each frame of RGB image in each video in the training set into the Inception-v2 network in turn, and sequentially output the depth feature map X _k with a size of 7 × 7 × 1024 for each frame in each video, where k represents the video The serial number of the sampled image in the middle, k=1,2,...,60;

(3) Construct the feature mapping matrix:

(3a) Using a spatial vectorization function, encode each depth feature map as a low-dimensional vector f _k with dimension 1024, k=1,2,...,60;

其中，T表示转置操作；(3b) Arrange the low-dimensional vectors corresponding to the 60-frame sampled images of each video into rows according to the time sequence of the frames to obtain a two-dimensional feature mapping matrix

Among them, T represents the transpose operation;

(4) Generate temporal interactive attention matrix:

(4a) Using the formula B=M ^T M, generate a correlation matrix B of M, where the value of the i-th row and the j-th column in the matrix represents the sum of the two low-dimensional vectors corresponding to the i-th and j-th sampled images in the video the degree of correlation between;

(4b) Normalize the correlation matrix B to obtain a temporal interactive attention matrix A with a size of 60×60;

(5) Generate a temporal interactive attention weighted feature matrix:

生成时间交互注意力加权特征矩阵

其中，γ表示一个初始化为0的用于平衡MA和M两项的比例参数；Use the formula

Generating Temporal Interactive Attention Weighted Feature Matrix

Among them, γ represents a scale parameter initialized to 0 for balancing MA and M;

(6) Generate a multi-layer temporal interactive attention weighted feature matrix:

生成

的相关性矩阵

对

进行归一化处理，得到尺寸为60×60的多层时间交互注意力矩阵

(6a) Using the formula

generate

The correlation matrix of

right

Perform normalization to obtain a multi-layer temporal interactive attention matrix of size 60×60

生成多层时间交互注意力加权特征矩阵

其中，

表示一个初始化为0的用于平衡

和

两项的比例参数；(6b) Using the formula

Generating Multilayer Temporal Interactive Attention Weighted Feature Matrix

in,

Represents a value initialized to 0 for balancing

and

two scale parameters;

(7) Obtain the feature vector of the video:

Input the multi-layer temporal interactive attention weighted feature matrix of each video to the fully connected layer, and output the feature vector of the video;

(8) Behavior recognition on video:

全连接层的参数、softmax分类器的参数，直至交叉熵损失函数收敛为止，得到训练好的各个参数；(8a) Input the feature vector of each video into the softmax classifier, and use the back-propagation gradient descent method to iteratively update the parameters γ and

The parameters of the fully connected layer and the parameters of the softmax classifier are obtained until the cross entropy loss function converges, and the trained parameters are obtained;

(8b) Sample 60 frames of RGB images at equal intervals for each video to be identified, scale the size of each frame to 256×340, and then perform center cropping to obtain 60 frames of RGB images with a size of 224×224. The RGB image is input into the Inception-v2 network, and the depth feature map of the video to be recognized is output;

(8c) adopt the same processing method as step (3) to step (7) to process the depth feature map of each to-be-recognized video, obtain the feature vector of the video, and input each feature vector into the trained softmax classification In the device, the behavior recognition results of each video are output.

Compared with the prior art, the present invention has the following advantages:

First, because the present invention constructs a feature mapping matrix, the feature mapping matrix contains the time information of 60 sampled images in the video and the spatial information of each sampled image, which overcomes the problem that only 7 frames of RGB images are sampled in the prior art. The loss of information makes it impossible to model more complete time dynamic information, so that the present invention can more fully retain time series information and obtain features with more expressive ability.

Second, since the present invention proposes a temporal interactive attention matrix, the temporal interactive attention matrix is obtained by calculating the correlation degree between the low-dimensional features of different sampled images in the feature mapping matrix, which overcomes the problem of ignoring the video content in the prior art method. The interdependence between different frames, thereby losing part of the global information, enables the technology proposed in the present invention to fully explore the global information, thereby improving the accuracy of behavior recognition.

Description of drawings

Figure 1 is a flow chart of the present invention.

Detailed ways

The specific steps of the present invention will be further described below in conjunction with FIG. 1 .

Step 1. Generate a training set.

The RGB videos containing N behavior categories in the video dataset are selected to form a sample set, each category contains at least 100 videos, and each video has a certain behavior category, where N>50. Each video in the sample set is preprocessed to obtain the corresponding RGB image of the video, and the RGB images of all preprocessed videos are formed into a training set. The preprocessing refers to sampling 60 frames of RGB images at equal intervals for each video in the sample set, scaling the size of each frame of RGB image to 256×340 and then cropping to obtain a video with a size of 224×224. 60 frames of RGB images.

Step 2. Obtain the depth feature map.

Input each frame of RGB image in each video in the training set into the Inception-v2 network in turn, and sequentially output the size of each frame in each video as a 7×7×1024 depth feature map X _k , where k represents the The serial number of the sampled image, k=1,2,...,60.

Step 3. Build the feature map matrix.

Due to the high dimensionality of feature maps, it is challenging to jointly analyze the information of densely sampled images in videos, and mapping the feature map to a low-dimensional vector can reduce the computational complexity and facilitate the joint analysis of densely sampled images. Take the kth sampled image in the rth video as an example to illustrate how to encode the depth feature map of the sampled image of the video into a low-dimensional vector with a dimension of 1024:

Among them, f _r,k represents the low-dimensional vector corresponding to the k-th sampled image in the r-th video, V( ) represents the space vectorization function, and X _r,k represents the k-th sampled image in the r-th video. Corresponding to the image Depth feature map, X _r,k,ij represents the value of the i-th row and j-th column of X _r,k , ∑ represents the summation operation, H and W represent the total number of rows and columns of X _r,k , respectively.

其中，f_k表示第k个采样图像的低维向量，k＝1,2,...,60，T表示转置操作。Arrange the low-dimensional vectors corresponding to the 60-frame sampled images of each video into rows according to the time order of the frames to obtain a two-dimensional feature map matrix

Among them, f _k represents the low-dimensional vector of the k-th sampled image, k=1, 2, . . . , 60, and T represents the transpose operation.

The number of columns of the matrix M is equal to the total number of sampled images corresponding to each video, and the number of rows is equal to the dimension of the low-dimensional vector.

The feature map matrix contains the temporal information of the video and the spatial information of each sampled image, which enables the method to jointly analyze the densely sampled images in the video.

Step 4. Generate the temporal interaction attention matrix.

Generate the correlation matrix of M B=M ^T M, the value of the i-th row and the j-th column in B represents the degree of correlation between the two low-dimensional vectors corresponding to the i-th and j-th sampled images in the video. After normalization, a temporal interaction attention matrix A of size 60×60 is obtained.

The following takes the sampled image of the ith frame and the sampled image of the jth frame as an example to illustrate how to calculate the element A _ij of the ith row and the jth column of the temporal interactive attention matrix A through the degree of correlation between the two frame images. The specific calculation formula is as follows :

Among them, A _ij measures the degree of correlation between the sampled image of the ith frame and the sampled image of the jth frame. M _i and M _j represent the column vector composed of elements in the i-th column and the column vector composed of the elements in the j-th column in the feature mapping matrix M respectively, and their physical meanings are the i-th sampled image and the j-th sampled image in the video respectively. Transpose of low-dimensional vectors. If the low-dimensional vectors of the two frames are more similar, the larger A _ij is, indicating that the correlation between the two frames is stronger.

Calculate all elements in the temporal interactive attention matrix A in the same way. The ith row in A represents the correlation degree between the sampled image of the ith frame of the video and all the sampled images in the video. Therefore, the temporal interaction attention matrix models the correlation between video frames, which helps to more fully explore the global information in the video.

Step 5. Generate a temporal interactive attention weighted feature matrix.

生成时间交互注意力加权特征矩阵

其中，γ表示一个初始化为0的用于平衡MA和M两项的比例参数。Use the formula

Generating Temporal Interactive Attention Weighted Feature Matrix

where γ represents a scaling parameter initialized to 0 for balancing MA and M.

Step 6. Generate a multi-layer temporal interactive attention-weighted feature matrix.

生成

的相关性矩阵

对

进行归一化处理，得到尺寸为60×60的多层时间交互注意力矩阵

再利用公式

生成多层时间交互注意力加权特征矩阵

其中，

表示一个初始化为0的用于平衡

和

两项的比例参数。Use the formula

generate

The correlation matrix of

right

Perform normalization to obtain a multi-layer temporal interactive attention matrix of size 60×60

Reuse formula

Generating Multilayer Temporal Interactive Attention Weighted Feature Matrix

in,

Represents a value initialized to 0 for balancing

and

A scale parameter for both terms.

Multi-layer temporal interactive attention applies temporal interactive attention again to the temporal interactive attention weighted feature matrix to explore richer temporal dynamics.

Step 7. Obtain the feature vector of the video.

The multi-layer temporal interactive attention weighted feature matrix of each video is input into a fully connected layer with 1024 output neurons, and the feature vector of the video is obtained.

Step 8. Perform behavior recognition on the video.

全连接层的参数、softmax分类器的参数，直至交叉熵损失函数收敛。Input the feature vector of each video into the softmax classifier, and use the back-propagation gradient descent method to update γ,

The parameters of the fully connected layer, the parameters of the softmax classifier, until the cross-entropy loss function converges.

Sample 60 frames of RGB images at equal intervals for each video to be recognized, scale the size of each frame to 256 × 340, and then perform center cropping to obtain 60 frames of RGB images with a size of 224 × 224, and input each frame of RGB image. To the Inception-v2 network, output the depth feature map of the video to be recognized.

The depth feature map of each to-be-recognized video is processed using the same processing method as step 3 to step 7 to obtain the feature vector of the to-be-recognized video, input each feature vector into the trained softmax classifier, and output each feature vector. The behavior recognition results of each video.

Claims (4)

1. A behavior recognition method based on feature mapping and multi-layer temporal interactive attention, characterized in that, a feature mapping matrix containing temporal information of video and spatial information of each sampled image is constructed; temporal interactive attention is proposed, The temporal interactive attention matrix is obtained by calculating the correlation degree between the low-dimensional vectors of different sampled images in the feature mapping matrix. The specific steps of the method include the following: (1) Generate a training set: (1a) Select the RGB videos containing N behavior categories in the video data set to form a sample set, each category contains at least 100 videos, and each video has a definite behavior category, where N>50; (1b) Preprocess each video in the sample set to obtain the RGB image corresponding to the video, and compose the RGB image of all the preprocessed videos into a training set; (2) Generate a deep feature map: Input each frame of RGB image in each video in the training set into the Inception-v2 network in turn, and sequentially output the depth feature map X _k with a size of 7 × 7 × 1024 for each frame in each video, where k represents the video The serial number of the sampled image in the middle, k=1,2,...,60; (3) Construct the feature mapping matrix: (3a) Using a spatial vectorization function, encode each depth feature map as a low-dimensional vector f _k with dimension 1024, k=1,2,...,60; (3b) Arrange the low-dimensional vectors corresponding to the 60-frame sampled images of each video into rows according to the time sequence of the frames to obtain a two-dimensional feature mapping matrix

Among them, T represents the transpose operation; (4) Generate temporal interactive attention matrix: (4a) Using the formula B=M ^T M, generate a correlation matrix B of M, where the value of the i-th row and the j-th column in the matrix represents the sum of the two low-dimensional vectors corresponding to the i-th and j-th sampled images in the video the degree of correlation between; (4b) Normalize the correlation matrix B to obtain a temporal interactive attention matrix A with a size of 60×60; (5) Generate a temporal interactive attention weighted feature matrix: Use the formula

Generating Temporal Interactive Attention Weighted Feature Matrix

Among them, γ represents a scale parameter initialized to 0 for balancing MA and M; (6) Generate a multi-layer temporal interactive attention weighted feature matrix: (6a) Using the formula

generate

The correlation matrix of

right

Perform normalization to obtain a multi-layer temporal interactive attention matrix of size 60×60

(6b) Using the formula

Generating Multilayer Temporal Interactive Attention Weighted Feature Matrix

in,

Represents a value initialized to 0 for balancing

and

two scale parameters; (7) Obtain the feature vector of the video: Input the multi-layer temporal interactive attention weighted feature matrix of each video to the fully connected layer, and output the feature vector of the video; (8) Behavior recognition on video: (8a) Input the feature vector of each video into the softmax classifier, and use the back-propagation gradient descent method to iteratively update the parameters γ and

The parameters of the fully connected layer and the parameters of the softmax classifier are obtained until the cross entropy loss function converges, and the trained parameters are obtained; (8b) Sample 60 frames of RGB images at equal intervals for each video to be identified, scale the size of each frame to 256×340, and then perform center cropping to obtain 60 frames of RGB images with a size of 224×224. The RGB image is input into the Inception-v2 network, and the depth feature map of the video to be recognized is output; (8c) adopt the same processing method as step (3) to step (7) to process the depth feature map of each to-be-recognized video, obtain the feature vector of the video, and input each feature vector into the trained softmax classification In the device, the behavior recognition results of each video are output. 2. the behavior recognition method based on feature mapping and multi-layer time interactive attention according to claim 1, is characterized in that, described in step (1a) is carried out preprocessing to each video in the sample set refers to, Each video in the sample set is sampled with 60 frames of RGB images at equal intervals, and the size of each frame of RGB image is scaled to 256×340 and then cropped to obtain a 60-frame RGB image with a size of 224×224. 3. the behavior recognition method based on feature map and multi-layer time interactive attention according to claim 1, is characterized in that, the space vectorization function described in step (3a) is as follows:

Among them, f _r,k represents the low-dimensional vector corresponding to the kth sampled frame in the rth video, V( ) represents the space vectorization function, and X _r,k represents the kth sampled frame in the rth video. Corresponding to the frame Depth feature map, X _r,k,ij represents the value of the i-th row and j-th column of X _r,k , ∑ represents the summation operation, H and W represent the total number of rows and columns of X _r,k , respectively. 4 . The behavior recognition method based on feature mapping and multi-layer temporal interactive attention according to claim 1 , wherein the number of output neurons of the fully connected layer in step (7) is set to 1024. 5 .

Images