CN111325155A - Video action recognition method based on residual 3D CNN and multimodal feature fusion strategy - Google Patents

Abstract

The invention relates to a video action classification method based on residual 3D CNN and multimodal feature fusion, and belongs to the field of computer vision and deep learning. First, the traditional C3D network connection method is changed to residual connection; the kernel decomposition technology is used to disassemble the 3D convolution kernel to obtain a spatial convolution kernel, and multiple time kernels of different time scales in parallel, and then in the spatial convolution kernel After inserting the attention model, the A3D residual module is obtained and stacked into a residual network. Build a dual-stream action recognition model, input RGB image features and optical flow features into the spatial flow network and temporal flow network, and extract the multi-level convolution feature layer features, and then use the multi-level feature fusion strategy to fuse the two networks to achieve The spatial and temporal features are complementary; finally, the global video action descriptor after fractional fusion is reduced by PCA, and then the SVM classifier is used to complete the action classification.

Description

Video Action Recognition Based on Residual 3D CNN and Multimodal Feature Fusion Strategy method

technical field

The invention belongs to the fields of computer vision and deep learning, and relates to a video action recognition method based on a residual 3D CNN and a multimodal feature fusion strategy.

Background technique

Today's digital content is essentially multimedia information that includes text, audio, images, video, and more. Especially images and videos, with the prevalence of sensors and the proliferation of mobile devices, conveying information through video actions as a way of communication has also gradually become popular, starting to become a new way of communication between Internet users. In order to explore and understand multimedia information more deeply and intelligently, the scientific research field is increasingly encouraging the development of advanced video understanding technologies. Representation learning is fundamental to the success of these technological advances. In recent years, with the rise of Convolutional Neural Network (CNN), especially in the image field, the Deep Convolutional Neural Network uses multiple different convolution kernels combined with the information capture mechanism of the local receptive field to traverse the previous layer. The feature planes capture local features of different granularities. As the number of layers deepens, these extracted salient features are combined and compressed. Different feature layers cover different levels of visual perception feature representation. Therefore, with its superior learning of visual apparent features ability, has been widely recognized in the field of representation learning. The success of Convolutional Neural Networks (CNN) has demonstrated the high ability of Convolutional Neural Networks to learn visual representations. For example, the residual network achieved a top-5 error rate of 3.57% on the ImageNet test set, breaking the best recognition performance known to humans before. However, a video frame is a time-series image, and the large dynamic changes and processing complexity make it difficult for the velocity model to learn a powerful and general spatiotemporal representation.

At present, the main method is to expand the convolution kernel of CNN from 2D to 3D, and train a new 3D CNN. By expanding a temporal dimension on the basis of 2D CNN, the network can not only extract each video image The visual appearance features that exist in the frame, and the dynamic information between consecutive frames can be captured. However, while the 3D convolution kernel improves the performance of the model, the expensive computational cost in network training has also become a problem to be solved. Taking a widely used 11-layer 3D CNN, the C3D network as an example, the model size reaches 321MB. With the increase of the quadratic method of model parameters, it is imperative to study the effective replacement of 3D convolution kernels. Furthermore, in the current two-stream action recognition model, the spatial flow network and the temporal flow network lack interaction before the final decision fusion, and the representation capabilities accumulated in multiple network layers are not fully developed. There are relatively few studies on effectively realizing the complementarity of spatial and temporal features. Therefore, how to conduct research on the difficulty of training C3D model parameters and the limitation of shallow network representation ability, effectively improve the ability and efficiency of 3D convolutional neural network model to process video actions, and how to fully and effectively realize dual-stream network fusion and complementarity , improving the performance of recognition is a very important task.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a video action classification method based on residual 3D CNN and multimodal feature fusion.

To achieve the above object, the present invention provides the following technical solutions:

A video action classification method based on residual 3D CNN and multimodal feature fusion, including the following steps:

S1: Based on the traditional Convolutional 3D Neural Networks (C3D), the connection mode of each convolution module is changed to residual connection, and identity mapping is introduced;

S2: In the residual module, the original 3D convolution kernel is decomposed into spatial kernels and multiple parallel multiscale temporal transform layers (MTTL) using 3D kernel decomposition technology to reduce model parameters. Then, Embed the attention model (Convolutional block attention module, CBAM), get a new residual module (A3D block);

S3: By stacking A3D blocks and pooling layers, adjust the input and output settings of each module to complete the construction of the final A3D residual network;

S4: Use the designed A3D convolutional residual neural network model to build a spatiotemporal dual-stream recognition model, and use the two modalities of RGB video image and optical flow image as network input respectively;

S5: Combine multi-stage feature fusion and decision fusion methods (multi-stage fusion methods), first fuse the features of different layers in the temporal network and the spatial network at the feature level, and then use the decision-level weight fusion strategy to weigh the classes of multiple softmax classifiers Fractional vector to achieve fractional-level decision fusion;

S6: Principal component analysis (PCA) dimensionality reduction algorithm is then used to reduce the dimensionality and decorrelation of the fused feature descriptors, and finally complete the classification and recognition of video actions through a multi-class SVM classifier.

Further, in step S1, the sequential direct connection between each feature module in the original C3D is changed to a residual connection, which specifically includes:

The original input x _n-1 of the feature module, that is, the identity mapping Indentitymapping, and the sum of its output are taken as the new output _yn , which is expressed as _yn =R*(x _n-1 ,W)+x _n-1 , where W represents the trainable parameter in the residual module, and the variable residual value in the network training is fitted through the residual mapping R* combined with the original input x _n-1 , R*+x _n-1 represents the shortcut connection, guaranteeing the Layer information is not easily lost when it propagates deeper into the network, avoiding gradient dispersion and gradient explosion.

Further, the 3D kernel decomposition described in step S2 includes:

Using the 3D kernel decomposition technology, the 3×3×3 convolution kernel is decomposed along the spatial and temporal dimensions to obtain a 1×3×3 spatial convolution kernel and a 3×1×1 time convolution kernel, Reduce model parameters; at the same time, in order to solve the shortcoming of a single time capture scale when the model processes time series frame image feature information, the present invention enriches the time kernel scale, and incorporates 1×1×1 and 2×1×1 different scale time kernels , a multi-scale temporal transform layer (MTTL) is designed to improve the model's ability to extract multi-granularity temporal information in the temporal domain.

Further, as described in step S2, an attention module CBAM is introduced into the residual module. CBAM is divided into channel attention module (CAM) and spatial attention module (SAM), wherein

In the channel attention model, the input feature F∈R ^C×W×H (where C, W, H respectively represent the number of feature plane channels, width and height) are passed through max pooling and average pooling respectively. (avgpool), compress the spatial dimension, and then use the multi-layer perceptual layer (MLP) to obtain the channel weights, and finally add them together. Through the relu activation layer, they are mapped to each feature channel of the input feature to achieve a reasonable attention score for the input feature channel. Assignment, the process calculation is expressed as: M _c =relu{MLP(max pool(F))+MLP(avgpool(F))}, M _c is the output of CAM, that is, the salient feature after channel weighting;

In the spatial attention model, the channel dimension of Mc is also compressed through _maxpool and average pooling (avgpool), and two-channel features with channel saliency are obtained by concatenating two feature descriptors, and then use A convolution operation Conv calculates Conv[max pool(F), _avgpool (F)} to obtain the spatial weight, which is normalized and added to Mc to obtain the spatial saliency feature; since CAM and SAM are complementary in spatial attention, making CBAM can realize all-round screening of feature space information; in the residual module, the CBAM model directly receives the output of the spatial kernel as input, giving the model an effective feature screening mechanism.

Further, the construction process of the dual-stream recognition model described in step S4 is as follows:

The A3D convolutional residual neural network is used as the basic model of the dual-stream network, and the RGB image features and the corresponding optical flow features are used as the input of the spatial and temporal flow networks respectively; the optical flow features are obtained by using the spatial pyramid model (Spatial). pyramid networks, SpyNet), the model is directly connected to the dual-stream network, and participates in training with the time-stream network and the spatial network through the back-propagation of the gradient to fine-tune its own parameters. Different from hand-crafted methods to extract optical flow information, optical flow computed from learning networks is more flexible to characterize action classification in real-world scenes.

Further, the multi-level feature fusion and decision fusion method described in step S5 specifically includes:

From the different feature layers of the A3D convolutional residual neural network, including A3D_2a, A3D_3a, A3D_5a and softmax layers, derive multi-level complementary features f _i ^* , f _i , where f _i ^* , f _i represent from the temporal flow network and The multi-level features of the spatial flow network, and then the derived features are combined with the corresponding temporal flow and spatial flow features by weighted summation to weigh the contribution of the dual-flow network, that is, to calculate F _i =W _i [f _i , f _i ^* ], where F _i , W _i are the output of the i-th layer feature fusion and the corresponding weight fusion parameter matrix, denoted as α _i , β _i ; then the weighted fusion features pass through a 1×1×1 volume The accumulation layer and the maximum pooling layer, after sofmax, get the decision score generated by the fusion features of each layer, and then perform a fractional weight fusion on the decision score of each layer to obtain a feature descriptor with strong representation.

The beneficial effects of the present invention are: compared with the original C3D model, the spatiotemporal dual-stream A3D convolutional residual neural network proposed by the present invention can achieve higher recognition efficiency with fewer model parameters, and at the same time a deeper network The model is further improved in feature representation, which can further improve the accuracy of action classification.

Other advantages, objects, and features of the present invention will be set forth in the description that follows, and will be apparent to those skilled in the art based on a study of the following, to the extent that is taught in the practice of the present invention. The objectives and other advantages of the present invention may be realized and attained by the following description.

Description of drawings

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be preferably described in detail below with reference to the accompanying drawings, wherein:

1 is a flowchart of a video action classification method based on residual 3D CNN and multimodal feature fusion according to the present invention;

Figure 2 is a C3D model diagram;

Figure 3 is a schematic diagram of 2D convolution and 3D convolution operations;

Figure 4 is a structural diagram of CBAM;

Figure 5 is a schematic diagram of the A3D residual module;

Figure 6 is a structural diagram of the A3D convolutional residual neural network;

FIG. 7 is a diagram of the overall dual-stream action recognition model.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments are only used to illustrate the basic idea of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

Among them, the accompanying drawings are only used for exemplary description, and represent only schematic diagrams, not physical drawings, and should not be construed as limitations of the present invention; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings will be omitted, The enlargement or reduction does not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the accompanying drawings may be omitted.

The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms “upper”, “lower”, “left” and “right” , "front", "rear" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must be It has a specific orientation, is constructed and operated in a specific orientation, so the terms describing the positional relationship in the accompanying drawings are only used for exemplary illustration, and should not be construed as a limitation of the present invention. situation to understand the specific meaning of the above terms.

As shown in FIG. 1, the present invention provides a video action classification method based on residual 3D CNN and multi-modal feature fusion. First, the present invention extracts the first 20 frames of images in the video, and crops all input frames into 112× 112 size, as the input of the network, the batch size used is 20 videos. As an early classic 3DCNN model, the C3D convolutional neural network includes 5 convolutional modules and 2 fully connected layers, and a total of 11 shallow models. The specific C3D model structure is shown in Figure 2. During training, the model propagates gradients and updates parameters through a single connection in which the output of the front layer is sequentially connected to the back layer. The large model parameters and insufficient model representation capabilities are the areas to be improved by the present invention. Next, the specific steps of the present invention are introduced in detail.

The construction process of A3D convolutional residual neural network:

(1) Establishing residual connection: The present invention changes the sequential direct connection between each feature module in the original C3D _to a residual connection. ) and its output as a new output y _n , the specific process is expressed as y _n =R*(x _n-1 ,W)+x _n-1 , where W represents the trainable parameter in the residual module, through the residual The mapping R* combines the original input x _n-1 to fit the variable residual value in the network training, R*+x _n-1 , which represents the shortcut connection, which ensures that the information of the previous layer is not easily lost when it propagates to the deeper layers of the network, avoiding Gradient dispersion and gradient explosion.

(2) 3D kernel decomposition: 2D convolution output lacks time domain information, while 3D convolution can capture both time domain and spatial domain information. The specific operation is shown in Figure 3. However, the heavy training parameters reduce the efficiency of network training. The invention uses the kernel decomposition technology to decompose the 3×3×3 convolution kernel along the space dimension and the time dimension to obtain a 1×3×3 spatial convolution kernel and a 3×1×1 time convolution kernel. , reduce the model parameters. At the same time, in order to solve the shortcoming of a single time capture scale when the model processes the feature information of time series frame images, the present invention enriches the time kernel scale and incorporates different scale time kernels of 1×1×1 and 2×1×1. Multiscale temporal transform layers (MTTL) to improve the ability to extract multi-granularity temporal information in the temporal domain.

(3) Introduction of attention module: Following the above process, the present invention introduces an attention model CBAM into the residual module. CBAM is mainly divided into channel attention module (CAM) and spatial attention module (Spatial attention module). , SAM). The model structure can be seen in Figure 4. ①In the channel attention model, the input feature F∈R ^C×W×H (where C, W, H represent the number of feature plane channels, width and height, respectively) are passed through max pooling and average pooling respectively. (avgpool), compress the spatial dimension, and then use the multi-layer perceptual layer (MLP) to obtain the channel weight, and finally add it, through the relu activation layer, and then map it to each feature channel of the input feature to realize the attention score of the input feature channel. Reasonable allocation, the process calculation is expressed as: M _c =relu{MLP(max pool(F))+MLP(avgpool(F))}, M _c is the output of CAM, that is, the saliency feature after channel weighting. ② In spatial attention, two pooling methods are also used to compress the channel dimension of _Mc , and two-channel features with channel saliency are obtained by concatenating two feature descriptors, and then a convolution operation Conv is used to calculate Conv [max pool(F), avgpool(F)} gets the spatial weight, which is normalized and added to M _c to get the spatial saliency feature. Since CAM and SAM complement each other in spatial attention, CBAM can realize all-round screening of feature space information. In the residual module, the CBAM model directly receives the output of the spatial kernel as input, giving the model an effective feature screening mechanism.

(4) A3D residual module: With the above foreshadowing, in the residual module of the invention, we use kernel decomposition technology to reduce model parameters, design the temporal feature granularity of MTTL rich model capture, and introduce effective attention The model improves the robustness of the model, and combines these advantages to obtain the A3D residual module. The detailed structure is shown in Figure 5.

(5) Building an A3D convolutional residual neural network: the present invention replaces the A3D module with the convolution module at the corresponding position of the original C3D, and adjusts the corresponding dimension output, in order to keep the input and output dimensions of each convolution module consistent with the C3D . By stacking the A3D modules, we finally get a convolutional neural network structure with more layers, that is, the A3D convolutional residual neural network shown in Figure 6.

The construction process of the dual-stream recognition model:

(1) Deriving multi-modal features: The present invention uses A3D convolutional residual neural network as the basic model of the dual-stream network, and uses RGB image features and corresponding optical flow features as the input of spatial flow and temporal flow networks respectively. The acquisition of optical flow features is derived by using the Spatial pyramid network (SpyNet), which is directly connected to the dual-stream network, and participates in training together with the temporal flow network and the spatial network through the back-propagation of the gradient, and fine-tuning own parameters. Different from hand-crafted methods to extract optical flow information, optical flow computed from learning networks is more flexible to characterize action classification in real-world scenes.

(2) Multi-stage fusion methods (multi-stage fusion methods): Next, in the built dual-stream recognition network, the present invention uses different feature layers (A3D_2a, A3D_3a, A3D_5a) of the A3D convolutional residual neural network respectively. and softmax layer) to derive multi-level complementary features f _i ^* , f _i , where f _i ^* , f _i represent multi-level features from the temporal flow network and the spatial flow network, respectively, and then, the derived features are weighted and summed Fusion of the corresponding temporal flow and spatial flow features aims to balance the contribution of the dual-stream network, that is, to calculate F _i =W _i [f _i , f _i ^* ], where F _i , W _i are the outputs of the i-th layer feature fusion, respectively and the corresponding weight fusion parameter matrix (denoted in detail as α _i , β _i ). Then, the weighted and fused features pass through a 1×1×1 convolutional layer and a maximum pooling layer, and after sofmax, the decision scores generated by the fusion features of each layer are obtained. A fractional fusion of weights is used to produce feature descriptors with strong representational power. Finally, the feature vectors are de-correlated and de-redundant through PCA, and the obtained effective features are then entered into a multi-class SVM classifier to complete the final recognition task. The overall dual-stream action recognition model is shown in Figure 7.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent replacements, without departing from the spirit and scope of the technical solution, should all be included in the scope of the claims of the present invention.

Claims (6)

1. a video action classification method based on residual type 3D CNN and multimodal feature fusion, is characterized in that: comprise the following steps: S1: Based on the traditional convolutional 3D neural network C3D, the connection mode of each convolution module is changed to residual connection, and the identity mapping is introduced; S2: In the residual module, the original 3D convolution kernel is decomposed into spatial kernels and multiple parallel multi-scale time kernels MTTL by using 3D kernel decomposition technology to reduce the model parameters. Then, the attention model CBAM is embedded to obtain A new residual module A3Dblock; S3: By stacking A3D blocks and pooling layers, adjust the input and output settings of each module to complete the construction of the final A3D residual network; S4: Use the designed A3D convolutional residual neural network model to build a spatiotemporal dual-stream recognition model, and use the two modalities of RGB video image and optical flow image as network input respectively; S5: Combined use of multi-level feature fusion and decision fusion methods. First, the features of different layers in the temporal network and the spatial network are fused at the feature level, and then the class score vectors of multiple softmax classifiers are weighed through the decision-level weight fusion strategy to achieve the score level. decision fusion; S6: PCA dimensionality reduction algorithm is then used to reduce the dimensionality and decorrelation of the fused feature descriptors, and finally complete the classification and recognition of video actions through the multi-class SVM classifier. 2. the video action classification method based on residual type 3D CNN and multimodal feature fusion according to claim 1, it is characterized in that: in step S1, the mode of order direct connection between each feature module in the original C3D is changed to It is a residual connection, including: Take the original input x _n-1 of the feature module, that is, the identity map, and the sum of its output as a new output y _n , expressed as _yn =R ^* (x _n-1 ,W)+x _n-1 , where W Represents the trainable parameters in the residual module, through the residual mapping R ^* combined with the original input x _n-1 , fitting the variable residual value in the network training, R ^* +x _n-1 represents the shortcut connection, ensuring the front layer Information is not easily lost when propagating deeper into the network, avoiding gradient dispersion and gradient explosion. 3. the video action classification method based on residual type 3D CNN and multimodal feature fusion according to claim 1, is characterized in that: the 3D kernel decomposition described in step S2 comprises: Using the 3D kernel decomposition technology, the 3×3×3 convolution kernel is decomposed along the spatial and temporal dimensions to obtain a 1×3×3 spatial convolution kernel and a 3×1×1 time convolution kernel, Reduce model parameters; simultaneously incorporate 1×1×1 and 2×1×1 time kernels of different scales, and design a multi-scale time transition layer MTTL to improve the ability to extract multi-granularity time information in the time domain. 4. the video action classification method based on residual type 3D CNN and multimodal feature fusion according to claim 1, is characterized in that: described in step S2, introduce attention module CBAM in residual error module, CBAM is divided into. channel attention CAM and spatial attention SAM, where In the channel attention model, the input feature F∈R ^C×W×H , where C, W, H represent the number of channels, width and height of the feature plane, respectively, through max pooling and average pooling, respectively, to compress the space Dimension, and then use the multi-layer perceptual layer (MLP) to obtain the channel weight, and finally add it. Through the relu activation layer, it is mapped to each feature channel of the input feature to achieve a reasonable allocation of the attention score of the input feature channel. The process calculation is expressed as : M _c =relu{MLP(maxpool(F))+MLP(avgpool(F))}, M _c is the output of CAM, that is, the salient feature after channel weighting; In the spatial attention model, the channel dimension of _Mc is also compressed through maximum pooling and average pooling, and two-channel features with channel saliency are obtained by concatenating two feature descriptors, and then a convolution operation Conv is used to calculate Conv[maxpool(F), _avgpool (F)} gets the spatial weight, which is normalized and added to Mc to get the spatially significant features; since CAM and SAM complement each other in spatial attention, CBAM can realize the feature spatial information analysis. In the residual module, the CBAM model directly receives the output of the spatial kernel as input, giving the model an effective feature screening mechanism. 5. the video action classification method based on residual type 3D CNN and multimodal feature fusion according to claim 1, is characterized in that: the building process of the dual-stream recognition model described in step S4 is as follows: The A3D convolutional residual neural network is used as the basic model of the dual-stream network, and the RGB image features and the corresponding optical flow features are used as the input of the spatial flow and temporal flow networks respectively; the acquisition of the optical flow features is derived by using the spatial pyramid model SpyNet , the model is directly connected to the dual-stream network, and participates in training with the time-stream network and the spatial network through the back-propagation of the gradient to fine-tune its own parameters. 6. the video action classification method based on residual type 3D CNN and multi-modal feature fusion according to claim 1, is characterized in that: the multi-level feature fusion and decision fusion method described in step S5, specifically comprises: From the different feature layers of the A3D convolutional residual neural network, including A3D_2a, A3D_3a, A3D_5a and softmax layers, derive multi-level complementary features f _i ^* , f _i , where f _i ^* , f _i represent from the temporal flow network and The multi-level features of the spatial flow network, and then the derived features are combined with the corresponding temporal flow and spatial flow features by weighted summation to weigh the contribution of the dual-flow network, that is, to calculate F _i =W _i [f _i , f _i ^* ], where F _i , W _i are the output of the i-th layer feature fusion and the corresponding weight fusion parameter matrix, denoted as α _i , β _i ; then the weighted fusion features pass through a 1×1×1 volume The accumulation layer and the maximum pooling layer, after sofmax, get the decision score generated by the fusion features of each layer, and then perform a fractional weight fusion on the decision score of each layer to obtain a feature descriptor with strong representation.

Images