CN115836330A - Action recognition method and related products based on deep residual network - Google Patents

Abstract

An action recognition method based on a deep residual network and related products are provided. A first convolution module in the at least one convolution module receives a video segment as input. Traverse at least one convolutional module by: processing the above input with a 1D filter for motion-related features and a 2D filter for appearance-related features; shifting the above motion-related features along the temporal dimension A step size, subtracting the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain residual features; obtain the output of the above-mentioned convolution module as the input of the next convolution module, until at least one of the above-mentioned convolutions is traversed The last convolutional module in the module. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module. Thereby reducing the model size and reducing the computational cost.

Description

Action recognition method and related products based on deep residual network

technical field

The present disclosure relates to the technical field of neural networks, in particular to an action recognition method based on a deep residual network and related products.

Background technique

The resurgence of convolutional neural networks (CNNs) and large-scale labeled datasets has enabled unprecedented advances in image classification using end-to-end trainable networks. However, video-based human action recognition cannot yet be achieved based solely on CNN features. How to efficiently model temporal information, that is, identify temporal correlations and causal relationships, is a fundamental challenge.

Currently, there is a classical branch of research focused on modeling motion via human-designed optical flow, and a two-stream approach using optical flow modalities and red-green-blue (RGB) in separate streams is the most successful architecture. one. However, optical flow is computationally expensive.

Contents of the invention

The embodiment provides an action recognition method based on a deep residual network and related products, so as to reduce the size of the model of the deep residual network for video-based human action recognition and reduce the calculation cost.

In the first aspect, an action recognition method based on deep residual network is provided. The action recognition method based on the deep residual network is applied to a deep residual network system including at least one convolution module, each convolution module in the at least one convolution module includes at least one first convolution layer, and the above at least Each first convolutional layer in a first convolutional layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above method includes the following content. A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module.

In a second aspect, an action recognition device based on a deep residual network is provided. The action recognition device based on the deep residual network is applied to a deep residual network system including at least one convolution module, each convolution module in the at least one convolution module includes at least one first convolution layer, and the above at least Each first convolutional layer in a first convolutional layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above device includes a receiving unit, a processing unit, and an identifying unit. The above-mentioned receiving unit is used for receiving a video segment as an input in the first convolution module of the at least one convolution module. The above-mentioned processing is used to traverse the above-mentioned at least one convolution module by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; Shift a step along the time dimension, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain residual features; The output of the convolution module; the output of the above convolution is used as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. The identification unit is configured to identify at least one action included in the video clip based on the output of the last convolution module in the at least one convolution module.

In a third aspect, a terminal device is provided, and the terminal device includes a processor and a memory for storing one or more programs. The above-mentioned one or more programs are used to be executed by the above-mentioned processor, and the above-mentioned one or more programs include instructions for performing part or all of the operations in the method described in the first aspect.

In a fourth aspect, there is provided a non-transitory computer-readable storage medium for storing a computer program for electronic data exchange. The above computer program includes instructions for performing part or all of the operations in the method described in the first aspect.

In a fifth aspect, a computer program product is provided, and the computer program product includes a non-transitory computer-readable storage medium storing a computer program. The above computer program can cause the computer to execute part or all of the operations in the method described in the first aspect.

In the embodiment of this application, a new deep residual network system is provided. The above-mentioned deep residual network system includes at least one convolution module, each convolution module in the above-mentioned at least one convolution module includes at least one first convolution layer, and each first convolution module in the above-mentioned at least one first convolution layer The product layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above-mentioned action recognition method based on the deep residual network is applied to the above-mentioned deep residual network system. A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module. Therefore, in this application, a new convolution module that can be regarded as a pseudo-3D convolution module is proposed, in which the standard 3D filters in the related art are decoupled to form parallel 2D spatial filters and a one-dimensional time-domain filter. By using separable 2D and 1D convolutions instead of 3D convolutions, the model size and computational cost are greatly reduced. Furthermore, 2D convolutions and 1D convolutions are placed in different paths, allowing for different modeling of appearance-related features and motion-related features.

Description of drawings

In order to illustrate the technical solutions in the embodiments more clearly, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

Figure 1 is a schematic diagram of an RGB frame example (top) and a residual frame example (bottom).

Figure 2 is a schematic diagram of an exemplary detailed design of a deep residual network.

Fig. 3 is a schematic flowchart of an action recognition method based on a deep residual network according to an embodiment.

Fig. 4 is a schematic flowchart of an action recognition method based on a deep residual network according to an embodiment.

Fig. 5 is a schematic diagram of an exemplary detailed design of the proposed convolution module.

Fig. 6 is a schematic structural diagram of an action recognition device based on a deep residual network according to an embodiment.

Fig. 7 is a schematic structural diagram of a terminal device according to an embodiment.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the embodiment, the technical solution in the embodiment will be clearly and completely described below in conjunction with the drawings in the embodiment. Obviously, the described embodiment is only Embodiments of a part of the application, but not all of the embodiments. Based on this embodiment, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms "first", "second" and "third" in the description and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "include" and "have", as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes Other steps or elements inherent to the process, method, product or apparatus.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

The terminal devices involved in the embodiments of the present invention may include various handheld devices, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem with wireless communication functions, and various forms of user equipment (UE), Mobile Station (MS), Mobile Terminal, etc. For convenience of description, the devices mentioned above are collectively referred to as terminal devices.

In order to better understand the embodiments of the present application, the related technologies involved in the present application will be briefly introduced below.

The resurgence of convolutional neural networks (CNNs) and large-scale labeled datasets has enabled unprecedented advances in image classification using end-to-end trainable networks. However, video-based human action recognition cannot yet be achieved based solely on CNN features. How to efficiently model temporal information, that is, identify temporal correlations and causal relationships, is a fundamental challenge. There is a classic branch of research that focuses on modeling motion via human-crafted optical flow. In the context of deep learning, a two-stream approach using optical flow and RGB modalities in separate streams is one of the most successful architectures. However, given the high computational cost of optical flow, and the fact that two-stream approaches using optical flow and RGB modalities in separate streams are usually not learnable end-to-end together with optical flow, this architecture cannot methodologically make People are satisfied.

In this application, residual frames, i.e., the differences between adjacent RGB frames, are proposed as an alternative "lightweight" motion representation, which are combined with RGB modalities for video-based human action recognition. The reason why residual frames can be used together with RGB modality for video-based human action recognition can be formulated as follows. On the one hand, adjacent RGB frames largely share the information of stationary objects and background information, so residual frames usually mainly preserve motion-specific features, as shown in Fig. 1. Figure 1 shows an RGB frame example (top) and a residual frame example (bottom). As can be seen from Figure 1, RGB frames contain rich appearance information, while residual frames mainly retain salient motion information. On the other hand, the computational cost of residual frames is negligible compared to other motion representations such as optical flow.

Following the recent trend of developing three-dimensional (3D) convolutional models for video classification, in this application a novel efficient convolutional module is presented, which can be viewed as a pseudo-3D convolutional module, in which In the convolution module, the original 3D convolution is decoupled into 2D and 1D convolution. Furthermore, to further enhance the motion features and reduce the computational cost, residual information in the feature space, i.e. residual features representing the differences between temporally adjacent CNN features, can be exploited. In addition, the self-attention mechanism can also be used to recalibrate the appearance-related features and motion-related features based on their importance to the final task to further reduce the model size and computational cost, and avoid being unimportant to the final task. The influence of features on the calculation accuracy of the deep residual network system.

The deep residual network-based action recognition method and related products provided in this application are computationally efficient and have improved performance. Specifically, the proposed residual frame (or residual feature) is a lightweight alternative compared to other motion representations (such as optical flow), and the new convolution module also helps to significantly reduce the computational cost. In addition, experiments have demonstrated that using residual frames can significantly improve the accuracy of action recognition, which will be described in detail below.

In order for those skilled in the art to better understand the concept of residual features, the concept of residual frames is firstly introduced.

中减去参考帧

来形成，其中时间戳t₁和t₂之间的步长表示为s。更正式地，残差帧可以定义为：Assume a video clip x∈R ^T×H×W×C , where T, H, W represent the length, height and width of each frame, respectively, and C represents the number of channels. The residual frame can be obtained from the desired frame by

Subtract the reference frame from

to form, where the step size between timestamps _t1 and _t2 is denoted as s. More formally, a residual frame can be defined as:

Since adjacent video frames have significant similarity in static information, residual frames usually do not contain background information and appearance information of objects, but retain significant motion-related information. Therefore, residual frames can be regarded as a good source for extracting motion features. Furthermore, the computational cost of residual frames can be significantly reduced compared to other motion representations such as optical flow.

In reality, actions and activities contained in videos can be complex and may involve different motion speeds or motion durations. To deal with this uncertainty, consecutive residual frames can be stacked to form a residual segment, which can be defined as:

The above residual segments can capture fast motion on the spatial axis and slow/long duration motion on the time axis. Therefore, the residual segment is suitable for 3D convolution, in which both short-duration and long-duration motion information can be extracted simultaneously.

However, residual frames alone may not be sufficient for human action recognition since object appearance and background scenes can also provide important information for recognizing actions. For example, in the scene of applying eye makeup and applying lipstick, the movements of applying eye makeup and applying lipstick are similar, but the positions of the movements are different, one of which occurs around the eyes and the other around the lips. Therefore, both RGB frames and residual frames need to be utilized for action recognition. To this end, a new convolutional module is provided, which is a pseudo-3D convolutional module with the ability to simultaneously process RGB frames and residual frames.

The embodiments of the present application will be described in detail below.

Figure 2 is a schematic diagram of an exemplary detailed design of a deep residual network. As shown in Figure 2, the deep residual network may at least include: an input layer, at least one convolutional layer, a pooling layer, at least one fully connected layer and an output layer. Deep residual networks for action recognition based on video clips as input. The deep residual network system in this application is based on the deep residual network system shown in FIG. 2 .

Fig. 3 is a schematic flowchart of an action recognition method based on a deep residual network according to an embodiment. The action recognition method based on the deep residual network is applied to a deep residual network system including at least one convolution module, each convolution module in the at least one convolution module includes at least one first convolution layer, and the at least one first convolution layer Each first convolutional layer in a convolutional layer has at least one one-dimensional filter and at least one two-dimensional filter. As shown in Figure 3, the method includes the following contents.

302. A first convolution module in the at least one convolution module receives a video segment as an input.

Specifically, a video segment x ∈ R ^T×H×W×C can be received as an input to a deep residual network system, where T, H, and W denote the length, height, and width of each frame, respectively, and C denotes the number of channels. For RGB frames, the number of channels is 3, and these three channels represent red (R), green (G) and blue (B) respectively. When there are no other convolution modules or layers before the first of the at least one convolution modules, the video segment is passed to the first of the at least one convolution modules, and the A first of the at least one convolutional modules is received as input. When there are other layers before the first convolution module of the at least one convolution module, the video segment may first be processed by other layers and then passed to the first convolution module of the at least one convolution module , and is further received as an input at the first convolution module of the at least one convolution module.

The size of the filter can be expressed as T×H×W, where T represents the time dimension, H represents the height in the space dimension, and W represents the width in the space dimension. A 1D filter can be expressed as T × 1 × 1, where T is greater than 1, and a 2D filter can be expressed as 1 × H × W, where at least one of H and W is greater than 1. 1D filters are used for convolutions in the time dimension Product, 2D filter for convolution of spatial dimensions.

304. Traverse the at least one convolution module by performing the following operations.

3042. Use a one-dimensional filter to process the above input to obtain motion-related features, and use a two-dimensional filter to process the above input to obtain appearance-related features.

In this application a new convolutional module that can be regarded as a pseudo 3D convolutional module is presented. For the convolution module, the 3D filter in the related art is decoupled to form a parallel 2D spatial filter and a 1D time domain filter. Therefore, by replacing 3D convolutions with separable 2D convolutions and 1D convolutions, the model size and computational cost are greatly reduced, which is in line with the latest trend in the development of efficient 3D networks. Furthermore, 2D convolutions and 1D convolutions are placed in different paths, allowing for different modeling of appearance and motion features.

3044. Shift the motion-related feature by one step along the time dimension, and subtract the shifted motion-related feature from the motion-related feature to obtain a residual feature.

其可以定义为：Specifically, the idea of residual frames is extended from pixel level to feature level when modeling motion. Assuming that the output features from 1D temporal convolution are f ^m ∈ ^{RT'×H'×W'×C,} , the output features of 1D temporal convolution are shifted by a step along the time dimension, e.g. step 1, The residual feature can then be generated by subtracting the shifted motion-related feature f ^m (t+1) from the original motion-related feature f ^m (t)

It can be defined as:

3046. Acquire the output of the convolution module based on the appearance-related feature, the motion-related feature, and the residual feature.

其中f^s是保留外观信息的2D卷积的输出，f^m是1D卷积的输出，f^m和

保留了区别运动结构。Three features are created after the pseudo 3D convolution

where f ^s is the output of a 2D convolution preserving appearance information, f ^m is the output of a 1D convolution, f ^m and

The distinguishing motion structure is retained.

3048. Use the output of the above convolution as the input of the next (that is, subsequent) convolution module until the last convolution module in the at least one convolution module is traversed.

306. Based on the output of the last convolution module in the at least one convolution module, identify at least one action included in the video segment.

In the embodiment of this application, a new deep residual network system is provided. The above-mentioned deep residual network system includes at least one convolution module, each convolution module in the above-mentioned at least one convolution module includes at least one first convolution layer, and each first convolution module in the above-mentioned at least one first convolution layer The product layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above-mentioned action recognition method based on the deep residual network is applied to the above-mentioned deep residual network system. A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module. Therefore, in this application, a new convolution module that can be regarded as a pseudo-3D convolution module is proposed, in which the standard 3D filters in the related art are decoupled to form parallel 2D spatial filters and a one-dimensional time-domain filter. By using separable 2D and 1D convolutions instead of 3D convolutions, the model size and computational cost are greatly reduced. Furthermore, 2D convolutions and 1D convolutions are placed in different paths, allowing for different modeling of appearance-related features and motion-related features.

In one embodiment, the output of the above-mentioned convolution module is obtained in the following manner: by cascading the above-mentioned motion-related features, the above-mentioned residual features and the above-mentioned appearance-related features in the channel dimension, the cascaded features are obtained; the above-mentioned cascaded features are determined is the output of the above convolution module.

To facilitate the effective fusion of appearance features and motion features, the output features of pseudo-3D convolutions can be concatenated in the channel dimension to obtain cascaded features, which can be defined as:

代表级联。Among them, the symbol

Represents cascading.

In one embodiment, the output of the above-mentioned convolution module is obtained in the following manner: by cascading the above-mentioned motion-related features, the above-mentioned residual features and the above-mentioned appearance-related features in the channel dimension, the cascaded features are obtained; based on the above-mentioned cascaded features, the obtained a channel attention mask; and based on the above channel attention mask and the above cascaded features, obtaining attention features as the above output of the above convolution module.

To facilitate the effective fusion of appearance-related features and motion-related features, a channel self-attention mechanism can be further applied to recalibrate the output features. Specifically, the output features can be concatenated in the channel dimension to obtain cascaded features, which can be defined as:

代表级联。由于f^m∈R^{T’×H’×W’×C’}，f^s∈R^{T’×H’×W’×C’}，

则级联后，f∈R^{T’×H’×W’×3C’}。Among them, the symbol

Represents cascading. Since f ^m ∈ ^{RT'×H'×W'×C'} , f ^s ∈ ^{RT'×H'×W'×C'} ,

After cascading, f∈RT ^{'×H'×W'×3C'} .

Then, the channel attention mask M _att can be obtained based on the above cascaded features. In one embodiment, each convolution module in the at least one convolution module further includes a fully connected layer, and the channel attention mask described above can be obtained based on the cascaded features in the following manner. Global pooling is performed on the above cascaded features to obtain the pooled cascaded features, which can be expressed as pool(f). Multiply the pooled cascaded features by the weight matrix of the fully connected layer to obtain weighted cascaded features, which can be expressed as Wpool(f). Add the above weighted cascaded feature and bias to obtain the biased cascaded feature, which can be expressed as Wpool(f)+b. Use the Sigmoid function to process the above-mentioned bias cascade feature to obtain the above-mentioned channel attention mask, which can be expressed as σ(Wpool(f)+b). Therefore, the above channel attention mask M _att can be expressed as:

M _att =σ(Wpool(f)+b),

代表由单层神经网络(即上述卷积模块的全连接层)参数化的权重矩阵，

代表偏差项，pool是跨空间和时间对级联特征f进行平均的全局池化操作，σ代表sigmoid函数。通过通道注意力掩膜，实现了动态特性，该动态特征以输入特征和基于输入特征对最终任务的重要性的重新加权通道为条件。in,

Represents the weight matrix parameterized by a single-layer neural network (i.e., the fully connected layer of the above convolutional module),

Represents the bias term, pool is a global pooling operation that averages the cascaded feature f across space and time, and σ represents the sigmoid function. Through channel attention masks, dynamics are achieved that are conditioned on input features and reweighted channels based on the importance of the input features to the final task.

After obtaining the above channel attention mask, attention features can be further obtained based on the above channel attention mask and the above cascaded features. In one embodiment, the attention feature is obtained by performing channel-by-channel multiplication between the above-mentioned channel attention mask M _att and the above-mentioned cascaded feature f. In another embodiment, in order to further improve the robustness, the above-mentioned attention feature is obtained in the following manner: by performing channel-by-channel multiplication between the above-mentioned channel attention mask M _att and the above-mentioned cascade feature f to obtain intermediate features; Add the above-mentioned intermediate features and the above-mentioned cascade features to obtain the above-mentioned attention features, and the above-mentioned attention features can be defined as:

f _att ＝f⊙M _att +f

Among them, the symbol "⊙" means channel-by-channel multiplication. Therefore, residual connections can be implemented in the proposed convolutional module.

In one embodiment, each convolution module of the at least one convolution module further includes a second convolution layer before the at least one first convolution layer, and the second convolution layer includes ×1 three-dimensional filter, traversing the at least one convolution module further includes: using the above-mentioned one-dimensional filter to process the above-mentioned input to obtain motion-related features, and before using the above-mentioned two-dimensional filter to process the above-mentioned input to obtain appearance-related features, use The above-mentioned three-dimensional filter processes the above-mentioned input to reduce the dimensionality of the above-mentioned input. Processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with the above-mentioned two-dimensional filter to obtain appearance-related features include: processing the above-mentioned input after dimension reduction with the above-mentioned one-dimensional filter to obtain the above-mentioned motion-related features , using the above-mentioned two-dimensional filter to process the above-mentioned input after dimensionality reduction to obtain the above-mentioned appearance-related features.

Specifically, before being processed by the 1D filter and 2D filter of the next convolution module, the output of the previous convolution module can be processed by a 3D filter with a size of 1 × 1 × 1, so that the previous convolution can be reduced. The number of channels of the output of the convolution module, thereby reducing the dimensionality of the output of the previous convolution module (ie, the input of the next convolution module). The 3D filter may include at least one 3D filter. The number of 3D filters can be determined based on the desired dimensions. For example, in order to reduce the dimension, the number of 3D filters is less than the number of channels of the output of the previous convolution module; in order to restore the dimension, the number of 3D filters is equal to the number of channels of the output of the previous convolution module.

In one embodiment, each convolution module in the at least one convolution module further includes a third convolution layer located after the at least one first convolution layer and the second convolution layer, and the third convolution The layer includes a three-dimensional filter with a size of 1×1×1, and obtaining the output of the above-mentioned convolution module also includes: using the above-mentioned three-dimensional filter to process the above-mentioned cascaded features to increase the dimension of the above-mentioned cascaded features; The concatenated features serve as the output of the above convolutional modules.

In one embodiment, each convolution module in the at least one convolution module further includes a third convolution layer located after the at least one first convolution layer and the second convolution layer, and the third convolution The layer includes a three-dimensional filter with a size of 1×1×1, and obtaining the above-mentioned output of the above-mentioned convolution module also includes: processing the above-mentioned attention feature with the above-mentioned three-dimensional filter to increase the dimension of the above-mentioned attention feature; The aforementioned attention features serve as the output of the aforementioned convolutional modules.

Specifically, top 1×1×1 convolution and bottom 1×1×1 convolution are used to reduce and restore dimensionality.

Multiple experiments are further conducted, where a variant of ResNet-60 is developed by replacing all bottleneck blocks with the proposed convolutional modules.

For example, the performance of the technical solutions proposed in this application is evaluated using the UCF101 dataset, where UCF101 is a dataset consisting of 13,330 videos containing 101 action categories. For all experiments, the top-1 (top-1) accuracy and top-5 (top-5) accuracy on the split-1 validation set are reported.

Specifically, the evaluation of the effectiveness of different data modalities is performed by training an action classifier using RGB frames alone, residual frames alone, and a combined input of RGB frames and residual frames, respectively. Second, the effect of residual frame stride on action recognition is studied. Finally, an ablation study is also performed to investigate the effectiveness of individual components in the proposed convolutional module.

Performance comparison of different data modalities.

Table 1 shows the action recognition performance for various combinations of input modalities and network architectures. For experiments using only RGB frames or residual frames, only one stream in the data layer (i.e., the first convolutional layer) is kept, but the number of channels is doubled for fair comparison. From Table 1, it can be seen that using only residual frames is about 3% higher than using only RGB frames in terms of top-1 accuracy and top-5 accuracy, which shows that residual frames do contain significant features that are important for action recognition. sports information. When RGB frames and residual frames are used in different streams, the top-1 accuracy is further improved by 2.6% (from 83.0% to 85.6%), indicating that the two data modalities preserve complementary information. It is worth noting that, compared with the case of using standard 3D convolution in the related art, using the convolution module provided in this application not only significantly reduces floating-point calculations (flops) per second (increased from 163G to 40G), And it also provides better performance (85.6% vs. 85.0% in top-1 accuracy).

Table 1: Performance comparison for different input modalities and network architectures.

The effect of the step size s.

When generating residual frames, the step size s can be varied to capture motion features at different time scales. However, it is unclear what the optimal step size is for action recognition tasks. Therefore, the effect of the step size was studied, and the results of the study are shown in Table 2. Experiments are carried out for three settings respectively, where the input data are residual frames with stride s=1, 2, 4, respectively. From Table 2, it can be seen that the classification accuracy decreases as the step size increases. We conjecture that motion causes the spatial displacement of the same object between two frames, and that using a large stride may cause a mismatch between motion representations.

Table 2: Performance comparison corresponding to different residual frame steps s.

step size Val top-1 Val top-5 s=1 83.0% 98.3% s=2 82.7% 97.1% s=4 80.2% 96.0%

Ablation studies.

To verify the effectiveness of different components in the proposed convolutional module, an ablation study is performed. Without loss of generality, the model is trained with a combined input of RGB frames and residual frames (s = 1). Table 3 shows the performance comparison corresponding to various convolution module settings. As shown in Table 3, removing the self-attention mechanism leads to a 1.8% drop in top-1 accuracy (from 85.6% to 83.8%). Meanwhile, the performance also drops from 85.6% to 83.5% when ignoring the residual information in the feature space. If both the self-attention mechanism and the residual feature are removed, the top-1 accuracy will be further reduced to 82.1%. These results confirm that the channel self-attention mechanism and residual features are effective for improving action recognition performance.

Table 3: Performance comparison for different convolution module settings

method Val top-1 Val top-5 Convolutional modules without self-attention and residual features 82.1% 97.9% Convolutional modules without self-attention 83.8% 98.4% Convolutional modules without residual features 83.5% 98.1% Convolution module 85.6% 99.2%

Fig. 4 is a schematic flowchart of an action recognition method based on a deep residual network according to an embodiment. The action recognition method based on the deep residual network is applied to the deep residual network system, the deep residual network system includes at least one convolution module, and each convolution module in the at least one convolution module includes at least one first convolution layers, each of the at least one first convolutional layer has at least one one-dimensional filter and at least one two-dimensional filter. As shown in Figure 4, the method includes the following contents.

402. A first convolution module in the at least one convolution module receives a video segment as an input.

404. Traverse the at least one convolution module by performing the following operations.

40502, Use a three-dimensional filter to process the above input to reduce the dimension of the above input.

40504. Use the above-mentioned one-dimensional filter to process the above-mentioned input after dimensionality reduction to obtain motion-related features, and use the above-mentioned two-dimensional filter to process the above-mentioned input after dimensionality reduction to obtain appearance-related features.

40506. Shift the motion-related features by one step along the time dimension, and subtract the shifted motion-related features from the motion-related features to obtain residual features.

40508. Obtain cascaded features by cascading the above-mentioned motion-related features, the above-mentioned residual features, and the above-mentioned appearance-related features in the channel dimension.

40410. Perform global pooling for the above cascaded features, and obtain pooled cascaded features.

40412. Multiply the pooled cascaded features by the weight matrix of the fully connected layer to obtain weighted cascaded features.

40414. Add the above weighted cascaded features to the bias to obtain the biased cascaded features.

40416, use the Sigmoid function to process the above bias cascade features to obtain the channel attention mask.

40418, Obtain intermediate features by performing channel-by-channel multiplication between the above channel attention mask and the above cascaded features.

40430. Add the above-mentioned intermediate features and the above-mentioned cascade features to obtain attention features.

40422. Use the above-mentioned three-dimensional filter to process the above-mentioned attention feature to increase the dimension of the above-mentioned attention feature.

40424, use the above-mentioned attention features after dimensionality enhancement as the output of the above-mentioned convolution module.

40426. Use the output of the above convolution as the input of the next convolution module, until the last convolution module in the above at least one convolution module is traversed.

406. Based on the output of the last convolution module in the at least one convolution module, identify at least one action included in the video segment.

Fig. 5 shows an exemplary detailed design of the proposed convolution module. The proposed convolutional module can be integrated into any standard CNN architecture, such as ResNet. To process RGB frames and residual frames simultaneously, the raw data layer (i.e., the first convolutional layer) is modified into two streams with parallel building blocks that output appearance-related features and motion-related features, respectively. traits, each building block for each modal. The resulting features from the two streams are concatenated and passed to the next layer (i.e., subsequent convolutional modules). In the exemplary design of the proposed convolutional module, a 2D filter of size 3×3 and a 1D filter of size 1×1 are taken as examples. Specifically, the size of the filter can be expressed as T×H×W, where T represents the time dimension, H represents the height in the space dimension, and W represents the width in the space dimension.

Table 4 shows an exemplary detailed design of the deep residual network system. As shown in Table 4, the deep residual network system includes four convolution modules, denoted as res 2, res 3, res 4 and res 5. In an exemplary design of a deep residual network system, the size of the filter can be represented by {T×S ² ,C} to represent the time, space and channel dimensions, specifically, in Table 4, the size is 3× The 2D filter of 3 and the 1D filter of size 1×1 are taken as examples for illustration.

Table 4: Exemplary detailed design of the deep residual network system.

In the embodiment of this application, a new deep residual network system is provided. The above-mentioned deep residual network system includes at least one convolution module, each convolution module in the above-mentioned at least one convolution module includes at least one first convolution layer, and each first convolution module in the above-mentioned at least one first convolution layer The product layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above-mentioned action recognition method based on the deep residual network is applied to the above-mentioned deep residual network system. A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module. Therefore, in this application, a new convolution module that can be regarded as a pseudo-3D convolution module is proposed, in which the standard 3D filters in the related art are decoupled to form parallel 2D spatial filters and a one-dimensional time-domain filter. By using separable 2D and 1D convolutions instead of 3D convolutions, the model size and computational cost are greatly reduced. Furthermore, 2D convolutions and 1D convolutions are placed in different paths, allowing for different modeling of appearance-related features and motion-related features.

The above operations can refer to the detailed description of the network training operation in the action recognition method based on the deep residual network, which will not be described here.

The foregoing mainly introduces the solutions of the embodiments of the present invention from the perspective of executing the process on the method side. It can be understood that, in order to realize the above functions, the electronic device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that the present invention can be realized in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

In the embodiment of the present invention, the functional units of the first wireless earphone can be divided according to the above method example. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of the units in the embodiment of the present invention is schematic, and is only a logical function division, and there may be another division manner in actual implementation.

Fig. 6 is a schematic structural diagram of an action recognition device based on a deep residual network according to an embodiment. The device is applied to a deep residual network system, the deep residual network system includes at least one convolution module, each convolution module in the at least one convolution module includes at least one first convolution layer, and the at least one first Each first of the convolutional layers has at least one one-dimensional filter and at least one two-dimensional filter. As shown in FIG. 6 , the above-mentioned action recognition device based on deep residual network includes a receiving unit 602 , a processing unit 604 and a recognition unit 606 .

The receiving unit 602 is configured to receive a video segment as an input in the first convolution module of the at least one convolution module.

The processing unit 604 is configured to traverse the at least one convolution module by performing the following operations: process the input with a one-dimensional filter to obtain motion-related features, and process the input with a two-dimensional filter to obtain appearance-related features; The motion-related features are shifted by a step along the time dimension, and the above-mentioned shifted motion-related features are subtracted from the above-mentioned motion-related features to obtain residual features; based on the above-mentioned appearance-related features, the above-mentioned motion-related features, and the above-mentioned residual features, Obtain the output of the above convolution module; use the output of the above convolution as the input of the next convolution module, until the last convolution module of the at least one convolution module is traversed.

The recognition unit 606 recognizes at least one action included in the video segment based on the output of the last convolution module in the at least one convolution module.

In the embodiment of this application, a new deep residual network system is provided. The above-mentioned deep residual network system includes at least one convolution module, each convolution module in the above-mentioned at least one convolution module includes at least one first convolution layer, and each first convolution module in the above-mentioned at least one first convolution layer The product layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above-mentioned action recognition method based on the deep residual network is applied to the above-mentioned deep residual network system. A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module. Therefore, in this application, a new convolution module that can be regarded as a pseudo-3D convolution module is proposed, in which the standard 3D filters in the related art are decoupled to form parallel 2D spatial filters and a one-dimensional time-domain filter. By using separable 2D and 1D convolutions instead of 3D convolutions, the model size and computational cost are greatly reduced. Furthermore, 2D convolutions and 1D convolutions are placed in different paths, allowing for different modeling of appearance-related features and motion-related features.

In one embodiment, in terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned processing unit 604 is specifically configured to: obtain the concatenated features by concatenating the above-mentioned motion-related features, the above-mentioned residual features, and the above-mentioned appearance-related features in the channel dimension ; and determining the above-mentioned concatenated features as the above-mentioned output of the above-mentioned convolution module.

In one embodiment, in terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned processing unit 604 is specifically configured to: obtain the concatenated features by concatenating the above-mentioned motion-related features, the above-mentioned residual features, and the above-mentioned appearance-related features in the channel dimension ; Obtaining a channel attention mask based on the above-mentioned cascade features; and obtaining an attention feature as the above-mentioned output of the above-mentioned convolution module based on the above-mentioned channel attention mask and the above-mentioned cascade features.

In one embodiment, in terms of obtaining the above-mentioned attention feature based on the above-mentioned channel attention mask and the above-mentioned cascade feature, the above-mentioned processing unit 604 is specifically configured to: perform The channel-wise multiplication of , to obtain the above attention features.

In one embodiment, in terms of obtaining attention features based on the above-mentioned channel attention mask and the above-mentioned cascaded features, the above-mentioned processing unit 604 is specifically configured to: channel-by-channel multiplication to obtain intermediate features; and adding the above-mentioned intermediate features to the above-mentioned cascaded features to obtain the above-mentioned attention features.

In one embodiment, each convolution module in the above at least one convolution module further includes a fully connected layer, and in terms of obtaining the above channel attention mask based on the above cascaded features, the above processing unit 604 is configured to: for the above The cascaded features perform global pooling to obtain the pooled cascaded features; multiply the above pooled cascaded features by the weight matrix of the above fully connected layer to obtain weighted cascaded features; combine the above weighted cascaded features with partial The bias cascade feature is obtained by adding the offsets; and the above bias cascade feature is processed using the Sigmoid function to obtain the above-mentioned channel attention mask.

In one embodiment, each convolution module of the at least one convolution module further includes a second convolution layer before the at least one first convolution layer, and the second convolution layer includes ×1 three-dimensional filter, in traversing the at least one convolution module, the processing unit 604 is further configured to: use the one-dimensional filter to process the input to obtain motion-related features, and use the two-dimensional filter to process the above-mentioned Before inputting to obtain appearance-related features, the above-mentioned input is processed by using the above-mentioned three-dimensional filter to reduce the dimension of the above-mentioned input; wherein, the above-mentioned input is processed by using a one-dimensional filter to obtain motion-related features, and the above-mentioned input is processed by using the above-mentioned two-dimensional filter In terms of obtaining appearance-related features, the processing unit 604 is specifically configured to: use the above-mentioned one-dimensional filter to process the above-mentioned input after dimensionality reduction to obtain the above-mentioned motion-related features, and use the above-mentioned two-dimensional filter to process the above-mentioned input after dimensionality reduction to obtain The aforementioned appearance-related features.

In one embodiment, each convolution module in the at least one convolution module further includes a third convolution layer located after the at least one first convolution layer and the second convolution layer, and the third convolution The layer includes a three-dimensional filter with a size of 1×1×1. In terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned processing unit 604 is also used to: use the above-mentioned three-dimensional filter to process the above-mentioned concatenated features to increase the performance of the above-mentioned concatenated features. Dimensions; and the above-mentioned cascaded features after dimension-up are used as the output of the above-mentioned convolution module.

In one embodiment, each convolution module in the at least one convolution module further includes a third convolution layer located after the at least one first convolution layer and the second convolution layer, and the third convolution The layer includes a three-dimensional filter with a size of 1×1×1. In terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned processing unit 604 is also used to: process the above-mentioned attention feature with the above-mentioned three-dimensional filter to increase the performance of the above-mentioned attention feature Dimensions; and using the above-mentioned attention features after dimensionality enhancement as the output of the above-mentioned convolution module.

Fig. 7 is a schematic structural diagram of a terminal device according to an embodiment. As shown in FIG. 7 , a terminal device 700 includes a processor 701 , a memory 702 , a communication interface 703 , and one or more programs 704 stored in the memory 702 and executed by the processor 701 . The one or more programs 704 described above include instructions for implementing a deep residual network (ResNet) system. The above-mentioned deep residual network system includes at least one convolution module, each convolution module in the at least one convolution module includes at least one first convolution layer, and each first volume in the at least one first convolution layer The stack has at least one one-dimensional (1D) filter and at least one two-dimensional (2D) filter. The one or more programs 704 described above include instructions for performing the following operations.

A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module.

In the embodiment of this application, a new deep residual network system is provided. The above-mentioned deep residual network system includes at least one convolution module, each convolution module in the above-mentioned at least one convolution module includes at least one first convolution layer, and each first convolution module in the above-mentioned at least one first convolution layer The product layer has at least one one-dimensional filter and at least one two-dimensional filter, and the above-mentioned action recognition method based on the deep residual network is applied to the above-mentioned deep residual network system. A first convolution module in the at least one convolution module receives a video segment as input. Traversing the at least one convolution module above by performing the following operations: processing the above-mentioned input with a one-dimensional filter to obtain motion-related features, and processing the above-mentioned input with a two-dimensional filter to obtain appearance-related features; shifting the above-mentioned motion-related features along the time dimension One step, subtract the above-mentioned shifted motion-related features from the above-mentioned motion-related features to obtain the residual feature; based on the above-mentioned appearance-related features, the above-mentioned motion-related features and the above-mentioned residual features, obtain the output of the above-mentioned convolution module ; Using the output of the above convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed. At least one action included in the video segment is identified based on an output of a last convolutional module of the at least one convolutional module. Therefore, in this application, a new convolution module that can be regarded as a pseudo-3D convolution module is proposed, in which the standard 3D filters in the related art are decoupled to form parallel 2D spatial filters and a one-dimensional time-domain filter. By using separable 2D and 1D convolutions instead of 3D convolutions, the model size and computational cost are greatly reduced. Furthermore, 2D convolutions and 1D convolutions are placed in different paths, allowing for different modeling of appearance-related features and motion-related features.

In one embodiment, in terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned one or more programs 704 include instructions for performing the following operations. Obtaining the concatenated features by concatenating the motion-related features, the residual features and the appearance-related features in the channel dimension; and determining the concatenated features as the output of the convolution module.

In one embodiment, in terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned one or more programs 704 include instructions for performing the following operations. By cascading the above-mentioned motion-related features, the above-mentioned residual features and the above-mentioned appearance-related features in the channel dimension, the cascaded features are obtained; based on the above-mentioned cascaded features, the channel attention mask is obtained; and based on the above-mentioned channel attention mask and the above-mentioned levels Concatenated features, the attention features are obtained as the above output of the above convolution module.

In one embodiment, in terms of obtaining the above attention feature based on the above channel attention mask and the above cascade feature, the one or more programs 704 include instructions for performing the following operations. The aforementioned attention features are obtained by performing a channel-by-channel multiplication between the aforementioned channel attention mask and the aforementioned cascaded features.

In one embodiment, in terms of obtaining attention features based on the above-mentioned channel attention mask and the above-mentioned cascaded features, the above-mentioned one or more programs 704 include instructions for performing the following operations. The intermediate features are obtained by performing channel-by-channel multiplication between the above-mentioned channel attention mask and the above-mentioned concatenated features; and the above-mentioned attention features are obtained by adding the above-mentioned intermediate features and the above-mentioned concatenated features.

In one embodiment, each convolution module in the above at least one convolution module further includes a fully connected layer, and in terms of obtaining the above channel attention mask based on the above cascaded features, the above one or more programs 704 include using Instructions for performing the following operations. Perform global pooling for the above cascaded features to obtain pooled cascaded features; multiply the above pooled cascaded features by the weight matrix of the above fully connected layer to obtain weighted cascaded features; combine the above weighted cascaded features Adding the bias to obtain the bias cascade feature; and using the Sigmoid function to process the above bias cascade feature to obtain the above channel attention mask.

In one embodiment, each convolution module in the at least one convolution module is located in the second convolution layer before the at least one first convolution layer, and the second convolution layer includes a size of 1×1×1 In terms of traversing the at least one convolution module, the one or more programs 704 further include instructions for performing the following operations. Before using the above-mentioned one-dimensional filter to process the above-mentioned input to obtain motion-related features, and using the above-mentioned two-dimensional filter to process the above-mentioned input to obtain appearance-related features: use the above-mentioned three-dimensional filter to process the above-mentioned input to reduce the dimension of the above-mentioned input; wherein, in In terms of processing the input using a one-dimensional filter to obtain motion-related features, and using the two-dimensional filter to obtain appearance-related features, the one or more programs 704 include instructions for performing the following operations. Using the above-mentioned one-dimensional filter to process the above-mentioned input after dimension reduction to obtain the above-mentioned motion-related features, and using the above-mentioned two-dimensional filter to process the above-mentioned input after dimension-reduction to obtain the above-mentioned appearance-related features.

In one embodiment, each convolution module in the at least one convolution module further includes a third convolution layer located after the at least one first convolution layer and the second convolution layer, and the third convolution A layer includes a three-dimensional filter with a size of 1×1×1, and in terms of obtaining the above-mentioned output of the above-mentioned convolution module, the above-mentioned one or more programs 704 further include instructions for performing the following operations. Processing the above-mentioned concatenated features by using the above-mentioned three-dimensional filter to increase the dimension of the above-mentioned concatenated features; and using the above-mentioned concatenated features after dimensionality enhancement as the output of the above-mentioned convolution module.

In one embodiment, each convolution module in the at least one convolution module further includes a third convolution layer located after the at least one first convolution layer and the second convolution layer, and the third convolution The layer comprises a three-dimensional filter of size 1×1×1, and in obtaining the above output of the above convolution module, the one or more programs 704 include instructions for performing the following operations. processing the attention feature with the above-mentioned three-dimensional filter to increase the dimension of the attention feature; and using the dimension-uplifted attention feature as an output of the convolution module.

A non-transitory computer storage medium is also provided. The non-transitory computer storage medium is configured to store a program. When the program is executed, the program can be used to perform some or all operations of the deep residual network-based action recognition method described in the above method embodiments.

A computer program product is also provided. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The above computer program can cause the computer to execute some or all of the operations of the deep residual network-based action recognition method described in the above method embodiments.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. Because of the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the above units is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical or other forms.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the above-mentioned integrated units are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable memory. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory. Several instructions are included to make a computer device (which may be a personal computer, server or network device, etc.) execute all or part of the steps of the above-mentioned methods in various embodiments of the present invention. The above-mentioned memory includes: various media capable of storing program codes such as U disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), mobile hard disk, magnetic disk or optical disk.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash disk , a read-only memory (Read-Only Memory, ROM), a random access device (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The embodiments of the present application have been described in detail above, and specific examples are used here to describe the principle and implementation of the present application. The descriptions of the above embodiments are only used to help understand the method and core idea of the present application. At the same time, those skilled in the art can modify the specific implementation and application scope according to the idea of the present application. In any case, the content of this specification should not be construed as limiting the application.

Claims (20)

1. A method for motion recognition based on a depth residual error network, applied to a depth residual error network system including at least one convolution module, each convolution module of the at least one convolution module including at least one first convolution layer, each first convolution layer of the at least one first convolution layer having at least one-dimensional filter and at least one two-dimensional filter, the method comprising: receiving as input a video segment at a first of the at least one convolution modules;

traversing the at least one convolution module by performing the following operations:

processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features;

shifting the motion-related feature by one step along a time dimension, and subtracting the shifted motion-related feature from the motion-related feature to obtain a residual feature;

obtaining an output of the convolution module based on the appearance-related feature, the motion-related feature, and the residual feature;

taking the output of the convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed;

identifying at least one action included in the video segment based on an output of a last convolution module of the at least one convolution module.

2. The method of claim 1, wherein obtaining the output of the convolution module comprises:

obtaining a cascade feature by cascading the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension; and

determining the concatenated features as the output of the convolution module.

3. The method of claim 1, wherein obtaining the output of the convolution module comprises:

obtaining a cascade feature by cascading the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension;

acquiring a channel attention mask based on the cascade feature; and

based on the channel attention mask and the cascade feature, acquiring an attention feature as the output of the convolution module.

4. The method of claim 3, wherein obtaining the attention feature based on the channel attention mask and the cascade feature comprises:

the attention feature is obtained by performing a channel-by-channel multiplication between the channel attention mask and the cascade feature.

5. The method of claim 3, wherein obtaining the attention feature based on the channel attention mask and the cascade feature comprises:

obtaining an intermediate feature by performing a channel-by-channel multiplication between the channel attention mask and the cascade feature; and

and adding the intermediate feature and the cascade feature to obtain the attention feature.

6. The method of any of claims 3 to 5, wherein each of the at least one convolution module further comprises a fully-connected layer, and wherein obtaining the channel attention mask based on the concatenated features comprises:

performing global pooling on the cascade features to obtain pooled cascade features;

multiplying the pooled cascade features by the weight matrix of the full connection layer to obtain weighted cascade features;

adding the weighted cascade feature and the bias to obtain a biased cascade feature; and

and processing the bias cascade feature by using a Sigmoid function to obtain the channel attention mask.

7. The method of any of claims 1 to 6, wherein each of the at least one convolution module further comprises a second convolution layer located before the at least one first convolution layer, the second convolution layer comprising a three-dimensional filter having a size of 1 x 1, traversing the at least one convolution module further comprising:

prior to processing the input using the one-dimensional filter to obtain motion-related features and processing the input using the two-dimensional filter to obtain appearance-related features:

processing the input using the three-dimensional filter to reduce a dimension of the input;

wherein processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features comprises:

processing the reduced dimension input using the one-dimensional filter to obtain the motion-related feature, and processing the reduced dimension input using the two-dimensional filter to obtain the appearance-related feature.

8. The method of claim 7, wherein each of the at least one convolution module further includes a third convolution layer positioned after the at least one first convolution layer and the second convolution layer, the third convolution layer including a three-dimensional filter having a size of 1 x 1, the obtaining the output of the convolution module further comprising:

processing the concatenated features using the three-dimensional filter to increase a dimension of the concatenated features; and

and taking the cascade feature after the dimension is raised as the output of the convolution module.

9. The method of claim 7, wherein each of the at least one convolution module further includes a third convolution layer positioned after the at least one first convolution layer and the second convolution layer, the third convolution layer including a three-dimensional filter having a size of 1 x 1, the obtaining the output of the convolution module further comprising:

processing the attention feature with the three-dimensional filter to increase a dimension of the attention feature; and

using the attention feature after upscaling as the output of the convolution module.

10. An action recognition device based on a depth residual error network, applied to a depth residual error network system comprising at least one convolution module, each convolution module of the at least one convolution module comprising at least one first convolution layer, each first convolution layer of the at least one first convolution layer having at least one-dimensional filter and at least one two-dimensional filter, the device comprising:

a receiving unit for receiving as input a video segment at a first of said at least one convolution module;

a processing unit to traverse the at least one convolution module by performing the following operations:

processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features;

shifting the motion-related feature by one step along a time dimension, and subtracting the shifted motion-related feature from the motion-related feature to obtain a residual feature;

obtaining an output of the convolution module based on the appearance-related feature, the motion-related feature, and the residual feature;

taking the output of the convolution as the input of the next convolution module until the last convolution module in the at least one convolution module is traversed;

an identification unit to identify at least one action included in the video segment based on an output of a last convolution module of the at least one convolution module.

11. The apparatus according to claim 10, wherein, in obtaining the output of the convolution module, the processing unit is specifically configured to:

obtaining a cascade feature by cascading the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension; and

determining the concatenated features as the output of the convolution module.

12. The apparatus according to claim 10, wherein, in obtaining the output of the convolution module, the processing unit is specifically configured to:

obtaining a cascade feature by cascading the motion-related feature, the residual feature, and the appearance-related feature in a channel dimension;

acquiring a channel attention mask based on the cascade feature; and

based on the channel attention mask and the concatenated features, acquiring attention features as the output of the convolution module.

13. The apparatus of claim 12, wherein, in obtaining the attention feature based on the channel attention mask and the cascade feature, the processing unit is specifically configured to:

the attention feature is obtained by performing a channel-by-channel multiplication between the channel attention mask and the cascade feature.

14. The apparatus of claim 12, wherein, in obtaining an attention feature based on the channel attention mask and the cascade feature, the processing unit is specifically configured to:

obtaining an intermediate feature by performing a channel-by-channel multiplication between the channel attention mask and the cascade feature; and

and adding the intermediate feature and the cascade feature to obtain the attention feature.

15. The apparatus according to any one of claims 12 to 14, wherein each of the at least one convolution module further comprises a fully-connected layer, the processing unit being specifically configured to, in obtaining the channel attention mask based on the concatenated features:

performing global pooling on the cascade feature to obtain the pooled cascade feature;

multiplying the pooled cascade features by the weight matrix of the full connection layer to obtain weighted cascade features;

adding the weighted cascade feature and the bias to obtain a biased cascade feature; and

and processing the bias cascade feature by using a Sigmoid function to obtain the channel attention mask.

16. The apparatus of any of claims 10 to 15, wherein each of the at least one convolution module further comprises a second convolution layer preceding the at least one first convolution layer, the second convolution layer comprising a three-dimensional filter having a size of 1 x 1, the processing unit further to, in traversing the at least one convolution module: processing the input using the three-dimensional filter to reduce the dimensionality of the input prior to processing the input using the one-dimensional filter to obtain motion-related features and processing the input using the two-dimensional filter to obtain appearance-related features;

wherein, in processing the input using a one-dimensional filter to obtain motion-related features and processing the input using a two-dimensional filter to obtain appearance-related features, the processing unit is to: processing the reduced dimension input using the one-dimensional filter to obtain the motion-related feature, and processing the reduced dimension input using the two-dimensional filter to obtain the appearance-related feature.

17. The apparatus of claim 16, wherein each of the at least one convolution module further comprises a third convolution layer positioned after the at least one first convolution layer and the second convolution layer, the third convolution layer comprising a three-dimensional filter having a size of 1 x 1, the processing unit further to, in obtaining the output of the convolution module:

processing the cascaded feature using the three-dimensional filter to increase a dimension of the cascaded feature; and

and taking the cascade feature after the dimension is raised as the output of the convolution module.

18. The apparatus of claim 16, wherein each of the at least one convolution module further comprises a third convolution layer positioned after the at least one first convolution layer and the second convolution layer, the third convolution layer comprising a three-dimensional filter having a size of 1 x 1, the processing unit further to, in obtaining the output of the convolution module:

processing the attention feature with the three-dimensional filter to increase a dimension of the attention feature; and

using the attention feature after upscaling as the output of the convolution module.

19. A terminal device comprising a processor, a memory for storing one or more programs, wherein the one or more programs are for execution by the processor and comprise instructions for performing the method of any of claims 1 to 9.

20. A non-transitory computer readable storage medium storing a computer program for electronic data exchange, which when executed, causes a computer to perform the method according to any one of claims 1 to 9.

Images