CN115661596A - Short video positive energy evaluation method, device and equipment based on 3D convolution and Transformer - Google Patents

Abstract

The invention discloses a short video positive energy evaluation method, device and equipment based on 3D convolution and Transformer, and relates to the technical field of video violent behavior analysis. Energy evaluation", the method includes: obtaining a video clip, the frame number of the video clip is a preset frame number; performing feature extraction on the video clip based on a pre-trained 3D convolution model to obtain a plurality of feature vectors; The feature vector is position-encoded; the position-encoded multiple feature vectors are input to the pre-trained Transformer model to obtain an output vector; the output vector is input to the multi-layer perceptron model to calculate the positive energy of the video segment Score; this method is based on the 3D convolution model and the Transformer model to evaluate the positive energy of the short video, which has a good timing modeling effect, and can process videos that contain a large number of video frames for a long time. The present invention is also applied to the field of computer vision.

Description

Short video positive energy evaluation method, device and method based on 3D convolution and Transformer equipment

technical field

The invention relates to the technical field of video violent behavior analysis.

Background technique

In recent years, user-generated content video has exploded on platforms such as Kuaishou, Douyin, and Weishi, each supporting tens of millions and billions of users. Today's video shooting equipment is becoming cheaper and cheaper as the shooting level is getting higher and higher, which greatly reduces the cost of short video production. Therefore, a large number of short video users are not only short video consumers but also short video users. The creator of the video has led to a rapid increase in the number of short videos on the Internet, and short videos have quickly become the most important source of human information.

Due to the different purposes of different users creating videos, there are always some vulgar, negative or unhealthy short videos, resulting in the information disseminated by the short video content does not conform to the mainstream values of society

There are relatively few existing data sets for violent behavior analysis, and many studies are carried out on a single data set. In terms of algorithms, existing technologies often use 2D convolution technology + RNN (GRU, LSTM), single 3D convolution technology and 3D convolution technology + RNN (GRU, LSTM). A large number of experiments have shown that the above methods are not good enough in video processing. RNN technology has been proven to be much worse than Transformer model in timing modeling, and only using 3D convolution technology is too limited for video frame processing. Work with videos that contain a large number of video frames over a long period of time.

Contents of the invention

In order to solve the technical problems existing in the prior art, the present invention provides a short video positive energy evaluation method, device and equipment based on 3D convolution and Transformer, and perform positive energy evaluation on short videos based on the 3D convolution model and Transformer model , has a better timing modeling effect, and can handle videos that contain a large number of video frames for a long time.

A short video positive energy evaluation method based on 3D convolution and Transformer, including:

Obtain a video clip, the frame number of the video clip is a preset frame number;

Carry out feature extraction to described video segment based on pre-trained 3D convolution model, obtain a plurality of feature vectors;

performing position encoding on the feature vector;

Inputting a plurality of said feature vectors through position encoding into a pre-trained Transformer model to obtain an output vector;

The output vector is input to the multi-layer perceptron model, and the positive energy score of the video segment is calculated.

Further, the 3D convolution model is an R3D model, and the preset number of frames is a multiple of the number of frames that the R3D model can input each time.

Further, the 3D convolutional model includes a plurality of fully connected layers, and the last fully connected layer in the 3D convolutional model is recorded as the first fully connected layer;

Based on the pre-trained 3D convolution model, feature extraction is performed on each video clip to obtain multiple feature vectors, including:

Carry out center cropping to described video segment;

Segmenting the center-cropped video segment according to frame order to obtain a plurality of input frame groups;

Inputting a plurality of the input frame groups into the 3D convolution model in sequence, and recording the obtained input vector of the first fully connected layer as an intermediate feature vector;

A plurality of the intermediate feature vectors are input to the second fully connected layer to increase the dimension to obtain a plurality of feature vectors.

Further, the 3D convolution model is trained based on the data set Kinetics, and is trained based on the following parameters:

The 3D convolutional model is an R3D model with a depth of 18 layers, the number of iterations is 1M, and the initial learning rate is 1e-2. The entire training process is divided into 45 stages. The first 10 stages are warmed up, and the next ten stages The learning rate is reduced to one-tenth of the original.

Further, the Transformer model includes at least one Encoder Block, and the EncoderBlock includes a multi-head attention structure and a multi-layer perceptron structure;

Standardization is performed before the feature vector is input into each of the multi-head attention structures, and a residual connection is used after each of the multi-head attention structures;

Standardization is performed before the feature vector is input into each multilayer perceptron structure, and a residual connection is used after each multilayer perceptron structure;

The multilayer perceptron structure includes a third fully connected layer and a fourth fully connected layer, the third fully connected layer is used to expand the dimension of the feature vector to four times the original, and the fourth fully connected layer Used to restore the dimensions of the feature vectors to their original size.

Further, the Transformer model is trained based on the following parameters:

The loss function adopts MSE, the optimizer adopts AdamW optimizer, the number of layers of the Encoder Block is 24, and the heads of the multi-head attention structure are 16;

The Warm up method is used to warm up the learning rate, and Linear Warm up is selected as the specific Warmup strategy. The initial learning rate is set to 1e-5, the number of warm-up steps is set to 15, and the total number of training steps is 60.

Further, input a plurality of position-encoded feature vectors into the pre-trained Transformer model to obtain an output vector, including:

Input the position-encoded multiple feature vectors and classification head vector cls-token into the Transformer model;

The output corresponding to the classification header vector cls-token in the Transformer model is input to the final fully connected layer as a classification feature, and the 1024-dimensional feature is transformed into a one-dimensional vector to obtain an output vector.

Further, the preset number of frames is 96, the number of frames that can be input each time of the R3D model is 16, and the number of feature vectors is 6.

A short video positive energy evaluation device based on 3D convolution and Transformer, including:

A video acquisition module, configured to acquire video clips, the number of frames of the video clips is a preset number of frames;

Feature extraction module, for carrying out feature extraction to video segment based on pre-trained 3D convolution model, obtains a plurality of feature vectors;

A position encoding module, configured to perform position encoding on the feature vector;

An output calculation module, configured to input a plurality of position-encoded feature vectors into a pre-trained Transformer model to obtain an output vector;

The score calculation module is used to input the output vector to the multi-layer perceptron model to calculate the positive energy score of the video segment.

An electronic device includes a processor and a storage device, wherein a plurality of instructions are stored in the storage device, and the processor is used to read the plurality of instructions in the storage device and execute the above method.

The short video positive energy evaluation method, device and equipment based on 3D convolution and Transformer provided by the present invention at least include the following beneficial effects:

(1) Based on the 3D convolution model and the Transformer model, the positive energy evaluation of the short video is performed, and the 3D convolution model is used to extract the features of the video, and then the Transformer model is used to fuse the temporal features of the video, which has a good temporal modeling effect, And be able to handle videos that contain a large number of video frames for a long time;

(2) Use specific parameters to train the 3D convolution model and Transformer model, which can effectively improve the training efficiency while improving the training effect;

(3) When extracting features from video clips based on the 3D convolution model, the dimension-up operation is adopted to increase the complexity of the model and make the model have better fitting ability;

(4) The position encoding is performed on the feature vector extracted by the 3D convolution model, which effectively preserves the timing information between frames from being lost, and achieves a better timing feature fusion effect.

Description of drawings

Fig. 1 is the flowchart of an embodiment of the short video positive energy evaluation method based on 3D convolution and Transformer provided by the present invention;

Fig. 2 is a schematic structural diagram of an embodiment of the Encoder Block in the Transformer model provided by the present invention.

Detailed ways

In order to better understand the above-mentioned technical solution, the above-mentioned technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

It should be noted that the concept of "positive energy" in this embodiment is used to describe the emotional tendency of the short video, specifically referring to the content of the short video is not related to violent behavior, and does not involve vulgar, negative or unhealthy content.

Referring to Fig. 1, in some embodiments, a short video positive energy evaluation method based on 3D convolution and Transformer is characterized in that it includes:

S1. Obtain a video clip, the frame number of the video clip is a preset frame number;

S2. Perform feature extraction on the video clip based on the pre-trained 3D convolution model to obtain multiple feature vectors;

S3. Perform position encoding on the feature vector;

S4. Input a plurality of position-encoded feature vectors into a pre-trained Transformer model to obtain an output vector;

S5. Input the output vector into the multi-layer perceptron model, and calculate the positive energy score of the video segment.

In Figure 1, R3D is a 3D version of the ResNet model, FC is a fully connected layer, Position Embedding represents position encoding, and MLP Head represents a multi-layer perceptron.

As a preferred implementation manner, the 3D convolution model is an R3D model, and the preset number of frames is a multiple of the number of frames that the R3D model can input each time. The R3D model keeps the overall structure of the original 2D ResNet unchanged, expands the original 3×3 2D convolution to a 3×3×3 3D convolution, and replaces the pooling layer with 3D pooling. The input dimension of the R3D model is 3×16×112×112. Since the model parameters of the R3D model are relatively small, when the R3D model is used for 3D convolution, not only can better results be obtained, but also the speed of the model will not be slowed down.

The method provided in this embodiment uses a data set containing multiple negative energy behaviors for training, including fighting, blood, gunshots, explosions, smoking, etc. Compared with the existing single data set, the obtained model has more Good positive energy scoring effect, more comprehensive recognition.

Specifically, in step S2, feature extraction is performed on the video clip based on the pre-trained 3D convolution model to obtain multiple feature vectors, including:

S21. Perform center cropping on the video segment;

S22. Segment the center-cropped video segment according to frame order to obtain a plurality of input frame groups;

S23. Input a plurality of input frame groups into the 3D convolution model in sequence, and record the obtained input vector of the first fully connected layer as an intermediate feature vector;

S24. Input the plurality of intermediate feature vectors to the second fully connected layer to increase the dimension to obtain a plurality of feature vectors.

In a specific application scenario, the R3D model is used to extract features from video clips. The R3D model can input 16 frames each time. When obtaining video clips, a random sampling strategy is used for the complete video frames, and 96 frames of video clips are randomly selected from the entire video each time. 96 frames are the multiples of the number of frames that can be input by the R3D model each time, and then these 96 frames of video The segment is center-cut, and 96 frames of 112*112 video frames are obtained as the input of the network. Since the R3D model can only input 16 frames at a time, the video clips are divided according to the frame order to obtain 6 input frame groups, and each input frame group contains 16 frames. After the R3D model, 6 512-dimensional intermediate features are finally obtained. Vector, and then through the second fully connected layer to increase the dimension of the feature from 512 dimensions to 1024 dimensions.

Among them, the input of the last fully connected layer in the R3D model is used as the extracted feature, that is, the intermediate feature vector, and then the dimension is increased to obtain the feature vector for input to the transformer model in the subsequent steps. In step S24, the dimensionality of the obtained intermediate feature vector is increased based on the second fully connected layer, which increases the complexity of the model and makes the model have better fitting ability.

As a preferred embodiment, the 3D convolution model is trained based on the data set Kinetics, and is trained based on the following parameters: the 3D convolution model is an R3D model with a depth of 18 layers, the number of iterations is 1M, and the initial learning rate For 1e-2, the entire training process is divided into 45 stages, the first 10 stages are warmed up, and the learning rate is reduced to one-tenth of the original every ten stages.

Referring to FIG. 2 , in some embodiments, the Transformer model used includes at least one EncoderBlock, and the Encoder Block includes a multi-head attention structure and a multi-layer perceptron structure. Before the feature vector is input into each of the multi-head attention structures (Multi-Head Attention), a standardized Norm operation is performed, and after each of the multi-head attention structures, a residual connection is used; the feature vector is input into each of the multi-head attention structures. The multi-layer perceptron structure (MLP) is preceded by a standardized Norm operation, and each multi-layer perceptron structure is followed by a residual connection. The multilayer perceptron structure includes a third fully connected layer and a fourth fully connected layer, the third fully connected layer is used to expand the dimension of the feature vector to four times the original, and the fourth fully connected layer Used to restore the dimensions of the feature vectors to their original size. In Figure 2, Norm represents the normalization layer, MLP represents the multi-layer perceptron structure, and Multi-Head Attention represents the multi-head attention structure.

In some embodiments, the Transformer model is obtained by stacking multiple Encoder Blocks multiple times. For an Encoder Block, it mainly consists of a Multi-Head Attention (MSA) and an MLP structure, and also includes two operations of residual connection and layer normalization. It should be noted that layer normalization processes all features of each sample, while batch normalization processes all samples of each channel. Before inputting the data into the two structures of MSA and MLP, the layer normalization operation is used to normalize the data, and the residual connection is applied after the structure of each MSA and MLP. The MLP structure follows the MSA. The MLP structure contains two fully connected layers. The first fully connected layer expands the dimension of the feature to four times the original size, and the second fully connected layer restores the dimension of the feature to its original size. All activation functions use GELU (Gaussian Error Linear Unit). After inputting the position-encoded feature vector into the Transformer model, it will continue to propagate forward through multiple Encoder Blocks. After each Encoder Block, the sequence dimension remains unchanged, and finally the feature vector corresponding to cls-token as output.

In step S3, position coding is performed on the obtained feature vectors to effectively preserve the timing information between frames without being lost.

In a specific application scenario, six 1024-dimensional feature vectors are position-encoded to preserve the timing information between video blocks. Here, a learnable position-encoding vector is used, which is equivalent to a chapter table. There are N rows in the table, and the value of N is the same as the length of the input sequence. Each row represents a video frame, and the dimension of the sequence remains unchanged after adding position encoding.

In step S4, input the position-encoded multiple feature vectors into the pre-trained Transformer model to obtain an output vector, including:

S41. Input the plurality of position-encoded feature vectors and classification head vector cls-token into the Transformer model;

S42. Use the vector corresponding to the classification head vector cls-token in the Transformer model as an output vector.

As a better implementation, the Transformer model is trained based on the following parameters: the loss function uses MSE, the optimizer uses the AdamW optimizer, the number of layers of the Encoder Block is 24, and the heads of the multi-head attention structure are 16. The output corresponding to the classification header vector cls-token is input as a feature to the final fully connected layer, and the 1024-dimensional feature is transformed into a one-dimensional score. Among them, the AdamW optimizer is a variant of the Adam optimizer, which introduces weight decay and L2 regularization.

As a better implementation, the Warm up method is used to warm up the learning rate, and LinearWarm up is selected as the specific Warm up strategy. The initial learning rate is set to 1e-5, and the number of warm-up steps is set to 15 steps. The total number of training steps is 60, that is, the learning rate increases uniformly from 6.67e-7 to the initial learning rate 1e-5 in the first 15 epochs, and then decreases uniformly to 2.22e-7. The principle of the warm-up operation is to let the learning rate increase slowly from a small value, and then use a larger learning rate for training after the model changes little. In this way, the convergence speed of the model can be effectively accelerated, thereby improving the efficiency of model training.

In some embodiments, when the length of the video is longer, steps S1-S5 are repeated to extract and analyze multiple segments.

In some embodiments, a short video positive energy evaluation device based on 3D convolution and Transformer is provided, including:

A video acquisition module, configured to acquire video clips, the number of frames of the video clips is a preset number of frames;

Feature extraction module, for carrying out feature extraction to video segment based on pre-trained 3D convolution model, obtains a plurality of feature vectors;

A position encoding module, configured to perform position encoding on the feature vector;

An output calculation module, configured to input a plurality of position-encoded feature vectors into a pre-trained Transformer model to obtain an output vector;

The score calculation module is used to input a plurality of output vectors into the multi-layer perceptron model, and calculate the positive energy score of the video segment.

In some embodiments, an electronic device is provided, including a processor and a storage device, wherein a plurality of instructions are stored in the storage device, and the processor is configured to read the plurality of instructions in the storage device and execute the above method .

In a specific application scenario, the device for executing the above method is NVIDIA GTX 3090GPU, and the machine learning framework uses pytorch1.12.

In the existing violent behavior video analysis methods, most of them use the combination of 2D convolution technology and RNN model (GRU, LSTM), a single 3D convolution technology, or a combination of 3D convolution technology and RNN model. When using the above method to score short video positive energy, RNN technology has been proven to be much worse than the Transformer model in terms of time series modeling, and only using 3D convolution technology is too limited for the processing of video frames, and cannot handle long-term content containing a large number of video frames of video.

When combining the 3D convolution model and the Transformer model, there is a problem that the parameters of the Transformer model are relatively large and need to be solved. The short video positive energy evaluation method based on 3D convolution and Transformer provided in this embodiment uses the technical means of time-series modeling of the features of the video clip extracted by R3D, and realizes the application of the 3D convolution model and the Transformer model to the short video In the positive energy evaluation, it can not only have a better timing modeling effect, but also be able to process videos that contain a large number of video frames for a long time. In order to highlight the advantages of this technology, compared with three models, namely C3D model + GRU, C3D model + Transformer, R3D model + GRU, it is found that the effect of this technology is significantly better than the other three models, which proves that the R3D feature extractor The effectiveness of combining with the Transformer model.

While preferred embodiments of the invention have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is appreciated. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims (10)

1. A short video positive energy evaluation method based on 3D convolution and a Transformer is characterized by comprising the following steps:

acquiring a video clip, wherein the frame number of the video clip is a preset frame number;

performing feature extraction on the video clip based on a pre-trained 3D convolution model to obtain a plurality of feature vectors;

performing position coding on the feature vector;

inputting a plurality of feature vectors subjected to position coding into a pre-trained transform model to obtain an output vector;

and inputting the output vector to a multilayer perceptron model, and calculating to obtain a positive energy fraction of the video clip.

2. The method of claim 1, wherein the 3D convolution model is an R3D model, and the preset frame number is a multiple of a number of frames that the R3D model can input at a time.

3. The method of claim 2, wherein the 3D convolution model includes a plurality of fully connected layers, and the last fully connected layer in the 3D convolution model is denoted as a first fully connected layer;

carrying out feature extraction on each video segment based on a pre-trained 3D convolution model to obtain a plurality of feature vectors, wherein the feature vectors comprise:

performing center cropping on the video clip;

dividing the video clips subjected to center cutting according to a frame sequence to obtain a plurality of input frame groups;

inputting the plurality of input frame groups into the 3D convolution model in sequence, and recording the obtained input vector of the first full-connection layer as a middle characteristic vector;

and inputting the plurality of intermediate feature vectors into a second full-connection layer for dimensionality increase to obtain a plurality of feature vectors.

4. The method of claim 2, wherein the 3D convolution model is trained based on a dataset Kinetics and based on the following parameters:

the 3D convolution model is an R3D model with 18 layers of depth, the iteration times are 1M times, the initial learning rate is 1e-2, the whole training process is divided into 45 stages, the first 10 stages are warmed up, and the learning rate of every ten stages is reduced to one tenth of the original learning rate.

5. The method of claim 4, wherein the transform model comprises at least one Encoder Block, wherein the Encoder Block comprises a multi-head attention structure and a multi-layer perceptron structure;

before the feature vector is input into each multi-head attention structure, standardization operation is carried out, and residual connection is adopted after each multi-head attention structure;

before the feature vectors are input into each multi-layer perceptron structure, standardization operation is carried out, and residual connection is adopted after each multi-layer perceptron structure;

the multilayer perceptron structure comprises a third full connection layer and a fourth full connection layer, wherein the third full connection layer is used for expanding the dimensionality of the characteristic vector to four times of the original dimensionality, and the fourth full connection layer is used for restoring the dimensionality of the characteristic vector to the original size.

6. The method of claim 5, wherein the Transformer model is trained based on the following parameters:

the loss function adopts MSE, the optimizer adopts an AdamW optimizer, the number of layers of the Encoder Block is 24, and the number of heads of a multi-head attention structure is 16;

preheating the learning rate by adopting a Warm up method, selecting Linear Warm up as a specific Warm up strategy, setting the initial learning rate to be 1e-5, setting the preheating step number to be 15 steps, and setting the total training number to be 60 steps.

7. The method of claim 5, wherein inputting a plurality of the position-coded feature vectors into a pre-trained transform model to obtain an output vector comprises:

inputting a plurality of feature vectors subjected to position coding and a classification head vector cls-token into a Transformer model;

and inputting output corresponding to the classification head vector cls-token in the Transformer model as classification features to a final full-connection layer, and converting 1024-dimensional features into one-dimensional vectors to obtain output vectors.

8. The method according to claim 2, wherein the preset frame number is 96, the number of frames that the R3D model can input at a time is 16, and the number of feature vectors is 6.

9. A short video positive energy evaluation device based on 3D convolution and a Transformer is characterized by comprising:

the video acquisition module is used for acquiring a video clip, and the frame number of the video clip is a preset frame number;

the feature extraction module is used for extracting features of the video clips based on a pre-trained 3D convolution model to obtain a plurality of feature vectors;

the position coding module is used for carrying out position coding on the feature vector;

the output calculation module is used for inputting the plurality of feature vectors subjected to position coding into a pre-trained Transformer model to obtain an output vector;

and the score calculating module is used for inputting the output vector to the multilayer perceptron model and calculating to obtain the positive energy score of the video clip.

10. An electronic device comprising a processor and a memory means, wherein a plurality of instructions are stored in the memory means, and wherein the processor is configured to read the plurality of instructions from the memory means and to perform the method according to any one of claims 1 to 8.

Images