CN117392582A - Multi-mode video classification method and system - Google Patents

Abstract

The invention discloses a multi-modal video classification method based on parallel ResNet18 of speech and visual modes and a global linkage gating mechanism, which includes the following steps: S1: Use two resnet18 encoders with consistent structures to extract speech respectively. Feature representation of image modalities and used as input features for gated cross-modal feature fusion. S2: Design a globally linked gating mechanism, which consists of a cross-modal gating fusion module and an objective function based on gating auxiliary loss. It is the core of balancing different modalities. S3: The first part is a cross-modal gating fusion module designed based on the GRU gating principle. It accepts the feature input of voice and image modalities, automatically adjusts the input proportions of different modalities, and further performs interactive fusion, and outputs the fused Cross-modal features. S4: In the second part, the single-modal loss is used as the auxiliary loss, and the gating parameters in the first part are processed to a certain extent and used as the weight of the auxiliary loss to form the adaptive gating adjustment mechanism of the model.

Description

A multi-modal video classification method and system

Technical field

The present invention relates to the technical field of multi-modal video classification, and specifically refers to a multi-modal video classification method and its system.

Background technique

The development of multi-modal learning has entered the period of multi-modal deep learning. In recent years, deep learning (Deep Learning, DL) has been widely used and achieved great results in image recognition, machine translation, sentiment analysis, natural language processing (NLP) and other fields. Many research results show that in order for machines to perceive the world around them more comprehensively and efficiently, they need to be given the ability to understand, reason and fuse multi-modal information. And because people live in an environment where multiple fields are integrated, what they hear is Sound, physical objects seen, and smells are all modalities, so researchers have begun to focus on how to fuse multi-domain data to achieve heterogeneous complementarity. For example, research on speech recognition shows that the visual modality provides the lips of the mouth. Internal motion and articulatory information, including opening and closing, help improve speech recognition performance. It can be seen that using comprehensive semantics of multiple modes is of great significance to deep learning research.

Multimodal data generally provide more information than unimodal data, so learning with multimodal data should match or outperform unimodal data. However, in some cases, multimodal models that use a joint training strategy to optimize a unified learning goal for all modalities may be inferior to single-modal models. This phenomenon is due to the fact that various modes tend to converge at different speeds, resulting in one mode reaching the fitting state while other modes have not yet been fitted, which is a modal imbalance problem.

Contents of the invention

In view of the shortcomings of the existing technology, the present invention proposes a multi-modal video classification method and its system, which not only effectively integrates multiple video types, but also effectively solves the problem of multi-modal imbalance.

In order to solve the above technical problems, the technical solution of the present invention is:

A multi-modal video classification method based on parallel ResNet18 of speech and visual modalities and a global linkage gating mechanism, including the following steps:

S1: Use two resnet18 encoders with the same structure to extract feature representations of speech and image modalities respectively, and use them as input features for gated cross-modal feature fusion.

S2: Design a globally linked gating mechanism, which consists of a cross-modal gating fusion module and an objective function based on gating auxiliary loss. It is the core of balancing different modalities.

S3: The first part is a cross-modal gating fusion module designed based on the GRU gating principle. It accepts the feature input of voice and image modalities, automatically adjusts the input proportions of different modalities, and further performs interactive fusion, and outputs the fused Cross-modal features.

S4: In the second part, the single-modal loss is used as the auxiliary loss, and the gating parameters in the first part are processed to a certain extent and used as the weight of the auxiliary loss to form the adaptive gating adjustment mechanism of the model.

Preferably, the specific steps of step S1 include:

S101: Preprocess the original video data of the relevant data set, extract the speech information in the video and convert it into a spectrogram as the input of the speech modality. At the same time, taking into account the differences in the data sets, uniformly sample 3 frames from the video as visual Modal input;

S102: The speech mode and the visual mode each use two identical ResNet18 as encoders to extract features from the input data of the speech and visual modes respectively:

in Is input from the speech (audio) mode,/> is the speech modality speech encoder based on ResNet18, θ ^a is the encoder parameter, and H ^a is the speech modality feature extracted by the encoder; similarly/> Is input from the visual modality,/> is the visual modality visual encoder based on ResNet18, θ ^v is the encoder parameter, and H ^v is the visual modality feature extracted by the encoder;

Preferably, the specific steps of step S2 include:

First, the cross-modal fusion module is used to interactively fuse the two input features. After the fused cross-modal feature ^Hav , speech modality feature ^Ha , and visual modality feature ^Hv are classified and predicted, the cross-entropy formula is used Three independent losses ^av , loss ^a , and loss ^v are calculated. Finally, the single-modal loss is used as the auxiliary loss, and the gating parameters after certain processing are used as the weight of the auxiliary loss to form an objective function based on the gating auxiliary loss.

Preferably, the specific steps of step S3 include:

λ=Sigmod(UH ^a +VH ^v )

H ^av = (1-λ)·H ^a +λ·H ^v

Where U and V are trainable variables, λ is the gating parameter controlling how much visual information can be retained, Sigmod is the activation function, ^{and Hav} is the fusion feature of different modalities.

Preferably, the specific steps of step S4 include:

After taking the average dimensionality reduction of the gating parameter λ in the first part of the cross-modal fusion module, the reciprocal form is used as the weight of the corresponding modal loss, achieving a negative correlation between the modal loss and the modal retained information. That is to say, when the loss of a single mode is relatively smaller (the mode has reached the fitting state), the model will correspondingly reduce the information retention of the mode. At the same time, the loss of the other single mode is relatively larger (the mode has not yet reached the fitting state). reaches the fitting state), the model will correspondingly improve the information retention of this mode;

λ ^- =mean(λ)

where λ ^- is the averaged one-dimensional gating parameter, β is a hyperparameter, loss ^a is the cross-entropy loss of a single speech modality, loss ^v is the cross-entropy loss of a single visual modality, and loss ^av is multi-modal fusion The final cross-entropy loss.

The invention also provides a multi-modal video classification system based on parallel ResNet18 and global linkage of voice and visual modalities and global linkage gating mechanism, including a parallel multi-modal feature extraction module, a cross-modal gating fusion module, and a gate-based gating fusion module. Control the objective function and classification prediction module of auxiliary loss;

The parallel multi-modal feature extraction module is used to extract initial features of speech and visual modalities, and encode them as input features for modal fusion;

The cross-modal gating fusion module and the objective function based on gating-assisted loss jointly construct a global gating adjustment mechanism, which will judge the fitting status of each modality based on the loss of a single modality and adjust it adaptively. The proportion of information of different modalities in cross-modal fusion solves the problem of modal imbalance in multi-modal models;

The classification prediction module performs classification prediction on single modal features and fused modal features. After the model training is completed, the classification prediction of the fused modal features is used as the final prediction result.

The invention has the following characteristics and beneficial effects:

Using the above technical solution, a general global gating adjustment structure is designed for multi-modal video classification tasks, which not only effectively fuses the feature information of multiple modalities, but also pays more attention to the imbalance problem during modal fusion.

In addition, in order to further improve the complementarity between fused modal features, the objective function based on gated auxiliary loss simply and cleverly solves the multi-modal imbalance problem and effectively improves the performance of multi-modal fused features.

The present invention is very helpful for multi-modal video classification tasks and can significantly improve the classification prediction accuracy of multi-modal video classification tasks.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.

Figure 1 is a flow chart of the multi-modal video classification method of the present invention based on the parallel ResNet18 of speech and visual modalities and the global linkage gating mechanism;

Figure 2 is a schematic model diagram of the multi-modal video classification method and its system based on the parallel ResNet18 and global linkage gating mechanism of speech and visual modalities;

Figure 3 is a schematic diagram of a single ResNet infrastructure that constitutes ResNet18;

Figure 4 is a schematic structural diagram of the designed gate-based modal fusion module;

Figure 5 is a structural block diagram of the multi-modal video classification system based on the parallel ResNet18 of speech and visual modalities and the global linkage gating mechanism.

Detailed ways

It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", " The orientations or positional relationships indicated by "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and The simplified description is not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms “first”, “second”, etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, features defined by "first," "second," etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

In the description of the present invention, it should be noted that, unless otherwise clearly stated and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense. For example, it can be a fixed connection or a detachable connection. Connection, or integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be an internal connection between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

The present invention provides a multi-modal video classification method. As shown in Figures 1-4, the parallel ResNet18 of voice and visual modalities and the global linkage gating mechanism, referred to as RGGL, include the following steps:

S1: Use two resnet18 encoders with the same structure to extract feature representations of speech and image modalities respectively, and use them as input features for gated cross-modal feature fusion;

Preliminary processing and feature extraction of dataset data includes the following steps:

S101: Preprocess the original video data of the relevant data set, extract the speech information in the video and use librosa to convert it into the corresponding spectrogram as the input of the speech modality. At the same time, taking into account the differences in the data set, uniformly sample from the video 3 frames serve as input to the visual modality;

S102: The speech mode and the visual mode each use two identical ResNet18 as encoders (the encoder for the speech mode changes the input channel of ResNet18 from 3 to 1, and the rest remains unchanged.) From speech and visual modes respectively Extract features from input data of visual modality:

in Is input from the speech (audio) mode,/> is the speech modality speech encoder based on ResNet18, θ ^a is the encoder parameter, and H ^a is the speech modality feature extracted by the encoder; similarly/> Is input from the visual modality,/> is the visual modality visual encoder based on ResNet18, θ ^v is the encoder parameter, and H ^v is the visual modality feature extracted by the encoder;

At the same time, this article uses SGD (0.9momentum, 1e-4weight decay) as the optimizer, with an initial learning rate of 1e-3, multiplied by 0.1 every 60 rounds.

S2: Design a globally linked gating mechanism, which consists of a cross-modal gating fusion module and an objective function based on gating auxiliary loss. It is the core of balancing different modalities.

S3: The first part is a cross-modal gating fusion module designed based on the GRU gating principle. It accepts the feature input of voice and image modalities, automatically adjusts the input proportions of different modalities, and further performs interactive fusion, and outputs the fused Cross-modal features.

First, the features of speech and visual modalities are accepted, added through a fully connected layer, and then activated by the Sigmod function to obtain a multi-dimensional gating parameter λ. The information retention of speech and visual modalities is then controlled by the gating parameter λ;

S4: In the second part, the single-modal loss is used as the auxiliary loss, and the gating parameters in the first part are processed to a certain extent and used as the weight of the auxiliary loss to form the adaptive gating adjustment mechanism of the model. After taking the average dimensionality reduction of the gating parameter λ in the first part of the cross-modal fusion module, the reciprocal form is used as the weight of the corresponding modal loss, achieving a negative correlation between the modal loss and the modal retained information. That is to say, when the loss of a single mode is relatively smaller (the mode has reached the fitting state), the model will correspondingly reduce the information retention of the mode. At the same time, the loss of the other single mode is relatively larger (the mode has not yet reached the fitting state). reaches the fitting state), the model will correspondingly improve the information retention of this mode;

λ ^- =mean(λ)

where λ- is the averaged one-dimensional gating parameter, β is a hyperparameter, loss ^a is the cross-entropy loss of a single speech modality, loss ^v is the cross-entropy loss of a single visual modality, and loss ^av is multi-modal fusion The final cross-entropy loss.

It can be understood that loss ^a is the cross entropy loss of a single speech modality, loss ^v is the cross entropy loss of a single visual modality to assist loss ^av is the cross entropy loss after multi-modal fusion to complete the main task.

It should be noted that in this embodiment, in step S4, the hyperparameter β is set to 0.1 in this experiment. This is, loss ^a , loss ^v , loss ^av are the losses calculated using cross entropy for different modal features. The objective function based on gated loss will be used after 30 rounds of model training. The objective function for the first 30 rounds is loss = ^lossav . Combined with Figure 2, the entire model is divided into four major blocks, the multi-modal feature extraction module, the first part of the gated cross-modal feature fusion module, the classification prediction module and the second part of the objective function based on gated auxiliary loss.

Using fully connected layers (FC) as a classifier for classification prediction, the classification results of three different modal features are obtained, among which the classification results of cross-modal features are used as the final classification results.

In the embodiment of the present invention, after the model training is completed, referring to Figure 5, a multi-modal video classification system based on gated cross-modal feature fusion is provided, including:

The multi-modal feature extraction module is used to extract features of speech and visual modalities and encode them as input features for modal feature fusion.

The gated cross-modal feature fusion module performs a balanced fusion of input multi-modal features based on the information expression characteristics of single-modal features.

The classification prediction module performs final classification prediction on the output features after balanced fusion.

The present invention conducts experiments on two public data sets CREMAD and VGGSound. To quantitatively evaluate the performance of RGGL, accuracy acc and mean average accuracy map are used as evaluation metrics.

Table 1

Audio-only is the result of using only speech data and using the speech modality model.

Visual-only is the result of using only visual data and using the visual modality model.

The fusion method of modal features in the Baseline model is simple addition, and the basic objective function is used.

The results in Table 1 show that the performance of the video classification model RGGL based on the parallel ResNet18 of speech and visual modalities and the global linkage gating mechanism of the present invention on the CREMA-D and VGGSound data sets is significantly higher than the Baseline model. Especially when comparing Audio-only and Baseline data on the CREMAD data set, it can be found that after using the multi-modal model, the performance does not increase but decreases. It can be inferred that there is a modal imbalance problem, but after using RRGL, acc increased by nearly 15%. This shows that the global gating mechanism composed of a cross-modal gating fusion module and an objective function based on gating-assisted loss can solve the modal imbalance problem of multi-modal models and significantly improve the performance of multi-modal video classification models. .

The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principle and spirit of the invention, various changes, modifications, substitutions and modifications can be made to these embodiments, including components, and still fall within the protection scope of the invention.

Claims (8)

1. A multi-modal video classification method, characterized by including the following steps: S1. Input the preprocessed video data and use two resnet18 encoders with the same structure to extract speech modal features and visual modal features respectively; S2. Construct a global linkage gating mechanism. The global linkage gating mechanism is composed of a cross-modal gating fusion module and an objective function based on gating auxiliary loss; S3. Take the speech modality features and visual modality features as input, obtain the cross-modal features through the cross-modal gating fusion module, and obtain the gating parameters through the Sigmod activation function, and use the gating parameters to control the speech modality and visual modality. Stateful information retention, S4. Classify and predict the speech modal features, visual modal features and cross-modal features through the fully connected layer, and use the cross entropy formula to calculate three independent loss ^av , loss ^a , and loss ^v ; S5. Use the loss of one mode among the three independent loss ^av , loss ^a , and loss ^v as the auxiliary loss, and use the processed gating parameters as the weight of the auxiliary loss to form an objective function based on the gating auxiliary loss. 2. A multi-modal video classification method according to claim 1, characterized in that, in step S1, the video data includes voice modality and visual modality, The acquisition method of the speech mode is: extract the speech information in the video and convert it into a spectrogram as the speech mode; The visual mode extraction method is: uniformly sampling 3 frames in the video as the visual mode. 3. A multi-modal video classification method according to claim 2, characterized in that, in step S1, the method for extracting speech modal features and image modal features is: The speech mode and visual mode respectively use two identical ResNet18 as encoders to extract features from the input data of the speech and visual modes respectively: in is input from the speech modality,/> is the speech modality speech encoder based on ResNet18, θ ^a is the encoder parameter, and H ^a is the speech modality feature extracted by the encoder; similarly/> is input from the visual modality,/> is the visual modality visual encoder based on ResNet18, θ ^v is the encoder parameter, and H ^v is the visual modality feature extracted by the encoder. 4. A multi-modal video classification method according to claim 1, characterized in that, in step S3, the cross-modal features obtained after fusion are controlled by gating parameters to control the information of speech mode and visual mode. reserve. 5. A multi-modal video classification method according to claim 1, characterized in that the specific method of step S3 is: accepting the characteristics of the speech mode and the visual mode, and adding them after passing through a fully connected layer. After being activated by the Sigmod function, a multi-dimensional gating parameter λ is obtained, and then the gating parameter controls the information retention of the speech mode and the visual mode. The expression is as follows: λ=Sigmod(UH ^a +VH ^v ) H ^av = (1-λ)·H ^a +λ·H ^v Among them, U and V are trainable variables, λ is the gating parameter, sigmod is the activation function, H ^a is the speech modality feature, H ^v is the visual modality feature, and H ^av is the cross-modality feature. 6. A multi-modal video classification method according to claim 4, characterized in that, in the step S5, the processing method of the gating parameter is: after averaging and reducing the dimension of the gating parameter λ, in the form of a reciprocal As the weight of the corresponding modal loss. 7. A multi-modal video classification method according to claim 5, characterized in that in step S5, the expression of the objective function based on gated auxiliary loss is as follows: where λ ^- is the averaged one-dimensional gating parameter, β is a hyperparameter, loss ^a is the cross-entropy loss of a single speech modality, loss ^v is the cross-entropy loss of a single visual modality, and loss ^av is multi-modal fusion The final cross-entropy loss. 8. A multi-modal video classification system, characterized by including: It includes a parallel multi-modal feature extraction module, a cross-modal gated fusion module, an objective function based on gated auxiliary loss, and a classification prediction module; The parallel multi-modal feature extraction module includes two resnet18 encoders with the same structure, used to extract the initial features of speech modality and visual modality; The cross-modal gating fusion module and the objective function based on gating-assisted loss jointly construct a global gating adjustment mechanism, which is used to determine the fitting status of each modality based on the loss of a single modality and adaptively Adjust the proportion of information of different modalities in cross-modal fusion; The classification prediction module performs classification prediction on single modal features and fused modal features. After the model training is completed, the classification prediction of the fused modal features is used as the final prediction result.