CN115982652A - A Cross-Modal Sentiment Analysis Method Based on Attention Network - Google Patents

Abstract

The invention belongs to the fields of natural language processing, computer vision and emotion analysis technology, and discloses a cross-modal emotion analysis method based on an attention network, which comprises the following steps: step 1: extracting picture characteristics, picture text characteristics and aspect characteristics; step 2: the extracted picture text features enter modality updating layers, each modality updating layer comprises a modality alignment module and two modality updating modules, each modality is aligned in the modality alignment module, the aligned modalities enter the modality updating modules, and the interactive picture features and text features are finally obtained by utilizing the correlation of different modalities to supplement step by step; and step 3: performing multi-mode fusion on the step picture characteristics and the text characteristics by adopting a self-attention mechanism; and 4, step 4: and performing concat operation on the picture characteristics, the picture text characteristics and the multi-mode characteristics to predict emotion. The method makes full use of the information interaction among the cross-modal, and is beneficial to improving the accuracy of emotion prediction.

Description

A cross-modal sentiment analysis method based on attention network

Technical Field

The present invention relates to the fields of natural language processing, computer vision and sentiment analysis technology, and specifically to a cross-modal sentiment analysis method based on an attention network.

Background Art

With the development of various online social platforms and network technologies, users have more diverse ways to express their opinions on the Internet. More and more users choose to use videos, pictures or articles to express their emotions and opinions. How to analyze the emotional tendencies and public opinion orientations contained in these multimodal information has become a challenge in the field of sentiment analysis. However, due to the heterogeneity and asynchrony of multimodal data, it is not easy to fuse multimodal information. In terms of heterogeneity, different modalities exist in different feature spaces. In terms of asynchrony, the inconsistent sampling rates of time series data of different modalities make it impossible to obtain the best mapping between different modalities. There are many studies on multimodal analysis, and the specific methods can be summarized into the following two categories: one is to use cross-modal attention to provide soft mapping between different modalities, thereby modeling the asynchrony of multimodal data. However, this type of method does not consider the heterogeneity of multimodal data. The other type considers the heterogeneity of multimodal data. The methods in this category divide each modality into a shared part of the modality and a private part of the modality, which are represented by different neural networks. The limitation of these methods is that they do not consider the asynchrony between different modes.

Summary of the invention

In order to solve the problems of multimodal heterogeneity and heterogeneity, the present invention proposes a cross-modal sentiment analysis method based on attention network, which adopts modal alignment module and modal update module, and uses attention mechanism to perform cross-modal interaction, so as to improve the accuracy of multimodal sentiment analysis.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

The present invention is a cross-modal sentiment analysis method based on attention network, which specifically includes the following steps:

Step 1: Extract the image features and image text features corresponding to the input image text;

Step 2: The extracted image and text features enter the modality update layer. Each of the modality update layers includes a modality alignment module for aligning the representation space and two modality update modules. Each modality is aligned in the modality alignment module and enters the modality update module after alignment. The features are gradually supplemented by utilizing the correlation of different modalities, and finally the image features and text features after interaction are obtained.

Step 3: The interactive image features and text features obtained in step 2 are fused multimodally using the self-attention mechanism to obtain multimodal features;

Step 4: Concat the image features and image text features in step 1 with the fused multimodal features in step 3 to perform sentiment prediction.

Preferably, step 2 specifically includes the following steps:

Step 2.1: The modality alignment module aligns the feature spaces of different modalities before modality interaction to obtain multimodal information;

和

即文本更新模块和图片更新模块，为了使文本和视觉特征更专注于给定方面的信息部分并抑制不太重要的部分，在模态更新层的第一层采用了方面引导的注意力方法，具体过程如下：Step 2.2: The aligned multimodal information enters the modality update module to gradually enhance each modality. Each modality update layer contains two modality update modules.

and

That is, the text update module and the image update module. In order to make the text and visual features focus more on the information part of a given aspect and suppress the less important part, the aspect-guided attention method is adopted in the first layer of the modality update layer. The specific process is as follows:

代表生成的目标模态的隐藏表征，I^A代表方面特征向量，b⁽¹⁾代表可学习参数，

表示可变参数，

表示模态向量；in

represents the generated hidden representation of the target modality, I ^A represents the aspect feature vector, b ⁽¹⁾ represents the learnable parameter,

Indicates variable parameters.

represents the mode vector;

Calculate the normalized attention weights:

对目标模态的特征向量进行加权平均，得到新的目标模态向量

Using attention weights

Perform weighted averaging on the eigenvectors of the target mode to obtain a new target mode vector

的具体过程如下：Step 2.3: In order to capture the two-way interaction between different modalities and enhance the interaction between modalities, the modality update module introduces a cross-modal attention mechanism and a self-attention mechanism to enhance the target modality.

The specific process is as follows:

Among them, * represents the target modality to be enhanced, and α represents the supplementary modality. If the target modality is text, then the supplementary modality is the image. The formula is as follows:

Among them, SA ^mul , CMA ^mul and Att represent the multi-head self-attention mechanism, the multi-head cross-modal attention mechanism and the normalization function and the additive attention mechanism. In order to better integrate the image and text modalities, the present invention uses the additive attention mechanism, which is specifically expressed as follows:

和

Where G, _Wc , _bc represent learnable parameters, and the weight of each modality update module is dynamically calculated through the additive attention mechanism, so as to achieve the purpose of information interaction between the two modalities, and finally obtain the enhanced multimodal sequence

and

In order to learn the deep abstract representation of multimodal features, GRU is used to combine the results of the interactive attention mechanism with the input of the current layer. In the nth layer, the cross-modal attention mechanism and the self-attention mechanism are first used to obtain the enhanced multimodal sequence, and then GRU is used to obtain new text and image features. Among them, n does not include the first layer. The first layer uses the aspect-guided attention mechanism. The specific process is as follows:

为目标模态向量，n代表层数。Among them: SA ^mul represents the multi-head self-attention mechanism,

is the target mode vector, and n represents the number of layers.

Preferably, in step 3, the image features and text features obtained in step 2 are multimodally fused using a self-attention mechanism, which is specifically expressed as follows:

均表示多模态序列，FC是融合多模态函数。in:

Both represent multimodal sequences, and FC is the fusion multimodal function.

Preferably, step 4 is specifically as follows: performing a concat operation on the text features, the image features and the fused multimodal features in steps 1 and 3 to obtain three feature representations E _mul as input data:

E _mul =concat(X _mul ,X _L ,X _V )

Use a fully connected network to fuse the data features, and use a softmax classifier in the last layer to predict sentiment. The sentiment prediction calculation formula is as follows:

P = softmax(W _m E + b _m )

Where _Wm represents the weight of the fully connected layer, _bm represents the bias, and P represents the sentiment prediction.

Preferred: The specific process of extracting image features using the VGG16 network is as follows:

Step 11: Input: Input 224*224*3 image pixel matrix;

Step 12: Convolution pooling: The input image pixel matrix undergoes 5 rounds of convolution pooling operations. The size of each convolution kernel is 3*3*w, where w represents the matrix depth. After convolution, multiple feature maps are obtained through the activation function ReLU, and the maximum pooling is used to filter local features. The convolution calculation formula is as follows:

f _j =R(X _i *K _j +b)

Where R represents the ReLU activation function, * represents the convolution operation, b represents the bias term, and _Kj represents the convolution kernel of different matrix depths;

Step 13: Full connection: After three full connections, a 1*1*1000 image feature representation vector is obtained;

Step 14: Finally, the image feature vector is obtained through the pre-trained VGG16 network and represented by X _Vp = {X _V1 , X _V2 …X _Vn }.

Preferably: In step 1, the Bert pre-training model is used to obtain the image text features. The specific process is as follows:

Step 21: Text preprocessing: Preprocess meaningless words and symbols in network terms, and delete words that do not affect the judgment of the text sentiment tendency as stop words;

Step 22: Use the pre-trained Bert model to extract the word vector sequence of the input text. After the input text is segmented and labeled, it takes the word sequence as input and passes through the word embedding, segment embedding, and position embedding of the Bert model. It flows layer by layer in the stack and finally generates the word vector of the text feature, which is represented by _XLp = { _XL1 , _XL2 … _XLn }.

Preferably, the method for extracting aspect features of a given aspect phrase in step 1 is as follows:

Given an aspect phrase A = {A ₁ , A ₂ …A _n }, first use word embedding to obtain the word embedding vector a _j , and then use the bidirectional LSTM model to learn the hidden representation V _j of each aspect word embedding vector:

Then take the average of all hidden representations _Vj as the final aspect feature vector V ^A :

The beneficial effects of the present invention are:

(1) The sentiment analysis method of the present invention utilizes a modal alignment module and a modal update module, and adopts an attention mechanism to perform cross-modal interaction, thereby improving the accuracy of multimodal sentiment analysis.

(2) The modality update module layer of the present invention includes a modality alignment module and a modality update module. The modality alignment module is used to align feature sequences of different modalities, which facilitates interaction between modalities.

(3) The modality update module uses a multi-head self-attention mechanism and a cross-modal attention mechanism to strengthen the interaction between modalities and fully integrate the shared and private characteristics of different modalities.

(4) In order to preserve the rich features between modalities, the present invention fuses the fused multimodal features with the initial modal features again and then performs sentiment classification.

(5) The present invention makes full use of cross-modal information interaction, which helps to improve the accuracy of emotion prediction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the sentiment analysis method of the present invention.

FIG. 2 is a diagram showing the structure of the sentiment analysis method of the present invention.

FIG. 3 is a diagram of a modal update module of the present invention.

DETAILED DESCRIPTION

The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

As shown in Figures 1-3, the present invention is a cross-modal sentiment analysis method based on an attention network, and proposes a cross-modal sentiment analysis model based on a visual attention network. The information interaction between the two modalities of image and text is enhanced through a modal update layer, thereby enhancing the robustness and accuracy of the model. Specifically, the cross-modal sentiment analysis method includes the following steps:

Step 1: Extract the image features corresponding to the input image text, image text features and aspect features of the given aspect phrase.

The VGG16 network is used to extract image features. The VGG16 network consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The convolutional layer obtains the image feature map through convolution. The point multiplication method is used on the image representation matrix. The convolution kernel slides with a certain step size and multiplies each element at the corresponding position and the input matrix unit. Finally, the feature map of the image based on the current convolution kernel is obtained. The pooling layer reduces the dimension of the feature map after convolution and uses the maximum pooling to screen local features. Finally, the fully connected layer integrates the output features of the upper layer.

The specific process of extracting image features is as follows:

Step 11: Input: Input 224*224*3 image pixel matrix;

Step 12: Convolution pooling: The input image pixel matrix undergoes 5 rounds of convolution pooling operations. The size of each convolution kernel is 3*3*w, where w represents the matrix depth. After convolution, multiple feature maps are obtained through the activation function ReLU, and the maximum pooling is used to filter local features. The convolution calculation formula is as follows:

f _j =R(X _i *K _j +b)

Where R represents the ReLU activation function, * represents the convolution operation, b represents the bias term, and _Kj represents the convolution kernel of different matrix depths;

Step 13: Full connection: After three full connections, a 1*1*1000 image feature representation vector is obtained;

Step 14: Finally, use the pre-trained VGG16 network to obtain the image feature vector

X _Vp ={X _V1 ,X _V2 ..X _Vn }.

The Bert pre-training model is used to obtain image text features. The specific process is as follows:

Step 21: Text preprocessing: Preprocess meaningless words and symbols in network terms, and delete words that do not affect the judgment of the text sentiment tendency as stop words;

Step 22: Use the pre-trained Bert model to extract the word vector sequence of the input text. After the input text is segmented and labeled, it takes the word sequence as input and passes through the word embedding, segment embedding, and position embedding of the Bert model. It flows layer by layer in the stack and finally generates the word vector of the text feature, which is represented by _XLp = { _XL1 , _XL2 … _XLn }.

The extraction method of aspect features for a given aspect phrase is as follows:

Given an aspect phrase A = {A ₁ , A ₂ …A _n }, first use word embedding to obtain the word embedding vector a _j , and then use the bidirectional LSTM model to learn the hidden representation V _j of each aspect word embedding vector:

Then take the average of all hidden representations _Vj as the final aspect feature vector V ^A :

Step 2: The extracted image text features enter the modality update layer. Each of the modality update layers includes a modality alignment module for aligning the representation space and two modality update modules. Each modality is aligned in the modality alignment module and enters the modality update module after alignment. By gradually supplementing the correlation between different modalities, the image features and text features after interaction are finally obtained.

Step 2.1: The modality alignment module aims to align the feature spaces of different modalities before modal interaction. First, the single modality representations of multiple modalities are mapped to the same storage space. The specific representation is as follows:

代表文本向量，Memⁿ代表存储空间向量，θ代表参数，

代表对齐后的模态向量，f(·)代表模态向量和存储空间向量的交换函数，n代表第n层模态更新层，模态对齐模块具体计算过程如下：in

represents the text vector, Mem ⁿ represents the storage space vector, θ represents the parameter,

represents the aligned modal vector, f(·) represents the exchange function between the modal vector and the storage space vector, and n represents the nth modal update layer. The specific calculation process of the modal alignment module is as follows:

K＝Mem ⁿ ·W _K

Where _Wq and _WK represent the parameters of the linear transformation, Q _* represents the vector representation of the two modal linear transformations, K represents the size of the storage space, and the similarity calculation formula between the modal vector and the storage space vector is as follows:

The weight of the jth storage vector is expressed as:

The storage space vector is expressed as follows after linear transformation:

V＝Mem ⁿ ·W _v

W _v represents the learnable parameters, and the query vector is obtained by calculating the memory space vector and weight:

Where: *∈{L,V} represents image features and text features, and V _*j represents the memory space vector.

和

即文本更新模块和图片更新模块，为了使文本和视觉特征更专注于给定方面的信息部分并抑制不太重要的部分，在模态更新层的第一层采用了方面引导的注意力方法，具体过程如下：Step 2.2: The aligned multimodal information enters the modality update module to gradually enhance each modality. Each modality update layer contains two modality update modules.

and

That is, the text update module and the image update module. In order to make the text and visual features focus more on the information part of a given aspect and suppress the less important part, the aspect-guided attention method is adopted in the first layer of the modality update layer. The specific process is as follows:

代表生成的目标模态的隐藏表征，I^A代表方面特征向量，b⁽¹⁾代表可学习参数，

表示可变参数，

表示模态向量；in

represents the generated hidden representation of the target modality, I ^A represents the aspect feature vector, b ⁽¹⁾ represents the learnable parameter,

Indicates variable parameters.

represents the mode vector;

Calculate the normalized attention weights:

对目标模态的特征向量进行加权平均，得到新的目标模态向量

Using attention weights

Perform weighted averaging on the eigenvectors of the target mode to obtain a new target mode vector

的具体过程如下：Step 2.3: In order to capture the two-way interaction between different modalities and enhance the interaction between modalities, the modality update module introduces a cross-modal attention mechanism and a self-attention mechanism to enhance the target modality.

The specific process is as follows:

Among them, * represents the target modality to be enhanced, and α represents the supplementary modality. If the target modality is text, then the supplementary modality is the image. The formula is as follows:

Among them, SA ^mul , CMA ^mul and Att represent the multi-head self-attention mechanism, the multi-head cross-modal attention mechanism and the normalization function and the additive attention mechanism. In order to better integrate the image and text modalities, the present invention uses the additive attention mechanism, which is specifically expressed as follows:

和

Where G, _Wc , _bc represent learnable parameters, and the weight of each modality update module is dynamically calculated through the additive attention mechanism, so as to achieve the purpose of information interaction between the two modalities, and finally obtain the enhanced multimodal sequence

and

Step 2.3 In order to learn the deep abstract representation of multimodal features, GRU is used to combine the results of the interactive attention mechanism with the input of the current layer. In the nth layer, the cross-modal attention mechanism and the self-attention mechanism are first used to obtain the enhanced multimodal sequence, and then GRU is used to obtain new text and image features. Among them, n does not include the first layer. The first layer uses the aspect-guided attention mechanism. The specific process is as follows:

为目标模态向量，n代表层数。Among them: SA ^mul represents the multi-head self-attention mechanism,

is the target mode vector, and n represents the number of layers.

Step 3: The image features and text features obtained in step 2 are fused in a multimodal manner using the self-attention mechanism, as shown below:

均表示多模态序列，FC是融合多模态函数。in:

Both represent multimodal sequences, and FC is the fusion multimodal function.

Step 4: Perform a concat operation on the image features and image text features in step 1 and the fused multimodal features in step 3 to obtain three feature representations E _mul as input data:

E _mul =concat(X _mul ,X _L ,X _V )

Use a fully connected network to fuse the data features, and use a softmax classifier in the last layer to predict sentiment. The sentiment prediction calculation formula is as follows:

P = spftmax (W _m E + b _m )

Where _Wm represents the weight of the fully connected layer, _bm represents the bias, and P represents the sentiment prediction.

In order to maintain richer features between modalities, the present invention uses L2 loss as the loss function, which is as follows:

Where α represents a hyperparameter.

The present invention makes full use of cross-modal information interaction, which helps to improve the accuracy of emotion prediction.

Claims (8)

1. A cross-modal emotion analysis method based on an attention network is characterized by comprising the following steps: the cross-modal emotion analysis method comprises the following steps of:

step 1: extracting picture characteristics corresponding to an input picture text, picture text characteristics and aspect characteristics of a given aspect phrase;

step 2: the extracted picture text features enter modality updating layers, each modality updating layer comprises a modality alignment module and two modality updating modules, each modality is aligned in the modality alignment module, the pictures enter the modality updating modules after being aligned, and finally the picture features and the text features after interaction are obtained by utilizing the correlation of different modalities to be supplemented step by step;

and step 3: performing multi-mode fusion on the interactive picture features and text features obtained in the step 2 by adopting a self-attention mechanism to obtain multi-mode features;

and 4, step 4: and (4) performing concat operation on the picture features and the picture text features in the step (1) and the fused multi-modal features in the step (3) to perform emotion prediction.

2. The method for cross-modal emotion analysis based on attention network as claimed in claim 1, wherein: the step 2 specifically comprises the following steps:

step 2.1: the modal alignment module aligns feature spaces of different modes before modal interaction to obtain multi-modal information;

step 2.2: the aligned multi-modal information enters a modal updating module to gradually enhance each modal, and each modal updating layer comprises two modal updating modules

And &>

Namely a text updating module and a picture updating module, a first layer of a modal updating layer adopts an aspect-guided attention method, and the specific process is as follows:

wherein

Hidden representation of the generated target modality, I ^A Representative facet feature vector, b ⁽¹⁾ Representing a parameter that can be learned by a user,

represents a variable parameter, is selected>

Representing a modality vector;

calculating a normalized attention weight:

using attention weights

Carrying out weighted average on the characteristic vectors of the target mode to obtain a new target mode vector

Step 2.3: in order to capture the bidirectional interaction among different modes and enhance the interaction among the modes, the mode updating module introduces a cross-mode attention mechanism and a self-attention mechanism and enhances the target mode

The specific process is as follows:

wherein, represents the target modality to be enhanced, α represents the supplementary modality, and if the target modality is text, the supplementary modality is a picture, and the formula is as follows:

wherein, SA ^mul ，CMA ^mul And Att partial tables represent a multi-head self-attention mechanism, a multi-head trans-modal attention mechanism, a normalization function and an additive attention mechanism, and the additive attention mechanism is used and is specifically expressed as follows:

wherein G, W _c ，b _c Representing learnable parameters, and the weight of each mode updating module is obtained by dynamic calculation through an additive attention mechanism, so that the aim of information interaction between two modes is fulfilled, and finally a strong multi-mode sequence is obtained

And

3. the attention network-based cross-modal emotion analysis method of claim 2, wherein: in step 2.3, in order to learn the deep abstract representation of the multi-modal features, the GRU is adopted to combine the result of the interactive attention mechanism with the input of the current layer, in the nth layer, the cross-modal attention mechanism and the self-attention mechanism are used to obtain an enhanced multi-modal sequence, and then the GRU is used to obtain new text and picture features, which specifically includes the following steps:

wherein: SA ^mul Representing a multi-head self-attentive mechanism,

for the target mode vector, n represents the number of layers.

4. The method for cross-modal emotion analysis based on attention network as claimed in claim 1, wherein: in the step 3, the picture features and the text features obtained in the step 2 are subjected to multi-modal fusion by using an attention mechanism, which is specifically represented as follows:

wherein:

all represent a multimodal sequence, FC is a fused multimodal function.

5. The attention network-based cross-modal emotion analysis method of claim 1, wherein: the step 4 specifically comprises the following steps: performing concat operation on the text features, the picture features and the fused multi-modal features in the steps 1 and 3 to obtain a representation E containing three features _mul As input data:

E _mul ＝concat(X _mul ,X _L ,X _V )

performing feature fusion on the data by using a full-connection network, and performing emotion prediction by using a softmax classifier at the last layer, wherein the emotion prediction calculation formula is as follows:

P＝softmax(W _m E+b _m )

wherein W _m Weight representing fully connected layer, b _m Representing bias and P representing emotion prediction.

6. The method for cross-modal emotion analysis based on attention network as claimed in claim 1, wherein: the method for extracting the aspect features of the aspect phrases given in the step 1 comprises the following steps:

given aspect phrase a = { a = { [ a ] ₁ ,A ₂ …A _n Get word embedding vector a first using word embedding _j Then, a bidirectional LSTM model is adopted to learn hidden representation V of each aspect word embedding vector _j ：

Then all hidden tokens V are taken _j As the final aspect feature vector V ^A ：

7. The attention network-based cross-modal emotion analysis method of claim 1, wherein: in the step 1, a VGG16 network is adopted to extract picture features, the VGG16 network is composed of 13 convolution layers, 5 pooling layers and 3 full-connection layers, and the specific process of extracting the picture features by adopting the VGG16 network is as follows:

step 11: inputting: inputting 224 x 3 matrix of image pixels;

step 12: convolution pooling: the input image pixel matrix is subjected to 5 rounds of convolution pooling, the size of each convolution kernel is 3 x w, w represents the depth of the matrix, a plurality of feature maps are obtained through an activation function ReLU after convolution, and local features are screened by adopting maximum pooling, wherein the convolution calculation formula is as follows:

f _j ＝R(X _i *K _j +b)

where R represents the ReLU activation function, which represents the convolution operation, b represents the bias term, K _j Convolution kernels representing different matrix depths;

step 13: fully connecting: obtaining 1 × 1000 image feature representation vectors through three times of full connection;

step 14: finally, the picture feature vector is obtained through the pre-trained VGG16 network

X _Vp ＝{X _V1 ,X _V2 …X _Vn Represents it.

8. The attention network-based cross-modal emotion analysis method of claim 1, wherein: in the step 1, a Bert pre-training model is adopted to obtain the picture text characteristics, and the specific process is as follows:

step 21: text preprocessing: preprocessing is carried out aiming at nonsense words and symbols in the network words, and words which do not influence the judgment of the text emotional tendency are used as stop words and deleted;

step 22: extracting a word vector sequence of an input text by adopting a pretrained Bert model, inputting the text, segmenting and labeling the input text, inputting the input text by using the word sequence, performing word embedding, segment embedding and position embedding of the Bert model, flowing layer by layer in a stack, finally generating a word vector of text characteristics, and using X to use _Lp ＝{X _L1 ,X _L2 …X _Ln Denotes.

Images