CN116541579A - Aspect-level emotion analysis based on local context focus mechanism and conversational attention - Google Patents

Abstract

The invention provides an aspect-level emotion analysis based on a local context focus mechanism and conversational attention, comprising: s1, constructing an analysis model; step S2, modeling words in the local context form sequence and the global context form sequence by the BERT pre-training layer respectively to obtain preliminary local context characteristics and preliminary global context characteristics; s3, at a feature extraction layer, utilizing a local context focus mechanism, further extracting local context features by combining a context feature dynamic masking technology with a talking attention mechanism, and extracting global context features by using the talking attention mechanism; s4, at a feature learning layer, fusing local context features and global context features to obtain fusion vectors, and extracting features of the fusion vectors by adopting a conversational attention mechanism; and S5, acquiring a result of the aspect emotion analysis according to the characteristics of the fusion vector at the output layer. The invention can better capture the emotion contained in different aspects.

Description

Aspect-level Sentiment Analysis Based on Local Context Focus Mechanism and Conversational Attention

technical field

The present invention relates to the field of opinion mining, in particular to aspect-level sentiment analysis based on local context focus mechanism and conversational attention.

Background technique

With the rapid development of the Internet, various online platforms have emerged one after another. From news, blogs to forums, the participation of users in the Internet is getting higher and higher. Users express their views on some events by browsing hot information on the Internet. and attitude. At the same time, various products and entertainment methods are also displayed to users through the Internet. After purchasing and experiencing, users will post a large number of comments expressing their own opinions on products and services. These text data with user opinions are very important data resources, and it is of great significance to analyze these text data with user opinions. For example, merchants can analyze the data to obtain users' preferences and deficiencies for a certain product, so as to find better directions for improvement and increase product sales; after an emergency occurs, by analyzing people's comments on the event Data, so as to better control the direction of public opinion; when the government introduces new policies, it analyzes the opinions of netizens to judge whether the proposed policies are effective, so as to make adjustments.

With the continuous advancement of social media, sentiment analysis has high theoretical significance and application value in the field of natural language processing. Sentiment analysis is mainly to classify the different emotional expressions in the text. Most of the previous research on sentiment analysis is a kind of coarse-grained analysis, which cannot meet the needs of more accurate and fine-grained analysis. For example, for a certain product, analyze the advantages and disadvantages of the product from different aspects. Aspect-level sentiment analysis is different from the previous coarse-grained research, which can analyze the emotional polarity of different aspects in a sentence, so it has become an important research direction in the field of sentiment analysis. For example: "The price of this house is good, but the location is terrible", there are two aspect words in this comment on the house, namely "price" and "location", and the corresponding opinion words are " good” and “terrible”, where “good” represents positive emotions and “terrible” represents negative emotions. In this case, aspect-level sentiment analysis can more fully capture the sentiment of different aspects expressed in the text. It can be seen that the research on aspect-level sentiment analysis has very important research significance and value.

With the introduction of the pre-training model, the pre-training model headed by BERT has received extensive attention. More and more researchers have used this model in the field of aspect-level sentiment analysis, proving that the BERT pre-training model is effective for aspect-level sentiment analysis tasks. It works. However, aspect-level sentiment analysis research in the prior art is all about predicting the emotional polarity of different aspects in a sentence, and lacks consideration of the relationship between emotional polarity and local context. In addition, most studies combine single attention or multi-head attention mechanisms, but the operations of each head in the multi-head attention mechanism are independent of each other, so the existing aspect-level sentiment analysis methods in the language processing model There is still room for improvement.

Contents of the invention

The purpose of the present invention is to propose an aspect-level sentiment analysis based on the local context focus mechanism and conversation attention, establish a relationship between emotional polarity and local context, and connect independent heads in the multi-head attention mechanism to obtain a stronger The attention design has achieved better results in language processing models, and can better capture the emotions contained in different aspects.

The present invention is realized through the following technical solutions:

Aspect-level sentiment analysis based on local context focus mechanism and conversation attention, including the following steps:

Step S1, constructing an analysis model including BERT pre-training layer, feature extraction layer, feature learning layer and output layer;

Step S2, the BERT pre-training layer processes the corpus to be analyzed into a local context form sequence and a global context form sequence, and models the words in the local context form sequence and the global context form sequence respectively to obtain preliminary local context features and preliminary global context features /> Step S3. In the feature extraction layer, use the local context focus mechanism to further extract local context features through the context feature dynamic masking technology combined with the conversation attention mechanism, and use the conversation attention mechanism to extract global context features, specifically including:

Step S31, according to the formula Calculate the semantic relative distance of the local context form sequence, where i represents the position of the word in the local context form sequence, F _a represents the position of the aspect word in the local context form sequence, and n represents the length of the aspect word in the local context form sequence;

Step S32, using context feature dynamic masking technology to help the model capture local context features, and obtain local context features Among them, M=[V ₁ , V ₂ ,...V _n ] is a masking matrix for masking non-local contextual features, /> Be the mask vector of each context word in the local context form sequence, i=1,2,...,n, a is the semantic relative distance threshold, E is a 1 vector whose length is n, and O is a 0 vector whose length is n;

Step S33, using the chat attention mechanism to further extract local context features Using Conversation Attention Mechanism to Extract Global Context Features/>

Step S4, in the feature learning layer, fuse the local context feature O ^l and the global context feature O ^g to obtain a fusion vector, and use the conversation attention mechanism to extract the features of the fusion vector Step S5, at the output layer, according to the characteristics of the fusion vector /> Get the results of aspect-level sentiment analysis. Further, in the step S2, preprocessing the corpus to be analyzed into a local context form sequence and a global context form sequence is specifically: processing the corpus to be analyzed into a local context form sequence X ^l =[CLS]+context+[SEP ], global context form sequence X ^g =[CLS]+context+[SEP]+aspect word+[SEP] form, where [CLS] can be used as the semantic representation of the entire sentence.

Further, in the step S2, the BERT pre-training layer adopts the first BERT training model BERT ¹ to model X ¹ to obtain Using the second BERT training model BERT ^g to model X ^g to get Wherein, the first BERT training model BERT ^l and the second BERT training model BERT ^g are independent of each other.

Further, in the step S4, the feature of the fusion vector is in, O ^lg = [O ^l ; O ^g ], W ^lg represents the weight coefficient matrix, b ^lg represents the bias vector, /> is a fully connected layer.

Further, in the step S5, the output layer is a nonlinear layer, and the features of the fusion vector Enter the nonlinear layer and use the softmax function to predict: /> where, /> is the analysis result, W _o is the weight matrix, b _o is the bias vector.

The present invention has following beneficial effect:

The present invention fully considers the importance of local context on emotional polarity, uses the first BERT training model and the second BERT training model that are independent of each other to extract local and global context features, and then uses the context features in the local context focus mechanism to dynamically The mask layer combines the conversation attention mechanism to further capture the local context features, and then fuses with the global information and then inputs it to the nonlinear layer for sentiment analysis, achieving stronger attention design and achieving better results in the language processing model , which can better capture the emotions contained in different aspects.

Description of drawings

The present invention will be described in further detail below in conjunction with the accompanying drawings.

Fig. 1 is a flowchart of the present invention.

Fig. 2 is an analysis model diagram of the present invention.

Fig. 3 is a dynamic mask diagram of context features in the present invention.

Fig. 4 is a graph showing the influence of semantic relative distance on analysis accuracy in the present invention.

Fig. 5 is a graph showing the influence of semantic relative distance on MF1 value in the present invention.

Fig. 6 is a graph showing the influence of the present invention on the analysis accuracy.

Fig. 7 is an image diagram of MF1 according to the present invention.

Detailed ways

As shown in Figure 1 and Figure 2, the aspect-level sentiment analysis based on the local context focus mechanism and conversational attention includes the following steps:

Step S1, constructing an analysis model (i.e. LCFTHA model) comprising BERT pre-training layer, feature extraction layer, feature learning layer and output layer;

Step S2, the BERT pre-training layer processes the corpus to be analyzed into a local context form sequence and a global context form sequence, and models the words in the local context form sequence and the global context form sequence respectively to obtain preliminary local context features and preliminary global context features />

Specifically, the corpus to be analyzed is processed into a local context form sequence X ^l = [CLS] + context + [SEP], a global context form sequence X ^g = [CLS] + context + [SEP] + aspect words + [SEP] In the form of , [CLS] can be used as the semantic representation of the entire sentence, and [SEP] is the separator of the sentence processed by the BERT pre-training layer;

The first BERT training model BERT ^l is used to model X ^l respectively to obtain The second BERT training model BERT ^g models X ^g to get /> Wherein, the first BERT training model BERT ^l and the second BERT training model BERT ^g are independent of each other;

Step S3. In the feature extraction layer, use the local context focus mechanism to further extract local context features through the context feature dynamic masking technology combined with the conversation attention mechanism, and use the conversation attention mechanism to extract global context features, specifically including:

Step S31, according to the formula Calculate the semantic relative distance of the local context form sequence (semantic relative distance is based on the concept of Token-Aspect pair, that is, the position of the word in the sentence and the position of the aspect word, describing the distance between Token and Aspect, which can be understood as the distance between the two How many words are separated between them), where i represents the position of the word in the local context form sequence, F _a represents the position of the aspect word in the local context form sequence, n represents the length of the aspect word in the local context form sequence, and D _i represents the i-th The distance between the position of a word and the target facet;

Step S32, using context feature dynamic masking technology to help the model capture local context features, and obtain local context features Among them, M=[V ₁ , V ₂ ,...V _n ] is a masking matrix for masking non-local contextual features, /> is the mask vector of each context word in the local context form sequence, when D _i ≤ a, V _i represents the local context, i=1,2,...,n, a is the semantic relative distance threshold, E is the length 1 vector of n, O is a 0 vector of length n;

Among them, as shown in Figure 3, in addition to the local context features, the context dynamic feature mask layer will shield the non-local context features learned by the layer, the features of the output position pointed by the dotted line arrow will be masked, and the output position pointed by the solid line arrow The feature will be preserved, POS is represented as the position of the output, and the context feature dynamic mask sets the feature of all positions of the non-local context to a zero vector;

Step S33, using the chat attention mechanism to further extract local context features Using Conversation Attention Mechanism to Extract Global Context Features/>

By connecting the independent attention heads of the multi-head attention mechanism, that is, re-integrating multiple attention heads with a parameter matrix to form multiple mixed attentions, each newly obtained mixed attention is fused The original attention of each head, namely: O _THA = concat[{O ⁽¹⁾ ,O ⁽²⁾ ,...,O ^(h) } W ^WH ], where concat[] is used to connect two or Multiple arrays, O ⁽¹⁾ = P ⁽¹⁾ V ⁽¹⁾ , O ⁽²⁾ = P ⁽²⁾ V ⁽²⁾ ,..., O ^(h) = P ^(h) V ^(h) , P ⁽¹⁾ =softmax(J ⁽¹⁾ ),P ⁽²⁾ =softmax(J ⁽²⁾ ),...,P ^(h) =softmax(J ^(h) ), J ^(h) means introducing a linear mapping between each attention head before the softmax operation, O ^(h) means Attention (Q ^(h) , K ^(h) , V ^(h) ), Represents the reduction factor, h represents the number of attention, W ^WH represents the weight matrix, λ _hh represents the trainable parameter matrix;

The chat attention mechanism can rebalance the masked local contextual features and avoid the unbalanced feature distribution mechanism after the dynamic masking of contextual features, the formula The THA () in is to adopt O _THA =concat[{O ⁽¹⁾ ,O ⁽²⁾ ,...,O ^(h) } W ^WH ] to calculate, and the specific calculation process is the prior art;

Step S4, in the feature learning layer, fuse the local context feature O ^l and the global context feature O ^g to obtain a fusion vector, and use the conversation attention mechanism to extract the features of the fusion vector

Specifically, the fusion vector is characterized by where, /> O ^lg = [O ^l ; O ^g ], W ^lg represents the weight coefficient matrix, /> b ^lg represents the bias vector, /> d _n and d _h represent the number of rows and columns of O ^l and O ^g respectively, /> is a fully connected layer;

Step S5, at the output layer, according to the characteristics of the fusion vector Obtain the results of aspect-level sentiment analysis;

Specifically: the output layer is a nonlinear layer, and the features of the vector will be fused Enter the nonlinear layer and use the softmax function to predict: /> where, /> is the analysis result, W _o is the weight matrix, b _o is the bias vector.

In terms of experiments, the present invention verifies the validity of the model in this paper by using two data sets of Restaurant14 and Laptop14 and a Twitter data set. Among the comment information in these three public data sets, the aspect words in each sentence correspond to three different emotional polarities, and the data details are shown in Table 1.

Table 1 Dataset distribution

Some hyperparameter settings of the LCFTHA model: the learning rate of the BERT layer is 0.00002, the number of iterations is 20, the batch size is 32, the maximum sentence length is 85, the number of conversational attention heads is 12, the L2 regularization weight is 0.01, BERT Model selection bert-base-uncased (open source bert pre-training model).

The present invention selects seven baseline models for comparative experiments to verify the validity of the LCFTHA model. Two commonly used evaluation indicators in aspect-level sentiment analysis tasks are mainly used to verify the effect of the model. The first evaluation index is Accuracy, which indicates the proportion of correctly predicted samples to all samples in the task. The correct prediction may be a positive sample or a negative sample, that is:

Among them, TP represents the positive samples that are predicted correctly, TN represents the negative samples that are correctly predicted, FP represents the positive samples that are incorrectly predicted, and FN represents the negative samples that are incorrectly predicted.

Among them, P represents the precision rate in the emotion category, R represents the recall rate in the emotion category, and C represents the total number of emotion categories.

By conducting experiments on three public datasets of Restaurant14, Laptop14 and Twitter, and comparing seven baseline models, the results are shown in Table 2.

Table 2 Model accuracy and macro average (%)

Among them, on the three public datasets Restaurant14, Laptop14 and Twitter, the LCFTHA model proposed in this paper is superior to the baseline model in accuracy and MF1 value, and its accuracy is greatly improved compared with other models.

In the model LCFTHA, the semantic relative distance is an important factor affecting the extraction of local context features, so the present invention analyzes the influence of the semantic relative distance on the three data sets through experiments. It can be seen from Figure 4 and Figure 5 that the semantic relative distance When the threshold is 2, the analysis accuracy and MF1 value (MF1 represents the average performance of the model under different emotional categories) of the Restaurant14 data set are optimal. For the Laptop14 dataset, when the threshold of semantic relative distance is 7, the analysis accuracy and MF1 value are optimal. For the Twitter dataset, the threshold of semantic relative distance is the same as that of the Laptop14 dataset. When the threshold is 7, the analysis accuracy and MF1 value are optimal.

Finally, the present invention carried out ablation experiments on three public data sets to verify the importance of each module in the LCFTHA model, where w/o is the abbreviation of without, and FLL is the abbreviation of feature learning layer. The results are shown in the table 3.

Table 3 Ablation experiment results (%)

It can be seen from the observation of Table 3 that after combining the global and local module components, it can be seen that the effect is better than that of a single local or global module. On this basis, it can be seen that the performance of the model is further improved by adding FLL, and the experimental results show that the FLL layer is of great significance to the design of the model. The importance of each model component is more intuitively represented by a histogram, as shown in Figure 6 and Figure 7.

To sum up, the present invention proposes an aspect-level sentiment analysis model LCFTHA based on local context focus mechanism and conversation attention mechanism. First, use two BERT pre-training models to extract local and global context features; then, use the preliminary local features extracted by BERT to further capture local context features through the CDM layer in the local context focus mechanism combined with the conversation attention mechanism, A conversational attention encoder is deployed for global context features to learn global context features; finally, after the feature learning layer fuses local and global context features, it is input to the nonlinear layer for sentiment analysis. Experiments on three public datasets of Restaurant14, Laptop14 and Twitter prove that the performance of the LCFTHA model is better than the baseline models in other aspect-level sentiment analysis tasks.

The above is only a preferred embodiment of the present invention, so it cannot limit the scope of the present invention, that is, equivalent changes and modifications made according to the patent scope of the present invention and the content of the specification should still belong to the present invention covered by the patent.

Claims (5)

1. Aspect-level sentiment analysis based on local context focus mechanism and conversational attention, characterized in that: comprising the following steps: Step S1, constructing an analysis model including BERT pre-training layer, feature extraction layer, feature learning layer and output layer; Step S2, the BERT pre-training layer processes the corpus to be analyzed into a local context form sequence and a global context form sequence, and models the words in the local context form sequence and the global context form sequence respectively to obtain preliminary local context features and preliminary global context features /> Step S3. In the feature extraction layer, use the local context focus mechanism to further extract local context features through the context feature dynamic masking technology combined with the conversation attention mechanism, and use the conversation attention mechanism to extract global context features, specifically including: Step S31, according to the formula Calculate the semantic relative distance of the local context form sequence, where i represents the position of the word in the local context form sequence, F _a represents the position of the aspect word in the local context form sequence, and n represents the length of the aspect word in the local context form sequence; Step S32, using context feature dynamic masking technology to help the model capture local context features, and obtain local context features Among them, M=[V ₁ , V ₂ ,...V _n ] is a masking matrix for masking non-local contextual features, /> Be the mask vector of each context word in the local context form sequence, i=1,2,...,n, a is the semantic relative distance threshold, E is a 1 vector whose length is n, and O is a 0 vector whose length is n; Step S33, using the chat attention mechanism to further extract local context features Using Conversation Attention Mechanism to Extract Global Context Features/> Step S4, in the feature learning layer, fuse the local context feature O ^l and the global context feature O ^g to obtain a fusion vector, and use the conversation attention mechanism to extract the features of the fusion vector Step S5, at the output layer, according to the characteristics of the fusion vector Get the results of aspect-level sentiment analysis. 2. The aspect-level sentiment analysis based on local context focus mechanism and conversational attention according to claim 1, characterized in that: in the step S2, the corpus to be analyzed is preprocessed into a local context form sequence and a global context form sequence Specifically: process the corpus to be analyzed into local context form sequence X ^l = [CLS] + context + [SEP], global context form sequence X ^g = [CLS] + context + [SEP] + aspect word + [SEP] In the form of , [CLS] can be used as the semantic representation of the entire sentence. 3. The aspect-level sentiment analysis based on local context focus mechanism and conversation attention according to claim 2, characterized in that: in the step S2, the BERT pre-training layer adopts the first BERT training model BERT ¹ to X ¹ modeled Using the second BERT training model BERT ^g to model X ^g to get /> Wherein, the first BERT training model BERT ^l and the second BERT training model BERT ^g are independent of each other. 4. The aspect-level sentiment analysis based on local context focus mechanism and conversation attention according to claim 1 or 2 or 3, characterized in that: in the step S4, the feature of the fusion vector is in, O ^lg = [O ^l ; O ^g ], W ^lg represents the weight coefficient matrix, b ^lg represents the bias vector, /> is a fully connected layer. 5. according to claim 1 or 2 or 3 described aspect-level sentiment analysis based on local context focus mechanism and conversation attention, it is characterized in that: in described step S5, described output layer is a nonlinear layer, and described Features of fused vectors Enter the nonlinear layer and use the softmax function to predict: /> where, /> is the analysis result, W _o is the weight matrix, b _o is the bias vector.