CN108647191B - A Sentiment Dictionary Construction Method Based on Supervised Sentiment Text and Word Vectors - Google Patents

Abstract

The invention proposes an emotion dictionary construction method based on supervised emotion text and word vector, which includes three stages: data processing stage, word vector emotion embedding stage, and emotion dictionary generation stage. This method uses a neural network to generate word vectors, embeds emotions into the word vectors, mines the internal relationship between words, and then builds a word relationship graph, uses a label propagation algorithm to propagate sentiment labels, and automatically builds a domain-specific sentiment dictionary. The invention solves the problem that the sentiment dictionary constructed by artificial and knowledge base-based methods is inaccurate when dealing with sentiment analysis tasks in specific fields.

Description

A Sentiment Dictionary Construction Method Based on Supervised Sentiment Text and Word Vectors

technical field

The invention relates to the field of sentiment analysis, in particular to a construction method of sentiment dictionary based on supervised sentiment text and word vectors.

Background technique

With the rapid development of the Internet, the popularity of various network platforms such as Weibo, Tieba, and forums has provided people with many opportunities to speak out. The resulting public textual data is plentiful, readily available, and has enormous commercial and social value. Sentiment analysis techniques come to the fore in order to capture the sentimental tendencies of people towards things or events in these texts.

Sentiment dictionaries have always been an important tool for sentiment analysis. An excellent sentiment dictionary can greatly improve the effect of sentiment analysis. Usually, as the application domain changes, the emotion embodied by the word will also change accordingly. Therefore, when dealing with domain-specific sentiment analysis tasks, it becomes time-consuming and labor-intensive to manually organize sentiment lexicons, and an automated method is needed to construct sentiment lexicons. The existing automatic construction methods of sentiment dictionary are divided into two categories, namely knowledge base-based methods and corpus-based methods. Knowledge base-based methods rely on existing semantic knowledge bases. The definitions of a large number of words and the relationships between words (such as synonyms and antonyms) are recorded in these manually organized knowledge bases. The knowledge base-based method constructs a sentiment dictionary with high accuracy and generality through these existing knowledge. However, for Chinese, a well-organized knowledge base is relatively scarce, so this method cannot be well applied to the construction of Chinese sentiment dictionaries. At the same time, the sentiment dictionary generated by this method is relatively general, and cannot well solve the problem of emotional changes of words in different fields. Corpus-based methods can be used to generate domain-specific sentiment lexicons. This kind of method processes the corpus text and mines the relationship between words in the corpus, such as conjunction relationship, co-occurrence relationship, etc. It generates sentiment lexicons by setting rules or using statistical methods to cluster closely related words together. This type of method often only considers the simple relationship of words in the text, ignoring the complexity of the text itself, such as some complex syntax and the influence of negative words, etc., which will affect the effect of this type of method.

SUMMARY OF THE INVENTION

Purpose of the invention: Aiming at the shortcomings of the automatic construction method of emotional dictionary based on corpus, the present invention proposes a construction method of emotional dictionary based on supervised emotional text and word vector, using neural network to generate word vector, and mining the inner relationship between words and words , and then generate a sentiment dictionary.

Technical scheme: The technical scheme proposed by the present invention is:

A sentiment dictionary construction method based on supervised sentiment text and word vectors, including steps:

(1) Obtaining a text dataset D, the text dataset D includes positive sentiment texts with positive sentiment marks and negative sentiment texts with negative sentiment marks;

(2) Preprocess the text in the text data set; construct a vocabulary V, and fill in the words in the preprocessed text data set into the vocabulary V one by one;

(3) The SO-PMI method is used to calculate the emotional tendency value of each word in the vocabulary V, and the emotional mark of the corresponding word is determined according to the emotional tendency value:

Among them, lable _w represents the sentiment mark of the word w, and SO-PMI(w) represents the sentiment tendency value of the word w;

(4) Construct an improved skip-gram model with word-level supervision. The improved skip-gram model takes the words in D as input data to predict the context and sentiment tags of the words; calculate the loss function loss _{context when predicting the context} , and loss function loss _word when predicting sentiment labels;

The expressions of loss _context and loss _word are:

loss _context (w _t )=-∑ _{-k≤j≤k,j≠0} logp(w _t+k |w _t )

where w _t represents a word, w _t ∈ D; {w _tk ,…,w _t-1 ,w _t+1 ,…,w _t+k } represents a set of predicted context words, and the set includes the predicted words The first k words and the last k words of w _t , p(w _t+j |w _t ) represents the probability of w _t+j being predicted as the context of w _t , p(pos|w _t ) means that w _t is predicted is the probability of having a positive sentiment label, p(neg|w _t ) represents the probability that _wt is predicted to have a negative sentiment label;

(5) Construct a convolutional neural network model as a text-level supervision model. The text-level supervision model uses the text in the text data set D as input data to predict the emotional tags of the text; calculate the emotional tags of the predicted text and the actual emotions of the text. Loss function loss _doc between tokens:

表示d_i的情感标签；p(pos|d_i)表示d_i被预测为具有正面情感标记的概率，p(neg|d_i)表示d_id_i被预测为具有负面情感标记的概率；Among them, d _i represents the text, d _i ∈ D;

represents the sentiment label of d _i ; p(pos|d _i ) represents the probability that d _i is predicted to have a positive sentiment label, and p(neg|d _i ) represents the probability that d _i d _i is predicted to have a negative sentiment label;

(6) Set the joint loss function:

loss=α ₁ ·loss _context +α ₂ ·loss _doc +α ₃ ·loss _word

In the formula, α ₁ , α ₂ , and α ₃ are the weight coefficients of loss _context , loss _doc , and loss _word respectively;

为输入数据，利用反向传播算法训练联合损失函数，得到具有情感嵌入的词向量；(7) Take the text dataset D, the sentiment label of words, label _w , and the sentiment label of text

For the input data, use the back-propagation algorithm to train the joint loss function to obtain the word vector with sentiment embedding;

(8) constructing a word relation graph G according to the word vector with emotional embedding obtained in step (7);

(9) Select some words in the word relation graph G as seed words, and mark the sentiment labels for the seed words. The sentiment labels include positive, derogatory and neutral; then use the label propagation algorithm to spread the sentiment labels of the seed words in the relation graph G. , to generate a sentiment dictionary.

Further, the calculation formula of the emotional tendency value is:

Among them, SO-PMI(w) represents the sentiment tendency value of the word w, pos represents the positive sentiment text, neg represents the negative sentiment text, p(w|pos) represents the probability that the word w appears in the positive sentiment text, p(w| neg) represents the probability that the word w appears in the negative sentiment text.

Further, the improved skip-gram model with word-level supervision includes an input layer, a projection layer, and an output layer. The input layer is the word _wt in the text data set D, and the projection layer projects the word _wt into a word vector C. (w _t ), the output layer predicts the context and sentiment label lable _w of _{wt t} according to C(wt ), respectively.

Further, the text-level supervision model includes: an input layer, a convolution layer, a pooling layer, and a fully connected layer, wherein the input layer is the text d _i in the text data set D; Extract multiple feature vectors from d _i and send them to the pooling layer; the pooling layer selects the most important feature vector from the feature vector through the MaxPoolingOverTime operation and outputs it to the fully connected layer; the fully connected layer predicts through the softmax function according to the received feature vector Sentiment token for input text d _i

Further, the specific steps of constructing the word relation graph G include:

1) For vocabulary V, extract the verbs, adjectives and adverbs in it to form a new vocabulary V';

2) Construct the word relation graph G, and use the words in V' as the vertices in G;

3) For each word _wi in V', calculate the Euclidean distance between _wi and all other words in V' in the word vector space obtained in step (7), select the k words with the nearest Euclidean distance, The edge between _wi and the k words is established in the relational graph G, and the weight calculation formula of the edge is:

Among them, w _ij represents the weight of the edge between words _wi and w _j , _xi and x _j are the word vectors of words _wi and w _j respectively, euclidean_dis( _xi , x _j ) represents the distance between _xi and x _j The Euclidean distance; σ is a constant parameter, used to control the value of w _ij .

For words other than the m words closest to word wi, let w _{ij =} 0

Beneficial effect: Compared with the prior art, the present invention has the following advantages:

The invention generates an emotion dictionary based on a supervised corpus, uses a neural network to generate word vectors, mines the inner connection between words, uses a label propagation algorithm to propagate emotion labels, and automatically constructs an emotion dictionary in a specific field. The invention not only avoids the deficiency that the knowledge base-based sentiment dictionary construction method cannot be used for sentiment analysis in a specific field, but also strengthens the consideration of the complex relationship of words in this article compared with other corpus-based methods. Finally, the automatic construction of emotion dictionary is realized.

Description of drawings

Fig. 1 is the overall flow chart of the present invention;

Fig. 2 is the structure diagram of the improved skip-gram model;

Figure 3 is a structural diagram of a convolutional neural network model.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

Fig. 1 shows the overall flow of the present invention. The present invention is mainly divided into three stages: data processing stage, word vector emotion embedding stage, and emotion dictionary generation stage. The specific steps of each stage are described in detail below in conjunction with accompanying drawings 1 to 3. describe.

1. Data processing stage (steps 1-3):

表示文本d_i的情感，其中d_i＝0表示负面情感标签，d_i＝1表示正面情感标签。Step 1 is data acquisition, that is, to obtain a text dataset D marked with emotional labels. The emotional labels of the text in D are divided into positive and negative, and the label is used.

represents the sentiment of the text d _i , where d _i =0 represents a negative sentiment label, and d _i =1 represents a positive sentiment label.

Step 2 is data preprocessing. First, use the open source tool jieba to segment the text and tag it, and then use the stop word list to remove the stop words in the text to obtain the word sequence. The text dataset of is denoted as D={d ₁ ,d ₂ ,...,d _n }.

Step 3 is to use the SO-PMI method to calculate the approximate sentiment of the word. The calculation formula of the SO-PIM method is:

Among them, SO-PMI(w) represents the sentiment tendency value of the word w, pos represents the positive sentiment text, neg represents the negative sentiment text, p(w|pos) represents the probability that the word w appears in the positive sentiment text, p(w| neg) represents the probability that the word w appears in the negative sentiment text.

Define lable _w to represent the word w sentiment token,

Second, the word vector emotion embedding stage (steps 4-6):

其中，Step 4 is the construction of a word-level supervision model, that is, an improved Skip-gram model is constructed, and word-level emotion supervision data is used to train word vectors. The model consists of an input layer, a projection layer, and an output layer. The input layer is a word _wt in the training data, the projection layer projects the word _wt into a word vector representing C( _wt ), and the output layer uses C( _wt ) to predict the context and sentiment tags of _wt respectively

in,

The loss function for predicting the context of _wt is:

The loss function when predicting sentiment labels is:

Among them, w _t represents the word, w _t ∈ D; k represents the range of the predicted context, {w _tk ,...,w _t-1 ,w _t+1 ,...,w _t+k } represents the predicted context word set, The set includes the first k words and the last k words of the predicted word w _t ; p(w _t+j |w _t ) represents the probability that w _t+j is predicted as the context of w _t , p(pos|w _t ) represents the probability that wt is predicted to have positive sentiment, and p(neg|w _t ) represents the probability that _{wt is} predicted to have negative sentiment.

其损失函数为：Step 5 is the construction of a text-level supervised model, that is, to construct a convolutional neural network model. The convolutional neural network model consists of an input layer, a convolutional layer, a pooling layer, and a fully connected layer. Among them, the input is a piece of text d _i in the text dataset D, the convolution layer extracts multiple feature vectors from the text d _i through the feature extractor, and the pooling layer extracts the most important feature vectors from the feature vectors through the Max Pooling Over Time operation. The feature vector is used as the output, and the fully connected layer predicts the sentiment tag of the input text d _i through the softmax function

Its loss function is:

where p(pos|d _i ) represents the probability that d _i is predicted to have positive sentiment, and p(neg|d _i ) represents the probability that d _i is predicted to have negative sentiment.

Step 6 is to jointly train the model, that is, to combine the word-level supervised model and the text-level supervised model, and set the loss function as:

loss=α ₁ ·loss _context +α ₂ ·loss _doc +α ₃ ·loss _word

为输入数据，使用随机梯度下降和误差反向传播算法对loss进行优化训练，获得具有情感嵌入的词向量。In the formula, α ₁ , α ₂ , and α ₃ are the weight coefficients of loss _context , loss _doc , and loss _word , respectively, which are used to control the weights of the three loss functions in the final loss function respectively. Take text dataset D, the sentiment label of words lable _w , the sentiment label of text

For the input data, the loss is optimized using stochastic gradient descent and error back-propagation algorithms to obtain word vectors with sentiment embeddings.

3. Sentiment dictionary generation stage (steps 7-9):

Step 7 is to construct a word relationship graph, and the specific steps are:

1) For vocabulary V, extract the verbs, adjectives and adverbs in it to form a new vocabulary V';

2) Construct the word relation graph G, and use the words in V' as the vertices in G;

3) For each word _wi in V', calculate the Euclidean distance between _wi and all other words in V' in the word vector space obtained in step (7), select the k words with the nearest Euclidean distance, The edge between _wi and the k words is established in the relational graph G, and the weight calculation formula of the edge is:

Among them, w _ij represents the weight of the edge between words _wi and w _j , _xi and x _j are the word vectors of words _wi and w _j respectively, euclidean_dis( _xi , x _j ) represents the distance between _xi and x _j The Euclidean distance; σ is a constant parameter, used to control the value of w _ij .

For words other than the m words closest to word wi, let w _{ij =} 0

Step 8 is to use the label propagation algorithm to spread emotional labels, and the specific steps are:

1) Manually mark a small number of seed emotional words, that is, manually mark a small number of words with positive, derogatory and neutral meanings as seed words;

2) Define the label matrix Y. The label matrix Y is a matrix of size |V′|×3. Each row in Y corresponds to a word in the vocabulary V′, and the three columns represent that the word is positive, derogatory and neutral. The probability. The label matrix is initialized according to the artificially labeled seed sentiment words; the initialization method adopted in this embodiment is: for the artificially labeled words in 1), if it is a compliment, the corresponding row is initialized to [1,0,0], if For derogatory meaning, it is initialized to [0,1,0], if it is neutral, it is initialized to [0,0,1]; for words that are not manually labeled in 1), the corresponding line is initialized to [0,0,0] ;

3) Define the probability transition matrix T such that

4) Spread emotional labels: Y=TY

5) Re-initialize the label probability distribution of the manually labeled data in the label matrix Y according to the initialization method in 2)

6) If the label matrix Y converges, stop the iteration, otherwise go to step 4).

Step 9 is to generate emotional words, that is, according to the result of the label propagation algorithm in step 8, the words in the vocabulary V' are divided into positive, derogatory and neutral according to the probability in the label matrix Y;

Step 10 is the end.

Figure 2 is a structural diagram of the word-level supervision model in step 4, and its specific structure and settings are as follows:

1) Set the word vector dimension to 100 and the size of the context window to 3; initialize the word vector matrix W, whose size is |V|×100, and the i-th row represents the word vector of the i-th word in the vocabulary V;

2) In the input layer, the word _wt is selected from the text data set D as input, and its representation is the one-hot representation of _wt ;

3) In the projection layer, the vector form C(w _t ) of w _t is output from the word vector matrix W;

4) In the output layer, use C(w _t ) to predict the context of word wt {w _tk ,...,w _{t -1} ,w _t+1 ,...,w _t+k }. Its loss function is recorded as: loss _context (w _t )=-∑ _{-k≤j≤k,j≠0} logp(w _t+k |w _t );

其损失函数记为：

5) In the output layer, use C(w _t ) to predict the sentiment tag of word _wt by softmax function

Its loss function is recorded as:

6) End.

Figure 3 is a structural diagram of the text-level supervision model in step 5, and its specific structure and settings are as follows:

1) Set the word vector dimension m to 100, and initialize the word vector matrix W. Set the maximum text length input by the model to L according to the length of the text in the text dataset D;

2) In the input layer, select a piece of text d _i from the text dataset D. The word vector of each word in the text d _i is extracted from the word vector matrix W, connected to each other to form a two-dimensional matrix of size L×m, and the matrix is used as the model input;

3) In the convolution layer, use 200 filters to perform convolution operations to obtain feature vectors;

4) In the pooling layer, use the Max Pooling Over Time operation to extract the most important features from the feature vector as output;

其损失函数为：5) In the fully-connected layer, a fully-connected neural network layer is used, and the sentiment tag of the input text d _i is predicted by the softmax function

Its loss function is:

6) End.

To sum up, the present invention is based on supervised corpus, uses neural network to generate word vectors with emotional embedding, then mines the inner connection between words, uses label propagation algorithm to spread emotional labels, and automatically builds sentiment dictionary in specific fields . The invention not only avoids the deficiency that the knowledge base-based sentiment dictionary construction method cannot be used for sentiment analysis in a specific field, but also strengthens the consideration of the complex relationship of words in this article compared with other corpus-based methods. Finally, the automatic construction of emotion dictionary is realized.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (5)

1. A method for constructing an emotion dictionary based on supervised emotion text and word vectors is characterized by comprising the following steps:

(1) acquiring a text data set D, wherein the text data set D comprises a positive emotion text with a positive emotion mark and a negative emotion text with a negative emotion mark;

(2) preprocessing texts in the text data set; constructing a vocabulary V, and filling words appearing for the first time in the preprocessed text data set into the vocabulary V one by one;

(3) calculating the emotional tendency value of each word in the vocabulary V by adopting an SO-PMI method, and determining the emotional mark of the corresponding word according to the emotional tendency value:

wherein, table_wAn emotional tag representing the word w, SO-PMI (w) representing an emotional tendency value of the word w;

(4) constructing an improved skip-gram model with word level supervision, wherein the improved skip-gram model takes the words in the D as input data and predicts the context and emotion marks of the words; loss function loss in computing a prediction context_contextAnd loss function loss in predicting emotion mark_word；

loss_contextAnd loss_wordAre respectively:

wherein, w_tMeaning term, w_t∈D，

Meaning word w_tThe sentiment mark of (2); { w_t-k，…，w_t-1，w_t+1，…，w_t+kIndicates the predicted set of context words, including the predicted word w_tThe first k words and the last k words; p (w)_t+j|w_t) Watch (A)Word w_t+jIs predicted as w_tProbability of context of (p (pos | w)_t) Denotes w_tProbability of being predicted to have positive emotion marker, p (neg | w)_t) Denotes w_tA probability of being predicted to have a negative sentiment marker;

(5) constructing a convolutional neural network model as a text-level supervision model, wherein the text-level supervision model takes a text in a text data set D as input data and predicts emotion marks of the text; calculating a loss function loss between the predicted emotion mark of the text and the actual emotion mark of the text_doc：

Wherein d is_iRepresenting text, d_i∈D；

Denotes d_iThe sentiment tag of (1); p (pos | d)_i) Denotes d_iProbability of being predicted to have positive emotion marker, p (neg | d)_i) Denotes d_iA probability of being predicted to have a negative sentiment marker;

(6) setting a joint loss function:

loss＝α₁·loss_context+α₂·loss_doc+α₃·loss_word

in the formula, alpha₁、α₂、α₃Are respectively loss_context、loss_doc、loss_wordThe weight coefficient of (a);

(7) text data set D and emotion mark table of words_wEmotion marking of text

Training a joint loss function by using a back propagation algorithm for inputting data to obtain a word vector with emotion embedding;

(8) constructing a word relation graph G according to the word vector with emotion embedding obtained in the step (7);

(9) selecting partial words in the word relation graph G as seed words, and marking emotion labels for the seed words, wherein the emotion labels comprise commendation, derogation and neutrality; and then, propagating the emotion labels of the seed words in the relational graph G by using a label propagation algorithm to generate an emotion dictionary.

2. The method as claimed in claim 1, wherein the calculation formula of the emotional tendency value is as follows:

where SO-PMI (w) represents an emotional tendency value of word w, pos represents positive emotion text, neg represents negative emotion text, p (w | pos) represents a probability that word w appears in the positive emotion text, and p (w | neg) represents a probability that word w appears in the negative emotion text.

3. The method as claimed in claim 2, wherein the modified skip-gram model with word-level supervision comprises an input layer, a projection layer and an output layer, wherein the input layer is a word w in a text data set D_tProjection layer will be the word w_tProjected as a word vector C (w)_t) The output layer is based on C (w)_t) Separately predict w_tContext and emotion markup of

4. The method as claimed in claim 3, wherein the text level supervision model comprises: an input layer, a convolution layer, a pooling layer and a full-link layer, wherein the input layer is a text in the text data set DThis d_i(ii) a From the text d, the convolutional layer is passed through a feature extractor_iExtracting a plurality of feature vectors and sending the feature vectors to a pooling layer; selecting the most important characteristic vector from the characteristic vectors by the Pooling layer through Max Pooling Over Time operation and outputting the most important characteristic vector to the full connection layer; the full-connection layer predicts an input text d through a softmax function according to the received feature vectors_iIs marked with emotion

5. The method as claimed in claim 4, wherein the step of constructing the word relationship graph G comprises the following steps:

1) extracting verbs, adjectives and adverbs in the vocabulary V to form a new vocabulary V';

2) constructing a word relation graph G, and taking words in V' as vertexes in G;

3) for each word w in V_iCalculating w_iAnd (4) selecting m words with the nearest Euclidean distances from all other words in the V' in the word vector space obtained in the step (7), and establishing w in the word relation graph G_iAnd the weight calculation formula of the edge between the m words is as follows:

wherein, w_ijThe expression w_iAnd w_jWeight of edges in between, x_i、x_jAre respectively a word w_iAnd w_jWord vector of (1), euclidean _ dis (x)_i，x_j) Denotes x_i、x_jThe Euclidean distance between; σ is a constant parameter for controlling w_ijTaking the value of (A);

then the word w_iThe weight of the edge with the other words than the m words is set to 0.

Images