CN105868185A - Part-of-speech-tagging-based dictionary construction method applied in shopping comment emotion analysis - Google Patents

Abstract

The invention discloses a part-of-speech-tagging-based dictionary construction method applied in shopping comment emotion analysis. The method comprises the steps of conducting pre-treatment on text data of a shopping comment, in other words, conducting segmentation and word segmentation on a comment text, filtering out words which are not used any more, and partitioning shopping domains; constructing a basic emotion dictionary and a network buzzword emotion dictionary; taking a shopping comment corpus as a data set, conducting part-of-speech tagging on the data set, extracting words with the part-of-speech as habitually used words, adverbs and adjectives as candidate words, selecting new emotion words as domain emotion words by calculating the PTF-IDF values of the candidate words, and adding the domain emotion words to a domain emotion dictionary. The domain emotion dictionary is combined with the basic emotion dictionary and the network buzzword emotion dictionary, emotional characteristic screening and extraction are conducted on the shopping comment, and the emotion classification of the shopping comment is studied. It is shown through experiments that the method is high in accuracy rate, free of limitation of shopping domains and more suitable for practical application.

Description

A kind of dictionary creation method based on part-of-speech tagging in comment sentiment analysis of doing shopping

Technical field

The present invention relates to process or the conversion art of natural language be applicable to the data processing method of specific function, especially It relates to a kind of dictionary creation method based on part-of-speech tagging in comment sentiment analysis of doing shopping.

Background technology

Flourish along with the Internet, the rise of ecommerce has attracted increasing user to start shopping on the web, Experience " staying indoors " and " thing is inexpensively beautiful " that shopping online is brought.Meanwhile, user is also by online business The commodity bought are commented on by city, express the subjective opinion to commodity and suggestion.But, owing to shopping online does not has The restriction of region, also causes user directly to touch while offering convenience for user and specifically understands the matter of commodity Amount, is likely to result in online shopping mall and has any different the description of commodity with actual, make troubles for user.User only passes through Understand and purchased client's comment to commodity, make relatively reliable decision-making.Therefore, in order to enable users to convenient soon Prompt finds out rich valuable comment, browses the information oneself wanted, in order to promote that user improves to shopping online in businessman The satisfaction of service, carries out emotional semantic classification to shopping comment and just seems particularly significant.

Shopping comment is carried out emotional semantic classification, it is simply that be analyzed according to the Sentiment orientation expressed by comment text, process, Conclude and reasoning, it determines viewpoint that in comment, user is intended by, like, experience and to commodity or merchant service Attitude, and then provide the user more efficient and more reliably merchandise news, auxiliary user makes rational decision-making, Improve shopping online efficiency and service quality.At present, shopping is mainly commented by the research for comment text emotional semantic classification Opinion is divided into two classes, the i.e. comment of forward emotion and negative sense emotion to comment on.Also having part research is to be classified as three classes, the most just To emotion comment, the comment of negative sense emotion and neutral emotion comment.The emotional semantic classification research of shopping comment belongs to text emotion One branch of sort research.

At present, text emotion analysis is as natural language processing (Natural Language Processing, NLP) field The research direction of middle hot topic, has caused the widely studied analysis of scholar.Comment text is being carried out emotional orientation analysis Aspect, the technology that research uses both at home and abroad is broadly divided into following two big classes: one is method based on machine learning, and two are Based on sentiment dictionary or the method for semantic knowledge.Wherein, the method using sentiment dictionary is by by sentiment dictionary Search and add up positive emotion word and negative emotion word in comment text to be sorted and be used as the Main Basis that emotion differentiates, The emotion tendency of decision-making comment text to be sorted is i.e. carried out according to emotion word, if income value is canonical is judged to positive emotion, Otherwise it is negative emotion for negative, if income value is equal to zero, is considered as neutral emotion.Method based on sentiment dictionary can take Obtain preferable classification accuracy, but it be limited in that and depend on existing dictionary too much, it is impossible to identify and be not logged in word, Just cannot judge that the emotion of the text is inclined to and emotion intensity once text does not exist the word in sentiment dictionary.

Using the method for machine learning is then to be corpus and testing material by corpus labeling, by use maximum entropy, The grader such as support vector machine, naive Bayesian carries out emotion tendency classification to comment text.The method uses Algorithm complex is higher, needs the most suitable and corpus of tape label when training affective characteristics grader.

Publication No. CN104731923A, the sending out of entitled construction method of body dictionary " the Internet comment on commodity excavate " The deficiency that bright patent exists is mainly: one, is provided without the general vocabulary that disables, but by feature in experiment with computing data Frequency and document frequency, the peek high word of value, as stop words, is easily generated deviation in this process, loses and have feelings The word of sense tendency, impact experiment；Two, during it carries out dictionary structure, do not consider to remove name from the rolls outside word, other The impact that comment on commodity is analyzed by part of speech word.

Publication No. CN103207855A, entitled " fine granularity sentiment analysis system and the side for product review information Method " the deficiency that exists of patent of invention predominantly: one, need this sentiment analysis system of text training of a large amount of bands mark, and Periodically to be updated, to add a large amount of manpower and temporal consumption；Two, do not consider stop words, network flow lang Impact on sentiment analysis；Three, excessively rely on the matched combined dictionary in data base, make calculating process complicated, and also Do not consider different part of speech word Sentiment orientation in comment text.

In sum, the existing emotional semantic classification research towards shopping comment, its accuracy differentiated is insufficient for reality The demand of border application.

Summary of the invention

The technical problem to be solved be combine the most efficiently basis sentiment dictionary and machine learning shopping is commented The emotion of opinion effectively divides, so that final classification results has high-accuracy, and is not limited by shopping area, It is more suitable for actual application.

In order to solve above-mentioned technical problem, the present invention proposes word based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping Allusion quotation construction method, comprises the steps:

Step 1: shopping comment text is carried out data prediction；

Step 2: build basis sentiment dictionary；

Step 3: build network flow lang sentiment dictionary.

Step 4: use PTF-IDF (Part of speech Tag Frequency-Inverse Document Frequency) Method extracts the affective characteristics of shopping comment data collection, builds field sentiment dictionary；

Step 5: utilize described field sentiment dictionary, basis sentiment dictionary and network flow lang sentiment dictionary, shopping is commented Opinion carries out emotional semantic classification.

Further, above-mentioned data prediction includes the segmentation of comment text, participle, filtration stop words.

And, the segmentation of comment text, participle, filtration stop words specifically include following steps:

Step 1: read every comment, using Jieba participle instrument is independent word by described comment cutting；

Step 2: the word use after cutting is disabled vocabulary and filters.

Further, above-mentioned network flow lang sentiment dictionary be from representative large-scale Chinese website (such as from Sohu, Netease, Sina and Tengxun) in manually extracted several and use frequencies higher and containing the network of more apparent emotion tendency Popular word, network consisting popular word sentiment dictionary.

Further, the structure of aforementioned base sentiment dictionary, specifically include following steps:

Step 1: from existing representative sentiment dictionary, picks out and comprises the word of " " " " " obtaining " suffix with front The word in face merges, and is removed from it ambiguity or the word being of little use, composition candidate basis sentiment dictionary；

Step 2: each word in the sentiment dictionary of candidate basis utilizes threshold method according to from a search engine return Touching quantity sorts from big to small, removes the word that touching quantity is relatively low, composition basis sentiment dictionary.

Further, aforementioned use PTF-IDF method extracts the affective characteristics of shopping comment data collection, specifically includes as follows Step:

Step 1: use part-of-speech tagging (Part of Speech Tag) method, in extracting comment text part of speech for commonly use word, Adverbial word, adjectival word are as candidate word；

Step 2: calculate the PTF-IDF value of described each candidate word, IDF represents that described candidate word is at shopping comment language material Reverse document-frequency in storehouse, specific formula for calculation is as follows:

${\mathrm{PTF}}_{x,j}=\frac{{\mathrm{word}}_{x,j}}{{\mathrm{\Σ}}_k{\mathrm{word}}_{k,j}}$

Wherein, PTF_x,jSelected, after representing part-of-speech tagging, candidate word word obtained_xIn shopping comment corpus j Word frequency；Molecule word_x,_jRepresent candidate word word_xOccurrence number in comment corpus j, denominatorRepresent The sum that in corpus j, all words occur is commented on after participle；

${\mathrm{IDF}}_x={\log }_2\frac{\left|\mathrm{Re}views\right|}{\left|\left\{j:{\mathrm{word}}_x\∈{\mathrm{Review}}_j\right\}\right|+1}$

Wherein, IDF_xRepresent described candidate word word_xReverse document-frequency in shopping comment corpus j, molecule | Reviews | represents the sum of comment in comment corpus j, denominator | { j:word_x∈Review_j| represent comment language material Storehouse j comprises candidate word word_xComment number, if candidate word word_xIt is not present in commenting in corpus, then can The denominator causing this formula is zero, goes wrong to prevent from affecting calculating, and the method using denominator+1 finally calculates:

PTF-IDF(word_x)=PTF (word_x)×IDF(word_x)

Obtain the PTF-IDF value of each candidate word；

Step 3: in the field of described comment, uses threshold method to choose according to PTF-IDF value and makees more than the candidate word of threshold value For candidate field emotion word；

Step 4: calculate in number of times and the negative reviews language material of above-mentioned candidate field emotion word appearance in front comment language material Difference between the number of times occurred, if this difference is just, then candidate field emotion word is field, front emotion word；Otherwise, If this difference is negative, then candidate field emotion word is negative field emotion word；If difference is zero, then these candidate field feelings Sense word does not have Sentiment orientation, is added without field sentiment dictionary.

As preferably, when above-mentioned employing threshold method chooses the candidate word more than threshold value, threshold value selects 0.005.

Further, as the emotional semantic classification of last step in dictionary creation method based on part-of-speech tagging, specifically include Following steps:

Step 1: read every comment, utilizes art sentiment dictionary, basis sentiment dictionary and network flow lang emotion Affective characteristics chosen by dictionary；

Step 2: calculate every comment in positive emotion characteristic weighing and, negative emotion characteristic weighing and；

Step 3: calculate this comment positive emotion characteristic weighing and with negative emotion characteristic weighing and difference；

Step 3: if difference is just, then this comment belongs to positive emotion；If difference is negative, then this comment belongs to negative Face emotion；If zero, then this comment belongs to neutral, and specific formula for calculation is as follows:

Negative emotion feature in positive emotion this comment of feature-Σ in this comment of Sentiment orientation=Σ of shopping comment.

Compared with prior art, there is advantages that

1, dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping that the present invention proposes, effectively carries The high accuracy rate of shopping comment emotional semantic classification；

2, the proposed by the invention field sentiment dictionary built based on part-of-speech tagging according to the comment of Related shopper field, The accuracy rate obtaining comment classification is apparently higher than the accuracy rate using basis sentiment dictionary to reach.

Accompanying drawing explanation

Fig. 1 is the schematic flow sheet of one embodiment of the present of invention.

Fig. 2 is the idiographic flow schematic diagram of step B in Fig. 1.

Fig. 3 is to combine field sentiment dictionary, basis sentiment dictionary and network flow lang emotion in method proposed by the invention Dictionary, with comparative test result figure based on tradition sentiment dictionary.

Fig. 4 is the test result exemplary plot of three shopping area.

Detailed description of the invention

Below in conjunction with the accompanying drawings and the present invention is embodied as being described in further detail by embodiment.Institute in the present invention The embodiment described, the only a part of embodiment of the present invention rather than whole embodiments.

In a kind of comment sentiment analysis of doing shopping that Fig. 1 is that the present invention proposes the one of dictionary creation method based on part-of-speech tagging The schematic flow sheet of individual embodiment, comprises the steps:

A, respectively to hotel, books, PC domain shopping comment data carry out pretreatment, including comment segmentation, participle, Filter stop words.

Concrete, as in figure 2 it is shown, step A includes step:

A1, reading every comment, using Jieba participle instrument is independent word by described comment cutting；

A2, the word after cutting is used disable vocabulary and filter；

B, structure basis sentiment dictionary.

B1, from existing " knowing net " sentiment dictionary, delete ambiguity and the word being of little use, pick out and comprise " " " " word of " obtaining " suffix merges with word above, and be removed from it ambiguity or the word being of little use, composition Candidate basis sentiment dictionary；

B2, each word in the sentiment dictionary of candidate basis utilize Hits value that threshold method returns according to Google from greatly To little sequence, remove the word that Hits value is relatively low, composition basis sentiment dictionary；

C, structure network flow lang sentiment dictionary.

C1, artificial from the most popular large-scale Chinese microblogging site for service of Sohu, Netease, Sina and Tengxun these four Extract about 50 use frequency higher and contain the network flow lang of more apparent emotion tendency, network consisting popular word Sentiment dictionary.

D, build field sentiment dictionary towards different field.

Concrete, step D includes step:

D1, respectively read hotel, books, PC domain shopping comment corpus in every comment, to participle, stop Word filter after word, use part-of-speech tagging method, in extracting comment, part of speech is for commonly using word, adverbial word, adjectival Word is as candidate word；

Wherein, shopping comment specifically includes the word of following 28 kinds of parts of speech, as shown in table 1 after part-of-speech tagging.

Table 1 Chinese part-of-speech tagging

D2, commonly use word, adverbial word, the candidate word of adjective part of speech according to having of being extracted, calculate described each candidate The PTF-IDF value of word, wherein PTF-IDF=PTF*IDF, in formula, PTF represents that described candidate word is after part-of-speech tagging Occurrence number in this field shopping comment corpus, IDF represents that described candidate word is in shopping comment corpus Reverse document-frequency；

D3, calculate described shopping area comment on after participle, all words occur sum；Calculate described each candidate The number of times that word occurs in this field is commented on；The PTF value of each candidate word is obtained by formula (1) after being calculated；

D4, calculate in described shopping area the sum of comment；Calculate the comment comprising each candidate word in this field is commented on Number, obtains the IDF value of each candidate word by formula (2) after being calculated；

D5, calculated by formula (3) after obtain the PTF-IDF value of each candidate word；

D6, in the field of described comment, use threshold method to choose qualified according to the PTF-IDF value of each candidate word Candidate word is as field emotion word, composition field sentiment dictionary.

D7, the number of times calculating the appearance in front comment language material of candidate field emotion word and negative reviews language material occur Difference between number of times, if this difference is just, then candidate field emotion word is field, front emotion word；Otherwise, if should Difference is negative, then candidate field emotion word is negative field emotion word；If difference is zero, then this candidate field emotion word Not there is Sentiment orientation, be added without field sentiment dictionary.

E, utilize described field sentiment dictionary, in conjunction with basis sentiment dictionary, network flow lang sentiment dictionary, shopping is commented Opinion carries out emotional semantic classification.

Concrete, step E includes step:

E1, read every comment, utilize described field sentiment dictionary, basis sentiment dictionary and network flow lang emotion word Allusion quotation chooses affective characteristics；

E2, calculate every comment in positive emotion characteristic weighing and, negative emotion characteristic weighing and；

E3, calculate this comment positive emotion characteristic weighing and with negative emotion characteristic weighing and difference；

If E4 difference is just, then this comment belongs to positive emotion；If difference is negative, then this comment belongs to negative Emotion；If zero, then this comment belongs to neutral.

Present invention python is compiled test, the accuracy rate obtained and sorting algorithm based on tradition sentiment dictionary Contrasting, experimental result is as shown in table 2.

Table 2 experimental result

Field	Algorithm based on basis sentiment dictionary	The algorithm of the present invention
			Hotel	0.631	0.894
Books	0.641	0.851
			Computer	0.655	0.852

Obviously, classifying quality based on inventive algorithm is significantly better than classifying quality based on tradition sentiment dictionary, at three necks Territory: hotel, books, PC domain, has been respectively increased 26.3%, and 21%, 19.7%.

As shown in Figure 4, by described three fields, i.e. hotel, books, PC domain, it is subdivided into six class comments, its Middle transverse axis represents the field of shopping comment, longitudinal axis presentation class accuracy rate.By result it can be shown that the present invention proposes A kind of do shopping comment sentiment analysis in dictionary creation method based on part-of-speech tagging to shopping comment emotional semantic classification can obtain Preferably effect.

It should be noted that embodiment provided by the present invention only has schematically, the method illustrated in embodiment is also Can be realized by other compiling mode.Such as, the division in described shopping comment field, is only a kind of logic-based The division of function, can have other dividing mode during reality realizes；Can also in specific implementation process, In conjunction with the multiple steps in the present invention, some feature is ignored or does not performs.

Claims (8)

1. dictionary creation method based on part-of-speech tagging in a comment sentiment analysis of doing shopping, it is characterised in that comprise the steps:

Step 1: shopping comment text is carried out data prediction；

Step 2: build basis sentiment dictionary；

Step 3: build network flow lang sentiment dictionary；

Step 4: use PTF-IDF method to extract the affective characteristics of shopping comment data collection, build field sentiment dictionary；

Step 5: utilize described field sentiment dictionary, basis sentiment dictionary and network flow lang sentiment dictionary, shopping comment is carried out Emotional semantic classification.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 1, its feature exists The segmentation of comment text, participle, filtration stop words is included in described data prediction.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 2, its feature exists Following steps are specifically included in, the segmentation of described comment text, participle, filtration stop words:

Step 1: read every comment, using Jieba participle instrument is independent word by described comment cutting；

Step 2: the word use after cutting is disabled vocabulary and filters.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 1, its feature exists It is from representative large-scale Chinese website, manually to have extracted several use frequency relatively in described network flow lang sentiment dictionary Height the network flow lang containing more apparent emotion tendency, network consisting popular word sentiment dictionary.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 1, its feature exists In, the structure of described basis sentiment dictionary, specifically include following steps:

Step 1: from existing representative sentiment dictionary, picks out and comprises word and the word above of " " " " " obtain " suffix Language merges, and is removed from it ambiguity or the word being of little use, composition candidate basis sentiment dictionary；

Step 2: each word in the sentiment dictionary of candidate basis utilizes threshold method according to the hits returned from a search engine Amount sorts from big to small, removes the word that touching quantity is relatively low, composition basis sentiment dictionary.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 1, its feature exists Extract the affective characteristics of shopping comment data collection in, described use PTF-IDF method, specifically include following steps:

Step 1: using part-of-speech tagging method, in extracting comment text, part of speech is for commonly using word, adverbial word, adjectival word as time Select word；

Step 2: calculate the PTF-IDF value of described each candidate word, IDF represents that described candidate word is in shopping comment corpus Reverse document-frequency, specific formula for calculation is as follows:

${\mathrm{PTF}}_{x,j}=\frac{{\mathrm{word}}_{x,j}}{{\mathrm{\Σ}}_k{\mathrm{word}}_{k,j}}$

Wherein, PTF_x,jSelected, after representing part-of-speech tagging, candidate word word obtained_xWord frequency in shopping comment corpus j； Molecule word_x,_jRepresent candidate word word_xOccurrence number in comment corpus j, denominatorRepresent comments after participle The sum that in material storehouse j, all words occur；

${\mathrm{IDF}}_x={\log }_2\frac{\left|\mathrm{Re}views\right|}{\left|\right\{j:{\mathrm{word}}_x\∈{\mathrm{Review}}_j\left\}\right|+1}$

Wherein, IDF_xRepresent described candidate word word_xReverse document-frequency in shopping comment corpus j, molecule | Reviews | represents the sum of comment in comment corpus j, denominator | { j:word_x∈Review_j| represent in comment corpus j Comprise candidate word word_xComment number, if candidate word word_xIt is not present in commenting in corpus, then can cause this formula Denominator be zero, go wrong to prevent from affecting calculating, the method using denominator+1, finally calculate:

PTF-IDF(word_x)=PTF (word_x)×IDF(word_x)

Obtain the PTF-IDF value of each candidate word；

Step 3: in the field of described comment, uses threshold method to choose the candidate word more than threshold value as candidate according to PTF-IDF value Field emotion word；

Step 4: calculate number of times and appearance in negative reviews language material that above-mentioned candidate field emotion word occurs in front comment language material Difference between number of times, if this difference is just, then candidate field emotion word is field, front emotion word；Otherwise, if this difference is Negative, then candidate field emotion word is negative field emotion word；If difference is zero, then this candidate field emotion word does not have emotion and inclines To, it is added without field sentiment dictionary.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 6, its feature exists It is 0.005 in described threshold value.

Dictionary creation method based on part-of-speech tagging in a kind of comment sentiment analysis of doing shopping the most according to claim 1, its feature exists In, described emotional semantic classification, specifically include following steps:

Step 1: read every comment, utilizes art sentiment dictionary, basis sentiment dictionary to select with network flow lang sentiment dictionary Take affective characteristics；

Step 2: calculate every comment in positive emotion characteristic weighing and, negative emotion characteristic weighing and；

Step 3: calculate this comment positive emotion characteristic weighing and with negative emotion characteristic weighing and difference；

Step 3: if difference is just, then this comment belongs to positive emotion；If difference is negative, then this comment belongs to negative emotion； If zero, then this comment belongs to neutral, and specific formula for calculation is as follows:

Negative emotion feature in positive emotion this comment of feature-Σ in this comment of Sentiment orientation=Σ of shopping comment.