CN103593454A - Mining method and system for microblog text classification - Google Patents

Abstract

The invention relates to a mining method for microblog text classification, comprising the following steps: acquiring existing microblog data; analyzing and preprocessing the acquired microblog text; searching for a term set of the microblog text Traversing, removing stop word terms; doing development test CHI value calculation for each term in the original feature term set, and obtaining the N terms with the highest value as the feature term set, the original feature term The collection is the collection of terms of all microblog texts; the N terms are carried out for association rule mining, and the strongly associated terms of the feature terms in the microblog text are added to the feature term set of this microblog to improve the microblog Blog text classification accuracy. The invention also relates to a mining system oriented to microblog text classification. The invention can effectively simplify the mining complexity of association rules of original microblog texts, greatly reduce the amount of data to be analyzed, and improve the classification accuracy of microblog texts.

Description

Mining method and system for microblog text classification

technical field

The invention relates to a mining method and system for microblog text classification.

Background technique

Weibo has become one of the important platforms and media for people to socialize. There are more than 400 million Weibo users in China, while Twitter users are more than 500 million, and the daily sending volume of information exceeds 200 million. Facebook's second largest social networking site. In recent years, Weibo has become the birthplace of countless hot topics and trends. With the popularity of social networking sites such as Sina Weibo and Tencent Weibo in China, social media such as Weibo has not only become a platform for netizens to publish, share, and disseminate information, but also has accumulated large-scale Internet user behavior data. In May 2012, Lu Yi, deputy general manager of Sina Weibo Business Department, pointed out that Sina Weibo has more than 300 million registered users, of which 60% active users log in through mobile terminals, and users publish more than 100 million Weibo content every day on average . It can be seen that the amount of microblog data is increasing, so the mining of microblog data is feasible, innovative and practical, and has attracted extensive attention from domestic and foreign academic circles.

In microblog text classification, association rules can effectively improve the classification accuracy. Among them, the support of association rules in the data set (support) is the percentage of items in the data set that contain both X items and Y items, that is, the probability; the confidence (confidence) is the case that the items in the data set already contain X items, including Y items The percentage of is the conditional probability. If the minimum support threshold and the minimum confidence threshold are met. These thresholds are artificially set according to mining needs.

There are two main types of existing association rule algorithms: Apriori algorithm and FP-tree frequency set algorithm.

Apriori Algorithm: First find all frequency sets whose frequency of occurrence is at least the same as the predefined minimum support. Then strong association rules are generated from the frequency set, and these rules must satisfy the minimum support and minimum confidence. Then use the found frequency set to generate the desired rules, generating all rules that contain only the items of the set, where the right part of each rule has only one item. Once these rules are generated, only those rules that are greater than the minimum confidence given by the user are left, and all frequency sets are generated using a recursive method.

FP-Tree Frequency Set Algorithm: Using a divide-and-conquer strategy, after the first pass of scanning, the frequency set in the database is compressed into a frequent pattern tree (FP-tree), while still retaining the associated information, and then the FP-tree is divided into some conditional libraries, each library is related to a frequency set with a length of 1, and then these conditional libraries are mined separately. When the amount of original data is large, the division method can also be combined so that an FP-tree can be placed in the main memory. Experiments show that FP-growth has good adaptability to rules of different lengths, and has a huge improvement in efficiency compared with Apriori algorithm.

However, for short texts such as Weibo, the Apriori algorithm generates a large number of candidate sets, and may need to scan the database repeatedly, which greatly increases the mining complexity and mining time. Although the FP-tree frequency set algorithm can effectively improve the efficiency, it is still not efficient for short texts.

Contents of the invention

In view of this, it is necessary to provide a mining method and system for microblog text classification.

The present invention provides a mining method for microblog text classification. The method includes the following steps: a. obtaining existing microblog data; b. analyzing and preprocessing the obtained microblog text; c. The term set of the blog text is searched and traversed to remove the stop word term; d. The development test CHI value is calculated for each term in the original feature term set, and the N terms with the highest value obtained are used as features Term set, the original characteristic term set is the term set of all microblog texts; e. carry out association rule mining to described N term, the strongly associated term of the characteristic term in the microblog text is added to The characteristic word items of the microblog are concentrated to improve the classification accuracy of the microblog text.

Wherein, the microblog data includes: user ID, user name, and microblog text.

The step b includes removing special symbols such as punctuation marks, removing non-Chinese characters and word segmentation operations on the microblog text to obtain a set of words in the microblog text, and manually classifying the microblog.

The feature term sets are arranged according to the mutual information value, where N is user-defined, and N is less than the total number of terms.

The calculation method of the CHI value of the development test is: for each word, it is calculated separately: the number a of microblog texts containing this word under this classification; the number b of microblog texts not containing this word under this classification; The number c of Weibo texts that do not contain this word; the number d of Weibo texts that are not under this category and do not contain this word; z1=a*d-b*c; CHI=(z1*z1*float(N))/( (a+c)*(a+b)*(b+d)*(c+d).

Described step e comprises: traversing each microblog in the microblog data that obtains, carry out binary grouping to the feature word item set of each microblog; Set the threshold value of support degree and confidence degree; According to the set The threshold of support and confidence is based on the strong association rules, and the strongly associated terms of the feature terms in the microblog text are added to the feature term set of the microblog.

The present invention also provides a microblog text classification-oriented mining system, including an acquisition module, a preprocessing module, an extraction module, a calculation module, and a mining module electrically connected to each other, wherein: the acquisition module is used to acquire the existing microblog blog data; the preprocessing module is used to analyze and preprocess the obtained microblog text; the extraction module is used to search and traverse the term set of the microblog text, and remove stop word terms; The calculation module is used to calculate the development and inspection CHI value of each term in the original feature term set, and the N terms with the highest value obtained are used as the feature term set, and the original feature term set is all micro A set of terms in the blog text; the mining module is used to mine the N terms for association rules, and add strongly associated terms of the feature terms in the microblog text to the set of feature terms in the microblog to Improve the accuracy of microblog text classification.

Wherein, the microblog data includes: user ID, user name, and microblog text.

The preprocessing module is used to remove special symbols such as punctuation marks, non-Chinese characters and word segmentation operations on the microblog text to obtain a term set of the microblog text.

The feature term sets are arranged according to the mutual information value, where N is user-defined, and N is less than the total number of terms.

The invention is oriented to the mining method and system of microblog text classification, comprehensively considers the text structure of microblog, aims at the characteristics of short text of microblog text and the necessity of association rules of microblog text, and proposes a simple and effective method for microblog The association rule mining method of text classification, compared with the previous association rule mining method, the time complexity of the present invention is greatly reduced, the amount of data to be analyzed is greatly reduced, and the microblog text classification accuracy is significantly improved.

Description of drawings

Fig. 1 is the flow chart of the mining method facing microblog text classification in the present invention;

FIG. 2 is a hardware architecture diagram of the microblog text classification-oriented mining system of the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

Referring to FIG. 1 , it is a flow chart of a preferred embodiment of the mining method for microblog text classification in the present invention.

Step S401, acquiring existing microblog data. Specifically, obtain the existing data on the microblogging website. Limited by the analysis technology, this embodiment only acquires microblog data whose content is in Chinese. The microblog data includes: user ID, user name, and microblog text.

Step S402, analyzing and preprocessing the acquired microblog text. Specifically, each microblog text is initialized. After the microblog text is removed from special symbols such as punctuation marks, non-Chinese characters and word segmentation operations, the term set of the microblog text is obtained, and the Weibo is manually classified.

Step S403, performing feature extraction on the microblog text, that is, searching and traversing the term set of the microblog text to remove stop word terms.

Step S404, performing feature selection on the microblog data. Specifically, the development test CHI value is calculated for each term in the original feature term set, and the N terms with the highest values obtained are used as the feature term set. Wherein, the original characteristic term set is a term set of all microblog texts. The feature term sets are arranged according to the mutual information value, where N is user-defined, and N is less than the total number of terms.

The CHI value calculation method for the development test is as follows:

Calculated separately for each word: the number of microblog texts that contain this word under this category a; the number of microblog texts that do not contain this word under this category b; the number of microblog texts that do not contain this word under this category c ; The number d of Weibo texts that are not under this category and do not contain this word.

z1=a*d-b*c.

CHI=(z1*z1*float(N))/((a+c)*(a+b)*(b+d)*(c+d).

Step S405, performing association rule mining on the N terms. Specific steps are as follows:

1. Traverse each microblog in the obtained microblog data, perform binary grouping on the feature term set of each microblog, and add each binary group to MAP<(term x, term y) , count>, count is the number of occurrences of the binary group.

2. The number of occurrences of each term has been calculated in the process of selecting features, and the thresholds of support and confidence are set.

21. Filter the total number of microblogs whose count is less than the microblog data * the binary group with support set;

22. support(x=>y)=count/the total number of microblog data of microblog data;

23. confidence(x=>y)=count/(a+b).

3. According to the support and confidence thresholds set above, strong association rules are taken. The strongly associated terms of the feature terms in the microblog text are added to the feature term set of the microblog to improve the classification accuracy of the microblog text.

Referring to FIG. 2 , it is a hardware architecture diagram of the microblog text classification-oriented mining system of the present invention. The system includes an acquisition module, a preprocessing module, an extraction module, a calculation module and an excavation module electrically connected to each other.

The obtaining module is used to obtain existing microblog data. Specifically, the obtaining module obtains existing data on the microblog website. Limited by the analysis technology, this embodiment only acquires microblog data whose content is in Chinese. The microblog data includes: user ID, user name, and microblog text.

The processing module is used to perform denoising and enhancement pre-processing on the acquired images to prepare for later processing and screening. Specifically, the processing module respectively performs denoising processing and enhancement processing on the acquired image, so as to improve the recognition degree of the image.

The preprocessing module is used for analyzing and preprocessing the acquired microblog text. Specifically, the preprocessing module initializes each piece of microblog text, and after the microblog text removes special symbols such as punctuation marks, removes non-Chinese characters, and performs word segmentation operations, the word items of the microblog text are obtained Collect and manually classify the microblogs.

The extraction module is used to perform feature extraction on the microblog text, that is, the extraction module searches and traverses the term set of the microblog text to remove stop word terms.

The calculation module is used for feature selection of microblog data. Specifically, the calculation module calculates the development test CHI value for each term in the original feature term set, and the obtained N terms with the highest values are used as the feature term set. Wherein, the original characteristic term set is a term set of all microblog texts. The feature term sets are arranged according to the mutual information value, where N is user-defined, and N is less than the total number of terms.

The calculation module calculates and obtains the CHI value of the development inspection as follows:

Calculated separately for each word: the number of microblog texts that contain this word under this category a; the number of microblog texts that do not contain this word under this category b; the number of microblog texts that do not contain this word under this category c ; The number d of Weibo texts that are not under this category and do not contain this word.

z1=a*d-b*c.

CHI=(z1*z1*float(N))/((a+c)*(a+b)*(b+d)*(c+d).

The mining module is used to mine association rules for the N terms. details as follows:

Described mining module first traverses each microblog in the microblog data that obtains, carries out binary grouping to the feature word item set of each microblog, joins each binary group to MAP<(term x, word Item y), count>, count is the number of occurrences of the binary group.

Then, in the feature selection process, the number of occurrences of each term has been calculated, and the threshold of support and confidence is set: filter the total number of microblogs whose count is less than the microblog data * the binary group that has set support; support(x= >y)=count/the total number of microblogs in the microblog data; confidence(x=>y)=count/(a+b).

Finally, according to the support and confidence thresholds set above, strong association rules are taken. The strongly associated terms of the feature terms in the microblog text are added to the feature term set of the microblog to improve the classification accuracy of the microblog text.

Although the present invention has been described with reference to the current preferred embodiments, those skilled in the art should understand that the above-mentioned preferred embodiments are only used to illustrate the present invention, and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of principles shall be included in the protection scope of the present invention.

Claims (10)

1. towards a method for digging for microblogging text classification, it is characterized in that, the method comprises the steps:

A. obtain existing microblogging data;

B. the microblogging text obtaining is analyzed and pre-service;

C. the lexical item set of described microblogging text is carried out to search spread, remove stop words lexical item;

D. each lexical item in the set of primitive character lexical item is done to exploitation check CHI value and calculate, the N of a drawn mxm. lexical item is as feature lexical item collection, and the set of described primitive character lexical item is the lexical item set of all microblogging texts;

E. a described N lexical item is carried out to association rule mining, the feature lexical item that the strong associated lexical item of the feature lexical item in microblogging text is joined to this microblogging is concentrated, to improve microblogging text classification precision.

2. the method for claim 1, is characterized in that, described microblogging data comprise: user ID, user name, microblogging text.

3. method as claimed in claim 2, it is characterized in that, described step b comprises special symbols such as described microblogging text removal punctuation marks, removes non-Chinese character and participle operation, obtains the lexical item set of described microblogging text, and this microblogging is carried out to manual sort.

4. method as claimed in claim 3, is characterized in that, described described feature lexical item collection is arranged according to the height of mutual information value, and wherein N is User Defined, and N is less than total lexical item number.

5. method as claimed in claim 4, is characterized in that, described exploitation check CHI value calculating method is:

For each word, calculate respectively: the microblogging amount of text a that comprises this word under this classification; The microblogging amount of text b that does not comprise this word under this classification; The microblogging amount of text c that does not comprise this word under this classification; Not under this classification, and do not comprise the microblogging amount of text d of this word;

z1=a*d-b*c；

CHI=(z1*z1*float(N))/((a+c)*(a+b)*(b+d)*(c+d)。

6. method as claimed in claim 5, is characterized in that, described step e comprises:

Every microblogging in the microblogging data that traversal is obtained, carries out two tuples to the feature lexical item collection of every microblogging;

Set the threshold value of support and degree of confidence;

According to the support of setting and the threshold value of degree of confidence, get Strong association rule, the feature lexical item that the strong associated lexical item of the feature lexical item in microblogging text is joined to this microblogging is concentrated.

7. towards a digging system for microblogging text classification, it is characterized in that, this system comprises acquisition module, pretreatment module, extraction module, computing module and the excavation module of mutual electric connection, wherein:

Described acquisition module is used for obtaining existing microblogging data;

Described pretreatment module is for analyzing and pre-service the microblogging text obtaining;

Described extraction module, for the lexical item set of described microblogging text is carried out to search spread, is removed stop words lexical item;

Described computing module calculates for each lexical item of primitive character lexical item set being done to exploitation check CHI value, and the N of a drawn mxm. lexical item is as feature lexical item collection, and the set of described primitive character lexical item is the lexical item set of all microblogging texts;

Described excavation module is for a described N lexical item is carried out to association rule mining, and the feature lexical item that the strong associated lexical item of the feature lexical item in microblogging text is joined to this microblogging is concentrated, to improve microblogging text classification precision.

8. system as claimed in claim 7, is characterized in that, described microblogging data comprise: user ID, user name, microblogging text.

9. system as claimed in claim 8, is characterized in that, described pretreatment module, for described microblogging text being removed to the special symbols such as punctuation mark, being removed non-Chinese character and participle operation, obtains the lexical item set of described microblogging text.

10. system as claimed in claim 9, is characterized in that, described described feature lexical item collection is arranged according to the height of mutual information value, and wherein N is User Defined, and N is less than total lexical item number.

Images