CN106294689B - A kind of method and apparatus for selecting to carry out dimensionality reduction based on text category feature - Google Patents

Abstract

The present invention provides a method and device for dimensionality reduction based on text class feature selection. The method includes the steps of: obtaining the text to be processed; using HanLP to perform word segmentation to obtain multiple word items, and removing stop words in the word items; Count word frequency, term document frequency, and document word count; store term, term frequency, term document frequency, and document word count to form primary text vectors; perform information gain calculation on primary text vectors, sort according to the size of information gain, Vocabulary that meets the preset requirements is formed into a reference vector for feature selection; the text to be processed is reduced in dimension according to the reference vector to form a reduced-dimensional text vector. The device includes: an acquisition module, a word segmentation module, a statistics module, a vector module, an information gain calculation module and a dimension reduction module. The method and device perform text feature selection based on an information gain algorithm, and perform dimensionality reduction processing on feature word set vectors, thereby reducing the calculation burden caused by excessive dimensionality.

Description

A method and device for dimensionality reduction based on text class feature selection

technical field

The present invention relates to the technical field of machine learning, in particular to a method and device for dimensionality reduction based on text class feature selection.

Background technique

With the rapid development of the Internet and the continuous innovation of Internet-related technologies, the cost and efficiency of the entire society's informatization have undergone tremendous changes compared with ten or twenty years ago. In addition, the increasing popularity of the Internet has produced data in many different formats (text, multimedia, etc.) and many different data sources. In the face of massive information, people can no longer simply rely on manual processing of all information resources, but need auxiliary tools To help people better discover, filter and manage these electronic information data and resources.

Traditional software related to text processing is all about processing text files. However, with the emergence of various text formats, files carrying electronic information are no longer limited to a single file type. Especially with the development of the Internet, these formats Text also shows its own advantages, and the limitations of the processing system for single-format files are becoming more and more obvious.

The representation of the text is abstracted into the space vector of the feature word set. However, the original candidate feature word set is as high as hundreds of thousands of dimensions, and the high-dimensional text representation causes a huge computational burden.

Contents of the invention

The present invention provides a method and device for dimensionality reduction based on text feature selection to solve the above technical problems.

A method for dimensionality reduction based on text class feature selection provided by the present invention comprises steps:

Step A, obtain and store the detailed information of the data source text to be processed;

Step B, using HanLP to perform word segmentation on the data source text to obtain a plurality of terms, and remove stop words in the terms;

Step C, counting word frequency, term document frequency and document word count;

Step D, storing the term, term frequency, term document frequency and document word count and forming a primary text vector;

Step E, performing information gain calculation on the primary text vector to obtain the information gain amount of each term, sorting according to the size of the information gain amount, and forming a reference vector for feature selection from a plurality of words that meet the preset requirements;

Step F, reducing the dimensionality of the text to be processed according to the reference vector to form a dimensionality-reduced text vector.

Wherein, performing information gain calculation in step E includes steps:

Take each text as a category, and use the terms in the text as features, and calculate the amount of information gain according to the following formula

Among them, N represents the total number of categories, P(C _i ) represents the probability of category C _i appearing, P(t) represents the probability of feature (T) appearing, Indicates the probability that the feature (T) does not appear, P(C _i |t), indicates the probability that the text contains the feature (T) and belongs to the category C _i .

Wherein, in step E, where DF _T represents the document frequency of the feature (T);

where TF _i represents the frequency of occurrence of each term;

The embodiment of the present invention also provides a device for feature selection of text class for dimensionality reduction, including an acquisition module, a word segmentation module, a statistics module, a vector module, an information gain calculation module, and a dimensionality reduction module;

The obtaining module is used to obtain and store the detailed information of the data source text to be processed;

A word segmentation module, configured to use HanLP to perform word segmentation on the data source text to obtain a plurality of terms, and remove stop words in the terms;

Statistical module, used to count term frequency (occurrence frequency of each term) and term document frequency and document word count;

The vector module is used to store term, term frequency, term document frequency and document word count and form primary text vector;

The information gain calculation module is used to calculate the information gain of the primary text vector, obtain the information gain amount of each term, sort according to the size of the information gain amount, and form a reference vector for feature selection by a plurality of words that meet the preset requirements;

The dimensionality reduction module is configured to perform dimensionality reduction on the text to be processed according to the reference vector to form a dimensionality-reduced text vector.

Wherein, the information gain calculation module is used for:

Take each text as a category, and use the terms in the text as features, and calculate the amount of information gain according to the following formula

Among them, N represents the total number of categories, P(C _i ) represents the probability of category C _i appearing, P(t) represents the probability of feature (T) appearing, Indicates the probability that the feature (T) does not appear, P(C _i |t), indicates the probability that the text contains the feature (T) and belongs to the category C _i ;

where DF _T represents the document frequency of the feature (T);

where TF _i represents the frequency of occurrence of each term;

The embodiment of the present invention provides a method and device for dimensionality reduction based on text-based feature selection. The reference vector is obtained through steps such as HanLP word segmentation, removal of stop words, calculation of information gain by using terms as features, and sorting according to the amount of information gain. , and then perform dimension reduction processing on the document according to the reference vector, which is a document feature dimension reduction processing method based on the information gain algorithm, which reduces the dimension of the document feature word set, and reduces the hundreds of thousands of dimensional feature word sets. Calculate the burden.

Description of drawings

Fig. 1 is a schematic flow chart of an embodiment of a method for dimensionality reduction based on text class feature selection in the present invention;

FIG. 2 is a schematic flowchart of an embodiment of text feature selection provided by Embodiment 2 of the present invention.

Detailed ways

The embodiment of the present invention provides a method and device for dimensionality reduction based on text feature selection, which is an algorithm for text feature selection based on Information Gain (IG), which can be representative by extracting text features. , the most effective features to reduce the dimensionality of the dataset. In information gain, the measure of importance is to see how much information a feature can bring to the classification system. The more information it brings, the more important the feature is.

The embodiment of the present invention adopts HanLP word segmentation technology to carry out word segmentation to the text, and its principle is to build a sufficiently large dictionary database containing all possible Chinese words, and judge whether the Chinese character string of the Chinese text to be processed appears in the dictionary database, once found, then A word is recognized, and the word is segmented from the Chinese character string until the Chinese character string is segmented. HanLP has the characteristics of complete functions, high performance, clear structure, up-to-date corpus, and customization. While providing rich functions, HanLP's internal modules insist on low coupling, models insist on lazy loading, services insist on static provision, and dictionaries insist on publishing in plain text. But its biggest disadvantage is that the execution accuracy is completely dependent on the dictionary library, which needs to be updated.

The information gain method is to determine the amount of information provided by the category to which the text belongs by judging whether the feature appears or does not appear in the text. In the filtering problem, it is used to measure how much information contribution to the topic prediction is given whether a feature appears in the relevant text of a certain topic. By calculating the information gain, the features that appear frequently in positive samples and low in non-positive samples can be obtained. Information gain involves more mathematical theories and complex entropy theoretical formulas. The embodiment of the present invention defines it as the amount of information that a feature item can provide for the entire classification, regardless of the entropy of any feature and the entropy after considering the feature. difference. According to the training data, the embodiment of the present invention calculates the information gain of each feature item, deletes items with a small information gain, and sorts and screens the rest according to the information gain from large to small.

Embodiment one

Specifically, as shown in Figure 1, the method includes steps:

In step S110, the detailed information of the data source text to be processed is acquired and stored.

Obtain the detailed information of the data source text and store it in HDFS, and keep the backup for subsequent inspection or data retrospective use.

Step S111 , using HanLP to segment the data source text to obtain a plurality of lexical items, and remove stop words in the lexical items.

The effective information of a text is generally composed of nouns, adjectives, verbs, quantifiers and other content words, which category it belongs to is also mainly distinguished by these content words, and there are some words that frequently appear in all texts and words that have no actual meaning. Function words contribute little to text classification. These stop words usually do not have great practical significance, but they often appear in the text. If they are not removed, two texts with completely different contents may not be able to be distinguished because of the large amount of common information, and it will affect the The subsequent feature selection stage increases the computational overhead of the system, which ultimately affects the construction of the classifier. Therefore, by disabling the thesaurus, after the text is segmented, the words existing in the thesaurus are directly filtered out.

Step S112, counting word frequency, term document frequency and document word count.

Use HanLP to segment the text, count the word frequency and term (Term) document frequency and the number of words in the document. Among them, the term frequency is the frequency of each term (Term) appearing in all texts, the term document frequency is the frequency of each term appearing in a document, and the document word count is the number of terms contained in a document .

Step S113, storing the term, term frequency, term document frequency and document word count to form a primary text vector.

Store the term, the number of times the word appears (word frequency), and the document frequency of the term in the memory database to form a vectorized text, which is ready for information gain calculation and reading.

Step S114, perform information gain calculation on the primary text vector to obtain the information gain amount of each term, sort according to the size of the information gain amount, and form a plurality of words that meet the preset requirements into a reference vector for feature selection.

Calculate the information gain of the text vectors, sort them according to the amount of information, reserve N words according to the requirements, as the reference vectors for feature selection, and reduce the dimensionality of all texts according to the reference vectors to form the final dimensionality-reduced text vectors.

Mathematical definition of entropy: Suppose there is a variable X with n possible values, namely x ₁ , x ₂ ,…,x _n , and the probability of taking each value is P ₁ , P ₂ ,…,P _n , then the entropy of variable X is defined as:

Entropy of classification system: For a classification system, category C is a variable, its possible values are C ₁ , C ₂ ,...,C _n , and the probability of each category is P(C ₁ ), P(C ₂ ) ,…,P(C _n ), where n represents the number of categories. The entropy of a classification system is defined as:

Among them: P(C _i ) represents the probability of category C _i appearing, which can be estimated by dividing the number of records (number of documents) contained in category C _i by the total number of records (number of documents). which is:

Among them, N represents the total number of records, Indicates the number of records contained in category C _i .

Conditional entropy: Assuming that feature X has n possible values (x ₁ , x ₂ ,…, x _n ), then given X, the entropy of the system is defined as:

in,

Information gain is for each feature, that is, to look at a feature (T), how much information does the system have when it has it and when it does not have it, and the difference between the two is the information that the feature brings to the system Quantity, that is, gain.

The information gain brought by the feature (T) to the system can be written as the difference between the original entropy of the system and the conditional entropy after the feature (T) is fixed:

IG(T)＝H(C)-H(C|T)

In the text classification system, a feature (T) corresponds to a term, and it has only two values "appear" or "not appear". Use t to indicate that a feature (T) appears, and use Indicates that the feature (T) does not appear.

So:

Among them: P(t) represents the probability of feature (T) appearing, Indicates the probability that the feature (T) does not appear.

Expanding the formula further:

So IG(T) can be further expanded into:

The feature selection of the text is to extract important words from the entire text collection. There is no category concept, so the problem needs to be generalized and each text is regarded as a category. At this time, the number of categories is equal to the number N of texts in the text collection. Based on this assumption, the relevant parameters of the information gain formula are estimated.

Symbol Description:

N, represents the total number of texts, that is, the total number of categories;

P(C _i ), represents the probability of category C _i appearing, that is, the probability of text D _i appearing, equal to

P(t), which represents the probability of feature (T), divides the number of texts containing feature (T) by the total number of texts N, namely: where DF _T represents the document frequency of the feature (T);

Indicates the probability that the feature (T) does not appear, equal to 1-P(t);

P(C _i |t), represents the probability that the text contains features (T) and belongs to category C _i ; here, there may be two estimation methods:

Divide the number of texts containing the feature (T) and belonging to the category C _i by the total number of texts, and the value is 0 or

According to the Bayesian formula, P(t|C _i ) represents the probability of feature (T) appearing in category C _i , that is, the probability of feature (T) appearing in document D _i , using Among them, TF _i represents the frequency of occurrence of each term; TF _T represents the frequency of occurrence of each feature T.

Indicates the probability that the text contains the feature (T) and belongs to the category C _i ; here, there may be two estimation methods:

Divide the number of texts that do not contain the feature (T) and belong to the category C _i by the total number of texts, and the value is 0 or

According to the Bayesian formula, in

have to be aware of is:

When estimating P(t), this value may be 1, which would cause has a value of 0, so that Unable to calculate. So P(t) actually uses Make an estimate.

P(t|C _i ) adopts is estimated, if the value of TF _T is 0, this will be the value of 0 for this estimate. So actually use: Make an estimate.

The features mentioned in the embodiments of the present invention all refer to the terms of the text.

Those skilled in the art can determine the definition of each parameter according to the technical solutions of the embodiments of the present invention, and the embodiments of the present invention do not list all of them.

Step S115, reducing the dimensionality of the text to be processed according to the reference vector to form a dimensionality-reduced text vector.

The information gain algorithm proposed in the first embodiment of the present invention selects the features of the text, sorts and screens the features according to the importance of the features to the entire system, so as to achieve the purpose of dimensionality reduction and reduce the computational burden.

Embodiment two

In Embodiment 2 of the present invention, the main process of the method for dimensionality reduction based on text feature selection is the same as in Embodiment 1, wherein the process of text feature selection is shown in FIG. 2, including steps:

Step S210, obtaining initial text.

Step S211, obtain a tokenizer, and use the tokenizer to segment the initial text.

Step S212, obtaining a noun filter, and using the noun filter to perform noun screening on the word-segmented text to obtain a noun set.

In step S213, document frequency statistics are performed and stored in redis.

Step S214, perform word frequency statistics and store in redis.

Step S215, perform document forward indexing.

Step S216, perform IG calculation according to the statistical results of steps S213 and S214.

Step S217, persisting the obtained feature words.

Embodiment Three

Embodiment 3 of the present invention provides a device for dimensionality reduction based on text feature selection, including an acquisition module, a word segmentation module, a statistics module, a vector module, an information gain calculation module, and a dimensionality reduction module.

The obtaining module is used to obtain and store the detailed information of the data source text to be processed.

A word segmentation module, configured to use HanLP to perform word segmentation on the data source text to obtain multiple word items, and remove stop words in the word items.

The statistical module is used for counting term frequency (frequency of occurrence of each term), term document frequency and document word count.

The vector module is used to store the term, term frequency, term document frequency and document word count to form a primary text vector.

The information gain calculation module is used to calculate the information gain of the primary text vector to obtain the information gain amount of each term, sort according to the size of the information gain amount, and form a plurality of words that meet the preset requirements into a reference vector for feature selection.

The dimensionality reduction module is used to reduce the dimensionality of the text to be processed according to the reference vector to form a dimensionality-reduced text vector.

The representation of the text is abstracted into a space vector of the feature word set. However, the original candidate feature word set is as high as hundreds of thousands of dimensions. The high-dimensional text representation not only causes a computational burden on the one hand, but also has large feature redundancy. It will cause a decline in classification performance. Embodiments of the present invention provide a method and device for feature extraction based on an information gain algorithm, which reduces the dimension of the feature word set, reduces the corresponding calculation burden, and removes redundant features. classification performance.

It should be noted that the devices or system embodiments in the embodiments of the present invention may be implemented by software, or by hardware or a combination of software and hardware. From the perspective of hardware, in the framework of the hardware structure of the embodiment of the present invention, in addition to the CPU, memory, network interface, and non-volatile memory, the device where the device in the embodiment is located can generally include other hardware, such as responsible for The forwarding chip that processes the message, etc. Taking software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for operation by the CPU of the device where it is located.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (2)

1. A method for dimensionality reduction based on text class feature selection, characterized in that, comprising steps: Step A, obtain and store the detailed information of the data source text to be processed; Step B, performing word segmentation on the data source text to obtain a plurality of terms, and removing stop words in the terms; Step C, counting word frequency, term document frequency and document word count; Step D, storing the term, term frequency, term document frequency and document word count and forming a primary text vector; Step E, performing information gain calculation on the primary text vector to obtain the information gain amount of each term, sorting according to the size of the information gain amount, and forming a reference vector for feature selection from a plurality of words that meet the preset requirements; Step F, reducing the dimensionality of the text to be processed according to the reference vector to form a dimensionality-reduced text vector; Performing the information gain calculation in the step E includes steps: Take each text as a category, and use the terms in the text as features, and calculate the amount of information gain according to the following formula In the step E, where DF _T represents the document frequency of the feature (T); where TF _i represents the frequency of occurrence of each term; N, indicates the total number of texts, that is, the total number of categories; P(C _i ), represents the probability of category C _i appearing, that is, the probability of text D _i appearing, equal to P(t), which represents the probability of feature (T), divides the number of texts containing feature (T) by the total number of texts N, namely: where DF _T represents the document frequency of the feature (T); , indicating the probability that the feature (T) does not appear, equal to 1-P(t); P(C _i |t), represents the probability that the text contains features (T) and belongs to category C _i ; here, there are two estimation methods: Divide the number of texts containing the feature (T) and belonging to the category C _i by the total number of texts, and the value is 0 or According to the Bayesian formula, P(t|C _i ) represents the probability of feature (T) appearing in category C _i , that is, the probability of feature (T) appearing in document D _i , using Among them, TF _i represents the frequency of occurrence of each term; TF _T represents the frequency of occurrence of each feature T; Indicates the probability that the text does not contain features (T) and belongs to category C _i ; here, there are two estimation methods: Divide the number of texts that do not contain the feature (T) and belong to the category C _i by the total number of texts, and the value is 0 or According to the Bayesian formula, in have to be aware of is: When estimating P(t), P(t) may be 1, which will cause has a value of 0, so that cannot be calculated; so P(t) actually takes make an estimate; according to If the value of TF _T is 0, this will make the value of P(t|C _i ) 0; so P(t|C _i ) actually takes Make an estimate. 2. A text class feature selection device for dimensionality reduction, characterized in that it includes an acquisition module, a word segmentation module, a statistical module, a vector module, an information gain calculation module and a dimensionality reduction module; The obtaining module is used to obtain and store detailed information of the data source text to be processed; The word segmentation module is used to use HanLP to perform word segmentation on the data source text to obtain a plurality of word items, and remove stop words in the word items; The statistical module is used to count word frequency, term document frequency and document word count; The vector module is used to store and form primary text vectors with the term, term frequency, term document frequency and document word count; The information gain calculation module is used to perform information gain calculation on the primary text vector to obtain the information gain amount of each term, sort according to the size of the information gain amount, and form a plurality of words that meet the preset requirements into a feature selected datum vector; The dimensionality reduction module is configured to perform dimensionality reduction on the text to be processed according to the reference vector to form a dimensionality-reduced text vector; The information gain calculation module is used for: Take each text as a category, and use the terms in the text as features, and calculate the amount of information gain according to the following formula where DF _T represents the document frequency of the feature (T); where TF _i represents the frequency of occurrence of each term; N, indicates the total number of texts, that is, the total number of categories; P(C _i ), represents the probability of category C _i appearing, that is, the probability of text D _i appearing, equal to P(t), which represents the probability of feature (T), divides the number of texts containing feature (T) by the total number of texts N, namely: where DF _T represents the document frequency of the feature (T); Indicates the probability that the feature (T) does not appear, equal to 1-P(t); P(C _i |t), represents the probability that the text contains features (T) and belongs to category C _i ; here, there are two estimation methods: Divide the number of texts containing the feature (T) and belonging to the category C _i by the total number of texts, and the value is 0 or According to the Bayesian formula, P(t|C _i ) represents the probability of feature (T) appearing in category C _i , that is, the probability of feature (T) appearing in document D _i , using Among them, TF _i represents the frequency of occurrence of each term; TF _T represents the frequency of occurrence of each feature T; , represents the probability that the text does not contain features (T) and belongs to category C _i ; here, there are two estimation methods: Divide the number of texts that do not contain the feature (T) and belong to the category C _i by the total number of texts, and the value is 0 or According to the Bayesian formula, in have to be aware of is: When estimating P(t), P(t) may be 1, which will cause has a value of 0, so that cannot be calculated; so P(t) actually takes make an estimate; according to If the value of TF _T is 0, this will make the value of P(t|C _i ) 0; so P(t|C _i ) actually takes Make an estimate.