CN101408893A - Method for rapidly clustering documents - Google Patents

Abstract

The invention provides a fast document clustering method. The method is realized by the following steps: 1, a group of key words is extracted from each document by word frequency statistics; 2, the document is expressed to be a corresponding dimensional congregation of index value, the contained key words of the document are in the characteristic space of the congregation; 3, a nerve element in a self organization mapping model is expressed as a vector in the characteristic space; 4, the documents are input in sequence, and the similarity between the documents and all nerve elements is calculated; 5, the nerve element with maximum accumulated value is the winner; the winner and neighbor nerve elements adjust weight in current document direction; 6, an individual dimension that the nerve element is matched with the input document is adjusted while the weight of other dimensions are weakened; 7, all the documents are input, and the method is over. The invention utilizes a self-organization mapping clustering model to renovate the links of document quantization expression and similarity calculation, thus the calculation efficiency is greatly improved under the condition that the number of the documents is same and the clustering quality is maintained.

Description

A Fast Document Clustering Method

(1) Technical field

The invention relates to a document clustering technology, in particular to a fast document clustering method.

(2) Background technology

With the increasing popularity of the network and the remarkable achievements of information construction, people often need to face an astonishing number of natural language documents. The prominent problem is how to organize, concentrate and integrate the rich information and knowledge contained in them quickly and effectively. Processing, in order to improve the ability of human beings to grasp these massive amounts of information, and improve the level of cognition. Especially in recent years, the automatic sorting of users' personal documents, large-scale network information public opinion monitoring, topic tracking and detection technology, network public opinion trend tracking, and automatic classification of a large number of forum documents are inseparable from fast and high-quality research. Support for text clustering techniques.

Because it is generally believed that the complexity of the clustering algorithm is high, and the use of the well-known Salton vector space model to represent natural language documents is easy to cause the curse of dimensionality, its high computational overhead is recognized as a problem for text clustering when the number of processed documents reaches a large scale. It is one of the important problems to be solved in the practical application of class technology.

(3) Contents of the invention

The invention provides a fast document clustering method for overcoming the efficiency and low problems of the existing clustering methods due to high-dimensional feature quantization and frequent similarity calculation.

The purpose of the present invention is achieved like this: it realizes by following steps: one, utilize word frequency statistics to extract a group of keywords (as 10) from every document, be used to represent the main content of this article; Two, pass In one scan, the keywords of all the documents extracted are used to construct the feature vector space, and the document is expressed as a set of index values of the corresponding dimensions of the keywords contained in the feature space; 3. The neuron in the self-organizing map model Expressed as a vector on the feature space; 4. Input the document in sequence, and calculate the similarity between it and all neurons; 5. The neuron with the largest cumulative value is the winning neuron, and the neuron in its neighbor Document direction adjustment weights; 6. While adjusting the individual dimensions of the neuron that matches the input document, the weights of other dimensions are weakened to prevent documents of other topics from being wrongly mapped to the neuron; 7. All document inputs When you're done, it's over.

The present invention also has some technical characteristics:

1. The similarity calculation method is to calculate the cumulative value of the weight of the document keyword index on the relevant dimension of the neuron node.

The present invention aims at the problem of low efficiency of current text clustering due to high dimensionality and frequent calculation of similarity, and utilizes the self-organizing map clustering model to carry out innovations in links such as document quantitative representation and similarity calculation. When the number of processed documents is the same and the clustering quality is maintained, the computational efficiency is greatly improved.

The steps of the method of the present invention: different from the traditional way of representing documents as high-dimensional space vectors, this method first extracts some keywords (such as 10 keywords) from the documents, and the keywords are generated by performing high-frequency word statistics on the text important content words. Then, a scan is realized, and the work of dynamically constructing the feature space by using the keywords of all documents and simultaneously expressing the index of the keywords contained in the document directly in the vector space is completed. The neuron nodes in the self-organizing map model are represented as vectors in space. Although neuron nodes are still high-dimensional vectors, a large number of documents only contain indexes of a few keywords (such as 10), rather than traditionally represented as high-dimensional vectors (such as several thousand dimensions) as neuron nodes. . Therefore, the similarity calculation between documents and neuron nodes, which is frequently performed in the traditional clustering process, is simplified.

The similarity calculation method between the document and the neuron node is the cumulative value of the weight of the document keyword index on the relevant dimension of the neuron node. The neuron with the largest cumulative value is the winning neuron, and it and its neighbor neurons have the opportunity to adjust the weights to the current document. In order to prevent the wrong mapping of documents to neurons, the method of suppressing other dimensions is adopted, that is, while adjusting the individual dimensions of neurons and input documents, the weights of other dimensions are weakened to prevent wrong mapping of documents of other topics to the neuron.

The essence of the method of the present invention is to avoid a large number of redundant calculations in the traditional method, and the features are not compressed, so that the purpose of not affecting the clustering quality and greatly improving the clustering efficiency can be achieved.

Potential users of the present invention include: 1. Relevant national departments that need to carry out dynamic analysis and monitoring of large-scale text information flow on the network; 2. Many enterprise users, library and information institutions, and scientific research institutes engaged in document information retrieval and information management applications and research 3. A large number of individual users who need to organize and manage documents and quickly retrieve and browse (such as personal emails and clustering of various natural language documents).

The method of the present invention represents a document as a collection of several keyword indexes, the number of which is far less than the dimension of the feature space, which is generally several thousand. Neurons still continue the traditional approach. Since the similarity between documents and neuron nodes is frequently performed in the self-organizing map model, and the number of documents is generally much larger than the number of neuron nodes (generally it can be set to the number of document clusters that need to be generated), the calculation saved The overhead is substantial. Note that the method of the present invention does not compress the features, and the features used are exactly the same as the traditional method. The feature of the present invention is that a large number of redundant calculations can be eliminated by improving aspects such as feature quantitative representation and similarity calculation, so as to achieve the purpose of maintaining clustering quality and greatly improving efficiency. Clustering quality can be measured by clustering F value.

Calculation method of clustering F value: use clustering F value to evaluate the comprehensive quality of document clustering. For a clustering category r generated by clustering and the original predetermined category s, the definitions of recall rate recall and precision rate precision are respectively:

recall(r, s) = n(r, s)/n _s (1)

precision(r, s) = n(r, s)/n _r (2)

where n(r, s) is the number of common documents in the clustered category r and the predefined category s. n _r is the number of documents in the clustering category r, and n _s is the number of documents in the predefined category s. Define F(r, s) as

F(r, s)＝(2*recall(r, s)*precision(r, s))/((precision(r, s)+recall(r, s)) (3)

Then the overall evaluation function of the clustering result is

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; no</span>}}\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Here, n is the number of input documents for clustering. And n _i represents the number of documents in the predefined category i.

(4) Description of drawings

Fig. 1 is the overall schematic diagram of the inventive method;

Fig. 2 is a schematic diagram of the keyword extraction principle of the method of the present invention;

Fig. 3 is a schematic diagram of the similarity calculation principle of the method of the present invention.

(5) Specific implementation methods

The present invention will be further described below in conjunction with Fig. 1 to Fig. 3 and specific embodiment:

In the present invention, the document is represented as a set composed of several representative words, instead of being widely used as a vector in the same high-dimensional space as the model node, so that in the case of large-scale text clustering, the features of the document Indicates that the required memory consumption is greatly reduced. In this mode, two issues need to be properly dealt with: one is the construction of the vector space where the node vectors in the model are located; the other is how to effectively calculate the similarity due to the differences in the representation methods and dimensions of document and node vectors.

For the problem of vector space construction, there are two methods: one is to dynamically generate a vector space according to the actual situation of the samples to be clustered when the documents to be clustered are open domains, because even if the size of the processed documents is large, It is generally impossible to cover most of the vocabulary. However, if the entire vocabulary (such as the Chinese vocabulary) is used to directly construct the vector space, its dimension will be quite large. This increases computational overhead. However, the method of extracting feature words from processing sample documents to construct vector space will greatly reduce the sparse elements in model nodes and reduce the redundancy of vector representation. In this mode, only model nodes need to be represented as high-dimensional vectors, while documents are represented as a collection of a small number of representative words. Because the number of documents is generally much larger than the number of nodes, and the similarity between documents and nodes is frequently carried out in the self-organizing map model. Therefore, the computational overhead will be greatly reduced. In addition, another benefit of this representation is that the construction efficiency of the vector space is also improved.

The present invention takes following steps (the overall schematic diagram is shown in Fig. 1):

1) Perform word segmentation processing on each document, then filter out stop words, and perform word frequency statistics, and finally retain several high-frequency words as the keyword set of the document (see Figure 2);

2) After obtaining the keyword set of each document, first initialize a zero-length vector space, and then repeat the following operations: each keyword w in a document is read into the vector space (it can be a vector ), if found, record the position vec[i] of w in the vector, if not found, add an element at the end of vec, and record its position in the currently processed document vector. In this way, one scan can be implemented, and the vector space and the index of keywords in each document can be constructed at the same time. The approach of similar research is to construct the space first, and then represent the document as a vector in the high-dimensional space.

3) Initialize the neurons in the self-organizing map network as vectors on the constructed feature space;

4) Input the document and calculate the similarity between it and all neurons. For document d and any node n, it is only necessary to add the weights of certain dimensions of the node, for example:

vec[0001]+vec[0008]+vec[0009]+vec[0023];

这里的

是一个经验值，其具体的取值可以通过大量实践获得。(根据自组织映射模型的原理，可以在开始时取较大值，如0.03，以后减小，最后0.01)。通过这种处理，使得与文档d同一主题的文档与节点N相似度较大，而使得节点N在这一主题上的向量组成上得到强化。5) Assuming that the similarity between node N and document d is the largest, adjust node N corresponding to each dimension of document d, for example:

here

It is an experience value, and its specific value can be obtained through a lot of practice. (According to the principle of the self-organizing map model, a larger value can be taken at the beginning, such as 0.03, and then reduced, and finally 0.01). Through this processing, the similarity between the document with the same topic as document d and node N is greater, and the vector composition of node N on this topic is strengthened.

的同时，对其他维减少一个数，如(或者在每个节点第一次调整权值时将其他维一律清零)(参见图3)。6) Increase the number of dimensions (up to 10) that match the document

At the same time, reduce a number for other dimensions, such as (Or all other dimensions are cleared to zero when each node adjusts the weight for the first time) (see Figure 3).

7) After all documents are input, end.

The application process of the present invention is: the user inputs the search term to the search engine, and the search engine returns the found results through retrieval, and these returned documents will be used as the input of the clustering method of the present invention, and the returned results are processed by fast clustering Classification processing improves the visualization effect, thereby greatly improving retrieval efficiency.

Claims (2)

1, a kind of method for rapidly clustering documents is characterized in that it realizes by following step: one, utilize word frequency statistics to extract one group of keyword from every piece of document, be used to represent the main contents of this article; Two,, utilize the keyword structural attitude vector space of all documents that extract, and document is expressed as the set of the index value of the respective dimensions of its keyword that comprises on feature space by single pass; Three, the neuron in the self organizing maps model is expressed as vector on the feature space; Four, import document successively, and calculate the similarity between itself and all neurons; Five, the neuron of accumulated value maximum is the triumph neuron, and the neuron of itself and its neighborhood is adjusted weights to the current document direction; Six, when adjusting indivedual dimensions of neuron and input document coupling, the weights of other dimensions are weakened, be mapped on this neuron with the document mistake that prevents other themes; Seven, after all document inputs finish, finish.

2, a kind of method for rapidly clustering documents according to claim 1 is characterized in that described similarity calculating method is, calculates the accumulated value of the weights of document keyword index on the relevant dimension on the neuron node.

Images