JP4125951B2 - Text automatic classification method and apparatus, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an automatic text classification method and apparatus, a program, and a recording medium used for classifying a large amount of text information such as documents on the Internet.
[0002]
[Prior art]
For example, in a computer network such as the Internet, a large amount of digitized document information having an unspecified number of categories, senders, authors, senders, creation dates and the like is disclosed. Most of these document information is mainly text composed of character string information.
[0003]
In order to acquire necessary information and important information from such a large amount of unspecified document information, various analyzes are performed.
For example, a method called cluster analysis is often used to classify a large amount of information into several groups.
Cluster analysis is one of the multivariate analysis methods. “A mixture of heterogeneous things (individuals may be things or variables) is defined in some sense between them. "Similarity (similarity)" is a general term for collecting similar things and classifying them into several homogeneous communities (non-patent document 1).
[0004]
That is, similar information is linked and information related to each other is classified into a group called a cluster.
In the cluster analysis algorithm, the following processing is generally performed (see FIGS. 5 and 6).
(S1) Initial setting: When there are N pieces of data (d1, d2,..., DN), the cluster (C1, C2,. A set D = {C1, C2,..., CN} is formed.
[0005]
(S2) The cluster set D is searched, and the most similar clusters among them are extracted as the cluster set (Ci, Cj).
(S3) A new cluster Ck is generated from the clusters Ci and Cj, and Ck is added to the cluster set D.
[0006]
(S4) The clusters Ci and Cj are deleted from the cluster set D.
(S5) The above processes (S2) to (S4) are repeated until a predetermined end condition is satisfied.
As the termination condition, for example, “up to m clusters” or “when it is no longer determined to be similar by the value of similarity” in step S2 is assumed.
[0007]
When calculating the degree of similarity between a newly created cluster and other clusters, it is generally calculated using a representative value from the newly created cluster because of the calculation cost. This is called the “centroid method” or “median method”.
Cluster analysis is used for automatic document classification. For example, a large number of documents such as newspaper articles can be collected and used to classify documents having similar contents.
[0008]
In this case, the feature of each document is expressed as a vector (feature vector) in which the element has a one-to-one correspondence with a word existing in the document. The element corresponding to each word expresses the importance (weight) of the word with respect to the document by converting it into a real value.
When calculating the weight of a word, it is generally determined based on the appearance frequency of the word, the distribution ratio in the entire document set, the length of the number of characters, and the like (Non-Patent Document 2).
[0009]
Using the vectors obtained in this way, the similarity (distance) between the documents is defined. For example, an angle (cos θ) formed by vectors in the Euclidean space is used, and vectors having large values are regarded as similar to each other.
Based on such similarity, similar documents are classified into the same cluster, and the process is repeated until there is no similarity between the clusters. As a result, a large number of documents can be classified into several document groups.
[Non-Patent Document 1]
“Multivariate Statistical Analysis”, Tanaka, Wakimoto, Hyundai Mathematics, 1983 First Edition [Non-patent Document 2]
“Automatic Text Processing”, Gerard Salton, ADDISON-WESLEY pub. 1989
[0010]
[Problems to be solved by the invention]
In the conventional cluster analysis as described above, since recursive processing is performed, the number of clusters gradually decreases as processing proceeds. Therefore, if the processing is continued indefinitely, data that are not similar to each other are classified in the same cluster, and a classification error occurs.
[0011]
In order to eliminate the occurrence of such an error, the presence / absence of similarity between clusters is determined using a threshold or the like, and the process ends when the similarity is lost. However, if the threshold value is not appropriately determined according to the type of data to be handled, the situation is often the case that the process ends before the classification process is sufficiently performed, or data with no similarity is often mixed in the same cluster. is there.
[0012]
An object of the present invention is to provide an automatic text classification method and apparatus, a program, and a recording medium capable of improving classification accuracy.
[0013]
[Means for Solving the Problems]
Claim 1 sets a text composed of character string information as a processing target, inputs a plurality of N texts independent from each other, and sets each text to a number smaller than N based on the similarity between the texts. In the automatic text classification device for classifying into clusters having elements as elements, cluster group detection means for storing the degree of association between two different clusters in the association degree table and detecting the two clusters having the highest degree of association as a cluster group; , Between the elements of the plurality of elements included in one cluster of the cluster set and the elements of the plurality of elements included in the other cluster of the cluster set to obtain the similarity between the elements and store in the similarity table a similarity detection means, for the total number of combinations of elements with each other to determine the degree of similarity, number of combinations similarity is equal to or larger than the first threshold value A similarity rate calculation means for calculating the ratio of the similarity ratio comparing means for comparing the ratio of the similarity rate calculation means has calculated the second threshold value, the cluster if the ratio is less than the second threshold value A cluster group exclusion unit that does not integrate a group and a cluster group integration unit that integrates the cluster group and generates a new cluster if the ratio is equal to or greater than a second threshold value are provided.
[0014]
In a general cluster analysis, when a cluster set is detected, a new cluster is unconditionally generated, each original cluster is deleted, and the number of clusters gradually decreases. Therefore, there is a high possibility that a new cluster including a large number of elements (text information) that are small in relation to each other is generated. In claim 1, when a cluster set is detected, for a plurality of elements constituting each cluster of the cluster set, the similarity between the elements is obtained between the clusters and compared with a threshold value. When the degree of similarity between the elements included in each detected cluster is small, the cluster set can be prevented from being integrated as a new cluster.
[0015]
In addition, for example, by assigning 0 to the similarity between clusters detected as the cluster set, these clusters are prevented from being detected again as the same cluster set, and excluded from the similarity determination target between clusters. Can do. As a result, it is possible to prevent the clusters from growing steadily and to show combinations that are not related to elements belonging to the cluster, and it is possible to extract only clusters in which highly related elements are collected with high accuracy.
[0016]
Although the technique (group average method) of Non-Patent Document 1 is related to the technique of the present invention, the cost for calculation is smaller in the present invention based on the centroid method.
[0017]
Further, in claim 1 , when a cluster set is detected, whether the ratio of the similarity of each element included in the cluster satisfies the condition of the first threshold is larger or smaller than the second threshold Therefore, if the number of elements having a small relationship between elements is large compared to the number of elements having a large relationship between elements, the cluster set is classified into one cluster. You can cancel the integration. Therefore, highly accurate classification processing is possible.
[0020]
According to a second aspect of the present invention, there is provided a text classification program for causing a computer to function as each means constituting the automatic text classification apparatus according to the first aspect.
[0021]
By executing the program of claim 2 , the same operation as that of claim 1 can be realized.
[0022]
A third aspect of the present invention is a computer-readable recording medium on which the program of the second aspect is recorded.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
An embodiment of an automatic text classification method and apparatus, program, and recording medium of the present invention will be described with reference to FIGS. This form corresponds to all the claims.
[0024]
FIG. 1 is a flowchart showing this type of text classification processing. FIG. 2 is a block diagram illustrating a configuration example of an apparatus that handles document information. FIG. 3 is a schematic diagram showing a specific example of cluster generation. FIG. 4 is a schematic diagram illustrating a configuration example of the relevance table.
In this embodiment, the cluster set detection means, the element similarity detection means, the element similarity comparison means, and the cluster set exclusion means of claim 1 correspond to steps S13, S15, S16, and S20, respectively.
[0025]
Further, the cluster set detecting means, the element similarity detecting means, the similarity ratio calculating means, the similarity ratio comparing means, and the cluster set excluding means of claim 1 correspond to steps S13, S15, S16, S17, and S20, respectively. Further, the cluster set integrating means of claim 1 corresponds to S18 and S19.
In the apparatus shown in FIG. 2, it is assumed that the classification processing apparatus 10 performs classification processing of text document information. The classification processing device 10 is composed of, for example, a personal computer and a classification processing program that operates on the personal computer. The storage device 11 provided in the classification processing device 10 holds a large amount of collected document information. Further, a matrix representing the similarity between a plurality of independent document information is created on the classification processing device 10 as the similarity table 12. Further, information representing the degree of association between clusters is created on the degree of association table 13.
[0026]
In this example, the classification processing device 10 is connected to the Internet 20 and the LAN 30. Accordingly, the classification processing apparatus 10 has various servers 21 (1), 21 (2), 21 (3),... Connected to the Internet 20 and various servers 31 (1) connected to the LAN 30. , 31 (2),..., Document information, for example, an HTML format text data file (character string information) can be collected.
[0027]
The operation of the classification processing apparatus 10 will be described with reference to FIG.
In step S <b> 10, the classification processing apparatus 10 collects document information groups to be processed via the Internet 20 or the LAN 30.
Further, when a function necessary for calculating similarity between document information is not prepared in advance, or when a plurality of types of selectable functions are prepared, the classification processing apparatus 10 selects a function or inputs it directly. Prompt input to the operator and get one function. Furthermore, when each threshold value used in the process described later can be changed, the operator is prompted to input each threshold value.
[0028]
In step S11, the similarity between each document is calculated according to the function for all the document information collected in S10, and the resulting similarity matrix is registered in the similarity table 12. Note that half of the elements in the similarity table 12 are unnecessary because they are the same combination and the same similarity.
As the document information to be processed, for example, articles of newspapers published in a certain period are assumed.
[0029]
When obtaining such similarity between document information, a vector (feature vector) corresponding to a word existing in the document in a one-to-one relationship is detected, and the importance (weight) of each word for the document is a real value. To express. Further, when calculating the weight of a word, it may be determined based on the appearance frequency of the word, the distribution ratio in the entire document set, the length of the number of characters, and the like. Using the feature vectors obtained in this way, the degree of similarity (distance) between the documents is used, for example, using the angle (cos θ) between the vectors in the Euclidean space, and the vectors having large values are similar to each other. I reckon.
[0030]
In step S12, cluster analysis is initialized. That is, clusters with one element are sequentially created, and document information to be processed is assigned to each cluster one by one. For example, when N pieces of document information are to be processed, initially N cluster groups C1 to CN are created. These collections are set as a set D.
In step S13, two clusters are sequentially extracted for all combinations from the cluster groups C1 to CN included in the set D, and the most similar relationship between the clusters is detected as a cluster set among all the combinations. To do.
[0031]
Actually, using the center vector (centroid vector) of the feature required for each cluster, the combination of all clusters that has the shortest distance between the center vectors (highly related) is detected as a cluster set. .
The degree of association between the clusters in the set D is obtained, for example, as an association degree table 13 having a form as shown in FIG. In the example of FIG. 4, since the degree of association (0.35) between the clusters C2 and C3 is the largest (close), the combination of the clusters C2 and C3 is detected as a candidate cluster pair (C2, C3) in step S13. .
[0032]
In step S14, it is identified whether an end condition is satisfied. For example, by comparing the distance between clusters in the candidate cluster set detected in S13 with a predetermined threshold th1, it is identified whether or not the process should be terminated.
If the end condition is not satisfied, the processes after step S15 are repeatedly executed.
In step S15, the candidate cluster set (Ci, Cj) detected in S13 is included in each element (one or more document information) x included in the i-th cluster Ci and the j-th cluster Cj. Similarity S (x, y) is obtained for each combination of all the elements y. Since these similarities S (x, y) are calculated in advance in step S11 and stored in the similarity table 12, they can be obtained by referring to the similarity table 12.
[0033]
For example, when clusters C2 and C3 shown in FIG. 3 are detected as candidate cluster pairs, three document information exists as elements in cluster C2, and four document information exists as elements in cluster C3. There are 12 combinations of elements (x, y), and the similarity S (x, y) is obtained for each of these combinations.
[0034]
In step S16, each similarity S (x, y) is compared with the threshold th2, and the ratio R between the total number of combinations N1 in the candidate cluster set (Ci, Cj) and the number N2 that satisfies the condition of the threshold th2 is determined. Calculate (R = N2 / N1).
For example, since the candidate cluster set (C2, C3) shown in FIG. 3 is (N1 = 12), if the number N2 that satisfies the condition (S (x, y) ≧ th2) is 9, (R = 0.75).
[0035]
Then, the ratio R is compared with a threshold th3 (for example, 0.6). If (R ≧ th3), the process proceeds from step S17 to S18, and if (R <th3), the process proceeds to step S20.
That is, if (R ≧ th3), a candidate cluster set (Ci, Cj) is generated as a new cluster in step S18 and added to the set D. Further, since the clusters Ci and Cj that are the basis of the new cluster are unnecessary, they are deleted from the set D in step S19.
[0036]
On the other hand, if (R <th3), the candidate cluster set (Ci, Cj) is not adopted as the cluster set. Further, after that, the combination of the cluster Ci and the cluster Cj is processed in step S20 so as not to be the target of the relevance determination in step S13. For example, by changing the degree of association between the corresponding clusters Ci and Cj to 0 in the matrix of the degree-of-association table 13, these can be excluded from the relevance judgment target.
[0037]
For this reason, even if the distance between the center vectors of each cluster is close, if the ratio of documents with low similarity between document information is large among clusters, the combination of these clusters is integrated as a new cluster. Therefore, it is possible to suppress classification of documents with low similarity within the same cluster.
If the predetermined end condition is satisfied in step S14, the classification process is terminated, and the classification result is displayed on the classification processing apparatus 10.
[0038]
In addition, about the process in the classification | category processing apparatus 10, it can also implement | achieve with a dedicated hardware and can also be implement | achieved as a program run on a computer. The program can be read into a computer from a recording medium such as a CD-ROM.
[0039]
【The invention's effect】
As described above, according to the present invention, text information classification processing can be performed with higher accuracy than conventional cluster analysis.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating a text classification process according to an embodiment.
FIG. 2 is a block diagram illustrating a configuration example of an apparatus that handles document information.
FIG. 3 is a schematic diagram showing a specific example of cluster generation.
FIG. 4 is a schematic diagram illustrating a configuration example of an association degree table.
FIG. 5 is a flowchart showing a general cluster analysis process.
FIG. 6 is a schematic diagram showing a general cluster analysis process;
[Explanation of symbols]
10 Classification processing device 11 Storage device 12 Similarity table 13 Relevance table 20 Internet 21 Server 22 Database 30 LAN
31 server 32 database

Claims (3)

A cluster in which text composed of character string information is processed, a plurality of N texts independent from each other are input, and each text is a number of texts smaller than N based on the similarity between the texts In the automatic text classification device that classifies
A cluster set detecting means for storing a degree of association between two different clusters in a relation degree table, and detecting two clusters having the highest degree of association as a cluster set;
The similarity between the elements of the plurality of elements included in one cluster of the cluster set and the elements of the plurality of elements included in the other cluster of the cluster set are calculated and stored in the similarity table. Degree detection means;
A similarity ratio calculating means for calculating a ratio of the number of combinations whose similarity is equal to or greater than a first threshold to the total number of combinations of elements for which the similarity is obtained ;
Similarity ratio comparison means for comparing the ratio calculated by the similarity ratio calculation means with a second threshold;
A cluster group exclusion unit that does not integrate the cluster group if the ratio is less than a second threshold;
An automatic text classification device, comprising: a cluster set integration unit that generates a new cluster by integrating the cluster sets when the ratio is equal to or greater than a second threshold value . A text classification program for causing a computer to function as each means constituting the automatic text classification apparatus according to claim 1 . A computer-readable recording medium on which the program according to claim 2 is recorded.

Images