JP2002183171A - Document data clustering system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document clustering system which can classify document data into a number of clusters corresponding to a clustering object. SOLUTION: A group of feature vectors of documents that a feature vector generating means 103 generates is decomposed by singular-values and document similarity vectors 108 for computing the similarities among the documents are generated from the result 106 of the singular-value decomposition. A cluster generating means 110 computes the distance between an object document and the center of gravity of a cluster by using a document similarity vector, classifies the same object document for the 2nd time while increasing the number of dimensions of the document similarity vector used for the 1st classification, and compares the results of the both to determine a cluster having small variation as a stable cluster. A data selecting means 109 excludes documents of the stable cluster from the object to select object documents to be classified next by the cluster generating means, and repeats this trial. The classification is repeated in steps to determine the number of clusters corresponding to the object even when the number of clusters is not determined in advance.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document clustering system used in a document search system, a document filing system, and the like, and more particularly, it is possible to accurately classify documents by using a method of performing clustering step by step. It was made.

[0002]

2. Description of the Related Art Conventionally, as a method of classifying documents, there is known a method of extracting a keyword of a document as a characteristic of the document and automatically classifying the document based on the content of the keyword. The keyword is automatically extracted from the document using a document analysis method such as morphological analysis, or is added manually in advance. For example, JP-A-7-114
No. 572 discloses a method of automatically extracting words from a document by a document analysis method, obtaining a feature vector of a word in which a vector is associated with each word for each document, and classifying the document based on the similarity of the feature vectors. Have been.

[0003]

However, when documents are classified based on the degree of similarity, a large number of classification candidates are generated. Therefore, the criteria for selecting and adopting the classification candidates are unclear, and the classification is not clear. There is a problem that it becomes difficult.

Another problem is that it is difficult to understand what the classified results mean.

[0005] In the clustering performed by humans on a daily basis, for example, newspaper articles are first classified into political articles, economic articles, international articles, sports / entertainment articles, etc. The classification results often have a hierarchical structure, such as subdividing into economic policy articles, stock market articles, and corporate trend articles. This means that hierarchical clustering is more natural and easier for humans to understand than non-hierarchical clustering. Therefore, it is desirable to obtain a classification result having a hierarchical structure as much as possible even when performing clustering mechanically.

SUMMARY OF THE INVENTION The present invention addresses such a problem, and it is possible to accurately classify document data into a number of clusters corresponding to a clustering target, and to display a display showing the contents of the classified result. An object of the present invention is to provide a document clustering system that can associate documents and classify document data into a hierarchical structure.

[0007]

Therefore, the present invention comprises a document database storing machine-readable document data and a dictionary storing machine-readable words, and clusters the documents stored in the document database. In the document data clustering system, a feature vector creating unit that creates a feature vector of a document stored in a document database based on the appearance frequency of a word stored in the dictionary in the document, and a feature vector creating unit that creates the feature vector. Singular value decomposition means for singular value decomposition of a set of feature vectors, document similarity vector generation means for generating a document similarity vector for calculating similarity between documents from the result of singular value decomposition, and document similarity vector generation means All or some documents in the document database are matched by the set of document similarity vectors created by Cluster creating means for creating a cluster in the cluster, a cluster information table storing information of the created cluster, and clustering data for selecting a document to be clustered by the cluster creating means from the document database with reference to the cluster information table Selecting means, a cluster creating means calculates a distance between the document and the cluster centroid using a document similarity vector for the document to be clustered, and further, for the same document to be clustered, The second clustering is performed by increasing the number of dimensions of the document similarity vector used in the clustering within an appropriate range, and comparing the results of the two clusterings, a cluster with little change is determined as a stable cluster, and the clustering data Selection means assigned to stable cluster Remove the document from the clustered object, cluster creation means selects the object clustering to do next, between the cluster forming means clustering data selection means, and configured to repeat the attempt.

The cluster creating means calculates the distance between the document and the cluster centroid using the document similarity vector for the document to be clustered, and further calculates the first clustering document for the same document to be clustered. The second clustering is performed by increasing the number of dimensions of the document similarity vector used in the clustering within an appropriate range, and the results of the two clusterings are compared. The selection unit removes the document assigned to the unstable cluster from the clustering targets, selects a clustering target to be performed next by the cluster creation unit, and performs this trial between the cluster creation unit and the clustering data selection unit. It is configured to repeat.

Further, label extracting means for extracting a label for each cluster with reference to the document database, the dictionary, the singular value decomposition result, and the cluster information table is provided.
The label extracting means calculates a feature vector expressing a pseudo frequency of occurrence of the word at the center of gravity of the cluster, and calculates, from among documents assigned to the cluster, a vicinity of a word having a high frequency of appearance included in the feature vector. Is extracted as a label.

Further, there is provided a cluster hierarchical relationship determining means for setting a hierarchical relationship between clusters created by the cluster creating means, and the cluster hierarchical relationship determining means has determined that any cluster C is a stable cluster. When the number of dimensions at this time is defined as the stable dimension d (C) of the cluster, all documents belonging to the cluster C ′ having a lower stable dimension than the stable dimension d (C) of the cluster C and the center of gravity g (C) of the cluster C When the distance in the d (C) dimension with respect to is within a certain distance R (C), the cluster C
It is configured to be hierarchized into upper clusters.

Therefore, in the document data clustering system of the present invention, the number of dimensions of the document similarity vector is gradually increased, and the clustering is repeated in stages, so that the clustering can be performed even if the number of clusters is not determined in advance. The number of clusters according to the target can be determined.

Further, the label extracting means can extract a character string representing the content of the cluster from a document belonging to the cluster.

[0013] Further, the cluster hierarchy relation determining means can set a hierarchy between the clusters generated by the stepwise clustering, and generate a hierarchical relation between the clusters that matches human intuition.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment) As shown in FIG. 1, a document data clustering system according to a first embodiment has a document database 101 for storing document data.
A machine-readable dictionary 102 that stores words, a feature vector creating unit 103 that creates a feature vector from word appearance frequencies in documents to be clustered, a singular value decomposition unit 105 that performs singular value decomposition of feature vector data, Document similarity vector creating means 107 for creating a document similarity vector with reduced vector dimension, and clustering target data selecting means 109 for selecting a clustering target document
A cluster creating unit 110 for creating a cluster of documents to be clustered based on the document similarity vector, a cluster information table 111 for storing information of clusters created by the cluster creating unit 110, and a result display for displaying a clustering result Means 112.

In FIG. 1, reference numeral 104 denotes a feature vector set created and stored by the feature vector creation means 103; 106, a singular value decomposition result created and stored by the singular value decomposition means 105; Reference numeral 108 denotes a set of document similarity vectors created and stored by the document similarity vector creation means 107.

Next, the operation of the document data clustering system having the above configuration will be described with reference to FIGS.

First, a processing procedure until a document similarity vector is created from document data will be described. FIG. 3 is a flowchart showing this processing procedure.

Step 302: feature vector creating means 103
Creates a feature vector 104 of a document in the document database 101 based on the word appearance frequency statistical information with reference to the dictionary 102.

The feature vector creation means 103 refers to the dictionary 102, finds the appearance frequency of the words described in the dictionary 102 in the document, and uses the appearance frequency of each word as the value of the element of the feature vector for the document. The number of dimensions of the feature vector matches the number of all words in the dictionary, and the number of feature vectors matches the number of documents. The frequency of words that do not appear in the document is 0.

FIG. 4 is an example of a feature vector based on the frequency of appearance of a word in a document. In the document 1 with the document identifier 0001, the word 1 recorded in the dictionary 102 is 13 times and the word 2
Represents 0 times and word 3 appears 4 times.

The elements of the feature vector include not only the frequency of appearance but also a value obtained by dividing the frequency of appearance by the document length as shown in FIG. 5 or the frequency of appearance as shown in FIG. The value normalized by dividing by the total sum of the appearance frequencies of the words in the document, and the tf · in consideration of the appearance frequency in the document and the appearance frequency in the whole document widely used in the field of information retrieval.
An idf value or the like can be used. The element of the feature vector may be any value as long as it is statistical information calculated based on the appearance frequency.

Step 303: The singular value decomposition means 105 performs singular value decomposition on the feature vector set obtained in step 302.

When the feature vector set is represented by a matrix X, the number of rows is the number of feature vectors, and the number of columns is the number of dimensions of the feature vector, that is, the number of words in the dictionary 103. If the matrix X has rank r, number of rows m, and number of columns n, it can be decomposed into three matrices D, S, and T by singular value decomposition (T 'represents a transposed matrix of T). X = DST ′ (Equation 1) where S is r × r having a singular value of the matrix X as a diagonal element.
Where D is an m × r, T is an n × r column orthogonal matrix (TT ′ = DD ′ = I, I is a unit matrix). Thus, the singular value decomposition of the feature vector is performed, and the singular value decomposition result 10
Get 6.

When the singular value decomposition is performed, s elements can be extracted from the matrix S in descending order of diagonal elements to perform low-rank approximation. At this time, the dimension of the feature vector obtained by the low-rank approximation can be obtained. The number is smaller than the number of dimensions of the original feature vector set.

Step 305: Document similar vector creating means 1
In step 07, using the singular value decomposition result 106, a document similarity vector set for calculating the similarity between documents is created.

When the matrix expression X of the feature vector set 104 is used, the similarity between documents can be expressed by a matrix XX '. Therefore, the similarity matrix between documents can be transformed into XX ′ = DST′TSD ′ = DSSD ′ = (DS) (DS) ′ (Equation 2) from the singular value decomposition result given by (Equation 1). The matrix DS is an mxr matrix, the number of rows matches the number m of documents, and the number of columns matches the rank r of the matrix X.
The row vector of the matrix DS is defined as a document similar vector.

Since the diagonal elements of the matrix S obtained by the singular value decomposition correspond to the principal components in the principal component analysis, the greater the number of dimensions, the more information is provided.

Next, the processing procedure of the clustering processing in this system will be described. FIG. 2 is a flowchart showing this processing procedure.

Step 202: The clustering target data selecting means 109 adds all documents (all elements) of the document database 101 to the cluster information table 111 as clustering targets.

FIG. 7 shows a description example of the cluster information table 111. The number of records in this table is the document database
This is the number of documents in 101, and has at least a document identifier for specifying a document, a field of an identifier of a cluster to which the document is assigned, and a flag indicating that the document is to be clustered. In FIG. 7, step 202
As a result, it is shown that all the documents in the document database 101 are assigned to the cluster with the cluster identifier 0, and that all the documents are to be clustered.
Here, the cluster identifier 0 indicates a virtual cluster since clustering has never been performed.
As a clustering target flag, 1 indicates that the clustering is to be performed, and 0 indicates that the clustering is not to be performed.

Step 203: The cluster creating means 110 receives, as parameters, the number k of clusters created by one clustering, the increment Δk of the number of clusters, the number d of dimensions of the document similarity vector used at the time of cluster assignment, and the number of dimensions. Set the increment Δd.

The number k of clusters may be one or more and equal to or less than the number of documents in the document database. Cluster number increment Δk
May be 0 or its absolute value is from 1 to the number of documents in the document database. Dimension number d and dimension increment Δ
d may be equal to or smaller than the rank r of the matrix S calculated in step 303.

Step 204: The cluster creating means 110 checks the number of documents to be clustered with reference to the cluster information table 111. If no clustering has been performed, all documents are to be clustered. Therefore, the number of documents to be clustered matches the number of documents in the document database, and the process proceeds to step 205.

Step 205: Non-hierarchical clustering called a K-means method is performed using the document similarity vector set 108 obtained in step 305.

In the K-means method, clustering is performed by the following algorithm. First, k cluster centroid points are given as initial values. Documents to be clustered (=
The distance from each of the k cluster centroids is calculated for each element, and the element is assigned to the cluster having the closest cluster centroid. When the assignment has been completed for all the elements, the center of gravity is obtained from the elements for each of the k clusters, and this is set as a new center of gravity, and each element is assigned to the cluster again. Each element may be assigned to another cluster for each cluster assignment, but if all the elements have a certain number of mismatches between the previous cluster and the newly assigned cluster, the assignment will be reduced. finish.

In step 205, the cluster information table 1
For documents that are subject to clustering in 11,
Clustering is performed by the K-means method. Here, the distance calculation between the cluster centroid and the document similarity vector of each element,
And, when calculating the center of gravity, the first of the r-dimensional document similarity vectors
Elements from the dimension to the d-th dimension are used. At this time, the initial center of gravity of each cluster and a list of document identifiers of the documents included in the cluster are recorded. As a result of step 205, k clusters are generated.

Step 206: The number of used dimensions d is appropriately increased within a range less than r. If the increment of the number of dimensions d is Δd, d + Δd becomes d. For example, if d is 100 and the increment is 50, the new number of used dimensions is 150. Step 207: Similar to step 205, k clusters are created by the K-means method. In this clustering,
Since the number of dimensions to be used is larger than in step 205, clustering is performed using more information. At this time, the same point as that used in step 205 is used for the initial center of gravity. By using the same initial center of gravity, the cluster obtained in step 205 and step 207 are obtained.
Can be associated with the cluster obtained in step (1). At this time, a list of document identifiers of the documents included in the cluster is recorded.

Step 208: By comparing the result of the clustering in the step 205 with the result of the step 207 in which the information used for the clustering is increased and clustered, a cluster which does not require further clustering is determined. For a cluster in which the document assigned to the cluster does not change even when the amount of information is increased, it can be determined that the clustering result does not change even if more information is given.

In the determination procedure, first, for each of the clusters obtained in steps 205 and 207, the presence or absence of the assigned document is compared, and the degree of coincidence is checked.

FIG. 8 shows a comparison of documents assigned as a result of two clusterings for a certain cluster. The second line is step 205, that is, the number of dimensions d
, And the third line indicates the document included in the cluster created using step 207, that is, the number of dimensions d + Δd. In the figure, 0 indicates that the document is not included in the cluster, and 1 indicates that the document is included in the cluster. The degree of coincidence may be given by the ratio of the number of documents commonly included in two clustering results to the total number of documents.

In FIG. 8, a total of 100 documents are obtained by the first clustering, and a total of 11 documents are obtained by the second clustering.
This example shows a case where zero documents are allocated and the number of commonly included documents is 95. At this time, the degree of coincidence is 95 / (100 + 110) = 0.45. For each cluster, an appropriate threshold is given, and the one with a high degree of coincidence is regarded as a stable cluster, and the other clusters are regarded as unstable clusters.

In addition to determining an unstable cluster based on the degree of coincidence, an unstable cluster may be determined using the ratio of the number of documents assigned to the cluster to the total number of documents.

Step 208: Next, the clustering target flag of the cluster information table 111 is set to be unnecessary because the document included in the stable cluster does not require further clustering. The documents included in the unstable cluster are subjected to re-clustering without changing the clustering target flag. In the cluster information table 111, the cluster number assigned the second time is recorded in the cluster number field of the document included in the stable cluster.

FIG. 9 shows an updated cluster information table.
111 shows an example. Document identifier 0001, 0003
Is the stable cluster 1001 and the document identifier 00
04, m are assigned to stable cluster 1002. Since the document identifiers 0002 and 0005 are to be re-clustered, the clustering target flag is set to 0.

Step 209: After step 208, the number k of created clusters is increased by Δk given in step 203.
When Δk is 0, the number of clusters does not change.

Then, the process proceeds to step 204, where the number of documents to be clustered is checked with reference to the cluster information table 111 again.

If it is determined in step 208 that all clusters are stable clusters, since there is no document to be clustered, the process proceeds to step 210, and the clustering process ends. If there is a document to be clustered, steps 205 to 208 are repeated again. However, the number of dimensions of the document similarity vector used at the time of cluster assignment is d + Δd.

FIG. 10 is a conceptual diagram showing the change of the cluster created from step 201 to step 210 described above. On the left side, the number of clusters k and the number of dimensions of the document similarity vector used at the time of clustering in step 207 are shown. k is set to 7. Δk is 0. The first clustering shows that three stable clusters (squares in the figure) and four unstable clusters (hatched squares) have been created. The next stage shows the result of clustering with documents assigned to the four unstable clusters as clustering targets.

As described above, by repeating the clustering, the unstable cluster is eliminated in the fourth stage.
Indicates that the cluster creation processing has been completed.

As described above, in the document data clustering system of this embodiment, the singular value decomposition of the document data is performed, the information used for calculating the distance between documents is gradually increased, and the clustering is repeated stepwise. By doing so, it is possible to determine the number of clusters according to the clustering target even if the number of clusters is not determined in advance, and it is possible to generate a cluster in which only similar documents are aggregated. .

(Second Embodiment) In the second embodiment,
A system for labeling a character string representing the content of the result of clustering will be described.

As shown in FIG. 11, the system includes a document database 1101 for storing document data and a machine-readable dictionary 1102 for storing words, as in the first embodiment.
And feature vector creating means 1103 for creating a feature vector
Singular value decomposition means 11 for singular value decomposition of a feature vector set
05, a document similarity vector creating means 1107 for calculating a document similarity vector for calculating the similarity between documents based on the singular value decomposition result, and a result display means 1112, and further, a label representing a cluster is provided. Label extraction means 1110 to extract
It has.

In FIG. 11, reference numeral 1104 denotes a feature vector set created by the feature vector creation means 1103, 1106 denotes a singular value decomposition result obtained by the singular value decomposition means 1105,
Reference numeral 1108 denotes a document similarity vector set created by the document similarity vector creating means 1107. Reference numeral 1109 denotes a cluster information table in which a document in the document database is assigned to one of a plurality of clusters by some method, and information similar to that of the first embodiment is stored.

The label extracting means 1110 includes a document database
With reference to 1101, dictionary 1102, singular value decomposition result 1106, document similar vector set 1108, and cluster information table 1109, a label representing a cluster is extracted. Reference numeral 1111 denotes a label extracted by the label extracting unit 1108.
The result display means 1112 displays the cluster information table 1109 and the extracted label 1111.

The operation of the document data clustering system having the above configuration will be described with reference to FIGS.

FIG. 12 is a flowchart showing the processing procedure of this system. Step 1202: The feature vector creation unit 1103 creates a feature vector set 1104 using the document database 1101 and the dictionary 1102. This procedure is the same as step 302 of the first embodiment (FIG. 3).

Step 1203: Next, singular value decomposition means 1105
Performs singular value decomposition based on the feature vector set to obtain a singular value decomposition result 1106. This procedure is the same as step 303 of the first embodiment (FIG. 3).

As a result of the singular value decomposition, the matrix expression X (m ×
n) are three matrices D (m × r), S (r × r), T
It is decomposed into (nxr). Here, m is the number of documents in the document database, n is the number of words in the dictionary, and r is the rank of the matrix X.

Step 1204: Document similar vector creation means
A step 1107 creates a document similarity vector 1108 from the singular value decomposition result. This procedure is a step of the first embodiment (FIG. 3).
Same as 304.

Step 1205: Next, label extracting means 1110
Shifts to label extraction processing from the cluster. Step 12
In 05, it is determined whether or not labels have been extracted from all clusters. If label extraction has been completed for all clusters, the process ends (step 1210).
The label extracting unit 1106 performs a label extracting process from the cluster for a cluster for which label extraction has not been completed (steps 1206 to 1209).

Step 1206: First, a centroid vector g of the cluster is obtained. Each element of the centroid vector g of the cluster is an average value of each element of the document similarity vector 1108 corresponding to the document assigned to the cluster. The number of dimensions of the cluster centroid vector matches the number of dimensions of the document similarity vector, and is r.

Step 1207: Next, using the centroid vector g (1 × r) of the cluster and the singular value decomposition result 1106,
The representative word vector h is obtained by the calculation method shown in (Equation 3). h = gST ′ (Equation 3) Since the right side of (Equation 3) is obtained by replacing the matrix D in (Equation 1) with the barycenter vector g, h represents a feature vector corresponding to the barycenter vector g of the cluster. become. Therefore, each element of the representative word vector h is stored in the dictionary 1102
The value of the element corresponds to the frequency of appearance with respect to the cluster centroid.

FIG. 13 shows an example of a representative word vector. In the figure, a representative word vector is surrounded by a dotted line, and words corresponding to each element are described on the left side.

Next, words corresponding to each element of the representative word vector are obtained with reference to the dictionary 1102, and these are set as a representative word set. For each representative word, the value of the corresponding element of the representative word vector is recorded as its score. The number of representative words is dictionary 11
Although the number of words matches the number of words in 02, words with a small score may be removed from the representative words.

Step 1208: Subsequently, using the representative word,
A character string around a representative word appearing in a document assigned to a cluster is extracted as a label candidate. here,
A character string sandwiched between any two representative words is set as a label candidate.

In the method of extracting a character string around a representative word as a label, for example, considering a part of speech of a representative word, a character string that starts with a representative word that is a proper noun and ends with a representative word that is a variable is a label candidate. And the like. The document from which the label is to be extracted may be all documents assigned to the cluster or some selected documents. As a document selection method, a method of calculating the distance between the centroid vector of the cluster and the document similarity vector, and selecting several documents from those having the shortest distance can be considered.

Step 1209: A score is calculated for each label candidate, and a label candidate having a large score is extracted as a label representing a cluster. Although there are several possible score calculation methods, there is a calculation method as shown in (Formula 4) as an example. (P × w1 + (1−p) w2) / L (Equation 4) Here, p represents the score weight of the representative word located at the head of the label candidate, and its value is 0 or more and 1 or less. w1, w
2 represents the score of the representative word located at the head of the label candidate and the score of the representative word located at the end. L is the length of the label candidate.

FIG. 14 shows an example of extracting a label candidate from a document and calculating its score. In the upper example, “living standard” is extracted as a character string sandwiched between “living” and “level”, and its score is calculated as P = 0.5 in (Equation 4). In the lower example, “cooperation utilizing specialty” is extracted as a character string sandwiched between “specialty” and “cooperation”, and its score is calculated.

As described above, in the document data clustering system of this embodiment, a character string representing the contents of the clustered result can be extracted as a label, and the cluster can be distinguished. This has the effect of providing information for estimating the contents.

(Third Embodiment) In the third embodiment,
A system will be described in which an unstable cluster is excluded from a target and re-clustering is repeated to generate a cluster in which only similar documents are aggregated.

In the first embodiment, when performing re-clustering, clusters that are stable are excluded from the target and re-clustering is performed. However, in the system of the third embodiment, the amount of information is increased. Re-clustering is performed excluding unstable clusters having a high degree of mismatch when clustering is performed.

In this system, as shown in FIG. 15, the clustering target data selecting means 109 removes the unstable clusters from the document database 101 or the clustering result, and sets the remaining clusters as the clustering target, and sets the identifier of the document data. The main parent clustering target data selection means 1091 for selecting the Other configurations are the same as those of the first embodiment (FIG. 1).

The operation of this system is shown in the flowchart of FIG. In FIG. 16, the procedure from step 202 to step 207 is the same as the procedure of the first embodiment (FIG. 2).

Step 208: By comparing the clustering result of step 205 with the result of step 207 obtained by increasing the information used for clustering and performing clustering, a cluster to be excluded from the cluster is determined. For a cluster in which the documents assigned to the cluster fluctuate greatly even if the amount of information is increased, it is determined that the clustering result will fluctuate further by providing more information.

In the determination procedure, first, for each cluster obtained in steps 205 and 207, the presence / absence of a document assigned to each cluster is compared, and the degree of coincidence is checked. FIG. 17 shows a comparison of documents assigned as a result of two clusterings for a certain cluster. The second line is Step 205, that is, the document included in the cluster created using the number of dimensions d is the third line.
That is, the documents included in the cluster created using the number of dimensions d + Δd are shown. In the figure, 0 indicates that the document is not included in the cluster, and 1 indicates that the document is included in the cluster. The degree of coincidence may be given by the ratio of the number of documents commonly included in two clustering results to the total number of documents. FIG. 17 shows a case where a total of 100 documents are allocated in the first clustering and a total of 110 documents are allocated in the second clustering, and the number of documents that are not common to each other is 40.
At this time, the degree of mismatch is 40 / (100 + 110) = 0.
It becomes 18.

For each cluster, an appropriate threshold value is given, those with high mismatch are excluded as unstable clusters, and the others are regarded as stable clusters. In addition to determining an unstable cluster due to the degree of inconsistency, an unstable cluster may be determined using the ratio of the number of documents assigned to the cluster to the total number of documents.

Next, for documents included in an unstable cluster, the clustering target flag of the cluster information table 111 is set to be unnecessary. Documents included in a stable cluster are set as re-clustering targets without changing the clustering target flag. In the cluster information field of the document included in the stable cluster in the cluster information table 111, the cluster number assigned for the second time is recorded.

FIG. 18 shows an example of the updated cluster information table 111. Document identifier 0001, 000
3 is the document identifier 000 in the stable cluster 1001
Since 4 and m are allocated to the stable cluster 1002 and are to be re-clustered, the clustering target flag is 1. Document identifier 0002, 0005
Are excluded from the re-clustering target, and the clustering target flag is set to 0.

From step 208 onward, the same processing as in FIG. 1 is performed. By doing so, a stable cluster can be further clustered.

FIG. 19 is a conceptual diagram showing a change in a cluster. On the left side, the number of clusters k and the number of dimensions of the document similarity vector used at the time of clustering in step 207 are shown. k is set to 7. Δk is 0. In the first clustering, three stable clusters (squares in the figure)
And four unstable clusters (hatched squares) have been created. The next stage shows the result of clustering with documents assigned to three stable clusters as clustering targets.

As described above, by repeating clustering, a cluster in which only similar documents are aggregated can be generated.

As described above, in the document data clustering system of this embodiment, the singular value decomposition of the document data is performed, the information used for calculating the distance between documents is gradually increased, and the clustering is repeated stepwise. By doing so, even if the number of clusters is not determined in advance, the number of clusters can be determined according to the clustering target, and a cluster in which only similar documents are aggregated can be generated. .

(Fourth Embodiment) In the fourth embodiment,
A document data clustering system that classifies document data hierarchically will be described.

The basic concept of hierarchization in this system is as follows. In the first embodiment, in the process of creating a cluster, while increasing the number of dimensions to be used, a stable cluster is removed as needed, and the remaining elements are clustered in a space having a larger dimension.
It realizes the gradual clustering.

In such a stepwise clustering,
There is a tendency for denser clusters to stabilize earlier and for more broad clusters to be generated later. Therefore, a denser cluster, that is, a cluster of documents related to a more specific theme, is generated earlier, and a sparse cluster, that is, a cluster of documents that are grouped under a more general theme, is It is easy to generate later.

In this case, setting a document cluster related to a more general theme as a higher-level cluster of a document cluster related to a more specific theme is also consistent with the intuition of hierarchies between clusters performed by humans.

Therefore, in order to introduce a natural hierarchical relationship between the clusters obtained step by step in the clustering process, a later stable cluster (having a higher stable dimension) must be stably first. (Low stable dimension)
It is desirable to be positioned higher in the hierarchy than in the cluster.

However, since it is natural intuition that the upper cluster covers the theme of the lower cluster,
It is desirable that the lower cluster be within a certain range from the center of gravity of the upper cluster.

In order to satisfy these requirements, referring to the center of gravity of each cluster and the dimension when the cluster is determined to be stable, the first stable cluster C ′ is later determined by the dimension d
In the case of being located within a certain distance from the center of gravity of the stable cluster C in (C), that is, in the dimension d (C), the distance from the center of gravity of the cluster C to the cluster C ′ is a certain distance indicating the size of the cluster C. If it falls within the range,
By positioning C as the upper cluster of C ′, the hierarchical relationship between the clusters generated stepwise is set.

Here, the center of gravity of each cluster and the distance representing the size of the cluster are calculable quantities that can be obtained mechanically by calculation between document feature vectors.

According to the above concept, the document data clustering system of this embodiment executes clustering.

As shown in FIG. 20, this system has a document database 1901 and a document database 1901 as in the first embodiment (FIG. 1).
Dictionary 1902, feature vector creation means 1903, feature vector set
1904, singular value decomposition means 1905, singular value decomposition result 1906, document similar vector creation means 1907, document similar vector set 1908,
It includes a clustering target data selection unit 1909, a cluster creation unit 1910, a cluster information table 1911, and a result display unit 1915, and further, based on the cluster information stored in the cluster information table 1911 and the document similarity vector set 1908, Cluster hierarchical relationship determining means 1912 for calculating the hierarchical relationship between the clusters, a cluster size information table 1913 for storing size information of each cluster, and a cluster hierarchical relationship table 1914 for registering a hierarchical relationship between clusters.
And

Here, in addition to the operation in the first embodiment, when a stable cluster is detected, the cluster creating means 1910 sets the used dimension at that time as the stable dimension of the cluster and sets the cluster size information table 1912 as the stable dimension. And the center of gravity of the cluster is stored in the same cluster size information table 1913 at the same time. Also, the result display means 19
Reference numeral 15 denotes a clustering result and a hierarchical relationship between clusters. 1901 to 1911 excluding this cluster creation means 1910
The internal configuration and operation up to this point are the same as in the first embodiment.

FIG. 21 is an example of the cluster size information table 1913. This table stores a cluster number, a stable dimension, a center-of-gravity vector, and a reference radius for each cluster. The stable dimension of cluster C is
This is the number of dimensions when the cluster C is removed from the clustering target as being stable in the clustering process. Hereinafter, this stable dimension is expressed as d (C).

The reference radius of the cluster C is the cluster centroid (g (C)) of the cluster C among the documents belonging to the cluster C.
), The distance in the d (C) dimension between the document farthest in the distance in the d (C) dimension and g (C) is R1 (C).
Further, among documents belonging to a cluster other than C whose stable dimension is d (C) or more, in a distance in the d (C) dimension,
When the distance between the document closest to g (C) and g (C) in the d (C) dimension is R2 (C), R1 (C) and R1 (C)
2 (C), whichever is smaller. Hereinafter, the reference radius of the cluster C is represented as R (C).

Here, the distance between the document or cluster centroids in the d (C) dimension is the distance in the d (C) dimension between the document similar vectors, and the document similar vector x1 = (x11, x12,. , X1d (C), ‥, x1r)
Dist (x1, x2) in the d (C) dimension between x2 = (x21, x22, ‥, x2d (C), ‥, x2r)
Is defined by dist (x1, x2) = {(x11−x21) ² + (x12−x22) ² + ‥ + (x1d (C) −x2d (C)) ² } ^0.5 (Equation 5)

FIG. 22 shows a cluster hierarchy relation table 1914.
FIG. 3 is a diagram showing an example of the above. The total number of generated clusters is Cma
Assuming that x is the same, the table is a Cmax × Cmax-dimensional matrix. In the (i, j) -th element, when the cluster of cluster number i is an upper cluster of the cluster of cluster number j, 1 is set. Otherwise, 0 is stored. FIG. 22 shows that the cluster 2001 is the cluster 1001
This indicates a state in which the cluster is registered as a higher-level cluster. Initially, all elements are zero.

The operation of the document data clustering system having the above configuration will be described with reference to FIGS.
FIG. 23 is a flowchart illustrating a clustering process and a hierarchical relationship calculation procedure between clusters.

Step 2202: means for creating feature vector 1903
Creates a feature vector 1904 for a document included in the document database 1901. Step 2203: The singular value decomposition means 1905 performs singular value decomposition based on the feature vector set to obtain a singular value decomposition result 1906. Step 2204: The document similar vector creation means 1907 creates a document similar vector set 1908 from the singular value decomposition result. The procedure up to this point is the same as that of the first embodiment (FIG. 3).
Same as 302 to step 305.

Step 2205: The clustering target data selecting means 1909 and the cluster creating means 1910 refer to the document similarity vector set 1908 and the document database 1901,
Perform clustering on all documents in the document database. This procedure is the same as steps 201 to 210 of the first embodiment (FIG. 2). However, every time a stable cluster is recognized (corresponding to the time point of step 208 in the first embodiment), the cluster creating means 1910 stores the number of used dimensions and the centroid vector of the cluster at that time point in the cluster size information table 1913. Register with.

Step 2206: Next, the cluster hierarchy relation determining means 1912 refers to the contents of the cluster information table 1911 and the cluster size information table 1913, calculates the reference radius of each cluster according to the above definition, and Register in the information table 1912. Step 2207: The cluster hierarchy relation determining means 1913 includes the cluster information table 1911 and the cluster size information table 19
13, the hierarchical relationship between the clusters is calculated, and the result is registered in the cluster hierarchical relationship table 1914.

FIG. 24 is a flowchart showing a procedure for calculating the hierarchical relationship between the clusters. Step 2302: First, all the elements of the cluster hierarchy relation table 1914 are initialized to 0 (there is no hierarchical relation). Step 2303: Next, the cluster size information table 1913
, And set the value of the dimension d as the minimum value of the registered stable dimensions. Step 2304: Next, the cluster size information table 1913
, A set of clusters whose stable dimension is d is obtained as a set to be compared. Hereinafter, the set to be compared is represented as S0.

Step 2305: Next, increase the dimension by the value of Δd used at the time of clustering. Step 2306: Referring to the cluster size information table 1913, the dimension resulting from the increase exceeds the maximum value of the stable dimension. Find out if. If so, the calculation of the hierarchical relationship between the clusters ends (step 2314). If not, step 2307: The cluster size information table 1913 is referred to, and a set of clusters having the current dimension as a stable dimension is obtained as a processing target set. Hereinafter, the set to be processed is represented as S1.

Step 2308: Next, after all the clusters belonging to the processing target set S1 are unprocessed, Step 2309: Until there are no unprocessed clusters in S1, Step 2310: One cluster is extracted from S1 Step 2311: The inclusion relationship between each cluster in S0 and the cluster to be processed is examined. This procedure will be described later in detail with reference to FIG.

Step 2312: The processed cluster to be processed is regarded as processed, and the procedure from step 2309 is repeated.

When the calculation of the hierarchical relationship with the clusters of S0 is completed for all the clusters of S1, step 2313: All the clusters of the processing target set S1 are added to the comparison target set S0, and the process returns to step 2305.

As described above, for each cluster, the hierarchical relationship between the cluster having a stable dimension lower than the stable dimension of the cluster, that is, the cluster determined to be stable earlier than the cluster is calculated.

FIG. 25 is a flowchart showing a procedure for obtaining a lower cluster. Using this FIG. 25, the cluster to be processed in step 2311 and the comparison target set S
The calculation of the hierarchical relation between each cluster of 0 will be described. Here, the cluster to be processed is C.

Step 2402: All clusters in the set to be compared are not compared. Step 2403: Next, the cluster size information table 1912
And the cluster centroid g of the cluster C to be processed
The center of gravity vector of (C) and the reference radius R (C) of the cluster C
And get

Step 2404: Next, it is checked whether there is an uncompared cluster in the set to be compared. If not, the calculation of the hierarchical relationship ends (step 2409). If there are uncompared clusters, Step 2405: One of them is set as a comparison target. The cluster to be compared is C ′.

Step 2406: Next, it is checked whether all the documents belonging to the cluster C 'to be compared fall within the range of the reference radius R (C) centered on the center of gravity g (C) of the cluster C to be processed. . However, regarding the distance, (Equation 5)
Calculate according to the definition.

Step 2407: As a result, if all are included, the cluster C to be processed is set as the upper cluster of the cluster C ′ to be compared, and Step 2408: The cluster C ′ to be compared is made to be already compared. In the inspection of step 2406, the document belonging to the cluster C ′ to be compared, and the center of gravity g of the cluster C to be processed
In the case where there is an object that does not fall within the range of the reference radius R (C) from (C), the process proceeds directly from step 2406 to step 24.
Move to 08. After step 2408, the process returns to step 2404 to check whether there is a next comparison target.

The above is the method of examining the hierarchical relationship between the cluster to be processed and each cluster of the set S0 to be compared.

As described above, in the document data clustering system according to the fourth embodiment, the singular value decomposition of the document data is performed, and the information used for calculating the distance between the documents is gradually increased. There is an effect that a natural hierarchical relation can be introduced between the clusters resulting from the clustering by using a computable amount.

In the above embodiment, even after the cluster C is determined to be the upper cluster of the cluster C ′, the elements of the cluster C ′ are not directly included in the cluster C. May be included in the upper cluster.

Further, the center of gravity of the upper cluster may be calculated again in consideration of the elements of the lower cluster.

In the above embodiment, a distance called a reference radius is defined and used when examining the hierarchical relationship between clusters. However, instead of the reference radius defined above, for example, the distance from the center of gravity of the cluster to be processed may be used. By using various distances defined in the d (C) dimension, such as the maximum value, the minimum value, or the average value of the distances to the documents belonging to the cluster, the radius in which 95% of the elements are included, and the like. Is also good.

[0118]

As is clear from the above description, in the document data clustering system of the present invention, the singular value decomposition of the document data is performed, and the information used for calculating the distance between the documents is gradually increased. By repeating clustering, even if the number of clusters is not determined in advance,
The number of clusters according to the clustering target can be determined, and a cluster in which only similar documents are aggregated can be generated.

Further, in the system provided with the label extracting means, a character string representing the contents of the clustered result can be labeled, and the clustering result can be shown in an easily understandable manner.

[0120] Further, by forming hierarchies between clusters generated by stepwise clustering, it is possible to generate a hierarchical relationship between clusters that matches human intuition.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the overall configuration of a document data clustering system according to a first embodiment;

FIG. 2 is a flowchart showing a clustering processing procedure of the document data clustering system according to the first embodiment;

FIG. 3 is a flowchart showing a processing procedure until a document similarity vector is created from a document database of the document data clustering system according to the first embodiment;

FIG. 4 is an example of a feature vector based on the appearance frequency of a word in a document in the document data clustering system according to the first embodiment;

FIG. 5 is an example of a feature vector based on a value obtained by dividing the frequency of appearance of the document data clustering system according to the first embodiment by a document length and normalizing the frequency;

FIG. 6 is an example of a feature vector based on a value obtained by dividing the appearance frequency of the document data clustering system according to the first embodiment by the sum of the appearance frequencies of words in a document and normalizing the divided frequency.

FIG. 7 is a description example of a cluster information table of the document data clustering system according to the first embodiment;

FIG. 8 shows an example of comparing documents assigned as a result of two clustering operations for a certain cluster in the document data clustering system according to the first embodiment;

FIG. 9 is an example of an updated cluster information table of the document data clustering system according to the first embodiment;

FIG. 10 is a conceptual diagram illustrating a change in a cluster of the document data clustering system according to the first embodiment;

FIG. 11 is a block diagram illustrating an overall configuration of a document data clustering system according to a second embodiment.

FIG. 12 is a flowchart illustrating a processing procedure of the document data clustering system according to the second embodiment;

FIG. 13 shows an example of a representative word vector of the document data clustering system according to the second embodiment;

FIG. 14 extracts a label candidate from a document of the document data clustering system according to the second embodiment,
An example of calculating the score,

FIG. 15 is a block diagram illustrating an overall configuration of a document data clustering system according to a third embodiment;

FIG. 16 is a flowchart illustrating a clustering processing procedure of the document data clustering system according to the third embodiment;

FIG. 17 shows an example in which documents assigned as a result of two clusterings are compared for a cluster of the document data clustering system according to the third embodiment;

FIG. 18 is an example of an updated cluster information table of the document data clustering system according to the third embodiment;

FIG. 19 is a conceptual diagram showing a change in a cluster of the document data clustering system according to the third embodiment;

FIG. 20 is a block diagram illustrating a configuration of a document data clustering system according to a fourth embodiment.

FIG. 21 is a diagram illustrating an example of a cluster size information table according to the fourth embodiment;

FIG. 22 is a diagram illustrating an example of a cluster hierarchy relation table according to the fourth embodiment;

FIG. 23 is a flowchart showing an operation procedure of the document data clustering system according to the fourth embodiment;

FIG. 24 is a flowchart illustrating a procedure for calculating a hierarchical relationship between clusters according to the fourth embodiment;

FIG. 25 is a flowchart illustrating a lower cluster calculation procedure according to the fourth embodiment.

[Explanation of symbols]

 101, 1101, 1901 Document database 102, 1102, 1902 Dictionary 103, 1103, 1903 Feature vector creation means 104, 1104, 1904 Feature vector set 105, 1105, 1905 Singular value decomposition means 106, 1106, 1906 Singular value decomposition result 107, 1107, 1907 Document similar vector creation means 108, 1108, 1908 Document similar vector set 109, 1909 Clustering target data selection means 110, 1910 Cluster creation means 111, 1109, 1911 Cluster information table 112, 1112, 1915 Result display means 1091 Main parent Clustering target data selection means 1110 Label extraction means 1111 Extracted labels 1912 Cluster hierarchy relation determination means 1913 Cluster size information table 1914 Cluster hierarchy relation table

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Takamasa Koyama 1006 Kazuma Kadoma, Kadoma-shi, Osaka Matsushita Electric Industrial Co., Ltd. F-term (reference) 5B075 ND03 NK06 NR12 PQ02 PR04 PR06 QM08 UU06

Claims (5)

[Claims] 1. A document data clustering system, comprising: a document database storing machine-readable document data; and a dictionary storing machine-readable words, wherein the document data clustering system clusters the documents stored in the document database. A feature vector creating means for creating a feature vector of the document stored in the document database based on the frequency of appearance of the word stored in the dictionary, and a set of feature vectors created by the feature vector creating means as singular values Singular value decomposition means for decomposing, document similarity vector creation means for creating a document similarity vector for calculating the similarity between documents from the result of the singular value decomposition, and document similarity created by the document similarity vector creation means By using a set of vectors, a cluster is formed for all or some documents in the document database. And cluster creation means for forming a cluster information table for storing information of cluster created,
Clustering data selecting means for selecting a document to be clustered by the cluster creating means from the document database with reference to the cluster information table, the cluster creating means, for a document to be clustered, Using the document similarity vector, the distance between the document and the cluster centroid is calculated, and for the same clustering target document, the number of dimensions of the document similarity vector used for the first clustering is increased within an appropriate range. Performing the second clustering, comparing the results of the second clustering, and discriminating a cluster with little change as a stable cluster, and the clustering data selecting means removes a document assigned to the stable cluster from a clustering target. Remove the cluster creation means A clustering target to be selected, and repeating this trial between the cluster creating means and the clustering data selecting means.
Clustering system. 2. A document data clustering system, comprising: a document database storing machine-readable document data; and a dictionary storing machine-readable words, wherein the document data clustering system clusters the documents stored in the document database. A feature vector creating means for creating a feature vector of a document stored in the document database based on the frequency of appearance of words stored in the dictionary, and a set of feature vectors created by the feature vector creating means as singular values Singular value decomposition means for decomposing, document similarity vector creation means for creating a document similarity vector for calculating the similarity between documents from the result of the singular value decomposition, and document similarity created by the document similarity vector creation means A cluster is formed for all or some documents in the document database by a set of vectors. And cluster creation means for forming a cluster information table for storing information of cluster created,
Clustering data selecting means for selecting a document to be subjected to clustering by the cluster creating means from the document database with reference to the cluster information table, wherein the cluster creating means, for a document to be clustered, Using the document similarity vector, the distance between the document and the cluster centroid is calculated, and for the same clustering target document, the number of dimensions of the document similarity vector used for the first clustering is increased within an appropriate range. Performing a second clustering, comparing the results of the two clusterings, and determining a cluster having a large change as an unstable cluster, and the clustering data selecting unit performs clustering on the documents assigned to the unstable cluster. Removed from the target, the cluster creation means A clustering target to be selected, and repeating this trial between the cluster creating means and the clustering data selecting means.
Clustering system. 3. The document database, the dictionary,
A label extracting unit that extracts a label for each cluster by referring to the singular value decomposition result and the cluster information table, wherein the label extracting unit includes a pseudo appearance frequency of the word at a center of gravity of the cluster. 2. A feature vector expressing the expression is calculated, and a character string appearing around a word having a high appearance frequency included in the feature vector is extracted as a label from a document assigned to the cluster. Or the document data clustering system according to 2. 4. A cluster hierarchical relationship determining means for setting a hierarchical relationship between clusters created by the cluster creating means, wherein the cluster hierarchical relationship determining means determines that any cluster C is a stable cluster. When the number of dimensions at the time of determination is defined as the stable dimension d (C) of the cluster, all documents belonging to the cluster C ′ whose stable dimension is lower than the stable dimension d (C) of the cluster C and the center of gravity of the cluster C When the distance in the d (C) dimension from g (C) is within a certain distance R (C), the cluster C
2. The document data clustering system according to claim 1, wherein the document data clustering system is hierarchized into clusters higher than the cluster C ′. 5. The cluster hierarchy relation determining means includes: each document belonging to a cluster C; and a center of gravity g (C) of the cluster C.
Is the largest distance among the distances in the d (C) dimension with R1
(C), and the minimum distance in the d (C) dimension between each document belonging to a cluster other than C whose stable dimension is d (C) or more and the center of gravity g (C) of the cluster C is R2.
5. The document data clustering system according to claim 4, wherein when (C) is used, a smaller one of R1 (C) and R2 (C) is set as R (C).