JPH11250102A - Information retrieving method and its device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a speed without deteriorating sorting precision by stopping the calculation of an evaluation function in the middle and executing the calculation of an evaluation value in clustering so as to reduce the calculation amount of the evaluation value. SOLUTION: U (C1 ) is obtained for all inputted clusters S=(C1 , C2 ,..., CN) (S202, 203). Evaluation values E (Ci , Cj ) between the combinations (Ci ,j ) of all the clusters are obtained to the middle (S204). E (Ci , Cj ) is continually obtained for combinations to a high-ordered t-number (t is a positive integer) (S205). The combination of clusters maximizing E (Ci , Cj ) is merged as that of the highest similarity. Then, a cluster Ck with the clusters Ci , Cj a slave node is prepared (S206). Thus, clustering is advanced toward the upper layer of the clusters by setting the cluster S to be S=S-Ci -j +Ck (S207).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information retrieval method and apparatus, and more particularly to an information retrieval method and apparatus for speeding up calculation of evaluation values in clustering using Bayesian clustering.

[0002]

2. Description of the Related Art FIG. 6 is a block diagram of a conventional information retrieval apparatus. Usually, an information search device that searches for document information held by a plurality of computers 63 connected to the Internet 61 is positioned as an information search server 62. The Internet 61 is connected to a huge number of computers 63 having further pages and clients 64 desiring to search. The information search server 62 includes a computer 63
Manages the page information which is the URL of the page held by the client, and the URL of the page which matches the information specified by the client 64.
L is provided as a search result.

The information search server 62 has a content database 621, a cluster database 622, and a control unit 623. Content database 6
21 stores a plurality of page information, and the cluster database 622 stores node information for clustering the page information.

Further, the control unit 623 includes a leaf node information selecting unit 623a, a partial cluster generating unit 623b, a recursive clustering unit 623c, and a page updating / searching unit 6.
23d. Leaf node information selecting means 623a
Is for selecting a predetermined number of optimal page information from a plurality of page information. The partial cluster generation unit 623b is for generating a cluster by assigning the remaining unselected page information to a leaf node similar to the cluster. Recursive clustering means 62
3c is for instructing the leaf node information selecting means 623a and the partial cluster generating means 623b to be repeated again toward the leaf node direction of the generated cluster. The page updating / searching unit 623d is for adding and updating page information to the generated cluster and for searching for page information from the cluster.

FIG. 7 is a flowchart showing a recursive clustering function for generating a cluster in a conventional information retrieval apparatus. This is similar to a recursive function (not shown) generally used to search the information structure of a binary tree, and the input (step 701) is a pointer to a node indicating a page set. When constructing a cluster, it is assumed that a root node to which all pages are assigned is input.

First, the number of pages assigned to the input node is determined (step 702). If the number of pages is equal to or greater than the predetermined maximum number of max, a cluster located below the node is generated. On the other hand, if the number of pages is less than the predetermined number of pages, the similarity is increased and clustering is performed for all rounds (step 70).
8).

The number of pages assigned to a node is ma
If x or more, the cluster generation function is called (step 702). This cluster generation function clusters pages located below the pointer of the input node. The output of this function is a pointer to the root node of the generated partial cluster.

Next, clustering is performed by recursively calling each leaf node of the generated cluster (step 704). First, it is determined whether there is a page assigned to a certain leaf node (step 705). If there is an assigned page, the self-function (step 706) is recursively called to advance clustering toward the lower layer of the cluster. Thereafter, the root node of the cluster obtained by the recursive clustering function is merged as a leaf node (step 707).

FIG. 8 is a flowchart showing the cluster generation function. This cluster generation function is divided into two processing stages: a leaf node selection stage and a partial cluster generation stage. The leaf node selecting step is to select a predetermined number of optimal pages that have a minimum code length when clustering is performed, from a plurality of pages. In the partial cluster generation step, the selected page is assigned as a leaf node and the remaining pages not selected are assigned to similar leaf nodes to complete the cluster.

First, from a plurality of pages assigned to the input node pointer (step 801),
Select max page sets P [t] (step 8)
03). The larger the max number, the larger the classification unit for one time. t is 1 each time a series of processing is repeated.
It is a number that increases by one. From the flow of the recursive clustering stage, at least max pages or more should have been allocated to the node when this cluster generation function is called.

Next, the selected page set P [t] is clustered by a known algorithm (step 8).
04). Since the clustering is performed by determining the similarity in the max number of round robins, the calculation amount does not significantly increase.

Then, for the cluster of the generated page set P [t], the remaining pages not selected are allocated to similar leaf nodes of the cluster (step 805).

Next, a code length L is added to the generated cluster.
[T] is obtained (step 806). In optimizing a set of information, MDL (Minimum Description Length creteri
on) Based on the criterion, the code length is selected so as to minimize the code length of the classification result. Here, the code length L is obtained as the sum of the information amount L _{1 of the} node required for the cluster and the code length L ₂ required for classification from the number of pages assigned to each leaf node.

The encoding of the binary tree itself is performed by visiting the tree in order of precedence, outputting 1 when the internal node is visited, and outputting 0 when the leaf node is visited. Information amount L ₁ of node, the number of leaf nodes (= max) (the k positive integer) k and the internal node number required to describe itself of the tree becomes L ₁ = 2k-1.

[0015] Therefore, if the probability that information is selected leaf node i the number of pages allocated to the leaf node i from n _i and all the pages and the _{p i = n i / Σ j} n j, each leaf node The code length necessary for classification is L ₂ = Σn _i log p _i from the number of pages assigned to. As a result, L ₁ + L ₂
Is obtained as the code length L of the cluster.

The code length L of the cluster obtained here is
Minimum code length L stored by previous iteration
Compare with _min . Is smaller than the minimum code length L _min of the obtained code length L [t] is stored, L [t] is stored as L _min (step 807).

By repeating these series of processes a predetermined number of times c (step 802), a page set P [t] having the minimum code length is selected. Since the selection of the page set is performed at random, the larger the number c is, the more the optimum page set can be selected.

In the cluster generation step, the page set P [t] selected in the leaf node selection step is clustered according to the similarity (step 808), and the remaining unselected pages are generated. Assigned to a similar leaf node of the cluster (step 809).
In this way, a cluster having the minimum code length L _min is generated.

In the conventional information retrieval apparatus based on the clustering method described above, when the number is equal to or more than a predetermined max, clustering is performed at a high speed with a slight decrease in similarity accuracy, and a predetermined m is obtained.
If the number is smaller than ax, the similarity is increased and clustering is performed on a brute force basis. For this reason, clusters can be generated while balancing generation time and similarity accuracy.

In such a conventional information retrieval apparatus,
As a similarity search method of a document, there is a statistical method of ranking search target documents according to a probability regarding compatibility between an input document and a search target document. As one of the methods, there is a method of Bayesian clustering a set of documents using the probability of occurrence of words in the document. This method is described in "Makoto IMAYAM
A, Takenobu TOKUNAGA, "Cluster-Based Text Categori
zation; A Comparisonof Categoly Search Strategie
s ", Proc. of the Annual International ACM SIGIR Co
nference on Research and Develpment in Information
Rctricval, pp.273-280, 1995 ", and the inventor's" Evaluation of Clustering Method for Large Number of Documents ", Information Processing Society of Japan 55th Annual Convention (late 1997),
3-208, 1997.

The Bayesian clustering in these documents regards a set of documents as a cluster and sequentially merges the clusters. When merging clusters, an evaluation function E (c _i , c _j ) having two clusters c _i , c _j to be merged as arguments is used, and E (c
_i, c _j) is determined a set of c _i, c _j that maximizes merges c _i, c _j.

According to the program list shown in the former document, E (c _i , c _j ) is calculated as follows.

E (c _i , c _j ) = M (c _i , c _j ) −U (c _i ) −U (c _j )

In the case of this calculation method, U (c _i ∪c _j ) = M (c
_{Since i} , c _j ) holds, U (c _i ) is calculated as M (c _i , c _j ). Here, M (c _i , c _j ) is obtained as follows.

1: function M (c _i , c _j ) 2: out = 0.0; 3: forall document d∈cluster c _i ∪c _j 4: tmp = 0.0; 5: forall word w∈d 6: tmp = tmp + rate (w, d, c _i ∪c _j ); 7: out = out + log (tmp); 8: return out; 9: endfunc

Where, rate (w, d, c) = (relative frequency of w in d) (relative frequency of w in c) / (relative frequency of w in all documents);

FIG. 9 is a flowchart showing this method.

The input (step 901) includes clusters c _i and c _j
Is a frequency table of documents included in. First, for each of the input clusters c _i and c _j (c), all words in the document d included in the clusters c _i and c _j (steps 902 and 903) are sorted by the relative frequency in all the documents (step 904-
906). After accumulating the values of these results, taking the log and accumulating the result, E (c _i , c _j ) is output (step 907,
908).

[0029]

In such a conventional method, most of the prior art algorithms include a loop (lines 5 and 6) inside M (c _i , c _j ). Time is spent. Then, it is necessary to calculate the value of rate for all the words in all the documents in the cluster in all the cluster pairs being generated, and it takes a lot of calculation time to process a large number of documents.

The present invention has been made to solve these problems, and can greatly reduce the amount of calculation of the evaluation value. In addition, the calculation of the evaluation function is more complicated than the calculation in a random order.
An object of the present invention is to provide an information retrieval method and apparatus that can perform high-speed processing without deteriorating classification accuracy by sorting and calculating based on relative frequencies in all documents.

[0031]

According to the present invention, a plurality of computers having document information are connected to a network, and a plurality of index information of the document information is stored according to the present invention. And a control unit for searching and updating the document information using the content database, wherein the control unit is a leaf node of a cluster from among the plurality of pieces of the document information. Leaf node information selecting means for selecting a predetermined number of pieces of information, partial cluster generating means for allocating the remaining unselected information to similar leaf nodes, and leaf node information selecting means and partial cluster generating means Recursive clustering means for instructing the cluster to be repeated in the direction of the leaf nodes; In an information search apparatus having a page updating / searching unit for adding and updating page information to a formed cluster and searching for page information from the cluster, the control unit stops calculation of an evaluation function halfway, and evaluates in clustering. It is characterized by having an evaluation value calculation means for performing value calculation. Further, the evaluation value calculating means may be provided in the partial cluster generating means and / or the recursive clustering means. According to the apparatus of the present invention having the above-described configuration, it is possible to realize an information retrieval apparatus that can increase the speed without deteriorating the classification accuracy.

Further, when a set of documents is regarded as a cluster, and when the clusters are sequentially merged, a combination of clusters that maximizes the evaluation function is obtained by using an evaluation function having the two clusters as arguments. In the information search method in the clustering using Bayesian clustering for merging the clusters, it is also characterized in that the calculation of the evaluation function is stopped halfway and the evaluation value is calculated in the clustering. Further, the calculation of the evaluation function is stopped halfway, and the calculation of the evaluation function is continuously performed for combinations up to a predetermined number. Further, when calculating the evaluation function, the calculation is performed by sorting in order of the relative frequency in all documents, the relative frequency in each document, and the relative frequency in each cluster. Therefore, the calculation amount of the evaluation value can be significantly reduced.

Therefore, according to the present invention, the amount of calculation of the evaluation value can be greatly reduced, and the evaluation function is sorted by the relative frequency in all the documents as compared with the calculation in a random order. By performing the calculation, it is possible to provide an information retrieval method and an information retrieval apparatus capable of increasing the speed without deteriorating the classification accuracy.

[0034]

Embodiments of the present invention will be described below with reference to the drawings. First, when the value of out does not increase so much during execution of the third to seventh lines of the Bayesian clustering calculation method in the conventional example, the value of M (c _i , c _j ) is small even if calculation is performed to the end, Although it is unlikely that the cluster pair c _i , c _j maximizes the evaluation function, it needs to be calculated in the conventional example.

Therefore, in the information search method according to the present invention, the evaluation functions M and E are used for all the words in the cluster.
Instead of calculating the evaluation function M, the value of the evaluation function E is predicted at a stage where the evaluation function M is obtained halfway, and the calculation of the evaluation function E is temporarily stopped for a cluster pair having a low estimated value of the evaluation function E. Then, the value of the evaluation function E may be calculated only for a cluster pair whose estimated value of the evaluation function E is higher than a predetermined value. Therefore, the evaluation function E is calculated using all the words in the document.
Instead of calculating

Sorting by relative frequency in a document, and a word set of a specified ratio (r) in descending order of frequency, or

Sorting by relative frequency in the cluster (c _i ), and a word set of a specified ratio (r) in descending order of frequency, or

Sorting by the relative frequency in the cluster (c _i ∪c _j ), and a word set of a specified ratio (r) in descending order of frequency, or

A set of words of a specified ratio (r) in descending order of frequency, sorted by relative frequency in all documents

Then, a combination having a high evaluation value is predicted.

However, in order to maintain the classification accuracy of the evaluation function E,
If the number of word types of each document in the cluster is equal to or smaller than the specified size (s), the evaluation value is calculated for all the words.

Next, an information retrieval apparatus according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration of an information search device according to an embodiment of the present invention. In the figure, the same components as those in FIG. 6 are denoted by the same reference numerals. As a different constituent requirement, 11 is an evaluation value calculating means, which calculates M (c _i , c _j ) halfway while leaving only the top t, and calculates the intermediate result m (i, c _j ) of M (c _i , c _j ). j, d) gives M (c _i ,
E (c _i , c _j ) is obtained from c _j ), U (c _i ), U (c _j ), and M
(c _i, c _j) seeking _{M (c i, c j)} , U (c i), from _{U (c j) E (c} i,
c _j ). The evaluation value calculation unit 11 may be included in the partial cluster generation unit 623b and / or the recursive clustering unit 623c.

Next, a case where the function for calculating the evaluation value E until the ratio of the number of words becomes r is sorted at the relative frequency in all documents will be described with reference to the flowchart shown in FIG.

The input (step 201) is a cluster set S, r, s, t. First, the input cluster S =
U (c _i ) is obtained for all clusters in (c ₁ , c ₂ ,..., C _N ) (steps 202 and 203). Finding all cluster combinations (c _i, c _j) the evaluation value E (c _i, c _j) between the halfway (step 204). E (c _i , c _j ) is continuously obtained for up to t (t is a positive integer) combinations (step 205). A cluster combination that maximizes E (c _i , c _j ) is merged with the highest similarity. Then, a cluster _ck having the clusters c _i and c _j as child nodes is created (step 206). Therefore, let the cluster S be S = S
Clustering proceeds toward the upper layer of the cluster as -c _i -c _j + c _k (step 207).

FIG. 3 shows the evaluation values in the case of sorting by relative frequency in all the documents. The horizontal axis shows the ratio of the number of word types, and the vertical axis shows the value of the evaluation function E. In this figure, five documents (d ₁ , d ₂ , d ₃ ,
Using d ₄ , d ₅ ), E (d ₁ , d ₂ ), E (d ₁ , d ₃ ), E (d ₁ , d ₄ )
, E (d ₁ , d ₅ ). As can be seen from the figure, E increases almost in proportion to the ratio of the number of words. This indicates that the final value of the evaluation function E can be accurately estimated by using a certain value of the arbitrary ratio r.

The case where the function for calculating the evaluation function E until the ratio of the number of words becomes the ratio r is sorted by the relative frequency of all the documents will be described below.

1: function newM (c _i , c _j ) 2: out = 0.0; 3: forall document d∈ (c _i ∪c _j ) {4: if (d The number of word types N (d)> s 5: N _x = rN (d) 6: else N _x = N (d); 7: Sort words in d by relative frequency in all documents; 8: tmp = 0.0; 9: for (w = 1; w ++; w <= N _x ) 10: tmp = tmp + rate (F (w), d, c _i ∪c _j ); 11: m (i, j, d) = tmp; 12: out = out + log (tmp); 13:} 14: return out; 15: endfunc

M (i, j, d) will be used later. F (w) is
The frequency of the w-th most frequent word.

Next, a function for calculating M (c _i , c _j ) for obtaining the evaluation value E (c _i , c _j ) until the ratio of the number of words becomes r becomes a relative frequency in all documents. The case where sorting is performed will be described with reference to the flowchart shown in FIG.

The input (step 401) is the clusters c _i , c _j
Is a frequency table of documents included in. First, the cluster c _i ∪c _j
The N _x if the document d each word type number N (d) is greater than s included in (step 402, 403) as rN (d) (step 404-406), the words in the number of types N (d)
There words in the document d is smaller than s N _x as N (d) sort by relative frequency in all documents (step 407),
Further, the above rate is calculated for the top N words of the document d (steps 408 and 409), and the intermediate values m (i, j, d) are stored after accumulating the result values (step 410).
The log is taken and accumulated to output M (c _i , c _j ) (steps 411 and 412).

After that, evaluation values are continuously obtained for the top t combinations specified from the evaluation values having the highest evaluation values, and the combination which is finally the maximum is obtained.

The functions used when obtaining the evaluation value are shown below.

1: function cntM (c _i , c _j ) 2: out = 0.0; 3: forall document d∈ (c _i ∪c _j ) {4: tmp = m (i, j, d); 5: if (the number of word types of d: N (d)> s) ｛6: for (w = rN (d) +1; w ++; w <= N (d)) 7: tmp = tmp + rate (F (w) , d, c _i ∪c _j ); 8:} 9: out = out + log (tmp); 10:} 11: return out; 12: endfunc

Next, a description will be given of a case in which evaluation values are successively obtained for the top t combinations specified from the one having the highest single evaluation value, and finally the maximum combination is obtained with reference to the flowchart shown in FIG. .

The input (step 501) is a cluster pair c _i , c _j having t higher evaluation values E ′ (c _i , c _j ). First, for each of the input clusters c _i , c _j (c) (steps 502, 503, 504), the number of types of words N (d) for each document d included in the clusters c _i , c _j is smaller than s. If there is, the intermediate result m (i, j, d) is used as it is, and the number of word types N
If (d) is greater than s (step 505), the frequency table F
For the remaining words F (N _x +1),..., F (d), rate is calculated and accumulated (steps 506 and 507). The values of the result are taken as a log and further accumulated to output M (c _i , c _j ) (steps 508 and 509).

The configuration of each embodiment described above is merely an example, and combinations of the embodiments are also possible, and the combination can be arbitrarily configured.
The embodiments described above are merely examples of the present invention and are not intended to limit the present invention. The present invention can be implemented in other modified forms and modified forms. Therefore, the scope of the present invention should be defined only by the appended claims and their equivalents.

[0057]

As described above in detail, according to the present invention, the amount of calculation of the evaluation value can be greatly reduced. In addition, in the calculation of M (c _i , c _j ) for obtaining the evaluation function, sorting accuracy is calculated by sorting by relative frequency in all documents, as compared with calculation in random order. It is also possible to increase the speed without deterioration.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an information search device according to the present invention.

FIG. 2 is a flowchart showing that sorting is performed at a relative frequency in all documents according to the present invention.

FIG. 3 is a characteristic diagram showing a relationship between a word count ratio and an evaluation value according to the present invention.

FIG. 4 is a flowchart showing a process in the case of performing the t top combinations specified from the one with the highest evaluation value in the present invention.

FIG. 5 is a flowchart showing a process in a case where a finally maximum combination is obtained in the present invention.

FIG. 6 is a block diagram showing a configuration of a conventional information search device.

FIG. 7 is a flowchart showing processing contents of a conventional recursive clustering function.

FIG. 8 is a flowchart showing processing contents of a conventional cluster generation function.

FIG. 9 is a flowchart showing a conventional Bayesian clustering method.

[Explanation of symbols]

 11 Evaluation value calculation means 61 Internet 62 Information search server 63 Computer 64 Client 621 Content database 622 Cluster database 623 Control unit 623a Leaf node information selection means 623b Partial cluster generation means 623c Recursive clustering means 623d Page update / search means

Claims (7)

[Claims] 1. A set of documents is regarded as a cluster, and when merging the clusters sequentially, a combination of clusters that maximizes the evaluation function is obtained using an evaluation function having two clusters as arguments. An information search method in clustering using Bayesian clustering for merging clusters, wherein the calculation of the evaluation function is stopped halfway and the evaluation value is calculated in clustering. 2. The information search method according to claim 1, wherein the calculation of the evaluation function is stopped halfway, and the calculation of the evaluation function is continuously performed for up to a predetermined number of combinations. 3. The information retrieval method according to claim 1, wherein when calculating the evaluation function, the relative frequency in all documents, the relative frequency in each document, and the relative frequency in each cluster are sorted and calculated in this order. . 4. A content database storing a plurality of computers having document information connected to a network and storing index information of the plurality of document information, and a control unit for searching and updating the document information using the content database. An information retrieval apparatus having: a control unit configured to select a predetermined number of pieces of information to be leaf nodes of a cluster from a plurality of pieces of the document information; And a recursive clustering means for instructing to be repeated in the direction of the leaf nodes of the cluster generated by the leaf node information selecting means and the partial cluster generating means,
In an information search device having a page update / search unit for adding and updating page information to a generated cluster and searching for page information from the cluster, the control unit stops calculation of an evaluation function halfway, An information retrieval device comprising an evaluation value calculation means for performing evaluation value calculation. 5. The information retrieval apparatus according to claim 4, wherein the evaluation value calculation means stops the calculation of the evaluation function in the middle and continuously calculates the evaluation function for up to a predetermined number of combinations. 6. The evaluation value calculation means, when calculating the evaluation function, sorts and calculates in order of relative frequency in all documents, relative frequency in each document, and relative frequency in each cluster. 4. The information search device according to 4. 7. The information retrieval apparatus according to claim 4, wherein said evaluation value calculation means is provided in said partial cluster generation means and / or said recursive clustering means.