JPH11259509A - Information retrieval and classification method and system therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To sufficiently reduce a calculation amount required for clustering and to retrieve and classify information at a high speed. SOLUTION: A retrieval technique using retrieval interrogation and a clustering technique are combined to retrieve and classify information. Especially, by a term correlation extraction part 1d, knowledge on the correlation of terms is extracted beforehand from a set of documents inside a document file 10a (a correlation degree is imparted to a term highly correlated to the term for respective terms) and registered in a term correlation file 10d. Then, in the case of retrieving the document corresponding to the retrieval interrogation of a user, terms highly correlated to an input term are extracted from the term correlation file 10d and clustered based on the correlation degree, and documents retrieved corresponding to the retrieval interrogation are further clustered based on the cluster of the terms.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for searching and classifying information by clustering, and more particularly to an information search and classification method and an information search and classification system suitable for high-speed classification of search results.

[0002]

2. Description of the Related Art With the advance of computer (computer) technology, an information retrieval system for retrieving a document meeting a user's information request from a database storing a large amount of documents has been used in various fields. . In such an information search system, it is generally necessary for a user to input a search question using a term indicating an information request.

[0003] However, it is not always easy for a user to recall a term that accurately represents an information request. In addition, the user often comes up with a few terms at most. Therefore, in a system in which a large number of documents are stored, the number of documents to be searched is considerably large, and some of the documents are compatible documents and some are non-conforming documents.

[0004] In order to deal with such a problem, there is also a system which calculates the relevance of a search query and a document and outputs the relevance in descending order of relevance. However, not all conforming documents will be ranked higher. Therefore, the user load for selecting a suitable document from the retrieved documents is extremely large.

[0005] Instead of such a system responding to a search question, a system based on a new concept of searching for a document that meets an information request while browsing the entire database has been proposed. This technique is suitable for situations where the user has an information request but the content is not always clear.
g, et al., "Scatter / Gather: A Cluster-based Appro
ach to Browsing Large Document Collections, "Proce
edings of the 15th International Conference on Inf
ormation Retrieval, pp. 318-329 (1992 June, Copen
hagen) is an example.

In this technique, the system clusters documents stored in a database based on the similarity between documents and presents the clustered documents to a user. The user selects a cluster that is likely to be relevant to the information request. Based on this selection, the system performs further clustering on the documents of the selected cluster and presents the result to the user. Through such an interaction, it is possible to narrow down to clusters containing documents that meet the information request.

However, this technique has the following problems. That is, starting from the clustering of the entire database, initially, it is often the case that only clusters that include quite various documents and have unclear characteristics are obtained. Also, interactivity is the essence of this technology, but it is not easy to achieve a response time that does not compromise it.

[0008] The amount of calculation in the clustering process is as follows. n elements process clustering (documentation here) generally including similarity calculation for all element pairs, the computational complexity of the order of n ^2.
In order to reduce the amount of calculation, in Scatter / Gather, n
A first step of clustering m elements selected at random from n elements to generate a cluster seed without clustering the elements collectively; and (nm) elements by There is proposed a technique of dividing into a second step of distributing to the closest cluster.

By doing so, the first step has a calculation amount of the order of m ² , and the second step has a calculation amount of the order of (nm) × c. The amount of calculation is small. Here, c is the number of cluster types. However, in this technique, m must be a value proportional to n in order to obtain a good clustering result. Therefore, it is not a sufficient solution in terms of real-time document clustering.

[0010]

The problem to be solved is that the amount of calculation required for clustering cannot be sufficiently reduced with the conventional technology. An object of the present invention is to solve the problems of the conventional technology and to provide an information search / classification method and an information search / classification system capable of high-speed information search / classification.

[0011]

In order to achieve the above object, an information retrieval / classification method and an information retrieval / classification system according to the present invention are configured to perform retrieval and classification of information by combining a retrieval technique based on a retrieval query and a clustering technique. In particular, knowledge relating to term correlation is extracted in advance from a set of documents stored in a document file and registered in advance (for each term, a term having a high correlation with this term is given a degree of correlation). When searching a document in response to a search query entered by a user using terms, terms having a high correlation with the terms input by the user are extracted from terms registered in advance, and the degree of correlation between the terms is extracted. , And further, based on the cluster of the term, cluster the documents searched in response to the search query. Then display this document cluster and wait for the user's selection,
It repeats term clustering and document clustering for the cluster selected by the user. As described above, starting from the user's search question, the cycle of clustering and cluster selection of the related documents searched is repeated based on the term used in the search question and the correlation term, so that the processing time required for clustering is reduced. Thus, the user can efficiently reach the matching document.

[0012]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of the configuration of the information search and classification system of the present invention according to the present invention, and FIG. 2 is a block diagram showing an example of the configuration of a computer constituting the information search and classification system in FIG. It is.

As shown in FIG. 2, the information retrieval and classification system of the present embodiment includes a processing unit 21 having a CPU (Central Processing Unit) for performing computer processing according to a storage program system, a storage unit 22 including a RAM, and a hard disk. The document to be searched and classified stored in
1, a document input device 23, an input device 24 such as a keyboard and a mouse, and a CRT (Cathode Ray).
Tube) and the like.

The processing device 21 executes the processing of the storage subsystem 1 and the search subsystem 2 in FIG.
The storage device 22 stores the document files 10a,
Document characteristic file 10b, index file 10c,
And the term correlation file 10d is stored. The document input device 23 is a device for inputting document data, and in addition to a hard disk drive, a CD-ROM drive, a floppy disk drive, or the like is used depending on the medium from which the document data is obtained. With the input device 24 and the display device 25, the user performs an interactive operation with the search subsystem.

In the processing unit 21 of the computer having the above-described structure, a program in which the procedure of the information search and classification method according to the present invention is recorded on an optical disk via a CD-ROM drive or the like is loaded. Each processing unit (document registration unit 1a, document feature extraction unit 1b, index file generation unit 1c, term correlation extraction unit 1d) in storage subsystem 1 and related term selection unit 2a, term cluster generation unit 2b in search subsystem 2 Modules such as a document search unit 2c, a document cluster generation unit 2d, and a document cluster display control unit 2e are configured.

Hereinafter, the configuration of the information retrieval / classification system of this embodiment will be described with reference to FIG. In FIG. 1, reference numeral 1 denotes a storage subsystem for generating and storing files used for information search and classification in advance, and 2 performs information search and classification processing using each file stored in the storage subsystem 1. The search subsystem.

The storage subsystem 1 includes a document registration unit 1a,
It has a document feature extraction unit 1b, an index file generation unit 1c, and a term correlation extraction unit 1d.
Document file 10a and document feature file 10 respectively
b, index file 10c, term correlation file 1
0d is registered. Before describing the processing of each unit (document registration unit 1a, document feature extraction unit 1b, index file generation unit 1c, term correlation extraction unit 1d) of storage subsystem 1, FIG.
This will be described with reference to FIG.

FIG. 3 is an explanatory diagram showing an example of the record configuration of the document file in FIG. 1. FIG. 4 is an explanatory diagram showing an example of the record configuration of the document characteristic file in FIG.
6 is an explanatory diagram showing an example of a record configuration of the index file in FIG. 1, and FIG. 6 is an explanatory diagram showing an example of a record configuration of the term correlation file in FIG.

As shown in FIG. 3, the document file 10a
Is the document ID 31 and the text data 3 which is the content of the document.
2 is included. This file has a structure in which a record having the document ID can be directly accessed using the document ID as a key. A file with such a structure
It can be realized by various techniques such as a method of creating a B-tree index and a method by hashing, and these techniques are well-known, and thus description thereof is omitted.

As shown in FIG. 4, the document characteristic file 10
b includes a record including a document ID 41 and a term vector 42 of the document. The term vector 42 is a list of pairs of terms 42a and weights 42b of the terms. This file has a structure that allows direct access to records using the document ID 41 as a key.

As shown in FIG. 5, the index file 1
0c is composed of a record including the term 51 and the term appearance document information 52. The term appearance document information 52 includes a document ID 52a and a weight 52 of the term 51 in the document.
b is a list of pairs. This is a file having a structure in which records can be directly accessed using the term 51 as a key.

As shown in FIG. 6, the term correlation file 1
0d is composed of a record including the term 61 and the correlation term information 62. The correlation term information 62 is a term 62
It is a list of a set of a and a correlation 62b. This is a file having a structure in which records can be directly accessed using the term 61 as a key.

Next, the processing of each unit (document registration unit 1a, document feature extraction unit 1b, index file generation unit 1c, term correlation extraction unit 1d) of the storage subsystem 1 in FIG. 1 will be described. The document registration unit 1a is the document input device 2 of FIG.
3 is read one by one, a document ID is assigned, and the document ID is output to the document file 10a together with the text data of the document.

The document feature extraction unit 1b includes a document file 10
Each document in a is read, terms and their appearance frequencies are obtained from the text data, and the weight of each term in each document is calculated based on the result. Then, for each document, select a term whose weight is equal to or greater than a predetermined threshold,
The weight is output to the document feature file 10b together with the weight. Here, the term weight is tf−idf (term frequency−inver
The calculation is based on the concept of se document frequency.

That is, the weight of the term tj in the document di
Calculate w (di, tj) by the following equation. w (di, tj) = fi, j × log (N / nj) Here, fi, j is the appearance frequency of the term tj in the document di,
N is the number of documents in the document file, and nj is the number of documents in which the term tj has appeared.

The index file generator 1c reads the document characteristic file 10b and generates an inverted file. That is, the document feature file 10b stores a list of terms and sets of weights of the terms in the document for each document, whereas the index file generating unit 1c stores a document and a document in the document for each term. The list of the weight set of the term is stored. Such a record is created and output to the index file 10c.

The term correlation extraction unit 1d outputs the document file 1
The document in 0a is read, a set of terms that co-occur and the frequency of co-occurrence are obtained from the text data 32, and the degree of correlation between the terms is calculated based on the result. For each term,
A term having a degree of correlation equal to or greater than a predetermined threshold is selected and output to the term correlation file 10d together with the degree of correlation.

Co-occurring terms are extracted based on the definition of window co-occurrence. That is, the term t in the text
When the term t ′ appears at a position within W words from the appearance position of, it is considered that t and t ′ co-occur. W is the size of the window, whose value is determined in advance. While moving the window along the text, a set of co-occurring terms is extracted and its frequency is counted.

The mutual information is calculated as the degree of correlation between the terms. That is, the degree of correlation α (ti, tj) between the term ti and the term tj is calculated by the following equation (Equation 1).

(Equation 1) Here, f (ti) is the appearance frequency of the term ti, and g (ti, tj) is the co-occurrence frequency of the term ti and the term tj.

Next, the search subsystem 2 in FIG. 1 will be described with reference to FIG. FIG. 7 is a block diagram showing a detailed configuration example of the search subsystem in FIG. The search subsystem of this example is a search question reading unit 2
f, a document search unit 2c, a related term selection unit 2a, a term cluster extraction unit 2b, a document cluster generation unit 2d, a document cluster display control unit 2e, a cluster number reading unit 2g, and a term set / document set reduction unit 2h. ing.

The search query reading unit 2f and the cluster number reading unit 2g start processing after waiting for a user input from the input device 24 in FIG. In addition, related term selection section 2a,
The term cluster extraction unit 2b, the document search unit 2c, the document cluster generation unit 2d, the document cluster display control unit 2e, and the term set / document set reduction unit 2h start processing when necessary data is received from another unit.

Hereinafter, each part of these search subsystems (related term selection section 2a, term cluster extraction section 2b,
Before describing the processes of the document search unit 2c, the document cluster generation unit 2d, the document cluster display control unit 2e, the search query reading unit 2f, the cluster number reading unit 2g, and the term set / document set reduction unit 2h), (Search query data 10e, search document set data 10f,
Related term set data 10g, term cluster data 1
0h, the document cluster data 10i, and the cluster number data 10j) will be described with reference to FIGS.

FIG. 8 is an explanatory diagram showing an example of the structure of the search question data in FIG. 7, FIG. 9 is an explanatory diagram showing an example of the structure of the search document set data in FIG. 7, and FIG. FIG. 11 is an explanatory diagram showing an example of the configuration of term cluster data in FIG. 7, FIG. 12 is an explanatory diagram showing an example of the configuration of document cluster data in FIG. 7, and FIG. FIG. 13 is an explanatory diagram showing a configuration example of the cluster number data in FIG.

As shown in FIG. 8, the search question data 10e
Is one logical operator 81 and one or more terms 82
Consists of The value of the logical operator 81 is “AN
D "," OR ", or blank. As shown in FIG. 9, the search document set data 10f includes zero or more document IDs.
91.

As shown in FIG. 10, the related term set data 10g is composed of zero or more terms 101.
As shown in FIG. 11, the term cluster data 10h is
It consists of records corresponding to each of L clusters.
Each record has a cluster number 111 and one or more terms 1
12 is comprised.

As shown in FIG. 12, the document cluster data 10i is composed of records corresponding to each of L clusters, and each record has a cluster number 121,
It is composed of one or more sets of a document ID 122 and a similarity 123. As shown in FIG.
j is composed of one or more cluster numbers 131.

Next, the processing operation of each unit in the search subsystem shown in FIG. 7 for such data will be described.
The search question reading unit 2f reads the search question from the input device 24 in FIG. 2, and outputs values to the logical operator 81 and the term 82 of the search question data 10e. In this example, the format of the search question is limited to three types: one term, logical product of plural terms, and logical sum of plural terms. The values of the logical operator 81 and the term 82 in each case are as follows.

When the search query is one term, that is, when q = t ₁ , the logical operator 81 is blank and t ₁
Is set in the term 82. When the search query is a logical product of a plurality of terms, that is, when q = t ₁ ∧t ₂ ∧ ‥ ∧tp, the logical operator 81 is AND, and t ₁ , t ₂ , ‥, tp
Is set in the term 82. When the search query is a logical sum of a plurality of terms, that is, when q = t ₁ ∨t ₂ ∨ ‥ ∨tp, the logical operator 81 is OR, and t ₁ , t ₂ , ‥, tp is set in the term 82. I do.

The document retrieval section 2c retrieves the retrieval question data 10e
Is read, and the record of the index file 10c is searched using the term 82 as a key, and the relevance between the document whose document ID is described in the term appearance document information 52 of FIG. 5 and the search query is calculated. Is output to the search document set data 10f.

Relevance R between document (d) and search query (q)
(d, q) is calculated as follows according to the value of the logical operator 81. If the logical operator is blank, ie q
= For _{t 1, R (d, q} ) = w (d, t 1) logic if the operator of the AND, i.e., q = t ₁ ∧t ₂ ∧
{In the case of tp, R (d, q) = min {w (d, t ₁ ), w (d, t ₂ ), ‥, w (d, tp)} When the logical operator is OR, that is, q = t ₁ ∨t ₂
In the case of ‥ ∨tp, R (d, q) = max {w (d, t ₁ ), w (d, t ₂ ), ‥, w (d, tp)} where w (d, t) The values are as follows:

The document ID 52a in the record of the index file 10c searched using the term t as a key
If (d) is included, the corresponding weight 52b is set to the value of w (d, t). If the document ID 52a in the record of the index file 10c searched using the term t as a key does not include the document d, the value of w (d, t) is set to 0.

The determination of the retrieval document set based on the relevance R (d, q) obtained in this manner is performed by previously defining the upper limit M of the size of the retrieval document set and the lower limit θ _{1 of the} relevance. . That is, within a range that satisfies R (d, q)> θ 1, R (d,
Select up to M documents in descending order of q). The document IDs of the selected documents d ₁ , d ₂ ,.
Output as f.

The related term selecting section 2a reads the search question data 10e, searches for a record in the term correlation file 10d using the term 82 in FIG. 8 as a key, and
, The degree of correlation between the term included in the correlation term information 62 and the search query is calculated, and a term having a high degree of correlation is output to the related term set data 10g.

Correlation degree A between term (t) and search query (q)
(t, q) is calculated as follows according to the value of the logical operator 81 in FIG. If the logic operator is blank, that is, when the _{q = t 1, A (t} , q) = α (t, t 1) logic if the operator of the AND, i.e., q = t ₁ ∧t ₂ ‥
In the case of ∧tp, A (t, q) = min {α (t, t ₁ ), α (t, t ₂ ), ‥, α (t, tp)} When the logical operator is OR, that is, q = t ₁ ∨t ₂ ‥ ∨
In the case of tp, A (t, q) = max {α (t, t ₁ ), α (t, t ₂ ), {, α (t, tp)}

Here, the value of α (t, ti) is as follows. Term 6 in the correlation term information 62 of the record of the term correlation file 10d searched using the term ti as a key
If the term t is included in 2a, the corresponding correlation 62
Let b be the value of α (t, ti). If the term t is not included in the term 62a in the correlation term information 62 of the record of the term correlation file 10d searched using the term ti as a key, the value of α (t, ti) is set to 0.

The determination of the related term set based on the correlation A (t, q) is performed by previously defining the upper limit K of the size of the related term set and the lower limit θ _{2 of the} correlation. That is, A
(t, q)> θ ₂ in a range satisfying, A (t, q) selects the maximum of K terms in descending order of. The selected terms t ₁ , t ₂ ,
‥, tk are output as related term set data 10g.

The term cluster extraction unit 2b reads the related term set data 10g, and further calculates the degree of correlation between the terms belonging to the related term set by using the term correlation file 10g.
Read from d. Based on this, the related term set is clustered, and the obtained cluster is output as term cluster data 10h. In this example, a group averaging method, which is one of the coherent clustering methods, is used as a clustering method. The number of clusters to be generated is a predetermined value L.

The related term set to be clustered is {t
₁ , t ₂ , {, tk}. At this time, L
Generate clusters. First, k clusters C ₁ = {t ₁ }, C ₂ = {t ₂ }, ‥, and Ck = {tk} are created as an initial state. Then, the following processing is repeated until the total number of clusters becomes equal to L. A pair of clusters having the highest degree of correlation is selected and merged into one cluster.

Here, the degree of correlation αc (C, D) between clusters C and D
Is defined by the following equation (Equation 2). Here, α (t, u) is the degree of correlation between the two terms t, u.

(Equation 2)

The document cluster generation unit 2d reads out a term vector from the document feature file 10b for each document whose document ID is described in the search document set data 10f, and determines the similarity with each cluster in the term cluster data 10h. Is calculated. A document cluster using the term cluster as a seed is generated by assigning each document to a term cluster having the highest similarity. The result is output to the document cluster data 10i.

The similarity between a term vector of a document and a term cluster is calculated as follows. First, the term vector of the document d (the term vector 42 in FIG. 4) read from the document feature file 10b is converted into the following vector expression d for calculating the similarity. Here, it is assumed that the related term set is {t ₁ , t ₂ , ‥, tk}. Document d
If term vector 42 includes term ti, its weight is used as the value of the i-th element of vector d . If the term vector 42 of the document d does not include the term ti, the value of the ith element of the vector d is set to 0.

Next, the vector expression c of the term cluster C
Is created as follows. Again, the related term set
{t ₁ , t ₂ , ‥, tk}. Term Cluster C
Contains the term ti, the value of the ith element of the vector c is set to 1. The term cluster C is term t
If i is not included, the value of the i-th element of the vector c is set to 0
To The two vectors d and c created in this way
, The similarity ρ (d, C) between the document d and the term cluster C is calculated by the following equation. ρ (d, C) = d × c / (| d | × | c |)

The document cluster display control unit 2e outputs a digest to each of the document clusters in the document cluster data 10i to the display device 25 in FIG. The digest is a list of terms characterizing the document cluster and a document representing the document cluster. As the former, a record of the term cluster data 10h having the same cluster number as the document cluster is output. As the latter, a document having a document ID 122 corresponding to the maximum value of the similarity 123 in FIG. 12 in the record of the document cluster of the document cluster data 10i is adopted.

The text data of the document (text data 32 shown in FIG. 3) is read from the document file 10a, and the first part (for a predetermined number of characters) is output. FIG. 14 shows an example of the document cluster digest output in this manner. FIG. 14 is an explanatory diagram showing a display example of the document cluster digest displayed by the document cluster display control unit in FIG. In this example, the search and classification results when “semiconductor” is designated as the term in the search question data are shown. For example, as the cluster 1, each term cluster (“Japan-US”, “Trade”,...), And document 1 and document 2 are displayed as document clusters together with part of their contents.

Returning to the description of FIG. 7, the cluster number reading unit 2g reads the cluster number input by the user from the input device 24 of FIG. 2, and sets a value in the cluster number data 10j. The term set / document set reduction unit 2h converts the cluster number data 10
The record having the same cluster number as the cluster number in j is read. Then, the union of the terms described in the read record is obtained and output to the related term set data 10g. Similarly, a record having the same cluster number as the cluster number in the cluster number data 10j is read from the document cluster data 10i, the union of the document IDs written in the read record is obtained, and the search document set data 10f Output to

The processing of each unit in the search subsystem has been described above. The entire search subsystem performs the following processing. When the user inputs a search query from the input device 24 of FIG. 2, the search query reading unit 2f, the document search unit 2c, the related term selection unit 2a, and the term cluster extraction unit 2
b, the document cluster generation unit 2d and the document cluster display control unit 2e operate sequentially to classify the documents having a high degree of relevance with the search query into a plurality of clusters, and display the result of the classification into a plurality of clusters.
To be displayed.

When the user selects a cluster likely to be related to the information request from the displayed clusters and inputs the number from the input device 24, the cluster number reading unit 2
g, the term set / document set reduction unit 2h, the term cluster extraction unit 2b, the document cluster generation unit 2d, and the document cluster display control unit 2e sequentially operate to classify the selected cluster into smaller clusters. The result is displayed on the display device 25. The user can search for a suitable document by repeating such a process of selecting a cluster and subclassifying as necessary.

The calculation amount of the clustering is as follows. The processing unit introduced in this example for clustering the searched documents includes a related term selection unit 2a, a term cluster extraction unit 2b, and a document cluster generation unit 2d.
It is. The calculation amount of the related term selecting unit 2a is on the order of t, where t is the number of terms included in the search query. The amount of calculation of the term cluster extraction unit 2b is as follows.
The number of terms of the relevant term set is of the order of k ² and the k. Further, the calculation amount of the document cluster generation unit 2d is:
If the number of documents in the retrieval document set is m and the number of term clusters is L, the order is m × L.

Here, the values of t and L are much smaller than m. Further, k may be set to a constant value, which can be sufficiently smaller than a value that should be allowed as the number m of documents in the search document set. Moreover, the amount of calculation by the term cluster extraction unit 2b is k ²
Is small. This is because the degree of correlation between terms has been calculated and stored in the term correlation file 10d. From the above, it is clear that the calculation amount of the document clustering of this example is smaller than the method of directly clustering the retrieval document set (the calculation amount is on the order of m ² ).

The advantages of this embodiment are as follows. When the document cluster is generated, a list of terms characterizing the document cluster has already been obtained. Therefore, it is not necessary to separately perform a process of creating a digest from a document cluster.

FIGS. 15 and 16 are flowcharts showing one embodiment of the information retrieval / classification method of the present invention. First, in FIG. 15, the following processing is performed as a storage subsystem. The term appearing in each document in the document file 10a and the frequency of appearance are extracted by the document feature extraction unit 1b in FIG. 1 (step 1501), and the weight of the term in each document is calculated based on the term (step 1501).
2) A term vector having a set of terms and weights whose weight is equal to or greater than a predetermined value as an element is output to the document feature file 10b (step 1503). Then, the term correlation extraction unit 1d extracts a set of co-occurring terms and a co-occurrence frequency from the document in the document file (step 150).
4), calculate the correlation between terms based on the co-occurrence frequency, and output the result to the term correlation file 10d (step 15).
05).

After accumulating the data in this way, FIG.
At 6, the following processing is performed as a search subsystem. When the user inputs a search question (step 150)
6), this is read by the search question reading unit 2f in FIG. 7 (step 1507). Next, the document search unit 2c
With reference to the index file 10c, the relevance between the search query and the document is calculated, a document having a high relevance is selected, and the selected document ID is output as the search document set data 10f (step 1508). Further, the related term selecting unit 2a refers to the term correlation file 10d to select a term having a strong correlation with the search query, and outputs the term as the related term set data 10g (step 150).
9).

Further, the term cluster extraction unit 2b clusters the related term set data 10g with reference to the term correlation file 10d, extracts a plurality of term clusters composed of mutually strongly correlated terms, and extracts the term cluster data 10h. (Step 15
10). Then, the term cluster is read by the document cluster generation unit 2h from the document feature file 10b for each document in the search document set data 10f (step 1511), and the similarity between the term vector and each term cluster is calculated (step 1512). By assigning a document to the term cluster having the highest similarity, a document cluster using the term cluster as a seed is generated (step 1513).

Then, for each of the generated document clusters, the document cluster display control section 2e displays the seed term cluster and the document having the highest similarity to the term cluster (step 1514). The user sees this display and selects one or several clusters likely to be relevant to the information request (step 1).
515).

Then, the cluster number reading unit 2g reads the cluster number selected by the user (step 1516), and the term set / document set reduction unit 2h reads
The union of the term clusters that became the seed of the selected document cluster is obtained, the related term set data 10g is replaced with this (step 1517), and the union of the selected document clusters is obtained, and the search document set is obtained. Data 10f
Is replaced with this (step 1518). Thereafter, the process returns to step 1510, and the clustering using the related term set data and the search document set data thus replaced is repeated.

As described above with reference to FIGS. 1 to 16, in the information search and classification system and the information search and classification method of the present embodiment, the whole is divided into the storage subsystem 1 and the search subsystem 2, As shown in FIG. 1, the system 1 includes a document feature extraction unit 1b that extracts a term vector characterizing the content of each document in the document file 10a, and analyzes documents in the document file 10a to obtain knowledge on term correlation. It is configured to include a term correlation extracting unit 1d that extracts and stores the extracted term correlation file 10d in the term correlation file 10d.

As shown in FIG. 7, the search subsystem 2 is a search question reading unit 2 for reading a user's search question.
f, a document search unit 2c for searching for a document strongly related to the read search query, a related term selection unit 2a for selecting a term having a strong correlation with the search query as a related term with reference to the term correlation file 10d, and a term correlation A term cluster extraction unit 2b that extracts a plurality of term clusters from a set of related terms by referring to a file, calculates the similarity between the term vector of the retrieved document and the extracted term cluster, and makes the document most similar A document cluster generating unit 2d that generates a document cluster by assigning the term cluster to a term cluster having a large degree, a term cluster that has become a seed of the document cluster as a digest of the generated document cluster,
A document cluster display control unit 2e for displaying (part of) a document having the highest similarity with the term cluster; a cluster number reading unit 2g for allowing the user to select one or some of the document clusters; The term set / document set reduction unit 2h replaces the set of related terms with the union of the term clusters used as the seeds of the selected document clusters, and replaces the set of search documents with the union of the selected document clusters.

With such a configuration, in the information search and classification system of the present example, when a user inputs a search query, a document having a high degree of relevance to the search query is searched, and a cluster consisting of documents having high similarity to each other is searched. To display the digest of the cluster. Then, from the displayed cluster,
When the user selects a cluster that is likely to be relevant to the information request, the process is returned, a smaller cluster is extracted from the selected cluster, and the digest is displayed.

As described above, by combining the term-based search and clustering, starting from the search question recalled by the user, the cycle of clustering and cluster selection is repeated, and the cluster including the conforming document is narrowed down step by step. As a result, the user can efficiently reach the matching document. Further, high-speed clustering that is not so affected by the number of retrieved documents is possible, and does not impair interactivity unlike the related art. still,
The present invention is not limited to the embodiment described with reference to FIGS. 1 to 16, and can be variously modified without departing from the gist thereof.

[0070]

According to the present invention, the amount of calculation required for clustering can be sufficiently reduced, high-speed information search and classification can be performed, and the user can obtain desired information without feeling the stress of interactive operation. Information can be searched efficiently.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a configuration according to the present invention of an information search and classification system of the present invention.

FIG. 2 is a block diagram showing a configuration example of a computer constituting the information search and classification system in FIG.

FIG. 3 is an explanatory diagram illustrating an example of a record configuration of a document file in FIG. 1;

FIG. 4 is an explanatory diagram showing an example of a record configuration of a document feature file in FIG. 1;

FIG. 5 is an explanatory diagram showing an example of a record configuration of an index file in FIG. 1;

FIG. 6 is an explanatory diagram showing an example of a record configuration of a term correlation file in FIG. 1;

FIG. 7 is a block diagram showing a detailed configuration example of a search subsystem in FIG. 1;

FIG. 8 is an explanatory diagram showing a configuration example of search question data in FIG. 7;

FIG. 9 is an explanatory diagram showing a configuration example of search document set data in FIG. 7;

FIG. 10 is an explanatory diagram showing a configuration example of related term set data in FIG. 7;

FIG. 11 is an explanatory diagram showing a configuration example of term cluster data in FIG. 7;

FIG. 12 is an explanatory diagram showing a configuration example of document cluster data in FIG. 7;

FIG. 13 is an explanatory diagram showing a configuration example of cluster number data in FIG. 7;

14 is an explanatory diagram illustrating a display example of a document cluster digest displayed by the document cluster display control unit in FIG. 7;

FIG. 15 is a storage subsystem processing part of a flowchart showing one embodiment of the information search / classification method of the present invention.

FIG. 16 is a search subsystem processing part of a flowchart showing an embodiment according to the information search and classification method of the present invention.

[Explanation of symbols]

1: Storage subsystem, 1a: Document registration unit, 1b: Document feature extraction unit, 1c: Index file generation unit, 1d:
Term correlation extraction unit, 2: search subsystem, 2a: related term selection unit, 2b: term cluster generation unit, 2c: document search unit, 2d: document cluster generation unit, 2e: document cluster display control unit, 2f: search query Reading unit, 2g: cluster number reading unit, 2h: and term set / document set reduction unit, 10a: document file, 10b: document feature file, 10c: index file, 10d: term correlation file, 10e: search query data, 10f : Search document set data, 10g: Related term set data, 10
h: term cluster data, 10i: document cluster data, 10j: cluster number data, 21: processing device, 2
2: storage device, 23: document input device, 24: input device,
25: display device, 31: document ID, 32: text data, 41: document ID, 42: term vector, 42a:
Term, 42b: weight, 51: term, 52: term appearance document information, 52a: document ID, 52b: weight, 61:
Term, 62: correlation term information, 62a: term, 62
b: correlation degree, 81: logical operator, 82: term, 9
1: Document ID, 101: Term, 111: Cluster number, 112: Term, 121: Cluster number, 122:
Document ID, 123: similarity, 131: cluster number.

Claims (12)

[Claims] 1. An information search and classification apparatus for searching and classifying a document strongly related to an input search query from a set of documents stored in a document file based on a relationship with a term included in the search query. A method for extracting and registering information relating to term correlation from a set of documents stored in the document file in advance, and a set of documents searched as being strongly related to the search query, And a second step of generating a document cluster using the information related to the term correlation, and outputting a search classification result for each of the document clusters. 2. The information retrieval and classification method according to claim 1, wherein the generation of the document cluster in the second step is repeated for documents included in a document cluster selected by a user from the output document cluster. An information search and classification method characterized by the following. 3. An information search and classification system for searching and classifying a document strongly related to an input search query from a set of documents stored in a document file based on a relation with a term included in the search query. A first step of extracting and registering information relating to term correlation from a set of documents stored in the document file in advance, and using the information relating to term correlation to be included in the search query. A second step of selecting a term related to the selected term and generating a term cluster from the selected set of terms; and a set of documents searched as being strongly related to the search query using the term cluster as a seed. And a third step of generating a document cluster from the above, and outputting a search classification result for each document cluster. 4. The information search and classification method according to claim 3, wherein the second step relates to a term related to a term included in the search query and relates to a correlation of the terms extracted in the first step. An information search and classification method, wherein the term cluster is generated based on information and the term cluster is generated from a set of the selected terms based on information relating to the correlation of the terms. 5. The information retrieval and classification method according to claim 3, wherein the third step comprises: setting a set of terms previously associated with the retrieved document with the term An information search and classification method, wherein a similarity to a cluster is determined, and the document cluster is generated by assigning the document to a term cluster having the highest similarity. 6. The information retrieval and classification method according to claim 3, wherein a document included in a document cluster selected by a user from the output document clusters,
And generating the term cluster using information relating to the correlation of the terms in the second step for the term class serving as the seed of the document cluster selected by the user, and the term in the third step An information retrieval and classification method characterized by repeatedly generating the document cluster using a cluster as a seed. 7. The information search and classification method according to claim 3, wherein a term cluster that has become a seed of the document cluster is output as one information of a search and classification result for each of the document clusters. An information search and classification method characterized by the following. 8. The information search and classification method according to claim 1, wherein the first step comprises:
Determine a set of terms that co-occur in a predetermined size in the document and a co-occurrence frequency, calculate the degree of correlation between the terms based on the co-occurrence frequency, and calculate the degree of correlation in which the degree of correlation exceeds a predetermined value. An information search and classification method, wherein sets are assigned with a degree of correlation and associated, and information on the correlation of the terms is extracted. 9. The information search and classification method according to claim 1, wherein a part of the documents included in the document cluster is output as one information of a search and classification result for each document cluster. An information search and classification method characterized by the following. 10. An information retrieval and classification system for retrieving and classifying, from a set of documents stored in a document file, a document strongly related to an input search query based on a relation with a term included in the search query. A registration means for extracting and registering information relating to term correlation from a set of documents stored in the document file in advance, and a term related to a term included in the search query selected from the registration means;
Term cluster generating means for generating a plurality of term clusters composed of mutually strongly correlated terms, and searching for a document strongly related to the search query from the document file, assigning to the term cluster having the highest similarity, Document cluster generating means for generating a cluster of documents using the term cluster as a seed; and presenting, to a user, for each cluster of the document, a cluster of the term serving as a seed of the cluster of the document and a part of the document. Means for searching and classifying information. 11. The information search and classification system according to claim 10, wherein said registration means obtains a set of terms that co-occur in a predetermined size of the document and a co-occurrence frequency, and based on the co-occurrence frequency. Means for calculating a degree of correlation between the terms, assigning a set of terms in which the degree of correlation exceeds a predetermined value by assigning a degree of correlation, and extracting information relating to the correlation of the terms. Information retrieval and classification system. 12. The information retrieval and classification system according to claim 10, wherein the term used for generating the term cluster by the term cluster generating means and the document by the document cluster generating means. Means for narrowing down the documents used for generating the cluster to the terms and documents of the cluster selected by the user from the clusters of the documents presented by the presenting means, and presenting by the presenting means and selecting the document cluster by the user The information retrieval and classification system is characterized in that the document cluster is narrowed down by repeating the above.