KR100407081B1 - Document retrieval and classification method and apparatus - Google Patents

Abstract

In a document retrieval and classification system, a retrieval operation is performed on a database of documents according to the search conditions entered by the user to pick up the intended retrieval documents. The user is allowed to enter a plurality of classification standards in response to the retrieved documents picked up by the search operation. Classification criteria are converted into search conditions. The similarity between the search documents picked up by the search operation and the converted search conditions obtained from the classification standards is calculated. Then, the attribute of each retrieved document picked up by the retrieval operation for each category is calculated with reference to the similarity, whereby each retrieved document is classified into the category having the highest attribute.

Description

Document retrieval and classification method and apparatus

The present invention relates to a document retrieval and classification method for retrieving intended documents from a database storing a set of electronic document data. The present invention also relates to a document retrieval and classification system for performing the document retrieval and classification method of the present invention.

The present invention can be applied to various document information stored in a database such as an information storage medium loadable thereon as well as memory devices installed in personal computers, office computers, word processors, and the like.

Recent developments in the field of data communications, including e-mails, electronic catalogs, and electronic publications, provide vast amounts of document information that users can access and use. In addition, the number of Internet users is increasing significantly. Thus, there is an increasing demand for retrieving or collecting intended documents from a huge database. At the same time, there is an increasing demand for classification of intended picked up documents.

However, according to the conventional document retrieval and classification system, the search conditions and classification standards are typically fixed in advance or according to the user's preferences. In this respect, conventional document retrieval and classification systems have been fixed in terms of search conditions and classification standards.

It is an object of the present invention to improve flexibility in document retrieval and classification.

Another object of the present invention is to allow users to arbitrarily change the perspective of search conditions and classification standards.

It is yet another object of the present invention to enable users to perform search and classification operations in accordance with an immediate decision in response to a search result.

Another object of the present invention is to realize automatic classification to help the intellectual activities of users.

In order to achieve the above and other related objects, the present invention provides a first document retrieval and classification system, which includes input / output means for allowing a user to input search conditions and classification standards. The search means calculates a similarity between the search conditions and the search conditions picked up by the search operation and to perform a search operation on the database of documents according to the search conditions composed of any words or arbitrary strings. To provide. Search result storage means is provided for storing a search document picked up by a search operation. Classification standard conversion means are provided for converting classification standards into search conditions. The classification standard is expressed as a set of any words or any strings. Search result classification means is provided for classifying search documents picked up by a search operation according to a plurality of classification standards.

Thus, the present invention can provide flexible document retrieval and classification to assist intellectual activities during document retrieval and classification.

According to preferred embodiments of the present invention, the retrieval means is responsive to the retrieval conditions entered by the user via the input / output means and executes a retrieval operation on the database of documents in accordance with the retrieval conditions entered by the user. . The search result storage means stores the search documents picked up by the search operation of the search means. The classification standard converting means responds to the plurality of classification standards input by the user through the input / output means, and generates the converted search conditions obtained from the input classification standards. The search means calculates the similarity between search documents stored in the search result storage means picked up by the search operation and the converted search conditions. Then, the search result classification means calculates the attribute of each retrieved document picked up by the search operation for each classification standard according to the similarity calculated by the search means, and performs classification of the documents.

With this device, a user can arbitrarily enter search conditions when they have words or strings they have liked during the search operation. In addition, users can optionally sort the intended search results.

According to a preferred embodiment of the present invention, the input / output means allow the user to input a plurality of classification standards each composed of any words or any set of strings, and the classification standard conversion means may be any words or Convert a set of arbitrary strings to search conditions.

In this device, users can enter any words or any strings they have in mind as classification standards (ie, terms of classification). Thus, great flexibility is given when setting up aspects of classification.

According to preferred embodiments of the present invention, the document retrieval and classification system further comprises keyword retrieval means for extracting keywords from any sentence or document. In this case, the keyword retrieving means responds to the plurality of classification standards represented by any sentence input by the user via the input / output means, and extracts the keywords from the input sentences. The classification standard converting means then converts the extracted set of keywords into search conditions.

In this device, users can directly enter any sentence belonging to the intended field as a classification standard. This makes it possible to express complex classification aspects. Thus, the setting of the aspects of classification can be performed flexibly in many respects.

According to preferred embodiments of the present invention, the input / output means allow the user to specify a plurality of documents serving as a plurality of classification standards. The designated documents are selected from the retrieved documents picked up by the retrieval operation. The keyword retrieving means extracts a keyword from the designated document. The classification standard converting means then converts the extracted set of keywords into search conditions.

In this apparatus, after the users have identified the search documents picked up by the search operation, the users can select the retrieved documents themselves or some by expressing the aspects of classification. Thus, the setting of the aspects of classification can be easily performed.

The present invention also provides a second document retrieval and classification system, which includes input / output means for allowing a user to enter retrieval conditions. Search means are provided for performing a search operation on a database of documents according to a search condition composed of any words or any strings and for calculating the similarity between the search documents picked up by the search operation and the search conditions. . Search result storage means is provided for storing a search document picked up by a search operation. Keyword search means is provided for extracting keywords from the search documents picked up by the search operation. An automatic keyword classification means is provided for automatically classifying the extracted keywords into a plurality of clusters. Classification standard conversion means are provided for converting classification standards into search conditions. Each classification standard is a set of keywords classified into each cluster. Search result classification means is provided for classifying the set of retrieved documents picked up by the search operation according to the classification standards.

Thus, the present invention can provide an automatic document retrieval and classification system to assist intellectual activities during document retrieval and classification.

According to preferred embodiments of the present invention, the retrieval means is responsive to the retrieval conditions entered by the user via the input / output means and executes a retrieval operation on the database of documents in accordance with the retrieval conditions entered by the user. . The search result storage means stores the retrieved documents picked up by the search operation of the search means. The keyword retrieving means extracts keywords from the retrieved documents picked up by the retrieval operation. The automatic keyword classification means classifies the classified keywords into a plurality of clusters. The classification standard conversion means generates converted search conditions obtained from classification standards, which is a set of keywords classified into each cluster. The search means calculates the similarity between search documents stored in the search result storage means picked up by the search operation and the converted search conditions. The search result classification means then calculates an attribute of each retrieved document picked up by the search operation for each classification standard with reference to the similarity calculated by the search means, thereby performing automatic classification.

In this apparatus, it is possible to automatically extract aspects of classification of search results without relying on input of classification standards by the user. Users can automatically get sorting perspectives they don't think about. As a result, document classification can be efficiently assisted.

In addition, the present invention provides a first document search and classification method, which performs a search operation on a database of documents in accordance with search conditions input by a user to pick up intended searched documents; Allowing a user to enter a plurality of classification standards of the classifications in response to the retrieved documents picked up by the operation, converting the classification standards into search conditions, and searching documents picked up by the search operation; Calculating the similarity between the converted search conditions obtained from the classification standards, calculating the attributes of each retrieved document picked up by the search operation for each classification with reference to the similarity, and thereby resolving each retrieved document. Classifying the classification with the highest attribute.

In this way, users can enter search conditions arbitrarily when they have words they have in mind during the search operation. In addition, the user can optionally sort the intended search results. Thus, the present invention can assist in intellectual activity during document retrieval and classification.

According to a preferred embodiment of the present invention, when the user enters any words or any set of strings for the classification standard of each classification, any words or any strings entered are converted into search conditions and The similarity between the retrieved documents picked up by the retrieval operation and the converted retrieval conditions is calculated.

In this way, users can enter any words or any strings of interest as classification standards (ie, terms of classification). Thus, great flexibility is given in setting the aspects of classification.

According to preferred embodiments of the present invention, when a user inputs any sentence serving as a classification standard of each classification, the keywords are converted into a search condition, and the searched document and the converted search picked up by the search operation are converted. The similarity between the conditions is calculated.

In this way, users can directly enter any sentence belonging to the intended field as a classification standard. This makes it possible to express complex classification aspects. Thus, the setting of the aspects of classification can be performed flexibly in many respects.

According to preferred embodiments of the present invention, a user specifies a plurality of documents among search documents picked up by a search operation, each of which serves as a standard of classification. In this case, keywords are extracted from the extracted documents. The set of extracted keywords is extracted from the retrieved conditions. The extracted keyword set is converted into search conditions. Then, the similarity between the search documents picked up by the search operation and the converted search conditions is calculated.

In this apparatus, after the users confirm the search documents picked up by the search operation, the users can select the retrieved documents themselves or part thereof as representing the points of view of the classification. Thus, the setting of the aspects of classification is easily performed.

The present invention also provides a second document search and classification method, which performs a search operation on a database of documents according to search conditions input by a user to pick up intended search documents, and a search operation. Extracting keywords from the retrieved documents picked up by the user, classifying the extracted keywords into a plurality of clusters, and similarity between the retrieved documents picked up by the search operation and the converted search conditions obtained from the extracted keywords Calculating the attributes of each of the retrieved documents picked up by the retrieval operation for each classification with reference to the similarity, thereby classifying each retrieved document into the classification with the highest attribute.

In this way, it is possible to automatically extract aspects of classification of search results without relying on input of classification standards by the user. Users can automatically get sorting perspectives they don't think about. No special effort is needed. As a result, it becomes possible to assist with document classification efficiently.

1 is a block diagram showing a schematic structure of a document retrieval and classification system according to a first embodiment of the present invention;

Fig. 2 is a diagram showing a search result obtained by the document search and classification system according to the first embodiment of the present invention.

3 is a diagram showing a search result based on a classification standard according to the first embodiment of the present invention;

4 is a diagram showing an example of an attribute according to the first embodiment of the present invention;

Fig. 5 is a diagram showing a document classification result according to the first embodiment of the present invention.

6 is a block diagram showing a schematic configuration of a document retrieval and classification system according to a second embodiment of the present invention;

Fig. 7 is a block diagram showing the schematic arrangement of a document retrieval and classification system according to a third embodiment of the present invention.

* Description of the symbols for the main parts of the drawings *

21: input / output section 22: classification standard conversion section

23: search unit 24: document storage unit

25: search result storage unit 26: search result classification unit

The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Like reference numerals refer to like parts throughout. An embodiment of the present invention is based on documents containing Japanese. Thus, the following description includes kanji or katakana followed by English in parentheses.

First embodiment

1 is a functional block diagram showing a schematic arrangement of a document retrieval and classification system for implementing a document retrieval and classification method according to a first embodiment of the present invention.

In the document retrieval and classification system shown in FIG. 1, the input / output unit 21 allows users to input search conditions and classification standards, and also outputs search results and classification results. The searcher 23 calculates the similarity between the searched documents and the searched conditions. The search result storage unit 25 stores search results such as searched documents. The classification standard conversion section 22 receives the classification standards supplied from the input / output section 21 and converts the input classification standards into search conditions that can be processed by the search section 23. The search result classification unit 26 classifies the documents searched according to the classification standards with reference to the similarity calculated by the search unit 23.

In the following, details of the document retrieval and classification process according to the first embodiment are described.

First, the user inputs search conditions into the input / output unit 21. For example, the following logical (boolean) equation 1 can be given as search conditions.

(米 OR コ メ OR 政策) -------- (1)

Here, 米 is kanji representing rice, コ メ is katakana representing rice, and 政策 is kanji representing policy.

The searching unit 23 searches for documents stored in the document storing unit 24 based on the search conditions. The search unit 23 may perform a search based on search conditions composed of arbitrary words or arbitrary strings. In addition, the searcher 23 may calculate a similarity between the search result and the search conditions.

This type of retrieval unit may include a full sentence retrieval unit capable of retrieving all documents in which a specified word exists, as disclosed in Japanese Patent Laid-Open No. 9-319766.

The similarity between the search conditions and the retrieved document (ie, the search result) Dj can be expressed by the following equation.

S (Dj) = {fij x (1-log (di / N))}

here, Is the sum of the variable "i", "fij" represents the frequency (or degree) of occurrence of each word "ti" in the document Dj, "di" represents the number of documents in which the word "ti" appears, "N" represents the total number of documents retrieved.

The above expression represents taking the sum of the similarities of the respective words "ti" included in the search conditions.

This is commonly referred to as similarity calculation based on the inner product scale and word weighting based on the “TFIDF” method.

Now, in the specific search document Dj, it is assumed that the search words in the current search conditions have the following occurrence frequency. That is, "fij" is given as

米 (rice) 3

コ メ (rice (Katakana)) 2

政策 (policy) 1

On the other hand, in all documents stored in the document storage unit 24, the number of documents including each search word is as follows. That is, "di" is given by

米 (rice) 5000

コ メ (rice (katakana)) 1250

Policy 2500

When N = 10000, the similarity S (Dj) of Dj is calculated in the following manner.

S (Dj) = 3 x (1-log (5000/10000))

+ 2 x (1-log (1250/10000))

+ 1 x (1-log (2500/10000))

= 6 + 8 + 3 = 17

2 shows the details of the search results, where the number of documents in each retrieved document is shown along with the content and similarity of each retrieved document. According to Fig. 2, all ten documents are picked up according to the search conditions described above and ranked in order of similarity. The similarity of each document is normalized by assigning a maximum value of 100. The search result is stored in the search result storage unit 25, and the user can know the search result through the input / output unit 21.

The user can view the search results and then perform a new search, or sort the search results currently obtained.

If the user wants to classify the search result shown in FIG. 2, the user can input a plurality of classification standards through the input / output unit 21. According to the first embodiment, classification standards are words representing aspects of classification. For example, the user may input the following words as classification standards via the input / output unit 21.

Classification Standard 1: コ メ (rice; Katakana), 米 (rice price), 新 食糧 法 (new Staple Food Control Act)

Classification Standard 2: North Korea, China, U.S.-North Korea Talk

Classification Standard 3: 米 國 (U.S.), 米 軍 (U.S.force)

The classification standard conversion unit 22 converts the input classification standards into search conditions that can be processed by the retrieval unit 23.

For example, it is desirable to use AND to link the resulting logic to Formula 1 above, which makes a logical expression by properly combining the words entered as classification standards, and serves as the most recent search conditions. The following shows examples of converted search conditions.

Search condition1: (コ メ OR 米 價) AND (米 OR COMOME OR 政策)

Search condition2: (北 朝鮮 OR 中國) AND (米 OR コ メ OR 政策)

Search condition 3: (米 國 OR 米 軍) AND (米 OR コ メ OR 政策)

Linking the most recent search conditions (ie, logical expression 1) with classification criteria by AND is desirable to reduce the scale of the object to be searched.

Next, the searching unit 23 performs a search according to the search conditions 1 to 3 described above to obtain a search result shown in FIG.

As shown in FIG. 3, the search results obtained by the search conditions 1 to 3 are subsets of the entire set of retrieved documents shown in FIG. 2. In FIG. 3, the similarity attached to each retrieved document indicates the similarity value calculated for the corresponding search condition. Now assume that S (i, j) represents the similarity of the document "i" to the search condition (i.e., the classification standard) "j".

Next, the search result classification unit 26 calculates the attribute T (i, j) of the document "i" with respect to the classification standard "j". For example, the following equation is used to calculate the property T (i, j).

T (i, j) = C.S (i, j) + (1-C) .100 (S (i, j) / S (i, k)) --------- (2)

here, Is the sum for the variable "k" and C is a constant in the range 0 <C <1.

Equation (2) is merely an example for obtaining the attribute T (i, j). The method of calculating the attribute T (i, j) is not limited to the above expression (2).

4 shows the attribute T (i, j) calculated according to equation (2) above under the condition that C = 0.5 applies to documents 1 to 10 and classifications 1 to 3.

The search result classification section 26 uses the following equation (3) to identify the classification with the highest attribute T (i, j) for each document "i".

c (i) = max {T (i, j)} ---------------------------------- (3 )

Here, "max" represents the maximum value for the variable "j".

Finally, the search result classification unit 26 outputs a result that the document "i" belongs to the classification (i). This result is displayed or notified to the user via the input / output unit 21.

FIG. 5 shows an example output of the final classification based on the example shown in FIG. 4.

As described above, the search results shown in FIG. 2 (ie, the full set of retrieved documents) may be classified into a plurality of subsets according to classification standards 1 to 3, which are words entered by the user. .

According to the above example, the slip sheet "米" is one of the search elements given in the logical formula (1). However, the Japanese "米" has several meanings. Thus, in the retrieved documents shown in Fig. 2, some documents doc. # 6, # 10 include "米" representing "rice", while other documents (doc. # 3, # 4). (# 5, # 7, # 9) includes "米" for US (ie, US). However, the user can classify these documents into different classifications by entering appropriate classification standards.

In addition, when the user enters search conditions or classification standards, the user can arbitrarily select search words without particular attention for complex words such as "新 食糧 法" and "米 朝 協議" consisting of a plurality of single words. have.

In addition, after a classification is completed according to given classification standards, a user may designate a subset corresponding to one classification standard as a new object for another classification (ie, fine classification).

As described above, the first embodiment of the present invention provides a flexible document retrieval and classification method and system. According to the first embodiment, a retrieval operation is performed on a database of documents according to a retrieval condition input by a user to pick up the intended retrieved document. The user is allowed to input the classification standards of the plurality of classifications in response to the retrieved document picked up by the search operation. Classification criteria are converted into search criteria. The similarity between the search documents picked up by the search operation and the converted search condition obtained from the classification standard is calculated. Then, the attribute of each retrieved document picked up by the retrieval operation for each category is calculated with reference to the similarity, whereby each retrieved document is classified into the category having the highest attribute.

According to the first embodiment of the present invention, users can arbitrarily enter search conditions when they have words they like during the search operation. In addition, the user can optionally sort the intended search results. Thus, the first embodiment can assist in intelligent activities during document retrieval and classification.

Further, according to the first embodiment of the present invention, when the user enters arbitrary words or a set of arbitrary strings for the classification standard of each classification, any words or non-specified strings entered are search conditions. The similarity between the retrieved documents picked up by the retrieval operation and the converted retrieval conditions is calculated.

This makes it possible for users to enter any words or any strings they have in mind as classification standards (ie, aspects of classification). Thus, great flexibility is given when setting up aspects of classification.

Second embodiment

The second embodiment provides a document retrieval and classification system wherein the classification standards are selected from sentences representing aspects of classification.

6 shows a functional block diagram showing a schematic arrangement of a document retrieval and classification system according to a second embodiment of the present invention.

In the document retrieval and classification system shown in Fig. 6, the input / output section 11 allows a user to input search conditions and classification standards, and outputs search results and classification results. The document storage unit 15 stores the documents. The search unit 14 calculates the similarity between the searched documents and the search conditions. The search result storage unit 16 stores search results such as searched documents. The keyword detector 12 receives a sentence representing the viewpoints of the classification inputted by the user through the input / output unit 11, and detects keywords from the received sentence. The detected keywords serve as classification standards. The classification standard conversion unit 13 receives the keywords (that is, classification standards) transmitted from the keyword detection unit 12 and converts the input keywords into search conditions that can be processed by the retrieval unit 14. The search result classification unit 17 classifies the documents searched according to the classification standards with reference to the similarity calculated by the search unit 14.

The details of the document retrieval and classification process according to the second embodiment are described below.

First, the user inputs search conditions into the input / output section 11. Now, as in the first embodiment, it is assumed that search conditions defined by the logical formula 1 are input. Then, the search result shown in FIG. 2 is obtained.

When the user wants to classify the search result shown in FIG. 2, the user may input a plurality of classification standards through the input / output unit 11. According to a second embodiment, the classification standards are a sentence indicating aspects of classification, a reference number identifying the retrieved documents or an integral part of the retrieved documents.

For example, the user may input the following sentences as classification standards through the input / output unit 11.

Classification Standard 4: コ メ 市場 や 政府 の 米 價 政策 に つ い て (Regarding the rice market and the rice price policy of the government)

Classification Standard 5: Attitude of U.S. against North Korea and China

Classification Standard 6: Problems of U.S. forces in Korea and Japan

In response to the input of these sentences, the document retrieval and classification system of the second embodiment performs the following processing.

The keyword detection unit 12 extracts words that appear in each sentence based on a morphologic analysis and selects keywords (ie, essential words) of each sentence. .

Regarding the selection of keywords in each sentence, the frequency (or degree) of occurrence of each word in all documents stored in the dictionary in the document storage unit 15 is checked, and then the keywords are assigned to the "TFIDF" word weighting method or the like. It is preferred to be selected based on this. Such a word weighting method is disclosed in, for example, "Word Weighting Principle Based on Frequency-of-occurrence Information" by Umino, Library and Information Science, No. 26 (1988).

Moreover, as another method for selecting keywords, it is desirable in Japanese documents to take into account differences in character forms such as katakana, hiragana and kanji in extracting words (ie strings) from sentences. This is efficient for detecting compound words or new words not registered in the dictionary.

Of course, it is also desirable to combine the two methods as appropriate.

According to the second embodiment, keywords of each sentence are selected by referring to a dictionary database. Now, assume that the following words are extracted from the classification standards 4 to 6 described above.

Classification standard 4 ': Come (rice), city (government), government (government), 米 價 政策 (rice price policy)

Classification Standard 5 ': North Korea, China, 米 國 (U.S.)

Classification Standard 6 ': Korea 國 (Korea), 日本 (Japan), 米 軍 (U.S.force) 問題 (Problem)

Then, the classification standard conversion section 13 converts the classification standards 4 'to 6' into search conditions that can be processed in the reception section 14 in the same manner as the processing performed in the classification standard conversion section 22 of the first embodiment. do.

Also, after viewing the search results shown in Fig. 2, the user can designate document numbers of documents searched as classification standards in the following manner.

Classification Standard 7: 1,2

Classification Standard 8: 4,5

Classification Standard 9: 9

In response to the input of these reference numbers, the document retrieval and classification system of the second embodiment performs the following processing.

The keyword detector 12 reads the texts of the documents designated by the reference numbers (ie, classification standards) from the document storage unit 15, and extracts the keywords included in the designated documents.

Extraction of keywords may be performed in the same manner as the example described above of extracting keywords from sentences. Alternatively, it is preferable to extract the keywords from each document in advance, and to store the extracted keywords together with the corresponding documents in the document storage unit 15. In this case, the keyword detector 12 reads the keywords stored in the document storage 15 with reference to the reference numbers of the designated documents (i.e., classification standards).

Now, assume that the following words are extracted from the classification standards 7-9 described above.

Classification Standard 7 ': rice, stock, food, rice price, agricultural cooperative association, product, farmer, rice crop , 消費者 (consumer), 米 (rice)

Classification Standard 8 ': North Korea, North Korea, Conference, Korea, Talk, US, US-Korea, Problem, North-south ), 朝鮮 半島 (Korean Peninsula), 米 軍 (USforce)

Classification Standard 9 ': Okkinawa, US, Federation, Investigation, Return, Official Document, Material, Provisional Disposition, District court, decision

Then, the classification standard conversion section 13 converts the classification standards 7 'to 9' into search conditions that can be processed in the retrieval section 14 in the same manner as the processing executed in the classification standard conversion section 22 of the first embodiment. To convert.

After completion of the conversion of the classification standard to the search condition, the document search and classification system of the second embodiment executes the same processing as that disclosed in the first embodiment.

As described above, the search result shown in FIG. 2 (ie, the full set of search documents) is an integral part of sentences indicating aspects of classification, reference numbers identifying searched documents, or search documents entered by the user. It may be classified into a plurality of subsets according to phosphorus classification standards 4 to 6 (or 7 to 9). Thus, the user can perform the classification in various ways. For example, a user can flexibly and selectively use complex and simple perspectives when sorting searched documents.

As described above, according to the second embodiment of the present invention, when a user inputs an arbitrary sentence serving as a classification standard of each classification, keywords are extracted from the sentence, and the set of extracted keywords is a search condition. The similarity between the retrieved documents picked up by the retrieval operation and the converted retrieval conditions is calculated.

According to the second embodiment of the present invention, users can directly enter any sentence belonging to the intended field as a classification standard. This makes it possible to express complex classification points of view. Thus, setting aspects of classification can be performed flexibly in many respects.

Further, according to the second embodiment of the present invention, the user designates a plurality of documents among the searched documents picked up by the search operation, and the designated documents serve as a classification standard of each classification. At this time, keywords are extracted from the extracted documents. The set of extracted keywords is converted into search conditions. Then, the similarity between the search documents picked up by the search operation and the converted search conditions is calculated.

After the users have identified the retrieved documents picked up by the retrieval operation, the users can select the retrieved documents themselves or portions thereof as representing the aspects of classification. Thus, setting the aspects of classification can be easily performed.

Third embodiment

The third embodiment provides a document retrieval and classification system characterized in that classification standards are automatically determined and retrieved documents are automatically classified.

7 is a functional block diagram showing a schematic configuration of a document search and classification system according to a third embodiment of the present invention.

In the document search and classification system shown in Fig. 7, the input / output section 71 allows users to input search conditions and classification standards, and also outputs search results and classification results. The document storage unit 76 stores the documents. The search unit 75 calculates the similarity between the searched documents and the search conditions. The search result storage unit 77 stores search results such as searched documents. The keyword detection unit 72 detects a keyword from the retrieved document stored in the search result storage unit 77. The automatic keyword classification unit 73 classifies the set of detected keywords into a plurality of clusters. The classified keywords of each cluster serve as classification standards. The classification standard conversion unit 74 receives the keywords (that is, classification standards) transmitted from the automatic keyword classification unit 73 and converts the input keywords into search conditions that can be processed by the retrieval unit 75. The search result classification unit 78 classifies the searched documents according to the classification standards with reference to the similarity calculated by the search unit 75.

Hereinafter, details of the document retrieval and classification process according to the third embodiment will be described.

First, the user inputs search conditions into the input / output unit 71. Now, as in the first embodiment, it is assumed that search conditions defined by the logical formula 1 are input. And, the search result shown in FIG. 2 can be obtained.

The third embodiment differs from the first and second embodiments described above in that the classification standards are automatically determined without depending on the input of the classification standards by the user.

Hereinafter, the automatic classification according to the third embodiment will be described in detail. First, the keyword detection unit 72 detects keywords of each retrieved document stored in the search result storage unit 77. Details of the extraction of keywords are described in the second embodiment described above of the present invention. It is also possible to use the keyword extraction method disclosed in Japanese Patent Laid-Open No. 9-176822.

Next, the automatic keyword classification unit 73 classifies the set of detected keywords into a plurality of subsets. With regard to automatic keyword classification, the following method is used.

Now assume that document storage 76 stores all of the n documents D1 through Dn, where all of the m words W1 through Wm appear, respectively.

In this case, the next n-th vector Vj can be introduced into each word Wj.

Vj = (e1, e2, e3, ....., en)

The following equation (4) shows the calculation of each vector component (e1) (i = 1, ..., n).

ei = TFi (Wj) x log (n / DF (Wj)) -------------- (4)

Here, TFi (Wj) represents the frequency (or degree) of occurrence of the word Wj in the document Di, and DF (Wj) represents the number of documents in which the word Wj appears.

The vector Vj is preferably normalized such that its length is one.

In this way, the vectors V1 to Vm can be obtained for m words respectively.

Next, a plurality of word groups G1 to Gp are considered. Each word group consists of specific words that appear frequently in a particular field of document. Each word group can be generated manually or automatically using the occurrence distribution of words in a dictionary or large document.

In this case, the next n th vector VGK may be introduced into each word group Gk.

VGK = (e'1, e'2, e'3, ....., e'n)

The following equation (5) shows the calculation of each vector component (e'1) (i = 1, ..., n).

e'i = TFi (Gj) x log (n / DF (Gj) ----------------- (5)

Here, TFi (Gj) represents the frequency (or degree) of occurrence of words belonging to the group Gj of the document Di, and DF (Gj) represents the number of documents in which a certain word belonging to the group Gj appears.

The vector VGK is preferably normalized such that its length is one.

In this way, vectors VG1 to VGp may be obtained for p word groups, respectively.

The similarity between each word Wj and word group Gk can be obtained by the inner product of the vector Vj and the vector VGk.

By using the vectors and the similarity calculation, automatic classification of keywords can be easily realized. For example, suppose now that there are three word groups G1, G2, G3, which are frequently used in the following fields.

G1: Internal Combustion Engines for Automobiles

G2: aviation accident

G3: Internet

The search unit 75 searches for documents related to the "engine". At this time, the keyword detector 72 extracts the following keywords.

Gasoline (gasoline), 事故 (accident), WWW, fuel consumption, retrieval, Explosion, airport (Airport), URL

The similarity of a word for each word group G1, G2, G3 is calculated in the following manner.

S (Gasoline) = (0.8,0.0,0.2)

S (事故 (accident)) = (0.2,0.6,0.3)

S (WWW) = (0.1,0.2,0.8)

S (fuel consumption) = (0.7,0.1,0.2)

S (retrieval) = (0.0,0.2,0.6)

S (Explosion) = (0.4,0.6,0.1)

S (空港 (Airport)) = (0.0,0.9,0.2)

S (URL) = (0.1,0.0,0.9)

At this time, each keyword is considered to belong to a word group having the highest similarity. Therefore, all of the extracted keywords are classified into respective word groups G1, G2, and G3 in the following manner.

G1: gasoline (gasoline), fuel consumption

G2: 事故 (accident), 暴發 (Explosion), 空港 (Airport)

G3: www, retrieval, url

The keyword groups thus obtained by the automatic keyword classification section 73 are input to the classification standard conversion section 74.

When the number of word groups G is large (e.g., 100), or when the number of keyword groups serving as classification standards needs to be reduced (e.g., 2), the automatic keyword classification unit 73 It works in the following way:

First step: The sum of the weights of the keywords classified for each word group G is taken and the next obtained sum is regarded as the score of this word group.

Second step: A predetermined number of groups are selected by continuously considering the highest value of the scores.

According to the above example,

Score of G1: 0.8 + 0.7 = 1.5

Score of G2: 0.6 + 0.6 + 0.9 = 2.1

Score of G3: 0.8 + 0.6 + 0.9 = 2.3

Therefore, when the number of keyword groups needs to be reduced to two, the automatic keyword classification unit 73 selects the groups G2 and G3 in consideration of the score of each word group.

When the automatic keyword classification unit 73 performs the above described processing, the set of keywords extracted from the retrieved documents can be automatically classified into a plurality of groups. According to the above example, the following classification standards are obtained.

Classification standard 10: gasoline (fuel comsumption)

Classification Standard 11: Accident, Explosion, Airport

Classification Standard 12: WWW, retrieval, URL

Thereafter, the classification standard conversion section 74 converts the classification standards 10 to 12 into search conditions that can be processed by the search section 75 in the same manner as the processing performed by the classification standard conversion section 22 of the first embodiment. .

After completion of the conversion of classification standards into search conditions, the document retrieval and classification system of the third embodiment executes the same processing as that disclosed in the first embodiment.

As described above, the third embodiment automatically determines the field of words that frequently appear in the retrieved documents. Detected fields are considered as classification standards. Therefore, it is possible to perform document classification according to the nature of the search results. In other words, the third embodiment provides simplified document classification.

The keyword groups obtained by the automatic keyword classification unit 73 can be seen once by the user via the input / output unit 71. The user can modify or correct the indicated keyword groups. Next, the classification standard conversion unit 74 converts the modified keyword groups (that is, the modified classification standards) into search conditions. This classification process makes it possible for the user to know classification points which are not considered by the user. As a result, the third embodiment helps the document classification task efficiently.

As described above, the third embodiment of the present invention provides a method and system for automatic document retrieval and classification. The retrieval operation is performed on a database of documents according to the search conditions entered by the user to pick up the intended retrieved documents. The keywords are extracted from the retrieved documents picked up by the search operation. The searched keywords are classified into a plurality of clusters. The set of extracted keywords belonging to each cluster are converted into search conditions. The similarity between the search documents picked up by the search operation and the converted search conditions obtained from the extracted keywords is calculated. Then, for each category, the attributes of each retrieved document picked up by the retrieval operation are calculated with reference to the similarity, thereby classifying each retrieved document into the category with the highest attribute.

According to the third embodiment of the present invention, it is possible to automatically extract classification aspects of the search result without relying on input of classification standards by the user. Users can automatically get classification views they don't think about. As a result, it becomes possible to assist in document classification work efficiently.

The present invention is susceptible to various modifications and changes without departing from the spirit and scope of the invention. The described embodiments are provided for illustrative purposes only and are not intended to be limiting. The spirit of the invention is defined by the appended claims. Many modifications are possible without departing from the spirit and scope of the invention.

Claims (12)

delete delete delete delete delete In a document retrieval and classification system, Input / output means for allowing a user to enter search conditions, Retrieving means for performing a retrieval operation on a database of documents according to the retrieval conditions composed of arbitrary words or arbitrary strings, and calculating the similarity between the retrieved documents picked up by the retrieval operation and the retrieval conditions , Search result storage means for storing the searched documents picked up by the search operation, Keyword detecting means for extracting keywords from the retrieved documents picked up by the searching operation, Automatic keyword classification means for automatically classifying the extracted keywords into a plurality of clusters, Classification standard conversion means for converting classification standards into search conditions, wherein each of the classification standards is a set of keywords classified into respective clusters, and A search result classification means for classifying the set of retrieved documents picked up by the search operation according to the classification standards. The method of claim 6, The retrieval means is responsive to the retrieval conditions entered by the user via the input / output means, and executes a retrieval operation on a database of documents in accordance with the retrieval conditions entered by the user, The search result storage means stores the retrieved documents picked up by a search operation of the search means, The keyword detecting means extracts keywords from the retrieved documents picked up by the search operation, The automatic keyword classification means automatically classifies the extracted keywords into a plurality of clusters, The classification standard conversion means generate converted search conditions obtained from classification standards, each of which is a set of keywords classified into each cluster, The retrieval means calculates a similarity between the retrieved documents picked up by the retrieval operation and stored in the retrieval result storage means and the converted retrieval conditions, The search result classification means calculates an attribute of each retrieved document picked up by the search operation for each classification standard with reference to the similarity calculated by the search means, thereby performing document classification. And classification system. delete delete delete delete In the document search and classification method, Performing a search operation on a database of documents according to the search conditions entered by the user to pick up the intended retrieved documents, Extracting keywords from the retrieved documents picked up by the search operation, Classifying the extracted keywords into a plurality of clusters; Converting the extracted set of keywords belonging to each cluster into search conditions, Calculating a similarity between the retrieved documents picked up by the search operation and the converted search conditions obtained from the extracted keywords, and Calculating a property of each document picked up by the search operation for each category with reference to the similarity, thereby classifying each retrieved document into a category having the highest attribute. Way.