JPH1145266A - Document retrieval device and computer readable recording medium recorded with program for functioning computer as the device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically extract a noun phrase relating to retrieval conditions used at the time of retrieval from a document obtained by the retrieval and to register it to a thesaurus dictionary for the retrieval. SOLUTION: This device selects a synonym corresponding to the inputted retrieval conditions 206 from the thesaurus dictionary 203 for the retrieval, extends the retrieval conditions 206 and retrieves a pertinent document. In this case, it is provided with a natural language processing module 200 for inputting the document retrieved in a retrieval engine 209 based on the inputted retrieval conditions 206 and generating a document set 204 including a noun phrase list in the inputted document and a relating word extraction engine 211 for inputting the document set 204, imparting a score corresponding to statistic information such as an appearing frequency and distribution, etc., in the document in a document DB 101 and the document retrieved in the retrieval engine 209 to the respective noun phrases, extracting the noun phrase of the score pertinent to selection conditions set beforehand and registering it to the thesaurus dictionary 203.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document in which a noun phrase related to a search condition used in a search is automatically extracted from a document obtained by the search and registered in a thesaurus for search. The present invention relates to a search device and a computer-readable recording medium storing a program for causing a computer to function as the search device.

[0002]

2. Description of the Related Art Generally, a document search apparatus for searching for a specific document from a document DB (database) storing a plurality of documents inputs a search condition such as a search formula or a search sentence, and corresponds to the input search condition. The document is searched from the document DB.

In the above-described document search apparatus, a search is performed based on the input search condition. Therefore, a document described using a vocabulary related to the vocabulary in the search condition, not the vocabulary itself of the search condition, In some cases, the search results did not correspond to the input search conditions and the search results were omitted.

Therefore, a thesaurus for search is prepared in advance, a synonym corresponding to the input search condition is extracted from the thesaurus, and a search is performed by adding a synonym extracted from the thesaurus to the input search condition. Thus, a document search apparatus has been proposed in which search results are prevented from being leaked.

[0005]

However, in the above-mentioned conventional technology, since the thesaurus for search is manually generated, it is difficult to manage the thesaurus in a state where the latest synonyms are always registered. There was a problem that it was difficult. In particular, with regard to advanced technology, new terms are constantly being created, and it has been difficult to continuously collect new terms and register them in the thesaurus. Then, if registration in the thesaurus dictionary is neglected, there is a problem that it is impossible to always obtain a highly accurate search result, no matter how much the search is performed using the thesaurus dictionary.

The present invention has been made in view of the above, and is intended to automatically extract a noun phrase related to a search condition used in a search from a document obtained by the search and register the same in a thesaurus dictionary. Accordingly, an object of the present invention is to reduce the labor for managing the thesaurus dictionaries.

Further, the present invention has been made in view of the above, and it is possible to obtain a highly accurate search result by keeping a thesaurus dictionary in a state where the latest synonyms are registered at all times. The purpose is to be able to.

[0008]

In order to achieve the above object, a document search apparatus according to claim 1 selects a synonym corresponding to an input search condition from a thesaurus for search, and based on the search condition and the synonym. What is claimed is: 1. A document search apparatus comprising: a search unit for searching a target document from a search target document, wherein the search unit inputs a document searched by the search unit, extracts a noun phrase in the input document, and And a noun phrase list generated by the noun phrase list generating unit, and a document and a search target searched by the searching unit for each noun phrase in the input noun phrase list Score assigning means for assigning a score according to statistical information such as an appearance frequency and a distribution in a document,
A noun phrase extraction unit for extracting a noun phrase of a score corresponding to a preset extraction condition based on the score given by the score giving unit, and a noun phrase extracted by the noun phrase extraction unit registered in the thesaurus dictionary Dictionary registration means to
It is provided with.

A document search device according to a second aspect of the present invention is the document search device according to the first aspect, further comprising: a document selection unit that selects a document from which the noun phrase is to be extracted from the documents searched by the search unit. Wherein the noun phrase list generation means extracts the noun phrase from the document selected by the document selection means to generate a noun phrase list.

According to a third aspect of the present invention, in the document search apparatus according to the first or second aspect, further, a noun phrase to be registered in the thesaurus dictionary is selected from the noun phrases extracted by the noun phrase extraction means. The dictionary registration means registers the noun phrase selected by the noun phrase selection means in the thesaurus dictionary.

According to a fourth aspect of the present invention, there is provided a computer-readable recording medium on which a program for causing a computer to function as each unit of the document search apparatus according to any one of the first to third aspects is recorded. is there.

[0012]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of a document search apparatus according to the present invention and a computer-readable recording medium storing a program for causing a computer to function as the apparatus. explain.

FIG. 1 is a system configuration diagram of a document search apparatus according to the present embodiment. The document search apparatus shown in FIG. 1 generates a transposed file 102 using a plurality of clients 100 that output search conditions for searching for a desired document, and a document in a document DB (database) 101, and uses a vector space method. Equipped with a search engine (for example, CLARIT of CLARITECH, etc.) that performs a search using
A search condition is input from the client 100, a synonym corresponding to the search condition is extracted from a prepared thesaurus dictionary, and the transposed file 1 is extracted based on the search condition and the synonym.
02, the relevant document is searched, and a noun phrase related to the input search condition is extracted from the searched document.
It comprises a search server 103 to be registered in a thesaurus dictionary, and a network 104 for connecting the client 100, the search server 103 and the like.

In FIG. 1, a document DB 101 stores a plurality of documents created by the client 100 or the like. The stored documents include word processing documents and SGM.
Any type of document, such as a structured document such as L or HTML, may be used. In the present embodiment, the document DB 10
1, the search target is not limited to the documents in the document DB 101.

The transposed file 102 defines the relationship between a plurality of documents in the document DB 101 and a plurality of index words extracted from each of the plurality of documents by a method described later, so that a certain index word is included in each document. It shows how important the vocabulary is by using a vector expression, so that a corresponding document can be searched using this index word.

Specifically, one document is divided into sub-documents consisting of a plurality of predetermined sentences,
The noun phrases serving as the above-mentioned index words are extracted from the sub-documents, and for each of the extracted noun phrases, statistical information such as the frequency of appearance in the sub-document and the distribution in the entire document DB 101 is obtained. To convert each subdocument into a vector representation. And
Based on the vector representation of the converted subdocument,
Generate a vector representation of the document. Inversion file 102
Is the document DB 101 thus expressed in vector.
This is for storing the document inside.

Each index word can be weighted according to the degree of importance in the corresponding document. Also, the vector expression of the document may be generated based on the vector expression of the sub-document when performing an actual search.

Client 100 and search server 10
Reference numeral 3 includes a personal computer, a workstation, and the like. FIG. 2 is a schematic block diagram illustrating the processing of the search server 103. The search server 103 registers the document in the document DB 101 in the transposed file 102, searches the document using the vector space method,
A process of extracting a noun phrase related to a search condition from a searched document and registering the noun phrase in the thesaurus dictionary 203 is performed.

In the search server 103, the document DB 10
The process of registering the document in the file 1 in the transposition file 102 is performed by the natural language processing module 200 and the database build component 205.

Specifically, the natural language processing module 200
Input a document from the document DB 101, and perform morphological analysis on the input document using a format recognition process, a dictionary 201 storing part-of-speech information and the like, and a grammar dictionary 202 for analyzing dependency of each word. The document set 204 including the parsing list and the noun phrase list for each sub-document is analyzed by performing a parsing process and a noun phrase extraction process.
Generate

Database Build Component 2
05, a document set 204 generated by the natural language processing module 200 is input, each sub-document in the input document set 204 is converted into a vector expression, and a document vector is generated based on the vector expression of the sub-document. An expression is generated and registered in the transposition file 102.

The document search process is performed by the natural language processing module 200, the query build component 207, and the search engine 209.

More specifically, the natural language processing module 200
Inputs the search condition 206 from the client 100,
Analysis processing such as morphological analysis, syntax analysis, and noun phrase extraction is performed using a dictionary 201 storing part-of-speech information and a grammar dictionary 202 for analyzing the dependency of each word.
Synonyms of the extracted noun phrase are extracted from the thesaurus dictionary 203 and added to the noun phrase of the search condition 206, and the search condition 206
Generate a document set 204 that includes a list of noun phrases.

Query build component 207
Inputs the document set 204 and sets the search condition 2
For each of the noun phrases constituting the part No. 06, statistical information such as the frequency of appearance in the search condition 206 and the distribution in the entire document DB 101 is obtained, and the query document 208 obtained by converting the search condition 206 into a vector expression using the obtained statistical information is obtained. Generate.

The search engine 209 inputs the query document 208 generated by the query build component 207, and converts the vector expression of each document in the transposed file 102 and the query document 208 (vector expression of the search condition 206). In comparison, a score corresponding to the degree of similarity with the query document 208 is assigned to each document, and a document list 210 to which a score exceeding a predetermined threshold is assigned is output as a search result.

Further, in the search server 103, the registration processing to the thesaurus dictionary 203 is performed by the natural language processing module 200 and the related word extraction engine 211.

More specifically, the natural language processing module 200
Inputs the document list 210 from the search engine 209, and based on the input document list 210, the document DB1
Enter the corresponding document from 01. For all input documents, morphological analysis, syntax analysis, and noun phrase extraction are performed using a format recognition process, a dictionary 201 storing part-of-speech information, and a grammar dictionary 202 for analyzing the dependency of each word. To generate the document set 204 including the noun phrase list for each sub-document described above.

The related word extraction engine 211 generates the document set 2 generated by the natural language processing module 200.
04, and for each noun phrase in the input document set 204,
The statistical data such as the appearance frequency in the set 204) and the distribution in the document DB 101 (transposed file 102) are calculated, and a score is assigned to each noun phrase based on the calculated statistical data. Then, a noun phrase having a score exceeding a preset threshold is extracted as a noun phrase related to the search condition 206, and the extracted noun phrases are grouped as one group in the thesaurus dictionary 20.
Register to 3.

Although FIG. 1 shows a configuration in which the document DB 101 and the transposed file 102 are independently connected to the network 104, a configuration in which these are directly connected to the search server 103 may be used. In FIG. 1, the document search device of the present embodiment is connected to a network 104.
However, the processing of the client 100 and the search server 103 may be performed by one computer.

Next, the operation of the document search apparatus having the above-described configuration will be described with respect to (1) a process of generating an inverted file,
This will be described in detail in the order of (2) document search processing and (3) registration processing in a thesaurus dictionary.

(1) Transposition File Generation Process FIG. 3 is a flowchart showing a transposition file generation process. The search server 103 stores the new document in the document DB1
01 (S301), a process for inputting this document and registering it in the transposition file 102 is started (S302).

In the search server 103, the natural language processing module 200 performs a process of analyzing the document input in step S302 (S303). Specifically, the input document is a word processing document, a structured document such as HTML, etc.
A process is performed to determine the format of the document. Thereafter, syntax analysis such as morphological analysis and dependency is performed by using the dictionary 201 and the grammar dictionary 202, and the document is divided into a plurality of sub-documents, and processing such as extracting a noun phrase from the divided sub-documents is performed.

Then, the natural language processing module 200
Is based on the result of the processing in step S303.
A noun phrase list is generated for each sub-document, and a document set 204 including the generated noun phrase list is generated (S304).

Thereafter, the database build component 205 receives the document set 204 generated by the natural language processing module 200, generates a vector representation of the document, and registers it in the transposed file 102 (S305).

Specifically, the document set 204
Transpose file 102 for each noun phrase of subdocument in
The statistical information such as the frequency of occurrence in the sub-document and the distribution in the entire document DB 101 is obtained as an index term, and the sub-document is converted into a vector expression using the obtained statistical information for each noun phrase. This process is performed for all the sub-documents in the document set 204, and based on the converted sub-document vector expression, a vector expression of the document is generated and registered in the transposed file 102. As a result, the document newly registered in the document DB 101 is converted into a vector expression and registered in the transposition file 102.

(2) Document Search Processing Next, the transposed file 102 generated as described above
A description will be given of a process of searching for a document based on the. FIG. 4 is a flowchart showing a document search process.

The search server 103 is connected to the client 100
When the search condition 206 is input from (S401), the natural language processing module 200 analyzes the search condition 206 (S402). The search condition 206 is a search sentence described in a natural language, and can be changed to a search expression, a set of keywords, or the like, depending on the configuration of the document search device. Specifically, the natural language processing module 200 performs syntactic analysis processing such as morphological analysis and dependency using the dictionary 201 and the grammar dictionary 202 for the search condition 206, and extracts a noun phrase from the search condition 206. If a synonym of the extracted noun phrase exists in the thesaurus dictionary 203, a process of extracting the corresponding synonym and adding it as a noun phrase of the search condition 206 is performed.

Then, the natural language processing module 200
Generates the document set 204 including the noun phrases extracted by performing the analysis processing in step S402 (S403).

Next, the query build component 207 inputs the document set 204 from the natural language processing module 200, and for each noun phrase constituting the input document set 204, the appearance frequency in the search condition 206, Statistical information such as distribution in the entire document DB 101 is obtained, and a query document 208 is generated by converting the document set 204 into a vector expression using the obtained statistical information (S404).

The search engine 209 inputs the query document 208 generated by the query build component 207, and converts the vector expression of each document in the transposed file 102 and the query document 208 (vector expression of the search condition 206). In comparison, a score corresponding to the degree of similarity with the query document 208 is given to each document (S405). That is, search processing using the vector space method is performed.

The score according to the similarity expresses the similarity between each document and the query document 208 based on the cosine distance. A document having a higher score is more similar to the query document 208. It represents that it is.

The search engine 209 selects a document having a score exceeding the threshold based on a preset score threshold, generates a document list 210 based on the selected document, and performs a search. The result is output (S406).

The client 100 is a search server 103
Input document list 210, and input document list 2
Based on 10, a list of searched documents is displayed on the screen (S 407). The user of the client 100 can display a document in the document DB 101 on the screen by selecting a desired document from the document list displayed on the screen.

In the client 100, documents are listed in order from the document having the highest ranking. Therefore, the documents that are most similar to the search condition 206 are displayed in order, and a criterion for the user to select a document can be provided.

(3) Registration Processing in Thesaurus Dictionary Next, processing for extracting a noun phrase related to the search condition 206 from the document searched in the above-described search processing and registering the extracted noun phrase in the thesaurus dictionary 203 will be described. I do. FIG. 5 is a flowchart showing a process for registering a noun phrase in a thesaurus dictionary.

When the above-described search processing is completed, the natural language processing module 200 inputs the document list 210 from the search engine 209 (S501). Then, based on the input document list 210, a corresponding document is input from the document DB 101 (S502).

When a document is input from the document DB 101, for all the input documents, a format recognition process, a dictionary 201 storing part-of-speech information and the like, and a grammar dictionary 202 for analyzing the dependency of each word are used. Perform analysis processing such as morphological analysis, syntax analysis, and noun phrase extraction (S
503).

Thereafter, based on the result of the analysis processing in step S503, a document set 204 including a noun phrase list for each sub-document is generated (S5).
04).

The related word extraction engine 211 generates the document set 2 generated by the natural language processing module 200.
04, and for each noun phrase in the input document set 204,
The statistical data such as the appearance frequency in the set 204) and the distribution in the document DB 101 (transposed file 102) are calculated (S
505).

After calculating the statistical data in step S505, the related word extraction engine 211 scores each noun phrase based on the obtained statistical data (S50).
6). This score indicates the importance of each noun phrase in the document 200 and the relevance to the noun phrase in the search condition 206. The higher the score, the higher the importance and relevance.

The related word extraction engine 211 proceeds to step S
Based on the result of the scoring performed in 506, a noun phrase with a score exceeding a preset threshold is extracted as a noun phrase related to the noun phrase in the search condition 206 (S50).
7). Here, a threshold value is used as a condition for selecting a noun phrase. However, instead of the threshold value, for example, a noun phrase having the top five scores may be extracted.

Thereafter, the related word extraction engine 211 registers the noun phrases extracted in step S507 as one group in the thesaurus dictionary 203 (S508). As described above, the noun phrase registered in the thesaurus dictionary 203 is used to extend the search condition 206 as a synonym of the noun phrase in the search condition 206 when performing a document search process. .

In step S508, when the extracted noun phrase is registered in the thesaurus dictionary 203, a comment such as the purpose of use of the extracted noun phrase can be registered at the same time.

When explaining the document search process,
Synonyms are automatically extracted from the thesaurus dictionary 203,
I decided to add it as a noun phrase in the search condition 206,
Synonyms may be added by a user operation of the client 100.

In the above-described registration processing in the thesaurus dictionary 203, the document list 210 as a search result is obtained.
Extract noun phrases from documents corresponding to
However, the client 100 may select a document from which a noun phrase to be registered in the thesaurus dictionary 203 is extracted. That is, the user selects a document to be used for registration processing in the thesaurus dictionary 203 from the document list displayed on the screen of the client 100 in step S407 of FIG. 4, extracts a noun phrase from the selected document, and extracts the noun phrase from the selected document. You can register to. As a result, a noun phrase is extracted from the document of the search result satisfying the user, and it is possible to prevent an unnecessary noun phrase from being registered in the thesaurus dictionary 203.

Further, the noun phrase selected in step S507 in FIG. 5 is registered in the thesaurus dictionary 203 as it is. A list of the selected noun phrases is displayed on the client 100 on the screen, and the noun phrase registered in the thesaurus dictionary 203 is displayed. The user may be allowed to select a phrase. As a result, similarly to the above, it is possible to prevent unnecessary noun phrases from being registered in the thesaurus dictionary.

As described above, according to the document search apparatus of the present embodiment, the noun phrase related to the search condition 206 used in the search is automatically extracted from the document obtained by the search, and is stored in the thesaurus dictionary 203. Since the registration can be performed, the labor for managing the thesaurus dictionary 203 can be reduced. In addition, since the thesaurus dictionary 203 can always keep the latest synonyms registered, it is possible to always obtain highly accurate search results. In particular, it is possible to particularly improve search accuracy when a search is performed by a method similar to the search performed previously.

Note that, among documents displayed as a list of search results, for a document that the user considers appropriate or unsuitable as the search result, the result can be fed back to the search server 103.
That is, the user can specify a positive weight, for example, “+” for a document that is considered to be appropriate as a search result, and a negative weight, for example, “−” for a document that is not appropriate for the search result. Can be specified.
As a result, if the input weight is a positive designation, the weight of the corresponding document in the transposed file 102 is strengthened,
If the input weight is a negative designation, the weight of the document is reduced.

Further, in the present embodiment, the search by the vector space method has been described as an example, but the search processing may be performed by a Boolean search.

Further, the document search device described in the present embodiment is realized by executing a prepared program on a computer or a workstation.
This program is recorded on a computer-readable recording medium such as a hard disk, a floppy disk, a CD-ROM, an MO, and a DVD, and is executed by being read from the recording medium by the computer.
This program can be distributed via the recording medium or via a network.

[0061]

As described above, according to the document search apparatus of the present invention (claim 1), a synonym corresponding to the input search condition is selected from the search thesaurus, and the synonym is selected based on the search condition and the synonym. A document search apparatus having a search means for searching for a corresponding document from a search target document, inputting the document searched by the search means, extracting a noun phrase in the input document, and generating a noun phrase list. The noun phrase list generating means to be generated and the noun phrase list generated by the noun phrase list generating means are inputted, and for each noun phrase in the input noun phrase list, the document searched by the searching means and the document to be searched are A scoring unit that assigns a score according to statistical information such as an appearance frequency and a distribution, and a noun of a score corresponding to a preset extraction condition based on the score assigned by the scoring unit. Because with a noun phrase extraction means for extracting a dictionary registration means for registering a noun phrase extracted with noun phrase extraction means thesaurus, and it is possible to reduce the effort required to manage the thesaurus. In addition, since the thesaurus dictionary can always keep the latest synonyms registered, it is possible to always obtain highly accurate search results. In particular, it is possible to particularly improve search accuracy when a search is performed by a method similar to the search performed previously.

Further, the document search device of the present invention (Claim 2)
According to the document search apparatus according to claim 1, further comprising a document selection unit for selecting a document from which a noun phrase is to be extracted from the documents searched by the search unit, wherein the noun phrase list generation unit includes a document selection unit. Since the noun phrases are extracted from the document selected by the means to generate a noun phrase list, the noun phrases are extracted from the document of the search result satisfying the user, and unnecessary noun phrases are registered in the thesaurus dictionary. Can be prevented.

A document retrieval apparatus according to the present invention (claim 3)
According to claim 1, the document retrieval device according to claim 1 or 2, further comprising a noun phrase selection unit that selects a noun phrase to be registered in a thesaurus dictionary from the noun phrases extracted by the noun phrase extraction unit. Since the noun phrase selected by the noun phrase selecting means is registered in the thesaurus dictionary, unnecessary noun phrases can be prevented from being registered in the thesaurus dictionary.

Further, according to a computer readable recording medium of the present invention (claim 4), a program for causing a computer to function as each means of the document search device according to any one of claims 1 to 3 Was recorded,
By running this program on a computer, the effort required to manage the thesaurus dictionary can be reduced, and the thesaurus dictionary can always be kept in the state where the latest synonyms are registered. A document search device that can obtain high search results can be realized.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram of a document search device according to an embodiment.

FIG. 2 is a schematic block diagram illustrating processing of a search server in the document search device according to the present embodiment.

FIG. 3 is a flowchart illustrating a process of generating a transposed file in the document search device according to the present embodiment.

FIG. 4 is a flowchart illustrating a document search process in the document search device according to the present embodiment.

FIG. 5 is a flowchart showing a process of registering a noun phrase in a thesaurus dictionary in the document search device according to the present embodiment.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 client 101 document DB 102 transposed file 103 search server 200 natural language processing module 201 dictionary 202 grammar dictionary 203 thesaurus dictionary 204 document set 205 database build component 206 search condition 207 query build component 208 query document 209 search engine 210 Document list 211 Related word extraction engine

Claims (4)

[Claims] 1. A document search method comprising: selecting a synonym corresponding to an input search condition from a thesaurus for search; and searching for a corresponding document from a search target document based on the search condition and the synonym. An apparatus, comprising: inputting a document searched by the search unit; extracting a noun phrase in the input document to generate a noun phrase list; and a noun generated by the noun phrase list generation unit. A phrase list is input, and a score is assigned to each noun phrase in the input noun phrase list according to statistical information such as the frequency of occurrence and distribution in the document searched by the search means and the search target document. Means, a noun phrase extracting means for extracting a noun phrase of a score corresponding to a preset extraction condition based on the score given by the score giving means, and the noun phrase A dictionary registration means for noun phrases extracted with means out is registered in the thesaurus, the document search apparatus characterized by comprising a. 2. The apparatus according to claim 1, further comprising: a document selection unit that selects a document from which the noun phrase is to be extracted from the documents retrieved by the retrieval unit. 2. The document search device according to claim 1, wherein the noun phrase is extracted to generate a noun phrase list. 3. The system according to claim 1, further comprising a noun phrase selecting unit for selecting a noun phrase to be registered in the thesaurus dictionary from the noun phrases extracted by the noun phrase extracting unit, wherein the dictionary registering unit selects the noun phrase selected by the noun phrase selecting unit. 3. The document search apparatus according to claim 1, wherein a noun phrase is registered in the thesaurus dictionary. 4. A computer-readable recording medium having recorded thereon a program for causing a computer to function as each unit of the document search device according to claim 1.