JP4691117B2 - Text search device, text search method, text search program, and recording medium recording the program - Google Patents

Description

  The present invention relates to a text search technique for searching text information requested by a user from a large amount of text information (for example, a large number of documents) using a plurality of words included in the text information and the number of documents as search conditions.

  2. Description of the Related Art Conventionally, a technique using a transposed file is known in order to search text information (text document, hereinafter simply referred to as a document) at high speed. The transposed file includes a transposed file in document ID order (for example, refer to Patent Document 1 and Non-Patent Document 1) in which documents are stored in order of document ID for each word, and an impact value order in which documents are stored in order of impact value for each word. There are transposed files (for example, see Non-Patent Documents 1 to 4). The impact value indicates the numerical value of the weight obtained based on the word appearance frequency (tf value) calculated in advance for each word in each text document and the total number of words included in the text document. In general, it is known that a text search apparatus that uses a transposed file in order of impact value has better performance than a text search apparatus that uses a transposed file in order of document ID.

FIG. 3 is an explanatory diagram schematically showing an outline of a conventional text search apparatus 300 that uses a transposed file in order of impact value constructed for each word. The text search device 300 includes a search processing device 320, an index storage device 330, and a word dictionary storage device 350. The search processing device 320 extracts the word group 402 included in the search condition 401 input from the terminal device 360 with reference to the word dictionary stored in the word dictionary storage device 350 and stores it in the index storage device 330. Using the transposed files 331 to 334 in the order of impact values, a candidate document to be output (candidate document) is searched, and the search result is output to the terminal device 360. In the example illustrated in FIG. 3, the search processing device 320 divides the phrase “Tokyo station” as the search condition 401 and extracts “Tokyo”, “no”, and “station” as the word group 402. Then, for the transposed files 331, 333, and 334 corresponding to each word, a predetermined score is calculated for each document in descending order of impact value, as indicated by arrows 403, 404, and 405. As a result, the document (d16) having the document ID “16” indicated by the reference numerals 413, 414, and 415 is searched from the three transposed files as a common document, and is output as a candidate document.
Japanese Patent Laid-Open No. 3-108064 (FIG. 1) Justin Zobel and Alistair Moffat, “Inverted Files for Text Search Engines”, ACM Computing Surveys, Vol. 38, No. 2, Article 6, July, 2006 Vo Ngoc Anh, Owen de Kretser, Alistair Moffat, “Impact Transformation: Effective and Efficient Web Retrieval”, Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p.35-42, 2001 Vo Ngoc Anh and Alistair Moffat, “Vector-Space Ranking with Effective Early Termination”, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p.3-10, 2002 Vo Ngoc Anh and Alistair Moffat, “Pruned Query Evaluation Using Pre-Computed Impacts”, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p.372-379, 2006

  However, among the words divided in the conventional text search device using the transposed file in order of impact value constructed for each word, particles such as “no” and “ga” are all text documents to be searched. Will appear through. Therefore, for example, as shown in FIG. 3, in order to search for a text document including both “Tokyo” and “station” in the transposed file 333 of “no”, all of the transposed file 333 of “no” There was a performance problem that the probability of scanning was high.

  Also, for example, when searching a document on the Web (document), when searching for a text document using a transposed file in order of impact value constructed for each word included in the document, PageRank (registration) There is a demand for searching text documents in consideration of the unique score of the document such as trademark). However, when searching for text documents using transposed files in order of impact value, it has not been known how to improve the performance of search processing by handling the original scores independent of words. .

  Therefore, an object of the present invention is to provide a text search technique that can solve the above-described problems and improve the processing performance of search processing when using a transposed file in order of impact value.

In order to solve the above-mentioned problem, the text search device according to claim 1 includes: a word appearance frequency calculated in advance for each word in each text document having a document ID; and a total number of words included in the text document. Including a transposed file in order of impact value constructed for each word using an impact value indicating a weight value obtained based on a transposed file in order of document ID constructed for each word included in each text document A text search device for searching for a text document using a plurality of transposed files as an index for document search, wherein a search expression including a plurality of words and the number of documents are input as specified search conditions. Means, word extraction means for extracting each word included in the inputted search expression based on a word dictionary, and all texts to be searched for each word Word group classification for classifying each extracted word into a word group whose document appearance frequency is lower than a predetermined threshold and a word group that is not so based on the document appearance frequency calculated in advance through the document It means, as the search of the first stage, the lower word groups than the document frequency is the predetermined threshold searches a candidate document to be output candidate by using the inverted file of the impact value order, second As a step search, for a word group whose document appearance frequency is equal to or higher than the predetermined threshold value , the document ID of the candidate document searched in the first step search is obtained using the transposed file in the document ID order. by searching on the document ID order of the inverted file, the candidate document search means for searching for a candidate document to be output candidates be output to confirm the candidate documents at the time of satisfying the inputted retrieval conditions Characterized in that it comprises a candidate document output means.

In order to solve the above problem, the text search method according to claim 7 includes a word appearance frequency calculated in advance for each word in each text document having a document ID, a total number of words included in the text document, and , A transposed file in order of impact value constructed for each word using an impact value indicating a weight value obtained based on the above, a transposed file in order of document ID constructed for each word included in each text document, and A text search method of a text search device for searching the text document as a search target using a plurality of transposed files including a document search index, the text search device comprising: search condition input means; word extraction Means, a word group classification means, a candidate document search means, and a candidate document output means. A search condition input step for inputting a search expression including a word and the number of documents as a specified search condition, and a word for extracting each word included in the input search expression based on a word dictionary by the word extraction means Based on the document appearance frequency calculated in advance through all the text documents to be searched for each word by the extraction step and the word group classification means, the extracted frequency of each word is set to a predetermined threshold. The document frequency is lower than a predetermined threshold as a first-stage search by the word group classification step for classifying into a word group lower than the value and a word group that is not, and the candidate document search means the word groups to explore the candidate document to be output candidate by using the inverted file of the impact values in order, as the search of the second step, teeth the document frequency of occurrence of a predetermined For had values above word group by utilizing the inverted file of the document ID order, the document ID of the first stage has been searched candidate documents in search by searching on the document ID order of the inverted file, A candidate document searching step for searching for a candidate document as an output candidate; and a candidate document output step for determining and outputting the candidate document when the input search condition is satisfied by the candidate document output means. It is characterized by performing.

  According to the text search device according to claim 1 or the text search method according to claim 7, the text search device has a higher performance for a word group that is included in the search formula and has a low document appearance frequency throughout the entire document. A supplementary document is searched using a transposed file in order of high impact value, and then candidate documents are narrowed down using a transposed file in order of document IDs for the remaining words included in the search formula. As described above, the text search apparatus appropriately uses the two types of transposed files according to the frequency of appearance of documents through all the documents to be searched for words included in the search formula. When a transposed file in document ID order is used alone as in the prior art, performance is generally not as good as when a transposed file in order of impact value is used alone as in the prior art. However, the text search apparatus of the present invention narrows down candidate documents to some extent using the transposed file in the order of impact values in the first stage, and then uses the transposed file in the order of document ID in the second stage. Therefore, in the second stage, it is not necessary to simply search for each document in descending order of the document ID, and it is possible to appropriately skip other than the document IDs of the documents narrowed down in the first stage. As a result, it is possible to avoid all scans in the transposed file in document ID order. Thereby, the text search apparatus can improve performance compared with the past.

  Further, the text search device according to claim 2 is the text search device according to claim 1, wherein the candidate document search means performs the impact on the word group whose document appearance frequency is lower than the predetermined threshold. First search means for reading a transposition file in order of values, calculating a word-dependent document score depending on the search formula for each document in descending order of impact value, and searching for the candidate document based on the calculated word-dependent document score And a number determination means for determining whether or not the number of searched candidate documents is larger than the specified number of documents, and the number of searched candidate documents is greater than the specified number of documents. When it becomes large, the transposition file in the document ID order for the word group whose document appearance frequency is equal to or higher than the predetermined threshold is read, and the first search means The candidate document is obtained by alternately using second search means for calculating a score for each document while searching for a candidate document that matches the document ID of the searched candidate document, and the first search means and the second search means. And narrowing-down control means for determining a rank within the specified number of documents as the rank of the candidate document.

  According to this configuration, as a first stage, the text search apparatus searches for candidate documents using the transposed file in order of impact value until the searched candidate documents reach the designated number of documents. For this reason, it is possible to search documents of the number of documents requested by the user who has designated the number of documents at high speed. After the specified number of documents is exceeded, as a second step, the score for each document is calculated while matching the candidate document searched from the transposed file in document ID order with the candidate document searched in the first step. To do. Therefore, for example, by providing a maximum score that can be taken by each document as a score for each document, a cut-off process for removing a predetermined number of documents from candidate documents that have reached the designated number of documents is performed. It can be carried out. Then, the text search device executes the first stage again starting from the number of candidate documents decreased due to the cut-off, and in order of impact value until the searched candidate documents reach the designated number of documents. The candidate document is searched using the transposed file. The same applies hereinafter. As a result, it is possible to output a suitable candidate document selected from a wider range of documents as a candidate document having a high score without being biased in the text document group to be searched.

  The text search device according to claim 3 is the text search device according to claim 2, wherein the first search means performs the impact on the word group whose document appearance frequency is lower than the predetermined threshold. A first processing unit that reads a transposition file in order of values and calculates a word-dependent document score depending on the search formula in descending order of impact value for each document; and a document-specific score order calculated for each text document in advance. The document-specific score corresponding to the document for which the word-dependent document score is calculated is read from the transposed file in the document-specific score order, and the read document-specific score and the calculated word-dependent document score And second processing means for searching for the candidate document using a linear sum.

  According to such a configuration, as a first step, the text search apparatus, when searching for candidate documents until the number of searched candidate documents reaches the designated number of documents, is a transposed file in order of impact value by the first processing means. Candidate documents are searched using a linear sum of the word-dependent document score calculated by using the document-specific score read from the transposition file in the document-specific score order. Therefore, since the text search device uses a linear sum of the word-dependent document score and the document-specific score, unlike in the past, the word-dependent document score (document-specific score) is searched as if It can be handled in the same manner as if it is a kind of score (word-dependent document score) depending on the word included in the expression. Here, the document-specific score is a score that does not depend on words included in the document, and is, for example, PageRank (registered trademark). In other words, the text search device of the present invention is set by setting the document-specific score such that, for example, a Web document that is viewed and used by more users among Web documents (documents) has a higher value. If the text document to be searched is a Web document (document), for example, a document that includes words included in a specified search expression from Web documents that are viewed and used by more users Can be output as a candidate document. As a result, the text search apparatus of the present invention can improve the performance of the search process as compared to the conventional case.

  According to a fourth aspect of the present invention, there is provided the text search device according to the third aspect, in which the unique score calculation means for calculating a document-specific score for each text document and the calculated document-specific score. And a document unique score transposed file construction means for constructing a transposed file in the document unique score order.

  According to such a configuration, the text search apparatus calculates a document-specific score for each text document to be searched before performing the search process, constructs a transposed file in the document-specific score order, and constructs the document-specific score thus constructed. It is possible to store and hold sequential transposed files. Accordingly, the text search apparatus uses the transpose file in the document unique score order stored in the storage device that stores and holds the transpose file in the document unique score order, for example, searches the text information stored in the accumulation device as a search target. Can be executed.

  The text search device according to claim 5 is the text search device according to any one of claims 1 to 4, wherein the text search device according to the document ID is constructed for each word included in each text document. The transposed file is one in which the maximum impact value is held for all the text documents to be searched for the word.

  According to such a configuration, in the search process using the transposed file in the order of the impact value, the text search apparatus performs the search process using the transposed file in the order of the document ID so as to cut off based on the impact value. Can be cut off based on the maximum impact value held by the transposed file in document ID order. Therefore, when a transposition file in document ID order is used in combination with a transposition file in impact value order, a cut-off can be performed when any type of transposition file is used. This makes it possible to output an appropriate candidate document selected from a wider range of documents in the text document group to be searched.

  Further, the text search device according to claim 6 is the text search device according to any one of claims 1 to 5, wherein the transposed file in the order of the impact value, the transposed file in the order of the document ID, Index construction means for constructing a text document, the index construction means for reading a text document group as a search target, and extracting a word from each of the read text documents based on a word dictionary A document feature amount calculating means for calculating a total number of words and a word appearance frequency of each word as a feature amount of words included in the document, and calculating a document appearance frequency through all text documents to be searched for each word; The document appearance frequency is compared through all the text documents to be searched, and the document appearance frequency of each word is a predetermined threshold. An appearance frequency determining means for determining whether or not the document appearance frequency of the word is lower than the predetermined threshold value, the transposed file in the order of the impact value for the word is used for the document search. When the first inverted file construction unit constructed as an index and the document appearance frequency of a word are equal to or higher than the predetermined threshold value, the maximum impact value is calculated for all the text documents to be searched for the word. And second transposed file constructing means for constructing a transposed file in the document ID order including the calculated maximum impact value as the index for document retrieval.

  According to such a configuration, the text search apparatus calculates the document appearance frequency through all documents for each word from the text document group to be searched before performing the search process, and builds based on the calculated document appearance frequency. As the appropriate type of the transposed file, one of the transposed file in the order of impact values and the transposed file in the order of document ID is selected and the selected transposed file is constructed. Therefore, the text search apparatus can accumulate and hold the calculated document appearance frequency of each word, the transposed file in the order of impact values, and the transposed file in the order of document ID. Therefore, the text search device uses the document appearance frequency and each transposed file stored in the storage device that stores and holds the document appearance frequency, the transposed file in the order of the impact value, and the transposed file in the order of the document ID, for example, in the storage device. Search processing can be executed using the stored text information as a search target.

  A text search program according to claim 8 is a program for realizing the function of the text search device according to any one of claims 1 to 6 by a computer. By being configured in this way, a computer in which this program is installed can realize each function based on this program.

  A computer-readable recording medium according to a ninth aspect stores the text search program according to the eighth aspect. By being configured in this way, a computer equipped with this recording medium can realize each function based on a program recorded on this recording medium.

  According to the present invention, the text search device searches for a document using a transposed file in order of impact value when the document appearance frequency is low throughout all text documents to be searched for words included in the search expression. On the other hand, when the frequency of appearance of documents through all text documents is high, the transposition file in order of document IDs that can be skipped is used, which has the effect of speeding up the search process. Therefore, the processing performance of the search process can be improved when using a transposed file in order of impact value.

  Hereinafter, the best mode for carrying out a text search apparatus and text search method of the present invention (hereinafter referred to as “embodiment”) will be described in detail by dividing it into a first embodiment and a second embodiment with reference to the drawings. To do.

(First embodiment)
[Configuration of text search device]
FIG. 1 is a block diagram schematically showing a text search apparatus according to the first embodiment of the present invention.
The text search apparatus 1 includes a transposed file in order of impact value constructed for each word using an impact value in each text document having a document ID, and a transposed in order of document ID constructed for each word included in each text document. A text document is searched as a search target by using a plurality of transposed files including the file as an index for document search. Here, the impact value indicates a numerical value of the weight obtained based on the word appearance frequency (tf value) calculated in advance for each word in each text document and the total number of words included in the text document.

  The text search device 1 includes, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), and an input / output interface. As shown in FIG. 1, the text search device 1 includes an index construction device 10, a search processing device 20, an index storage device 30, a document storage device 40, and a word dictionary storage device 50 as text search engines. I have.

  The text search device 1 is a search condition (search formula, number of documents) input from an input device M (see FIG. 6) such as a mouse or a keyboard connected to a terminal device 60 such as a personal computer used by a user via a cable. Enter. The text search apparatus 1 can also input user search conditions received via a communication network such as the Internet.

  The index construction device (index construction means) 10 reads a text document to be searched stored in the document storage device 40, and performs morphological analysis of the text document using the word dictionary stored in the word dictionary storage device 50. Then, the word is extracted, a transposed file is constructed, and the constructed transposed file is written in the index storage device 30. In this embodiment, the transposed file is indicated by an index. The index indicates an index indicating which word is in which document in order to improve the data search speed.

  The search processing device 20 inputs the number of documents specified by the user (number of results to be returned) and the search formula (a plurality of words) from the terminal device 60, analyzes the search formula, extracts a word group, and indexes The transposed file corresponding to the extracted word is read from the storage device 30, the candidate document satisfying the search condition is searched, and the search result (candidate document) is output to the terminal device 60.

The index storage device 30 stores a transposed file (index), and is composed of, for example, a general hard disk.
The document storage device 40 accumulates a plurality of search target documents (text documents, text information) to be searched, and is composed of, for example, a general hard disk. A document ID (identification information) is assigned to the search target document.
The word dictionary storage device 50 stores a word dictionary, and is composed of, for example, a general hard disk. The index storage device 30, the document storage device 40, and the word dictionary storage device 50 can also be configured with one or more external storage devices.

[Overview of text search device search processing]
FIG. 2 is an explanatory diagram schematically showing an outline of the text search apparatus according to the first embodiment of the present invention. In FIG. 2, the index construction device 10 and the document storage device 40 shown in FIG.

  The search processing device 20 extracts the word group 102 included in the search condition 101 input from the terminal device 60 with reference to the word dictionary stored in the word dictionary storage device 50 and stores it in the index storage device 30. Using the transposed files 71, 72, and 74 in the order of impact values and the transposed file 73 in the order of the document ID, the candidate document to be output is searched, and the search result is output to the terminal device 60. Compared with the conventional text search device 300 shown in FIG. 3, the text search device 1 uses the transposed file 73 in the document ID order together with the transposed files 71, 72, and 74 in the order of impact values. Are different.

  In the example shown in FIG. 2, the search processing device 20 of the text search device 1 divides the phrase “Tokyo station” as the search condition 101, and sets “Tokyo”, “no”, “station” as the word group 102. Is extracted. Then, the search processing device 20 corresponds to the document appearance frequency (df value) of each word, and as a first step, for the transposed files 71 and 74 in the order of impact values for “Tokyo” and “Station”, reference numeral 103 is used. , 104, a predetermined score is calculated for each document in descending order of impact value (descending order). As a result, the document (d16) having the document ID “16” indicated by the reference numerals 105 and 106 is searched from the two transposed files 71 and 74 as a common document.

  Then, as a second stage, the search processing device 20 does not scan the transposed file 73 in the document ID order for “no” one by one from the document (d1) with the document ID “1”. As indicated by the arrow 107, the document ID is skipped as appropriate, and as indicated by the arrow 108, the document IDs searched in the transposed files 71 and 74 in order of impact value are reached in ascending order of document ID (ascending order). Until a document (d16) whose document ID indicated by reference numeral 109 is “16” is searched, and when it is found, a predetermined score is calculated for each document. Thereafter, the first stage and the second stage are repeated to output the designated number of document documents. The number of transposed files (indexes) is not limited to the illustrated number (4).

  Next, a detailed configuration and operation of the text search device 1 for realizing the search processing by the search processing device 20 of the text search device 1 shown in FIG. 2 will be described. Hereinafter, for convenience of explanation, the configuration and operation of the index construction device 10 (see FIG. 1) and the search processing device 20 will be described in detail.

[Configuration of index building device]
FIG. 4 is a block diagram schematically showing an example of the index construction apparatus shown in FIG.
The index construction device 10 constructs a transposed file in the order of impact values and a transposed file in the order of document ID, and includes a document reading unit 11, a document feature amount calculating unit 12, an appearance frequency determining unit 13, 1 transposition file construction means 14 and second transposition file construction means 15 are provided.

The document reading unit 11 reads a text document group to be searched from the document storage device 40.
Based on the word dictionary stored in the word dictionary storage device 50, the document feature quantity calculation means 12 extracts words from each text document read by the document reading means 11 and uses them as the feature quantities of the words included in the text document. The total number of words and the word appearance frequency (tf value: Term Frequency) of each word are calculated, and the document appearance frequency (df value: Document Frequency) through all text documents to be searched is calculated for each word. . The df value indicates the number of documents including the word. The calculated df value (document appearance frequency) is stored in a predetermined area of the word dictionary storage device 50, for example.

  The appearance frequency determining means 13 compares the document appearance frequency of each word through the entire text document group for each word, and determines whether the document appearance frequency is lower than a predetermined threshold value for each word. In the present embodiment, the appearance frequency determination unit 13 selects an unprocessed word and determines whether or not the df value of the selected word is equal to or greater than a predetermined threshold value.

  When the document appearance frequency of a word is lower than a predetermined threshold, the first transposed file construction unit 14 constructs a transposed file in order of impact value for the word as an index for document search. The first transposed file construction means 14 can construct a transposed file in the order of impact values by a known method. The details of the construction of the transposed file in order of impact value are described in Non-Patent Documents 1 to 4, for example. The first inverted file construction unit 14 stores the constructed inverted file 31 in the order of impact values in the index storage device 30. The transposed file 31 in order of impact value is a transposed file that lists documents (document IDs) in descending order of impact value. In FIG. 4, only one transposed file 31 in order of impact value is illustrated, but it is provided for each word.

  When the document appearance frequency of a word is equal to or higher than a predetermined threshold, the second transposed file construction unit 15 calculates the maximum impact value for the word through the entire text document group, and includes the calculated maximum impact value. The transposed file in the document ID order is constructed as a document search index. The second transposed file construction means 15 can construct a transposed file in document ID order by a known method. Details of the construction of the transposed file in document ID order are described in Non-Patent Document 1, for example. The second transposed file construction unit 15 stores the constructed transposed file 32 in the document ID order in the index storage device 30. The transposed file 32 in document ID order is a transposed file that lists documents (document IDs) in ascending order of document IDs. In FIG. 4, only one transposed file 32 in the order of document ID is illustrated, but it is provided for each word.

  The document reading means 11, the document feature quantity calculating means 12, the appearance frequency determining means 13, the first inverted file constructing means 14, and the second inverted file constructing means 15 are stored in a predetermined program stored in the storage means by the CPU. This is realized by expanding and executing in the RAM.

[Operation of index building device]
The operation of the index construction apparatus 10 shown in FIG. 4 will be described with reference to FIG. 5 (refer to FIG. 4 as appropriate). FIG. 5 is a flowchart showing the operation of the index construction apparatus shown in FIG. The index construction apparatus 10 reads a text document group to be searched by the document reading unit 11 (step S1), extracts a word from each text document by the document feature amount calculation unit 12, and extracts the word contained in each document. The total number, tf value, and df value are calculated (step S2).

  Then, the index construction device 10 selects the unprocessed word for which the transposed file is not constructed by the appearance frequency determination unit 13 (step S3). Then, the index construction device 10 determines whether or not the df value of the selected word is greater than or equal to a predetermined threshold value by the appearance frequency determination unit 13 (step S4). When the df value of the selected word is equal to or greater than the predetermined threshold (step S4: Yes), the index construction device 10 calculates the maximum impact value for the word by the second transposed file construction means 15, A transposed file in the document ID order including the impact value is constructed (step S5).

  On the other hand, when the df value of the selected word is lower than the predetermined threshold value in step S4 (step S4: No), the index construction device 10 uses the first inverted file construction unit 14 to determine the order of impact values for the word. Is constructed (step S6). Subsequent to step S5 or step S6, the index construction device 10 determines whether or not all words have been processed by the appearance frequency determination means 13 (step S7). When there is an unprocessed word (step S7: No), the index construction device 10 returns to step S3. On the other hand, when all the target words are processed (step S7: Yes), the index construction device 10 ends the process of constructing the transposed file (index).

  Specifically, in FIG. 2, transposed files 71, 72, and 74 in the order of impact values are constructed for “Tokyo”, “Tomato”, and “Station” when the index is constructed. The transposed file 73 in the document ID order is constructed. However, in FIG. 2, the maximum impact value is not shown in the transposed file 73 in the document ID order. Here, an example of the structure of the transposed file in the document ID order including the maximum impact value is shown in FIG. The text search device 1B shown in FIG. 14 is different in that the transposed file 76 in the document ID order is stored in the index storage device 30. The transposed file 76 in the document ID order is the same as the transposed file 73 in the document ID order shown in FIG. 2 except that the left end includes the maximum impact value (s12.0). Here, the column including the maximum impact value (s12.0) is not limited to the left end.

[Configuration of search processing device]
FIG. 6 is a block diagram schematically showing an example of the search processing device shown in FIG. As shown in FIG. 6, the search processing device 20 includes search condition input means 21, word extraction means 22, word group classification means 23, candidate document search means 24, and candidate document output means 25. .

  The search condition input means 21 inputs a search expression including a plurality of words and the number of documents as designated search conditions. In this embodiment, the search condition input means 21 inputs a search condition from the terminal device 60 connected to the text search device 1 with a cable. For example, the search condition may be received from the terminal device 60 via a communication network such as the Internet. The user of the terminal device 60 operates the input device M such as a mouse or a keyboard, for example, and inputs the search formula and the number of documents to the terminal device 60 as a search condition.

The word extraction unit 22 extracts each word included in the input search formula based on the word dictionary stored in the word dictionary storage device 50.
The word group classification unit 23 determines each word extracted by the word extraction unit 22 based on a document appearance frequency calculated in advance as a word group having a document appearance frequency lower than a predetermined threshold and a word that is not so. They are classified into groups. In this embodiment, based on the df value (document appearance frequency) obtained at the time of index construction and stored in a predetermined area of the word dictionary storage device 50, for example, the word group classification means 23 extracts the extracted word group. It classifies into a word group with a high df value (document appearance frequency) and a low word group.

  Candidate document search means 24 searches for candidate documents that are output candidates using a transposed file in order of impact value for a word group whose document appearance frequency is lower than a predetermined threshold, and then the document appearance frequency is predetermined. For a word group equal to or greater than the threshold value, a candidate document as an output candidate is searched using a transposed file in document ID order. In the present embodiment, the candidate document search means 24 is a module that executes candidate document search processing. As shown in FIG. 6, the first search means 241, the number determination means 242, and the second search are included as submodules. Means 243 and narrowing control means 244 are provided.

  The first search means 241 reads the transposed file 31 in the order of impact value for a word group whose word appearance frequency is lower than a predetermined threshold (word group having a low df value), and searches for each document in descending order of impact value. A word-dependent document score depending on the document is calculated, and candidate documents are searched based on the calculated word-dependent document score. Here, for the word-dependent document score, for example, a predetermined ranking function based on a search expression such as tf-idf or BM25 can be used. In addition, when the first search unit 241 searches for candidate documents, the k-best (scores with the highest score) of the scores that each document can take can be calculated to cut off the search process. . For example, Non-Patent Document 4 describes the details of the search processing cut-off method.

  The number determination unit 242 determines whether or not the number of candidate documents searched by the first search unit 241 is larger than the number of documents specified by the search condition. In the present embodiment, when the number of candidate documents searched by the first search unit 241 becomes larger than the number of documents specified by the search condition, the number determination unit 242 notifies the second search unit 243 of that fact. Output to.

  When the number of candidate documents searched by the first searching unit 241 is larger than the designated number of documents, the second searching unit 243 determines a word group (df value) having a document appearance frequency equal to or higher than a predetermined threshold. A transposed file 32 in the document ID order for the high-word group), and calculates a score for each document while searching for candidate documents that match the document ID of the candidate document searched by the first search means 241. . When searching for candidate documents, the second search unit 243 scans the transposed file 32 in document ID order while skipping appropriately. Details of the method of scanning the transposed file 32 in the document ID order while skipping appropriately are described in Non-Patent Document 1, for example. Further, when searching for a candidate document, the second search unit 243 performs a cut-off using the maximum impact value held in the transposed file 32 in document ID order.

  The narrowing-down control unit 244 narrows down candidate documents by alternately using the first search unit 241 and the second search unit 243, and determines a rank within the number of documents designated as the rank of candidate documents. In the present embodiment, the narrowing down control unit 244 determines whether or not the order of the number of documents designated by the search condition has been determined. For example, Non-Patent Document 4 describes a method in which the narrowing-down control unit 244 identifies candidate documents that satisfy the search expression condition and performs the final stage cut-off process using the number of results specified by the user. Can be used. In the method described in Non-Patent Document 4, a three-stage cut-off process is performed.

  The candidate document output means 25 determines and outputs the candidate document when the input search condition is satisfied. In the present embodiment, the candidate document output unit 25 outputs a list of candidate documents to the terminal device 60 connected to the text search device 1 with a cable. For example, a list of candidate documents may be transmitted to the terminal device 60 via a communication network such as the Internet. The terminal device 60 outputs and displays the acquired list of candidate documents on the output device D such as a liquid crystal display and presents it to the user.

  The search condition input means 21, the word extraction means 22, the word group classification means 23, the candidate document search means 24, and the candidate document output means 25 are arranged such that the CPU expands a predetermined program stored in the storage means to the RAM. It is realized by executing.

[Operation of search processing device]
The operation of the search processing device 20 shown in FIG. 6 will be described with reference to FIG. 7 (refer to FIG. 6 as appropriate). FIG. 7 is a flowchart showing the operation of the search processing device shown in FIG. First, the search processing device 20 inputs a search expression and the number of documents as specified search conditions by the search condition input means 21 (step S11: search condition input step). Then, the search processing device 20 uses the word extraction means 22 to analyze the search formula and extract the included word group (step S12: word extraction step). Then, the search processing device 20 classifies the extracted word group into a word group having a high df value (appearance frequency) and a word group having a low df value (appearance frequency) by the word group classification means 23 (step S13: word group classification step). Then, the search processing device 20 executes candidate document search processing by the candidate document search means 24 (step S14: candidate document search step). Then, the search processing device 20 determines and outputs the candidate document as a search result when the input search condition is satisfied by the candidate document output means 25 (step S15: candidate document output step).

<Candidate document search process>
Next, the candidate document search process in step S14 will be described with reference to FIG. 8 (see FIG. 6 as appropriate). FIG. 8 is a flowchart showing details of the candidate document search process shown in FIG. First, the candidate document search unit 24 uses the first search unit 241 to read the transposed file 31 in order of impact value for a word group having a low df value, calculate a word-dependent document score, and select a candidate document based on the calculated score. Search is performed (step S21). Then, the candidate document search unit 24 determines whether or not the number of candidate documents exceeds the designated number of documents by the number determination unit 242 (step S23). If the number of candidate documents exceeds the designated number of documents (step S23: Yes), the candidate document search means 24 uses the second search means 243 to transpose the file 32 in the document ID order for the word group having a high df value. (Step S25), and a candidate document matching the document ID of the candidate document searched in step S21 is searched (step S27). In step S27, when searching for a candidate document, the second search unit 243 scans the transposed file in the document ID order while skipping appropriately, and calculates a score for each document for each candidate document while matching the document ID. .

Subsequently, the candidate document search unit 24 determines whether the rank control unit 244 determines the order of the number of documents specified by the search condition (step S29). When the rank of the designated number of documents is determined (step S29: Yes), the candidate document search unit 24 ends the process. On the other hand, if the order of the designated number of documents has not been determined (step S29: No), the candidate document search unit 24 returns to step S21 and repeats the processing.
In step S23, if the number of candidate documents does not exceed the number of documents specified by the search condition (step S23: No), the candidate document search unit 24 returns to step S21, in order of impact value. The transposed file is further read to search for other candidate documents. Thereby, as a candidate document having a high score, it is possible to output an appropriate candidate document selected from a wider variety of documents without being biased in a document group to be searched.

  Note that the text search device 1 can also be realized by executing a text search program that causes a general computer to function as the search processing device 20. The text search device 1 can also be realized by executing a text search program that causes a general computer to function as the index construction device 10 and the search processing device 20 described above. These programs can be distributed via a communication line, or can be written and distributed on a computer-readable recording medium such as a CD-ROM.

  According to this embodiment, when the df value of a word included in the search formula is low, the text search device 1 searches for a document using the transposed file 31 in order of impact value, while the df value is high. For words, the transposition file 32 in order of document IDs that can be skipped is used, so that the search process can be speeded up. Therefore, in the text search apparatus that uses the transposed file 31 in the order of impact values, the processing performance of the search process can be improved. As a result, consumption of resources (disk, memory, CPU) can be suppressed.

(Second Embodiment)
[Overview of text search device search processing]
FIG. 9 is an explanatory diagram schematically showing an outline of a text search apparatus according to the second embodiment of the present invention. The text search device 1A includes an index construction device 10A, a search processing device 20A, an index storage device 30, a document storage device 40 (see FIG. 1), and a word dictionary storage device 50. In FIG. 9, the document storage device 40 is omitted and displayed in order to explain the outline of the search processing of the text search device 1A. Moreover, about the structure similar to FIG. 2, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The index construction device 10A constructs a transposition file 75 in the document unique score order in the document order with the document unique score in descending order in addition to the transposed file in the order of impact values and the transposition file in the document ID order. The transposed file in document-specific score order is a transposed file that stores a score for each document that does not depend on words included in the search formula. Here, as the score for each document that does not depend on the word, the score for each document (Web page) calculated by using the data of the number of clicks for each Web page obtained from the search server log, or PageRank (registered trademark) Etc. For PageRank (registered trademark), for example, (Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd, “The PageRank Citation Ranking: Bringing Order to the Web”, Jan. 29, 1998, <URL: http: / /WWW-db.stanford.edu/~backru/pageranksub.ps>).

  The search processing device 20A uses the transposed files 71, 72, 74 in order of impact values, the transposed file 73 in order of document ID, and the transposed file 75 in order of document unique score, which are stored in the index storage device 30. The candidate document to be output is searched, and the search result (candidate document) is output to the terminal device 60. This search processing device 20A differs from the search processing device 20 of the first embodiment in the first stage search method.

  Specifically, in the example shown in FIG. 9, the search processing device 20A of the text search device 1A uses the transposed files 71 and 74 in order of impact value for “Tokyo” and “Station” as the first stage. Then, a predetermined score is calculated for each document, and then the sum of the document unique score read from the transposed file 75 in the document unique score order and the calculated score is obtained, and a candidate document is searched based on the total score. . As a result, the document (d16) having the document ID “16” indicated by the reference numerals 105, 106, and 111 is searched from the three transposed files 71, 74, and 75 as a common document.

  Next, a detailed configuration and operation of the text search device 1A for realizing the search processing by the search processing device 20A of the text search device 1A shown in FIG. 9 will be described. In the following, for convenience of explanation, the configuration and operation of the index construction device 10A and the search processing device 20A will be described in detail.

[Configuration of index building device]
FIG. 10 is a block diagram schematically showing an example of the index construction device shown in FIG. The index construction apparatus 10A shown in FIG. 10 has the same configuration as the index construction apparatus 10 shown in FIG. 4 except that it includes a unique score calculation means 16 and a document unique score transposition file construction means 17. The same reference numerals are given to the components, and description thereof is omitted.

The unique score calculation means 16 reads a text document group to be searched and calculates a document-specific score for each text document.
The document unique score transposed file construction unit 17 constructs the transposed file 33 in the document unique score order based on the document unique score calculated by the unique score calculation unit 16. The document unique score transposed file construction means 17 stores the constructed transposed file 33 in the document unique score order in the index storage device 30.

[Operation of index building device]
The operation of the index construction apparatus 10A shown in FIG. 10 for constructing the transposition file 31 in the order of impact values and the transposition file 32 in the order of document ID is the same as that of the index construction apparatus 10 of the first embodiment, and the description thereof is omitted. Will be described with reference to FIG. 11 (see FIG. 10 as appropriate). FIG. 11 is a flowchart showing an operation of constructing a document unique score transposed file by the index construction apparatus shown in FIG. The index construction apparatus 10A reads the text document group to be searched by the unique score calculation means 16 (step S31), and calculates a document-specific score for each text document by the unique score calculation means 16 (step S32). Then, the index construction apparatus 10A constructs the transposed file 33 in the document unique score order by the document unique score transposed file construction means 17 (step S33). It should be noted that the index construction apparatus 10A may construct the transposition file 33 in the document unique score order before or after constructing the transposition file 31 in the impact value order and the transposition file 32 in the document ID order. You may build it in parallel with the file.

[Configuration of search processing device]
FIG. 12 is a block diagram schematically showing an example of the search processing device shown in FIG. The search processing device 20A has the same configuration as that of the search processing device 20 shown in FIG. 6 except that the configuration of the first search means 241A is different.
The first search unit 241A includes a first processing unit 251 and a second processing unit 252 as shown in FIG.

  The first processing unit 251 reads the transposed file 31 in order of impact value for a word group whose word appearance frequency is lower than a predetermined threshold (a word group having a low df value), and searches the document in descending order of impact value. The word-dependent document score depending on the expression is calculated.

  The second processing means 252 reads a document-specific score corresponding to the document for which the word-dependent document score has been calculated by the first processing means 251 from the pre-constructed transposition file 33 in the document-specific score order. The candidate document is searched using a linear sum of the score and the word-dependent document score calculated by the first processing means 251. For example, for a document, if the document-specific score is “5” and the word-dependent document score is “10”, the second processing unit 252 sets the total score of the document as “15” and selects a candidate. Search for a document. The linear sum is not limited to simple addition, but includes addition after weighting.

[Operation of search processing device]
The search processing device 20A shown in FIG. 12 is the same as the search processing device 20 of the first embodiment, except that only the candidate document search processing (see step S14, see FIG. 7) is different in the overall operation. Since it operates, description of the overall operation will be omitted, and the candidate document search processing will be described with reference to FIG. 13 (refer to FIG. 12 as appropriate). FIG. 13 is a flowchart showing details of candidate document search processing by the search processing device shown in FIG.

  The candidate document search unit 24 reads the transposed file 31 in order of impact value for the word group having a low df value by the first processing unit 251 of the first search unit 241A, and calculates a word-dependent document score (step S21a). Then, the candidate document search means 24 reads the transposed file in the document unique score order by the second processing means 252 of the first search means 241A, the read score, and the score (word-dependent document score) calculated in step S21a. A candidate document is searched based on the linear sum of (step S21b). The following operation is the same as the candidate document search process described with reference to FIG. However, in step S27b, the candidate document search unit 24 searches for candidate documents that match the document ID of the candidate document searched in step S21b by the second search unit 243.

  According to the present embodiment, the text search apparatus 1A is linear between the word-dependent document score calculated using the transposed file 31 in the order of impact values and the document-specific score read from the transposed file 33 in the document-specific score order. Candidate documents can be searched using sums. Therefore, the text search apparatus 1A similarly applies the score of a document that does not depend on a word (document-specific score) as if it is a kind of score (word-dependent document score) that depends on a word included in a search expression. Can be handled. As a result, when the document to be searched for is a Web document (document), for example, the text search device 1A includes the specified search expression from Web documents that are viewed and used by more users. It is possible to output a document including a word as a candidate document.

  As mentioned above, although each embodiment of this invention was described, this invention is not limited to these, It can implement in the range which does not change the meaning. For example, in each embodiment, the text search device 1 (1A) has been described as the best mode including the index construction device 10 (10A). However, the index construction device 10 (10A) is not essential, and a transposed file constructed in advance. And the df value calculated in advance may be stored. Similarly, the text search device 1 (1A) may not necessarily include the document storage device 40. Further, the text document to be searched (search target document) does not need to be stored in one document storage device 40, and may be distributed and arranged in a plurality of storage devices on the network.

  In each embodiment, three words are exemplified as the search condition. However, the number of words input by the search condition may be plural. Moreover, not only the phrase of a word but a sentence can also be used as a search expression. The words included in the search expression are not limited to nouns and particles, but may be other parts of speech such as adjectives. Further, the language of the word included in the search expression is not limited to Japanese, but may be another language such as English, French, Chinese.

  In the second embodiment, the index construction apparatus 10A creates only one transposed file 33 in the document unique score order. However, a plurality of them may be created. For example, a transposed file with a score for each document (Web page) calculated using data on the number of clicks for each Web page obtained from a search server log and a transposed file with a PageRank (registered trademark) score are created. can do. In this case, the second processing unit 252 reads the document-specific score corresponding to the document for which the word-dependent document score is calculated by the first processing unit 251 from the two transposed files in the document-specific score order. A candidate document is searched for using a linear sum of the document-specific score and the word-dependent document score calculated by the first processing means 251. At this time, the two kinds of document-specific scores can be weighted separately and then added to the word-dependent document score. Also in this case, it is possible to output a document including a word included in a designated search expression as a candidate document from among Web documents that are browsed and used by more users.

It is a lineblock diagram showing typically the text search device concerning a 1st embodiment of the present invention. It is explanatory drawing which shows typically the outline | summary of the text search device concerning 1st Embodiment of this invention. It is explanatory drawing which shows the outline | summary of the conventional text search apparatus typically. It is a block diagram which shows typically an example of the index construction apparatus shown in FIG. It is a flowchart which shows operation | movement of the index construction apparatus shown in FIG. It is a block diagram which shows typically an example of the search processing apparatus shown in FIG. It is a flowchart which shows operation | movement of the search processing apparatus shown in FIG. It is a flowchart which shows the detail of the candidate document search process shown in FIG. It is explanatory drawing which shows typically the outline | summary of the text search device concerning 2nd Embodiment of this invention. It is a block diagram which shows typically an example of the index construction apparatus shown in FIG. It is a flowchart which shows the operation | movement which builds the document original score transposition file by the index construction apparatus shown in FIG. It is a block diagram which shows typically an example of the search processing apparatus shown in FIG. It is a flowchart which shows the detail of the candidate document search process by the search processing apparatus shown in FIG. It is explanatory drawing which shows an example of the transposition file of document ID order.

Explanation of symbols

1 (1A, 1B) Text search device 10 (10A) Index construction device (index construction means)
DESCRIPTION OF SYMBOLS 11 Document reading means 12 Document feature-value calculation means 13 Appearance frequency discrimination means 14 1st transposition file construction means 15 2nd transposition file construction means 16 Original score calculation means 17 Document original score transposition file construction means 20 (20A) Search processing apparatus 21 Search condition input means 22 Word extraction means 23 Word group classification means 24 Candidate document search means 241 (241A) First search means 242 Number of cases determination means 243 Second search means 244 Narrow down control means 25 Candidate document output means 251 First processing means 252 Second processing means 30 Index storage device 40 Document storage device 50 Word dictionary storage device 60 Terminal device M Input device D Output device

Claims (9)

Constructed for each word using an impact value indicating a weight value obtained based on the word appearance frequency calculated in advance for each word in each text document having the document ID and the total number of words included in the text document. A plurality of transposed files including a transposed file in order of impact value and a transposed file in order of document ID constructed for each word included in each text document, as the index for document search, A text search device for searching,
A search condition input means for inputting a search expression including a plurality of words and the number of documents as a specified search condition;
Word extraction means for extracting each word included in the input search formula based on a word dictionary;
Based on the document appearance frequency calculated in advance through all the text documents to be searched for each word, each of the extracted words is not a word group whose document appearance frequency is lower than a predetermined threshold. Word group classification means for classifying into word groups;
As a first stage search, for a word group whose document appearance frequency is lower than a predetermined threshold, a candidate document that is an output candidate is searched using the transposed file in the order of impact value, and a second stage search is performed. As for the word group whose document appearance frequency is equal to or higher than the predetermined threshold , the document ID of the candidate document searched in the first stage search is used as the document ID using the transposed file in the document ID order. Candidate document search means for searching for candidate documents to be output candidates by searching on sequential transposed files;
A text search apparatus comprising candidate document output means for determining and outputting the candidate document when the input search condition is satisfied. The candidate document search means includes:
Read the transposed file in order of impact value for a word group whose document appearance frequency is lower than the predetermined threshold, calculate a word-dependent document score depending on the search formula for each document in order of high impact value, First search means for searching for the candidate document based on the calculated word-dependent document score;
A number determination means for determining whether or not the number of searched candidate documents is larger than the specified number of documents;
When the number of searched candidate documents is larger than the designated number of documents, the transposed file in the document ID order for the word group whose document appearance frequency is equal to or higher than the predetermined threshold is read. Second search means for calculating a score for each document while searching for a candidate document that matches the document ID of the candidate document searched by the first search means;
And narrowing down control means for narrowing down the candidate documents by alternately using the first search means and the second search means, and determining a rank within the designated number of documents as the rank of the candidate documents. The text search device according to claim 1. The first search means includes
A transposition file in the order of impact value is read for a word group whose document appearance frequency is lower than the predetermined threshold value, and a word-dependent document score depending on the search formula is calculated in descending order of impact value for each document. Processing means;
The document-specific score corresponding to the document for which the word-dependent document score has been calculated is read from the transposed file in the document-specific score order that is built in advance in the document-specific score order calculated for each text document, and the read The text search apparatus according to claim 2, further comprising: a second processing unit that searches for the candidate document using a linear sum of a document-specific score and the calculated word-dependent document score. Unique score calculating means for calculating a document-specific score for each text document;
4. The text search apparatus according to claim 3, further comprising: a document unique score transposed file construction unit that constructs a transposed file in the document unique score order based on the calculated document unique score.   The transposed file in document ID order constructed for each word included in each text document holds a maximum impact value for all the text documents to be searched for the word. The text search device according to any one of claims 1 to 4. Index construction means for constructing the transposed file in the impact value order and the transposed file in the document ID order,
The index construction means includes:
A document reading means for reading a text document group to be searched;
A word is extracted from each read text document based on the word dictionary, and the total number of words and the word appearance frequency of each word are calculated as the feature amount of the word included in the text document, and the search target is set for each word. A document feature amount calculating means for calculating a document appearance frequency through all text documents;
Appearance frequency determination means for comparing the document appearance frequency for each word through all the text documents to be searched, and determining whether the document appearance frequency of each word is lower than a predetermined threshold;
First transposed file construction means for constructing a transposed file in order of impact value for the word as an index for the document search when the document appearance frequency of the word is lower than the predetermined threshold;
When the document appearance frequency of a word is equal to or higher than the predetermined threshold, the maximum impact value is calculated for all the text documents to be searched for the word, and the document ID including the calculated maximum impact value The text search apparatus according to claim 1, further comprising: a second transposed file construction unit that constructs a sequential transposed file as the document search index. Constructed for each word using an impact value indicating a weight value obtained based on the word appearance frequency calculated in advance for each word in each text document having the document ID and the total number of words included in the text document. A plurality of transposed files including a transposed file in the order of impact values and a transposed file in the order of document IDs constructed for each word included in each text document as an index for document search, A text search method of a text search device for searching a text document,
The text search device includes search condition input means, word extraction means, word group classification means, candidate document search means, and candidate document output means,
A search condition input step of inputting, by the search condition input means, a search expression including a plurality of words and the number of documents as a specified search condition;
A word extraction step of extracting each word included in the inputted search expression based on a word dictionary by the word extraction means;
Based on the document appearance frequency calculated in advance through all text documents to be searched for each word by the word group classification unit, the extracted word is lower in the document appearance frequency than a predetermined threshold value. A word group classification step for classifying into a word group and a non-word group;
As a first-stage search , the candidate document search means searches for candidate documents that are output candidates using a transposed file in the order of impact values for a word group whose document appearance frequency is lower than a predetermined threshold. Then, as a second stage search, for the word group having the document appearance frequency equal to or higher than a predetermined threshold, using the transposed file in the document ID order, the candidate documents searched in the first stage search are searched. A candidate document search step of searching for a candidate document as an output candidate by searching the document ID on the transposed file in the document ID order ;
And a candidate document output step for determining and outputting the candidate document when the input search condition is satisfied by the candidate document output means.   A text search program for realizing the function of the text search device according to any one of claims 1 to 6 by a computer.   A computer-readable recording medium on which the text search program according to claim 8 is recorded.

Images