JP5145202B2 - Document search apparatus and document search program - Google Patents

Description

  The present invention relates to a technique for retrieving an electronic document corresponding to a keyword from a group of electronic documents.

  With the spread of computerization of documents and the explosive spread of the Internet, users of the Internet and corporate networks can browse a large amount of electronic documents. For such a large number of electronic documents, a search system (full-text search system) that can quickly search for a document that satisfies a search request expressed by a user is widely used. As a general expression method of a search request, a method of specifying a word string (keyword string) included in an electronic document to be searched is used.

  Since the number of electronic documents that contain a specific keyword is a part of the whole document, every time a search request is entered, the existence of the keyword is checked for every document that is stored. The process is repeated many times, which is inefficient. For this reason, Non-Patent Document 1 proposes a method of speeding up using a word group as an index word, a document group including the word, and an index called a word transposition index having an occurrence position of the index word in each document. Has been.

  When a large number of electronic documents are to be searched, when a plurality of keywords are input, it is common to execute a document search (AND search) that includes at least one of each keyword. On the other hand, instead of handling a plurality of keywords individually, there is a demand for performing a document search by phrase that retains adjacent information and order information in a search request for those keywords.

  However, when searching for a document in which a phrase appears using only a simple transposed index, there is a problem that the load for calculation is large. Because, in order to confirm that the words constituting the phrase appear adjacent in order, the values of the inverted index (transposed list) obtained using each word constituting the phrase as a key are merged, This is because it is necessary to sequentially check the list of results and check for each document whether each word constituting the phrase appears adjacently in the requested order.

  In response to the above problem, by treating not only the word-by-word index but the entire phrase as if it were one word, the index corresponding to the phrase (hereinafter referred to as the phrase transposed index) is maintained. There are known methods for reducing the computational load for searches involving phrases.

However, if phrase transposition indexes are prepared for all phrases, a huge amount of storage device is required, which is not practical. For this reason, it is necessary to select what phrase is to be stored for a limited amount of phrase transposed index based on some criteria. Thus, as a phrase selection criterion, Non-Patent Document 2 proposes a method for storing phrases that frequently appear in the past search history in a phrase transposed index.
“Distributed high-speed information collection / full-text search system InfoBee / Evangelist”, Hiroshi Takeno, Takashi Inoue, NTT R & D, vol. 52, no. 2, 2003, pp 78-84. “Fast Phrase querying with combined indexes”, Hugh E. et al. Williams, Justin Zobel and Dirk Bahr, ACM TOIS, vol. 22, no. 4,2004

  In the technique of Non-Patent Document 2, a phrase frequently used for search is preferentially stored in the phrase transposed index. However, with this method, there is a possibility that the calculation load of a search including a phrase cannot be reduced efficiently.

  That is, in the first place, the magnitude of the calculation load when the phrase search is handled only by the word transposed index depends on the length of the transposed list of each word constituting the phrase. Therefore, there has been a problem that the calculation load reduced by storing a phrase composed of words having a short transposed list is small.

  Therefore, in view of such a problem, the present invention has an object to solve the problem of efficiently selecting a phrase to be stored in the phrase transposed index and efficiently reducing a calculation load in a search using the index.

  In order to solve the above-described problem, the present invention sets a phrase that is frequently used as a target for generating an inverted index, in view of the fact that it takes a long time to check the connection relation of each word constituting the phrase. At this time, the phrase to be indexed is determined based on the idea that the shortest of the transposed lists of the words constituting the phrase is the required processing amount and is proportional to the processing time.

  Specifically, according to the first aspect of the present invention, when searching for an electronic document including a word instructed to be searched from a user terminal, a word transposition index for storing information related to the word and the electronic document, and a plurality of words A phrase search apparatus that uses a phrase transposition index that stores related information between a phrase and an electronic document, the phrase that is included in the search history is extracted, and the electronic document that includes each of the extracted phrases is transposed to the word A calculating means for obtaining a calculation amount when searching using an index; and a determining means for determining a phrase to be stored in the phrase transposed index based on the calculated calculation amount of each phrase and the appearance frequency in the search history. It is characterized by providing.

  The invention according to claim 2 is characterized in that the calculation means refers to the word transposition index with each word constituting a phrase extracted from the search history, and acquires a transposed list of each word as the related information. The length of the shortest transposed list among the obtained transposed lists is obtained as the calculation amount.

  The invention according to claim 3 is characterized in that the determining means calculates a score of each phrase using the calculation amount and the appearance frequency, and determines a phrase to be stored in the phrase transposed index according to the score. It is said.

  According to a fourth aspect of the present invention, there is provided a document search program, wherein a computer is caused to function as each means constituting the document search device according to any one of the first to third aspects.

  According to the first to fourth aspects of the present invention, the phrase to be indexed is appropriately selected in consideration of the usage frequency of the phrase and the calculation load required for checking the connection of the words constituting the phrase. Processing load on processing is reduced, and processing time is shortened.

  FIG. 1 shows a configuration example of a document search apparatus 1 according to an embodiment of the present invention. The document search apparatus 1 is communicably connected to a user terminal 2 that instructs a search condition (keyword) via a network and a content server S that stores a search target electronic document group.

  Here, it is assumed that the document search device 1 is configured as a server (for example, a search engine) that searches for content and the like existing in the content server S on the Internet. The document search apparatus 1 may be, for example, a computer that can be connected to a network and can execute processing logic for document search. The document search apparatus 1 is connected to a network other than the Internet such as an in-house LAN (Local Area Network). May be.

  The user terminal 2 only needs to have a user interface such as a browser that can be connected to a network. For example, a personal computer (PC) or a mobile phone is applicable. With this user terminal 2, the user transmits a keyword and performs a document search. Note that a plurality of user terminals 2 are normally connected to the document search apparatus 1.

  The document search apparatus 1 has an index generation function for generating an inverted index in advance, and a search engine function for searching an electronic document using the generated inverted index.

  The index generation function includes: document collection means 3, document database 4, index generation means 5, word transposition index database 6, phrase / frequency extraction means 7, search history database 8, merge load calculation means 9, storage phrase in FIG. The determination unit 10 and the phrase transposed index database 11 are used. The search engine function is executed by the search execution means 12.

  The functions of the means 3 to 12 are realized by the cooperation of computer hardware and software. The document retrieval apparatus 1 includes a normal component of a computer, for example, a rewritable memory (RAM) that temporarily stores processing data of each of the units 3 to 12, a communication device used for network connection, A processing unit (CPU: Central Processor Unit, etc.) for controlling the means 3 to 12 and a processing unit and a storage unit such as a hard disk drive device are provided, and each database 4.6.8.11 is stored on the hard disk drive device. Has been built. Hereinafter, the index generation processing of each of the means 3 to 11 will be described with reference to the flowchart of FIG.

  S01: First, the document collection means 3 accesses the content server S through the communication device, collects electronic document groups to be searched, and stores them in the document database 4. The function of the document collection unit 3 is realized by a program (crawler or the like) that collects an electronic document group from the content server S.

  S02: The index generating means 5 generates a word transposed index (word transposed index) by referring to the electronic document group stored in the document database 4 and stores it in the word transposed index database 6.

  The word transposition index is an index including a document number (document ID) including the word and an occurrence start position of the word in each document using the word as an index key. This index is called an inverted list, and its length usually differs depending on the number of documents in which each word appears. An example of data in the word transposition index database 6 is shown in Table 1.

  The example of Table 1 shows the number of documents and document IDs including the words “today”, “but”, “docomo”, and “NTT”. The index generating means 5 and the word transposed index database 6 can use existing methods widely used in information retrieval.

  S03: The phrase / frequency extraction means 7 refers to the search history stored in the search history database 8 and extracts a phrase used as a keyword in the past and its frequency.

  In the search history database 8, search requests (keywords) input by the user so far with respect to the document search apparatus 1 are stored with date and time information. A data example of the search history database 8 is shown in Table 2.

  In the example of Table 2, the date and time of the search history of “weather”, “NTT Docomo”, “Next Generation”, and “camera” are shown.

The phrase / frequency extraction means 7 extracts a phrase that appears in the search history within a certain period (for example, one month), and checks the appearance frequency F _i of each extracted phrase. Here, the number of appearances in the search history within a certain period will be described as the appearance frequency F _i .

  Specifically, keywords including quotation marks (“” and “”) are extracted from the search history within a certain period. Then, an associative array (hash table) A of “phrase → appearance count” in which the inside of the quotation marks in the keywords is a phrase and the appearance count is associated with each phrase is generated.

  In addition, although the present Example demonstrated what includes an explicit quotation mark, the existing technique which extracts the part which can be regarded as a phrase implicitly from a keyword besides this can also be used. When a phrase is extracted by such a method, there is a possibility that a word sequence that is not originally a phrase is mistakenly regarded as a phrase. Although it is preferable that there is no such error, the present invention is applicable even if an error is included.

  S04: The merge load calculating means 9 refers to the search history stored in the search history database 8 and appears in the search history within a certain period in the same manner as the phrase / frequency extracting means 7. Extract the phrase. Then, a calculation load expected when a document including each extracted phrase is inspected using the word transposed index database 6 is calculated.

  That is, in order to confirm that the phrase appears in the document, it is necessary to confirm that the words constituting the phrase remain in order and appear in the document adjacent to each other. For this purpose, first, a transposed list of each word is obtained from the word transposed index database 6 using each word constituting the phrase as an index key, and the values of the obtained transposed lists are merged. Next, it is necessary to sequentially check whether or not each word constituting the phrase appears adjacently in the requested order for all the documents included in the merged list. Therefore, the calculation load in this case depends on the length of the list obtained by merging the transposed list of each word.

Here, the calculation load is calculated based on the idea that it is proportional to the length of the shortest transposed list among the transposed lists of each word. That is, the merge load calculation unit 9 calculates the shortest list length among the transposed lists of the words constituting the phrase as the merge load C _i of the phrase. Then, for each phrase extracted from the search history, an associative array (hash table) B of “phrase → merged load” in which the merged load C _i is associated is generated.

  S05: The storage phrase determining means 10 determines that the associative array Referring to B, the phrase for storing the index in the phrase transposed index database 11 is determined. Here, the phrase for which the inverted index is to be generated is selected in consideration of the usage frequency of the phrase and the calculation load required to confirm the word connection relation.

That is, the stored phrase determining means 10 refers to the associative array A and reads the appearance frequency F _i in the search history within a certain period of the phrase P _i . Further, referring to the same time the associative array B, reads the merge load C _i clause P _i. And using these, the phrase storage score S _i of the phrase P _i is calculated by the following equation (1).

Here, α represents a weight, and it is possible to arbitrarily set which of the phrase appearance frequency F _{i and the} merged load C _i is given priority. This expression (1) only needs to be defined in a program or the like.

The stored phrase determining means 10 sets phrase transposition indexes to be generated in order from the phrase having the largest phrase storage score S _i calculated in this way. As a result, phrases that frequently appear in the search history and phrases that have a large merge load when a search is performed using a word inverted index can be preferentially generated as a phrase inverted index generation target. The phrase storage score S _i and the both associative arrays A. B may be stored in the memory (RAM) or the hard disk drive device for processing.

  S06: The phrase data for which the phrase inverted index is to be generated is transmitted from the stored phrase determining means 10 to the index generating means 5, and the index generating means 5 uses the phrase inverted index (phrase inverted index) of each phrase. Is generated. The generated phrase transposed index is stored in the phrase transposed index database 11 within a range not exceeding a preset storage capacity. Note that the index generation processing described in S01 to 06 may be performed at regular intervals, whereby both the index databases 6.11 are updated to the latest state.

  The phrase transposed index database 11 generated in this way is used when the search execution means 12 searches for an electronic document with the keyword received from the user terminal 2. Here, when the keyword is a phrase and the keyword is present in the phrase inverted index database 11, the search execution means 12 searches the phrase inverted index database 11 and returns the result to the user terminal 2.

  At this time, the phrase transposition index stored in the phrase transposition index database 11 is preferentially generated for a phrase that is frequently used and requires processing time for confirmation of word connection relations. Reduces the processing time when searching for documents.

  If the keyword received from the user terminal 2 is a word or a phrase that does not exist in the phrase transposition index database 11, the search execution means 12 further searches the word transposition index database 6, and the result To the user terminal 2.

  The present invention can also be provided as a document search program that causes a computer to function as each means 3 to 12 of the document search apparatus 1. This program may cause the computer to realize all the functions of the respective means 3 to 12, or may realize a part of the functions.

  This program is provided to the computer by downloading from a website or the like. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, Blu-ray Disk (registered trademark). It may be provided to a computer.

1 is a configuration diagram of a document search apparatus according to an embodiment of the present invention. The processing flow figure of the same index generation.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Document search device 2 ... User terminal 3 ... Document collection means 4 ... Document database 5 ... Index generation means 6 ... Word transposition index database 7 ... Phrase / frequency extraction means 8 ... Search history database 9 ... Merge load calculation means 10 ... Store Phrase determination means 11 ... Phrase transposed index database 12 ... Search execution means S ... Content server

Claims (4)

A word transposition index for storing related information between a word and an electronic document when searching for an electronic document including a word instructed to be searched from a user terminal;
A document search device using a phrase composed of a plurality of words and a phrase transposition index for storing information related to an electronic document,
Calculating means for extracting a phrase included in the search history and obtaining a calculation amount when searching the electronic document including each extracted phrase using the word transposed index;
Determining means for determining a phrase to be stored in the phrase transposed index based on the calculated amount of each calculated phrase and the appearance frequency in the search history;
A document search apparatus comprising: The calculating means refers to the transposed index with each word constituting a phrase extracted from the search history,
The document search apparatus according to claim 1, wherein the transposed list of each word is obtained as the related information, and the length of the shortest transposed list among the obtained transposed lists is obtained as the calculation amount. The said determination means calculates the score of each phrase using the said calculation amount and the said appearance frequency, and determines the phrase stored in the said phrase transposition index according to this score. The document search apparatus according to item 1.   A document search program for causing a computer to function as each means constituting the document search device according to claim 1.

Images