JP2010277542A - Document search device and document search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document search device, capable of searching a document satisfying a search request at high speed while reducing the computational load of search processing. <P>SOLUTION: The document search device uses, in search of an electronic document including a phrase instructed to be searched from a user terminal, a word transposition index storage means 108 which stores relevancy information for electronic documents with each word and a phrase transposition index storage means 107 storing relevancy information for electronic document with each phrase composed of a plurality of words. The search device includes a processing time estimation means 103 which estimates processing times required to extract phrases included in search history information and search electronic documents including each of the extracted phrases by use of word transposition indexes in the word transposition index means 108; and a storage phrase determination means 104 which determines a phrase to be stored in the phrase transposition index storage means 107 based on the processing time of each phrase estimated by the estimation means 103 and the appearance frequency thereof in the search history information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an apparatus for accumulating documents and retrieving documents according to requested contents from documents accumulated according to user requested contents at a high speed.

  With the spread of computerization of documents and the explosive spread of the Internet, users in networks and companies can browse a large amount of documents. A search system (full-text search system) that can search a document satisfying a search request expressed by a user at high speed for such a large number of documents is widely used. As a general expression method of a search request, a method of specifying a word string (keyword string) that is included in a search document is used.

  Since the number of documents that contain a specific keyword is a part of the entire document, checking for the existence of a keyword for every document that is stored every time a search request is entered The process is repeated many times and the efficiency is poor. For this reason, a method of speeding up using a word group as an index word and using an index called a word transposition index having a document group including the word and an appearance position of the index word in each document (for example, Non-Patent Document 1) Are widely used.

  When a large number of documents are targeted, when a plurality of keywords are input, it is common to perform a document search (AND search) that includes at least one of each keyword in the document. On the other hand, instead of handling a plurality of keywords individually, there is a demand for performing a document search including a phrase that holds adjacent information and order information in a search request for those keywords.

  However, when searching for a document in which a phrase appears using only a simple transposed index, there is a problem that the load for calculation is large. Because, in order to confirm that the words constituting the phrase appear adjacent in order, the values of the inverted index (transposed list) obtained using each word constituting the phrase as a key are merged, This is because it is necessary to sequentially check the list of results and check whether they appear adjacent in the requested order.

Hiroshi Takeno, Takashi Inoue, “Distributed high-speed information collection / full-text search system InfoBee / Evangelist”, NTT R & D, vol. 52, no. 2, 2003, pp 78-84. Hugh E.E. Williams, Justin Zobel and Dirk Bahr, “Fast phrase querying with combined indexes”, ACM TOIS, vol. 22, no. 4, 2004, pp. 1-17. Takashi Inoue, Yukio Uematsu, Yoshihito Yasuda, Ryoji Kataoka, “Phrase index retention strategy in full-text search system”, IEICE Technical Report, DEIM Forum 2009, 2009, pp. 1-5

  In response to the above problem, by treating the entire phrase as if it were a single word, not just an index with words as words, and maintaining an index corresponding to the phrase (hereinafter referred to as a phrase transposed index) There are known methods for reducing the computational load for searches involving phrases.

  However, if all possible phrases are added to the phrase transposed index, the necessary storage device becomes enormous and is not realistic.

  For this reason, it is necessary to select what phrase to store for a limited amount of phrase transposed index based on some criteria.

  As a phrase selection criterion, a method of storing a high-frequency phrase in a search history using a past search history is conventionally known (for example, see Non-Patent Document 2).

  While the above method is known to have a certain effect, the amount of load varies when phrase search is dealt with only a normal inverted index. Since it is not limited, an optimal phrase cannot be selected from the viewpoint of calculation load. On the other hand, Non-Patent Document 3 and the like have reported a method of selecting a more suitable phrase from the viewpoint of calculation load by using the actual measurement time when the phrase is processed using only the transposed list.

  The above method is effective when the actual measurement time is obtained when the phrase processing is performed using only the transposed list. However, there is a problem that such actual measurement time is not always available. This is because in a system in which a phrase transposed index has already been introduced, since an ordinary inverted index is not used for a phrase included in the index, an actual measurement time cannot be obtained when processing is performed using only an ordinary inverted list. In addition, the actual measurement time is not always available in an environment where the load caused by introducing the time measurement mechanism into the system cannot be ignored.

  An object of the present invention is to provide a document search apparatus and a document search program that can solve the above-described problems of the prior art, reduce the calculation load of search processing, and search documents that satisfy a search request at high speed.

  In order to achieve the above object, the document search apparatus of the present invention includes a word transposed index storage means for storing related information between a word and an electronic document when searching for an electronic document including a word instructed to be searched from a user terminal. , A document search device that uses phrase transposition index storage means for storing related information between a phrase composed of a plurality of words and an electronic document, and extracts phrases included in the search history information, Inference means for estimating a processing time when searching an electronic document including a word transposition index storage means in the word transposition index storage means, processing time of each phrase estimated by the estimation means and search history information And a stored phrase determining means for determining a phrase to be stored in the phrase transposed index storage means based on the appearance frequency.

  According to the present invention, even when the measurement mechanism of the actual measurement time cannot be used, the processing time required for the calculation can be estimated, and the phrase to be stored in the phrase transposed index can be appropriately selected, so that the calculation load of the search process is reduced. , Processing time is shortened.

The block diagram which shows one embodiment of this invention. The flowchart of the index production | generation function in one embodiment of this invention. Explanatory drawing of the search log | history storage database in one embodiment of this invention. Explanatory drawing of the word transposition index in one embodiment of this invention. Explanatory drawing of the measurement result database in one embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments.

  FIG. 1 shows a configuration example of a document search apparatus 100 according to an embodiment of the present invention. In FIG. 1, reference numeral 101 denotes a search history storage database in which a search request history input by the user to the search device 100 is stored.

  Reference numeral 102 denotes a phrase / frequency extracting unit that refers to the search history in the search history storage database 101 and extracts a phrase used as a search phrase in the past and its frequency.

  Reference numeral 103 denotes a processing time estimation unit as an estimation unit of the present invention that estimates a processing time when a phrase from the phrase / frequency extraction unit 102 is input and the phrase is processed using only the word transposed index. .

  104 receives a set of phrases obtained by the phrase / frequency extraction unit 102 and stores it in the phrase transposed index storage unit 107 based on the phrase appearance frequency and the processing time estimated by the processing time estimation unit 103. This is phrase storage determining means for determining a phrase to be performed.

  Reference numeral 105 denotes transposed index generation means for generating a word transposed index and a phrase transposed index for each document stored in the document database 106.

  Reference numeral 107 denotes a phrase transposed index storage unit that stores information related to a phrase composed of a plurality of words and an electronic document.

  Reference numeral 108 denotes a word transposed index storage unit that stores information related to a word and an electronic document.

  A search execution unit 109 performs a search using the document in the document database 106, the phrase transposed index of the phrase transposed index storage unit 107, and the word transposed index of the word transposed index storage unit 108.

  Reference numeral 110 denotes an actual measurement result database in which the correspondence between the statistical values of the inverted index constituting the phrase and the actual measurement time in the search for the past phrase is stored.

  111 is a coefficient learning means for learning a coefficient to be used in the processing of the processing time estimation means 103 using the information in the actual measurement result database 110.

  The functions of the phrase / frequency extraction means 102, processing time estimation means 103, stored phrase determination means 104, transposed index generation means 105, search execution means 109, and coefficient learning means 111 are achieved by, for example, a computer.

  The document search apparatus 100 having the above configuration has an index generation function for generating a word transposed index and a phrase transposed index in advance, and a search function for searching for a document using the generated index.

  The index generation function is executed by the phrase / frequency extraction means 102, processing time estimation means 103, coefficient learning means 111, stored phrase determination means 104, and transposed index generation means 105 in FIG. The search function is executed by search execution means 109.

  A flowchart of the index generation function is shown in FIG. Hereinafter, description will be made based on the flowchart of FIG.

  Step S01 The index generation function uses the phrase / frequency extraction means 102 to refer to the search history stored in the search history storage database 101 and extract a phrase used as a search phrase in the past and its frequency.

  The search history storage database 101 stores, for example, as shown in FIG. 3, search requests that have been input by the user so far with time information in the search system.

  The phrase / frequency extraction unit 102 extracts phrases that have appeared in the search history within a certain past period (for example, one month), and examines the number of appearances of each phrase. Specifically, for the phrase extraction, the inside of the search history quotes including quotes (“˜” or “˜”) is recognized as a phrase. In association with the number of occurrences of each phrase, an associative array having the number of times as a value using the phrase as a key is created.

  In the present embodiment, an example including an explicit quotation mark has been described. However, in addition to this, an existing technique for extracting a portion that can be implicitly regarded as a phrase from a search term can be used.

  When these implicit phrases are extracted, a word sequence that is not originally a phrase may be mistakenly regarded as a phrase. Although it is preferable that there is no such error, the present invention itself can be applied even if an error is included. Therefore, an extraction method that includes an error may be applied.

  Step S02 The index generation function creates a transposed index of the transposed index storage unit 108 using the transposed index generation unit 105.

  For example, as shown in FIG. 4, the word transposition index is an index including a word as an index key, a value including a document number including the word, and a word appearance start position in each document. An existing method widely used in information retrieval can be used.

  Step S03 The index generation function uses the coefficient learning unit 111 to determine a parameter to be used by the processing time estimation unit 103.

  The coefficient learning unit 111 learns a coefficient to be used in a processing time estimation function to be described later, using the correspondence between the statistical value of the inverted index constituting the phrase and the actual processing time in the past phrase search. . For example, information stored in the actual measurement result database 110 as shown in FIG. 5 is used as the correspondence between the statistical value of the inverted index constituting the phrase and the actual measurement processing time.

  The coefficient learning is performed by multiple regression analysis in which processing time is an objective variable and each statistical value is an explanatory variable. Specifically, ridge regression or the like can be used.

The following values are used as the statistical values.
1. 1. Number of words that make up a phrase 2. Minimum length of transpose list 3. Maximum length of transposed list 4. Minimum number of documents in transpose list 5. Maximum number of documents in transposition list 6. Minimum value of the average number of appearances obtained from the appearance position in the transposed list The coefficient corresponding to the maximum value of these seven statistical values of the average value of the number of occurrences obtained from occurrence position in the inverted list _{α i (i = 1, ..} , 7) that.

  Step S04 The index generation function uses the processing time estimation means 103 to estimate the processing time when each phrase is searched using only the word transposed index.

  The processing time estimation means 103 estimates a processing time when a phrase is input and the phrase is processed using only the word transposition index.

That is, the processing time estimation unit 103 first refers to the word transposition index storage unit 108 with each word constituting the phrase extracted from the search history storage database 101 by the phrase / frequency extraction unit 102, and stores each transposed list of each word. Get the statistics of the transpose list. As the statistical value, the following values are used as in the coefficient learning unit 111.
1. 1. Number of words that make up a phrase 2. Minimum length of transpose list 3. Maximum length of transposed list 4. Minimum number of documents in transpose list 5. Maximum number of documents in transposition list 6. Minimum value of the average number of appearances obtained from the appearance position in the transposed list The maximum value of the average number of appearances obtained from the appearance position in the transposed list. These statistical values s ₁ ,. . . , S ₇ and the coefficients α _i,. . . , Α ₇ , the processing time required when the input phrase is processed using only the word transposed index is obtained and output as follows.

α ₁ s ₁ + α ₂ s ₂ +. . . + Α ₇ s ₇ (1)
Step S05 The index generation function determines a phrase to be stored in the phrase transposed index storage means 107 using the stored phrase determination means 104.

  The stored phrase determination means 104 receives the phrase set obtained by the phrase / frequency extraction means 102 as input, and determines the phrase set to be stored in the phrase transposed index storage means 107.

For clause P _i in the set of clauses, phrases stored in the phrase P _i by using the frequency F _i in the search history in a certain period of P _i, the estimated processing time T _i of P _i calculated by the processing time estimation function The score S _i is calculated by the following formula: S _i = F _i × T _i (2)
Here, in order from the largest value of S _i , the phrases are stored within a range not exceeding the size of a predetermined phrase index.

  Step S06 The index generation function creates a phrase transposed index of the phrase transposed index storage means 107 using the transposed index generation means 105.

  The phrase transposition index of the phrase transposition index storage means 107 is an index having a phrase as an index key, a document number including the phrase, and a word appearance start position in each document as values. It has the same structure as the word transpose index.

  Next, the search function uses the search execution means 109 to search for a document.

  Needless to say, the present invention can be realized by configuring a part or all of the functions of each means in the document search apparatus of the present embodiment with a computer program and executing the program using the computer. The program for realizing the function is recorded on a computer-readable recording medium such as FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, It can be recorded on a CD (Compact Disk) -ROM, a DVD (Digital Versatile Disk) -ROM, a CD-R, a CD-RW, an HDD, a removable disk, etc., and stored or distributed. It is also possible to provide the above program through a network such as the Internet or electronic mail.

DESCRIPTION OF SYMBOLS 100 ... Document search apparatus 101 ... Search history storage database 102 ... Phrase / frequency extraction means 103 ... Processing time estimation means 104 ... Stored phrase determination means 105 ... Inverted index generation means 106 ... Document database 107 ... Phrase inverted index storage means 108 ... Word Transposed index storage means 109 ... Search execution means 110 ... Actual measurement result database 111 ... Coefficient learning means

Claims (4)

When searching for an electronic document including a word instructed from the user terminal, word transposed index storage means for storing related information between the word and the electronic document, and related information between the phrase consisting of a plurality of words and the electronic document A document search device that uses phrase transposition index storage means for storing,
A guess means for extracting a phrase included in the search history information, and estimating a processing time when searching for an electronic document including each of the extracted phrases using a word transposed index in the word transposed index storage means;
And a storage phrase determination unit that determines a phrase to be stored in the phrase transposed index storage unit based on the processing time of each phrase estimated by the estimation unit and the appearance frequency in the search history information. Document retrieval device. The inference means refers to a word transposition index in the word transposed index storage means with each word constituting a phrase extracted from the search history information, obtains a transposed list of each word as the related information, and obtains the The document search apparatus according to claim 1, wherein the processing time is obtained by regression analysis using a statistic of each transposed list. The said stored phrase determination means calculates the score of each phrase using the said estimated processing time and the said appearance frequency, and determines the said phrase to store according to this score. Document retrieval device. A document search program for causing a computer to function as each means according to any one of claims 1 to 3.

Images