JP2012014646A - Document retrieval device, document retrieval method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document retrieval device capable of performing retrieval at high speed without causing omission of the retrieval.SOLUTION: The document retrieval device includes: an index creation part 6 which uses a character string of N characters obtained by dividing a document, or a character string of N+1 or more characters with high use frequency in the retrieval as a key, and creates a mixed N-gram index for holding an identifier of a document including the key, and an appearance position of the key in the document; and a retrieval part 3 which outputs a list of documents including the input retrieval key by referring to the mixed N-gram index. The retrieval part 3 retrieves, when a key which matches the retrieval key does not exist in the mixed N-gram index, the mixed N-gram index by each character string obtained by dividing the retrieval key into character strings of the N characters, when the key which matches each character string exists, determines whether or not the document includes the retrieval key based on the appearance position of the key in the document including the key, and outputs a list of documents determined to include the retrieval key.

Description

  The present invention relates to a document search apparatus, a document search method, and a program.

  A full-text search technique for searching for a document including an input keyword from a large number of document files stored in a storage device is known. In a document search apparatus that performs full-text search, a method of creating an index of a document file to be searched in advance and collating the keyword of this index with an input search keyword at the time of search is common.

  Conventional document search apparatuses are roughly divided into two types according to the index creation method. One is a morphological analysis method that uses a dictionary to divide a document into words and uses the words as index keys, and the other is an N-gram that divides a document into N characters and uses them as index keys. It is a method.

  Although the morphological analysis method has an advantage that the search speed is fast, the index creation speed is slow. In addition, if an incorrect word is cut out when creating an index, there is a problem that a search omission occurs. In order to avoid cutting out wrong words, it is necessary to maintain a dictionary used for index creation.

  The N-gram method is widely used because it has a high index creation speed and few search omissions. However, the search key input at the time of searching is divided into N-character keys, and index search with each key is performed. Since the results need to be integrated, the search speed is slow. Note that in the N-gram system, the performance deteriorates when N is too small or too large, and therefore the bi-gram system that divides a document into two characters is widely used.

  Japanese Patent Application Laid-Open No. 2004-133830 discloses an information search method for searching text data in a full text, a search target text is divided into words, a word having word information indicating a word break and having N characters from the divided text. A method is described in which a character string index with information is created and a search word is searched by a character string search or a word search or both from the word information-added character string index.

JP 2001-034623 A

  In the method described in Patent Document 1, an attempt is made to reduce search noise by examining word breaks, but there is a possibility that search omissions may increase as compared with the normal N-gram method.

  Accordingly, an object of the present invention is to obtain a document search apparatus that does not cause a search omission and can perform a search at a high speed.

  The document search apparatus according to the present invention uses, as a key, a character string of N (N is a natural number) characters generated by dividing a document, or a character string of N + 1 characters or more whose frequency used as a keyword during a search exceeds a certain condition. An index creation unit that creates a mixed N-gram index that holds an identifier of a document including the key, and an appearance position of the key in the document, and a search input by referring to the mixed N-gram index A search unit that outputs a list of documents including a key, and the search unit first searches the mixed N-gram index with the search key itself, and there is a key that matches the search key. Outputs a list of documents including the key, and if there is no key that matches the search key, the search key is set to N characters. The mixed N-gram index is searched for each divided character string, and when there is a key that matches each character string, based on the appearance position of the key in the document including the key, It is determined whether or not the document includes the search key, and a list of documents determined to include the search key is output.

  According to the present invention, it is possible to obtain a document search apparatus that does not cause a search omission and can perform a search at a high speed.

The block diagram which shows the structure of the document search apparatus by embodiment of this invention. The figure explaining operation | movement of the document search apparatus by embodiment of this invention. The flowchart of operation | movement of the document search apparatus by embodiment of this invention. The flowchart of operation | movement of the document search apparatus by embodiment of this invention. The flowchart of operation | movement of the document search apparatus by embodiment of this invention.

Next, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
Embodiment FIG. 1 is a block diagram showing a configuration of a document search apparatus 10 according to an embodiment of the present invention.
The document search apparatus 10 includes an input unit 1, an output unit 2, a search unit 3, a search key index storage unit 4, a mixed bi-gram index (mixed N-gram index) storage unit 5, and an index creation unit 6. . Note that the document search apparatus 10 uses a bi-gram method that divides a document into two characters at a time among the N-gram methods.

  The input unit 1, the output unit 2, the search unit 3, and the index creation unit 6 represent modules of operations performed by a computer processor according to a program. The search key index storage unit 4 and the mixed bi-gram index storage unit 5 are realized by a memory or a hard disk.

  The input unit 1 is realized by a keyboard, a mouse, or the like that allows a user to input a search key when executing a search.

  The output unit 2 is realized by a display device such as a display that displays a list of hit document files and the like as search results.

  The search unit 3 refers to the mixed bi-gram index storage unit 5 and outputs a list of document files including the search key input via the input unit 1 as a result. The search unit 3 first refers to the mixed bi-gram index storage unit 5 with the search key itself, and if there is a document file having a keyword that matches the search key, notifies the output unit 2 of the hit document file in the form of a list. To do.

  If there is no document file having a keyword that matches the search key and the length of the search key is longer than two characters, the search key is divided into two characters and the mixed bi-gram index storage unit 5 is referred to. The document file including the search key is acquired based on the file ID of the hit record and the appearance position of the character, and the acquired document file is notified to the output unit 2 in the form of a list. Further, when the search key is longer than two characters, the search key is registered in the search key index storage unit 4.

  The search key index storage unit 4 stores an index that holds the number of searches and the last search date and time using the search key used for the search as a key.

  The mixed bi-gram index storage unit 5 stores an index in which two characters are extracted from each character position and a search key used for the search as a key.

  The index creation unit 6 creates a mixed bi-gram index by scanning the document file and stores it in the mixed bi-gram index storage unit 5. That is, the index creation unit 6 refers to the search key index storage unit 4 in addition to the normal two-character key of the bi-gram index, and adds a key longer than two characters.

Next, the operation of the document search apparatus 10 will be described.
In a general search device, when a folder and time are specified, document files in the folder under the specified folder are periodically referenced. If there is an updated file, the updated content is used as an index. It is operated in a mechanism that reflects it. When a document file on the Web is to be searched, the index is updated by following the URL in the html file instead of following the folder. Here, for simplification, when a file is specified, the index is updated for the file.

First, the operation of the index creation unit 6 will be described with reference to FIGS.
The index creation unit 6 reads the document files file1. txt or file2. A mixed bi-gram index is created from txt and the like. As shown in FIG. The content of txt is “Today is a good weather. The content of txt is “Today is a good day ...”.

  First, the index creation unit 6 sets file1. txt and its ID (1) are received (FIG. 3: step S1).

  The correspondence between the file name and the ID is held in the “file ID correspondence table” shown in FIG. The file ID correspondence table can be held in the mixed bi-gram index storage unit 5, for example. Note that the file name may be directly stored in the mixed bi-gram index storage unit 5 instead of the file ID. However, if the character string is left as it is, the storage capacity becomes large and the processing efficiency is poor, so here the file name is converted to ID.

  Next, assuming that the same file has been updated, file1.file is already stored in the field holding “file ID = appearance position” in the mixed bi-gram index. If there is an ID of txt, it is deleted (step S2). That is, a pair of 1 = ˜ (˜ is arbitrary) is deleted. If the file ID = appearance position field becomes empty, the key itself is also deleted.

  Next, the document in the file is read into the buffer, and the index registration process is performed by shifting the characters one by one from the beginning (steps S3 to S7).

  First, file1. Register a two-character key from the beginning of txt. That is, a record of key: “today”, file ID = appearance position: 1 = 0 is registered in the mixed bi-gram index storage unit 5 (step S4).

  Next, a key larger than two characters is searched with reference to the search key index storage unit 4. That is, “today is good”, “today is good”, “today is good”, and the like are searched from the search key index storage unit 4. The maximum number of characters to be searched is determined in advance, for example, 8 characters.

  If a key larger than 2 characters exists in the search key index, the search frequency is more than a certain number (for example, 5), and the search date is within a certain period (for example, 10 days) from the present, the key is Register in the mixed bi-gram index storage unit 5.

  For example, if the key is “good weather” with the pointer advanced three from the beginning of the document, “good weather” in the search key index shown in FIG. 2 is hit, so the key: “good weather” The data of file ID = appearance position: 1 = 3 is registered in the mixed bi-gram index storage unit 5 (step S5).

  When the pointer is advanced by one character (step S6) and the end of the file is reached, registration is completed (step S7: NO).

Next, the operation of the search unit 3 will be described with reference to FIG.
First, the search part 3 acquires the search key which the user input via the input part 1 (step S11).

  Next, the search unit 3 searches the mixed bi-gram index storage unit 5 with the input search key (step S12).

  If there is a hit key in the input search key (step S13: YES), the file ID of the key = the file ID registered at the appearance position is converted into a file name and output to the output unit 2. (Step S15).

  When there is no hit key among the input search keys (step S13: NO), the search key is divided into two characters, the mixed bi-gram index storage unit 5 is searched for each key, and the hit A list of file IDs registered in the key to be acquired is acquired (step S14). Further, the file ID is converted into a file name and output to the output unit 2 (step S15).

  Further, when the search key is 3 characters or more (step S16: YES), the search key is registered in the search key index storage unit 4 (step S17).

Next, the update process of the search key index storage unit 4 will be described with reference to FIG.
First, the search part 3 acquires the search key which the user input via the input part 1 (step S21).

  Next, the search key index storage unit 4 is searched with the acquired search key (step S22).

  If there is a key hit in the search key index with the acquired search key (step S23: YES), the "search count" of the record of the key is incremented by 1, and the current date is registered in "search date" (Step S24).

  If there is no hit key in the acquired search key (step S23: NO), a record of the key is added to the search key index storage unit 4, and the current date and time is registered in “search date and time” (step S23). S25).

  As described above, according to the present embodiment, as a mixed bi-gram index, in addition to a normal bi-gram index key, in the case of a frequently used search key, the search key itself is used as an index key. By doing so, in the environment where the same search key is searched many times, the search speed can be improved without causing a search omission. This is particularly effective in an environment where the same character string is often used as a search key, such as in-house document search.

  The effect of the present invention will be described using a specific example. For example, a case where “good weather” is input as a search key will be described as an example. The contents of the mixed bi-gram index are shown in FIG.

  First, when the four-character search key “good weather” is not registered in the mixed bi-gram index storage unit 5, the search key is divided into two characters, and the mixed bi-gram index is stored for each key. Part 5 is searched (normal N-gram search).

  As a result, two records of key: “good”, file ID = appearance position: 1 = 3, 2 = 3, and key: “weather”, file ID = appearance position: 1 = 5 are hit. Among these, a search is made for a file ID that exists in common and whose appearance positions are connected in order. In this case, it is necessary to select the “weather” appearing position two characters after the “good” appearing position. Since the file ID of the key “good” = appearance position “1 = 3” and the file ID of the key “weather” = appearance position “1 = 5” satisfy this condition, the file ID = 1, that is, (file1.txt) It can be determined that “good weather” is included.

  As the size of the index grows, thousands of records can be hit for each key, slowing down the search. On the other hand, according to the present embodiment, when the search key “good weather” is frequently used, since it is registered in the mixed bi-gram index, the normal N-gram search process as described above is used. Can be omitted, and the search speed can be increased.

A part or all of the above embodiment can be described as in the following supplementary notes, but is not limited thereto.
(Supplementary note 1) A character string of N (N is a natural number) characters generated by dividing a document, or a character string of N + 1 characters or more whose frequency used as a keyword during search exceeds a certain condition is included as a key. An index creation unit for creating a mixed N-gram index that holds an identifier of a document and an appearance position of a key in the document;
A search unit that outputs a list of documents including the input search key by referring to the mixed N-gram index;
The search unit first searches the mixed N-gram index with the search key itself, and when there is a key that matches the search key, outputs a list of documents including the key, and the search key If there is no key that matches the character string, the search key is divided into N character strings, the mixed N-gram index is searched for each divided character string, and the key that matches each character string is found. If it exists, it is determined whether or not the document includes the search key based on the appearance position of the key in the document including the key, and a list of documents determined to include the search key is output. Document retrieval device.

(Appendix 2) The search unit
Create a search key index that holds the number of times the keyword is used, using the character string used as a keyword during the search as a key.
The index creation unit
The document search apparatus according to appendix 1, wherein the search key index is referred to, and a key of N + 1 characters or more in which the number of uses exceeds a predetermined condition is registered as a key of the mixed N-gram index.

(Supplementary note 3) A character string of N (N is a natural number) characters generated by dividing a document or a character string of N + 1 characters or more whose frequency used as a keyword when searching exceeds a certain condition is included as a key. Creating a mixed N-gram index that holds the identifier of the document and the appearance position of the key in the document;
Outputting a list of documents including the input search key by referring to the mixed N-gram index,
In the step of outputting the list of documents, first, the mixed N-gram index is searched with the search key itself, and if there is a key that matches the search key, a list of documents including the key is output. If there is no key that matches the search key, the search key is divided into character strings of N characters, the mixed N-gram index is searched for each divided character string, and each character string is searched. If there is a key that coincides with the document, it is determined whether or not the document includes the search key based on the appearance position of the key in the document including the key, and the document determined to include the search key Document search method that outputs a list of files.

(Appendix 4)
A character string of N (N is a natural number) characters generated by dividing a document, or a character string of N + 1 characters or more whose frequency used as a keyword during a search exceeds a certain condition, and an identifier of a document including the key, An index creation unit that creates a mixed N-gram index that holds the appearance position of the key in the document;
A program that functions as a search unit that outputs a list of documents including an input search key by referring to the mixed N-gram index,
The search unit first searches the mixed N-gram index with the search key itself, and when there is a key that matches the search key, outputs a list of documents including the key, and the search key If there is no key that matches the character string, the search key is divided into N character strings, the mixed N-gram index is searched for each divided character string, and the key that matches each character string is found. If it exists, it is determined whether or not the document includes the search key based on the appearance position of the key in the document including the key, and a list of documents determined to include the search key is output. ,program.

  1 input unit, 2 output unit, 3 search unit, 4 search key index storage unit, 5 mixed bi-gram index storage unit, 6 index creation unit, 10 document search device

Claims (4)

A character string of N (N is a natural number) characters generated by dividing a document, or a character string of N + 1 characters or more whose frequency used as a keyword during a search exceeds a certain condition, and an identifier of a document including the key, An index creation unit that creates a mixed N-gram index that holds the appearance position of the key in the document;
A search unit that outputs a list of documents including the input search key by referring to the mixed N-gram index;
The search unit first searches the mixed N-gram index with the search key itself, and when there is a key that matches the search key, outputs a list of documents including the key, and the search key If there is no key that matches the character string, the search key is divided into N character strings, the mixed N-gram index is searched for each divided character string, and the key that matches each character string is found. If it exists, it is determined whether or not the document includes the search key based on the appearance position of the key in the document including the key, and a list of documents determined to include the search key is output. Document retrieval device. The search unit
Create a search key index that holds the number of times the keyword is used, using the character string used as the keyword during the search,
The index creation unit
The document search apparatus according to claim 1, wherein the search key index is referenced and a key of N + 1 characters or more in which the number of uses exceeds a certain condition is registered as a key of a mixed N-gram index. A character string of N (N is a natural number) characters generated by dividing a document, or a character string of N + 1 characters or more whose frequency used as a keyword during a search exceeds a certain condition, and an identifier of a document including the key, And creating a mixed N-gram index that holds the location of the key in the document;
Outputting a list of documents including the input search key by referring to the mixed N-gram index,
In the step of outputting the list of documents, first, the mixed N-gram index is searched with the search key itself, and if there is a key that matches the search key, a list of documents including the key is output. If there is no key that matches the search key, the search key is divided into character strings of N characters, the mixed N-gram index is searched for each divided character string, and each character string is searched. If there is a key that coincides with the document, it is determined whether or not the document includes the search key based on the appearance position of the key in the document including the key, and the document determined to include the search key Document search method that outputs a list of files. Computer
A character string of N (N is a natural number) characters generated by dividing a document, or a character string of N + 1 characters or more whose frequency used as a keyword during a search exceeds a certain condition, and an identifier of a document including the key, An index creation unit that creates a mixed N-gram index that holds the appearance position of the key in the document;
A program that functions as a search unit that outputs a list of documents including an input search key by referring to the mixed N-gram index,
The search unit first searches the mixed N-gram index with the search key itself, and when there is a key that matches the search key, outputs a list of documents including the key, and the search key If there is no key that matches the character string, the search key is divided into N character strings, the mixed N-gram index is searched for each divided character string, and the key that matches each character string is found. If it exists, it is determined whether or not the document includes the search key based on the appearance position of the key in the document including the key, and a list of documents determined to include the search key is output. ,program.