JP2014186482A - Full-text search system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a full-text search system capable of reducing retrieval time by reducing N-gram index to be read.SOLUTION: The full-text search system segments a document into registration substrings and registers a character position list which stores a piece of character position information representing a position of a character in each document, and a document ID list which stores a start position of the character position information in the character position list and the document ID while associating with each other. The full-text search system segments a retrieval term into retrieval substrings, compares document ID lists corresponding to the retrieval substring with each other, and extracts all document IDs commonly included. By acquiring the start position of the character position information for each retrieval substring from the document ID list, the full-text search system acquires the relevant character position information only from the character position list. With respect to the respective document IDs, when the respective retrieval substrings adjacently appear in the same order as that of the retrieval term before segmentation, the full-text search system determines that the retrieval term is included in the document of the document ID.

Description

  The present invention relates to a full-text search apparatus for searching a document including a designated character string from a previously registered document group.

  As a method in which the full-text search device searches a document containing a search character string (hereinafter referred to as a search term) designated from a large-scale document database at high speed, a registered document is scanned and each document is previously stored. There is a method to create an index of the character string that appears.

  As an indexing method, the document ID and the appearance position where one character or two concatenated characters (hereinafter referred to as “N-gram term”) appear according to the character type are described in Non-Patent Document 1. A method for determining whether or not the positional relationship in the search term and the positional relationship in the N-gram index are equal (hereinafter referred to as “neighboring determination”) is disclosed as “N-gram index”) .

  Since the N-gram index has a data size several times that of the document group, the N-gram index is usually arranged on the hard disk. When performing a search, the search term is divided into N-gram terms, and an N-gram index for each N-gram term is read from the hard disk to determine adjacency. In addition, in order to enable a complicated search, information may be increased in the N-gram index. Patent Document 1 discloses a method of adding information on a structure in which characters appear in an N-gram index. Japanese Patent Laid-Open No. 2004-228620 implements a search that is aware of structural information for an XML document.

JP 2010-108191 A JP-A-8-194718

"A method of high-speed full-text search for Japanese documents" (The IEICE Transactions D-1 Vol. J75-D-1 No. 9 pages 836-846, September 1992)

  Characters that appear frequently in a document group (hereinafter referred to as “frequent characters”) increase the data size of the N-gram index of the character, so if frequent characters are included in the search term, a large frequent Since it is necessary to read the character N-gram index from the hard disk, it takes time to perform the search. Further, as in the known example 1, when the information is increased in the N-gram index for performing a complicated search, the amount of data read from the hard disk increases accordingly, and it takes time to perform the search in the same manner.

  To solve the problem that this search takes time, in Patent Document 2, an N-gram index having a small data size with one character added to a frequent character is created, and when performing a search, a small N-gram index is stored on the hard disk. A method of shortening the time for searching by reading from the document is disclosed.

  In Patent Document 2, the search time is shortened by reducing the size of the N-gram index read at the time of search using one character following the frequent character specified in the search term. It is effective only for one character, and is not effective when a frequent character in the search term appears last or only a frequent word.

  In addition, since it is necessary to create N-gram indexes for only the types of characters that appear next to frequently appearing characters in the document group, the size of the entire N-gram index is increased, and the creation takes time.

  An object of the present invention is to divide an N-gram index into two parts, a document ID and character position information, and use the characters other than the frequent characters specified in the search term, and use the N-gram index of the frequent characters read from the hard disk. It is an object of the present invention to provide a full-text search apparatus that shortens the time for performing a search by reducing the number of times.

In order to achieve the above object, the present invention provides the following configurations.
An aspect of the full-text search device according to the present invention is to divide each of a plurality of documents into registration partial character strings having a predetermined number of characters, and for each of the registration partial character strings, characters in each document of the registration partial character string. Means for registering a character position list that stores character position information indicating the position of the document, and a document ID list that stores the start position of the character position information in the character position list and the document ID of each document in association with each other;
The obtained search terms are divided into the same number of search partial character strings as the number of characters of the registration partial character string, and stored for each of the same registration partial character strings as each of the divided search partial character strings. Means for extracting all document IDs included in common in all the compared document ID lists by comparing the document ID lists with each other;
Means for respectively obtaining a start position of the character position information associated with each of the extracted common document IDs from the document ID list for each partial character string for search;
Means for acquiring only corresponding character position information from the character position list for each partial character string for search based on the start position of the acquired character position information;
For each of the extracted common document IDs, the search partial character strings appear adjacently in the same order as the search terms before the division based on the character position information acquired for each of the search partial character strings. And means for determining that the search term is included in the document with the document ID when each partial character string for search appears adjacently.

  The full-text search device of the present invention has the following effects. By minimizing the N-gram index read from the hard disk, the search process can be speeded up.

It is a figure which shows the system configuration | structure in an example to which this invention is applied. It is the figure which showed the data structure of the N-gram index in an example to which this invention is applied. It is the figure which showed the registration process of the document in an example to which this invention is applied. It is the figure which showed the search process of the document in an example to which this invention is applied. It is the figure which showed the specific data flow of the search process of the document in an example to which this invention is applied.

Embodiments of the present invention will be described below with reference to the drawings showing an example of the present invention.
FIG. 1 is a configuration diagram of an entire system including a full-text search apparatus according to the present invention.
110 is a search client, 120 is a communication line such as a LAN, 130 is a search server, and the search client 110 and the search server 130 are connected by a communication line 120.

The search server 130 is a full-text search device 131 that performs registration processing and document search processing of a plurality of documents to be searched, a search memory 132 such as a RAM used for searching, and an N created from a plurality of registered documents. The hard disk 133, which is a typical example of an auxiliary storage device that registers the gram index 136, has an N-gram index that is registered in the hard disk 133, and includes a document ID list 134 and a character position list 135. . The search server 130 is an appropriate computer and has a predetermined program installed therein, and functions as the full-text search device 131 when the CPU reads and executes the predetermined program in the memory.
In the following description of the drawings, reference numerals in FIG. 1 may be referred to.

  FIG. 2 is a diagram showing an example data structure of an N-gram index registered in the hard disk 133 of FIG. As shown in the figure, the N-gram index 136 includes N-gram terms 210 and 211, which are partial character strings for registration obtained by dividing each of a plurality of documents to be registered by a predetermined number of characters, N-gram terms 210, It is composed of document ID lists 134A and 134B and character position lists 135A and 135B created for each 211. FIG. 2 shows an example of an N-gram term obtained by dividing a document into characters. Only two N-gram terms “A” 210 and “M” 211 are shown as representatives of the divided N-gram terms. Yes.

  In the document ID lists 134A and 134B for each of the N-gram terms 210 and 211, the document ID of each of a plurality of documents and an offset indicating the start position of the character position information on the character position lists 135A and 135B are registered. . On the document ID lists 134A and 134B, the document IDs are registered in ascending order.

  In the character position lists 135A and 135B for each of the N-gram terms 210 and 211, the number of appearances indicating the number of occurrences of the N-gram terms 210 and 211 in each of a plurality of documents, and the number of occurrences in the document Character position information composed of character positions (the same number as the number of appearances) indicating the positions of the N-gram terms 210 and 211 is registered. The start position of the character position information in the character position lists 135A and 135B is registered as an offset in the document ID lists 134A and 134B. On the character position lists 135A and 135B, registration is performed so that the document IDs are in ascending order and the character positions are in ascending order.

  Here, for example, when the document “Amegawa” with the document ID “001” is registered, the document ID list “001” corresponding to the first character N-gram term “A” 210 is included in the document ID “001”. "And" 1 ", which is an offset on the character position list 135A, are registered, and the first offset of the character position list 135A is the number of occurrences of" a "in the document with the document ID" 001 "" 3 " 210a and character positions “1”, “3”, and “12” 210b are registered. In the document ID list 134A, the document ID “900” and the offset “4000” on the character position list 135A are registered, and the document ID “900” is added to the 4000th offset in the character position list 135A. The number of occurrences of “a” in the document “1” 210c and the character position “2” 210d are registered.

  Further, for example, when the document “Amega Fu” with the document ID “001” is registered, the document ID list “134” corresponding to the N-gram term “M” 211 of the second character has the document ID “001”. "1", which is an offset on the character position list 135B, is registered, and the first offset of the character position list 135B is the number of occurrences of "me" in the document with the document ID "001" "1" 211a And character position “2” 211b are registered. The document ID list 134B is registered with the document ID “900” and the offset “10000” on the character position list 135b, and the document ID “900” is included in the offset 10000th in the character position list 135B. The number of occurrences of “me” in the document “1” 211c and the character position “3” 211d are registered.

FIG. 3 is a flowchart showing registration processing in the full-text search apparatus 131 shown in FIG.
The document registration process is started by transmitting a registration process request and a document group to be registered from the search client 110 to the search server 130. (Step 310) The search server 130 assigns a document ID to each document to be registered.

  The search server 130 divides all the documents to be registered sent from the search client 110 into N-gram terms that are one-character or two-character registration partial character strings. When dividing, the character of the N-gram term and the character position on the document of the character of the N-gram term are stored as a pair. (Step 320)

Next, the N-gram term and character position group created in step 310 are classified for each N-gram term of the same character. (Step 330)
For each classified N-gram term, the character position stored at the end of the character position list 135 on the hard disk 133 is added. At this time, character positions are added in ascending order. When adding a character position, the last position before adding the character position is stored as an offset for each N-gram term (step 340).

  Next, in the document ID list 134, the document ID and the document ID are stored in step 340 at the end of the document ID list 134 of the same N-gram term as the N-gram term to which the character position list 135 is added in step 340. Add the offset. At this time, the document ID used in the search server 130 is used as the document ID so that the document IDs are always arranged in ascending order in the document ID list 134. (Step 350)

  When the document ID is added to the N-gram index for all N-gram terms appearing in a plurality of registration target documents, the registration process is completed. (Step 360)

  FIG. 4 is a flowchart showing a search process (hereinafter referred to as “character string search”) when a search character string of three or more characters is designated as a search term in the full-text search device. The character string search process will be described with reference to FIG.

  The character string search starts when a search term and a search execution request are transmitted from the search client 110 to the search server 130. (Step 410)

  The search server 130 divides the search term received from the search client 110 into N-gram terms that are one-character or two-character search partial character strings. (Step 420) The number of characters in the partial character string for search divided here is the same as the number of characters in the partial character string for registration divided in Step 320 at the time of registration shown in FIG.

  Next, one N-gram term is selected from a plurality of N-gram terms, and the document ID list of the N-gram term (hereinafter referred to as “reference document ID list”) is stored in the search memory 132 from the hard disk 133. Is read. (Step 430)

  Subsequently, another N-gram term is selected from the plurality of N-gram terms, and a document ID list (hereinafter referred to as “comparison document ID list”) is read from the hard disk 133 to the search memory 132. (Step 440)

  Since document IDs that appear in the reference document ID list and do not appear in the comparison document ID list do not include the search term specified in step 410 without comparing character positions, the reference document ID list and the comparison document are determined. The document IDs are narrowed down so that only document IDs appearing in both ID lists remain. The narrowed down result is arranged on the search memory as a new reference document ID list. (Step 450)

  If N-gram terms that have not been compared still remain, the process returns to step 440 to read the next comparison document ID list, and further narrow down the document IDs by comparing with the new reference document ID list. If there are no remaining N-gram terms to compare, go to step 470. (Step 460) In the reference document ID list finally obtained in this way, only the document IDs common to each of the plurality of document ID lists corresponding to the plurality of N-gram terms for search are all included. include.

  For the document IDs that appear in the reference document ID list on the memory, the character position information of each N-gram term is read from the character position list 135 of the hard disk 133 into the search memory 132. (Step 470) At this time, since the reference document ID list has the offset of the start position of the character position information added in Step 340, only the character position information of the document that may hit based on the offset is stored in the hard disk. You can read from 133. Compared with the document ID list 134, the character position list 135 has a data size ratio of “1: N-gram term average number of appearances in the document”, so that character position information having a large data size is read from the hard disk 133. The search processing can be speeded up by reducing the value of the conventional method.

  With reference to the character position information read in step 470, it is checked for each document ID whether each N-gram term appears adjacently in the same order as the search term before division (hereinafter referred to as “adjacent matching”). If it appears adjacently in the same order, the document ID is stored as a search hit. (Step 480) That is, it is determined that the search term is included in the document with the document ID.

  When the adjacent collation in step 480 is completed for all document IDs in the reference document ID list, a list of hit documents is returned to the search client 110 as a search result, and the search process is completed. (Step 490)

  4 will be described using the data structure of the N-gram index in FIG. 2 and a specific example of the search process in FIG.

  In this example, when the N-gram index in which a plurality of documents are registered is in the state of FIG. 2 described above, the search is performed with the search term “Ame” 510.

  First, “ame” 510 input from the search client 110 is decomposed into one-character N-gram terms “a” and “me” 520. (Step 410, Step 420)

  Next, “A” is selected as the first N-gram term, and the document ID list 134 A of “A” is read from the hard disk 133 into the search memory 132 as a reference document ID list. The reference document ID list 531 on the search memory 132 is the read “A” document ID list 134A. (Step 430)

  Next, “M” is selected as the remaining N-gram term, and the “M” document ID list 134B is read from the hard disk 133 into the search memory 132 as a comparison document ID list. The comparison document ID list 532 on the memory 530 is the “document” document ID list 134B read. (Step 440)

  Next, the reference document ID list 531 and the comparison document ID list 532 on the search memory 132 are compared, and a new reference document ID list 533 is created with the document IDs 001 and 900 appearing in both. In this example, it is assumed that the same document ID does not appear in the omitted parts of the document ID lists 531 and 532. (Step 450)

  Here, there is no N-gram term for which the document ID list has not yet been read. The new reference document ID list 533 includes only the common document IDs in the original two document ID lists 134A and 134B corresponding to the search N-gram terms “a” and “me”. . Subsequently, using the document IDs in the new reference document ID list 533 and the offset information on the character position list, that is, the character position information, each character position list 135A of “A” and “ME” in FIG. Only character position information of document IDs “009” and “900” is read from the hard disk 133 to the search memory 132 from the 135B and 135B. The read character position of the N-gram term “A” is the character position list 534, and the character position of “ME” is the character position list 535. (Step 460, Step 470) Here, only a part of the character position lists 135A and 135B in FIG. 2 can be read from the hard disk 133, so that the amount of data to be read is smaller than that of the prior art, and the search processing is performed. Can be speeded up.

  Next, looking at the character position lists 534 and 535, it can be seen that “A” appears in the first character 210b of the document ID 001, and “ME” appears in the second character 211b. ”510 is included. Similarly, since “A” appears in the second character 210d of the document ID 900 and “Me” appears in the third character 211d, it can be seen that the document ID 900 also includes the search condition “AME” 510. (Step 480)

  Finally, document IDs 001 and 900 are returned as search results to the search client 110, and the search process is terminated (step 490).

110: Search client
120: Communication line
130: Search server
131: Full-text search device
132: Search memory
133: Hard disk
134: Document ID list
135: Character position list
136: N-gram index
210: N-gram term "A" for registration
211: N-gram term "I" for registration
134A: Document ID list for “A”
134B: Document ID list for “Me”
135A: Character position list of “A”
135B: Character position list for “Me”
510: Search term
520: N-gram term for search divided into search terms
531: Document ID list for “A”
532: Document ID list for “Me”
533: Standard document ID list
534: Character position list of “A”
535: "Me" character position list

Claims (1)

Each of the plurality of documents is divided into registration partial character strings having a predetermined number of characters, and character position information indicating the positions of characters in each document of the registration partial character strings is stored for each of the registration partial character strings. Means for registering a character position list and a document ID list in which the start position of the character position information in the character position list and the document ID of each document are stored in association with each other;
The obtained search terms are divided into the same number of search partial character strings as the number of characters of the registration partial character string, and stored for each of the same registration partial character strings as each of the divided search partial character strings. Means for extracting all document IDs included in common in all the compared document ID lists by comparing the document ID lists with each other;
Means for respectively obtaining a start position of the character position information associated with each of the extracted common document IDs from the document ID list for each partial character string for search;
Means for acquiring only corresponding character position information from the character position list for each partial character string for search based on the start position of the acquired character position information;
For each of the extracted common document IDs, the search partial character strings appear adjacently in the same order as the search terms before the division based on the character position information acquired for each of the search partial character strings. And a means for determining that the search term is included in the document with the document ID when the partial character strings for search appear adjacent to each other.

Images