JP2007219620A - Text retrieval device, program, and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a text retrieval device, a text retrieval program, and a text retrieval method further reducing retrieval noise while minimizing an increase in the amount of retrieval processing or index capacity in a retrieval system using an N-gram (bi-gram) index. <P>SOLUTION: The text retrieval device is provided with a retrieval object text section for analyzing a retrieval object text to extract a retrieval word, and a retrieval analysis section for analyzing the retrieval word to extract the word. Words are extracted using respectively different analysis rules to carry out index formation and text retrieval. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a text search device, a text search program, and a text search method, and more particularly, to a text search device, a text search program, and a text search method for searching for words or sentences as keywords.

  The required items for the search service to search for the target text from multiple text sentences (search target text) include search performance, search accuracy (noise and search omission), index generation time, index capacity, etc. Are in a trade-off relationship such that search noise increases when search omission is reduced.

  For example, as one of the methods for reducing search omissions, there is a search method using a bi-gram index. In this search method, (a) a search method using an index that holds character appearance positions in a document, (B) Search noise is reduced by using a search method that uses an index that maintains the order of index words extracted from a document.

  However, in the search method of (a), since the index word position information is indicated, a byte length proportional to the length of the document is required, and the index capacity is larger than that of the search method of (b).

  Further, in the search method of (a), since the search is performed in consideration of the appearance position of characters, the amount of calculation including the character string length of the search word and the position information of the index word is increased, and the search of (b) is performed. Compared to the method, the search processing amount is also increased.

For this reason, when there is an overlapping pattern, a method of reusing a calculation result that has already been processed (for example, see Patent Document 1) or a method of dividing a text based on the ease of word boundaries (for example, Patent Document) 2), and a method of separately holding an index using a maximal word that does not overlap between index words (for example, see Patent Document 3).
Japanese Patent Laid-Open No. 10-307841 JP-A-10-320421 Japanese Patent Laid-Open No. 11-073429

  On the other hand, in the search method of (b), only the order of index words is used, so that search noise increases compared to the search method of (a).

For example, when extracting an index word from a document, when processing in units of character strings separated by a predefined delimiter,
(Example 1) The index of the text "Tokyo, ..." becomes "Tokyo", "Kyoto" ...
(Example 2) The index of the text “Tokyo, Kyoto,...” Is “Tokyo” “Kyoto”.
In this case, a comma is one of the delimiters.

  For this reason, the order of the index words is the same, and the text “Tokyo, Kyoto ...” is searched for the search word “Tokyo”, causing a search noise.

  In the above example, there is also a method of reducing the search noise by checking the last character of all index words in the “city” of the search word “Tokyo” and searching for “city” of “Kyoto”. However, the search time was proportional to the type of index word.

  In addition, there is a method of suppressing the search noise by separately searching the reverse order of the character sequence of the index word (for example, “Tokyo”, “Kyoto”, etc.), but there is a problem that the overall index capacity increases. .

  Therefore, in the present invention, in a search method using an N-gram (bi-gram) index, a text search device and a text search that can further reduce search noise while suppressing an increase in search processing amount and index capacity. It is an object to provide a program and a text search method.

  The present invention provides a text search apparatus that performs text search using an N-gram method, and uses search index text analysis means for analyzing search target text based on text analysis rules and extracting index terms, and the index terms. Index generation means for generating an index, search word analysis means for analyzing a search word included in a search condition based on a search word analysis rule, and extracting a word from the search word, and based on a word cut out from the search word Search condition generating means for generating an internal search condition, and search processing means for searching the index based on the internal search condition.

  In this configuration, in the N-gram method, the search target text and the search conditions are analyzed based on different analysis rules, so it is possible to cut out words from the search terms based on the analysis rules dedicated to the search terms. Search noise in the gram method can be reduced, and a one-character search in the N-gram index can be efficiently processed.

  Further, the text analysis rule is at least defined to cut out the last character of the processing unit character string in the search target text as the index word, and the search word analysis rule is that the search word includes two or more characters. If the last character is a non-ASCII character, it is desirable that at least the last character should not be cut out as a word.

  Next, the present invention provides a text search program for causing an information processing apparatus to execute a text search using an N-gram method, analyzing a search target text based on a text analysis rule, and extracting an index word; Generating an index using an index word; analyzing a search word included in a search condition based on a search word analysis rule; cutting out a word from the search word; and based on a word cut out from the search word The information processing apparatus is caused to execute a step of generating an internal search condition and a step of searching the index based on the internal search condition.

  In this configuration, in the N-gram method, the search target text and the search conditions are analyzed based on different analysis rules, so it is possible to cut out words from the search terms based on the analysis rules dedicated to the search terms. Search noise in the gram method can be reduced, and a one-character search in the N-gram index can be efficiently processed.

  Further, the text analysis rule is at least defined to cut out the last character of the processing target character string in the search target text as the index word, and the search word analysis rule is that the search word includes two or more characters. If the last character is a non-ASCII character, it is desirable that at least the last character should not be cut out as a word.

  In the present invention, in a text search method for performing text search using the N-gram method, a search target text is analyzed by a search target text analysis means based on a text analysis rule, and an index word is extracted, and the index word is used. The index generation means generates an index, the search word included in the search condition is analyzed by the search word analysis means based on the search word analysis rule, the word is cut out from the search word, and based on the word cut out from the search word Then, an internal search condition is generated by the search condition generation means, and the index is searched by the search processing means based on the internal search condition.

  In this configuration, in the N-gram method, the search target text and the search conditions are analyzed based on different analysis rules, so it is possible to cut out words from the search terms based on the analysis rules dedicated to the search terms. Search noise in the gram method can be reduced, and a one-character search in the N-gram index can be efficiently processed.

  Further, the text analysis rule is at least defined to cut out the last character of the character string of the search target text as the index word, and the search word analysis rule is that the search word is two characters or more and the last character It is desirable that at least it should be specified that the last character is not extracted as a word if a character is a non-ASCII character.

  In the present invention, in a search method using a bi-gram index, search noise can be further reduced while suppressing an increase in search processing amount and index capacity.

  FIG. 1 is a functional block diagram showing an example of the configuration of a text search apparatus according to the present invention.

  As shown in the figure, the text search apparatus 10 includes a search target text analysis unit 11 that extracts index words (keywords) used for text search from a search target text that is a target of text search, and an index generation unit that generates an index. 12, an index holding unit 13 that stores and holds an index in a storage device such as an HDD (Hard Disk Drive) (not shown), and each search word in a search condition that is a condition for searching for a text, A search word analysis unit 14 that performs segmentation, a search condition generation unit 15 that generates an internal search condition to be described later using a sequence of words extracted from the search word, and an index based on the internal search condition A search processing unit 16 for searching, and a search result extracting unit 17 for acquiring and outputting a text identifier of text required by the user from the index based on the search result are configured. That.

  Note that the text search device is realized by a combination of modules having each function or the like, and is also installed in an information processing device such as a general-purpose server. The text search method can realize the text search method according to the present invention. It can be realized using a program or the like.

  Here, FIG. 2 is a schematic diagram illustrating an example of a processing flow when an index word group is extracted from the search target text and an index is generated.

  First, the multi-byte alphanumeric characters in the search target text are unified to ASCII characters (half-width alphanumeric characters), and the processing for normalizing the index word is performed, such as unifying half-width katakana to full-width katakana.

  Next, the search target text is analyzed according to the following analysis rule (text analysis rule) to extract an index word group.

Text analysis rules:
・ Symbols and spaces are delimiters and not index words.
-Concatenated parts of ASCII and non-ASCII shall be index word separators.
-If the search word is two or more characters and the last character is a non-ASCII character, the word of only the last character is extracted as an index word.
-A processing unit character string in the search target text is determined based on the above delimitation rule.
-The following rules apply to processing unit character strings.
・ Consecutive ASCII characters shall be one index word.
・ Non-ASCII characters are indexed by bi-gram.
In the case of a non-ASCII character string of two or more characters, when extracting a bi-gram index word, the word of only the last one character is extracted as an index word.

  For this reason, for example, as shown in FIG. 2 (a), when the text “XYZ Co., Ltd. with a branch office in Tokyo has announced a new product” is analyzed as a search target, FIG. b) “Tokyo” “Kyoto” “In Tokyo” “Branch” “Branch” “Company” “Hold” “Hold” “XYZ” “Co” “Ltd” “ Extracts "new", "newly made", "product", "product", "departure", "announcement", "representation", "done", and "ta" as index word groups.

  Then, the extracted index word group is converted into an inverted file while maintaining the extraction order, and the index shown in FIG. 2C is generated.

At this time, it is desirable to generate at least the next index.
An index of a data structure capable of obtaining a text identifier in which an index word appears from one index word (note that it is desirable that all index words are sorted in the character code order).
An index of a data structure capable of obtaining an index word appearing in the text and an order of appearance from one text identifier.

  In addition, in the index of the data structure (the former) that can obtain the identifier information of the text in which it appears from the index word, the information of the text identifier and the appearance position (such as the index word) in the text The index of the data structure (the latter) that can determine the index word appearing in the text identifier and the order of appearance from the text identifier may be omitted.

  In addition, the data structure is not limited to the data structure shown in FIG. 2, and an arbitrary inverted data structure such as an inverted data structure conventionally used may be used. it can.

  FIG. 3 is a schematic diagram showing an example of the flow of processing when analyzing a search word and generating an internal detection condition. Note that a character string enclosed in double quotes indicates that the character string is one search term.

  First, the process to normalize the search word is performed. If the search word is 2 or more characters and the last character is a non-ASCII character, only the last one word is not extracted as an index word. The word is cut out by performing the same process as the process of extracting the index word group by analyzing the search target text described with reference to FIG.

  That is, a search word is analyzed according to an analysis rule (search word analysis rule) shown below to cut out a word.

Search term analysis rules:
-Index terms are extracted from search terms in the same way as search target text analysis rules. However, if the search word is two or more characters and the last character is a non-ASCII character, only the last character is not extracted.

  Next, a search condition (internal search condition) for searching for text in which words extracted from the search word continuously exist is generated.

  Therefore, for example, as shown in FIG. 3A, if “Tokyo” and “XYZ” are used as search conditions, the words “Tokyo” and “Kyoto” are extracted from “Tokyo” and “XYZ”. The word “XYZ” is extracted from the search condition, and the internal search condition is a search condition that “Tokyo” and “Kyoto” appear successively and “XYZ” appears.

  Also, as shown in FIG. 3B, when “Tokyo / Kyoto XYZ” is used as a search condition, the words “Tokyo”, “Kyo”, “Kyoto”, “Metro”, and “XYZ” are cut out, and the internal The search conditions are such that “Tokyo”, “Kyo”, “Kyoto”, “Miyako”, and “XYZ” appear successively.

  In the conventional bi-gram text search, the words “Tokyo” and “Kyoto” are both extracted as “Tokyo” or “Tokyo / Kyoto”. However, in the text search method of the present invention, the text including “Tokyo” in the index is detected by the search condition 1, and the text including “Tokyo / Kyoto” in the index is searched. Since search results corresponding to the search conditions are obtained, search noise can be reduced.

  Using the above configuration, the text search apparatus according to the present invention first analyzes the search target text to be searched by the search target text analysis unit 11 to extract index words, and generates the index generation unit 12 index. The index holding unit 13 stores and holds the index.

  Next, the search term analysis unit 14 extracts words from the search terms, and the search condition generation unit 15 generates internal search conditions.

  Then, the index is searched by the search processing unit 16 by a normal search method based on the internal search condition, and the text identifier of the text required by the user is acquired from the index by the search result extracting unit 17 and output.

  FIG. 4 is a flowchart showing a flow of processing when an index of search target text is generated in the search device according to the present invention.

  First, when a search target text is acquired (step 100), the search target text is analyzed based on an analysis rule for the search target text (step 101).

  Next, an index word group is extracted based on the analysis result, and an index is generated and stored (step 102).

  FIG. 5 is a flowchart showing the flow of processing when searching for the text to be searched based on the search condition input by the user in the search device according to the present invention.

  First, when the search condition input by the user is acquired (step 200), the search term is analyzed based on the analysis rule of the search term (step 201).

  Next, an internal search condition is generated based on the analysis result (step 202), an index is searched based on the internal search condition (step 203), and the search result is output (step 204).

  Note that the search processing in the present invention is not limited to the bi-gram method, and can be used for the N-gram method.

  Also, when a one-word search word is given, the search is performed by expanding the index word starting from that character.

  As described above, in the present invention, the search target text and the search word are conventionally analyzed based on the same analysis rule. The search target text analysis unit for extracting the index word from the search target text and the search word are used. Since two types of search word analysis units for extracting words are prepared and analysis is performed using different analysis rules, it is possible to make the order of the words cut out from the search words and the index words cut out from the text different.

  For example, if the search target text (string) is “Minato-ku, Tokyo,…”, “Tokyo”, “Kyoto”, “Minato-ku”, “Minato-ku”, “Ku” ... In the case of “...”, “Tokyo”, “Kyo”, “Kyoto”, “Miyako”, etc. are indexes.

  When the search term is analyzed in the same way as the analysis of the text to be searched (conventional method), the text “Tokyo”, “Kyoto” and “Miyako” appear continuously in the search term “Tokyo”. This means that the text “Minato-ku,…, Tokyo” cannot be searched.

  On the other hand, in the present invention, since the analysis rules are different between the analysis of the search word and the analysis of the search target text, when the search word is “Tokyo”, the words “Tokyo” and “Kyoto” are cut out, When the search term is “Tokyo, Kyoto”, the words “Tokyo”, “Kyo”, and “Kyoto” are extracted.

  Therefore, only the text “Tokyo, Kyoto,...” Is searched for the search word “Tokyo, Kyoto” and the search word “Tokyo, Kyoto” for the search word “Tokyo”.

  Therefore, in the present invention, it is possible to obtain a search result corresponding to a search word, and search noise can be reduced using a bi-gram (N-gram) method.

It is a functional block diagram which shows an example of a structure of the text search device in this invention. It is a schematic diagram which shows an example of the flow of a process at the time of producing | generating an index. It is a schematic diagram which shows an example of the flow of a process at the time of analyzing a search word and producing | generating an internal detection condition. It is a flowchart which shows an example of the flow of a process at the time of producing | generating the index of search object text. It is a flowchart which shows an example of the flow of a process at the time of searching a search object text based on the search conditions input by the user.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Text search apparatus 11 ... Search object text analysis part 12 ... Index production | generation part 13 ... Index holding | maintenance part 14 ... Search term analysis part 15 ... Search condition production | generation part 16 ... Search processing part 17 ... Search result extraction part

Claims (5)

In a text search device that performs text search using the N-gram method,
Analyzing the text to be searched based on text analysis rules and extracting index terms,
Index generating means for generating an index using the index word;
A search term analysis means for analyzing a search term included in the search condition based on a search term analysis rule and cutting out a word from the search term;
Search condition generating means for generating an internal search condition based on a word extracted from the search word;
A text search apparatus comprising: search processing means for searching the index based on the internal search condition. The text analysis rule is:
It is defined at least that the last character of the processing unit character string in the search target text is cut out as the index word,
The search term analysis rule is:
2. The text search apparatus according to claim 1, wherein if the search word is two or more characters and the last character is a non-ASCII character, at least the last character is not extracted as a word. In a text search program that causes a computer to execute a text search using the N-gram method,
Analyzing the text to be searched based on text analysis rules and extracting index terms;
Generating an index using the index terms;
Analyzing a search term included in the search condition based on a search term analysis rule, and cutting out a word from the search term;
Generating an internal search condition based on a word extracted from the search word;
A text search program for causing a computer to execute the step of searching the index based on the internal search condition. The text analysis rule is:
It is defined at least that the last character of the processing unit character string in the search target text is cut out as the index word,
The search term analysis rule is:
4. The text search program according to claim 3, wherein if the search word is two or more characters and the last character is a non-ASCII character, at least the last character is not extracted as a word. In a text search method that performs text search using the N-gram method,
The search target text is analyzed by the search target text analysis means based on the text analysis rules, and index words are extracted.
An index is generated by index generation means using the index word,
The search terms included in the search conditions are analyzed by the search term analysis means based on the search term analysis rules, and the words are extracted from the search terms,
An internal search condition is generated by a search condition generation unit based on a word extracted from the search word,
A text search method, wherein the index is searched by search processing means based on the internal search condition.

Images