JPH10307841A - Text retrieving device and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the efficiency of retrieving processing by making it possible to reuse an initial operation result at the time of next appearance when an overlapped pattern exists in a retrieving character string. SOLUTION: A two-gram index storing part 101 uses two continuous characters in a retrieved text as a key and stores a two-gram index registering the positions of the characters concerned in a retrieved text. An overlapped pattern detection part 103 extracts an overlapped pattern constituted of characters more than two continuous characters from the retrieving character strings stored in a retrieving word storing part 102. A position retrieving part 104 retrieves the existing position of the overlapped pattern detected by the detection part 103 and stores the retrieved result in a comparing operation result storing part 105. The position retrieving part 104 retrieves the existing positions of characters other than the overlapped pattern and acquires the existing position of the retrieving character string based on the retrieved results and the contents stored in the storing part 105.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a text retrieval apparatus and method for retrieving a character string from text data at high speed using an index. In particular, N-g
The present invention relates to a text search apparatus and method for speeding up search processing in a text search employing a ram index method.

[0002]

2. Description of the Related Art In a text search apparatus such as a full-text search apparatus that searches a document including a given search word for all texts in a document, a large number of texts are searched at a high speed. An index technique of creating an index in advance and performing a search using the index is used.

One of such index systems is N-
There is an index method called a gram index. The N-gram index method is an index that holds a position list listing the positions of the character strings serving as keys in the search target text, using consecutive N characters in the search target text as keys.

For example, if N is fixed to 2, the character string is 2
The index will be created by cutting each character. As an example, the character string “Gentle Agent” is as shown in FIG. For each character string obtained in this way, a 2-gram index is created by registering the appearance position in the document text. Compared to the method using one character as an index, the number of index patterns is increased, but the number of appearances of each index is significantly reduced, so that a faster search process can be performed.

Next, a search process using an N-gram index will be described. Here, the case where N is fixed to 2 will be described, but the basics are the same even if the value of N changes.

First, a search character string is cut off two characters at a time from the beginning, a 2-gram index is inquired, and its appearance position information is stored. FIG. 5 shows an example of extracting two characters from the search character string. First, a 2-gram index is searched for two characters "J" to extract position information. Next, for the next two characters,
Similarly, the position information is extracted, and only the character positions that are increased by 2 from the character position obtained in the previous “J” are checked.
Others are excluded because the character strings are not continuous.

As described above, all of the character strings of two characters obtained from the search character string that have been checked are the appearance positions including the search character string. If the length of the search character string is odd, position information is extracted from the character position index for the last character alone, and the same processing is performed. Here, the character position index is obtained by registering a character position in the searched text for each character in the searched text. That is,
When the N-gram index is used, the character position index is also used.

[0008]

However, in the above-mentioned prior art apparatus, it is costly to perform an operation for combining the front character string with the position information of the two following characters. This operation is
For example, if the number of appearances of the front character string “J” is N and the number of appearances of the rear character string “N” is M, the comparison operation of N × M occurrence positions is performed, so that N and M become large. If
The ratio occupied by the comparison operation increases.

The present invention has been made in view of the above-described problem, and checks whether or not there is a duplicate pattern that is the same partial character string from a search character string. It is an object of the present invention to provide a text search apparatus and a text search method that enable a search result to be reused at the next appearance and improve the efficiency of search processing.

[0010]

Means for Solving the Problems A text search apparatus according to the present invention for achieving the above object has the following arrangement.
That is, using a character in the searched text as a key, a holding unit for holding a first index in which the position of the character in the searched text is registered, and two or more consecutive characters from a given search character string Extracting means for extracting a duplicate pattern composed of characters and present in the search character string in an overlapping manner; and using the first index held by the holding means, the position of the duplicate pattern extracted by the extracting means. A first search unit for searching for a character string, and a second search unit for searching for an existing position of a character other than the overlapping pattern in the search character string using the first index held in the holding unit.
A search unit; and an obtaining unit that obtains an existing position of the search character string based on search results obtained by the first search unit and the second search unit.

Further, the text search method according to the present invention for achieving the above object comprises a character string consisting of two or more consecutive characters from a given search character string. An extraction step of extracting a pattern, and using the first index in which the position of the character in the searched text is registered using the character in the searched text as a key, the location of the duplicate pattern extracted in the extracting step A second search step of searching for a position of a character other than the overlapping pattern in the search character string using the first index, a first search step of searching for 2 Based on the search results obtained in the search process,
Obtaining an existing position of the search character string.

[0012]

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. In the following embodiment, the case where N of N-gram is fixed to 2 will be described, but the basics are the same even if the value of N changes.

FIG. 1 is a block diagram showing a configuration of a document processing apparatus according to an embodiment of the present invention. In the figure, 1
Reference numeral 01 denotes a 2-gram index holding unit that holds an index holding the position of the character in the searched text, using the character and the character string in the searched text as keys. A search word holding unit 102 holds a character string to be searched. Reference numeral 103 denotes a duplicate pattern detection unit that detects a duplicate pattern of two or more consecutive characters from the search character string. 104 is a 2-gram index holding unit 1
Using the 2-gram index stored in 01, a position including the search character string is searched from a match between the search character string stored in the search character string storage unit 102 and an arbitrary character string in the searched text. This is a position search unit to perform. 105
Reference numeral denotes a comparison calculation result holding unit that holds a search result of the overlap pattern detected by the overlap pattern detection unit 103 at the time of calculation by the position search unit 104. Reference numeral 106 denotes a search result holding unit that holds the search results searched by the position search unit 105.

In the above configuration, since a known method for generating a 2-gram index can be employed, a portion for generating the index is not shown.

FIG. 2 is a diagram showing a hardware configuration of the above-described document processing apparatus. In FIG. 1, reference numeral 201 denotes a CPU, which operates according to a program for implementing a procedure described later. Reference numeral 202 denotes a RAM, which is the search word holding unit 10 described above.
2. Comparison operation result holding unit 105, search result holding unit 106
And a storage area necessary for the operation of the program. Reference numeral 203 denotes a ROM, which stores a control program that implements a procedure described below. Reference numeral 204 denotes a disk device that implements the 2-gram index holding unit 101. An input unit 205 includes a keyboard and a mouse, and performs input of a search character string, input of a search start instruction, and the like. Reference numeral 206 denotes a display for displaying search results and the like. A bus 207 connects the above-described components.

In this embodiment, the control program is R
It is stored in the OM and executed by the CPU 201. However, the control program is stored in an external storage device such as the disk device 204, and the control program is loaded into the RAM 202 and executed by the CPU 201. Obviously, you can.

A description will be given of a text search process in the document processing apparatus according to the present embodiment having the above configuration.

FIG. 3 is a flowchart showing a processing procedure of the operation in the apparatus shown in FIG. 5 and 6
Is a specific example for explaining the search processing of the present embodiment. Hereinafter, referring to the flowchart of FIG. 3, FIG.
2-g that was made more efficient using overlapping patterns using the example of
5 shows a ram index search processing procedure.

First, in step S301, a duplicate pattern (duplicate pattern) in a search word is detected. Here, the word “gentle agent” is used as an example of a search character string (hereinafter, a search word). The character string “gent” is used twice in this search word. That is, “gent” is detected as an overlapping pattern. The overlapping pattern is not limited to one search word. For example, an overlap pattern ("material" in this example) is detected when two conditions (search terms) such as "carbon material AND functional material" are combined by AND / OR or the like.

If duplicates are detected in step S301, a search for duplicate patterns is performed in step S302. In this example, for "gent", the position index of the first two characters "je" is read,
Read the position index of "nt". Then, both indices are compared, and only the consecutive one, that is, the one whose latter position is larger than the former by 2 is written in the buffer. In this embodiment, the overlapping pattern starts from the beginning of the search word, but a position index is created from the overlapping pattern regardless of whether the overlapping pattern is at the beginning or not.

If the character string is longer, the result is compared with the position index of the next two characters, and the processing is continued until the end. If the length is odd, position information is obtained from the one-character index for the last character. In step S303, the position index written in the buffer is set in the comparison operation result holding unit 105. When there are a plurality of overlapping patterns, similarly, the field processing is executed for all the overlapping patterns, and the position indexes of all the overlapping patterns are set.

In step S304, similar search processing is performed on character strings other than the overlapping pattern. In this example, it is "(gent) rue (gent)". First, the position index of “Le.” Is read, and the position index of “A” is also read. Then, both indices are compared, and only the consecutive one, that is, the one whose latter position is larger than the former by 2 is written in the buffer.

In step S305, position indexes are combined for "gent" overlapping "gent" and "rue". In other words, the position index of “gent” and the position index of “rue” are compared and calculated, and only those that are continuous, in this example, those whose position of the latter is larger by 4 than the former, are left. And finally,
If the position index of “gent” is compared again with this buffer, and only the position index larger than the former by 8 is left, only the position of the search word remains.

On the other hand, if no duplication is detected in step S301, processing is performed as a normal search in step S306.

As described above, according to the present embodiment, in the conventional method, as shown in FIG. 5, five comparison operations are required for six partial character strings composed of two characters. As shown in FIG. 6, it is sufficient to perform four comparison operations on four partial character strings. As described above, the number of partial character strings and the number of comparison operations are reduced, so that the search processing can be performed efficiently and at high speed. That is, the calculation cost can be reduced by using the overlapping pattern in the search word, so that the search can be performed at higher speed.

<Other Embodiments> (1) In the above embodiment, although the search processing is closed only once, patterns with high frequency are statistically extracted and compared with those patterns. You may comprise so that a result may be hold | maintained. In a search in which a specific pattern is frequently used, the search cost can be significantly reduced. However, when the searched document is updated, the held result needs to be discarded.

FIG. 7 is a block diagram for explaining a configuration for holding an index for an overlapping pattern that can support a plurality of search processes. In the figure, the same components as those in FIG. 1 are denoted by the same reference numerals. Reference numeral 301 denotes a statistical processing unit,
The appearance frequency of the overlapping pattern held in the comparison calculation result holding unit 105 is obtained. Reference numeral 302 denotes a duplicate pattern index holding unit which stores a duplicate pattern index in which the position of the duplicate pattern whose appearance frequency exceeds a predetermined value in the searched text is registered in the statistical processing unit 301.

The position search unit 104 is provided with a duplicate pattern detection unit 1
It is checked whether or not the duplicate pattern detected in 03 is registered in the duplicate pattern index holding unit 302, and if it is registered, position information is directly obtained from the duplicate pattern index. That is, it is possible to omit the search for the duplicate pattern using the 2-gram index.

In the above description, an index is generated using the entire character string of the overlapping pattern as a key, but the present invention is not limited to this. For example, it is also possible to generate an index using a character string obtained by dividing a character string of an overlapping pattern into N characters at a time. For example, if the overlapping pattern "gent" is divided into two characters, "je" and "nt"
Since the two character strings appear, the statistical processing unit 301 acquires the appearance frequencies of these two character strings. Then, when the appearance frequency exceeds a predetermined value, position information using the character string as a key may be added.

(2) In the above-described embodiment, the position index is obtained first because the overlapping pattern is at the head. However, the calculation order may be performed when the overlapping pattern appears. . Of course, calculations may be performed in parallel.

(3) In the above embodiment, the nested structure of the overlapping patterns is not described, but it is clear that the processing described in the above embodiment can be applied even if the overlapping patterns have the nested structure. . For example, in the example of "(natural material OR natural material) AND carbon material", "material" overlaps at three places, and "Nara material" overlaps at two places. In such a case, first, for the “material” included in the “nature material”, the position index of the overlapping pattern is obtained, and the result is compared with the one character “nature” to calculate the position of the “nature material”. What is necessary is just to ask for an index.

(4) In the above embodiment, a Japanese character string is taken as an example. However, a similar process can be performed on a character string in a language other than Japanese. It goes without saying that it is good.

(5) It should be noted that the present invention provides a plurality of devices (for example, a host computer, an interface device, a reader,
Printer, etc.)
The present invention may be applied to a device including one device (for example, a copying machine, a facsimile device, etc.).

An object of the present invention is to provide a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to provide a computer (or CPU) of the system or apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided on a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instructions of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0039]

As described above, according to the present invention,
Check if there is a duplicate pattern that is the same substring from the search string, and if there is a duplicate pattern, the search result can be reused at the next appearance, so search processing Efficiency is improved.

[0040]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a document processing apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a hardware configuration of the above-described document processing apparatus.

FIG. 3 is a flowchart showing a processing procedure of an operation in the device shown in FIG. 1;

FIG. 4 is a diagram illustrating a character string extraction mode when a 2-gram index is generated.

FIG. 5 is a diagram illustrating an example of extracting two characters from a search character string.

FIG. 6 is a specific example illustrating a search process according to the embodiment.

FIG. 7 is a block diagram illustrating a configuration of a document processing apparatus according to another embodiment of the present invention.

Claims (15)

[Claims] 1. Characters in a search target text are used as keys,
Holding means for holding a first index in which the position of the character in the search target text is registered; and two or more consecutive characters from a given search character string, which overlap with the search character string. Extracting means for extracting an existing overlapping pattern, and a first index held in the holding means,
Using a first search unit that searches for the position of the duplicate pattern extracted by the extraction unit, and a first index stored in the storage unit,
A second search unit that searches for a position of a character other than the overlapping pattern in the search character string; and a search character string based on search results obtained by the first search unit and the second search unit. A text search device comprising: an obtaining unit configured to obtain an existing position of a document. 2. The method according to claim 1, wherein, when N is an integer of 2 or more, a position of each character string in a search target text is registered using a character string composed of N characters as a key. The text search device according to claim 1, wherein: 3. The method according to claim 1, wherein the first search unit obtains a position in the searched text of a character string obtained by dividing the overlapping pattern into N characters by referring to the first index. The text search apparatus according to claim 2, wherein the existence position of the overlapping pattern is obtained based on the position. 4. The apparatus further comprises: a second holding unit that holds a second index for registering an existing position in the text to be searched, using the duplicate pattern obtained by the first search unit as a key, wherein the first search unit includes: 2. The text search apparatus according to claim 1, wherein a search is made for the location of the duplicate pattern extracted by the extraction means in the search target text, based on the first index and the second index. 5. The apparatus according to claim 4, wherein the second holding unit obtains statistics of the appearance of the overlapping pattern, and uses the overlapping pattern whose appearance frequency exceeds a predetermined value as a key of the second index. Text search device. 6. A statistical means for acquiring an appearance frequency of each character string obtained by dividing the overlapping pattern into N characters, and a character string whose appearance frequency exceeds a predetermined value in the statistical means is used as a key. And a second holding unit for holding a second index for registering a position in the searched text, wherein the first searching unit is configured to divide the overlapping pattern into N characters at a time. The text search apparatus according to claim 3, wherein a position in the searched text is obtained based on the first and second indexes. 7. The text search apparatus according to claim 1, further comprising a generation unit configured to generate a first index stored in the storage unit from the search target text. 8. An extracting step of extracting a duplicate pattern that is composed of two or more consecutive characters from a given search character string and that overlaps the search character string; and a character in the search target text. A first search step of searching for an existing position of the duplicate pattern extracted in the extraction step using a first index in which the position of the character in the searched text is registered, using A second search step of searching for an existing position of a character other than the overlapping pattern in the search character string, based on search results obtained in the first search step and the second search step, An acquisition step of acquiring the location of the search character string. 9. The text search method according to claim 8, further comprising a generation step of generating the first index. 10. The method according to claim 1, wherein, when N is an integer of 2 or more, a position of each character string in the search target text is registered using a character string composed of N characters as a key. The text search method according to claim 8, wherein: 11. The first search step obtains a position in the search target text of a character string obtained by dividing the overlapping pattern into N characters by referring to the first index. The text search method according to claim 10, wherein the existence position of the overlapping pattern is obtained based on the position. 12. The method according to claim 12, further comprising a holding step of holding a second index in which the positions of the overlapping patterns obtained in the first searching step are registered by using the overlapping patterns as keys, wherein the first searching step includes: 9. The text search method according to claim 8, wherein a search is made for a location of the duplicate pattern extracted in the extraction step based on the second index. 13. The text search according to claim 12, wherein, in the holding step, statistics of appearing duplicate patterns are obtained, and a duplicate pattern whose appearance frequency exceeds a predetermined value is used as a key of the second index. Method. 14. A statistical step of acquiring an appearance frequency of each character string obtained by dividing the overlapping pattern into N characters, and a character string whose appearance frequency exceeds a predetermined value in the statistical step is used as a key. The search method further includes a holding step of holding a second index for registering a position in the search target text, wherein the first search step is performed by dividing the overlap pattern into N characters at a time. Is obtained by referring to the first and second indexes,
11. The position of the duplicate pattern in the searched text is acquired based on the positions.
Text search method described in. 15. A computer-readable memory storing a control program for searching for a searched text with a given search character string, the control program comprising: two consecutive characters from a given search character string. Register the code of the extraction process, which consists of the above characters and extracts the duplicate pattern that exists in the search character string, and the position of the character in the search target text using the character in the search target text as a key A code of a first search step of searching for the position of the duplicate pattern extracted in the extraction step using the obtained first index; and a code other than the duplicate pattern in the search character string using the first index. The search character based on a code of a second search step for searching for the location of the character, and search results obtained in the first search step and the second search step A computer-readable memory characterized by comprising a code acquisition step of acquiring location of.