JP2006179019A - Document retrieval device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quicken retrieval in retrieving a document by using a conditional formula including a plurality of retrieval character strings. <P>SOLUTION: In retrieving a character string based on a conditional formula including a plurality of retrieval character strings, the signatures of all retrieval character strings included in the conditional formula are extracted, and a cursor for scanning a bit map bit-sliced according to bits whose values are 1 of each signature is prepared, and retrieval is executed by making the cursor scan the bit map according to the contents of the conditional formula in parallel. Thus, it is possible to obtain a retrieval result only by scanning a signature file once even when the retrieval character strings included in the conditional formula are increased, and to quicken retrieval. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a document search apparatus, and more particularly, to a document search apparatus that searches for a document including a character string specified using a signature file.

  Conventionally, word processors and personal computers using word processing software have been used to create and edit documents by handling multiple character types such as English and Japanese, and document data input to these devices A document search device is used to search for a partial character string.

  As this type of document search device, for example, there is a document search device described in Japanese Patent Application Laid-Open No. 7-244671. By using a binary bit pattern signature extracted from a character string by a certain method. A document search was being performed. The bit position where “1” is set in the binary bit pattern of this signature is to digitize the characters and words that make up the character string, and to hash the value from 0 to the maximum value of the bit position. Is gained by. For example, if there is “Copy” as a character string, “Co”, “Pi”, “-” can be digitized as “5”, “7”, “12” using character codes. Indicates the bit position, and “1” stands at the “5”, “7”, and “12” -th positions, and becomes “00000100001”. This pattern of 0 and 1 is called a bitmap (bit pattern), and a signature is constituted by this bitmap.

  The signature extraction method is described in Non-Patent Document 1. According to this non-patent document 1, a signature called a word signature is created for each word constituting document data, and a signature obtained by superimposing the signature is used as the document data signature. Here, superimposing is an operation of taking a logical sum of bit values at the same position in a plurality of signatures and extracting a column of values of each logical sum as a new signature.

  In addition, in order to be able to search for partial character strings constituting words, the document data is divided into character strings having a certain number of characters while having overlapping portions, and each character string is divided in the same manner as the word signature described above. There is a way to superimpose signatures.

  There is also a method in which longer document data is divided into logical blocks such as sentences and paragraphs, and a plurality of signatures extracted from each block are associated with one document. Here, the signature extracted from the block is called a block signature.

  When signatures are used to search for documents that contain search strings, signatures with the same bit pattern may be extracted from different strings, so documents that do not contain search strings are detected as search results. Sometimes. This document is called false drop. On the other hand, a document including a search character string is called an actual drop.

  In a conventional document search apparatus, a signature is extracted for each document, and each signature is collectively stored in a file called a signature file. Signature files are roughly classified into two types according to the signature storage method. One is a method of simply storing signatures in order. A file structure by this method is called a sequential structure. The other is a method in which each bit of the signature is stored in a separate bitmap for each bit position. A file structure by this method is called a bit slice structure. A signature file having a bit slice configuration is described in Non-Patent Document 2.

  In addition, as described in Non-Patent Document 3, there is a method of using run-length coating or the like as a method of compressing a sequential signature bitmap.

  In a conventional document search device, when searching by a conditional expression including a plurality of search character strings connected by logical operators AND and OR, such as "high speed or printer", the signature file is scanned for each search character string. And the final search result is obtained by performing a set operation on each search result.

“Access Method of Text” (Christos Faloutsos, Computing Surveys, Vol. 17, No. 1, March 1985, pp 49-74) "Partial-Match Retrieval via Method of Superimposed Codes" (Charles S. Roberts, Proceedings of the IEEE. Vol.67, No.12, December, 1979, pp1624-1979) "Description and Performance Analysis of Signature File Methods for Office Filing" (Christos Faloutsos, ACM Transaction Office Information Systems, Vol.5, No.3, July 1987, pp.237-257)

  However, in such a conventional document search apparatus, when a search is performed using a conditional expression including a plurality of search character strings, the number of scans increases as the number of search character strings included in the conditional expression increases. Therefore, the search time becomes longer and the search performance is lowered.

  The present invention has been made in view of the above, and an object of the present invention is to provide a document search apparatus capable of speeding up a search when searching for a document using a conditional expression including a plurality of search character strings. .

  In order to solve the above-mentioned problems and achieve the object, the invention according to claim 1 is directed to a document search apparatus for searching document data including a search character string using a binary bit pattern signature. A signature extraction processing unit that extracts a document, and the document data is divided into blocks that are character strings of a predetermined number of characters, and the signature is extracted from the character strings that constitute each block by using the signature extraction processing unit. A document registration processing unit to be stored in a signature file as a signature, a partial character string of a predetermined number of characters is extracted from the search character string, and a signature of the search character string is extracted from the partial character string using the signature extraction processing unit And stored in the signature file based on the signature of the search string. A document search processing unit that searches for the block signature, wherein the document search processing unit is a conditional expression that includes a plurality of search character strings that are connected by logical operators. A signature is extracted from all the search character strings included in the search character string, and is associated with each search character string that scans a bit-sliced bitmap according to a bit whose extracted signature value is 1. A cursor is prepared, and searching is performed while each cursor is scanned in parallel according to the contents of the conditional expression.

  According to a second aspect of the present invention, in the document retrieval apparatus according to the first aspect, the cursor is a number of bits whose value is 1 in a bitmap that is bit-sliced according to the cursor for each retrieval character string. Are arranged in ascending order.

  The invention according to claim 3 is the document search device according to claim 1, wherein the conditional expression is a logical expression format in which a literal associated with the search character string is a disjunctive disjunction. It is characterized by being converted to the conjunction standard form.

  According to a fourth aspect of the present invention, in the document retrieval device according to the first aspect, when the primary cursor corresponding to the search character string reaches the end of the bit-sliced bitmap, the primary cursor corresponds to the primary cursor. It is characterized in that a conjunction with a search character string as a literal is removed.

  According to the invention of claim 1, when searching for a character string based on a conditional expression including a plurality of search character strings, the signatures of all the search character strings included in the conditional expression are extracted, and each signature is extracted. A search character string included in a conditional expression for preparing a cursor that scans a bit-sliced bitmap according to a bit whose value is 1 and scanning the cursor in parallel according to the contents of the conditional expression Even if the number of files increases, it is possible to obtain a search result by scanning the signature file once, and it is possible to speed up the search when searching a document using a conditional expression including a plurality of search character strings. There is an effect.

  Moreover, according to the invention concerning Claim 2, there exists an effect that search efficiency can be improved.

  Further, according to the invention of claim 3, by setting the conditional expression in the conjunction standard form, it is possible to determine whether the whole conditional expression is true or false by the truth of any of the conjunctions. Play.

  Moreover, according to the invention concerning Claim 4, there exists an effect that a conditional expression can be simplified.

  Embodiments 1 to 4 of the present invention will be described below with reference to the drawings.

  First, before entering the description of each embodiment, a schematic configuration of a document search apparatus common to each embodiment will be described. FIG. 1 is a block diagram showing a schematic configuration of a document search apparatus according to the present embodiment. In FIG. 1, a document search apparatus includes an input unit 1, a processing unit 2, a character string input processing unit 3, a record identifier calculation processing unit 4, a document search processing unit 5, a document output processing unit 6, a storage position calculation processing unit 7, The document registration processing unit 8, signature extraction processing unit 9, output unit 10, data unit 11, signature file 12, record file 13, etc.

  The character code of the target search character string and the document data of the registered document is EUC (Extended Unix Interchange) even if the number of bytes of each character is uniform, as in ASCII (American Standard Code Information Interchange). Thus, a character code in which 1-byte, 2-byte, and 3-byte characters are mixed may be used. Note that the character code for internal processing used in the character string input processing unit does not have to be the same as the character code to be converted (for example, EUC). In the present invention, a case where EUC is used will be described.

  As shown in FIG. 1, the search character string input from the input unit 1 and the document data of the registered document are converted from the input character code into EUC by the character string input processing unit 3 of the processing unit 2. Further, the document data that is an actual drop at the time of retrieval is converted from EUC to an output character code by the document output processing unit 6. Therefore, the search character string and the document data of the registered document are always processed as an EUC character string in the processing unit 2, and the document data is always stored in the record file 13 of the data unit 11 as an EUC character string.

  As described above, the document search apparatus according to the present embodiment has a function of registering and searching for a document, and includes four parts: an input unit 1, a processing unit 2, a data unit 11, and an output unit 10. . The processing unit 2 includes a character string input processing unit 3, a signature extraction processing unit 9, a document search processing unit 5, a record identifier calculation processing unit 4, a document output processing unit 6, a document registration processing unit 8, and a storage location calculation process. Although it is provided with a unit 7 and is not illustrated, it further includes a cursor movement processing unit, a conditional expression clearing unit, and a conditional expression expectation calculating unit, which will be described later.

  The data unit 11 includes a signature file 12 and a record file 13. When registering a document, the input unit 1 accepts input of document data to be registered. The input document data is converted into a predetermined character code by the character string input processing unit 3 and then passed to the document registration processing unit 8. . The document registration processing unit 8 first stores the document data in the record file 13 of the data unit 11. Second, the storage position calculation processing unit 7 is used to calculate the signature storage position from the identifier of the record storing the document data. Third, the document data is divided into blocks having a certain number of characters, and a block signature is extracted from each block using the signature extraction processing unit 9. However, adjacent blocks have overlapping portions with a certain number of characters. Fourth, the block signature is stored in a predetermined storage position in the signature file 12 of the data part 11.

  As for the document search operation, first, a search character string is received by the input unit 1. The input search character string is converted into a predetermined character code by the character string input processing unit 3, and then the document search processing unit 5. Passed to. First, the document search processing unit 5 extracts a partial character string within a limited number of characters from the search character string, extracts a signature from the partial character string using the signature extraction processing unit 9, and extracts the signature. Is the signature of the search string. Here, the limited number of characters is the number of characters in the overlapping portion between the blocks constituting the registered document data. Secondly, the bit for which “1” is set in the search signature is checked, and in the signature file of the data part 11, the bit-sliced bitmap corresponding to the previously checked bit is referred to, and the search character string The storage location of the block signature extracted from the block that is determined to contain the is obtained.

  Third, the record identifier calculation processing unit 4 is used to obtain the value of the record identifier from the storage location of the block signature. Fourth, in the record file 13 of the data part 11, the document data is obtained by referring to the record corresponding to the record identifier. Fifth, the obtained document data is transferred to the document output processing unit 6. The document output processing unit 6 checks whether or not the passed document data really includes the search character string, removes the false drop, converts the actual drop into a predetermined character code, and then outputs the document data to the output unit 10. To pass. The output unit 10 outputs the passed document data.

(Embodiment 1)
In the first embodiment, when searching for a character string with a forward match and a backward match as a condition using the processing unit 2 of the document search apparatus, the following search character string p1 with a forward match and a backward match as a condition: p2 and search target character strings t1, t2, and t3.
p1 = "High speed"
p2 = "Printer"
t1 = "High-speed printer"
t2 = "Metropolitan Expressway"
t3 = "Printer output"

In these character strings, hypothetical characters for distinguishing between the beginning and the end are “^” (characters indicating the beginning of the character string) and “$” (characters indicating the end of the character string). The signature is extracted after converting to a character string as shown below.
p1 '= "^ High speed"
p2 '= "Printer $"
t1 '= "^ High-speed printer $"
t2 '= "^ Metropolitan Expressway $"
t3 '= "^ Printer output $"

  As described above, in the first embodiment, since the signature is extracted including the virtual character for distinguishing between the beginning and the end of the character string, the search character strings p1 ′ and p2 ′ are used as the condition. In any case, only the “high-speed printer” at t1 ′ can be searched. This means that t1 and t2 are search results under the p1 condition and t1 and t3 are search results under the p2 condition as in the conventional example. As a result, search performance and search speed can be improved.

(Embodiment 2)
In the second embodiment, when the document data is divided into one or more partial character strings (blocks) using the processing unit 2 of the document search device, and the signature corresponding to each block is extracted, the number of characters in each block Is divided so as to be constant. This will be described with reference to FIG. 2. FIG. 2A shows a comparative example in which document data is divided by a certain number of characters, and FIG. 2B shows that the number of characters in each block is less than the appropriate number of characters. In addition, an example of the second embodiment in which the number of characters in each block is divided to be uniform is shown. As shown in FIG. 2A, when the number of characters in a block to be divided is determined to be constant in advance, the number of characters in the last block (block 4 in the figure) is changed to a block other than the last (block in the figure) for each document data. 1, 2, 3) or less. In such a situation, if the number of characters in an appropriate block is L, the number of characters in blocks 1, 2, and 3 is L1, and the number of characters in block 4 is L2, the magnitude relationship between them is as follows. May be a inequality sign.
L2 ≦ L ≦ L1

  In the case of the above inequality relationship, in the block signature, more than the necessary number of bits are allocated to the block 4, while the number of bits is insufficient for the blocks 1, 2, and 3. Invited false drops.

  In contrast, in the second embodiment, as shown in FIG. 2B, the document data is divided by equalizing the number of characters in each block so that the number of characters in each block is L or less, and the signature is Therefore, the above inconvenience can be solved and false drop can be reduced.

(Embodiment 3)
In the third embodiment, using the processing unit 2 of the document search apparatus, a signature is extracted from document data with many character types such as kanji, hiragana, and katakana like Japanese using the above-described N-gram method. At this time, instead of cutting out arbitrarily adjacent N characters (N is an integer equal to or greater than 2), arbitrarily adjacent N characters having the same character type are cut out to extract a signature. By doing this, the string "Ricoh's high-performance camera" in the previous example is
It is possible to cut out like Ricoh / High Performance / Camera. Compared with Ricoh / Cor // High / High / High Performance / Performance / Camera / Camera, Since there is no need to assign signature bits to extra cutouts such as (High / High) and (Performance / Capability), the probability of false drops can be greatly reduced. it can.

  As described above, according to the third embodiment, when the document data is extracted by using the N-gram method and the signature is extracted, only the same character type is extracted. Compared to the conventional case, it can be greatly reduced. Thus, if the types of character strings are sufficiently reduced, the signature length (number of bits) can be made to match the type. When such a signature is created, when the search character string is N characters, the bitmap to be scanned is only one bit slice, and no false drop occurs.

In the document search apparatus according to the third embodiment, if the search character string is “printer” and the 3-gram method is used,
The signature is submitted from a character string that is unlikely to be a search character string, such as pudding / printer / intermediate. In such a case, instead of using only the 3-gram method, 4- By using together with other methods such as the gram method and the 5-gram method, a suitable effect can be obtained as in the above case.

(Embodiment 4)
In the fourth embodiment, the processing unit 2 of the document search apparatus is used to search using a conditional expression including a plurality of search character strings connected by logical operators AND and OR, such as “high speed or printer”. Even in this case, the search speed can be increased. That is, first, a signature is extracted from all search character strings (patterns) included in the conditional expression, and second, a bit-sliced bitmap corresponding to a bit whose extracted signature value is 1 is used. By preparing a plurality of cursors to be scanned and thirdly scanning them while moving them in accordance with the contents of the conditional expression, the search result can be obtained by scanning the signature file 12 only once. It becomes possible. In addition, since the cursor is always scanned in the forward direction, there is an advantage that it can cope with a bitmap compressed by the MH (Modified Huffman) method or the like.

  FIG. 3 is a diagram showing the relationship between the search character string pattern, the cursor, and the bit-sliced bitmap. Here, the cursor has a one-to-one correspondence with the bit-sliced bitmap. That is, only one cursor is prepared even if there are a plurality of signatures extracted from the search character string included in the conditional expression having a value of 1 at a certain bit position. By doing so, even if the search character string included in the conditional expression increases, the bit-sliced bitmap can be scanned once.

  The cursor is associated with each search character string (pattern). In addition, the cursors are arranged in order of good search efficiency for each pattern. The order in which the search efficiency is good is an order in which the number of bits having a value of 1 is small in the bit-sliced bitmap according to the cursor. Thereby, search efficiency can be improved.

  In each search character string, the first cursor is called the primary cursor, and at the time of search, the primary cursor having the smallest bitmap position is selected, and the bit whose value is “1” is selected by the cursor. I will look for it.

  If it is assumed that signatures are found at positions indicated by the primary cursors after the primary cursors are moved, it is determined whether or not the document data from which the signatures are extracted satisfies the conditional expression.

  The conditional expression is converted into the conjunction standard form in advance. The conjunction standard form represents a form of a logical expression in which a literal is a disjunction (sum) of conjunction (product). Here, the literal is associated with one search character string. For example, the conditional expression “(high-speed OR high-definition) AND printer” is converted into the conjunction standard form “(high-speed AND printer) OR (high-definition AND printer)”. As described above, by setting the conditional expression in the conjunction standard form, it is possible to determine whether the entire conditional expression is true or false based on any truth of the conjunction.

  Also, when the primary cursor corresponding to a certain search character string reaches the end of the bit-sliced bitmap, the conditional expression is removed by removing the conjunction that uses the search character string corresponding to the primary cursor as a literal. It can be simplified. In other words, if it is found that the signature of the search character string “high performance” is not found, after that, the conditional expression may be searched as “high-speed AND printer”.

  Assuming that the corresponding signature is found at the position indicated by each primary cursor, and it is determined that the document data from which these signatures are extracted satisfies the conditional expression, the primary cursor for each search string For subsequent cursors, it is checked whether the value of the bit position is “1”, and it is determined whether the conditional expression is satisfied.

  As described above, search results are obtained while repeatedly moving the cursor. The details are shown in FIGS. 4 to 7 in the flowcharts. Each of these flowcharts is a flowchart of the cursor movement processing unit (FIGS. 4 and 5), the conditional expression settlement unit (FIG. 6), and the conditional expression expectation calculation unit (FIG. 7). And the conditional expression expectation calculation part. In each of these drawings, the search character string constituting the conditional expression is referred to as a pattern, and each of these patterns has position, checked, and match as attribute values. The initial value of position and checked is “0”, and “0” represents the virtual head position of the bit-sliced bitmap. The initial value of match is FALSE.

  For the function search of step 45 in FIG. 4, the cursor specified by the argument is advanced to the bit whose next value is “1”, and the bit position is returned. However, if a bit whose next value is “1” is not found, a negative value is returned. Thereby, in step 46, whether to go to step 47 or step 53 is determined depending on whether h is positive or negative.

  In these flowcharts, search results and candidate document data are represented by t. If t is indefinite, the value is null (step 49). The attribute value of t includes start and end, and represents the beginning and end of the signature storage position according to t.

  In step 68 in FIG. 5, a set of signatures corresponding to t is expressed as a family signature.

  As described above, in the processing unit 2 of the document search apparatus according to the fourth embodiment, when searching for a character string based on a conditional expression including a plurality of search character strings, all the search characters included in the conditional expression are searched. Prepare a cursor that scans a bit-sliced bitmap according to the bit whose bit is 1 in each signature value, and searches the cursor while scanning in parallel according to the contents of the conditional expression. Therefore, even if the number of search character strings included in the conditional expression increases, it is possible to obtain a search result by scanning the signature file only once, and the search speed can be increased.

Although the document search apparatus according to the present embodiment has been described with respect to the case of performing a document search, the present invention is not limited to this and can be applied to a more general search apparatus. For example, you can assign a bit to the signature to distinguish gender, blood type,
"Gender = Male AND Blood Type = AB AND (Home Prefecture = Saitama OR Home Prefecture = Yamanashi)"
By doing so, it becomes possible to process the contents of the conditional expression at high speed.

It is a block diagram which shows schematic structure of the document search device concerning this Embodiment. FIG. 10 is a diagram illustrating a document data division state when document data is divided and a block signature is extracted in the second embodiment. It is a figure which shows the relationship between the pattern of the search character string which concerns on Embodiment 4, a cursor, and the bit-sliced bitmap. 14 is a flowchart of a cursor movement processing unit according to the fourth embodiment. 14 is a flowchart of a cursor movement processing unit according to the fourth embodiment. It is a flowchart of the conditional expression liquidation part which concerns on Embodiment 4. FIG. 10 is a flowchart of a conditional expression expectation calculation unit according to the fourth embodiment.

Explanation of symbols

5 Document Search Processing Unit 8 Document Registration Processing Unit 9 Signature Extraction Processing Unit 12 Signature File

Claims (4)

In a document search device for searching document data including a search character string using a binary bit pattern signature,
A signature extraction processing unit for extracting a signature;
Document registration that divides the document data into blocks that are character strings of a predetermined number of characters, extracts a signature from the character strings that constitute each block using the signature extraction processing unit, and stores the signature as a block signature in a signature file A processing unit;
A partial character string having a predetermined number of characters is extracted from the search character string, a signature of the search character string is extracted from the partial character string using the signature extraction processing unit, and the signature based on the signature of the search character string is extracted. A document search processing unit for searching the block signature stored in the file;
With
When the search character string is a conditional expression including a plurality of search character strings connected by a logical operator, the document search processing unit extracts and extracts a signature from all the search character strings included in the conditional expression. Preparing a cursor associated with each search character string for scanning a bit-sliced bitmap according to a bit having a value of 1 for each signature, and setting the cursors in parallel according to the contents of the conditional expression Search while scanning
A document search apparatus characterized by that. The cursor is arranged in ascending order of the number of bits having a value of 1 in the bitmap sliced according to the cursor for each search character string.
The document search apparatus according to claim 1, wherein: The conditional expression is converted into a conjunction standard form that is a form of a logical expression in which a literal associated with the search character string is a disjunctive choice,
The document search apparatus according to claim 1, wherein: When the primary cursor corresponding to the search character string reaches the end of the bit-sliced bitmap, the conjunction that uses the search character string corresponding to the primary cursor as a literal is removed.
The document search apparatus according to claim 1, wherein:

Images