JP3596696B2 - Information retrieval device - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an information retrieval apparatus which performs search based on the search key, in particular wildcard search, of possible information retrieval apparatus of flexible search for partial matching or the like.
[0002]
[Prior art]
A hash method is well known as a high-speed search method in a perfect match search based on a key. However, it is considered that the hash method makes it difficult to perform a flexible search such as a wildcard search due to the data structure. The hash method has been proposed to suppress a reduction in search speed due to collision (for example, Japanese Patent Laid-Open No. Japanese Patent Application Laid-Open Nos. 271525 and 04-358266) and those proposed to speed up the apparatus by applying a hash method (for example, Japanese Patent Application Laid-Open No. 05-061910). No proposal for partial match search has been made at all, and hitherto, other methods such as full-text search had to be used in order to perform a flexible search with a search device using a hash method.
[0003]
[Problems to be solved by the invention]
The hash method requires at least a hash table in addition to the actual data. The hash table is approximately the size of the pointer length * N, where N is the number of words different from the keyword. Here, if another search method is used in combination for a partial match search, it is necessary to additionally have index data, and there is a disadvantage that the data size becomes very large. In addition, if a full-text search method that directly refers to actual data is used, no extra index data is required, but since all data is referenced, the search speed becomes extremely slow when the size of the actual data increases, and practical There is a disadvantage that the speed cannot be obtained.
[0004]
An object of the present invention is to provide a search device capable of performing a flexible search such as a wildcard search without adding a data structure or referring to all data in a hash method in order to solve these problems. I do.
[0005]
[Means for Solving the Problems]
In the information registration method and the information search method according to the present invention, a hash function for limiting a range of a hash value according to information of each element constituting a search key of a partial match search is used, and all hash values belonging to the limited range are used. A means for performing a search on a value is provided. The search for a plurality of hash values may be configured to perform the search repeatedly, or may be configured to perform the search by parallel calculation.
[0006]
[Action]
At the time of data registration, a hash function that guarantees that a hash value falls within a range corresponding to an element constituting a key is used. Then, in the partial match search, when a partial key is given, a hash function obtains a range corresponding to a component of the partial key. Subsequently, a search is performed using all the hash values in this range. Since a key including a given partial key is not registered outside this range, it is not necessary to search for extra data and no additional data structure is required. In this way, in the hash method, not only a perfect match search but also a partial match search can be realized.
[0007]
That is, when the hash value for referencing the hash table when performing a search is, for example, 0 to 15, all records can be searched by searching for all the numbers from 0 to 15. If the range of is limited to, for example, an odd number, the retrieval amount is halved. By devising the hash function in this manner, a partial match search can be performed.
[0008]
BEST MODE FOR CARRYING OUT THE INVENTION
FIG. 1 shows an example of a simplified hash function for explaining the principle of the hash search method of the present invention. FIG. 2 shows an example of performing a search using the hash function of FIG.
[0009]
In the example of FIG. 1, it is assumed that the elements constituting the search key are A, B, C, and D. As a hash function, one bit of a hash value is assigned to each of A, B, C, and D. When "A" is present in the key element, the bit field A is set to 1 and " If “A” does not exist, a function for setting the bit field A to 0 is used. This means that the hash value when "A" is present is between 8 and 15, and the hash value when "A" is not present is limited to a value between 0 and 7. The same effect is obtained even if 0 and 1 are reversed. If the range of the hash value can be similarly limited, it is not necessary to perform the bit operation. In the data registration and perfect match search, this operation is performed on all of the bit fields A, B, C, and D. For example, the hash value for “ABC” is 1110, that is, 14 in binary. Subsequently, as shown in FIG. 2, the hash table is referred to, and the data “Sashisuse Soso ...” corresponding to the hash value 14 is searched.
[0010]
In the example of FIG. 1, “AB *” represents a partial key, and indicates that a search is performed for all keys in which A is included in a part of the key and B is included in a part of the key. This is a condition that “A” exists in the search key and “B” exists in the search key. In the example of the hash function in FIG. 1, the hash value when “A” exists is The property that the value is always between 8 and 15 and the hash value when “B” exists is always either a value between 4 and 7 or a value between 12 and 15 can be used. When the logical product of these two conditions is calculated, it is understood that the hash values corresponding to “AB *” are four, 12, 13, 14, and 15.
[0011]
In the operation for obtaining the actual hash value, the bit fields for the determined “A” and “B”, that is, the bit fields A and B are set to 1, and the bit fields for the undetermined elements are all combinations of 0 and 1. Just fine. As a result, four values of 12, 13, 14, and 15 are obtained. For each of them, as shown in FIG. 2, by referring to the hash table, data such as "Aioueo ...", "Kakikukeko ...", "Sashisusesoso ...", and "Tatsutsuto ..." are retrieved. .
[0012]
Hereinafter, an operation using an embodiment in which the hash search method of the present invention is applied to an electronic dictionary will be described. FIG. 3 is a configuration example of an electronic dictionary to which the hash search method of the present invention is applied. FIG. 4 is a configuration example of a data structure for applying the hash search method of the present invention to an electronic dictionary. FIG. 5 shows a schematic flow of the operation.
[0013]
The electronic dictionary shown in FIG. 3 includes a CRT 21 for displaying various data on a display screen, a CRT driver 22 for controlling display on the CRT 21, a keyboard 23 for inputting commands, character strings, numerical values, and the like. A mouse 24 which is a pointing device, a keyboard / mouse driver 25 which outputs various data by a user's operation of the keyboard 23 and the mouse 24, a disk device 26, a disk device driver 27, a main storage device 28, a CPU (central processing unit) Device 29).
[0014]
The disk device 26 is a secondary storage device for storing a large amount of data, and stores a chained index, a real data file, and the like to be described later. The input and output of data to and from the disk device 26 is controlled by a disk device driver 27.
[0015]
The main storage device 28 stores an application program, data such as characters and numerical values input from the keyboard 23 and the mouse 24, and a hash table described later. Further, a program for realizing a flowchart described later is also stored.
[0016]
The CPU 29 is a circuit that controls the entire system and performs arithmetic processing on predetermined data based on various commands, and executes data search processing based on a flowchart described later. Actually, the CPU 29 executes the processing of the flowchart in accordance with the program stored in the main storage device 28. The basic configuration and operation for realizing this search processing are the same as those disclosed in Japanese Patent Application Laid-Open No. 05-28194.
[0017]
FIG. 4 shows an outline of a data structure in the electronic dictionary. The data structure of FIG. 4 is basically composed of a hash table 11, a chained index 12, and a real data file 13.
[0018]
In the electronic dictionary of the present embodiment, as shown in FIG. 4, in addition to a search from ordinary headwords (“a”, “A”, etc.), the meaning of a word (“one”, It is possible to search from katakana notation (“e”, “a”, etc.) of pronunciations, etc. Therefore, the number of keys / records is very large, and records for a certain key are also Not unique.
[0019]
First, the basic operation of a search by the hash method with a chain in the electronic dictionary will be described.
[0020]
First, the hash table 11 holds, for example, 3 bytes of a pointer from the address indicated by the hash value h (k) of the search key k to the chained index 12. If the corresponding key has not been registered, FFFFFFH is held. However, H means hexadecimal notation. That is, FFFFFFH indicates that all 24 bits are "1".
[0021]
Next, details of the chained index 12 will be described. The chained index 12 is a set of index records having information corresponding to data records on a one-to-one basis. FIG. 6 shows the structure of the index record. The index record is composed of a pair of chain pointers _{1 to n} (CP _1-n ) and key identifiers 1 to n (KD _1-n ) related to all keys set in the record, and a pointer to the data file (data Pointer (DP)). Note that the key identifier is a copy of the registered word itself.
[0022]
The chain pointer is a pointer to the next index record for the corresponding key, and a list of registered keys (data connection) having the same hash value is formed by collision.
[0023]
Here, the collision means that different search keys have the same hash value by the hash function. As a solution in this case, the present embodiment employs a chain pointer.
[0024]
The head of each list is directly pointed to from the hash table 11, and when there is a collision, each pointer holds the address of the next chain pointer, as shown in FIG. Is stored. The chain pointer is represented by, for example, 3 bytes and can take a value from 000000H to 7FFFFFH.
[0025]
The key identifier exists immediately after the chain pointer, and is used for detecting whether or not the input key corresponds to the registration key, that is, detecting collision of hash values. The description rule of the key identifier in this embodiment is shown below.
[0026]
(I) In a normal search key, the character string of the registration key is stored after 81H indicating the start of the key identifier, and then 82H indicating the end of the key identifier is stored.
[0027]
(Ii) If the registration key is the same as that immediately before the chain, the description is omitted. If omitted, the next chain pointer or delimiter is stored at the position where the key identifier should be. Since the first byte of the chain pointer is 7FH at the maximum and the delimiter is FFH, even if omitted, it can be clearly distinguished from 81H with a key identifier.
[0028]
By introducing a key identifier in this way, it is not necessary to refer to actual data for checking a collision. A field for storing a keyword in the actual data may be provided, and a collision may be checked by referring to the keyword field of the actual data.
[0029]
In the data pointer, an address indicating the head of the actual data record in the actual data is stored in 3 bytes following the delimiter FFH. When the data pointer is FFFFFFH, it indicates that the data record has been deleted.
[0030]
The actual data file 13 is a set of data records, and the data records have the following format.
[0031]
(Headword) (Header break [NULL]) (Contents) (Record break [LF: Line Feed])
When 80H is used as the key identifier as described above, the key is identified by referring to the headword of this data record. The actual data file is referred to only when the key identifier 80H of the chained index file is used.
[0032]
The actual data file 13 (content part) is an actual dictionary description part. However, in this embodiment, information of a search key is not included by a key identifier. Any text is fine. Here, “flat” means data consisting only of a character code that does not include a field delimiter symbol. The data record is a single text file.
[0033]
Next, an algorithm of data search using the above-described electronic dictionary will be described in detail with reference to the flowchart of FIG.
[0034]
First, after initialization (step 101), the hash value h of the search key is obtained, and the contents of the position h of the hash table are read as the index position ip (step 102). Note that the initialization here means that a flag and the like used internally are initialized, and a buffer of the key identifier is also cleared. Next, it is determined whether ip = FFFFFFH (step 103). Here, if ip = FFFFFFH, it is known that the key is an unregistered key, and the process is terminated. When ip = FFFFFFH is not satisfied, 3 bytes from the position ip on the chained index 12 are read from the file of the chained index as the chain position cp (step 104), and the value of the next byte of the chain position cp (key identifier It is determined whether or not (storage position) ≧ 80H (step 105).
[0035]
In the case of less than 80H or FFH, the previous one is used because it is the same as the one before the chain. If it is 80H or more, a keyword is read (step 106). Thus, the collision is determined. Next, it is determined whether or not the current index record corresponds to the key based on kr (step 107). If so, skip to FFH on the chained index 12 and use the next 3 bytes as the data pointer dp. Read (step 108). The determination in step 107 is made by calculating the logical sum of the lower byte of the input character string for which the hash value has been obtained and 80H, and determining whether or not it matches the key identifier. Subsequently, it is determined whether or not dp = FFFFFFH (step 109). Here, if dp = FFFFFFH, there is a data record, so from the position dp on the data file to 0AH (= LF) is added to the list of results according to the format of the data record (step 110). Next, it is determined whether or not cp = 0 (step 11l). If cp = 0, the process is terminated. Otherwise, since the chain is continuing, cp is substituted into ip (step 112), and the process proceeds to step 104. Return.
[0036]
Next, an operation in the case of performing a partial match search in an electronic dictionary performing a search by the hash method with a chain as described above will be described.
[0037]
In the present embodiment, the remainder when the appearance position (the first character, the second character,...) Of the characters constituting the keyword is divided by 3, and the code (character code) of the characters constituting the keyword are represented by 5. It is assumed that a hash value is obtained from the remainder when divided. Specifically, a 15-bit hash value is divided into a 5-bit field of a (3n + 1) th character, a (3n + 2) th character, and a (3n + 3) th character, where n is an integer of 0 or more, The code is x, and each 5-bit value is the logical sum of 2 raised to the power of (xmod5) (2 ^xmod5 ). Note that xmod5 represents a remainder when divided by 5 of x.
[0038]
FIG. 9 shows an example of calculating the hash value of the character string “abcd” by this method. In a perfect match search and data registration, this value is used as it is as a hash value.
[0039]
In the example shown in FIG. 9, the remainder obtained by dividing the character code 61H (hexadecimal) (97 in decimal) of the first character 'a' by 5 is 2, which is 2 ² = 4. Therefore, field 1 is ORed with 00100 (binary number) (4 in decimal number) and the original value of field 1 00000 (binary number), and becomes 00100 (binary number). Similarly, the character code of the fourth character 'd' is 64H (100 in decimal), and the remainder obtained by dividing the character code by 5 is 0, so that 2 ⁰ = 1. Therefore, the logical sum of 00001 (binary number) and field 1 (00100) is obtained. As a result, field 1 becomes 00101 in binary.
[0040]
In this embodiment, as described above, an identifier indicating information about a key is added to the chain pointer in the index with chains in FIG. 4, and a check at the time of collision is performed using the key identifier.
[0041]
In the partial match search, the part where the bit is 1 is fixed at 1 and the part where the bit is 0 is set as a combination of both 0 and 1, and a set of hash values is obtained and all parts are obtained. Perform a search. When using the appearance position of a character, it is necessary to pay attention to the start position of the character string. For example, in the case of “* abcd *”, the appearance position of “a” is the (3n + 1) th character such as “abcde”, the (3n + 2) th character such as “zabcd”, and “yzabcd” ( Since there are all cases of the (3n + 3) -th character, three patterns shifted by one character are required.
[0042]
FIG. 5 shows a schematic flow in the case of performing a partial match search using a hash function.
[0043]
First, the hash value of the given search key is obtained (step 151), and the contents of the corresponding position in the hash table are read (step 152). If a key identifier is added to the chain pointer, a collision is detected based on the key identifier (step 153), and the search result is stored (step 154). Steps 152 to 154 are repeated until the search of all hash values is completed (step 155). When the search of all hash values is completed, the search result is output and displayed (step 156).
[0044]
Next, FIG. 10 shows a detailed flow when a partial match search is performed using a hash function. The operation will be described taking a case where the partial key is “abcd” as shown in FIG. 9 as an example. Here, the part where the bit is 1 is called a “fixed bit”, and the part where the bit is 0 is called an “undefined bit”.
[0045]
S1: First, when a partial key is given, the character start position of the partial key is initialized. That is, the character start position is set to the (3n + 1) th character first. Since the hash value is irrelevant to n, n = 0 and the start position = 1.
[0046]
S2: Subsequently, a bit fixed by the partial key is calculated. When the start position is 1, as shown in FIG. 9, bit 2 of field 1, bit 0 of field 1, bit 3 of field 2, and bit 4 of field 3 are fixed bits.
[0047]
S3: Next, the fixed bit string is stored as a fixed bit string list. This is because when a search is performed with the start position of a character shifted, combining fixed bits and undefined bits may result in the same hash value as already searched. This is to save the trouble of searching twice.
[0048]
S4: Next, all the undefined bits are set to 1 as initial values. That is, all bits except for bit 2 of field 1, bit 0 of field 1, bit 3 of field 2, and bit 4 of field 3 are set to 1. Hereinafter, the indefinite bit is treated as one of the number of bits excluding the number of fixed bits. Thus, for example, a combination of 11 bits of 0 and 1 excluding 4 bits that are fixed bits from 15 bits can be covered by a binary number from 0 to 2047 (decimal number).
[0049]
S5: Connect the undefined bit and the fixed bit. That is, a fixed bit is inserted between a binary number representing an indefinite bit. FIG. 11 shows an example in which an undefined bit and a fixed bit are connected. That is, the indefinite number of bits created in S4 or S7b described later is placed at the original bit position and fixed bits are inserted.
[0050]
S6: If the hash value generated by the concatenation becomes equal to or smaller than the minimum value, the process jumps to S12. Here, since there is no possibility that the search key does not have any single character, it is impossible that there is no bit in the field of the first character that has any one. Therefore, the search is started from a large numerical value, and the time when the field of the first character becomes 0 is set as the minimum value, that is, the end condition for obtaining the hash value.
[0051]
S7: Compare the fixed bit list up to the previous character start position stored in S3 with the hash value, and if all the bits of the hash value at the same bit position as the fixed bits of the fixed bit list are 1, Since it is included in the hash value pattern searched previously (S7a), the next undefined bit is obtained (S7b), and the process returns to S5. Note that the determination in S7a is a determination for preventing a redundant search from occurring because the already searched combination may appear due to the combination of the undefined bits 0 and 1. In S7b, a value obtained by subtracting 1 from the numerical value represented by the undefined bit string is newly set as the undefined bit string.
[0052]
S8: Read the hash table with the obtained hash value.
[0053]
S9: If there is no registered word in the index (S9), since there is no corresponding data record, the undefined bit is changed by the same processing as S7b to obtain the next undefined bit (S7b), and the process returns to S5.
[0054]
S10: Whether or not a partial key is included is compared based on the key identifier, and if a partial character string is included, it is registered in a search result list (S10a).
[0055]
S11: Skip to the next chain pointer in the next index, and return to S9.
[0056]
S12: The character start position is shifted by one.
[0057]
S13: If the character start position is 3 or less, return to S2. This is for searching for all of the cases where the character start positions are 1, 2, and 3.
[0058]
S14: The search result registered in the result list is displayed.
[0059]
With the above procedure, the partial match search can be performed even in the hash method. Furthermore, in the case of a forward match search, only the case where the character start position is 1 needs to be searched. Further, in the case of the backward matching search, as shown in FIG. 12, when performing the determination based on the key identifier, the search may be performed from the backward character. In addition, in the case of front-back matching, the partial key of the front matching is set to the character start position, and the partial key of the back matching is shifted to obtain a hash value and search. Similarly, a flexible search such as "* A * B *" can be easily realized by setting the wildcard to *.
[0060]
Although the present embodiment is configured to repeatedly search for each hash value, each hash value may be searched in parallel using a computer capable of parallel calculation. Absent.
[0061]
【The invention's effect】
As described above, according to the present invention, in the hash method, which is a high-speed search method, a flexible search such as a wild card search cannot be performed only by an exact match search, a data structure is added to all data. Partial search can be performed without referring to. Further, in the search method of the present invention, all partial match searches such as front match, back match, front / back match, intermediate match, etc. can be realized without adding a data structure.
[Brief description of the drawings]
FIG. 1 is an explanatory diagram showing an example of a hash function capable of performing a partial match search.
FIG. 2 is an example showing the principle of partial match search in the hash method.
FIG. 3 is a block diagram showing a configuration example of an electronic dictionary to which a hash search method according to the present invention is applied.
FIG. 4 is an explanatory diagram showing an example of a data structure for applying the hash search method of the present invention to an electronic dictionary.
FIG. 5 is a schematic flowchart of a partial match search of an electronic dictionary to which the hash search method of the present invention is applied.
FIG. 6 is a diagram showing a structure of an index record.
FIG. 7 is a diagram showing a list of chain pointers in an index record.
FIG. 8 is a flowchart illustrating an algorithm of data search using an electronic dictionary.
FIG. 9 is an explanatory diagram showing an example of obtaining a hash value using an example of a hash function capable of performing partial match search.
FIG. 10 is a detailed flowchart showing an example of implementing a partial match search.
FIG. 11 is an explanatory diagram showing an example of obtaining one of a plurality of hash values corresponding to a partial key.
FIG. 12 is an explanatory diagram illustrating an example of a key identification method in a backward match search.
[Explanation of symbols]
11 hash table, 12 index with chain, 13 actual data file, 21 CRT, 22 CRT driver, 22 keyboard, 24 mouse, 15 keyboard / mouse driver, 26 disk device, 27 disk device Driver, 28: Main storage device, 29: CPU

Claims (2)

Input means for inputting a search key which is a word to be searched;
The hash value is divided into a plurality of fields, which are partial areas, a search key input by the input unit is received, and a partial hash value of the size of the field based on the character code of each character constituting the search key Calculating means for storing a partial hash value in a corresponding field based on an appearance position of each character, limiting a possible value of the hash value, and obtaining a set of one or more hash values;
An information search apparatus comprising: a search unit configured to perform a search based on the input search key with respect to one or a plurality of sets of hash values obtained by the calculation unit. The arithmetic means sequentially stores partial hash values of each character constituting the input search key in association with a predetermined number of fields which are partial areas of the hash value. When the number of characters exceeds the number of the fields, when storing the partial hash value of the excess characters, the position of the field to be stored is obtained by the remainder of the appearance position of each character with respect to the number of the fields, and the obtained field 2. The information retrieval apparatus according to claim 1, wherein a logical sum of the hash value and the hash value already stored in the file is taken and the data is stored again .

Images