JPH09212524A - Entire sentence retrieval method and electronic dictionary formation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a retrieving time and to keep the retrieving time constant concerning any character string. SOLUTION: An index is expressed by the set of the low-order bytes (LOW vector) and the set of the high-order bytes (HIGH vector) of the code of each character in a character string. At the time of retrieving an optional retrieval character string, a first character is collated first (steps S1 and S2). When there is no index, the non-existence of a pertinent place is judged at this point of time. Next, the set of the low-order byte and the high-order byte of the retrieval character string is prepared and collation with the LOW vector is executed first (steps S3 and S5). When the LOW vector is not pertinent, the non-existence of a pertinent place is judged. When the LOW vector is pertinent, collation with a HIGH vector is executed (steps S6 and S7). When the HIGH vector coincides and there is information of the pertinent place in the HIGH vector, the pertinent place is fetched and retrieval is finished.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a full-text search method for searching a character string from a search document and a computerized dictionary device for searching registered words in a computerized dictionary.

[0002]

2. Description of the Related Art With the spread of computers and word processors, a large amount of documents are being digitized, and C
With the spread of electronic publication by D-ROM, electronic mail, and news, such information has become available. Thus, as the digitization of information progresses and the amount of digitized documents handled increases, various means for efficiently and easily handling these are required. One of them is information retrieval technology that quickly finds only what is needed from scattered documents. There is a great need for this technology because it is possible to make better use of the resources of electronic documents and improve work efficiency. In order to cope with such needs, a full-text search system technique has been attracting attention instead of a keyword search method in which a keyword assigned to a document is used for a search.

In addition, as a large number of documents are digitized,
The opportunity to read electronic documents has increased, and the work of document editing on computers has become more frequent. Therefore, there is a need to look up a dictionary on a computer.

[0004]

The simplest idea of the above full text search system is to find the search pattern string by scanning the document sequentially. Various high-speed character string collation algorithms have been researched and developed so far, but the speed becomes slower as the amount of data increases due to sequential scanning, which is not suitable for a large-scale search system.

Therefore, in order to speed up the search, a full-text search method in which a search auxiliary file is generated and a search is performed using the search auxiliary file is drawing attention. The implementation using this auxiliary file is roughly divided into two types. One is a method of creating an index by extracting words and phrases like the index of a book, and further research is underway to compress the index size by performing secondary processing of the index.
This method is very effective when words such as English are written separately, but in the case of Japanese, first,
Morphological analysis is required to extract words. However, since morphological analysis is ambiguous and requires manual checks, it is difficult to generate a complete system automatically. Further, since it is possible to search only by the word / phrase registered in this method index, there is a possibility that even an existing character string cannot be found depending on the specified method. For example, if a compound noun, such as a compound noun, is index-registered with each noun, it will not be found even if the compound noun is specified during the search.

On the other hand, the second method is a method in which a document is simply regarded as a list of characters and any character string can be searched. In this method, the index size tends to be larger than in the above method, but there is no unnaturalness in the search such as a sentence search, and there are few search omissions. Therefore, the user can search with a conceived word and phrase, and can solve the above-mentioned problem of compound nouns. However, in some conventional systems, a tree structure is used for the index, but the depth of the tree affects the search time, and it is difficult to constantly maintain the search speed. From such a point, it has been desired to realize a full-text search method capable of keeping the search time constant.

Further, the conventional electronic dictionary is a so-called Japanese syllabary dictionary, which is a method of registering words, in which codes are simply sorted according to size like a Japanese dictionary, or sorted by hashing. Many of them use a method similar to the pulling method, and by matching this with a character string, the meaning was taken out. However, with such a classification method, words that start with the same character string as "electronic *", such as "electronic computer" and "electronic dictionary", are classified close to each other, and it takes time to identify them. I will end up.
From this point of view, it has been desired to realize a digitized dictionary device in which registered words are taken out quickly and the search time is constant regardless of the words.

[0008]

The present invention employs the following structure to solve the above-mentioned problems. <Structure of claim 1> The index of the character string appearing in the search document is represented by a set of lower byte and upper byte of the code of each character in the character string, and in the search document of the character string corresponding to this set. When the search is performed with an arbitrary search character string by providing the information indicating the corresponding place of, the set of the low-order bytes and the set of the high-order bytes of the code of each character in the search string are created, and the index is calculated from these sets. It is characterized by referring to and extracting the corresponding place in the search document.

<Explanation of Claim 1> Looking at the constituent elements of a Japanese character consisting of 2 bytes, the upper byte has information representing the character type, and the lower byte is assigned as a code representing the character itself. For example, the double-byte character "mo"
When considering 2 bytes in “J”, it is considered that the character codes 2562 and 2528 are compared with each other, and the lower byte and the upper byte are connected to each other.
With 25, the corresponding element can be narrowed down faster. Also, when we check the frequency of use of characters, hiragana
The number of katakana is overwhelmingly large, and even when considering kanji, the frequency of use in actual documents is second for JIS level 1 kanji.
Overwhelmingly more than level kanji. Therefore, when considering a combination of character strings, the upper bytes of each character are often common.

Therefore, the invention of claim 1 pays attention to the lower byte that contains the character features, and creates the character string as a lower byte set and an upper byte set of four characters, respectively.
Take a method to proceed with the search. Therefore, the distribution of combinations can be dispersed, the candidates can be narrowed down quickly, and the search speed can be improved.

It should be noted that the reason why the four characters in the character string are used is that when considering a general document, it is possible to narrow down considerably if there are four characters, and it is also appropriate as a processing unit of a device that executes a search. However, the number of characters is not particularly limited, and the number of characters can be appropriately set.

<Structure of Claim 2> In the full-text search method according to Claim 1, first, a search is performed by a set of lower bytes of a code of each character in a search character string, and then a set of higher bytes. It is characterized by that.

<Explanation of Claim 2> In the invention of Claim 2, for example, in the case of "mo" and "di", the lower byte 6238 is inspected first. If there is no matching set of low-order bytes in this check, it is possible to obtain a search result with no corresponding place at that time. That is, although there are innumerable combinations following "mo", it is considered that there are actually few character strings starting with the character code string 6238 of the lower byte.
By inspecting 238 first, these two characters are almost fixed in one inspection.

<Structure of claim 3> In the full-text search method according to claim 1 or 2, the search first refers to the index with the first character of the search character string, and then refers to the index with the remaining character string. It is characterized by doing.

<Explanation of Claim 3> In the invention of Claim 3, the first character of the character string is hashed, a new character string is generated from the remaining character string, and the relevant candidates are narrowed down. Therefore, the search process can be performed faster than inspecting a combination of a plurality of characters from the beginning. In this case,
The first character of the character string is usually the first character of the character string, but it is not particularly limited to this and may be a specific character such as the last character.

<Structure of Claim 4> In the full-text search method according to any one of Claims 1 to 3, the index has a tree structure whose root is the first character in the character string appearing in the search document. It is a feature.

<Explanation of Claim 4> In the invention of claim 4,
Since the index has a tree structure and the second and subsequent characters are a set of low-order bytes and high-order bytes,
For the nodes below the root, the unevenness of the tree depth is distributed,
In addition, there are fewer branches to finalization. as a result,
The processing of tracing the index tree is reduced, and the search speed can be improved.

<Structure of claim 5> In the full-text search method according to claim 3 or 4, when a LOW vector of an index, which is a set of lower bytes of a character string appearing in a search document, is referred to, first one It is characterized in that reference is made by characters and then the remaining character strings are referenced.

<Explanation of Claim 5> In the invention of Claim 5, as in Claim 3, even in the LOW vector, the first character in the vector, that is, the second character in the search character string is hashed and the rest is left. A new character string is generated with the character string of and the applicable candidates are narrowed down. In the tree structure, the number of elements of the LOW vector especially for the second to fifth characters is expected to be large.
Therefore, by using the second character of the search character string as a hash value, access can be speeded up, and the search processing can be performed faster than inspecting a combination of a plurality of characters in the LOW vector.

<Structure of Claim 6> A true meaning registration database having a meaning of a registered word and information on the length of a character string of the meaning, and a character code of the registered word registered in the true meaning registration database. When a search word is given, an index that is a set of low-order byte and high-order byte is provided, and a set of low-order bytes and a set of high-order bytes of the code of each character in the search word are created. And a search unit that retrieves a registered word corresponding to the search word.

<Explanation of Claim 6> In the invention of claim 6, when the registered word registered in the true meaning registration database is taken out, the same full-text search method as that of the invention of claims 1-5 is used. When an arbitrary search word is given, the search unit creates a set of lower-order bytes and a set of higher-order bytes of the search word, and collates the set with the index.

Therefore, candidates can be narrowed down quickly, the search time can be shortened, and any word can be searched at a uniform speed. In addition, since the length information is provided in correspondence with the registered words in the true meaning registration database, it is possible to grasp how much data is present when transferring or displaying the search results, for example. Access can be done quickly.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. First, the principle of the full-text search method and the electronic dictionary device of the present invention will be described. The characters used in Japanese documents are about 10,000 words even with only the JIS code, and Japanese characters are composed of a combination of many of these characters. Therefore, even if the character string is not very long, it can be said that it is a part of a huge combination in terms of probability. Therefore, if the pattern of this character combination is used as a clue, it can be considered that it is easy to specify or narrow down the part to be obtained from the data. Then, when introducing the idea of a combination of character strings in such character string matching, it is desirable that the probability distribution of the combination is uniform without being biased.

Therefore, as a method for speeding up the narrowing,
Take advantage of the relationship between two characters. For example, this will be described by taking a combination with the double-byte characters "MO" and "J" as an example. Each character code is as follows. Mo: 2562 J: 2538

When considered with 2 bytes, 2562, 2528
Rather than seeing that, 6238 and 2525, in which the lower byte and the upper byte are respectively connected, can narrow down the corresponding element faster. That is, looking at the constituent elements of a Japanese character consisting of 2 bytes, the upper byte is the character type (Hiragana,
Information such as katakana), and the lower byte is assigned as a code representing the character itself.

When the frequency of use of characters is examined, the number of hiragana and katakana is overwhelmingly higher than that of kanji. For example,
There are innumerable combinations following "mo", but the lower byte 6
By inspecting 238 first, these two characters are almost fixed in one inspection. That is, the character code string 62 of the lower byte
It seems that there are actually few strings starting with 38.

Regarding the kanji, the frequency of use in actual documents is higher in JIS 1st level kanji than in JIS 2nd level.
It is overwhelmingly higher than the standard Kanji. Therefore, even in the case of Kanji, by paying attention to the lower byte than the upper byte, the distribution of combinations can be dispersed, and the candidates can be narrowed down at the first stage.

As described above, the present invention pays attention to the low-order bytes that contain the character features in this way, and creates a character string as a low-order byte set and a high-order byte set of four characters, and takes a method of proceeding with the search. The reason why four characters are used here is for the following reason.

That is, as described above, the larger the number of characters in the character string, such as two characters rather than one character and three characters rather than two characters, the stricter the condition for narrowing down the character string (information for narrowing down). Will increase). Therefore, considering a general document, it is possible to narrow down considerably if there are four characters, and it is also appropriate as a processing unit of the device that executes the search. is there.

<< Specific Example 1 >> FIG. 1 is a flow chart showing a specific example 1 of the full-text search method of the present invention. Prior to this, an index for realizing this specific example will be described.

FIG. 2 is an explanatory diagram of indexes for realizing the full-text search method. As shown in the figure, the index of this specific example is realized by a tree structure. In the figure, 1 is a leading character hash table, L1 and L2 are L
The OW vectors, H1 and H2, are HIGH vectors.

The leading character hash table 1 is a one-dimensional array of 64 Kbytes, which is used to refer to the leading character code of each character string as an index, and has an area for all character codes. When one or more corresponding character strings are present in the search document in each character area, the LOW vector L1 of the second and subsequent characters of the corresponding character string.
If there is information to follow, and the character string that has that character code at the beginning does not exist in the search document,
Contains NULL.

The LOW vectors L1 and L2 are 4k + 2, 4k + 3, 4k + 4, 4 in the corresponding character string in the search document.
It is a set consisting of a permutation of the lower 1 byte of the k + 5th character. Note that k ≧ 0 here. Also, the LOW vector L
In 1 and L2, a set of lower bytes for 4 characters and information (pointers) to the following HIGH vectors H1 and H2 are entered.

Further, in the tree structure, especially 2 to 5 of the character string are
It seems that the number of elements of the first LOW vector L1 for the character is large. In order to find the corresponding element earlier than this, hashing is performed by using the code of the second character as a hash value to speed up access.

FIG. 3 is an explanatory diagram of this LOW vector.
That is, the internal structure of the LOW vector is similar to the relationship between the leading character hash table 1 and the LOW vector L1 shown in FIG. 2, and the leading character (= 4k + 2) in the LOW vector L1.
The table A is configured with the (character) as a hash value. The lower byte of 4k + 3 to 4k + 5 characters is the vector B.

The HIGH vectors H1 and H2 are 4k + 2, 4k + 3, 4k + 4, 4k + 2, 4k + 3, and 4k + 4 in the corresponding character string in the search document.
It is a set consisting of a permutation of the upper 1 byte of the 4k + 5th character. Note that k ≧ 0 here. In addition, in the HIGH vectors H1 and H2, together with a set of upper bytes for these four characters, and when a corresponding character string further follows, as identification information D, information (pointer) to the LOW vector of the remaining character string However, when the location of the corresponding character string is confirmed by the 4k + 5th character, the information of the relevant location is entered.

Next, the full-text search method of this example will be described with reference to the flowchart of FIG. 1. The first character of the input character string S (n) of length n is taken out and the leading character hash table 1 is referred to by that character code (step S1). Here, the search character string S (n)
As an example, the character string is “American football”.

Since the first character of "American football" is "A", the first character hash table 1 is referred to check whether it exists (step S2). That is,
If the value at the position of "a" is a value larger than 0 (a value that is not NULL), it is the value of the LOW vector L1, so the LOW vector L1 is set as the current LOW vector, and k
= 0 (step S3). On the other hand, if it is 0, it means that the character string S (n) does not appear in the search document, and the search ends.

2. From character string S (n), 4k + 2,4k +
Take out the 3rd, 4k + 4, 4k + 5th character and divide into the lower byte and upper byte of each character,
Store each in search vectors L and H (step S
4). The search vectors L and H are a vector for searching the LOW vectors L1 and L2 and a vector for searching the HIGH vectors H1 and H2, respectively. In this case, the lower byte and the upper byte of "Merikan" are used. Is equivalent to Here, the code of each character is M = 2561, L = 2
Since 56A, f = 252B, and h = 2573, search vector L = 616A2B73 and search vector H = 2525.
2525.

3. The current LOW vector L1 vector and the search vector L are matched (step S5). That is, a set of lower bytes of "Melican""616A2B7"
3 ”is present in the LOW vector L1. If it exists, the information for the HIGH vector H1 is obtained, and it is set as the current HIGH vector (step S
6). On the other hand, if there is no equivalent, the character string S (n)
Indicates that it does not appear in the search document, and the search ends.

4. Matching between the current HIGH vector H1 and the search vector H (step S7). That is,
Similar to the matching with the LOW vector L1 described above,
A set of high-order bytes of "Merikan""25252525"
Is present in the HIGH vector H1. If there is no match as a result of the search, it indicates that the character string S (n) does not appear in the search document, and the search ends. Also,
If there is an equal value, the identification information D is obtained,
This identification information D is taken out (step S8).

5. The content of the identification information D is the LOW vector L2.
If it is a pointer to (step S9), LOW
Vector L2 is the current LOW vector, and k = k
+1 as above 2. (Step S10). In this case, if there are multiple "American ..." in the search document, the corresponding place is not specified by the fifth character, so
Further, a pointer to the LOW vector L2 is stored. Therefore, this time, the search vectors L and H are set to 6 of the character string.
The above search is performed as a set of the lower byte and the upper byte of "Football" which is the 9th character.

6.6 to 9th character search results in HIGH
If the identification information D extracted from the vector H2 indicates the corresponding place, the information of the corresponding place is taken out (step S11), and the search ends. In this case, the search character string S (n) has the remaining character strings, but since the location in the search document is specified up to the above-mentioned "football", the search ends. That is, if it is found in the front partial match of a character string that only one combination of the character strings exists, the character string is identified.

As described above, according to the first specific example, since the depth of the index tree can be reduced, it is possible to search with a small number of branches. As a result, the time required for branching can be reduced,
The search speed can be improved.

<< Specific Example 2 >> FIG. 4 is a block diagram of an electronic dictionary device which is a specific example 2. In FIG. The device shown in the figure comprises a search unit 10 and a true meaning registration database 20. Search unit 10
Is a search using the full-text search method of the specific example 1 described above, and the true meaning registration database 20 has information on the meaning part of a registered word as a dictionary and the length of a character string of that meaning. Is a database for storing.

FIG. 5 is a diagram showing an index structure of the electronic dictionary device. The index of this specific example has the same basic structure as the index of the specific example 1 shown in FIG. That is, the leading character hash table 1 is a 64-Kbyte one-dimensional array for referencing the leading character code of a registered word as an index, and areas for all character codes are provided. Then, in each character area, one or more registered words have a true meaning registration database 2
If it is within 0, LOW after the second character of the registered word
If the information for continuing the vector L1 is included, and if the registered word having the character code at the head does not exist in the true meaning registration database 20, NULL is included.

The LOW vectors L1 and L2 are 4k + 2 and 4k + in the registered words in the true meaning registration database 20.
It is a set of permutations of the lower 1 byte of the 3rd, 4k + 4, 4k + 5th words. Note that k ≧ 0 here. In addition, the LOW vectors L1 and L2 include a set of lower bytes for four characters, followed by HIGH vectors H1 and H2.
Information (pointer) is entered. Also in this specific example, in the LOW vector L1, as shown in FIG. 3, the code of the first character (the second character in the registered word) is hashed as a hash value.

The HIGH vectors H1 and H2 are 4k + 2 and 4k in the registered word in the true meaning registration database 20.
This is a set of permutations of the upper 1 byte of the +3, 4k + 4, 4k + 5th character. Note that k ≧ 0 here. In addition, in the HIGH vectors H1 and H2, when a registered word continues along with the set of the upper bytes for these four characters,
As the identification information D, the information (pointer) to the LOW vector of the remaining words, and when the location of the registered word is determined by the 4k + 5th character, the information of the corresponding location is entered.

Next, the search method of the electronic dictionary device of this example will be described. FIG. 6 is a flowchart of the search method of this specific example. 1. The first character of the input character string S (n) of the search word of length n is extracted (step S1), and the leading character hash table 1 is referred to by the character code (step S2).
Here, if the number is greater than 0, it is a LOW vector L
Since it is the information of 1, the LOW vector L1 obtained from this information is set as the current LOW vector, and k = 0 is set (step S3). On the other hand, if the value is 0, it means that the character string S (n) has not been registered, and the search ends.

2. From character string S (n), 4k + 2,4k +
Take out the 3rd, 4k + 4, 4k + 5th character and divide into the lower byte and upper byte of each character,
Store each in search vectors L and H (step S
4).

3. The current LOW vector L1 vector and the search vector L are matched (step S5). If they are equal to each other, the information for the HIGH vector H1 is obtained, so that it is set as the current HIGH vector. On the other hand, if they are not equal, it means that the character string S (n) is not registered, and the search ends.

4. Matching between the current HIGH vector H1 and the search vector H (step S6). If there is no match as a result of matching, the character string S (n)
Indicates that it does not appear in the search document, and the search ends.

5. If the information obtained in the case of equality is the information of the LOW vector L2, it is set as the current LOW vector, and k = k + 1 is set to 2. It returns to (step S7, S8).

6. If n ≦ 4k + 5 (step S
9) The information obtained at the same time is the information to the true meaning registration database 20. Therefore, the corresponding meaning and its information length are retrieved from the true meaning registration database 20 (step S10), and the search ends. On the other hand, if n> 4k + 5, it means that the character string is not registered, and the search ends.

As described above, according to the second specific example, it is possible to disperse the bias of the index trees forming the set of registered words, and the candidates can be narrowed down quickly. As a result, the search processing time can be shortened, and any word can be searched at a uniform speed. Further, since the length of the information is held at the head of the true meaning registration database 20, when transferring or displaying the search result,
Since it is possible to grasp how much data is available, it is possible to quickly access.

[Brief description of drawings]

FIG. 1 is a flowchart showing a specific example 1 of the full-text search method of the present invention.

FIG. 2 is an explanatory diagram of indexes for realizing a specific example 1 of the full-text search method of the present invention.

FIG. 3 is an LO in a specific example 1 of the full-text search method of the present invention.
It is explanatory drawing of W vector.

FIG. 4 is a block diagram of an electronic dictionary device of the present invention.

FIG. 5 is a diagram showing an index structure of the electronic dictionary device of the present invention.

FIG. 6 is a flowchart of a search method in the electronic dictionary device of the present invention.

[Explanation of symbols]

 1 first character hash table 10 search unit 20 true meaning registration database L1, L2 LOW vector H1, H2 HIGH vector

Claims (6)

[Claims] 1. An index of a character string appearing in a search document is represented by a set of a lower byte and an upper byte of a code of each character in the character string, and in the search document of the character string corresponding to the set. When performing a search with an arbitrary search character string with the information indicating the corresponding place of, the set of the low-order bytes and the set of the high-order bytes of the code of each character in the search string are created. A full-text search method that refers to an index and extracts the corresponding location in a search document. 2. The full-text search method according to claim 1, wherein the search is first performed by a set of low-order bytes of a code of each character in the search character string, and then by a set of high-order bytes. Full text search method. 3. The full-text search method according to claim 1, wherein the search first refers to the index with the first character of the search character string, and then refers to the index with the remaining character strings. Full-text search method. 4. The full-text search method according to claim 1, wherein the index is 1 in a character string appearing in a search document.
A full-text search method characterized by a tree structure whose root is the first character. 5. The full-text search method according to claim 3 or 4, when referring to a LOW vector of an index, which is a set of lower-order bytes of a character string appearing in a search document, first perform reference by one character. , Then, the full text search method characterized by referring to the remaining character string. 6. A true meaning registration database having information on the meaning of a registered word and the length of a character string of the meaning, a character code of the registered word registered in the true meaning registration database, lower byte and upper byte. When a search term is given by providing an index represented by a set of bytes, a set of low-order bytes and a set of high-order bytes of each character code in the search term are created, and the set refers to the index. An electronic dictionary device, comprising: a search unit that retrieves a registered word corresponding to the search word.