JP4183767B2 - Character string search device and search method thereof - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an apparatus and a method for searching for an arbitrary character string from a text file. In particular, an index file is created by mechanical character string division for a text file to be searched, and an arbitrary file is searched using the index file. The present invention relates to a character string search apparatus and method that can efficiently search a character string.
[0002]
[Prior art]
In general, in the field of computers and word processors, a technique for retrieving an arbitrary character string from a text file (document file) is indispensable. In particular, recently, since the amount of text (documents) handled by computers and the like has become enormous, there has been an increasing demand for searching for a predetermined character string reliably and efficiently.
[0003]
Also, in the field of electronic communication, many users have posted messages on the communication network, so in order to search for the information that they want from the large amount of information posted, any number of texts can be selected. There was a need to develop a technology that efficiently searches for strings.
[0004]
By the way, in Western languages, since a space is inserted between words, it is easy to extract a word from a series of texts using this space as a marker. Technically speaking, in Western languages, one character is one byte, and it is easy to detect a space or the like by a byte representing it. Therefore, if the index file is created by extracting in advance the word and the position of the word in the text with a space as a delimiter for the text to be searched, it will try to search by referring to the index file It is possible to quickly search for the presence or absence of a word or character string and the position where it exists.
[0005]
However, although Japanese, Korean, and Chinese have punctuation marks, words that are composed of words and words in a row and one character is represented by multiple bytes (thus these languages It is not easy to create an index file.
[0006]
That is, in the case of Japanese or the like, it is difficult to determine at which position the character string is divided and extracted as a word from simple byte collation. For this reason, in these multibyte languages, a word cannot be extracted from a character string as a word unless a dictionary is prepared and a sentence is analyzed.
[0007]
Therefore, the conventional multi-byte language character string search is mainly performed by one of the following three methods.
[0008]
(1) Method by simple search As is generally done in the field of word processors, when searching for a character string without creating an index file, it is a method of searching for a matching character string for the entire text.
[0009]
(2) Keyword search method Like a certain database, an index file is created by a user specifying keywords in advance for a given text, and a character string is searched using the index file. It is a method to do.
[0010]
(3) Method of full-text search using index file Prepare a dictionary, automatically divide the text into words using a technique such as morphological analysis, create an index file, and use the index file to create text This is a method of searching for a character string from the full text.
[0011]
[Problems to be solved by the invention]
However, the conventional character string search method described above cannot efficiently perform a character string search from a large text that becomes more prominent in the near future.
[0012]
That is, in the above simple search method, the same character string is searched one by one from the beginning of the text. Therefore, it takes too much time to search for a large text and is not suitable for practical use.
[0013]
Next, in the above keyword search method, since the user has to input a keyword for text, it takes time and effort to input, and there is a problem that a character string other than the input keyword cannot be searched.
[0014]
Finally, in the full-text search method using the index file, it takes time to parse the text, and the parse analysis cannot completely divide words. For example, with the current parsing technology, it has been difficult to extract words consisting of kanji and hiragana such as “New Party PRESTO”. Furthermore, if new words that are generated one after another are not registered in the dictionary, new words cannot be extracted as words, so the dictionary must be maintained continuously.
[0015]
For this reason, there has been a demand for the development of a simple technique for easily searching an arbitrary character string from text including a large number of characters, including preparation for searching.
[0016]
Thus, the problem to be solved by the present invention is that an index file for character string search can be automatically created without performing dictionary maintenance or parsing, and a character string for searching for an arbitrary character string using this index file. A search device and a search method thereof are provided.
[0018]
[Means for Solving the Problems]
The character string search device according to the present invention is:
A search processing unit and an index generation unit;
The index generation unit inputs text to be searched, and divides the search target text into a character string of the length of the search character string input by the search processing unit, leaving each character as it is. , To create an index by adding information on the position where the character string appears in the search target text to each divided character string, create an index file by sorting these indexes,
The search processing unit searches the index for the same character string as the search character string.
In addition , the character string search method according to the present invention is as follows:
The search target text and the search character string are input, the search target text is divided into character strings of the length of the search character string, and the characters are divided as they are. An index is created by adding information on a position where the character string appears in the search target text to the character string, and the index is sorted.
A character string identical to the search character string is searched from the index.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of a character string search device and a search method thereof according to the present invention will be described below with reference to the accompanying drawings.
[0020]
FIG. 1 shows the configuration of a character string search apparatus according to the present invention and the flow of processing. As shown in FIG. 1, a character string search device 1 according to the present invention includes an index generation unit 2 and a search processing unit 3. The index generation unit 2 inputs the text 4 to be searched and processes this by a method described later to automatically create an index file 5.
[0021]
On the other hand, the search processing unit 3 inputs a search character string 6 to be searched, performs a search process described later to generate a search character string, and refers to the index file 5 to obtain a search character string. The position 7 is output.
[0022]
Next, processing in the index generation unit 2 and the search processing unit 3 will be further described.
[0023]
FIG. 2 shows a flow of processing for index generation in the index generation unit 2.
As shown in FIG. 2, the index generation unit 2 first inputs text to be searched (step 100) and divides it into fixed-length character strings (step 110).
[0024]
In other words, when the text to be searched is input, the index generation unit 2 sets a fixed-length character string (the length of this character string is fixed length) regardless of the syntax (word, particle, conjunction, etc.). Divided).
[0025]
For example, assuming that the text to be searched is “having a dictionary and word division function” and the fixed length is n = 3, in step 110,
"Dictionary"
"Calligraphy and simple"
"Ya word"
"Words"
"Word division"
"Split machine"
"Split function"
"Function"
“Having Noh”
"I have"
"Have"
"Yes"
"Ru"
Are divided into 13 fixed-length character strings.
[0026]
Next, information on the position at which the fixed-length character string appears, that is, a numerical value indicating the number of characters from the first character of the search target text to the first character of the fixed-length character string is added (step 120).
[0027]
Speaking of the above search target text “having a dictionary and word segmentation function”
"Dictionary, 0"
"Written and simple, 1"
"Ya word, 2"
“Words, 3”
"Word division, 4"
"Split machine, 5"
"Split function, 6"
“Function, 7”
“Has Noh, 8”
"I have 9"
"Has 10"
"Yes, 11"
"Ru, 12"
In this way, 13 indexes are generated by pairing each fixed-length character string and its position information.
[0028]
The position information is not limited to the number of characters from the first character of the search target text to the first character of the fixed-length character string, and may be the number of characters from the last character of the search target text, or may be given as a constant function. Good.
[0029]
Next, these indexes are rearranged in a certain order by the first character (this operation is called sorting) (step 130).
[0030]
In the example of the index “dictionary, 0”, ..., “ru, 12”,
"Yes, 11"
"Ya word, 2"
"Ru, 12"
"I have 9"
"Split function, 6"
“Function, 7”
"Word division, 4"
"Dictionary, 0"
"Written and simple, 1"
“Words, 3”
“Has Noh, 8”
"Split machine, 5"
"Has 10"
So sort. The sorted index is output as an index file (step 140).
[0031]
What should be noted in the above index generation process is that, according to this processing method, it is possible to create an index mechanically from text without preparing a dictionary and performing difficult parsing. An index can be generated. This index can be easily searched by sorting as will be described later.
[0032]
Next, the method of using the index, that is, the processing by the search processing unit 3 will be described with reference to FIG.
As shown in FIG. 3, the search processing unit 3 inputs a search character string (step 200), determines its length, and distributes subsequent processing by comparing it with a fixed length (step 210).
[0033]
First, the case where the length m of the search character string is equal to the fixed length n will be described.
If the length m of the search character string is equal to the fixed length n, the same character string as the search character string is searched from the index file (step 220).
[0034]
For example, a case where a search character string “word division” is searched from “having a dictionary and a word division function” that created the index file corresponds to this.
[0035]
That is, since the length of the search character string “word division” is 3 characters, m = 3, which is equal to the above-described fixed length n = 3 (m = n). In this case, the same character string may be searched from the sorted index described above. Since position information is added to the character string of the index, the position of the character string can also be known.
[0036]
What should be noted here is that the index file sorts the indexes as described above, so it is not necessary to search all of them, and the index “word division, 4” with “word” as the first character is immediately searched. Is what you can do. As a result, the search can be performed much more efficiently than the conventional simple search method.
[0037]
In the above example, by obtaining the index “word division, 4”, it is possible to know that there is a character string “word division” fourth from the first character of the text.
[0038]
Next, a case where the length m of the search character string is smaller than the fixed length n will be described.
When the length m of the search character string is smaller than the fixed length n, a wild card character string is created by supplementing the search character string with a wild card (step 230), and the corresponding character string is searched from the index file (step 230). 240).
[0039]
For example, this corresponds to the case where the search character string “division” is searched from “having a dictionary and word division function” in the above example. In this case, since the length of the search character string “divided” is two characters, m = 2, which is smaller than the above-described fixed length n = 3 (m <n).
[0040]
At this time, the wild card character string “divided *” is searched from the index file. Here, “*” is a wide card character, and the portion corresponding to the wild card character may be an arbitrary character.
[0041]
In the example of the index file “dictionary or 0,...,“ Ru, 12 ”, an index“ divider, 5 ”can be obtained as a character string corresponding to“ divide * ”. This makes it possible to know that the search character string “divided” exists fifth from the first character of the text.
[0042]
Here, it should be noted that when m <n, the wildcard character * is added to the end of the search character, and the corresponding portion of the index sorted by the first character can be immediately accessed. Since the index is created starting from each character of the text, the above method does not cause a search omission.
[0043]
Finally, a case where the length m of the search character string is larger than the fixed length n will be described.
When the length m of the search character string is larger than the fixed length n, the search character string is divided into fixed length character strings (step 250), a phrase expression to be described later is created (step 260), and the corresponding phrase is obtained from the index file. The expression is searched (step 270).
[0044]
For example, the case of searching for a search character string “having a word division function” from “having a dictionary and word division function” in the above example corresponds to a case where m> n. First, create a phrase expression from "word division function". Here, the phrase expression is a search character string (this is called a phrase) including a character string “XXX” and a character string “△△△”, and a character string “XXX” and a character string “Δ △△”. When the first characters of “” appear apart from each other by p characters, this is expressed as “XXX” <p> “ΔΔΔ”, and this “XX” <p> “ΔΔΔ” is expressed as a phrase expression. That's it. In addition, when p is smaller than n (= 3), the character string “XXX” and the character string “ΔΔΔ” are partly or entirely overlapped. It can be expressed in the above phrase formula.
[0045]
To search for a search character string including the character string “XXX” and the character string “△△△”, search the index “XXX” and the index “△△△” separated by p characters. That's fine.
[0046]
In the example of searching the search character string “word split function” from “having a dictionary and word split function”, from “word split function”
"Word"<3>"Distributionfunction"<1>"Function"
Alternatively, a phrase expression such as “words” <2> “divider” <2> “function” is created. Here, the two phrase expressions are equivalent to each other, and the phrase expression only needs to cover the entire search character string.
[0047]
Next, the index corresponding to the phrase expression is searched from the index “dictionary, 0,...,“ Ru, 12 ”.
[0048]
As a result, the indexes “word portion, 3” to “function, 7” are searched, and it can be known that the search character string appears as the third character from the first character of the search target text.
[0049]
According to the phrase search function, character strings longer than the fixed index length can be searched using an index file prepared in advance, and the target character string can be searched quickly because the index file is sorted.
[0050]
The description of the above-described embodiment is completed. In the above-described embodiment, an index file is created with a fixed length for a search target text, and a search character string is searched using the index file. However, different search methods are possible using the method of the present invention. The search method will be described below.
[0051]
The different search method is a method of dividing the text to be searched according to the length of the search character string when searching without creating an index file in advance.
[0052]
This method is effective for texts that are relatively small and have a short shelf life.
[0053]
According to this method, when a search character string is input to a predetermined search target text, the search target text is divided with the length of the character string as a fixed length, and an index is created.
[0054]
In this case, creation of an index at the end of the text that is less than the length of the search character string is omitted. This is because it is clear that such an index does not match the search character string.
[0055]
For example, when searching for a character string “having a word division function” from a text “having a dictionary and a word division function”, a fixed length n = 10 (= m) is set, and “having a dictionary and a word division function” The following index is created.
[0056]
“Dictionary and word segmentation function is 0”
“Has a book and word division function, 1”
“And with word-splitting function, 2”
"Has word division function, 3"
At this time, it is not necessary to create an index of 9 characters or less, that is, “having a word division function, 4” to “ru, 12”. This is because it is clear that these do not include the search string.
[0057]
Then sort the index. In the above example, they are rearranged as follows.
[0058]
“And with word-splitting function, 2”
“Dictionary and word segmentation function is 0”
“Has a book and word division function, 1”
"Has word division function, 3"
If the same character string as the search character string “having the word division function” is searched from the index in this state, the position of the desired character string can be known. In the above example, it can be known that the character string “having the word division function” exists in the third character from the first character of the text.
[0059]
What should be noted here is that according to this method, it is not necessary to create an index for all texts, and the use efficiency of the storage device can be increased, and a long search character string is searched. In this case, it is not necessary to create an index for a character string shorter than n at the tail of the text, the creation of the index is simple, and the search method of the character string is discovery of the same character string. is there.
[0060]
Therefore, it is an effective search method for a relatively small amount of text that has a short storage period, and can be searched very efficiently particularly when searching for a long character string.
[0061]
【The invention's effect】
As is clear from the above description, the character string search device and the search method according to the present invention are mechanically performed without parsing the text to be searched, and thus without preparing a dictionary for parsing. It is possible to create indexes composed of character strings of a certain length and sort these indexes.
[0062]
When searching for a character string using this index, a search character of any length can be obtained regardless of the length of the search character string by any of the same character string search, wildcard character string search, and phrase expression search. You can search columns.
[0063]
As a result, even when a large number of large-capacity texts are searched, an index is created by mechanical processing, and an arbitrary character string can be searched quickly and reliably.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a character string search device according to the present invention and a flow of processing thereof.
FIG. 2 is a flowchart showing processing in an index generation unit of the character string search device according to the present invention;
FIG. 3 is a flowchart showing processing in a search processing unit of the character string probing device according to the present invention.
[Explanation of symbols]
1 Character string probing device 2 Index generation unit 3 Search processing unit 4 Text to be searched 5 Index file 6 Search character string 7 Character string to be searched and its position

Claims (2)

A search processing unit and an index generation unit;
The index generation unit inputs text to be searched, and divides the search target text into a character string of the length of the search character string input by the search processing unit, leaving each character as it is. , To create an index by adding information on the position where the character string appears in the search target text to each divided character string, create an index file by sorting these indexes,
The character string search apparatus, wherein the search processing unit searches the index for the same character string as the search character string. The search target text and the search character string are input, the search target text is divided into character strings of the length of the search character string, and the characters are divided as they are. An index is created by adding information on a position where the character string appears in the search target text to the character string, and the index is sorted.
A character string search method, wherein the same character string as the search character string is searched from the index.

Images