KR101694179B1 - Method and apparatus for indexing based on removing vowel - Google Patents

Abstract

According to one aspect of the present invention, a method for generating an index based on removing a vowel includes: a step of allowing an index generation device to remove a vowel in a keyword of a document and converting the vowel into a keyword composed of a consonant; and a step of generating the index for the document by using the converted keyword.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001]

The present invention relates to a method and apparatus for generating an index by removing a vowel for a fuzzy keyword search. More particularly, the present invention relates to a method and apparatus for extracting a keyword from a document and removing the vowel to create an index for the document using the extracted keyword.

In today's flood of information, it is very important to search web pages or documents quickly. Indexing is the process of extracting a lexicon from a document for quick retrieval and constructing a data structure. Conversely, given a particular keyword, it is called an inverted index that tells you which keyword appears in which document.

When a user enters a specific word, it is possible to quickly search for and provide a document having the word as a keyword through an inverse index. This is done by using the indexes built for each document. In addition to the keywords extracted from the document during the process of indexing the document, you can add words related to the keywords to the document's index. Here, the relevant word is a change in the form of a word according to a tense or a part-of-speech or a typo frequently inputted.

For example, if you want to show a document that contains the word come, even if you enter the word came in the search term, when you create the index of the document that contains come, you must add came to the index. Likewise, words such as teach and teacher can be added to a single index. You can also add calendar 's "calender" to a single index.

In this way, a keyword having relevance to the keyword is added to the index to perform the search, which is called fuzzy keyword search or fuzzy search. For the fuzzy keyword search, there are algorithms such as Levenshtein distance (edit distance), wild-card, and K / n-gram. However, these algorithms require large indexes and have poor performance.

Therefore, there is a need for an index generation method that can improve the search performance while reducing the size of the index.

SUMMARY OF THE INVENTION The present invention provides a method and apparatus for extracting a keyword from a document and creating an index for the document using the extracted keyword by removing a collection of keywords and creating an index.

It is another object of the present invention to provide a method and apparatus for creating an auxiliary index according to the length of a keyword from which a vowel is removed in the process of eliminating a keyword vowel and creating an index.

The technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems which are not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method of generating a collection removal index, the method comprising: a step of removing an vowel from a keyword of a document and converting the keyword into a keyword composed of consonants; And generating an index for the document using the converted keyword.

In one embodiment, the step of converting into the keyword consisting only of the consonants may include removing one of the consonants consecutively appearing as duplicates in the extracted keyword.

In another embodiment, the step of generating an index for the document may further include generating an auxiliary index larger than the index if the length of the converted keyword is less than a predetermined value.

In another embodiment, the predetermined value is three.

In yet another embodiment, generating the index for the document may include generating the index by performing a K-gram on the converted keyword.

In yet another embodiment, the step of performing the K-gram to generate the index may include comparing the length L of the converted keyword to a predetermined value m, The step of performing K-grams with K = m for the converted keyword may be included.

In yet another embodiment, the step of performing the K-gram to generate the index may include comparing the length L of the converted keyword to a predetermined value m, And performing a K-gram to set K = L to the converted keyword, if K is less than or equal to L.

In another embodiment, the step of performing K-grams with K = L for the transformed keyword may include: if the L is greater than 1, converting K-grams with K = L-1 to the transformed keywords And performing further steps.

According to another aspect of the present invention, there is provided an apparatus for generating a collection elimination-based index, the apparatus comprising: at least one processor; a memory for loading a computer program executed by the processor; a storage for storing an index for the document; . ≪ / RTI > Here, the computer program may include an operation of removing a vowel from a keyword of the document and converting the vowel to a keyword composed of consonants, and an operation of generating an index for the document using the converted keyword.

In one embodiment, the operation of generating an index for the document may include an operation of bi-gramming the converted keyword to generate the index.

In another embodiment, the operation of generating an index for the document may include an operation of further generating an auxiliary index larger than the index if the length of the converted keyword is less than a predetermined value.

In another embodiment, the operation of generating an index for the document may include an operation of generating the index by performing a K-gram on the converted keyword.

In another embodiment, the operation of performing the K-gram to generate the index may include an operation of comparing whether the length L of the converted keyword is greater than a predetermined value m, , It may include an operation of performing a K-gram with K = m for the converted keyword.

In another embodiment, the operation of performing the K-gram to generate the index may include an operation of comparing whether the length L of the converted keyword is greater than a predetermined value m, , It may include an operation of performing a K-gram to set K = L to the converted keyword.

The effects according to the embodiment of the present invention are as follows.

By removing a vowel from a keyword and using a keyword consisting only of consonants in the search, the size required to create an index can be reduced. In addition, although the length of the keyword is reduced, the performance of the search can be improved because the characteristic of the keyword can be reflected well.

In addition, in the process of generating a keyword composed only of consonants by removing the vowel, the damage due to the modification of the keyword can be compensated by using the auxiliary index. In other words, if the length of the word becomes too short by removing the vowel, the search efficiency can be improved by utilizing the auxiliary index.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood to those of ordinary skill in the art from the following description.

1A to 1C are diagrams for explaining a conventional index generation method.
2 is a flowchart illustrating an index generation method based on a vowel removal method according to an embodiment of the present invention.
FIG. 3 is a pseudocode explaining a method of generating an index based on a vowel removal method according to an embodiment of the present invention.
4A to 4C are diagrams for explaining a fuzzy search using an edit distance method.
5A to 5C are diagrams for explaining a fuzzy search using a wildcard method.
6A to 6C are diagrams for explaining a fuzzy search using the K-gram method based on K = 3.
7A to 7C are diagrams for explaining a fuzzy search using a bi-gram method.
8A to 8C are diagrams for explaining a fuzzy search using a NOVEL method according to an embodiment of the present invention.
9A to 9C are diagrams for explaining accuracy according to each method when searching a first document including a plurality of keywords.
10A and 10B are diagrams for explaining the accuracy of each method with respect to a first data set including a plurality of keywords.
11 to 12 are graphs and tables showing test results for the first data set and the second data set including a plurality of keywords.
13 is a diagram illustrating an example of a hardware configuration of an index generation apparatus based on a vowel removal method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise. The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification.

It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / Or additions.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1A to 1C are diagrams for explaining a conventional index generation method.

FIG. 1A is an exemplary diagram for explaining a method of generating an index based on a wild-card. Referring to FIG. 1A, three keywords chemistry, phenomenon, and come can be seen on the arrows. When an edit distance is set to 1, an index for three keywords is created as shown below the arrow in FIG. 1A.

Here, edit distance refers to a method of determining the degree of similarity between two words. When comparing two words, the edit distance is determined based on how many times the insertion, deletion, and substitution operations are performed on the string. That is, the minimum number of editing operations required to convert a source string to a target string is called edit distance. A more detailed description of the editing distance can be found on the web page at https://en.wikipedia.org/wiki/Edit_distance .

For example, * chemistry, which is illustrated by the index of chemistry at the bottom of FIG. 1A, can be obtained by simply deleting any alphabet in the place marked with *, which is a target string, chemistry. Likewise, che * istry can substitute m for any alphabet in the place marked with * to get the target string chemistry.

Thus, the index shown at the lower part of FIG. 1A consists of the words chemistry and the words that can obtain the keywords of phenomenon and come, in order to perform an edit operation once in a portion marked with an asterisk. The words displayed in the index are indexes composed of words having an edit distance of 1 because the keyword that is the object string can be obtained by one editing operation.

Let's look at the size of the index according to the word length. In the example of FIG. 1A, if the word 'come' is short, the word 'come' has a length L of 4. If you want to build an index of come, you need to do the following: * come with no editing and {* come, c * ome, co * me, com * Alternatively, you can build an index with {* ome, c * me, co * e, com *} where you can get a come replace. That is, the index for come is {come, * come, c * ome, co * me, com * e, come *, * ome, c * me, co * e, com *}. The size of the index at this time is 10.

That is, according to the wildcard method, the word "come" having a length of 4 makes an index having a size of 1 + 5 + 4 = 10. Expressing it as 1 + (L + 1) + L = 2L + 2. As can be seen from the formula, according to the wildcard method, the longer the word length, the larger the size of the index becomes.

This is the case with the wildcard method. In the case of the edit distance method, which is the mother of the wildcard method, 26 alphabets from the actual a to z are included in the place, so the required index size is 1 + { L + 1) + L} * 26 = 52L + 27, which requires a larger size index than the wild card method. That is, if you create an index by the edit distance method for come, the index from acome to zcome is added to the index instead of * come, and from coma to comz instead of com *.

In summary, according to the edit distance method, a 52L + 27 size index is required to construct an index of edit distance 1 in the edit distance method for a word whose length is L, and when the wild card method is used, Two size indexes are required.

This is the size of the index when the editing distance is 1 and the size of the index becomes exponentially larger when the editing distance is larger than 1. As the size of the index grows, the amount of data to be compared in the search increases, which results in an increase in the cost and time consumed in the search, thereby deteriorating the search performance.

1B is an exemplary diagram for explaining a method of generating an index based on a K-gram (k-gram). The K-gram is a method of constructing an index by dividing the keyword into K-sized syllables. 1B shows an example in which an index is constructed on the basis of a case where the value of K is 3.

For example, if you create an index based on chemistry with K = 3, you have indexes of {#ch, che, hem, emi, mis, ist, str, try, rt # do. In the case of a phenomenon, the index has {#ph, phe, hen, eno, nom, ome, men, eno, non, on #}. Similarly, in the case of come, we have {#co, com, ome, me #} as an index.

As shown in FIG. 1B, the front and back boundaries # are added so that the size of the index of the keyword of length L is L. In the case of the wild card method, the size of the index can be reduced to about half by using K-gram, compared with the index of the keyword having the length L of 2L + 2.

In the K-gram method, a case where K = 1 is called a uni-gram, a case where K = 2 is a bi-gram, a case where K = 3 is a tri- It is called a tri-gram. Let's take a case of using a bi-gram together with FIG. 1C.

1C is an exemplary diagram for explaining a bi-gram based index generation method. By bi-grams, the index is constructed by dividing the keyword into two syllables.

As shown in FIG. 1C, in the case of chemistry, {ch, he, em, mi, is, st, tr, ry} en, no, on} as an index. Finally, in the case of come, {co, om, me} are indexed.

If we create a bi-gram index of a word with length L, we will create an index with L-1 size. We can see that the size of K-gram is similar to that of K-gram. When using K-grams or bi-grams, the size of the index can be reduced by about half compared with the wildcard method. Therefore, in recent years, many methods of building indexes based on K-gram have been used.

Nevertheless, the K-gram also has a disadvantage in that the size of the index increases in proportion to the length of the word, and the search performance is poor when the punctuation is searched. To overcome this problem, we propose a new index construction method. The index construction method proposed by the present invention is based on the K-gram, which uses a vowel removal and an auxiliary index.

2 is a flowchart illustrating an index generation method based on a vowel removal method according to an embodiment of the present invention.

As we have explained in the previous section on editing distance method, wildcard method, and K-gram method, the longer the keyword length, the larger the size of the index. Therefore, to reduce the size of the index, you must reduce the length of the keyword. However, if the length of the keyword is reduced, the size of the index becomes smaller, but the size of the index directly affects the search. Therefore, it is important how to reduce the length of the keyword.

To effectively reduce the length of a keyword, you need to know about information retrieval. There are two concepts in information retrieval: precision and recall. Suppose that there are a total of 32 documents containing a specific A keyword. At this time, when the keyword A was inputted through the search engine, 20 documents were searched as a search result. Of the 20 documents retrieved, 16 documents were related to the A keyword, and 4 documents were related to the keyword A Let's assume.

In this case, the accuracy is 16/20 * 100 = 80%, which is the ratio of the document related to the actual A keyword among the 20 documents fetched as the search result. And the recall rate refers to the document retrieved from related documents, so it has a value of 16/32 * 100 = 50%. That is, the accuracy is the ratio of what is true to what is true, and the recall is the ratio of what is actually true to what is true.

The better the accuracy, the more similar each keyword in the index is to the keyword. For example, if the index of the come keyword is constructed with the keyword {come} equal to the keyword, the retrieved document contains all the come, so the accuracy becomes high. Instead, documents containing came could not be retrieved, so the recall rate drops.

Therefore, the size of the index is increased by adding words similar to the keyword to the index by using the wildcard method or the K-gram method in order to retrieve the document including the change of the word or the tense according to the parts and tense. By constructing {co, om, me} with an index of come using a bi-gram, even if it does not include come, indexes of me will be able to search for documents that have come. That is, the recall rate can be increased.

In summary, the more similar the keywords in the index are to the keywords, the higher the accuracy. The larger the index size, the higher the recall rate. However, if the size of the index is increased to increase the recall rate, the accuracy may be lowered, and the time and cost of the search may increase due to the size of the index.

As a result, the size of the index is proportional to the length of the keyword, so the size of the index can be reduced by reducing the length of the keyword. Instead, in order to increase accuracy, the characteristics of the keywords should be preserved. This is because the similarity between the words included in the keyword and the index is increased. In order to do this, it is necessary to look at the pattern in which the typos mainly occur.

If you look at the most common typo in English, the first is to display the vowel as a different vowel. This is due to mispronunciation. For example, you may enter realize as "relize" or because as "becouse". Statistically, typographical errors caused by mistyped vowels account for about 60% of all typographical errors.

Next, there may be cases of incorrect input. For example, you may want to type the same character as "betwween" or "betweeen" instead of between. Or, the spelling of a vowel may be changed back and forth. For example, typing "becuase" as the reason or "solider" as the solider is a typo.

Next, there is a typo caused by homophones with similar meanings. For example, you might enter rite instead of right or reed instead of read. These typo are mainly typographical errors. You can find more information on this at https://en.wikipedia.org/wiki/Commonly_misspelled_English_words.

As we have seen so far, most errors in English (60%) occur in vowels, and others include duplicate input or order confusion. In this paper, we propose a method to reduce the length of a keyword by eliminating vowels. That is, for the keyword come, we remove the vowels a and e and build the index based on "cm".

Reducing the length of a keyword by removing vowels has another advantage in addition to the fact that it reflects the main pattern of typos. That is, removing vowels and leaving consonants better reflects the characteristics of the keywords. English has 26 alphabets, of which {a, e, i, o, u} are five vowels and the other 21 are consonants.

If the length of the keyword is reduced by removing the consonants instead of decreasing the length of the keyword by removing the vowel, five keywords {a, e, i, o, u} The keyword becomes similar to the keyword and the discrimination power is lowered.

In other words, when the consonants are removed, the characteristics of the keywords are less reflected than when the vowels are eliminated. In other words, if the length of the keyword is reduced by removing the vowel, the remaining word becomes a word consisting of only 21 consonants. Therefore, when compared with other keywords, the discrimination becomes rather higher and reflects the characteristic of the original keyword more accurately. .

In summary, the number of vowels in the alphabet is smaller, so if you delete the vowels and leave only the consonants, you can increase the accuracy by reflecting the characteristics of the keywords better. It can be seen that reducing the length of the keyword by eliminating the vowel increases the accuracy and the recall rate.

In this paper, we propose a method of constructing an index by removing a set of keywords when constructing an index from a keyword. Since the vowel is removed, it is called the NOVEL method, focusing on the No-VoeEL. Hereinafter, NOVEL refers to a method of eliminating a vowel to reduce the length of a keyword and construct an index.

In addition, since there are many typos in vowels in English, and the kinds of vowels are small, a method of removing vowels has been proposed. In other words, other kinds of languages based on the alphabet other than English, and non-alphabetic languages such as Chinese and Korean are widespread than most vowels. Considering that there are a lot of mistakes in inputting wrong vowels The NOVEL method proposed by the present invention is still valid.

The following NOVEL method is explained in English, but it is not intended to limit the application of the NOVEL method to other languages, as this is only for convenience of understanding. Under this assumption, the NOVEL method will be described with reference to the flow chart of FIG. 2.

Referring to FIG. 2, a sentence is first extracted from a document (S1000). The process of extracting a sentence from a document uses a punctuation or a separator. Next, the keyword is extracted from the sentence (S2000). Here, the keyword is a word having a high degree of importance by evaluating the degree of importance among the words included in the document.

Various algorithms can be applied to evaluate the importance of words in a document. For example, by using an algorithm such as TF-IDF (Term Frequency-Inverse Document Frequency), the importance can be evaluated in consideration of the frequency of occurrence of the keyword in the document and the frequency of occurrence in other documents. Or meta-data such as tags included in the document can be used to evaluate the importance. Or the words included in the document can be clustered on the basis of meaning to evaluate the importance according to the size of the cluster. In addition to the methods mentioned above, various algorithms can be applied to evaluate the importance of words included in a document and to extract keywords.

Next, only vowels are extracted by removing vowels from the keyword (S3000). That is, remove the vowel from the keyword to create a keyword consisting only of consonants. In the case of baseball, "bsbll" is a new keyword created by extracting only consonants. In the case of telephone, "tlphn" is a new keyword created by extracting only consonants.

In the case of chemistry, phenomenon, and come, which were used in FIG. 1A, when the vowel is removed, "chmstry", "phnmnn", and "cm" become new keywords extracted only consonants. There may be additional processing in the process of creating new keywords by removing vowels and extracting only consonants.

For example, as we have seen, one of the main patterns of typographical errors is that characters are duplicated. Therefore, it is possible to consider a method of reducing the length of a keyword by eliminating redundant input characters and correcting them with a single character. For example, you can apply a method of reducing two "ll" s to a single "l" in baseball. In addition, the method of eliminating redundant input may be applied in parallel with the method of eliminating vowels.

For example, let's apply de-duplication and vowel removal to baseball at the same time. In this case, you can reduce the length of the keyword in the same way as baseball> basebal> bsbl. The original keyword basebll is an 8-syllable word, but the changed keyword "bsbl" is a 4-syllable word. In the example of baseball, the advantage of the NOVEL method that can reduce the length of the keyword while preserving the characteristics of the keyword is confirmed. This improves the accuracy and the recall rate.

It is desirable to remove duplicate characters before eliminating the vowel because removing the vowels and extracting the keywords with consonants will increase the probability of the same consonant appearing. However, this is only a suggestion, and duplicate removal can be performed after vowel removal.

Returning to FIG. 2, after extracting the keyword consisting only of consonants by removing the vowel, the index is constructed by applying the K-gram to the keyword. The process of constructing an index by applying a K-gram to a keyword is similar to a method of constructing an index using a conventional K-gram.

However, another feature of the NOVEL method proposed by the present invention also appears here. That is to apply the K-gram one more time to construct the secondary index. For example, suppose you apply the K-gram for the keyword come to the K = 3.

Since the conventional K-gram method does not change the keyword, it builds indexes of {#co, com, ome, me #} for the word come. However, in the NOVEL method proposed by the present invention, since the index is constructed by removing the vowel, the keyword is changed after changing the word to cm for the come word. If we construct an index based on K = 3 for cm, {cm}.

As can be seen, removing the vowel has the advantage of reducing the size of the index by reducing the length of the keyword, but the size of the index is too small and the recall is lowered. In order to compensate for this, when the length of a new keyword composed only of consonants is small, the total size of the index can be increased by constructing an auxiliary index by performing (K-1) -gram in addition to K-gram. This can increase the recall rate.

Here, 3 is proposed in the present invention as a value of K which determines whether to construct an auxiliary index. This is because the average length of words is the most efficient when K = 3, based on English. However, this is based on English, and the NOVEL method proposed in the present invention can be applied to other languages. Accordingly, when the NOVEL method is applied to other languages, the value of K, which is a standard, can be changed.

Returning to FIG. 2, if the value of m is greater than 3 (S4000), an index of 3-grma is generated (S5000). In this case, the index constructed with 3-gram is enough to search. However, if the value of m is smaller than 3, the index of the m-gram is generated first (S6100). If the value of m is larger than 1 (S6300), an index of (m-1) (S6500). In other words, the secondary index is to be constructed.

Here, it is natural that step (S6300) of confirming once again whether the value of m is larger than 1 is because m-1 becomes 0 and 0-gram can not be performed when the value of m is 1. That is, the step of confirming once again whether the value of m is greater than 1 (S6300) is a necessary step in logic.

In the example of FIG. 2, the index is constructed by removing the vowel and applying K-grams to the keyword consisting only of consonants. In addition to K-grams, the index can be constructed by applying any other method. For example, an index can be built by applying the wildcard method. Alternatively, the index may be constructed by applying the bi-gram method. The flow chart of Fig. 2 does not exclude the application of other methods other than K-grams.

The features of the NOVEL method proposed in the present invention described in FIG. 2 are summarized as follows. First, by reducing the length of the keyword, the overall size of the index is reduced. This reduces the cost and time required for fuzzy search. That is, the search performance can be improved.

In addition, by removing the vowels and using the keyword consisting only of consonants, the length of the keyword can be reduced efficiently. This allows you to better reflect the characteristics of the keyword in the index and improve its accuracy. Next, when the length of the keyword is smaller than the preset value, an additional sub-index is constructed to prevent the size of the index from becoming unnecessarily small. This can increase the recall rate. That is, retrieval efficiency can be improved through collection elimination and secondary indexes.

FIG. 3 is a pseudocode explaining a method of generating an index based on a vowel removal method according to an embodiment of the present invention.

Referring to FIG. 3, a sentence L is extracted from a document Dk (line 2), and a keyword Wi is extracted from a sentence L (line 3). For the extracted keyword Wi, the vowel is removed to create a new keyword Wv (line 5). If the length of the new keyword Wv is greater than 3, 3-gram is performed to construct an index (line 8). In the case of 3 and 2, m-gram (line 14) and (m- line 15) to construct an index. If m is 1, 1-gram is performed to construct an index.

The pseudo code shown in FIG. 3 is shown only in the form of a code, but the characteristic of only the NOVEL method called the vowel removal and the secondary index described in the flowchart of FIG. Now let's look at the NOVEL method through the actual case of FIGS. 4A to 10B.

4A to 4C are diagrams for explaining a fuzzy search using an edit distance method.

In the case of FIG. 4A, "kehmeesdree" is input as the pronunciation of the chemistry, and "finominan" is input as the pronunciation of the phenomenon is typified in the case of FIG. 4B. Finally, in the case of FIG. 4c, "came", which is a past type of come, is illustrated. Suppose that fuzzy search is performed using the edit distance method for each case.

Referring to FIG. 4A, to go from the source string "kehmeesdree" to the destination string chemistry, 1. replace k with c> 2. remove e> 3. add e after h> 4. replace e with i> 5. delete e> 6 . replace d with t> 7. replace e with y> 8. delete e.

If you decide to judge similarity based on the editing distance of 1, "kehmeesdree" and chemistry are determined to be similar because they are larger than the standard value of 1 because the editing distance is 8. In other words, even if you search by typing "kehmeesdree", you can not see the search result that contains chemistry.

Similarly, referring to FIG. 4b, to go from the source string "finominan" to the destination string phenomenon, 1. replace f with p> 2. add h> 3. replace i with e> 4. replace i with e> 5. replace a with It is necessary to go through five editing operations of o.

If we decided to judge the similarity based on the editing distance of 1, "finominan" and phenomenon are larger than the reference value of 1 because the editing distance is 5, and it is judged that they are not similar. In other words, even if you search for "finominan", you can not see the search result that contains the phenomenon.

Referring to FIG. 4C, it can be seen that, in order to go from the source string "came" to the destination string come, it is necessary to perform the editing operation of the first step of 1. replace a with o. Since we decided to judge similarity based on edit distance 1, if you enter "came", you can search for documents containing similar come.

As shown in FIGS. 4A to 4C, when the index is constructed and the document is searched using the edit distance method, the search performance and efficiency are inferior. That is, the size of the index is increased, the performance of the search is deteriorated, and the necessary documents may not be properly retrieved.

5A to 5C are diagrams for explaining a fuzzy search using a wildcard method.

Referring to FIG. 5A, although the size of the index is reduced compared with the edit distance method, it is still not possible to match chemistry with "kehmeesdree". That is, even if you type "kehmeesdree", you can not search documents containing chemistry keywords.

Similarly, referring to FIG. 5B, since "finominan" and the phenomenon do not match, it can be seen that even if "finominan" is input, the document containing the phenomenon can not be retrieved.

Finally, referring to FIG. 5C, it can be seen that "came" and "come" match in the index with "c * me" being the same. That is, according to the wildcard method, "came" can be used to search for documents containing come.

Comparing the distance editing method and the wildcard method, the size of the index can be greatly reduced from 52L to 2L. As the size of the index is reduced, search time and cost can be saved, which improves search performance. However, you can see that the accuracy of the search is lowered, as in the case of "kehmeesdree" or "finominan" entered as pronounced.

6A to 6C are diagrams for explaining a fuzzy search using the K-gram method based on K = 3.

Referring to FIG. 6A, although the size of the index is smaller than that of the wildcard method, it is still not possible to match chemistry with "kehmeesdree". That is, even if you type "kehmeesdree", you can not search documents containing chemistry keywords.

Similarly, referring to FIG. 6B, it can be seen that "finominan" and the phenomenon do not match, so that even if "finominan" is input, the document containing the phenomenon can not be retrieved.

Finally, referring to FIG. 6C, "came" and "come" can be seen as matching "me #" in the index. That is, according to the wildcard method, "came" can be used to search for documents containing come.

Compared to the wildcard method and the K-gram method, the size of the index can be reduced by half from about 2L to L. As the size of the index is reduced, search time and cost can be saved, which improves search performance. However, you can see that the accuracy of the search is lowered, as in the case of "kehmeesdree" or "finominan" entered as pronounced.

7A to 7C are diagrams for explaining a fuzzy search using a bi-gram method.

Referring to FIG. 7A, although the size of the index is smaller than that of the wildcard method, it can be seen that "kehmeesdree" does not match chemistry. That is, even if you type "kehmeesdree", you can not search documents containing chemistry keywords.

Next, referring to FIG. 7B, it can be seen that "finominan" and phenomenon are matched because "no" and "om" are the same in the index. That is, according to the bi-gram method, "finominan" can be used to search for documents containing a phenomenon.

Finally, referring to FIG. 6C, it can be seen that "came" and "come" are matched because "me" is the same in the index. That is, according to the wildcard method, "came" can be used to search for documents containing come.

When comparing the K-gram method and the bi-gram method, the size of the index is approximately equal to L. However, since the length of the word included in the index used in the bi-gram is shorter than the length of the word included in the index used in the K-gram, the recall rate increases but the accuracy decreases.

8A to 8C are diagrams for explaining a fuzzy search using a NOVEL method according to an embodiment of the present invention.

Referring to FIG. 8A, it can be seen that the "kehmeesdree" and the chemistry respectively generate K-grams based on the new keywords "khmsdr" and "chmstry" that have been removed from the vowels. The length of the keyword was 11 syllables and 9 syllables respectively before elimination, and after elimination of the vowel, it was reduced to 6 syllables and 7 syllables.

If you construct a 3-gram index for a keyword created by extracting only consonants, "hms" is common, and you can enter "kehmeesdree" as a keyword to search for documents containing keywords.

Next, referring to FIG. 8B, it can be seen that the "finominan" and the phenomenon respectively generate K-grams based on the new keywords "fnmnn" and "phnmnn" The length of the keywords was 9 syllables and 10 syllables before the vowel removal, respectively. However, after elimination of the vowels, the syllables decreased to 5 syllables and 6 syllables.

If you build an index by applying 3-grams to keywords created by extracting only consonants, "nmn" and "mnn" are common, and if you enter "finominan" as a search term, you can search for documents containing a phenomenon.

Finally, referring to FIG. 8C, a 2-gram is performed on the basis of the same keyword "came" and "vowel", in which "v" and " &Quot;, < / RTI > Because the keywords are the same as the removed vowels, all of the indexes are matched, and if you enter "came" as the search term, you can search for documents that contain the keyword "come".

The K-gram method and the bi-gram method are compared with the NOVEL method according to an embodiment of the present invention as follows.

First, the size of the index is K-gram or the bi-gram method generates an index having a size of L, which is approximately the length of the keyword. On the other hand, NOVEL statistically removes vowels from a keyword and creates a new keyword consisting only of consonants, which removes about 50% of the syllables and generates a 0.5L-size index. Since the size of the index is reduced by half, it is possible to compare the index constructed with the keyword extracted from the document and the index constructed by the keyword inputted by the user, and reduce the cost and time consumed in searching the document by half. That is, the search performance can be improved.

Next, as you can see from the examples of "kehmeesdree" and "finominan", removing a vowel can better reflect the characteristics of the keyword. That is, although the size of the index is reduced, the retrieval rate can be increased. The advantages of the NOVEL method in addition to the recall rate are shown in FIGS. 9A to 9C in terms of accuracy.

9A to 9C are diagrams for explaining accuracy according to each method when searching a first document including a plurality of keywords.

Referring to FIG. 9A, it can be seen that the first document has five keywords {house, mark, worry, motor, melon} as keywords. Assume that the user enters "homework" as the search term and searches.

If a first document having {house, mark, worry, motor, melon} as a keyword is constructed with an index based on K = 3 as shown in FIG. When the user enters "homework" as the search term, the index of the "homework" becomes {#ho, hom, ome, mew, ewo, wor, ork, rk # "rk #" and "wor" are common.

That is, even though the first document does not have "homework" as a keyword, if the user inputs "homework", the first document can be retrieved. That is, the accuracy of the search may be lowered. The same is true for K-grams as well as bi-grams.

Referring to FIG. 9B, an index is constructed using a bi-gram of the first document and compared with the keyword "homework " inputted by the user. Similarly, "ho", "rk", "wo", and "me" are common among the index of the document and the index of the search input by the user. Therefore, even if the first document does not have "homework" homework ", the first document can be searched for.

In contrast to this, a search using the NOVEL method proposed by the present invention will be described. In the case of the NOVEL method, "hs" is the house where the vowel is removed, "mrk" is the mark removing the vowel, "wrry" is the vowel removed vowel, "mtr" To build an index.

When the user enters "homework" as a search term, the search term also creates an index with a new search term "hmwrk" that eliminates the vowel, and compares it with the index of the document. In this case, since there is no common index between the two indexes, even if the user inputs "homework" as a search term, the first document is not displayed in the search result. That is, the accuracy of the search can be increased.

By using the NOVEL method, which is limited in the present invention, both the accuracy of search and the recall rate can be secured. That is, the efficiency of search can be increased. In addition, by reducing the size of the index by reducing the length of the keyword, search performance can be increased.

10A and 10B are diagrams for explaining the accuracy of each method with respect to a first data set including a plurality of keywords.

Referring to FIG. 10A, it can be seen that the user inputs "come" or "because" as a search word. In this case, if an index for a query is generated based on K = 3, the index is as shown below.

Here, "#co", the first index for "come", shows that documents including conditions from the communist can be retrieved even if the keyword "come" is not included. Similarly, "#be", the first index for "because" shows that documents including being from besides can be retrieved even if the keyword "come" is not included.

The first data set is a set of keywords composed of tens of thousands of words. As shown in FIG. 10A, it can be seen that, by using a K-gram, other words can be matched have. In other words, the accuracy of the search can be significantly reduced.

Referring to FIG. 10B, it can be seen that the indexes of the words are compared according to the NOVEL method proposed in the present invention for the same first data set. In the case of the first search term "come", an index is created based on the "cm" from which the vowel is removed. In the case of the second search term "because", an index is created based on the "bcs" from which the vowel is removed.

In this case, when each index is compared with {cm, c, m} and {bcs, bc, cs} and many other words included in the first data set, the index "cm" , Only four words are matched in the first data set: become, camera, come, and outcome. Can be seen to be significantly reduced compared to the K-gram method of FIG. 10A. Similarly, in the case of "bcs", which is the first index of "because", only the matching word is the search term in the first data set.

As described above, when the NOVEL method proposed in the present invention is used, it is possible to more accurately reflect the characteristics of the keyword while reducing the length of the keyword, thereby improving the accuracy of the search.

In summary, the NOVEL method proposed by the present invention is a fuzzy search method that increases search efficiency by ensuring accuracy and recall of search, and increases search performance by reducing the size of an index.

11 to 12 are graphs and tables showing test results for the first data set and the second data set including a plurality of keywords.

Referring to FIG. 11, it can be seen that the size of an index in the case of generating an index according to each method for two data sets is represented by a graph. The first data set and the second data set used here are test data composed of a set of tens of thousands of keywords.

As described in the previous formula, for the edit distance method, a 52L + 27 size index is generated for a keyword having a length of L. [ The index of each keyword is generated by the edit distance method for the first data set and the second data set including tens of thousands of keywords, and the union of the generated indexes is obtained as shown in FIG. That is, when the duplicated index is removed, it can be seen that an index of a size of 1,040,000 for the first data set and an index of 161,000 for the second data set are generated. The size of the index is so large that the performance of the search is significantly degraded.

For the wildcard method, create a 2L + 2 index for a keyword of length L. Therefore, when using the wildcard method, the size of the index can be reduced considerably compared with the edit distance method. Indeed, we have created an index of about 42,000 for the first data set and about 6600 for the second data set. Compared to the edit distance method, the size of the index is greatly reduced, but the index is still large and the search performance is degraded.

Next, an index of approximately L size is generated for a keyword having a length of L for K-gram and bi-gram. Therefore, it can be expected to be about half that of the wildcard method. Indeed, in the case of K-gram, we created an index of 24,000 for the first data set and about 370,000 for the second data set. In the case of bi-grams, we created an index of 15,000 for the first data set and for 24,000 for the second data set. Compared to the wildcard method, the size of the index is greatly reduced, but the size of the index is still large.

On the other hand, in the case of the NOVEL method proposed in the present invention, a 0.5-L index is generated for a keyword having a statistically L length. Indeed, it can be seen that an index of 0.8 thousand for the first data set and about 130 thousand for the second data set is generated. It can be seen that the size of the index is reduced to about half that of the K-gram method and the bi-gram method.

As shown in FIG. 11, the NOVEL method proposed by the present invention generates an index of the smallest size for the first data set and the second data set including tens of thousands of keywords. The smaller the size of the index, the faster the search can be performed. Therefore, the NOVEL method proposed by the present invention shows the best search performance.

FIG. 12 is a table showing the results of testing the K-gram method and the NOVEL method for the first data set and the second data set.

11, the NOVEL method calculates 8161 indexes for the first data set and 12926 indexes for the second data set, as shown in FIG. 11, Respectively. In contrast, the K-gram method based on K = 3 generated 24410 indexes for the first data set and 37424 indexes for the second data set. The search performance is also better because the index of the NOVEL method is smaller.

In the table of FIG. 12, the second row, Single Subindex with Multiple Keywords, is a result of counting the number of corresponding indexes when multiple keywords have the same index. That is, in the case of the NOVEL method, 1105 indexes among the 8616 indexes generated for the first data set are indexes generated by multiple keywords, and 1194 indexes among 12926 indexes generated for the second data set It is an index generated by duplication of keywords.

In contrast, in the case of the K-gram, 1442 indexes among 24410 indexes generated for the first data set are duplicates generated by a plurality of keywords, and 2330 indexes out of 37424 indexes generated for the second data set It is an index generated by duplication of several keywords.

If keywords such as "come" and "camera" shown in FIG. 10B that generate some identical indexes are input as search terms, documents containing other keywords can be searched, which reduces the accuracy of the search.

That is, in the table of FIG. 12, the larger the value of the second row, the lower the accuracy of the search. In general, the NOVEL method is lower than the K-gram method. In other words, the NOVEL method shows that the accuracy of the search is better.

In the table of FIG. 12, the third row, Average Keywords under Single Subindex, represents the average number of keywords having each index of the second row. That is, in the case of the NOVEL method, 1105 indexes share a keyword for the first data set, and five keywords are linked to one index on average for 1105 indexes.

That is, when "come" is input as a search term, five keywords having an index common to "come" including "camera" can be searched. In other words, the larger the value of the third row, the less accurate the search. Here again, the NOVEL method has a smaller value than that of the K-gram method, but the NOVEL method has a better search accuracy.

In the table of FIG. 12, the fourth row Maximum Keywords under Single Subindex shows a case where the number of keywords connected to one index is the largest. In this case as well, the larger the value of the third row, the lower the accuracy of the search. Overall, the NOVEL method is lower than the K-gram method. In other words, the NOVEL method shows that the accuracy of the search is better.

In the table of FIG. 12, the last row Unique Subindex represents the number of indices unique to each keyword. The greater the number of unique indexes, the higher the accuracy of the search results. However, here, since the indexes of the NOVEL method and the K-gram method are different from each other, it is necessary to focus on the ratio of the unique index to the total index by each method.

According to the first data set, the NOVEL method has 1864 indexes out of 8616 indexes as a unique index, which is 21.6%. In the K-gram method, among the 24410 indexes, the number of unique indexes is 2160, which is 8.8% . Since the ratio of the NOVEL method is higher than that of the NOVEL method, the accuracy of the NOVEL method is high.

As we have seen so far, the NOVEL method, which removes vowels and generates additional secondary indexes when generating indexes of keywords, improves the retrieval accuracy while increasing the retrieval efficiency. , And the search performance can be secured by reducing the size of the index.

13 is a diagram illustrating an example of a hardware configuration of an index generation apparatus based on a vowel removal method according to an embodiment of the present invention.

13, the index generation device 10 based on the vowel removal method may include one or more processors 510, memory 520, storage 560, and interface 570. The processor 510, the memory 520, the storage 560, and the interface 570 transmit and receive data via the system bus 550.

The processor 510 executes a computer program loaded into the memory 520 and the memory 520 loads the computer program from the storage 560. [ The computer program may include a keyword extraction operation 521, a vowel collection operation 523, and an index generation operation 535. [

The keyword extraction operation 521 may receive the document via the interface 570 and store it in the document 561 of the storage 560. The keyword may be extracted by applying a method such as TF-IDF to the stored document and stored in the keyword 563 of the storage 560 through the system bus 550.

The collection removal operation 523 may load the keyword 563 stored in the storage 560 into the memory 520 via the system bus 550 and then remove the vowel to generate a new keyword. In addition to eliminating vowels, deduplication can be used to remove duplicate characters.

The index creation operation 525 may generate an index by applying a K-gram or the like to a new keyword from which the vowel is removed, and store the index in the index 569 of the storage 560 through the system bus 550. At this time, an additional sub-index may be additionally generated according to the length of the keyword from which the vowel is removed, and may be stored in the index 569 of the storage 560.

The index 569 constructed for each document 561 can be utilized in a process of searching for and providing a document including an input search term when a specific keyword is input as a search term.

13 may be software or hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). However, the components are not limited to software or hardware, and may be configured to be addressable storage media, and configured to execute one or more processors. The functions provided in the components may be implemented by a more detailed component, or may be implemented by a single component that performs a specific function by combining a plurality of components.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Claims (15)

Removing the vowel from the keyword of the document and converting it into a keyword composed of consonants; And
Wherein the index generating device generates an index for the document using the converted keyword,
Wherein the step of generating an index for the document comprises:
And if the length of the converted keyword is smaller than a predetermined value, generating an auxiliary index larger than the index.
How to create a collection removal based index. The method according to claim 1,
The method of claim 1,
Wherein consonants consecutively appearing as duplicates in the keyword include removing one of them.
How to create a collection removal based index. delete The method according to claim 1,
The preset value is 3,
How to create a collection removal based index. The method according to claim 1,
Wherein the step of generating an index for the document comprises:
And generating the index by performing a K-gram on the converted keyword.
How to create a collection removal based index. 6. The method of claim 5,
The step of performing the K-gram to generate the index comprises:
Comparing whether the length L of the converted keyword is greater than a predetermined value m; And
And performing a K-gram with K = m for the transformed keyword if L is greater than m,
How to create a collection removal based index. 6. The method of claim 5,
The step of performing the K-gram to generate the index comprises:
Comparing whether the length L of the converted keyword is greater than a predetermined value m; And
Performing a K-gram to set K = L to the converted keyword if L is less than or equal to m,
How to create a collection removal based index. 8. The method of claim 7,
The step of performing a K-gram with K = L for the converted keyword,
Further comprising performing a K-gram with K = L-1 for the converted keyword if L is greater than 1,
How to create a collection removal based index. One or more processors;
A memory for loading a computer program executed by the processor; And
A storage for storing a document and an index for the document,
The computer program comprising:
An operation of removing a vowel from the keyword of the document and converting it into a keyword composed of consonants; And
And generating an index for the document using the converted keyword,
An operation for generating an index for the document includes:
And generating an auxiliary index larger than the index if the length of the converted keyword is smaller than a preset value.
A device for generating an index based on collection removal. 10. The method of claim 9,
An operation for generating an index for the document includes:
And generating an index by performing a bi-gram on the converted keyword.
A device for generating an index based on collection removal. 10. The method of claim 9,
The preset value is 3,
A device for generating an index based on collection removal. 10. The method of claim 9,
An operation for generating an index for the document includes:
And generating an index by performing a K-gram on the converted keyword.
A device for generating an index based on collection removal. 13. The method of claim 12,
The operation of performing the K-gram (K-gram)
Comparing whether the length L of the converted keyword is larger than a predetermined value m; And
And performing a K-gram of K = m for the converted keyword if L is greater than m,
A device for generating an index based on collection removal. 13. The method of claim 12,
The operation of performing the K-gram (K-gram)
Comparing whether the length L of the converted keyword is larger than a predetermined value m; And
And performing a K-gram to set K = L to the converted keyword if L is less than or equal to m,
A device for generating an index based on collection removal. A computer program stored on a medium for carrying out the method of any one of claims 1, 2 or 4 to 8 using a computer.

Images