KR20020054254A - Analysis Method for Korean Morphology using AVL+Trie Structure - Google Patents

Abstract

PURPOSE: A Korean morpheme analysis method is provided to analyze morphemes of the Korean by using a structure of a dictionary(AVL, Adelson-Velskii and Landis + Trie, reTrieval) for reducing a load of a word cluster engine with an input list generator of a word clustering engine. CONSTITUTION: The method comprises steps of extracting words from a sentence by separating words based on a space, a comma or other special characters(100), checking if each separated word is included in a compressed stop word dictionary formed of words having a high appearance frequency among no noun words, i.e. verbs, adjectives or adverbs by the full matching algorithm(103), if the word is not included in the compressed stop word dictionary, checking if the word is included in a noun dictionary by the longest matching algorithm(104), if the word is included in the compressed stop word dictionary, checking the next word(102), if the word is not included in the noun dictionary, determining if there occurs an inappropriate error at the words remaining after extracting the words included in the noun dictionary by referring to a general noun dictionary and a dependency morpheme dictionary(105), if there occurs an inappropriate error, registering the corresponding word at a non-registered word process module, extracting the nouns included in the noun dictionary(106), checking if there exists a list of adjacent nouns(107), designating the adjacent nouns as a noun phrase and checking if the noun phrase is available(108), and transmitting the extracted nouns or noun phrases to an application database(109).

Description

Analysis Method for Korean Morphology using AVL + Trie Structure}

The present invention relates to a Korean morpheme analysis method using a dictionary structure (AVL + Trie) and a computer-readable recording medium recording a program for realizing the method. More particularly, the present invention relates to an input list generator of a word clustering engine. The present invention relates to a Korean morphological analysis method using a dictionary structure (AVL + Trie) for reducing the performance burden of a cluster engine and a computer-readable recording medium recording a program for realizing the method.

As the Internet industry expands greatly and a huge amount of knowledge and information flows, the tendency to use the Internet is seeking a more convenient and personalized service route.

In the case of a retrieval system, the focus is shifting to a technology viewpoint that effectively provides useful and accurate information to users, rather than how much data it holds. The emerging clustering retrieval technology clusters highly related queries and documents for the queries thrown by information users and provides them with the user query search results.

The additional search results provided by the clustering engine can recognize related words and similar words when users are new to the new knowledge, thereby reducing the initial concept recognition burden.

Clustering retrieval technology that has an effect of reducing initial concept recognition burden is largely document clustering and word clustering technology. Among them, word clustering technology clusters words only to build a database and provides related documents through internal search. Therefore, the clustering database construction time is short, and the existing developed search engine can be utilized.

Looking at the prior art of the morpheme analyzer according to the field of application, the type of corpus, the language used are as follows.

First, the trend of application fields to which morphological analyzer is applied is as follows.

The morphological analyzer has reached the practical stage in most application systems, regardless of English or Korean, and English language processing research that has relatively few sentence structure transformations or morphological changes is more advanced than Korean language processing research.

The morphological analyzer, which has reached the level of practical use, has various algorithms and reference dictionaries depending on the purpose and extent of each application system applied (eg, spell checker, information retrieval, translation system, speech recognition system).

Morphological analyzers developed for spell checkers should be considered for accuracy first. Morphological analyzers developed for information retrieval require high speed and accuracy, while stemmers developed for translation systems are more likely to be used than translation speeds. Complete conversion matching, including idiom processing, should be done. Therefore, if the morphological analysis algorithm and the calculation result used in one system are shared in another system, performance should be reduced in terms of efficiency.

The morpheme analyzer for search engines directly related to the present invention can also be classified into various purposes, and an effective morpheme analyzer for each purpose is required. That is, the morpheme analyzer is a morpheme analyzer as an index word extractor for building an index database of a search engine, a morpheme analyzer as a preprocessor for automatic document abstract extraction, a morpheme analyzer as a preprocessor for a natural language query processor, or a word cluster engine as in the present invention. It is used in various ways such as a morpheme analyzer as an input word extractor.

Since the morphological analyzer can be used for such various purposes, some modular morphological analyzers have been developed so that the morphological analyzer used in one system can be used in other systems with portability. The development of customized individual morphological analyzers is mainly done.

Next, the trend of corpus applied morphological analyzer is as follows.

Corpus can vary depending on the degree of processing, by document characteristics, and by what it contains. First, depending on the degree of processing, the raw sentence corpus, which classifies documents according to raw raw corpus, corpus tagged tagging, corpus tagged tagging, categorized corpus, and basic verb list, In addition, depending on the characteristics of the document, there may be a short-length corpus such as a newspaper or a magazine and a corpus having a long length and various plots such as a novel. In addition, according to the contents contained in each field, information and communication, semiconductor, politics, economics, etc. can be categorized to form a corpus.

Currently, Korean corpus includes KTset 96, ETRI-Enlightening corpus from the Korea Economic Newspaper, corpus used by the 1st Morphological Analyzer and Tagger Evaluation Contest, and Cork cork from KORDIC. However, there has been no case of building a cluster engine by constructing a basic corpus of documents that contain specialized contents in the information and communication field.

The morphological analysis algorithm and dictionary structure applied to Korean are as follows.

At present, Korean morpheme analyzers in practical use can expect significant changes in processing speed depending on the algorithm used and the structure of the dictionary to be referred to. In particular, since Korean includes a large number of compound nouns, it must be fast to store and retrieve these compound nouns, and there must be a module that can handle the occurrence of unregistered words in the dictionary.

Typical algorithms include the longest matching method, the shortest matching method, the tabular parsing method, the head-tail division method, and the syllable-based analysis method. based Analysis Method), two-level morphology, etc. The pre-structures include a Tri structure, a hash structure, and a B + tree structure.

In order to process the unregistered words in the dictionary, the existing invention has been proposed a morphological analysis method to reduce the frequency of occurrence of unregistered words in the dictionary by estimating and classifying and storing them by type in addition to the word list that can be registered in advance.

However, the morpheme analyzers have small unregistered words, or are slow in search, and have problems in specialized morpheme analysis and new word processing.

The present invention has been proposed to solve the problems described above, and the dictionary-based Korean morphological analysis method using a dictionary structure (AVL + Trie) to reduce the performance burden of the word clustering engine and to improve the quality performance; It is an object of the present invention to provide a computer-readable recording medium having recorded thereon a program for realizing the method.

1 is a flow chart of an embodiment of a morphological analysis method according to the present invention.

2 is a diagram illustrating an embodiment of an AVL + Trie dictionary structure according to the present invention.

According to an aspect of the present invention, there is provided a dictionary-based Korean morpheme analysis method using an AVL + Trie structure applied to a morpheme analysis apparatus, the method comprising: extracting a word in word units; A second step of searching for the extracted word by referring to a compression stop word dictionary; A third step of searching for the extracted word by referring to a noun dictionary; Checking a syllable remaining after the noun dictionary reference search to determine whether an unsuitable error has occurred; A fifth step of processing a non-registered word when a non-conformance error occurs as a result of the checking in the fourth step; And a sixth step of extracting a noun and storing the noun in the database if the non-conformance error does not occur as a result of the checking in the fourth step.

In addition, another method of the present invention includes a seventh step of temporarily storing stop words that are not in advance; An eighth step of checking whether the term is registered in the terminology dictionary by referring to the terminology dictionary; A ninth step of storing a stop word having a high frequency of occurrence in the compressed stop word dictionary if it is registered in the stop word dictionary as a result of the checking in the eighth step; And as a result of the checking of the eighth step, the tenth step of moving to the manager module considering the new word if it is not in the stopword dictionary.

On the other hand, the present invention, the morpheme analysis system having a large processor, the first function for extracting words in word units; A second function of searching for the extracted word by referring to a compression stopword dictionary; A third function of searching for the extracted word by referring to a noun dictionary; A fourth function of checking whether syllable errors have occurred by checking the remaining syllables after the noun dictionary reference search; A fifth function of processing non-registered words when a non-conformance error occurs as a result of confirming the fourth function; And a computer-readable recording medium having recorded thereon a program for realizing a sixth function of extracting a noun and storing it in a database if a non-conformance error does not occur as a result of confirming the fourth function.

In addition, the present invention, the seventh function for temporarily storing the stopwords that do not exist in advance; An eighth function of checking whether a term is registered in the terminology dictionary by referring to the terminology dictionary; A ninth function of storing a stop word having a high occurrence frequency in the compressed stop word dictionary if it is registered in the stop word dictionary as a result of confirming the eighth function; And a computer-readable recording medium having recorded thereon a program for further realizing the tenth function of moving to a manager module, which is regarded as a new word if it is not found in the stopword dictionary as a result of the checking of the eighth function.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating an embodiment of a morpheme analysis method according to the present invention.

The morpheme analyzer in the present invention is an input word extractor for the word cluster engine, which extracts key nouns from a document. The core noun category is not based on general nouns, but is formed based on technical technical terminology in the field of information and communication, providing a foundation for providing specialized information in a systematic and efficient interface.

As shown in FIG. 1, in the morphological analysis method according to the present invention, if there is a sentence "analysis of impact on information and communication technology" 101 in a document, a word is extracted from the sentence in word units (100). . Words are separated by spaces, commas, and other special symbols. The word extraction is processed in document unit and sentence unit, and it is used to derive correlations and key sentences between adjacent words.

Then, beyond the next word (102), words that have a high occurrence frequency among words that are not nouns (eg, verbs, adjectives, adverbs, interjections, etc.) are composed of a compression stopword dictionary 122 and a noun dictionary 131 ) Before you refer.

If the occurrence frequency of the stopword is higher than that of the noun word having the high appearance frequency, the speed improvement can be achieved by referring to the stopword having the high appearance frequency before referring to the noun dictionary 131. Therefore, the compression stop dictionary 122 is small in size, and the reference algorithm also uses a simple perfect match method.

The threshold value for the stop word selection with high occurrence frequency is determined based on Equation 1 below.

In Equation 1, "n" is a noun, "N" is a noun dictionary, "d" is a stopword, "D" is a stopword dictionary, "k" and "q" are the frequency of occurrence of each word, and " max "represents the highest frequency value, respectively.

The non-compression stop words are checked for nouns by referring to the noun dictionary 131 (104). Here, the noun dictionary is largely divided into two types. That is, nouns containing general nouns and terminology in the ICT field.

In the noun dictionary reference step, a noun for a key term is to be extracted from the word clustering engine.

The professional nouns use the shortest match method and the general nouns use the shortest match method.

The longest matching method is a method of selecting the longest word among the words that are to be found and a result of the comparison between the words stored in the dictionary and the words stored in the dictionary. Compared with the shortest matching method, when extracting nouns, the number of extracted words is small and significant compound nouns can be extracted. Therefore, the longest matching method is effective for extracting compound nouns from sentences containing compound nouns.

If a word registered as a general noun comes before a noun, that is, when a "Korean-Japanese" (general noun) and "machine translation" (special noun) are present, such as "Han-il Machine Translation", If the search result is tagged as a noun, the longest word is extracted, and if it is tagged as a general noun, the shortest word is extracted.

The nouns are selected first, and if the general nouns are located in the front of the nouns, the nouns except for the general nouns can be extracted.

Next, extract the nouns registered in the professional noun dictionary in the longest way from a word and check the remaining syllables with reference to the general noun dictionary (131) and the dependent morpheme dictionary (132). In step 105, it is determined whether a nonconformance error has occurred. Here, the reference algorithm is, for example, "machine translation" is stored in the noun dictionary in the phrase "in a machine translation system", and when "machine translation" is extracted in the longest way, the "syllable system" through the remaining syllable checks. Is a common noun, and "of" is a dependent morpheme.

If, in the remaining syllable check step, the result of the analysis to the end of the word to be analyzed (105), if a non-conformance error occurs, it is performed again from the noun dictionary reference process (104). If the noun object to be analyzed is no longer analyzed or if an unsuitable error occurs until the end of the analysis target word, the entire word is passed to the unregistered word processing module 120.

After confirming the result of analysis (105) to the end of the word to be analyzed, the noun is transmitted to the application database.

On the other hand, the processing for the word delivered to the non-registered word processing module 120 is as follows.

Among the existing methods, a method of reducing the occurrence frequency of unregistered words as much as possible by estimating and storing them by type in advance is estimated and stored. In other words, it is a non-registered word not included in the dictionary such as a person's name and a city name, but it cannot be processed when it appears in a document. Instead, the method provides a non-registered word processor 120 to determine whether to add a non-registered word to a stopword dictionary or to treat it as a new word in real time when the non-registered word appears by type.

The non-registered word processor 120 is largely composed of two types. Terminology automatic classification module 130 and the manager module 125. The word moved to the non-registered word processor 120 first moves to a temporary stop word dictionary of the stop word automatic classification module 130 to determine whether the corresponding word is included in the stop word dictionary 121 (124), and if so, the frequency Increase. If not listed in the dictionary, the word is passed to the manager module 125, and the part-of-speech judging authority is given to the manager so that the manager can process it at a predetermined time interval.

The stopword dictionary 121 includes verbs, adjectives, adverbs, other parts of speech and noun words designated by the administrator as unnecessary among nouns.

Conventionally, there is a method of extracting a noun from a word extracted by word units and, if the noun is a compound noun, divides it into a single noun. However, in this method, it is important to use compound nouns as they are lost when Korean compound nouns are separated into single nouns. Therefore, not only extracts a single noun in a word, but also provides a compound noun combination to extract a compound noun that can be formed over multiple words.

To this end, previously uncompressed stopwords, dependent morphemes, and non-registered meaningful morphemes are tagged after the corresponding morpheme for compound noun combination processing.

Consecutive noun word lists that do not include the tag are noun combinations, since they are likely to be a single compound noun even though they are separated and extracted in the word-word extraction stage. The combined noun adds to the result noun list by referring to a specialized noun dictionary which is a meaningful compound noun.

The list of nouns and the frequency of appearance that appear in each sentence of each document is prepared and transmitted to the word cluster processing module.

2 is a diagram illustrating an embodiment of an AVL + Trie dictionary structure according to the present invention.

As shown in FIG. 2, the AVL + Trie dictionary structure combines the advantages of the existing Trie (reTrieval) structure and the advantages of the Adelson-Velskii and Landis (AVL) tree structure. Here, Trie is a useful data structure for storing, retrieving, and deleting Korean, including many compound nouns, because the search speed can be kept constant even if the key is of variable length.

In order to combine the advantages of the AVL structure with the Trie structure, a single AVL tree is constructed in place of each Trie node, and a small character set of syllable units or phoneme units can be stored, searched, and deleted in a range. . That is, when configuring the Trie step by syllable unit, it is configured as shown in FIG. 2 and refers to the AVL structure 210 configured for the first syllable to find the first syllable.

Each node of the AVL structure has link information for next syllables 220, 230, and 240, and jumps to a linked AVL tree having next syllable information using the link information. Also, each node of each AVL tree sets a separation tag (Taq) for syllables that can be separated into a single noun for spacing errors and separation into a single noun.

The AVL tree can use memory for storing, retrieving and inserting small amounts of data, which can significantly speed up prior use.

An example of the process of searching for a word using the AVL + Trie structure will be described below.

For example, the words contained in the dictionary are "information processing", "dependent form", "tag", "information communication", "single noun", "stopword", "synthetic", "informatization", "information search", If "precision" and the word you are looking for are "information processing", the first step searches for "jeong" in the first AVL tree 210 for the first syllable, and the second step retrieves the second syllable that "jeong" links to. Go to the second AVL tree 220 for and search for "Destination".

Next, go to the third AVL tree 230 for the third syllable that "Bo" is linking in the third step and search for "Look", and in the fourth step, the fourth AVL tree for the fourth syllable that "She" is linking to After moving to 240, "Lee" is searched.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be apparent to those of ordinary knowledge.

As described above, the present invention has the effect of reducing the initial concept recognition burden in searching and reducing the performance burden of the word cluster engine.

Claims (8)

In the Korean morpheme analysis method using a dictionary structure (AVL + Trie) applied to the morpheme analysis device, Extracting a word in a word unit; A second step of searching for the extracted word by referring to a compression stop word dictionary; A third step of searching for the extracted word by referring to a noun dictionary; Checking a syllable remaining after the noun dictionary reference search to determine whether an unsuitable error has occurred; A fifth step of processing a non-registered word when a non-conformance error occurs as a result of the checking in the fourth step; And A sixth step of extracting a noun and storing it in a database if a non-conformance error does not occur as a result of the checking in the fourth step; Morphological analysis method comprising a. The method of claim 1, A seventh step of temporarily storing stop words that are not in the dictionary; An eighth step of checking whether the term is registered in the terminology dictionary by referring to the terminology dictionary; A ninth step of storing a stop word having a high frequency of occurrence in the compressed stop word dictionary if it is registered in the stop word dictionary as a result of the checking in the eighth step; And As a result of the checking of the eighth step, if the dictionary does not exist in the stopwords, the tenth step is regarded as a new word and moved to the manager module. Morphological analysis method further comprising. The method according to claim 1 or 2, The second step, An eleventh step of checking whether the extracted word is completely identical with the extracted stopword dictionary; A twelfth step of proceeding to a process of extracting the next word if it is a perfect match as a result of the eleventh step; And A twelfth step of proceeding to the third step if the check result of the eleventh step does not match; Morphological analysis method comprising a. The method according to any one of claims 1 to 3, The third step, A thirteenth step of checking whether the extracted word is the longest match by referring to a noun dictionary; A fourteenth step of proceeding to the fourth step if the longest match is found as a result of the checking of the thirteenth step; And A fifteenth step of processing the unregistered language if it is not the longest match as a result of the checking of the thirteenth step Morphological analysis method comprising a. The method according to any one of claims 1 to 4, The process of referring to the noun dictionary, A morphological analysis method characterized by checking as the longest match method in a noun dictionary of a specialized field and by using the shortest match method in a noun dictionary in a general field. The method of claim 2, The process of storing the stopword in a compression stopword dictionary, A morpheme analysis method comprising selecting a stop word having a high frequency of occurrence according to a threshold value determined as in Equation 1 and storing the stop word in the compressed stop word dictionary. [Equation 1] Where "n" is a noun, "N" is a noun dictionary, "d" is a stopword, "D" is a stopword dictionary, "k" and "q" are the frequency of occurrence of each word, and "max" is the highest frequency value. Respectively. In a morphological analysis system with a large processor, A first function of extracting words in word units; A second function of searching for the extracted word by referring to a compression stop word dictionary; A third function of searching for the extracted word by referring to a noun dictionary; A fourth function of checking whether a non-conformance error has occurred by checking a syllable remaining after the noun dictionary reference search; A fifth function of processing non-registered words when a non-conformance error occurs as a result of confirming the fourth function; And A sixth function of extracting a noun and storing the noun in a database if a non-conformance error does not occur as a result of checking the fourth function A computer-readable recording medium having recorded thereon a program for realizing this. The method of claim 6, A seventh function of temporarily storing stop words that are not in the dictionary; An eighth function of checking whether a term is registered in the terminology dictionary by referring to the terminology dictionary; A ninth function of storing a stop word having a high frequency of occurrence in the compressed stop word dictionary if it is registered in the stop word dictionary as a result of confirming the eighth function; And A tenth function for moving to the manager module by considering a new word if it is not found in the stopword dictionary as a result of confirming the eighth function A computer-readable recording medium that records a program for further realization.