KR100328963B1 - Korean stemming method and device thereof - Google Patents

Abstract

The present invention relates to a spacing error-tolerant Korean morpheme analysis method using an extended part-of-speech network, wherein an extended part-of-speech network incorporating the parts-of-speech and part-of-speech weights in addition to intra-word parts is used for generation and filtration of analysis candidates. As a result, analysis candidates can be generated even in a word where the spacing is abnormal as in a word error word. By calculating the suitability of each candidate by using the transition weight of the extended part-of-speech network, it is possible to obtain correct analysis results. It does not affect the analysis accuracy or analysis speed of words.

Description

Korean morphological analysis method and device therefor

The present invention relates to a spacing error-allowed Korean morphological analysis method and apparatus using an extended part-of-speech network, and in particular, a morphological analysis provided with a means for analyzing a word having a spacing error, that is, a word having a spacing error. It is about a method.

On the other hand, as a prior patent, the domestic patent "morphological analysis system and analysis method using the characteristics of the word structure [right holder Korea Telecommunications Corporation, Registration No. 123238, 97.09.11]" and "morphological analysis method using the frequency information [Applicant Samsung Electronics (Note) ), Publication No. 98-004120, 98.03.30] "and" Method of morphological analysis of Korean sentences [Applicant Geumseongsa, Publication No. 92-5023, 92.03.28] ", etc. The analysis using the table only analyzes the well-spaced words, which has a very narrow analysis range.

In addition, as a precedent paper, "Korean Spelling Correction System [Author Lee Young-sik, Chae Young-sook, Yoon Ae-sun, Kwon, Hyuk-chul, pp. 25-38, 1993]" published in the Journal of AI Research, published an article on the implementation of the spell checker. However, by using a dictionary to insert a space character at the boundary of the stem by dividing the morpheme, the error word can be restored to the original word, but the processing time is very simple by using the method of inserting a space by separating the words. It is long, and there is a problem in that it is impossible to deal with multiple word spacing errors.

In the prior art, since only normal words with a correct spacing are to be analyzed, when a spacing error occurs, that is, when two words to be spaced are added and written as one word, the analysis fails.

In order to solve this problem, the conventional technique uses a method of analyzing morphemes through estimation and finding word boundaries when the analysis fails.

First, in the estimation method, there is a method of estimating a word that has failed to be analyzed as a proper noun such as a person's name or a place name by considering it as a vocabulary that is not in the dictionary.

However, if there is a misleading error such as "using a powerful engine", and a word that fails to analyze is estimated as "using a powerful engine / proper noun", there will always be wrong results.

A more advanced estimation method is to use a clue of function words such as surveys and endings to estimate all the first strings as unknown nouns or unknown verbs (or adjectives). Since it is assumed to be a noun such as "a powerful engine, a noun, and a search", it is always necessary to give a false analysis result for the word of the error.

The word boundary classification method analyzes the input word until one unit word is formed, and if the remaining strings remain, it is regarded as an error word and is analyzed again from the next position.

In this method, one unit word can be formed at each syllable position in Korean language, so it is necessary to repeat the analysis process several times, and the same process must be repeated to determine whether it is an error word or not. It is difficult to secure the real-time performance of the processing due to a large computational load, and difficult to deal with errors with more than three words, making it difficult to use practically due to the poor analysis success rate.

In order to solve the above problems, an object of the present invention is to be able to analyze not only normal words without spacing errors but also spacing / pasting error words in the same manner.

Another object of the present invention is to ensure the real-time processing of the processing by performing the analysis without a decrease in the analysis processing speed as compared to the general case of analyzing a word with a spacing / pasting error only normal words .

Another goal is to minimize the number of parse failure words by analyzing spacing / paste error phrases that frequently appear in general documents.

Another object of the present invention is to minimize the number of estimated nouns that are forcibly estimated after the first analysis failure.

Another object is to provide human-oriented document processing technology by allowing the author to flexibly create a document without forcing the document creator to observe the spacing principle for the automatic processing of the document.

1 is a hardware block diagram according to the present invention,

2 is a flowchart of a word analysis method on a morpheme analysis apparatus to which the present invention is applied;

3 is a matrix structure diagram of an extended part-of-speech transition network used in the application of the morpheme analysis method of the present invention;

Figure 4 is an illustration of the morpheme analysis of the Korean word in accordance with the present invention.

* Explanation of symbols for main parts of the drawings

10: stemming device 11: dictionary storage device

12: expanded part-of-speech network 13: string storage device

14: analysis candidate generation module 15: analysis candidate storage device

16: fitness evaluation module 17: result filtration module

18: Final result storage device 20: String input device

30: display device

In order to achieve the above object, the present invention provides a morphological analysis apparatus for processing an error with a spaced word, and the candidate for morphological analysis by comparing dictionary information stored in a dictionary storage device with a string input through a string input device. If the stemming list creation process fails, but the stemming analysis module analyzes in 1-PASS without distinguishing between normal words and spacing error words, instead of attempting to modify or create a candidate stem list in the next step. ; And a result filtering module for filtering an analysis candidate to select an appropriate analysis candidate among the analyzed morphological analysis candidates and storing a final result, without distinguishing between normal words and abnormal words (suffix error words) in the morphological analysis module. When analyzing at -PASS, even when at least two words are written by using a transition network expressed by integrating the grammatical transition relationship and weight of normal word and abnormal word without using error type, case and pattern. It is characterized by analyzing by PASS method.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a hardware configuration diagram according to the present invention, and is largely composed of a morpheme analysis apparatus 10, a character string input apparatus 20, and a display apparatus 30.

The morpheme analysis apparatus 10 compares the string storage device 13 storing the string input through the string input device 20 with the stored information and dictionary information stored in the dictionary storage device 11. An analysis candidate generation module 14 for generating candidates, an analysis candidate storage device 15 for storing analyzed candidate results, a fitness evaluation module 16 for performing a fitness evaluation on stored analysis candidates, and an evaluation result And a result filtration module 17 for selecting suitable analysis candidates, and a final result storage device 18 for storing the filtered results.

First, the expanded part-of-speech transition network 12 will be described in detail with reference to FIG. 3.

In the present invention, the part-of-speech is information indicating which parts of speech may follow after a part of speech in a Korean word, and includes "intra-word transition" and "inter-word transition". have.

In-word transition indicates which part of speech can follow after a part of speech within a normal word, and between parts of a word followed by a part of a trailing word after the last part of the preceding word when two words are entered. It is a part-of-speech transition that is not observed within normal words.

In the extended part-of-speech transition network of FIG. 3, transition information is expressed using a matrix, where each row represents a pre-transition part-of-speech and each column represents a post-transition part-of-speech.

The matrix value indicates whether a part-of-speech transition can occur from a part-of-speech corresponding to a row to a part-of-speech corresponding to a column. If the transition cannot occur, the value is indicated by -∞.

When a transition can occur, a "transition weight" is set as an integer value in a matrix element, which is a numerical value expressing how appropriate the transition is for forming Korean words.

In principle, this value is equal to or greater than zero if the part-of-speech transition is a word transition, that is, if it conforms to the principle of forming Korean word phrases, or if it is attached to a word without a word between words. If it is a transition that can occur, it is configured to have a negative value less than zero. In this case, even in the case of an intra-word transition, the weight may be negative if the transition probability is small.

If the transition weight is -∞, it means that no transition can occur in any case.

For example, in the extended part-of-speech transition network matrix of FIG. 3, the matrix value of (NN, JO) is 1, which may be followed by JO (irradiation) morpheme after NN (noun) within a word, where the transition weight value is Indicates 1

Also, the matrix value of (JO, VV) is -5, which means that it is possible to follow VV (verb) after JO (irradiation), but it is less than 0, so it is a transition between words from word to word. Able to know.

(VV, NN), on the other hand, has a weight of -∞, indicating that NN (noun) can not be followed by VV (verb) in any case, regardless of error.

Since the part-of-speech transition network, also called word structure information or morpheme automata, has only intra-word transitions in the prior art, the word-to-word transition and transition weight information are subtracted from the extended part-of-speech transition network 12 used in the present invention. It is like the part-of-speech network used by technology. In the present invention, the inter-word transition and the transition weight information are allowed for the reason, of course, in order to deal with error.

For example, there are four words, such as "power", "good", "engine", "each", each of which is a correct word in terms of unit words, but four words are attached to one word, such as "using a good engine". If discarded, the conventional method does not allow the transition of (NN, AJ), (EM, NN), (JO, VV), etc. in the part-of-speech transition network, so that analysis cannot be performed.

However, the extended part-of-speech transition network 12 of the present invention allows the transition of (NN, AJ), (EM, NN), (JO, VV) and the like, so that such words can be analyzed.

However, as can be seen from the (JO, VV) transition, the part-of-speech transition itself is allowed, but the weighting value is less than 0, so that the transition network is configured to be distinguished from the normal part-of-speech part of speech transition.

As a result, the normal part-of-speech transition within the input word with correct spacing is given a score of 0 or higher, which increases the goodness-of-fit score of the whole word, and the part-of-speech transition occurring at the word boundary is given a negative score. Its function is to lower the goodness of fit.

However, although the score is low, -∞, that is, it is not treated as an inparable word, is different from the conventional technology.

2 is a flowchart of a word analysis method on a morpheme analysis apparatus to which the present invention is applied.

First, a word separated by a space is separated from a string input by a string input device (S1), and a substring of the separated word is searched in a dictionary to generate a candidate analysis list.

Because each part string can be analyzed by several parts of speech, the combination of them can produce a large number of candidate candidates for analysis.

In this case, however, the list without the part-of-speech transition in the extended part-of-speech network is excluded from the candidate for analysis.

The set of analysis candidate lists thus created is called L = {l ₁ , ..., l _n } (S2), and _i is initialized to 1 to evaluate the goodness of fit for each analysis candidate list l _{i that} is an element of L ( S3).

After determining whether i is greater than n (S4), if it is not large, a random analysis candidate list is obtained by l _i = {m ₁ / t ₁ , ..., m _ni / t _ni } (where li∈L and m _j are A stemmed string, t _j is a part-of-speech) (S5).

In list l _i , t ₁ is the word beginning (WB) morpheme, and t _ni is the word ending (WE) morpheme.

Since there are n _i elements in this list, a total of n _i -1 parts-of-speech transitions (t _j , t _{j + 1} ) (1≤j <n _i ) are generated, and it is determined whether j is greater than n _i (S6). If it is not large, adding the transition weight of the part-of-speech transition (t _j , t _{j + 1} ) to s _i , the goodness-of-fit value of l _i (S7), and then increasing the value of j (S8) to the end of the list, the candidate list The goodness-of-fit values s _i for l _i are calculated.

If j is greater than n _i after the determination (S6), the value of i is increased (S9) to continue calculating the goodness of fit of the next li.

When this process is repeated for all candidate analysis lists l _i of L (S4), a goodness-of-fit value s _i is calculated for each candidate list, from which a final result should be selected.

First, sort all candidates list l _i as the s _i in descending order for the selection of final results (S10) the largest value in the ordered s _i S ₁ is determined to greater than or equal to 0 (S11) S ₁ is greater than zero, the normal since available analysis results for Eojeol s _i ≥ 0 a l _i a is selected as the final result is (S12), s _i is less than zero ttuibut it available on analysis of the error s _i is as large sequence What is necessary is to select it as a final result (S13).

However, in the latter case ttuibut error Eojeol because there is more than Eojeol attached Eojeol length is long, and therefore the number of analyzes candidate also very much because all of Choosing the invention and difficulties of the car the little things than 9 and s ₁ useful to have somewhere Only analysis candidates are selected as the final result (S13).

The selected final result is stored in a separate storage device (S14), and then the result is output through a display process such as screen / printing (S15).

4 is a diagram illustrating an example of a morpheme analysis of a Korean word according to the present invention, which shows an example of analysis when an error word "using a powerful engine" is input.

First, a morpheme for each substring is obtained by referring to a dictionary (S1), and a plurality of analysis candidate lists are generated by the combination (S2).

However, in this process, it can be seen that a large number of candidate lists can be excluded in advance through access information or part-of-speech network information (S2).

Since the conventional invention does not allow the transition between words, virtually all candidates are excluded in this process, and the estimation results in an incorrect analysis result using "no" as a noun and "no" as a noun.

However, since the present invention uses an extended part-of-speech network that allows inter-word transitions, it can be seen that candidate morphological lists can be properly obtained even in a word having a spacing and pasting error (S2).

The goodness-of-fit value is calculated for the candidate morpheme list thus obtained (S4), and among these, the one with the high goodness-of-fit value is selected to obtain the final result (S10).

As described above, according to the analysis apparatus of the present invention, since the analysis candidate is generated for a word having a spacing error and the most suitable candidate is selected, the word without the spacing error is of course not included. You can also analyze words.

In addition, the conventional technology performs a large amount of performance deterioration because the processing is performed once the analysis fails, or the analysis is repeated several times to find the word boundary, but the present invention is the same as when processing an error-free word even for an error word. It allows you to do this in a way that allows you to analyze words with spaces and paste errors while showing little performance degradation.

In addition, the number of nouns that are forcibly estimated after a primary analysis can be greatly reduced, which can be effectively used for extracting index words from automatic indexing of large documents.

In addition, it allows for flexible document creation without compulsory document compulsory compliance for document processing, which can be effectively used to implement human-oriented document processing technology.

Claims (2)

In the stemming device for handling errors with words How to create a stemming candidate by comparing the string input through the string input device with dictionary information stored in the dictionary storage, but if the stemming list creation process fails, try to modify or create the candidate stem list again in the next step. A morphological analysis module for analyzing 1-PASS without distinguishing between normal words and spacing error words; A result filtration module for selecting an appropriate analysis candidate among the analyzed morphological analysis candidates and filtering the analysis candidates to store a final result, In the morphological analysis module, when analyzing 1-PASS without distinguishing between normal word and abnormal word (knit error word), the grammatical transition relationship between normal word and abnormal word without using error type, case, pattern, etc. A spacing error tolerant Korean morphological analysis device, characterized in that the analysis is performed by using the 1-PASS method even if at least two words are added by using a transition network in which weights are integrated. In the Korean morpheme analysis method that automatically analyzes the morphemes and parts of speech in the words of the input text, A first step of separating a word separated by a space from a string input by a string input device; A second step of searching for a substring of the separated word in a dictionary to generate a set of analysis candidate lists; A third step of expressing an arbitrary analysis candidate list as a part-of-speech transition (morpheme string / word ending morpheme) for each of the candidate lists which are elements of the analysis candidate list set; A fourth step of obtaining a goodness of fit value for the set of candidate lists by repeating the process of adding the transition weight of the part-of-speech transition resulting from the elements of the candidate list to the goodness of fit value of the set of analysis candidate lists to the end of the list; A fifth step of sorting the analysis candidate list set in descending order to select the final result of the calculated fitness values for each candidate list and selecting the fitness value as the final result when the sorted fitness values are greater than zero; A sixth step of selecting final results in ascending order of goodness of fit as an analysis result for spacing and pasting errors if the sorted goodness of fit value is smaller than zero; And a seventh step of outputting the result through a display process such as screen / printing after storing the selected final result in a separate storage device.