KR20080028655A - Method and apparatus for part-of-speech tagging - Google Patents

Abstract

A method and a device for tagging part-of-speech are provided to tag the part-of-speech correctly at high speed by reducing calculation quantity and error probability, as a morpheme of each word is analyzed when the part-of-speech is tagged, and a morpheme connection error in the word and between the words is reduced in a morpheme analysis result. A data storing part(130,140,150,160,180,190) stores a word preprocessing dictionary(131), a morpheme dictionary(141), part-of-speech connection information(151), an intra-word morpheme connection error list(161), part-of-speech trigram information(181), and a between-word morpheme connection error list(191). A morpheme analyzer(120) analyzes a morpheme by dividing the inputted sentence into words and checking whether the word is found in the word preprocessing dictionary, checks whether the morphemes can be connected by using part-of-speech connection information, and removes a morpheme connection error by using the intra-word morpheme connection error list. A part-of-speech tagger(170) selects the optimal morpheme analysis result, tags the part-of-speech of the optimal morpheme analysis result by using the part-of-speech trigram information, and removes an error from a part-of-speech tagging result by using the morpheme connection error list.

Description

Part of speech tagging device and tagging method {Method and Apparatus for part-of-speech tagging}

1 is a block diagram of a part-of-speech tagging apparatus of the present invention.

FIG. 2 is a diagram illustrating an example of a letter combination for performing morpheme analysis on 'gwasoG_L' by applying a tabular parsing method among conventional morpheme analysis methods.

FIG. 3 is a diagram illustrating a part-of-speech list result for 'speeding (gwasoG_L)' by the morpheme analysis of FIG. 2.

4 is a block diagram of a part-of-speech tagging device incorporating a first storage unit to a sixth storage unit in the part-of-speech tagging device of FIG. 1.

5 is a flowchart of a part-of-speech tagging method of the present invention.

Explanation of symbols on the main parts of the drawings

110: sentence input unit 120: morphological analysis unit

130: first storage unit 131: word preprocessing dictionary

140: second storage unit 141: stemming dictionary

150: third storage unit 151: parts of speech connection information

160: fourth storage unit 161: stemming combination error list in the word

170: part of speech tagging unit

180: fifth storage unit 181: parts of speech trigram

191: 6th storage unit 191: stemming error list between words

200: Part of speech tagging result output unit

The present invention relates to a part-of-speech tagging device and a tagging method, and more particularly, to a part-of-speech tagging device and a tagging method capable of outputting a part-of-speech tagging result with high speed and high accuracy.

Part-of-speech tagging refers to the process of assigning the correct part-of-speech information for each word, depending on the context in which the word is used in the sentence. Part-of-speech tagging is used in preprocessing to reduce the excessive burden in the parsing phase due to lexical ambiguity, and can be used for index and search words with high recall and accuracy in information retrieval systems. It is also recognized as an essential process in all areas of natural language processing such as machine translation, language information acquisition tools, spell checking, and dictionary construction.

In general, parts of speech tagging can be classified into statistical approach and rule-based approach, and hybrid approach that combines statistical and rule-based approaches. have.

The statistical-based part-of-speech tagging method probabilistically solves the neutrality problem of natural language processing by analyzing the raw or tagged corpus including real-world natural language usages and sub information and using the probability obtained by extracting statistical information about natural language. Way. This method can be applied to almost all linguistic phenomena, but the lack of quantity and query corpus that can adequately represent real-world linguistic phenomena reduces accuracy due to lack of data.

Statistics-based part-of-speech tagging methods can be classified into methods using only lexical probabilities, methods using autonomous learning of HMM (Hidden Markov Model), methods using context probabilities and lexical probabilities of N-gram, and other methods using neural networks. And the use of a fuzzy network.

The rule-based part-of-speech tagging method finds common principles or deterministic rules applied to natural language and solves the neutrality problem of natural language processing by using knowledge-based approach or constraint-based approach. Also called Constraint-Based Approach. This method is generally not robust because it is difficult to obtain consistent deterministic rules and it is not easy to control many rules well, but it is highly accurate for the phenomena to which rules can be applied. Recently, research has been actively conducted to automatically learn the tagging rules based on the construction of a large amount of raw corpus and tagged corpus which have become available, and the improvement of computer performance.

The integrated part-of-speech tagging method seeks to complement the strengths and weaknesses of the statistical and rule-based approaches by using both probability and statistical information and linguistic knowledge extracted from large amounts of data.

Korean Patent Registration No. 284769 (Registration Date: December 22, 2000) discloses an example of using a statistical-based analysis method of the morpheme analysis method. However, this statistical-based analysis method has the advantage that it is easier to implement, extend, and manage than the rule-based part-of-speech tagging method by using the part-of-speech tagging method using both the state-based part-of-speech tagging method and the path-based part-of-speech tagging method. Because of the method of performing part-of-speech tagging and path-based part-of-speech tagging, merging the tagging results to remove non-fit parts of a word that have been assigned two parts of speech from the merged result, or assigning suitable parts of speech, Due to this, there is a problem that it is difficult to apply in a part where a quick response such as voice synthesis is required because the part-of-speech tagging is slow.

In addition, since the statistic-based analysis method applies full search by word or applies Viterbi search or Viterbi beam search, there is a morpheme error in a word or a word. If there is a stemming error in the liver, there is a limitation that accurate part-of-speech tagging cannot be performed.

Accordingly, the present invention is to solve such a problem, the object of the present invention can be applied to a place where a fast response, such as speech synthesis, part-of-speech tagging that can output a high-speed and high accuracy part-of-speech tagging results An apparatus and tagging method are provided.

In order to achieve the above object, the part-of-speech tagging apparatus according to the present invention comprises: a sentence input unit for receiving a sentence to perform part-of-speech tagging; Word preprocessing dictionary including information pre-morphologically analyzed for each word, morpheme dictionary, part-of-speech access information that is information on accessibility between morphemes, morphological combination error list in word to eliminate morphological association error in word, A data storage unit for storing the parts-of-speech trigram information and the word-to-word stemming error list for removing the word-to-word stemming error; After the sentence inputted through the sentence input unit is divided into words, it is checked whether a separated word exists in the pre-word pre-processing dictionary, and the morpheme analysis is performed. Then, the morpheme analysis results are combined using the parts-of-speech connection information. A morpheme analysis unit to check whether this is possible and to remove an error from the morpheme analysis result using the morpheme combining error list in the word; And a part-of-speech tagging unit that selects a part-of-speech analysis result selected from the morpheme analysis results using the part-of-speech trigram information and removes an error from the part-of-speech tagging result by using the morpheme combining error list. It is characterized by including.

On the other hand, to achieve the above object, the part-of-speech tagging method according to the present invention, (a) receiving a sentence to perform the part-of-speech tagging; (b) dividing the received sentence into a word, and performing a morpheme analysis by checking whether the separated word exists in a word preprocessing dictionary in which the information pre-analyzed by each word is recorded; (c) checking whether the morpheme analysis is possible by using the part-of-speech connection information on the result of the morpheme analysis; (d) removing an error from the morphological analysis result using an intra-word morpheme combining error list; (e) selecting a part-of-speech tag by selecting the morpheme analysis result that is optimally determined from the morpheme analysis results using the part-of-speech trigram information; And (f) removing an error from the part-of-speech tagging result by using a word-to-word morphological combination error list.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a part-of-speech tagging apparatus of the present invention.

As shown in FIG. 1, the part-of-speech tagging apparatus of the present invention includes a sentence input unit 110, a morpheme analysis unit 120, a first storage unit 130, a second storage unit 140, and a third storage unit. 150, a fourth storage unit 160, a part-of-speech tagging unit 170, a fifth storage unit 180, a sixth storage unit 190, and a part-of-speech tagging result output unit 200.

The sentence input unit 110 receives a sentence or the like for part-of-speech tagging. The sentence input method may be implemented in various forms. The sentence input method may be input in an already made text form or may receive an actual sentence from a user through a keyboard.

The morpheme analysis unit 120 receives a sentence from the sentence input unit 110 and performs a morpheme analysis by separating the input sentence into a word using the word preprocessing dictionary 131 stored in the first storage unit 130.

Here, the word preprocessing dictionary 131 includes information morphologically analyzed for each word in advance, so that the word morphological analysis can be performed immediately without searching for the morpheme dictionary described below.

Next, when it is determined that the separated word does not exist in the word preprocessing dictionary 131, the morpheme analysis unit 120 separates the word into a syllable and then the syllable is the second storage 140. Whether the morpheme dictionary 141 stored in the morpheme dictionary 141 or a combination of syllables and other syllables is present in the morpheme dictionary 141. Information and so on.

That is, the morpheme analysis unit 120 performs morpheme analysis on the input sentence using information stored in the first storage unit 130 and the second storage unit 140. Here, the morpheme analysis may be applied to the head-tail classification method, tabular parsing method, longest match method, shortest match method, syllable unit analysis method, and each morpheme analysis method for understanding of the present invention. Briefly described as follows.

Head-tail delimiting is a method of separating morphemes from words. The word is classified into unmodified head and deformed tail, and the coupling relationship between head and tail. Morphological analysis is performed by checking the part-of-speech connection information table.

Tabular parsing is a bottom-up morpheme analysis method that first searches a dictionary for substrings of all the Jamo units that make up a word, and extracts all possible lexical and grammatical morphemes. After that, we apply the Cocke-Younger-Kasami (CYK) algorithm to find the morphemes of words. In other words, the extracted morphemes are stored in a triangular table and a method of finding a list of all connectable morphemes using a part-of-speech connection information table to check the connection relations of the morphemes. When estimating unregistered words, the remaining parts other than the parts that can be interpreted by the bottom-up method are estimated.

The longest coincidence method is to select and examine the longest morpheme among the sets of morphemes constituting the word, and the shortest coincidence method is to first check the shortest morpheme.

The syllable unit analysis method uses statistical syllable characteristics. The syllable unit analysis method refers to a method of processing syllable units according to the characteristics of Korean characters, not phonemes.

On the other hand, the morpheme analysis unit 120 checks whether the morphemes can be combined using the part-of-speech analysis information 151 stored in the third storage unit 150 with respect to the result of the morpheme analysis. The connection information 151 is information used to confirm whether the morphemes can be combined.

That is, the part-of-speech connection information 151 is information that lists information that can be connected between parts-of-speech. For example, a noun can be followed by a noun, a parent or investigation can be followed, a verb is followed by a verb, and an adjective is followed by an ending. , A term after the ending means a list of information such as endings. The list of morphemes that can be combined is output according to the part-of-speech connection information 151.

In addition, when the list of morphemes that can be combined according to the part-of-speech connection information 151 is output, the morpheme analyzing unit 120 uses the morpheme combining error list 161 in the word stored in the fourth storage unit 160. The error is extracted and removed from the list of combinable morphemes, which not only increases the accuracy of morphological analysis but also reduces the amount of calculation due to morphological overanalysis, thereby speeding up the POS tagging speed.

Meanwhile, when the morpheme analysis of the input sentence is completed through the morpheme analyzer 120 as described above, the result of the morpheme analysis is transmitted to the part-of-speech tagging unit 170, and the part-of-speech tagging unit 170 is the fifth storage unit 180. The part-of-speech tag is selected by selecting one of the morpheme analysis results that are optimally determined from the morpheme analysis results using the part-of-speech trigram information 181 stored therein.

Here, the part-of-speech trigram information means a value used in a trigram model in which one word is affected by two previous words.

Also, when the part-of-speech tagging is performed using the part-of-speech trigram information 181, the part-of-speech tagging unit 170 uses the interstem morpheme combining error list 191 stored in the sixth storage unit 190. By extracting and removing parts of speech tagging errors, parts of speech tagging errors are reduced.

The part-of-speech tagging result output unit 200 performs a function of outputting the part-of-speech tagging result from the part-of-speech tagging unit 170 to a monitor or other output device.

That is, the part-of-speech tagging apparatus of the present invention performs morphological analysis for each word when tagging parts of speech, and reduces the morphological coupling error and the morphological coupling error between words in the morpheme analysis result. Compared to the conventional method of tagging, the amount of calculation and the possibility of error can be reduced, and the part-of-speech tagging can be performed accurately at a high speed. To facilitate the understanding of the present invention, a brief description of a conventional part-of-speech tagging method is performed. Is as follows.

FIG. 2 is a diagram illustrating an example of a letter combination for performing morpheme analysis on 'gwasoG_L' by applying a tabular parsing method among conventional morpheme analysis methods, and FIG. 3 is a morpheme of FIG. 2. This is a diagram showing a part-of-speech list result for 'speeding (gwasoG_L)' by analysis.

Referring to FIG. 2, in order to apply a tabular parsing method, first, all the hairs are separated and combinations of the hairs are extracted. Then, it is determined whether the extracted combinations of vowels exist in the morpheme dictionary, and if they exist in the morpheme dictionary (for example, 'and' 210), the parts of speech in the morpheme dictionary are listed.

In FIG. 2, the rows 220, 221, 222, and 223 or the columns 230, 231, 232, and 233, which extract the combination of the letters, are not actually used, and are not necessary in the combination of the letters. , Part 240 in which no Hangul exists is an unused part. Thus, with so many combinations, searching in the morpheme dictionary is a waste of memory.

As shown in FIG. 3, 'and' may be a noun 310, an adjective 311, a verb 312, and 'speeding' may be a noun 313. Similarly, 'k', 'e', 'e' and 'ㄹ' are found in the morpheme dictionary and concatenated to the column. Next, check whether '// noun' is connected to 'pronoun / noun' and '// noun' and 'pron / verb' are connected. To confirm the connection, the part-of-speech connection information is used.

That is, the morpheme analysis and the part-of-speech tagging method using the above Jamo combinations require a large amount of computation because it generates a combinable list of morphemes and performs morphological analysis and part-of-speech tagging thereof, thereby performing the part-of-speech tagging. There is a problem that it is difficult to apply where a fast response such as speech synthesis is necessary because of the slow speed.

On the other hand, in order to compensate for the problem of the part-of-speech tagging speed, the tabular parsing method is modified to form morphemes by syllable units, and then find whether the composed syllables are in the morpheme dictionary, and only the morphemes that can be connected by using the part-of-speech access information. There is also disclosed a method for finding and interpreting a. This method also requires a large amount of computation, and there is a problem in that the accurate part-of-speech tagging cannot be performed when there is a morpheme error in a word or a morpheme error between words.

In conclusion, the part-of-speech tagging system used in the small-capacity synthesizer is not only difficult to have a large-capacity morpheme dictionary for morphological analysis, but also requires a large amount of calculation even if the part-of-speech connection information is used for morphological analysis. There is a problem that it is difficult to apply where a quick response is needed.

In contrast, the present invention performs morphological analysis for each word when tagging parts of speech, and by reducing the morphological coupling error and the morphological coupling error between words in the morphological analysis results, the calculation amount and the probability of error are reduced, thereby enabling accurate part-of-speech tagging at high speed. The present invention can be efficiently used in natural language processing systems that require fast response such as speech synthesis, speech recognition, and multilingual translator.

4 is a block diagram of a part-of-speech tagging device incorporating a first storage unit to a sixth storage unit in the part-of-speech tagging device of FIG. 1.

As shown in FIG. 4, the simplified part-of-speech tagging device includes a sentence input unit 410, a morpheme analysis unit 420, a data storage unit 430, a part-of-speech tagging unit 440, and a part-of-speech tagging result output unit 450. Doing.

Is it possible to combine the word preprocessing dictionary including the information pre-morphologically analyzed for each word, the morpheme dictionary containing the morpheme, the category of the morpheme, and the frequency information about the connected form, and the morphemes? The parts-of-speech connection information used to verify the information, the morpheme combining error list in the word for removing the morpheme combining error in the word, the parts of the speech trigram information, and the morpheme combining error list for the removal of the morpheme combining error between the words It is preferably stored.

The remaining components are the same as the corresponding components of the part-of-speech tagging device shown in FIG. In addition, since the sentence input unit 410 and the part-of-speech tagging result output unit 450 are not essential components in FIG. 4, the part-of-speech tagging apparatus of the present invention may be implemented.

Hereinafter, a part-of-speech tagging method according to the present invention will be described in detail with reference to the accompanying drawings.

5 is a flowchart of a part-of-speech tagging method of the present invention.

First, the sentence to perform the part-of-speech tagging is input through the sentence input unit 110 (S510). The sentence input method may be implemented in various forms. The text input method may be input in the form of a text file that is already made, or may be input directly from the user through the keyboard.

Next, the morpheme analysis unit 120 is a word preprocessing dictionary 131 in which information morphologically analyzed for each word is recorded in advance, and a morpheme dictionary 141 in which information on a plurality of morphemes and their morphemes is recorded. ), The morpheme analysis is performed by using the part-of-speech connection information 151 indicating whether the morphemes are accessible (S520).

First, the morpheme analysis unit 120 performs morpheme analysis by dividing an input sentence into a word using a word preprocessing dictionary 131 stored in the first storage unit 130. It is preferable that the word preprocessing dictionary 131 includes information morphologically analyzed in advance for each word.

If it is determined that the separated word does not exist in the word preprocessing dictionary 131, the morpheme analyzer 120 divides the word into syllables, and the syllable is the second storage 140. The morpheme analysis is performed by checking whether the morpheme dictionary 141 stored in the hexadecimal dictionary 141 or a combination of syllables and other syllables exists in the morpheme dictionary 141.

Here, the morpheme dictionary 141 preferably includes information on morphemes, categories of morphemes, and frequency information on morphemes, and in addition to syllable-based tabular parsing as a morpheme analysis method, a head-tail (Head) -tail classification, tabular parsing, longest matching, shortest matching, and syllable unit analysis can be used.

 Next, the morpheme analysis unit 120 confirms whether the morpheme analysis is possible by using the parts of speech connection information 151 stored in the third storage unit 150 with respect to the result of the morpheme analysis. According to 151, a list of morphemes that can be combined is output.

Next, when the list of combinable morphemes is output, the morpheme analyzing unit 120 extracts an error from the list of combinable morphemes using the intra-word morpheme combining error list 161 stored in the fourth storage unit 160. Remove it (S530).

Next, the part-of-speech tagging unit 170 performs the part-of-speech tagging (S540).

At this time, inside the first sentence of the input sentence to determine the parts of speech using all the analysis results obtained from the morpheme analysis unit 120, and the next word is the parts of speech trigram (trigram) information stored in the fifth storage unit 180 The part-of-speech is determined based on the part-of-speech statistical information of the previous word and the trigram value using (181).

In other words, the part-of-speech tagging performs a full search inside the word, and the next word performs the part-of-speech tagging using the part-of-speech sequence information having the highest value of the previous word. From the result of 120, the result of the morphological analysis which is determined to be optimal is selected and part-of-speech tagged.

Next, the part-of-speech tagging unit 170 extracts and removes an error of the part-of-speech tagging using the word-to-word morpheme combining error list 191 stored in the sixth storage unit 190 (S550). Will be reduced.

Finally, the part-of-speech tagging result output unit 200 outputs the part-of-speech tagging result to a monitor or other output device (S560). The step S560 of outputting to the output device is not an essential component, and therefore, an omission method according to the method of the present invention may be implemented.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium.

The computer-readable recording medium may be a magnetic storage medium (for example, a ROM, a floppy disk, a hard disk, etc.), an optical reading medium (for example, a CD-ROM, DVD, etc.) and a carrier wave (for example, the Internet). Storage medium).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

As described above, according to the present invention, by increasing the execution speed and accuracy of the part-of-speech tagging, by performing the morpheme analysis for each word during the part-of-speech tagging, by reducing the morphological coupling error and the morphological coupling error between words in the morpheme analysis results, Accordingly, compared to the conventional part-of-speech tagging method, the amount of calculation and the possibility of error can be reduced, so that the part-of-speech tagging can be performed accurately at a high speed. Therefore, the part-of-speech tagging can be efficiently used in natural language processing systems such as speech synthesis, speech recognition, and multilingual translator. .

Claims (9)

A sentence input unit configured to receive a sentence for part-of-speech tagging; Word preprocessing dictionary including information pre-morphologically analyzed for each word, morpheme dictionary, part-of-speech access information that is information on accessibility between morphemes, morphological combination error list in word to eliminate morphological association error in word, A data storage unit for storing the parts-of-speech trigram information and the word-to-word stemming error list for removing the word-to-word stemming error; After the sentence inputted through the sentence input unit is divided into words, it is checked whether a separated word exists in the pre-word pre-processing dictionary, and the morpheme analysis is performed. Then, the morpheme analysis results are combined using the parts-of-speech connection information. A morpheme analysis unit to check whether this is possible and to remove an error from the morpheme analysis result using the morpheme combining error list in the word; And The part-of-speech tagging unit selects a part-of-speech analysis result selected from the morpheme analysis results using the part-of-speech trigram information and removes an error from the part-of-speech tagging result by using the morpheme combining error list. Part-of-speech tagging device, characterized in that. The method of claim 1, Part-of-speech tagging device further comprises a part-of-speech tagging result output unit for outputting the part-of-speech tagging result to the output device. The method according to claim 1, wherein the morpheme analysis unit, If it is confirmed that the separated word does not exist in the word preprocessing dictionary, the word is divided into syllables and then whether the syllable exists in the morpheme dictionary, or whether a combination of syllables and other syllables exists in the morpheme dictionary Part-of-speech tagging device, characterized in that to perform a morphological analysis by checking. The method of claim 1, wherein the part-of-speech tagging unit, In the part-of-speech tagging, the first word of the input sentence is used to determine the part-of-speech using all the morphological analysis results, and the next word is based on the part-of-speech statistical information and the trigram value of the previous word using the part-of-speech trigram information. Part of speech tagging device, characterized in that for determining. (a) receiving a sentence for part-of-speech tagging; (b) dividing the received sentence into a word, and performing a morpheme analysis by checking whether the separated word exists in a word preprocessing dictionary in which the information pre-analyzed by each word is recorded; (c) checking whether the morpheme analysis is possible by using the part-of-speech connection information on the result of the morpheme analysis; (d) removing an error from the morphological analysis result using an intra-word morpheme combining error list; (e) selecting a part-of-speech tag by selecting the morpheme analysis result that is optimally determined from the morpheme analysis results using the part-of-speech trigram information; And and (f) removing the error from the part-of-speech tagging result using a word-to-word morphological combination error list. The method of claim 5, (g) outputting the part-of-speech tagging result to an output device. The method of claim 5, wherein step (a) comprises: Receiving a sentence in a text file format or a part-of-speech tagging method comprising the step of directly inputting a sentence from a user via a keyboard. The method of claim 5, wherein step (b) If it is confirmed that the separated word does not exist in the word preprocessing dictionary, the word is divided into syllables, and then whether the syllable exists in the morpheme dictionary or a combination of the syllable and another syllable exists in the morpheme dictionary Part of speech tagging method comprising the step of performing a morpheme analysis. The method of claim 5, wherein step (e) In the part-of-speech tagging, the first word of the input sentence is used to determine the part-of-speech using all the morphological analysis results, and the next word is based on the part-of-speech statistical information and the trigram value of the previous word using the part-of-speech trigram information. Part of speech tagging method comprising the step of determining.

Images