KR20140079545A - Method for Multi-language Morphological Analysis and Part-of-Speech Tagging based on conventional decoding scheme - Google Patents

Abstract

The present invention relates to a method for part-of-speech tagging and, more specifically, to a method for multi-language morphological analysis and part-of-speech tagging based on a phrase-based decoding mode. According to the configuration of the present invention, tagging ambiguity can be significantly reduced by taking phrases in part-of-speech tagging units without selecting words with high tagging ambiguity in tagging units and, additionally, various qualities can be effectively used by generalizing an existing word-based HMM model and proposing a robust probabilistic framework using a phrase-based decoding algorithm.

Description

TECHNICAL FIELD [0001] The present invention relates to a multilingual morphological analysis and part-tagging method based on decoding,

More particularly, the present invention relates to a multi-lingual morpheme analysis and part-of-speech tagging method based on a phrase-based decoding method.

In addition, the present invention has the potential to significantly reduce the tagging ambiguity by taking a word having high ambiguity of tagging as a tagging unit instead of taking it as a tagging unit.

In addition, the present invention generalizes the existing word-based HMM model, suggests a robust stochastic framework that utilizes the phrase-based decoding algorithm, and can utilize various qualities effectively.

The most popular part - of - speech tagging method is a statistical and dictionary - based approach. It consists of probabilistic information learned from partly tagged corpus and a manually constructed prefix dictionary.

Most of these general word-based tagging approaches are word-based approaches, which allow to attach parts of speech to individual words or morpheme.

However, in this word-based method, since tagging is performed on a word-by-word basis, when the ambiguity of the word is high, the tagging ambiguity of the part-speech is greatly increased, and consequently, the performance of the part-speech tagging can be greatly reduced.

In order to solve this problem, in the existing Korean analysis, a preliminary analysis dictionary for a long word or a partial word is constructed, and when the input statement is given, I suggested how to do it.

However, such a word-based or partial-word-based approach fails to define an integrated probabilistic framework such as a decoder used in speech recognition and statistics-based translation in the process of finding the optimal analysis result by combining the results of previous analysis, There is a disadvantage in that the process of searching for optimal analysis results is not mathematically simple, heuristic, and can not effectively integrate various qualities.

In addition, there is a disadvantage that it is not clear how to extend to the multi - word beyond the boundary of the word and the partial word.

Also, there are limitations in that multilingual scalability is lacking because the applied language is limited to Korean.

1. Korean Patent No. 10-0282610

1. Shin, Joon-Cheol, and Oh, Chul-Young "Korean morphological analyzer using partial-phrase dictionary", Journal of Korean Information Science Society, vol 39/5, pp 415-2424, 2012.5

The present invention has been proposed in order to solve the problems described in the background art, and provides a multilingual morpheme analysis and part-of-speech tagging method based on a decoding method that overcomes technical limitations of a general method in terms of efficiency of parts tagging and multilingual scalability It has its purpose.

In order to solve the problems of the above-described word-based and word-based / partial-word-based analysis methods, the present invention considers that adjacent words can be a phrase, There is another purpose of providing a phrase-based part-of-speech tagging method of tagging as a unit.

In order to solve the above problems, the present invention solves the problems of the above-described word-based and word-based and partial-phrase-based analysis methods, The present invention provides a phrase-based part-of-speech tagging method for tagging a whole phrase as a unit.

The phrase-based tagging method includes a tag conversion model construction step of extracting a pair of sphere and corresponding analysis result from a corpus with part speech and constructing a tag conversion model;

Constructing a morpheme and a tag language model by extracting a morpheme column and a tag row from the corpus with part speech and constructing a morpheme and a tag language model; And

Performing decoding on the input statement using the tag conversion model, the morpheme, and the tag language model to determine an optimal analysis result;

And a control unit.

According to the constitution of the present invention, the tagging ambiguity can be significantly reduced by taking the phrases having a high ambiguity of tagging as a per-tagging unit instead of taking them as a unit of tagging.

Another advantage of the present invention is that the existing word-based HMM model is generalized, and a robust probabilistic framework utilizing the phrase-based decoding algorithm is presented, which can advantageously utilize various qualities effectively .

In addition, another effect of the present invention is that it has multi-language scalability that can be applied to all languages such as Korean, English, and Chinese.

1 is a flowchart illustrating an example of a phrase-based decoding scheme according to an embodiment of the present invention.
FIG. 2 shows an example of a phrase-based decoding process for an example of an input inquiry " go home " according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Like reference numerals are used for similar elements in describing each drawing.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. The term "and / or" includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Should not.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

Further, in one embodiment of the present invention, a process of learning a phrase-based tagging model from corpus attached to a part of speech and performing phrase-based decoding on an input statement will be described with an example.

1 is a flowchart illustrating an example of a phrase-based decoding scheme according to an embodiment of the present invention. Referring to FIG. 1, an embodiment of the present invention includes the steps of 1) extracting pairs of corresponding analysis results from corpus with part of speech, and learning a phrase-based tag conversion model, 2) extracting morphological and tag sequences from part- Learning the morpheme and tag language model; and 3) performing the phrase-based decoding on the input statement to determine the optimal analysis result.

First, a phrase and an analysis result pair are extracted for a partly-attached corpus 101, and a phrase-based tag conversion model is constructed from the obtained phrase and analysis result (steps 102 and 104).

In general, the corpus 101 with part-speech is an important basic data for natural language processing, information retrieval, and language-related information processing. To do this, you usually go through the steps of morphological analysis, automatic part-tagging, manual or automatic error correction.

At the same time, words and tag sequences are extracted from the corpus 101 with the same part-of-speech, and word and tag language models are constructed therefrom (steps 103 and 105).

After the learning step is completed, if an input statement is input (106), the phrase-based part tagging decoder is executed referring to the conversion model and the tag language model learned previously (steps S106 and S107).

Finally, the speech tagging result is obtained by the tagging decoder (step 108).

Here, the process of extracting the phrases and the pair of tag analysis results from the corpus (step 102) includes all possible n-gram word strings (for English) or n-character syllable strings (for Korean) limited to the maximum maxPhraseLength length in a given sentence, This candidate becomes a candidate.

maxPhraseLength is the maximum number of words in a word string, and the maximum syllable length in a syllable string. For example, in the case of the Korean word "go home", the syllable column shown in the following table can be the old candidate (only the subset is described).

Syllable column (phrase) Analysis result candidate (subset) At home House / NNG + to / JKB, house / NNB + to / JKB, house / NNP + to / JKB House Home / NNG, Home / VV, Home / NNB on To / JKB, to / NNG, to / IC At home House / NNG + to / JKB_sl / VV, house / to NNG + / JKB_ hear / VV Get in Enter / VV + d / ETM prong Forward / NNG, E / VV + EA / EF, E / VV + EA / EC field / XSN, S / VV, H / VV Get into the house Home / NNG + to / JKB_int / VV + / ETM Ra R / EF, R / EC Go EC / VX + / ETM, EC / EC + / VV + / ETM Go / VV + d / ETM, go / NNA, I'm coming in. Enter / VV + KIRA / EC

For example, in the case of English, a syllable column can be a candidate for a syllable, as shown in the following table (only a subset is described), as an example of a "good book for natural language processing" sentence.

phrase Analysis result candidate (subset) good good / RB, good / NN, good / RB, good / JJ book book / NN, book / VB good book good / JJ book / NN natural natural / JJ, natural / NN natural language processing natural / JJ language / NN processing / NN 언어 처리 language / NN processing / NN book for natural language processing book / NN for / IN language / NN processing / NN

Meanwhile, the process of extracting the morpheme and the tag sequence from the corpus 101 with part-of-speech (step 103) is a process of sequentially arranging the morpheme and tags appearing in a sentence with a given part-of-speech.

For example, if the correct answer analysis result for the above Korean "go home" is given as "house / NNG + / JKB / VV + 래 RA / EC", the extracted morpheme and tag sequences are shown in the following table.

Type Extracted sequence column Morpheme column Go home Tag column NNG JKB VV EC

If the correct answer analysis result is given as "good / JJ book / NN for / IN natural / JJ language / NN processing / NN" for the above "good book for natural language processing" The columns are shown in the following table.

Type Extracted sequence column Morphological column (word column) good book for natural language processing Tag column JJ NN IN JJ NN

라 하고, 대응하는 태그 분석 결과를

이라고 할 때, 구 기반 변환 확률

은 다음식과 같이 계산된다.The spout, building parsed based tags from the resulting pair conversion process model (step 104) there is a form of words based obtain the transition probability calculation process based on generalized manner of conventional HMM (H idden Markov Model), to obtain

And the corresponding tag analysis result

, The phrase-based conversion probability

Is calculated as follows.

는 품사 부착 코퍼스(101) 에서 구

의 분석 태그가

인 빈도수를,

는 코퍼스 상 분석 태그가

인 빈도수를 가리킨다. here,

In the corpus 101 with part speech,

Analysis tag of

However,

The corpus analysis tag

Quot;

Each old tag conversion model constructs an index inverse file separately, so that a fast reference can be made to the input text thereafter.

On the other hand, the process of constructing a word and / or tag language model from the extracted morpheme and / or tag sequence (step 105) may be performed using a smoothing technique widely applied in general statistical machine translation and speech recognition, .

에 확률 값

을, “형태소 언어 모델”은 임의의 형태소열

에 확률값

을 계산하는 모델이다.In more detail, the " tag language model "

Probability value

, &Quot; morpheme language model " is an arbitrary morpheme column

Probability

.

The phrase-based tagging decoder explains an example of a process of searching for an analysis result having the highest probability for a given input statement using a conversion model and a language model obtained through the above-described process (steps 104 and 105).

Fig. 2 is an example showing a sphere-based decoding process for an example of an input inquiry " Go home " according to an embodiment of the present invention. Referring to FIG. 2, the contents in the square box 200 show an analysis result chart for the syllable column, and an edge 210 connected between the square boxes indicates a combinable path between adjacent charts.

More specifically, the phrase-based tagging method can be considered as an algorithm to solve the optimization problem of the following.

는 태그 언어 모델을 이용하여 계산되고,

는 임의의 “태그 분석 결과”

에서 임의의 “구 분할 결과”

의 변환 확률로, 다음식과 같이 정의된다. Equation (2) is an optimization example using only a tag language model,

Is calculated using the tag language model,

Quot; tag analysis result "

Quot; segmentation result "

And is defined as the following equation.

는 구기반 태그 변환 확률로, 앞서 정의된 수학식 1을 이용하여 계산된다.
At this time,

Is calculated using Equation (1) defined above with the phrase-based tag conversion probability.

In addition, an example of adding the " morpheme language model " by expanding Equation (2) is as follows.

은 태그 분석 결과

에 대응되는 형태소열을 가리키며,

는 태그/형태소 언어 모델 각각의 확률 자질에 대한 가중치를 의미한다.here

Tag analysis result

≪ / RTI >< RTI ID = 0.0 >

Is a weight for each probability property of each tag / morpheme language model.

The phrases in the present invention are systematic concepts similar to phrases used in statistics-based translation instead of the phrases used in linguistics, and are defined differently for individual languages.

In the example of the present invention, an arbitrary word sequence is defined for a language such as English or French, and an arbitrary continuous syllable string (including a space) is defined for a language such as Korean and Chinese.

Especially in Korean, since the arbitrary syllable string is taken as a syllable, multiple syllables can be regarded as one syllable and can be simultaneously tagged in one unit, which is more common than the conventional word-based tagging method.

In particular, the multilingual morphological analysis and part-of-speech tagging method based on the decoding method according to the present invention can be implemented in a form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

The medium may be a transmission medium such as an optical or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, or the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

101: Corpus with parts
102: Sphere, analysis result pair extraction step
103: Morphological and tag column extraction step
104: Steps of building old-based tag conversion model
105: Construction of morpheme and tag language model
106: Entry step of input statement
107: Sphere-based tagging stage
108: Tagging result step

Claims (1)

A tag conversion model construction step of extracting a pair of corresponding analysis result pairs from a partly attached corpus and constructing a tag conversion model;
Constructing a morpheme and a tag language model by extracting a morpheme column and a tag row from the corpus with part speech and constructing a morpheme and a tag language model; And
Performing decoding on the input statement using the tag conversion model, the morpheme, and the tag language model to determine an optimal analysis result;
And a multilingual morpheme analysis and part-of-speech tagging method based on a decoding method.

Images