KR20140049148A - Method for part-of-speech tagging based on morpheme segmentation and apparatus thereof - Google Patents

Abstract

Disclosed are a word class tagging method based on morpheme separation and a device thereof. According to one embodiment of the present invention, the word class tagging method includes: a step of performing morpheme separation for a predetermined raw corpus; a step of searching for one or more documents similar to input text based on the result of the morpheme separation; a step of performing domain adaptation of a morpheme separation model and a word class tagging model using the searched similar documents; and a step of tagging word classes to the input text using the morpheme separation model and the word class tagging model to which domain adaptation is performed. The searching step uses a search database which is built by processing a group of documents in which the raw corpus is separated by morpheme based on the result of the morpheme separation or uses a plurality of clusters acquired by clustering the documents in which the raw corpus is separated by morpheme based on the result of the morpheme separation. By doing so, even if the word spacing among morphemes of the raw corpus is not clear, it is possible to perform domain adaptation of morpheme separation and word class tagging and to remove limitation of linguistic expandability. [Reference numerals] (AA) Start; (BB) End; (S110) Perform morpheme separation for a large-volume raw corpus; (S120) Search for documents similar to input text based on the morpheme separation result; (S130) Perform domain adaptation of a morpheme separation model and a word class tagging model using the searched similar documents; (S140) Tag word classes to the input text using the morpheme separation model and the word class tagging model to which domain adaptation is performed

Description

Part-of-speech tagging based on morpheme segmentation and its apparatus {Method for part-of-speech tagging based on morpheme segmentation and apparatus

The present invention relates to a part-of-speech tagging based on morphological division, and more specifically, by using only large-scale raw corpus that is not segmented and tagged, so that it can be applied to Korean or Chinese, which are not clearly defined. The present invention relates to a part-of-speech tagging method and apparatus therefor that can provide a domain adaptation scheme for a multilingual morpheme segmentation and part-of-speech tagging model.

The most commonly used part-of-speech tagging method is a statistical and dictionary-based approach, which consists of probabilistic information learned from the part-of-speech tagged corpus and a manually-built lexical dictionary. In order to effectively process sentences in various domains, the conventional part-of-speech tagging method according to the prior art technique is a) using a large amount of additional parts-of-speech corpus, or b) extending an existing lexical dictionary.

However, these conventional approaches increase the number of possible morphemes as candidates for analysis as the lexical dictionary scales, resulting in greatly increased morphological segmentation / part-of-speech tagging ambiguity.

In order to solve this problem, conventional systems have proposed a "domain adaptation method" of part-of-speech tagging, where the domain adaptation method does not use the original tagging model as it is, but modifies the probabilistic model to fit the domain of the input statement. As a way to reduce the morphological ambiguity that occurs, a) a method of bootstrapping a Hidden Markov Model (HMM) tagging model into a large primitive corpus, and b) a document-based local HMM model adaptation method have been proposed.

However, all of these conventional methods are limited to English-speaking languages with clear spacing between words, and thus there is a problem in that they cannot be extended to other languages. In particular, in the case of languages such as Korean or Chinese, the spacing between morphemes is not clear, which implies a morphological division problem.

Therefore, there is a need for a method that can cover not only part-of-speech tagging but also morphological division.

Korean Registered Patent No. 0784730 (Registration date 2007.12.05) Korean Laid-Open Patent No. 2011-0018140 (published 2011.02.23)

An object of the present invention, which was devised to solve the above problems, is to introduce a machine learning-based morpheme segmentation method, thereby enabling morphological segmentation and part-of-speech tagging even in languages where morphological spacing is not obvious in the target primitive corpus. The present invention provides a part-of-speech tagging method and an apparatus based on morpheme partitioning capable of domain adaptation.

Another object of the present invention is to provide a method and apparatus for tagging part-of-speech based on morphological division that can solve the limitation of linguistic extensibility by enabling domain adaptation of morphological division and part-of-speech tagging.

To achieve the above object, a part-of-speech tagging method according to an embodiment of the present invention comprises the steps of: performing morphological division on a predetermined raw corpus; Retrieving at least one similar document for an input statement based on the result of the morphological division performed; Performing domain adaptation on a morpheme division model and a part-of-speech tagging model using the retrieved similar document; And part-of-speech tagging the input statement using the domain-adapted morpheme division model and the part-of-speech tagging model.

The retrieving may include: processing a set of stemmed documents for the raw corpus based on the result of the stemming to build an index database; Generating a query by performing a morpheme division on the input statement; And searching for the similar document through the index database search using the generated query.

The generating may include obtaining a morpheme lattice structure by performing morpheme division on the input statement, and generating the query word by converting the obtained morpheme lattice structure.

The morpheme division may be performed based on a machine learning method including a conditional random field (CRF) for the raw corpus including the segmentation information.

In performing the domain adaptation, the domain adaptation may be performed by updating probability information applied to each of the morpheme division model and the part-of-speech tagging model with reference to the searched similar document.

The morpheme division model and the part-of-speech tagging model may use a Hidden Markov Model (HMM).

The retrieving may include clustering the stemmed document set for the raw corpus based on a result of the stemming to obtain a plurality of clusters; Performing stemming for the input statement; And retrieving a cluster corresponding to the stemmed input sentence among the plurality of clusters as the similar document.

A part-of-speech tagging apparatus according to an embodiment of the present invention includes a morpheme division unit that performs morphological division on a predetermined raw corpus; A retrieving unit for retrieving at least one or more similar documents for an input sentence based on the result of the morphological division performed; A domain adaptor for performing domain adaptation on the morpheme division model and the part-of-speech tagging model using the retrieved similar document; And a part-of-speech tagging part for tagging the input sentence using the domain-adapted morpheme division model and the part-of-speech tagging model.

Furthermore, the present invention further includes a database construction unit for processing an morphologically segmented document set for the raw corpus based on the result of the morphological segmentation and constructing an index database, wherein the searcher performs morphological segmentation on the input statement. The query may be generated, and the similar document may be searched through the index database search using the generated query.

Furthermore, the present invention further includes an acquisition unit for clustering the stemmed document set for the raw corpus based on the result of the morphological division to obtain a plurality of clusters, wherein the search unit is a stemmer for the input sentence. Partitioning may be performed, and a cluster corresponding to the stemmed partitioned input statement among the plurality of clusters may be searched as the similar document.

According to the present invention, by performing a morphological segmentation from a large-scale primitive corpus, by searching the document set most similar to the proposed input sentence and domain adaptation of the morphological segmentation / tagging model therefrom, beyond the limitation of the domain adaptation method limited to the conventional English-speaking language, The domain adaptation method can be applied to other languages such as Korean, Chinese, etc., where the boundary between morphemes is unclear.

Accordingly, the present invention can greatly broaden the multilingual scalability, thereby broadening the domain adaptation application range of part-of-speech tagging to practical multilingual-related applications requiring multilingual analysis such as machine translation.

1 is a flowchart illustrating an operation of a part-of-speech tagging method based on morpheme division according to an embodiment of the present invention.
Fig. 2 shows an operational flow diagram of an embodiment of step S120 shown in Fig.
3 shows the morphological division results for the input statement of "air conditioner temperature".
4 is a flowchart illustrating another embodiment of operation S120 of FIG. 1.
5 illustrates a configuration of a part-of-speech tagging apparatus based on morpheme division according to an embodiment of the present invention.
6 is a block diagram of a part-of-speech tagging apparatus based on morpheme division according to another embodiment of the present invention.

Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, the term "comprising" or " comprising " is intended to specify the presence of stated features, integers, steps, operations, elements, parts or combinations thereof, , But do not preclude the presence or addition of one or more other features, elements, components, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Hereinafter, a part-of-speech tagging method and apparatus thereof based on morpheme division according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 6.

1 is a flowchart illustrating an operation of a part-of-speech tagging method based on morpheme division according to an embodiment of the present invention.

Referring to FIG. 1, the part-of-speech tagging method according to the present invention performs a morpheme splitting module developed independently of a part-of-speech tagging to perform morphological splitting on a large-scale raw corpus composed of a web, a blog, and various newspaper articles (S110). .

In this case, the morpheme segmentation method used for corpus preprocessing may be obtained based on a machine learning scheme such as a conditional random field (CRF) in a corpus to which segment information is manually constructed.

Here, the CRF result may be classified into one of two states of each syllable divided and continued. In the case of "dividing", it means that the new morpheme starts from the next syllable, and in the case of "continuous", the next syllable is present. It may mean that it is included in the morpheme.

As the qualities used when applying the CRF method, basic dictionary information, front and rear left and right syllables (character) information, etc. may be used. <Table 1> below shows examples of qualities used in the morphological division process.

CRF Qualification Types Qualitative Value Meaning C ₀ Current syllable C ₀ C ₁ A bigram consisting of the current syllable and the next syllable C _-1 C ₀ A bigram consisting of the previous syllable and the current syllable C _-2 C _-1 C ₀ Trigram of previous two syllables and current syllable C ₀ C ₁ C ₂ Trigram with current syllable and next two syllables IsSuffixLexicon (C ₀ ) Is there a vocabulary for C ₀ as suffix? IsSuffixLexicon (C _-1 C ₀ ) Is there a vocabulary with suffix C _-1 C ₀ in the dictionary? IsPrefixLexicon (C ₀ ) Is there a vocabulary with C ₀ prefix in the dictionary? IsPrefixLexicon (C ₀ C ₁ ) Is there a vocabulary prefixed with C ₀ C ₁ in the dictionary?

Thereafter, at least one or more similar documents for a given input sentence are searched based on the result of the morpheme division performed by step S110 (S120).

In this case, the similar document search may obtain a morpheme lattice structure through morphological division of the input sentence, and search the top N documents most relevant to the input sentence using the obtained morpheme lattice structure.

In operation S130, domain information of the morpheme split model and the part-of-speech tagging model are collected by collecting the retrieved information of at least one similar document.

Here, the morpheme division model and the part-of-speech tagging model may use a Hidden Markov Model (HMM).

The domain adaptation method of the morpheme division model and the part-of-speech tagging model of step S130 may be performed by updating the probability information applied in each model with reference to the N-best search document.

Hereinafter, an example of a domain adaptation method in the case of using HMM as a morpheme division model and a part-of-speech tagging model will be described. For the sake of simplicity, the construction of the HMM based on the bigram transition probability is as follows.

[Equation 1]

Here, T means tag sequences t1, ..., tn, W means morpheme sequences w1, ..., wn, P (t | t ') means transition probability, and P (w | t) means lexical probability.

The lexical probability P (w | t) of the two probabilities may be approximated as in Equation 2 below.

&Quot; (2) &quot;

Where P (w) is the prior probability for morpheme w, P ₀ (w | t) is the lexical probability that was used before domain adaptation, and λ is the mixture ratio for the two probability information. Means a parameter.

When using HMM for morphological segmentation and part-of-speech tagging, the key to determining morphological segmentation is the lexical probability P (w | t). Therefore, the domain adaptation of the morphological segmentation model is to modify the prior probability P (w) by referring to the given N similar documents, and the domain adaptation model of P (w) is defined as Equation 3 below. Can be.

&Quot; (3) &quot;

Here, since the grid morpheme division results are assumed, the frequency is generally a real number rather than a binary number.

When the prior probability P (w) is calculated, the step of adapting P ₀ (w | t) and the transition probability P0 (t | t '), which are the remainder of the lexical probability, is performed. In this process, P ₀ (w | t) and the transition probability, the remainder of the lexical probability, are fixed by fixing the prior probability P (w) and performing an Expectation Maximization (EM) algorithm to maximize the probability of generating N search documents. P0 (t | t ') may be obtained, and may be obtained using bootstrapping schemes or the like already known to those skilled in the art.

As described above, the morphological division and part-of-speech tagging domain adaptation procedure can be summarized as follows.

1) P ₀ (w | t) learning: learning from a given part-of-speech tagged corpus

2) Similar Document Search: Search N similar documents for a given input statement

3) Morphological Segmentation Model Domain Adaptation: Changing the prior probability P (w) from N similar documents according to Equation 3 above

4) Tagging Model Adaptation: After fixing P (w), we construct an initial model from it, applying bootstrapping (or EM algorithm) to maximize the probability of generating N search documents, so that P ₀ (w | t ), Change P (t | t ')

The above-described domain adaptation process is an example of considering the HMM as a morpheme division model and a part-of-speech tagging model, and similar adaptation methods can be derived from a classification division and tagging model such as CRF.

When domain adaptation is performed, the part-of-speech tagging of the input statement is performed using the domain-adapted morpheme division model and the part-of-speech tagging model (S140).

At this time, the part-of-speech tagging of the input statement may be performed from the learned lexical probabilities P (w), P (w | t) and transition probabilities P (t | t ').

As described above, the part-of-speech tagging method according to the present invention performs morphological segmentation from a large-scale primitive corpus, retrieves a document set most similar to a given input sentence, and domain-adapts the morphological segmentation model and the part-of-speech tagging model therefrom, thereby making the boundary between morphemes unclear. The domain adaptation method can be applied to Korean, Chinese, and other languages, which can greatly increase multilingual scalability.

In addition, the present invention can extend the domain adaptation application range of part-of-speech tagging to practical multilingual-related applications requiring multilingual analysis such as machine translation.

In addition, since the present invention uses a lattice result that is not 1-best as a result of morpheme splitting, the domain-specific method proposed in the present invention is not susceptible to errors of morphological splitting and thus has various robustness results. Can be considered at the same time.

3 shows the morphological division results for the input statement of "air conditioner temperature".

Referring to FIG. 3, each vertex shown in FIG. 3 indicates a position of each syllable of an input sentence. In particular, the leftmost node is a special node and represents a start state of a sentence.

Here, the label of the edge connecting the two vertices may be shown as w: p, w is the stemming result of the syllables of the syllables in the vertex, p is a weight for the split result w The syllables between the two vertices are proportional to the posterior probability of dividing by w.

By utilizing the lattice structure as described above, it is possible to design a domain adaptation method that is robust to segmentation error by referring to more candidates than the simple 1-best or k-best morpheme segmentation results. The way of doing this can be regarded as a special case of the lattice structure.

2 is a flowchart illustrating an embodiment of operation S120 of FIG. 1.

Referring to FIG. 2, in the step S120 of retrieving a similar document, an index database is constructed by processing a set of stemmed documents for a large raw corpus by the stemming step S110 (S210).

Subsequently, for a given input statement, a morpheme lattice structure is obtained through morphological division, and a morpheme lattice structure for the obtained input statement is generated as a query word (S220).

When a query for the input statement is generated, the top N similar documents most relevant to the input statement are searched or obtained from an existing index database using a similar document search technique of information retrieval using the generated query (S230). .

In addition, as a step of performing a similar document search (S120), the entire raw corpus may be clustered and replaced with a method of classifying input statements. An example of the process will be described with reference to FIG. 4.

4 is a flowchart illustrating another embodiment of operation S120 of FIG. 1.

Referring to FIG. 4, the step S120 of retrieving similar documents may include document clustering on a stem set of documents that have been stemmed for a large-scale raw corpus by the stemming step S110. Acquire (S410).

Subsequently, for a given input statement, morphological division is performed to obtain a morphological lattice structure, and the document classification is applied to a plurality of clusters based on the divided morphological analysis results for the input statement to correspond to the divided input statement. By determining a cluster to replace, all document sets in the determined cluster are replaced or searched with N similar documents (S420 and S430).

5 illustrates a configuration of a part-of-speech tagging apparatus based on morpheme division according to an embodiment of the present invention.

Referring to FIG. 5, the part-of-speech tagging apparatus 500 includes a morpheme division unit 510, a database construction unit 520, a search unit 530, a domain adaptation unit 540, and a part-of-speech tagging unit 550.

The morpheme division unit 510 performs morphological division on a predetermined large-scale raw corpus.

In this case, the morpheme division unit 510 may perform morphological division based on a machine learning method including a conditional random field (CRF) with respect to the raw corpus including the partition information.

The database building unit 520 builds an index database by processing the stemmed document set for the raw corpus based on the result of the stemming.

The search unit 530 searches for at least one or more similar documents highly related to the input sentence based on the result of the morpheme splitting for the large-scale raw corpus.

In this case, the search unit 530 may generate a query by performing morphological partitioning on the input statement, and search at least one or more similar documents highly related to the input statement by searching the index database using the generated query.

The domain adaptation unit 540 performs domain adaptation on the morpheme split model and the part-of-speech tagging model using the retrieved at least one similar document.

In this case, the domain adaptor 540 may perform domain adaptation by updating probability information applied to each of the morpheme split model and the part-of-speech tagging model with reference to the retrieved similar document, and the morpheme split model and the part-of-speech tagging model may be HMM. (Hidden Markov Model) is available.

The part-of-speech tagging unit 550 parts-tagging an input statement using a domain-adapted morpheme division model and a part-of-speech tagging model.

6 is a block diagram of a part-of-speech tagging apparatus based on morpheme division according to another embodiment of the present invention.

Referring to FIG. 6, the part-of-speech tagging device 600 includes a morpheme division unit 610, an acquisition unit 620, a search unit 630, a domain adaptation unit 640, and a part-of-speech tagging unit 650.

The morpheme division unit 610 performs morphological division on a predetermined large-scale raw corpus.

In this case, the morpheme division unit 610 may perform morphological division based on a machine learning method including a conditional random field (CRF) with respect to the raw corpus including the partition information.

The acquirer 620 obtains a plurality of clusters by clustering the stemmed document set for the raw corpus based on the result of the stemming.

The search unit 630 searches for at least one or more similar documents highly related to the input sentence based on the result of the morpheme splitting for the large-scale raw corpus.

In this case, the search unit 630 obtains a morpheme lattice structure by performing a morpheme division on a given input statement, and applies a document classification to a plurality of clusters based on the result of the morpheme analysis split for the input statement. By determining the cluster corresponding to the divided input statement, all document sets in the determined cluster can be retrieved as N similar documents.

The domain adaptation unit 640 performs domain adaptation on the morpheme split model and the part-of-speech tagging model using the retrieved at least one similar document.

In this case, the domain adaptor 640 may perform domain adaptation by updating probability information applied to each of the morpheme split model and the part-of-speech tagging model with reference to the retrieved similar document, and the morpheme split model and the part-of-speech tagging model may be HMM. (Hidden Markov Model) is available.

The part-of-speech tagging unit 650 tags the input statement using the domain-adapted morpheme division model and the part-of-speech tagging model.

The part-of-speech tagging method according to an embodiment of the present invention may be implemented in the form of program instructions that may be executed by various computer means and may be recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In the present invention as described above has been described by the specific embodiments, such as specific components and limited embodiments and drawings, but this is provided to help a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations are possible from these descriptions.

Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

Claims (14)

Performing morphological splitting on a predetermined raw corpus;
Retrieving at least one similar document for an input statement based on the result of the morphological division performed;
Performing domain adaptation on a morpheme division model and a part-of-speech tagging model using the retrieved similar document; And
Part-of-speech tagging of the input statement using the domain-adapted morpheme split model and the part-of-speech tagging model
Part of speech tagging method comprising a. The method of claim 1,
The searching step
Processing the stemmed document set for the raw corpus based on the result of the stemming to build an index database;
Generating a query by performing a morpheme division on the input statement; And
Searching for the similar document through the index database search using the generated query word
Part of speech tagging method comprising a. 3. The method of claim 2,
The generating step
A part-of-speech tagging method comprising: obtaining a morpheme lattice structure by performing morpheme division on the input statement, and generating the query word by converting the obtained morpheme lattice structure. The method of claim 1,
The step of performing morphological division
The part-of-speech tagging method according to claim 1, wherein the morphological division is performed on the raw corpus including partition information based on a machine learning method including a conditional random field (CRF). The method of claim 1,
Performing the domain adaptation
The part-of-speech tagging method of claim 1, wherein the domain adaptation is performed by updating probability information applied to each of the morpheme division model and the part-of-speech tagging model with reference to the retrieved similar document. The method of claim 1,
The morpheme division model and the part-of-speech tagging model
Part-of-speech tagging method using HMM (Hidden Markov Model). The method of claim 1,
The searching step
Clustering the stemmed document set for the raw corpus based on the result of the stemming to obtain a plurality of clusters;
Performing stemming for the input statement; And
Retrieving a cluster corresponding to the stemmed input sentence among the plurality of clusters as the similar document;
Part of speech tagging method comprising a.
A morpheme division unit that performs morphological division on a predetermined raw corpus;
A retrieving unit for retrieving at least one or more similar documents for an input sentence based on the result of the morphological division performed;
A domain adaptor for performing domain adaptation on the morpheme division model and the part-of-speech tagging model using the retrieved similar document; And
A part-of-speech tagging part for tagging the input sentence using the domain-adapted morpheme division model and the part-of-speech tagging model
Part of speech tagging device comprising a. 9. The method of claim 8,
A database construction unit for processing an morpheme-divided document set for the raw corpus based on the result of the morphological division to construct an index database
Further comprising:
The search unit
A part-of-speech tagging device, characterized in that to generate a query by performing morphological partitioning on the input statement, and to search the similar document by searching the index database using the generated query. 10. The method of claim 9,
The search unit
A part-of-speech tagging device, characterized in that to obtain a morpheme lattice structure by performing morpheme division on the input statement, and convert the obtained morpheme lattice structure to generate the query word. 9. The method of claim 8,
The morpheme division
The part-of-speech tagging apparatus according to claim 1, wherein the morphological division is performed on the raw corpus including partition information based on a machine learning method including a conditional random field (CRF). 9. The method of claim 8,
The domain adaptation unit
The part-of-speech tagging apparatus according to the retrieved similar document, wherein the domain adaptation is performed by updating probability information applied to each of the morpheme division model and the part-of-speech tagging model. 9. The method of claim 8,
The morpheme division model and the part-of-speech tagging model
Part-of-speech tagging device, using HMM (Hidden Markov Model). 9. The method of claim 8,
An acquisition unit for clustering the stemmed document set for the raw corpus based on a result of the morphological division to obtain a plurality of clusters
Further comprising:
The search unit
A part-of-speech tagging apparatus, wherein the morpheme division is performed on the input sentence, and a cluster corresponding to the morphologically divided input sentence among the plurality of clusters is searched as the similar document.

Images