KR101740330B1 - Apparatus and method for correcting multilanguage morphological error based on co-occurrence information - Google Patents

Abstract

A multilingual morpheme analysis error correction apparatus and method based on word air information is disclosed. The multi-lingual morpheme analysis error correction method based on the word air information according to the present invention comprises steps of constructing a hypothetical candidate dictionary for the words in which the ambiguity occurs, extracting air information from the large-scale primitive corpus Generating a morpheme analysis result for the input sentence, and modifying the morpheme analysis result based on the air information.

Description

[0001] APPARATUS AND METHOD FOR CORRECTING MULTILANGUAGE MORPHOLOGICAL ERROR BASED ON CO-OCCURRENCE INFORMATION [0002]

The present invention relates to a multilingual morphological analysis error correction apparatus and method based on word air information. In particular, the present invention relates to a multilingual morpheme analysis error correction apparatus and method based on word air information that is not dependent on a specific language by introducing a morpheme analysis method based on an air information.

The morphological analysis method that is most widely used at present is a statistical and dictionary based approach. It consists of probabilistic information learned from partly tagged corpus and a manually constructed dictionary of words.

This conventional morpheme analysis method is mostly a word-based approach in which parts of speech can be attached to individual words or morpheme.

However, in this word-based method, the tagging is performed on a word-by-word basis. Therefore, when the degree of the word is high, the degree of ambiguity in determining the part of the word is greatly increased.

In particular, in the case of a language in which two or more morphemes are combined to form a single word, the ambiguity of the word is considerably increased, and the morphological analysis performance may be degraded by increasing the complexity in the morphological segmentation and tagging process .

In order to solve this problem, the existing morphological analysis methodology constructs a preliminary analysis dictionary for a longer word or partial word phrase than words, and combines the analysis candidate results of each preliminary analysis dictionary with the input sentence, And the like.

However, in the case of two or more analysis candidates in the preliminary analysis dictionary, these phrases or partial word-based schemes are used in combination with the integrated analysis such as decoders used in statistical based methods, Since the probabilistic framework can not be defined, the process of searching for the optimal analysis result is not mathematically simple, heuristic, and can not effectively integrate various qualities.

In addition, even if this methodology is applied, there are phenomena that it is difficult to generate correct analysis results because the size of context information that can be considered in the analysis process is limited.

For example, let's take a look at high-level sentences such as "Let's go to the mountain or take a rest at home."

In the sentence above, the phrase "to go" can be analyzed in three ways: "thin + you", "go +", and "+ go".

However, it is difficult to decide which one of these three analysis candidates is accurate without knowing the meaning of the sentence precisely, and it is difficult to solve the problem through the existing preliminary analysis dictionary or probabilistic method.

In addition to these problems, the method of combining with the new processing methodology in the existing morphological analysis process also becomes a problem in the multilingual scalability.

In addition, Korean Patent No. 0474823 describes a speech tagging apparatus and a method thereof in a natural language, but it is limited in that there is a restriction that the error correction data must be prepared by manual operation in advance.

Therefore, there is a need for a technique that overcomes the technical limitations of the existing methods described above in terms of efficiency of tagging tagging and multilingual scalability.

An object of the present invention is to improve the performance compared to the existing morphological analysis by solving the limitation of the multilingual scalability by introducing the morpheme analysis method based on the air information.

It is also an object of the present invention to provide a morphological analysis methodology or a technique which does not depend on a specific language, by taking a modification form of the morphological analysis result after analyzing the morpheme.

According to another aspect of the present invention, there is provided a method for correcting errors in multilingual morpheme analysis based on word air information, the method comprising the steps of: constructing an ambiguous candidate dictionary for a word in which ambiguity occurs; Extracting air information from a large-scale original corpus, generating a morpheme analysis result for the input sentence, and modifying the morpheme analysis result based on the air information.
At this time, the ambiguity candidate dictionary may include either a medium word existing in the large-scale raw corpus and a search or an end corresponding to the medium word.
In this case, the step of generating the morpheme analysis result may generate the most optimal analysis result among the candidates generated using the words existing in the input sentence and the ambiguous candidate dictionary.
At this time, the large-scale raw corpus can be extracted from at least one of a web site, a blog, and a newspaper article.
In this case, the step of modifying the morpheme analysis result may extract a word phrase existing in the ambiguity candidate from the morphological analysis result, and modify the morpheme analysis result based on the air information dictionary for the extracted word phrase.
At this time, the air information dictionary can extract words that are not found in structural ambiguity and morphological ambiguity among morphological analysis results of sentences, and can store the words.
In addition, according to an embodiment of the present invention, a multilingual morpheme analysis error correction apparatus based on word air information generates an analysis candidate for a word in which an ambiguity occurs in an input sentence, and outputs a morphological analysis result for each of the analysis candidates Generating morpheme analysis candidate generation module; An optimal solution selection module for selecting one analysis result using the ambiguous candidate dictionary generated based on the morpheme analysis result of each of the analysis candidates and the ambiguous word extracted from the large scale corpus; And an air information based modification module for modifying the morpheme analysis result based on the selected analysis result and the air information.
At this time, the ambiguity candidate dictionary may include one of an ambiguous word existing in the large-scale original corpus and a search or an end corresponding to the ambiguous word.
At this time, the large-scale raw corpus can be extracted from at least one of a web site, a blog, and a newspaper article.
At this time, the air information based correction module can extract a word phrase existing in the ambiguity candidate from the morphological analysis result, and modify the morphological analysis result based on the air information dictionary for the extracted word phrase.

According to the present invention, by introducing the morpheme analysis method based on the air information, it is possible to improve the performance compared to the existing morphological analysis by solving the limitation of the multilingual scalability.

In addition, according to the present invention, it is possible to provide a morphological analysis methodology or a technique which does not depend on a specific language, by taking a modification form of the morphological analysis result after analyzing the morpheme.

1 is a flowchart of a multilingual morpheme analysis error correction method based on word air information according to the present invention.
2 is a block diagram of a multilingual morpheme analysis error correction apparatus based on word air information according to the present invention.
3 is an example of error correction when a morpheme analysis error occurs according to the present invention.

The present invention will now be described in detail with reference to the accompanying drawings. Hereinafter, a repeated description, a known function that may obscure the gist of the present invention, and a detailed description of the configuration will be omitted.

Embodiments of the present invention are provided to more fully describe the present invention to those skilled in the art.

Accordingly, the shapes and sizes of the elements in the drawings and the like can be exaggerated for clarity.

1 is a flowchart of a multilingual morpheme analysis error correction method based on word air information according to the present invention.

Referring to FIG. 1, a multi-lingual morpheme analysis error correction method based on word air information according to the present invention includes a step of constructing a tentative candidate dictionary for a word in which a truism occurs, (S20) of extracting air information from a large-scale raw corpus with a sentence, generating a morpheme analysis result (S30) on the input sentence, and modifying the morpheme analysis result based on the air information (S40) .

2 is a block diagram of a multilingual morpheme analysis error correction apparatus based on word air information according to the present invention. 3 is an example of error correction when a morpheme analysis error occurs according to the present invention.

2, the multilingual morpheme analysis error correction apparatus 100 based on the word air information according to the present invention includes a morpheme analysis candidate generation module 110, an optimal solution selection module 120, and an air information based modification module 130, .

Here, the morpheme analysis candidate generation module 110 performs a function of generating all possible analysis candidates when the ambiguous analysis is possible in the input sentence.

In addition, the optimal solution selection module 120 performs a function of selecting only one analysis result that is the most probable among all possible analysis candidates.

The air information based correction module 130 performs a result verification of the air information based on the morpheme analysis result generated by the optimal solution selection module 120. If it is determined that there is an error in the morpheme analysis result, It performs error correcting function.

2 and 3, the overall operation of the multilingual morpheme analysis error correction method based on the word air information according to the present invention will be described.

In the step S10, a hypothetical candidate dictionary is constructed for the words in which the hypothesis occurs.

Specifically, referring to FIG. 2, a morpheme analysis candidate generation module, which is a morphological analysis result up to the time of selecting an optimal solution selection module, is executed for a large capacity raw corpus composed of a web, a blog, Construct an inner word dictionary. The Morphological Analysis Candidate Generation module generates all possible analysis candidates if possible, if possible, in the middle of the input sentence.

Input phrase:
Analysis Candidate: E / V + / E
Go / V + / E
Thin / A + N / E Input phrase:
Analysis Candidates: Thin / A + N / E
Go to / V + to / E
Is / V + / E Input phrase: Buying
Analytical Candidate: Sa / N + / J
Company / V + / E
Buy / V + / E Input
Analysis Candidate: State / N + / J
Note / V + / E
Line / V + / E V: verb, E: mother, N: noun, J: investigation

In other words, for the input word 'go', it is possible to perform three median analyzes with '/ V + / E', 'Gal / V + is / E', and 'A / N + A / E'.

In addition, for the input word 'Gonni', there are three possible analyzes of 'Gin / A + Nin / E', 'Gal / V + Ein', and 'Gin / V + Ein'.

In addition, for the input verb 'to buy', there are three possibilities for analysis of 'S / N + / J', 'S / V + / E' and 'S / V + / E' ', It is possible to perform three median analyzes of' main / N + / J ',' main / V + / E 'and' line / V + / E '.

The generated word dictionary analysis code is stored as shown in the following example.

Example: Key: Go, Value: / V + / E, Go / V + / E, Thin / A +

In step S20, the air information is extracted from the large-scale raw corpus with respect to the words in which the ambiguity occurs.

Specifically, an example of extracting the air information dictionary and the air information to be used for modifying the morphological analysis result of the air information based on the morpheme analysis result in step S40 will be described.

Here, the air information dictionary refers to a dictionary composed of a noun and a phrase. For example, an entry of the form 'apple / N + / J / V + da / E' constitutes an air information dictionary.

The air information dictionary is performed in contrast to the one for constructing the median word dictionary. That is, the air information dictionary is constructed using only analysis results that have no morphological or structural ambiguity. A specific example will be described with reference to Table 2 below.

Sentence: Let 's go to the mountain or go home. Mountain Go At home Go Mountain / N + on Thin / A + Ne / E
Go to / V + to / E
Is / V + / E Home / N + to / J / V + character / E

As shown in Table 2, in the case of 'acid + plus / minus / a', the air information is excluded because of morphological ambiguity.

In the selection of air information, various heuristics can be applied to judge the existence of structural ambiguity. Examples of the corresponding heuristic are 'noun + connection ending to list', 'noun + adjacency + adjunct connection adverb', 'noun + , 'Noun + classifier adverb ending mother'.

Other examples of extracting air information are shown in Table 3 below.

Sentence: Even if you are angry, let 's get angry. artist Even me Anger Subject Tue / N + / J
Painter / N Even if I / V + / E Tue / N + / J True / V + character

As shown in Table 3, in the case of 'painter', there are two analytical ambiguities such as 'pain +' and 'painter'. That is, only 'T / N + / J J / V' is extracted as air information.

However, if only the case where there is no such ambiguity is extracted as the air information, there is a problem that the air information such as 'fire + is out' can not be constructed.

In this case, the air information can be extracted by applying the following two methods.

1. How to extract air information when context can be solved through context

In the case of the sentence "Let's pray for anger even if it is angry" above, there is a doubt that "painter" and "painter" occur in the case of "painter" Since it is judged that the possibility of 'fire + a' 'is higher than the possibility of' painter ',' air + fire 'can be extracted as air information.

For example, if there is a basis for solving the ambiguity in the current sentence, extract the air information, and 2) there is a basis for solving the ambiguity before the present sentence. , Air information extraction, and 3) heuristic such as air information extraction may be applied when there is a basis for solving the problem after the present sentence.

2. How to extract air information including assistant

Although the air information description method described above is directed to only the question mark, when extracting the air information including the assistant, it is possible to extract the air information which is difficult to extract due to the ambiguity such as 'fire +.'

For example, it is possible to extract air information in the form of 'fire + manna' through the sentence such as' I am angry ', and through this, information that can solve the ambiguity of' fire + .

Examples of various types of heuristics including these auxiliary information are: 1) noun + auxiliary node 2) noun + adjunct termination noun 3) noun + auxiliary adjunct.

The extracted air information is stored in the air information dictionary, and the form stored in the air information dictionary is as follows.

Vocabulary air information 1: noun + inquiry

Home / N + / J / V 100

Lexical information 2: noun verb

House / N / V 150

Meaning air information 1: Meaning of noun +

$ Place + to / V 200

Meaning air information 2: Noun Meaning verb

$ Place / V 300

Here, the semantic air information is generated by referring to the lexical air information and the meaning of the noun of the lexical air information. In this case, except for cases where the noun semantics are ambiguous, .

Referring to FIG. 2, in step S30, the morpheme analysis result is generated for the sentence input through the optimal solution selection module 120. [

In this case, the optimal solution selection module 120 is a module for selecting only one analysis result which is the most probable among all possible analysis candidates. Table 4 below shows an example of the optimal solution selection result.

Input sentence Go to the mountain or go to your house. A word unit Mountain Go At home Go Morphological analysis
Candidate generation module Mountain / N + on Thin / A + Ne / E
Go to / V + to / E
Is / V + / E Home / N + to / J / V + character / E Optimal solution selection module Mountain / N + on Go to / V + to / E Home / N + to / J / V + character / E

Specifically, the optimal solution selection module 120 optimizes the results of the single analysis through the morpheme analysis candidate generation module 110 to 'acid / N +', 'house / N + / J' and '/ V + V / G + V 'among the three results of analysis through the morpheme analysis candidate generation module 100,' A / N + E ',' Gal / V + / E 'as the optimal solution.

The optimal solution selection module 120 has various methodologies as described above in selecting an optimal solution. Basically, various machine learning methodologies can be utilized, and various parameters to be used for selecting the optimal solution can be learned through the corpus with part-of-speech and applied in real time.

Referring to FIG. 2, in step S40, the morphological analysis result is modified based on the air information through the air information based correction module 130. FIG.

The air information based correction module 130 performs the result verification of the air information based on the morpheme analysis result generated by the optimum solution selection module 120. If it is determined that there is an error in the morpheme analysis result, . The following is an example of correcting morpheme analysis error of verb through the air information based correction module.

The air information based correction module 130 searches for a word dictionary existing in the word dictionary of 1) and 2) corrects a morpheme analysis error when referring to the air information dictionary for the corresponding word.

The air information dictionary information has a high weight in the order of "noun + inquiry term"> "noun verb"> "$ noun meaning + inquiry term".

3 is an example of error correction when a morpheme analysis error occurs according to the present invention.

Referring to FIGS. 2 and 3, an example of correcting an error when an error occurs in noun morpheme analysis will be described.

The result of the input sentence through the morpheme analysis candidate generating module 110 and the optimal solution selecting module 120 is referred to as an unmodified result 220 and the result through the air information based correction module is referred to as a corrected result 240. [

For example, if the pre-correction result is '/ N or / V + even though / E / N + is / J true / V + character / E' N + degrees / J ' existing in the air information dictionary 112 for the corresponding phrases by searching the existing " painter " I / V 'and' / N / V 'to detect and correct the occurrence of errors in morpheme analysis.

Therefore, the modified result 240 becomes 'J / V true / V true / V' even if 'Tue / N + is / J or V / V'.

As described above, the apparatus and method for correcting multi-lingual morpheme analysis based on the word air information according to the present invention are not limited to the configuration and method of the embodiments described above, All or some of the embodiments may be selectively combined.

100: Multilingual Morphological Analysis Error Correction Device Based on Word Air Information
110: Morphological analysis candidate generation module
120: Optimal solution selection module
130,230: air information based correction module
220: Results before correction
240: Result after modification
111: Candidate candidate dictionary
112: Air information dictionary

Claims (10)

A multilingual morpheme analysis error correction method performed by a multilingual morpheme analysis error correction apparatus based on word air information,
A step of constructing a tentative candidate dictionary for a word in which the tentative word occurs;
Extracting air information from a large-scale source corpus on the word having the above-mentioned ambiguity;
Generating a morpheme analysis result for the input sentence; And
And modifying the morpheme analysis result based on the air information,
Wherein the modifying the morpheme analysis result comprises:
Extracting a word phrase existing in the ambiguity candidate from the morpheme analysis result, and correcting the morpheme analysis result based on the air information dictionary for the extracted word phrase, wherein the multilingual morpheme analysis error based on the word air information Correction method. The method according to claim 1,
The second candidate dictionary
Wherein the word correcting unit corrects the correct morpheme error based on the word air information based on the word-of-mouth information. The method of claim 2,
The step of generating the morpheme analysis result
And generates the most optimal analysis result among the candidates generated by using the words existing in the input sentence and the ambiguous candidate dictionary, based on the word air information. The method of claim 3,
Wherein the large-scale raw corpus is extracted from at least one of a web site, a blog, and a newspaper article. delete The method according to claim 1,
The air information dictionary
The method comprising the steps of: extracting words that have not been found in structural ambiguity and morphological ambiguity among the morpheme analysis results of the sentence, and storing the words; and correcting errors in multilingual morpheme analysis based on the word air information. A morpheme analysis candidate generation module generating an analysis candidate for a word in which an ambiguity occurs in an input sentence and generating a morpheme analysis result for each of the analysis candidates;
An optimal solution selection module for selecting one morpheme analysis result using the ambiguous candidate dictionary generated based on the morpheme analysis result of each of the analysis candidates and the ambiguous word extracted from the large scale raw corpus; And
An air information based correction module for extracting a word phrase existing in the ambiguity candidate from the selected morpheme analysis result and correcting the morpheme analysis result based on the air information dictionary for the extracted word phrase,
Wherein the morpheme analysis error correction unit is based on word air information. The method of claim 7,
The second candidate dictionary
Wherein the word correcting unit corrects the morpheme morpheme error based on the word air information based on the word ambiguity. The method of claim 8,
The large-scale raw corpus
A word, a word, a web site, a blog, and a newspaper article. delete

Images