KR100484943B1 - Method for recognizing named entity in korean text - Google Patents

Abstract

The present invention relates to an apparatus and method for recognizing an entity name on Korean text. When the morpheme analyzer analyzes the morpheme of the input text and generates an HMM data structure of a list of entity name state information for each morpheme, the entity name recognizer After recognizing character traits, entity name qualities, combined nouns / adjunct qualities, and adjacent noun qualities for each morpheme, the probabilities of transfer of morphemes of the input text using the HMM entity name statistical information extracted by conducting the courting learning And lexical probabilities are respectively assigned, and the forward / backward probabilities considering the left and right contexts are recalculated and reflected on the relevant entity name status information for each morpheme vocabulary, and then the Viterbi algorithm Determine the optimal HMM entity name of the morphemes that make up the sentences. Which is to recognize,

Accordingly, in addition to the construction of statistical information through manually constructed learning data, it is possible to extend the construction of statistical information using plain text to obtain a result of recognizing individual names having proper performance for Korean.

Description

{Method for recognizing named entity in korean text}

The present invention relates to an apparatus and method for recognizing an entity name on various forms of natural language text, such as e-mail, newspaper articles, web pages, etc. More specifically, the Hidden Markov model (HMM) An apparatus and method for recognizing entity names on Korean text using CoTraining are disclosed.

As is known, entity name recognition refers to recognition of proper nouns such as names, place names, organization names, etc. on a specific text, and is required for various fields such as information extraction, information retrieval, and text mining.

In addition, the representative approach of the entity name recognition task as described above is a rule-based approach, a statistics-based approach, and a hybrid approach combining the two, in particular, the statistics-based approach uses the HMM to name the entity. As a recognizing method, it has advantages such as simplicity, scalability, robustness, etc. compared to the rule-based approach that requires a large amount of manual work and has domain-dependent limitations.

On the other hand, since entity name tagging text construction is essential for entity name recognition, and an entity name composed of proper nouns has data sparsity due to its characteristics, proper performance of the entity name recognition method by the statistical-based approach is achieved. Since it can be said that it is proportional to the cost of constructing a large-capacity tagging corpus, the statistical approach has the disadvantage of requiring a large-scale learning material.

Therefore, in recent years, a complementary approach to overcome the shortcomings of the statistics-based approach has emerged, and a representative complementary approach is augmented learning. The augmented learning method refers to an entity name recognition method using unlabeled text that is not manually performed to overcome the scarcity of manually constructed learning data, and a representative augmentation learning method is an entity name recognition method using coat lining. There is this.

For example, as an example of the entity name recognition method using the coat lining, an entity name constructing unit and left and right contexts are configured in a vector representation based on a chunking model and a naive bayes model, and the left-right view type is used. Korean language name recognition method has been developed to attempt to learn courting by dividing by type).

However, the Korean object name recognition method based on the chunking model and the Naive Bayes model has a disadvantage in that it cannot present the object name recognition result with proper performance because it uses an inappropriate Naive Bayes classifier for the Korean object name. .

In addition, as another example of the entity name recognition method using the coatlining, the courtlining is performed in the form of a decision list and only seven initial rules consisting of a spelling rule and a context rule. An approach for non-comparative learning has been developed that builds a model for recognizing individual names.

However, this method of recognizing individual names through comparative verb learning using only a small amount of initial rules takes a parsing approach when extracting context rules. There is a problem that it is not suitable to apply to.

Accordingly, an object of the present invention is to overcome the above-mentioned problems, and an object of the present invention is to recognize an entity name on Korean text through augmented learning using coat lining based on HMM. And providing a method.

The object name recognition apparatus on Korean text for achieving the object of the present invention, after composing the input text into a list of sentences, each sentence is composed of a morpheme list separated by a morpheme unit with a status cover, and then constitutes each sentence A morpheme analyzer for generating an HMM data structure which is a list of entity name state information for each morpheme of the morpheme list and storing it in a memory; A statistical information extractor for extracting HMM entity name statistical information from the entity name tagging text set; A coat-lining learner configured to extend the statistical information through the existing training data by extracting the HMM entity-name statistical information from the general learning text set by conducting a courting learning based on the HMM statistical information extracted from the entity-tagging text set; A statistical information DB for storing HMM entity name statistical information extracted by the statistical information extractor and the court training learner; And using the entity name dictionary DB, the combined noun / adjective dictionary DB, and the adjacency noun dictionary DB for the characterization of each morpheme analyzed by the morpheme analyzer, the entity name feature, the combined noun / adjective feature, and the adjacent noun feature. Recognize and assign the transition probability and the lexical probability to the morphemes of the input text using the HMM entity name statistical information extracted by the coat-training learner, and then, for the corresponding entity name state information for each morpheme vocabulary. Optimal HMM entity name state information path of morphemes that recomputes and reflects forward / backward probabilities considering left and right contexts and forms sentences of input text consisting of morpheme list through Viterbi algorithm It is composed of an entity name recognizer that recognizes an entity name by determining

The object name recognition method on Korean text for achieving the object of the present invention comprises the steps of: analyzing a morpheme of an input text by a morpheme analyzer to construct a morpheme list; Generating an HMM data structure of a list of entity name state information for each morpheme analyzed by the morphological analyzer; The entity name recognizer uses the entity name dictionary DB, the association noun / adjective dictionary DB, and the adjacency noun dictionary DB for character morphemes, entity names, association nouns / adjuncts, and adjacency for each morpheme analyzed by the morpheme analyzer. Recognizing noun qualities; Extracting, by the statistical information extractor, the HMM entity name statistical information from the entity name tagged text set; The coat-learning learner proceeds to coat-lining based on the HMM statistical information extracted from the entity-tagged text set to extract the HMM entity-name statistical information from the unlabeled text and expands the statistical information through the existing training data. Doing; Assigning, by an entity name recognizer, transition probabilities and lexical probabilities to the morphemes of the input text by using the HMM entity name statistical information extracted by the coat training learner; The entity name recognizer recalculating and reflecting forward / backward probabilities in consideration of left and right contexts with respect to corresponding entity name state information of each morpheme vocabulary to which a transition probability and a lexical probability are assigned; And recognizing the entity name by determining an optimal HMM entity name state information path of the morphemes constituting the sentences of the input text composed of the morpheme list through the Viterbi algorithm.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, the morpheme analyzer 10 divides an input text into a list of sentences, and forms each sentence into a morpheme list separated by a morpheme unit with a state mark, and then each stem of the morpheme list constituting each sentence. Create an HMM data structure, which is a list of entity name status information for, and store it in memory.

The morphological analyzer 10 checks whether or not the morpheme corresponds to the object name for each morpheme of the morpheme list, if it can be the first morpheme in the entity name, or is a morpheme that can be used inside the entity name, or Creates an HMM data structure with a list of entity name status information indicating the status of the last morpheme.

The statistical information extractor 20 extracts the HMM entity name statistical information from the entity name tagging text set and stores it in the statistical information DB 30.

The coat-learning learner 40 performs the coat-lining learning based on the HMM statistical information extracted from the object-tagged text set, extracts HMM individual-name statistical information from the general learning text set, and stores the statistical information in the statistical information DB 30. Boost statistical information through learning data.

The coat-training learner 40 determines the current learning view and the number of learning, randomly samples a certain amount of text set from the general learning text set, and then determines whether the currently designated learning view is a forward model for the random sampled text set. Based on the result of discriminating whether or not it is a word model, the courting learning progress direction is determined, and HMM individual name statistical information is extracted.

When the current learning view is a forward model, the coat-training learner 40 performs the HMM-based entity name tagging operation on the forward model based on the HMM statistical information extracted by the statistical information extractor 20 from the entity name tagging text set. If the current learning view is a backward model, the HMM-based entity name tagging operation for the backward model is performed, and then n entity name tagging results having the highest probability value are extracted from the entity name tagging result to learn new HMM statistics. Add it with data.

The coat-training learner 40 extracts n entity name tagging results having the highest probability value and adds them as new HMM statistical learning data, and then determines whether the current learning view is a forward model or a backward model. If the view is a forward model, the HMM statistical information about the backward model is extracted from the entity name tagging result and stored in the statistical information DB 30. Then, the backward model training data is updated and the current training view is changed to the backward model. After that, the process from the random sampling operation on the general training text set to the change of the current learning view to the backward model or the forward model is repeated until the current learning count becomes a predetermined learning count.

The coat-training learner 40 extracts n entity name tagging results having the highest probability value and adds them as new HMM statistical learning data, and then determines whether the current learning view is a forward model or a backward model. If the view is a backward model, the HMM statistical information of the forward model is extracted from the object name tagging result, stored in the statistical information DB 30, the forward model training data is updated, and the current training view is changed to the forward model. The process from the random sampling operation on the general training text set to the operation of changing the current learning view to a backward model or a forward model is repeatedly performed until the current learning number becomes a predetermined learning number.

The entity name recognizer 50 uses each entity morpheme analyzed by the morpheme analyzer 10 by using the entity name dictionary DB 51, the combined noun / adjective dictionary DB 52, and the neighbor noun dictionary DB 53. Recognizes character qualities, entity qualities, combined / adjunct qualities, and adjacent noun qualities.

The entity name recognizer 50 assigns a transition probability and a lexical probability to the morphemes of the input text using the HMM entity name statistical information extracted by the coat-training learner 40, and then applies the corresponding morpheme vocabulary to each morpheme vocabulary. The forward / backward probabilities considering the left and right contexts of the entity name state information are recalculated and reflected.

The entity name recognizer 50 uses the HMM entity name statistical information extracted by the coat-training learner 40 to identify two consecutive morphemes in the HMM data structure generated for the morphemes of the input text. Transition probabilities indicating variation probability values of information and lexical probabilities indicating probability values for corresponding entity name state information for each morpheme vocabulary are respectively assigned.

The entity name recognizer 50 recognizes an entity name by determining an optimal HMM entity name state information path of morphemes constituting sentences of an input text composed of a morpheme list through a Viterbi algorithm.

The apparatus for recognizing an entity name on Korean text according to the present invention configured as described above operates as follows.

2 to 5B, in order to recognize an entity name on Korean text, first, the morpheme analyzer 10 analyzes each sentence of the input text in a morpheme unit with a status mark to construct a morpheme list (S100). In operation S200, an HMM data structure consisting of a list of entity name state information for each morpheme analyzed is generated.

In this case, the morpheme analyzer 10 is for each morpheme of the morpheme list whether the morpheme corresponds to the entity name, if it can be the first morpheme in the entity name, whether the morpheme can be used in the entity name, Or create an HMM data structure with a list of entity name state information indicating the state of the entity's last morpheme.

For example, stemming from the sentence "he goes to school," he reads "he / nc", "ga / jc", "school / nc", "e / jc", "ga / pv", and "b / etf" Where the entity name is 'withdrawal / nc' and the rest do not correspond to the entity name. Thus, 'Retract / nc' becomes the entity name state PERSON.

In particular, since it is not possible to determine which entity name state after the morphological analysis step 100, in the embodiment according to the present invention, all the state information for the morpheme under the assumption that all states are possible in the morphological analysis step S100. After creating, we determine which state is most appropriate through the learned HMM statistics.

For reference, the entity name type according to the present invention and the HMM state type for each entity name type according to the present invention are shown as shown in FIG. 4A.

Referring to FIG. 4A, the entity name is PERSON, LOCATION, ORGANIZATION, DATE, TIME, PHONE, PERCENT, MONEY , Address (ADDRESS), quantity (QUANTITY), artifact (ARTIFACT).

In addition, the HMM state type means the entity name configuration state type that can be taken by the component morpheme of the input sentence. In other words, S_LOC refers to the state of morphemes that can come from the first vocabulary of the local name, C_LOC refers to the state of morphemes to come from the middle vocabulary of the local name, and E_LOC refers to the state names of morphemes that can come from the last vocabulary of the local name. For example, U_LOC refers to the case where a single morpheme is a local name.

For example, after morphological analysis of the sentence `` Daejeon National Museum '', status markers are attached to each morpheme in the form of 'National / S_LOC', 'Central / C_LOC', and 'Museum / E_LOC'. In case of 'match', HMM status type is determined in 'match / U_LOC' form.

After the HMM data structure is generated as described above, the entity name recognizer 50 uses the entity name dictionary DB 51, the combined noun / adjective dictionary DB 52, and the neighbor noun dictionary DB 52. Recognize character qualities, individual name features, combined nouns / affixes, and adjacent noun features for each morpheme analyzed by (10) (S300).

The textual feature, the entity name feature, the combined noun / adjective feature, and the adjacent noun feature for each of the morphemes are used by the entity name recognizer 50 to determine the transition probability and the lexical probability. As shown in 4b.

Referring to FIG. 4B, the qualities for the respective morphemes are character qualities 4b-1, entity qualities 4b-2, associative nouns and affixes 4b-3, and adjacent noun qualities 4b-4. It consists of.

The character feature 4b-1 is composed of specific feature values classified into numbers (DIGIT), a combination of numbers and letters (DIGIT & LETTER), Chinese characters (CHINESE), English letters (ALPHABET), and spelling (LETTER) features. For example, the vocabulary 'ras 21' has a ContainsDigitAndLetter feature value of the DIGIT & LETTER feature type.

The entity name feature 4b-2 depends on the entity name dictionary information. For example, if the vocabulary 'Gim Deuk-gu' exists in the life dictionary, it has the value of DicPerson. In this case, the statistical information is to present statistics using the feature value DicPerson instead of the vocabulary Kim Duk-gu.

The combined noun / adjunct feature 4b-3 is a qualification of frequently used words or affixes among the internal names of the individual names. For example, monetary units such as 'won' to 'won' are treated as qualities SuffixMoney.

The adjacent noun qualities 4b-4 are qualities of frequently used words among the constituent names adjacent constituent words. For example, in the president of Lee Kun Hoe, the president is a neighboring noun representing the status of PERSON and is treated as the feature value PositionPERSON.

After recognizing the qualities of the respective morphemes of the input text, the entity name recognizer 50 uses the HMM entity name statistical information extracted by the coat-training learner 40 to determine the probability of transition and the vocabulary of the input text. Each probability is assigned (S400).

In this case, the entity name recognizer 50 performs the coarse lining learning based on the HMM entity name statistical information extracted from the entity name tagging text set by the statistical information extractor 20 and the general training text. Transition probability that represents the probability of variation of the entity name state information for two consecutive morphemes in the HMM data structure generated for the input text morphemes using HMM entity name statistical information extracted from the set, and the corresponding for each morpheme vocabulary. Allocates lexical probabilities representing probability values for entity name state information.

In particular, the method of extracting the HMM entity name statistical information from the general training text set by performing the coatlining learning process 40 is performed as follows.

First, the coatlining learner 40 determines the current learning view and the number of learning (S410), randomly samples a predetermined amount of text set from the general learning text set (S415), and then coats the randomly sampled text set. In order to determine the learning progress direction, it is determined whether the currently designated learning view is a forward model or a backward model (S420).

In this case, if the current learning view is determined as a forward model, the coat-lining learner 40 may tag the HMM-based entity name for the forward model based on the HMM statistical information extracted by the statistical information extractor 20 from the entity name tagging text set. The operation proceeds (S425), and thus n entity name tagging results having the highest probability value are extracted from the obtained entity name tagging result and added as new HMM statistical training data (S435).

On the other hand, if the current learning view is determined as a backward model, the coat-lining learner 40 is based on the HMM based on the HMM statistical information extracted by the statistical information extractor 20 from the object name tagging text set. The entity name tagging operation is performed (S430), and n entity name tagging results having the highest probability value are extracted from the obtained entity name tagging result and added as new HMM statistical training data (S435).

At this time, the HMM statistical learning data type is composed of a combination of four types as shown in FIG. That is, it extracts in the form of MORP type 5a-1, root type 5a-2, POS type 5a-3, and FEATURE type 5a-4. It is then used to calculate probability values for individual names.

For example, in FIG. 5A, in the case of the feature type 5a-4, 'Indiana' has a feature value of DicLOC because it is in the local name dictionary, and in the case of 'Professor', the PositionPERSON value is used. Will have

In addition, referring to FIG. 5B, how the actual probability value is applied to the HMM from the HMM statistical learning data is as follows.

Referring to the morpheme type 5b-1, each of the learning view types of the HMM includes a forward model and a backward model, and each model is configured based on a product of a state transition probability and a lexical probability.

In the forward model, the state transition probability is Pr (S _i | S _i-1 , m _i-1 ) and the lexical probability is Pr (m _i | S _i , m _i-1 ). Here, since state S _i-1 of morpheme m _i-1 is NE_U and next state S _i is-, the state transition probability value is calculated as count (-, NE_U,) / count (-,). On the other hand, the lexical probability value is calculated as count (NE_U, of Stitches) / count (NE_U, of). For reference, the English letter NE means Name Entity.

The difference between the forward model and the backward model can be explained by the state transition probability of each model. In the forward model, the probability value is calculated based on the previous state when the probability value for the current state is obtained. In the forward model, the probability value is obtained based on the next state for the current state. In other words, the forward model is a probability model considering the left context and the backward model is a probability model considering the right context.

As described above, the apparatus and method for recognizing an entity name in Korean text according to the present invention applies the forward / backward modeling technique of the HMM to coat-training learning, and applies statistical information of the left context to plain text to apply statistical information of the right context. The HMM augmented learning model is presented by extracting statistical information of the right context by applying the statistical information of the right context to the plain text again.

After extracting n entity name tagging results having the highest probability values from the entity name tagging results and adding them as new HMM statistical training data (S435), the coat training learner 40 determines whether the current learning view is a forward model. In operation S440, the method determines whether the model is a backward model.

At this time, if the current learning view is determined as a forward model, the coat-training learner 40 extracts HMM statistical information about the backward model from the entity name tagging result and stores it in the statistical information DB 30 (S445). The backward model training data update operation is performed (S450) and the current training view is changed to the backward model (S455).

On the contrary, if the current learning view is determined as a backward model, the coat-training learner 40 extracts HMM statistical information about the forward model from the entity name tagging result and stores it in the statistical information DB 30 (S460). The forward model training data update operation is performed (S465), and the current training view is changed to the forward model (S470).

As described above, after changing the current learning view to a backward model or a forward model, the coat-lining learner 40 performs a random sampling operation on the general training text set until the current learning count becomes a predetermined learning count. The process from step S415 and the steps of changing the current learning view to the backward model or the forward model (S455 and S470) are repeatedly performed (S480).

After the transition probabilities and the lexical probabilities are assigned to the morphemes of the input text as described above, the entity name recognizer 50 performs information on the corresponding entity name state information for each morpheme vocabulary to which the transition probabilities and the lexical probabilities are assigned. The forward / backward probability considering the left and right contexts is recalculated and reflected (S500).

After recalculating the forward / backward probabilities as described above and reflecting the transition probabilities and the lexical probabilities to the assigned morphemes, the entity name recognizer 50 finally uses the Viterbi algorithm to determine the input text. The optimal HMM entity name state information path of the morphemes constituting the sentences is determined to recognize the entity name (S600).

As described above, the apparatus and method for recognizing an individual name on Korean text according to the present invention is configured to recognize the individual name on Korean text through augmented learning using coat lining based on the HMM. In addition to the construction of statistical information through the expansion of statistical information construction using plain text, there is an advantage in that it is possible to obtain the result of recognizing the entity name with the proper performance in Korean.

What has been described above is only one embodiment for implementing the apparatus and method for recognizing an object name on Korean text according to the present invention, and the present invention is not limited to the above-described embodiment, which is claimed in the following claims. Without departing from the gist of the present invention, anyone of ordinary skill in the art will have the technical spirit of the present invention to the extent that various modifications can be made.

1 is a diagram showing an apparatus for recognizing an entity name on Korean text according to the present invention.

Saint,

2 is a flowchart illustrating a method for recognizing an entity name on Korean text according to the present invention.

Climbing,

3 shows statistical information from the general training text set by the courtlining learner.

A flow chart illustrating a method of extracting

4A shows an entity name type and an HMM state type for each entity name type according to the present invention.

Diagram showing

Figure 4b is for determining the lexical probability and state transition probability of the HMM according to the present invention

Diagram showing feature composition,

5A illustrates a configuration and an example of a statistical learning data type according to the present invention.

graph,

Figure 5b is the actual probability value from the statistical learning data type according to the present invention

This chart shows if the HMM applies to the HMM.

<Description of the symbols for the main parts of the drawings>

10: stemmer 20: statistical information extractor

30: Statistical Information DB 40: Coat Raining Learning Machine

50: object name recognizer 51: object name dictionary DB

52: combined nouns / adjuncts DB 53: adjacent nouns dictionary DB

Claims (9)

delete delete delete delete delete delete A morpheme analyzer analyzing the morphemes of the input text to construct a morpheme list; Generating an HMM data structure of a list of entity name state information for each morpheme analyzed by the morphological analyzer; The entity name recognizer uses the entity name dictionary DB, the association noun / adjective dictionary DB, and the adjacency noun dictionary DB for character morphemes, entity names, association nouns / adjuncts, and neighbors for each morpheme analyzed by the morpheme analyzer. Recognizing noun qualities; Assigning, by the entity name recognizer, transition probabilities and lexical probabilities to the morphemes of the input text by using the HMM entity name statistical information extracted by the court training learner; The entity name recognizer recalculating and reflecting forward / backward probabilities in consideration of left and right contexts with respect to corresponding entity name state information of each morpheme vocabulary to which a transition probability and a lexical probability are assigned; And Recognizing the entity name by determining the optimal HMM entity name state information path of the morphemes constituting the sentences of the input text composed of the morpheme list through the Viterbi algorithm. Recognition method of the entity name on Korean text, characterized in that consisting of. 8. The method of claim 7, wherein the entity name recognizer assigns the transition probability and the lexical probability to the morphemes of the input text, respectively. The courtlining learner performs the courtlining learning based on the HMM entity name statistical information extracted from the object name tagging text set by the statistical information extractor and uses the HMM entity name statistical information extracted from the general training text set to form the input text. And a transition probability and a lexical probability are respectively assigned. 10. The method of claim 8, wherein the method for extracting the HMM entity name statistical information from the general training text set by performing the coatlining learner. Determining a current learning view and the number of learning and randomly sampling a predetermined amount of text set from the general learning text set; Determining whether the currently designated learning view is a forward model or a backward model to determine a direction of the training course for the random sampled text set; If the current learning view is determined as a forward model, the statistical information extractor performs a HMM-based entity name tagging operation on the forward model based on the HMM statistical information extracted from the entity name tagging text set and has the highest probability value from the entity name tagging result. extracting the n entity name tagging results and adding them as new HMM statistical training data; If the current learning view is determined as a backward model, the statistical information extractor performs a HMM based entity name tagging operation on the backward model based on the HMM statistical information extracted from the entity name tagging text set, thereby generating the highest probability value from the entity name tagging result. Extracting the n entity name tagging results with and adding it as new HMM statistical training data; Extracting the n entity name tagging results having the highest probability value and adding them as new HMM statistical learning data and determining again whether the current learning view is a forward model or a backward model; If the current training view is determined as the forward model, the HMM statistical information about the backward model is extracted from the object name tagging result and stored in the statistical information DB. Then, the backward model training data is updated and the current training view is forward modeled. Changing to; If the current training view is determined as the backward model, the HMM statistical information about the forward model is extracted from the object name tagging result, stored in the statistical information DB, and the forward model training data is updated and the current training view is changed to the forward model. Doing; And After changing the current training view to the backward model or the forward model, the random sampling operation step for the general training text set and the current training view are forward model or forward until the current training count becomes a predetermined training count. Repeat steps until you change to the model Recognition method of the entity name on Korean text, characterized in that consisting of.