KR100303171B1 - Morphological and syntactic analysis method using morpheme connection graph - Google Patents

Abstract

The present invention relates to natural language processing, including morphological analysis, part-of-speech tagging and syntax analysis.

In the morpheme analysis process according to the present invention, a morpheme connection node is configured as a data structure representing a morpheme, and the nodes are connected to each other to form a morpheme connection graph.

According to the present invention, various morphological analysis candidates appearing in the Korean language processing process can be represented in the form of a graph, thereby reducing the memory waste generated when representing the existing N-linked list and efficiently processing the searching process. .

Description

Morphological and syntactic analysis method using morpheme connection graph

The present invention relates to natural language processing, including morphological analysis, part-of-speech tagging, and syntax analysis.

According to the related art, various morphological candidates generated during natural language processing are represented as N-linked lists, which require a large amount of memory and decrease the efficiency of the search process due to the large number of linked lists. .

The present invention has been created to solve the above problems, by efficiently expressing intermediate products generated in the morphological analysis process of the natural language storage medium recording a morphological access graph that can efficiently perform the morphological analysis and syntax analysis process, It is an object of the present invention to provide a morphological analysis method using the morpheme connection graph.

1 illustrates the components of a morpheme connection node constituting a morpheme connection graph according to the present invention.

2 exemplarily illustrates a process of generating a morpheme access graph by combining morpheme access nodes.

Figure 3 illustrates a morphological analysis process using a morpheme connection graph according to the present invention.

Figure 4a shows an example of the use of the morpheme connection graph in the morpheme analysis process, Figure 4b shows the N-Best morpheme results.

FIG. 5A illustrates an example of using a morpheme connection graph in a part-of-speech tagging process, and FIG. 5B illustrates a 1-best tagging result.

6 illustrates triangular table packing at parsing initialization.

In order to achieve the above object, in a computer-readable recording medium recording a morpheme connection node constituting a morpheme connection graph representing intermediate products generated during the morpheme analysis process of the natural language input sentence according to the present invention, The access node includes a surface information field for storing one or more morphological surface layers; A dictionary information field for storing part-of-speech, main form, morphology, and access information of the morpheme; When the input sentence is divided into phonemes and assigned sequential numbers, respectively, a phoneme string start number field indicating a number where the first phoneme of the one or more morphemes is located and a phoneme indicating a number where the last phoneme of the one or more morphemes is located Column end number field; When the input sentence is divided into morphemes and assigned a sequential number, the morpheme start number field indicating the number where the first morpheme of the one or more morphemes is located and the morpheme end indicating the number where the last morpheme of the one or more morphemes are located. Number field; A morpheme probability field for storing lexical probability values of the one or more morphemes; A cumulative probability field for storing a cumulative probability value calculated using the lexical probability value, the context probability value and the syllable trigram probability value of the one or more morphemes; And previous nodes pointer fields that store pointers to all immediately preceding stemmed nodes connected to the at least one stem, and next nodes storing pointers to all stemmed nodes immediately after being connected to the at least one stem. And a pointer field, wherein the previous nodes pointer field and the next nodes point field have a priority queue structure arranged according to a probability value of the stem nodes, and thus the priority when obtaining the N-Best part-of-speech tagging result. It takes out one by one from the queue and outputs N tagged morphemes.

In the computer-readable recording medium recording the morpheme access node, the morpheme access node is configured to recover a memory space occupied by the morpheme access node when the morpheme access node fails to access through a connection test. A free storage space pointer field for storing a pointer; An all stem pointer field for storing a pointer to a previous part of stem steming node in which a part of speech is determined through a Viterbi search to find parts of speech tag tagged stem parts of 1-Best; A registered / unregistered information field for storing information for discriminating whether the one or more morphemes are registered or unregistered non-registered words; A serial / parallel tag information field for storing information on a parallel tag in which morphemes of different parts of speech are analyzed as parts of speech of a higher concept and a serial tag in which a plurality of morphemes are combined and analyzed as one morpheme; A morpheme number information field for storing the number of morphemes in the one or more morphemes; A word start morpheme node information field for storing whether the one or more morphemes are located at the beginning of a word; And a connection flag field for storing a result of performing a connection check using the connection check table.

In order to achieve the above object, a method of morphological analysis of a natural language input sentence using a morpheme connection graph according to the present invention includes (a) using information of a morpheme dictionary and a morpheme pattern dictionary in morpheme units of the input sentence. Forming a candidate morpheme connection node, and checking whether the candidate morpheme connection node is connected using the information in the connection table, and adding the candidate morpheme connection node to the morpheme connection graph. Transferring the memory space allocated to the access node to the memory manager to generate an initial morpheme connection graph; (b) a bit rate search is performed on a candidate morpheme access node added to the initial morpheme access graph using a lexical probability value, a context probability value, and a syllable trigram probability value to calculate a cumulative probability value, and the candidate is calculated using the calculated cumulative probability value. Reducing the number of morpheme access nodes to generate a filtered morpheme access graph; (c) repeating steps (a) and (b) for all morphemes of the input sentence to generate a tagged morphemes access graph that records an optimal path in the filtered morphemes connection graph; And (d) correcting a tagging error of the tagged stem graph by using a previously learned error correction rule to generate an error-corrected stem stem graph.

Morphological analysis is the basic process of Korean language processing. It is to separate each word of input sentence into morphological units, assign appropriate parts of speech from dictionary, and check connection between them using connection table. By using the information to give the parts of speech appropriate to each morpheme refers to the process of decomposing Korean into the smallest unit having meaning. In the present invention, a data structure representing a morpheme used in this process is defined as a morpheme connection node, and a structure in which the morpheme connection nodes are connected to each other is defined as a morpheme connection graph.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

First, let's look at the stem stem node which is the basic unit of the stem stem graph. As shown in Fig. 1, the stem stem node is composed of the following elements.

The <Free Memory Space Pointer> is a pointer used to transfer the space occupied by the stemming node to the memory manager when the stemming node fails to connect to the memory manager due to the connection check. That is, it is a pointer used for efficient memory management.

Next, the <former stem pointer> is a pointer used to indicate a previous stem stem node whose part of speech has been determined through a Viterbi search. Can be used to obtain a 1-Best POS-tagged tagged stem node.

Surface information is a space for storing a surface of an input morpheme.

<Registered / Unregistered Information> is a space for storing information for distinguishing whether the stem of the current node is a registered word or a non-registered word.

<Serial / Parallel Tag Information> can be determined as MP (unique noun), such as {MPN (personal proper noun), MPC (national proper noun), MPP (name proper noun), MPO (other proper noun)} These are called parallel tags, and multiple morphemes, such as '(ㄹ # 수 # 는)', are combined and analyzed as serial tags, called serial tags. to be.

The <morpheme number information> is a space for storing the number of stems included in the stem stem node, and usually has a value of 1. However, in case of '(ㄹ # 수 # able)', it has a value of 5.

<Dictionary information> is a place where information shown in the morpheme dictionary and the morpheme pattern dictionary is stored. In advance, parts of speech, main form, morphology, and connection information of morphemes are recorded. This content is recorded in <dictionary information> and used to construct a morpheme connection graph.

<Spoken start morpheme node information> is a space for storing whether the current stem stem node is the beginning of a word, and <connection flag> records whether the connection is possible as a result after performing the connection check using the connection check table. It is a space to do.

The <phoneme sequence start number> and <phoneme sequence end number> are spaces used to store the input sentence.

The <morphological start number> and <morphological end number> are storage spaces used for storing information that the current stem is from the first to the first stem.

<Morpheme probability> is the space where the lexical probability value of the current morpheme is stored, and <cumulative probability> calculates the cumulative probability value using the lexical probability value, context probability value, and syllable trigram probability value of the current node when performing Viterbi search. To store it.

Finally, <Previous Nodes Pointer> and <Next Nodes Pointer> are spaces for storing pointers to all stemming nodes before and after the current stemming node. Since this storage space has a priority queue structure arranged according to probability values of the stem nodes, the N-Best part-of-speech tagging results are taken out of the queue and used to output the parts-tagged N stem sequences. Can be.

I ate 0 One 2 3 4 5 6 7 8 … $ N ㅏ N ㅡ N # ㅂ ㅏ …

Next, let's take a look at the stemming graph through an example. For example, when the input sentence 'I have eaten' is input, as shown in FIG. 2, a morpheme connection graph in which connection nodes are connected to each other is created. Among the nodes of the stemming graph, let's take a look at the contents of the stemming connection node corresponding to '/ jS (secondary investigation)'. First of all, the <free memory pointer> is stored as NULL because the current node must be successfully connected and remain in memory. The <morpheme pointer> points to the 'na / T' pronoun to connect the '// T' pronoun determined by Viterbi search and the '/ jS' assistant in the stem stem graph. do. In <surface information>, the value of "is" is stored, and "/ jS (assistant survey)" is registered in advance, so <registration / non-registration information> is recorded as "registration word." '은 / jS (보조)' is neither a parallel morpheme nor a serial morpheme, so nothing is written in the <serial / parallel morpheme information>. 1 is recorded in <morph number information>, and tag information <main form> (this form) (connection information) is recorded in <dictionary information> as shown in FIG. '은 / jS (보조)' is not the first morpheme of the word, so the value of 'NO' is recorded in the <word start morpheme information>. If it were a word start morpheme, the value "YES" would have been stored. On the other hand, since the current node succeeded in the connection test, 1 is recorded in the <connection flag>. If it failed, 0 would have been recorded.

As shown in Table 1, when `` / jS '' is divided into phoneme strings, the start number 3 and the end number 5 are recorded respectively, and in the input sentence, `` / jS '' is defined as two. As the first morpheme, 2 is recorded for both <start stem number> and <end stem number>. The morphological probabilities contain pre-learned lexical probability values, and the cumulative probabilities store cumulative probability values determined by Viterbi search. Finally, pointers to previous and subsequent stemming nodes are stored in <previous node pointer> and <next nodes pointer>.

Figure 3 shows the morphological analysis process using the morpheme access graph. Here, the morphological analysis process is used not only in a narrow sense but also in a broad sense including a part-of-speech tagging process.

The morpheme analyzer 30 analyzes the morphemes for each morpheme constituting the input sentence, generates morpheme analysis candidates, and forms the generated morpheme analysis candidates in the form of a morpheme access node. At this time, the morpheme analyzer 30 searches the morpheme dictionary and the morpheme pattern dictionary to find out dictionary information and fills in the contents of each morpheme access node. In addition, the stem stem analyzer 30 refers to the information in the connection table, and the stem stem connection node to be connected is added to the stem stem connection graph, and the memory space of the stem stem connection node which is not connected is transferred to the memory manager. Create

Next, the statistical part-of-speech tagger 32 calculates a cumulative probability value by performing a Viterbi search using lexical probability values, context probability values, and syllable trigram probability values for each morpheme access nodes of the initial morpheme access graph. Pruning reduces the number of candidate stem nodes.

When the morphological analysis reaches the end of the sentence, the tagged morpheme graph is passed to the error corrector 34, and when the sentence ends, the morphological analyzer 30 performs the analysis on the next morpheme constituting the input sentence. Repeat until.

When the end of the sentence is reached, the error modifier 34 finds and corrects the tagging error in the tagged stem graph using the pre-learned error correction rule to complete the final stem stem graph. The final morpheme connection graph is passed to parser 36.

4A and 4B are each an exemplary diagram showing an initial morpheme access graph and a morpheme analysis result generated in the morpheme analysis process by the morpheme analyzer 30, respectively. The morpheme connection graph has all the morphological analysis candidates that can be divided into phoneme units and connected to each node. Figure 4a shows an initial morphological link graph for the input sentence 'I had a meal'. According to Figure 4a, in the process of making a candidate by stemming the phrase "I" "fly / DR (regular verb) + is / eCNMG (tubular malleable mother)," out / DI (irregular verb) + is / eCNMG It can be seen that there are three possible connection types: (the tubular malleable mother ') and' I / T (pronoun) + / jS (supplemental investigation) '. Unlike the trellis, the initial morpheme connection graph according to the present invention sets only a link that can be connected through a reference probability value and a connection check table as a candidate. Thus, 'fly / DR (regular verbs) + is / jS (supplemental investigation)', 'nada / DI (irregular verb) + is / jS (supplementary investigation)', 'I / T (pronoun) + is / eCNMG Candidates that occur in trellis, such as 'mother', are filtered through a connection check before they occur. Figure 4b shows the results of the morphological analysis of Pohang University Natural Language Processing Laboratory morpheme analyzer (POSTAG) to create a morpheme connection graph. In the analysis result table, phoneme unit information and syllable unit information of the analyzed morphemes for the input sentence are indicated by numbers in [] on the left. For example, for a morphological analysis candidate named 'Bob' estimated as a noun as a registered word, the phoneme information is input information from the seventh phoneme to the ninth phoneme from the beginning, and the syllable information is input information of the fourth syllable. For the input information, the stem stem connection graph is generated as a node to form a stem stem connection graph node 'Bob / MC' having information of MC <bob> (bob) [etc | bob]. That is, FIG. 4A is constructed as a morpheme connection graph reflecting the morphological analysis result of FIG. 4B.

5A and 5B are exemplary views illustrating tagged morpheme connection graphs and 1-Best tagging results generated during a part-of-speech tagging process of extracting the most accurate analysis result by the statistical part-of-speech tag 32, respectively. Statistical part of speech tagger 32 is responsible for extracting the most likely morphological analysis result from the filtered morphological analysis graph candidates using the connection test and the probability information. The POSAC Natural Language Processing Lab Parts of Speech Tagger generates all the possible paths of the stem stem connection graph starting from the 'START' node of the stem stem graph shown in FIG. 4A and ends with the 'END' node to select the best of them. Paths shown in bold in the figure of FIG. 5A show a morphological connection graph for the optimal paths.

6 is an exemplary view showing a result of embedding (Embedding) receiving the result of the morphological analysis for the example sentence 'I have a meal' shown in FIG. 6 shows a gradual syntax analysis process using the insertion of a morpheme connection graph as an intermediate structure. The parsing of the morpheme connection graph determines the index in the table of nodes of each morpheme connection graph through a morpheme connection graph embedding process that creates initial syntax information of the morphemes in the triangular table generated by the chart parsing. Show the result of changing the state index for the longest stem stem instead of the original frame index. After the index of each stem-connected graph node is changed, the distinct categories of each stem are created in each place of the triangle table. In this case, by combining the index information and the morpheme access information of the tables included in each syntax category, syntax combinations between syntax categories not combined in the morphological analysis result are limited.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed on a computer. And, it can be implemented in a general-purpose digital computer for operating the program using a medium used in the computer. The media may be stored such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optical reading media (e.g., CD-ROM, DVD, etc.) and carrier waves (e.g., transmission over the Internet). Media.

The recording medium (a) forms candidate morpheme access nodes using information of a morpheme dictionary and a morpheme pattern dictionary in units of morphemes of an input sentence, and checks whether the candidate morpheme access nodes are connected using information of a connection table. A module for adding the candidate stem stem connection node to the stem stem connection graph when the connection is established, and transferring the memory space allocated to the candidate stem stem connection node to a memory manager to generate an initial stem stem connection graph; (b) a bit rate search is performed on a candidate morpheme access node added to the initial morpheme access graph using a lexical probability value, a context probability value, and a syllable trigram probability value to calculate a cumulative probability value, and the candidate is calculated using the calculated cumulative probability value. A module for reducing the number of stemmed connection nodes to generate a filtered stemmed connection graph; (c) a module for repeating the (a) module and the (b) module with respect to all morphemes of the input sentence to generate a tagged morpheme access graph that records an optimal path in the filtered morpheme connection graph; And (d) program code for executing, on a computer, a module for correcting a tagging error of the tagged stemming graph using a previously learned error correction rule to generate an error corrected stemming graph.

Functional modules for implementing the present invention as described above can be easily implemented by programmers in the art to which the present invention belongs.

According to the present invention, various morphological analysis candidates appearing in the Korean language processing process can be graph-morphologically represented, thereby reducing the memory waste generated when the existing N-linked list is represented and efficiently processing the search process.

Claims (4)

A computer-readable recording medium recording a morpheme connection node constituting a morpheme connection graph representing intermediate products generated during a morphological analysis of a natural language input sentence. A surface information field for storing surface layers of one or more morphemes; A dictionary information field for storing part-of-speech, main form, morphology, and access information of the morpheme; When the input sentence is divided into phonemes and assigned sequential numbers, respectively, a phoneme string start number field indicating a number where the first phoneme of the one or more morphemes is located and a phoneme indicating a number where the last phoneme of the one or more morphemes is located Column end number field; When the input sentence is divided into morphemes and assigned sequential numbers, a morpheme start number field indicating a number where the first morpheme of the at least one morpheme is located and a morpheme end indicating a number where the last morpheme of the at least one morpheme are located. Number field; A morpheme probability field for storing lexical probability values of the one or more morphemes; A cumulative probability field for storing a cumulative probability value calculated using the lexical probability value, the context probability value and the syllable trigram probability value of the one or more morphemes; And Previous Nodes pointer field that stores a pointer to all immediately preceding stemmed nodes connected to the one or more stem cells and Next Nodes pointer that stores pointers to all stem stemmed nodes immediately after being connected to the one or more stem cells. With fields, The previous nodes pointer field and the next nodes point field are configured in a priority queue structure arranged according to probability values of morpheme access nodes. When the N-Best part-of-speech tagging result is obtained, the part-of-speech tag is taken out from the priority queue one by one. And a computer-readable recording medium having recorded morpheme access nodes. The method of claim 1, A free memory space pointer field storing a pointer for allowing the memory manager to recover the memory space occupied by the stem stem connection node if the stem stem connection node fails to connect through a connection check; An all stem pointer field for storing a pointer to a previous part of stem steming node in which a part of speech is determined through a Viterbi search to find parts of speech tag tagged stem parts of 1-Best; A registered / unregistered information field for storing information for discriminating whether the one or more morphemes are registered or unregistered non-registered words; A serial / parallel tag information field for storing information on a parallel tag in which morphemes of different parts of speech are analyzed as parts of speech of a higher concept and a serial tag in which a plurality of morphemes are combined and analyzed as one morpheme; A morpheme number information field for storing the number of morphemes in the one or more morphemes; A word start morpheme node information field for storing whether the one or more morphemes are located at the beginning of a word; And And a connection flag field for storing a result of performing a connection check using a connection check table. In the method of stemming a natural language input sentence using a morpheme connection graph, (a) forming a candidate morpheme access node using information of a morpheme dictionary and a morpheme pattern dictionary in the morpheme unit of the input sentence, and confirming whether the candidate morpheme access node is connected using the information of a connection table. Adding the candidate morpheme access node to the morpheme access graph, and if not, transferring the memory space allocated to the candidate morpheme access node to a memory manager to generate an initial morpheme access graph; (b) a bit rate search is performed on the candidate morpheme access node added to the initial morpheme access graph using a lexical probability value, a context probability value, and a syllable trigram probability value to calculate a cumulative probability value, and the candidate is calculated using the calculated cumulative probability value. Reducing the number of morpheme access nodes to generate a filtered morpheme access graph; (c) repeating steps (a) and (b) for all morphemes of the input sentence to generate a tagged morphemes access graph that records an optimal path in the filtered morphemes connection graph; And (d) correcting a tagging error of the tagged morpheme connection graph by using a previously learned error correction rule to generate an error-corrected morpheme connection graph. . A recording medium recording a morphological analysis program for morphological analysis of a natural language input sentence using a morpheme connection graph, (a) forming a candidate morpheme access node using information of a morpheme dictionary and a morpheme pattern dictionary in the morpheme unit of the input sentence, and confirming whether the candidate morpheme access node is connected using the information of a connection table. A module for adding the candidate morpheme connection node to the morpheme connection graph and, if not connected, transferring the memory space allocated to the candidate morpheme access node to a memory manager to generate an initial morpheme access graph; (b) a bit rate search is performed on the candidate morpheme access node added to the initial morpheme access graph using a lexical probability value and a syllable trigram probability value to calculate a cumulative probability value, and the candidate morpheme access node is calculated using the calculated cumulative probability value. A module for generating a filtered morpheme connection graph by reducing the number of cells; (c) a module for repeating the (a) module and the (b) module for all morphemes of the input sentence to generate a tagged morphemes connection graph that records an optimal path in the filtered morphemes connection graph; And (d) a morpheme analysis program using a morpheme connection graph, comprising a module for correcting a tagging error of the tagged morpheme connection graph by using a previously learned error correction rule to generate an error-corrected morpheme connection graph The computer-readable recording medium which recorded the data.