JP3027553B2 - Parser - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a syntax analyzer for automatically assigning a syntax structure to text data of a sentence including a character string by using a decision tree with a probability for determining a syntax structure.

[0002]

2. Description of the Related Art Conventionally, a relatively accurate part-of-speech assigning system (hereinafter referred to as a first conventional example) is disclosed in Prior Art Document 1.
“E. Brill et al.,“ Some Advances in Transformation
--Based Part of Speech Tagging ", Proceedings of the
Twelfth National Conferenceon Artificial Intellig
ence, pp.722-727, AAAI, 1994 "and prior art document 2
“B. Merialdo et al.,“ Tagging English Text with a
Probabilistic Model ", Computational Linguistics, 20-
2, pp. 155-171, 1994 ". In this part-of-speech giving system of this conventional example, part-of-speech is given to text data by referring to a dictionary for giving part-of-speech, which describes a set of a word notation and a part-of-speech label taken by the notation.

In the part-of-speech assigning system of the first conventional example, part-of-speech is assigned using a dictionary. Therefore, it is difficult to assign a part-of-speech to an unknown word not described in a dictionary item. There is a problem that processing for an unknown combination with is difficult. Furthermore, there is a problem that it is necessary to maintain the dictionary by changing the part of speech system used. There is also a part-of-speech device that uses heuristics (heuristically or empirically) to assign a part-of-speech label to a word without using a dictionary. However, there is a problem that the accuracy rate of part-of-speech assignment is relatively low.

[0004] In order to solve the above problems, the present applicant has compared the first conventional example without using a dictionary for assigning the part of speech in the patent application of Japanese Patent Application No. Hei 8-232993. A part-of-speech assigning device (hereinafter, referred to as a second conventional example) that can be assigned automatically and accurately. The second conventional example of the part-of-speech providing apparatus includes: (a) a spelling feature of each word, a feature based on how to be used in a sentence,
Using a plurality of attributes including hierarchical classification using mutual information of words, and having a tree structure in a binary tree format that is divided depending on the attribute value of each attribute, Tree that generates a decision tree with frequency probabilities by calculating and assigning frequency probabilities for a plurality of parts of speech to a leaf node that is an undivided node of the generated decision tree. Means, and (b) using the decision tree with frequency probability generated by the decision tree learning means, based on the text data composed of the input word strings, the top among the frequency probabilities assigned to the leaf nodes. A product in which a plurality of n frequency probabilities are selected and given to each word of the text data, and a part of speech having the maximum connection probability in the word string of the text data is determined and output as a correct part of speech. It is characterized in that a supplying means.

[0005]

Further, as an English sentence parsing apparatus, a prior art document 3 “MJCollins,“ A new st
atistical parser based on bigram lexical dependenc
ies ", the 34th Annual Meeting of ACL Proceedings, 19
In 1996, learning bigram statistical information of a main word from text data to which a syntax structure has been added and performing syntax analysis (hereinafter, referred to as a third conventional example) is disclosed.

However, in the third conventional example, there is a problem that information of a detailed syntax structure such as a context-free grammar with a feature structure cannot be added.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems, and to provide a part-of-speech more accurately and automatically than a conventional example without using a dictionary for giving a part-of-speech. It is an object of the present invention to provide a parsing apparatus capable of adding detailed syntax structure information such as a context-free grammar.

[0008]

According to the first aspect of the present invention, there is provided a parsing apparatus according to the present invention, wherein a spelling characteristic of each word and a usage in a sentence are determined based on parsed text data composed of a word string. Has a tree structure such as a binary tree format that is divided depending on the attribute value of each attribute, using a plurality of attributes including the feature according to and the hierarchical classification using mutual information of words. A part-of-speech tree with frequency probability is calculated by generating a part-of-speech decision tree for part-of-speech assignment and calculating and assigning frequency probabilities for a plurality of parts of speech to a leaf node which is an undivided node of the generated part-of-speech tree. Based on the part-of-speech decision tree learning means for generating a tree, and the number of words to be processed, the part-of-speech of the head word to be processed, the part-of-speech of the word immediately before the processing target, Mutual information Grammatical rule determination for assigning grammatical rules with a binary tree structure that is divided depending on the attribute value of each attribute using a plurality of attributes including hierarchical classification using A grammar rule decision tree with a frequency probability is generated by generating a tree and calculating and assigning frequency probabilities for a plurality of grammar rules to a leaf node which is an undivided node of the generated grammar rule decision tree. Based on the grammar rule decision tree learning means and the parsed text data consisting of word strings, the number of words to be processed, the part of speech of the subject word to be processed, the part of speech of the word immediately before the processing object, mutual information of words Each of the parsing states in the grammar rule assignment process has a tree structure of a binary tree format that is divided depending on the attribute value of each attribute using a plurality of attributes including a hierarchical classification using the quantity. Processed by By generating a processing direction decision tree for determining a direction and calculating and assigning frequency probabilities for a plurality of processing directions to a leaf node that is an undivided node of the generated processing direction decision tree, Using a processing direction decision tree learning means for generating a processing direction decision tree with probabilities, and a frequency-probability part of speech decision tree generated by the above-described part-of-speech decision tree learning means, text data consisting of a word string to be input is processed. Based on the frequency probabilities assigned to the leaf nodes, n higher-order frequency probabilities are selected and assigned to each word of the text data, and the maximum joint probability in the word string of the text data is selected. Is determined as a correct part-of-speech sequence, and then each parsing in the grammar rule assignment process is performed using a predetermined stack decoder algorithm. After selecting the purging state in which the connection probability for the word string in the state has the maximum connection probability, the processing direction decision tree with the frequency probability generated by the processing direction decision tree learning means is used to select the purging state. Parsing is performed by determining a processing direction and adding a grammar rule to the word string to be processed in accordance with the grammar rule decision tree with frequency probability generated by the grammar rule decision tree learning means in the parsing state in the determined processing direction. Syntax information adding means for adding information and outputting parsed text data.

According to a second aspect of the present invention, in the parsing apparatus according to the first aspect, when each of the decision tree learning means performs the division in the form of the binary tree, the division before the division by each of the attributes is performed. The difference (H ₀ −) between the entropy H ₀ representing the priority of the validity of the attribute of the
H) selects the largest attribute as an attribute of a division candidate, and when a predetermined division continuation criterion is satisfied, the decision tree is updated by division in the form of a binary tree.

Further, the parsing device according to claim 3 is
3. The parsing apparatus according to claim 2, wherein the division continuation criterion is that (I) a difference (H ₀ −H) in entropy at the time of division based on the selected attribute is equal to or greater than a predetermined entropy threshold Hth. And (II) the number of events of a set of the attribute after division based on the selected attribute, the attribute value thereof, and the part of speech is equal to or more than a predetermined event number threshold Dth.

Further, the parsing device according to claim 4 is a parsing device according to claim 1, 2 or 3, wherein
The syntax information assigning means selects a plurality of upper n frequency probabilities from the frequency probabilities assigned to the leaf nodes and assigns the selected frequency probabilities to each word of the text data. Is used to limit the part-of-speech candidates, leaving only the part-of-speech candidates whose connection probability to the word string of the text data being processed is equal to or greater than the predetermined connection probability, and the largest part of the word string of the text data at the end of the part-of-speech processing. A part-of-speech sequence having a connection probability is determined as a correct part-of-speech sequence.

[0012]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a syntax analysis system including a decision tree learning device 10 and a syntax information providing device 11 according to an embodiment of the present invention. This part-of-speech assignment system does not refer to a dictionary for part-of-speech assignment to English text data, and then assigns detailed syntactic information such as context-free grammar with feature structure after assigning the part of speech. (A) an attribute list stored in an attribute list memory 22 and an attribute list stored in a part-of-speech list memory 23 based on the part-of-speech text data stored in the part-of-speech text memory 21 By referring to the part-of-speech list and executing a part-of-speech decision tree learning process, which will be described in detail later, by learning, a part-of-speech decision tree with frequency probability is generated and stored in the part-of-speech decision tree file memory 25. An attribute list stored in an attribute list memory 22 and a grammar rule based on the part-of-speech-added text data stored in a text memory 21 A grammar rule decision tree with a frequency probability is generated by executing a grammar rule decision tree learning process, which will be described in detail later, with reference to the grammar rule list stored in the strike memory 24 to generate a grammar rule decision tree file memory. 26, and based on the part-of-speech-added text data stored in the part-of-speech-added text memory 21, an attribute list stored in an attribute list memory 22 and a grammar rule list stored in a grammar rule list memory 24. The processing direction decision tree learning processing described in detail later is executed to perform learning, thereby generating a processing direction decision tree with frequency probability and processing direction decision tree file memory 27.
And a (b) stack memory 12 connected to the syntax information providing device 11 and a part-of-speech decision tree with frequency probabilities stored in a part-of-speech decision tree file memory 25 and a grammar rule decision tree file memory The attribute list stored in the attribute list memory 22 using the grammar rule decision tree with frequency probability stored in the file 26 and the processing direction decision tree with frequency probability stored in the processing direction decision tree file memory 27, With reference to the part of speech list stored in the list memory 23 and the grammar rule list stored in the grammar rule list memory 24, the part of speech given to the text data stored and input in the text data memory 30 will be described in detail later. By executing syntax information adding processing including (FIG. 7) and grammar rule adding processing (FIG. 8), a part of speech is added and a grammatical rule is added. To, characterized in that a syntactic information providing device 11 which generates and stores parsed text data in the parsed text data 31. In the present embodiment, the text data is an English sentence composed of an English word string.

First, grammar rules and knowledge used in the syntax analysis system of the present embodiment will be described. First, detailed grammatical rules used in the present embodiment will be described. Many linguistic phenomena appearing in a corpus, which is a learning text database, have linguistic characteristics by being organized as a unit. Also,
This feature is often very useful information for parsing. However, it is difficult to learn such features only from a corpus to which syntax information has been added by skeleton parsing. We avoid this problem by using a detailed grammar. As the grammar, a context-free grammar with a feature structure is used. This is because it is considered that by giving features that do not appear in the child node of the grammar rule to the parent node, more detailed information can be given and the features of each feature are easy to use. The grammar system is 66
Expressed as a combination of the values of the features of the species (an average of 12 values), 2,843 types of tag sets are used as part of speech tags by adding information on semantic categories in addition to syntactic categories. . The number of rules based on this tag set is 1,155, and each rule has a first head,
Information of the second head is given.

Various features are used to obtain the correct structure from the structure that follows the grammatical rules described above. First, the feature of the value of each feature given to the grammar is used. In addition to these features, the vocabulary features of the words themselves and the features of the sentences are used. In various contexts, examining the statistical properties of the values taken by the features of a grammar rule can give an indication of which grammar rule is likely. The characteristics of the vocabulary of the word itself are very effective information for determining a tag for the word, and are also used as information of the first head word in a certain range of the syntax structure. The vocabulary information handled in the present embodiment is not information obtained from the dictionary, but is characterized by the suffix and prefix constituting the word, the number of characters of the word, and the like. The sentence features include the number of words contained in one sentence, punctuation,
Alternatively, it is characterized by whether the same word appears more than once. Further, in order to use contextual information, features related to the immediately preceding word, the end of a sentence, the beginning of a sentence, the first word in the range covered by the grammar rule to be processed, the last word, and the like are also available. To describe various features, grammarists (specialists who generate grammar rules)
Use a framework to describe the characteristics of the grammar.

Next, a statistical parsing method used in this embodiment will be described. In the present embodiment, in order to efficiently use the above-described features, statistical properties are calculated using a learning corpus, and learning is performed as a decision tree with probability.
In the decision tree used in this method, a minimum cost-complexity algorithm was used for pruning, and a forward-backward algorithm was used for smoothing.
This decision tree is a binary tree. Therefore, the above-mentioned features cannot be directly used as the information of the branch point of the decision tree. Therefore, a unique bit string of “0” and “1” is given to each feature value, and specific bits are used. In addition, branching is performed to determine whether the feature is a valid feature. Here, for example, when processing the word at the beginning of the sentence, the feature related to the immediately preceding word cannot be used.

The syntax analysis of the present method is regarded as a process of continuously constructing a state representing a partial parse tree. To move from one state to the next, one of the following processes is performed. The above-described decision tree is formed for each process. (A) After assigning tags to words and determining syntactic features,
Determine semantic features. (B) Determine whether the current processing target is the end of the component. (C) Add a grammar rule to the current component to be processed. There are several possibilities for the order of these processes. In the present embodiment, first, all words are tagged with parts of speech, and the analysis is performed from left to right in a bottom-up manner. I have. The number of candidates generated from the grammar is very large, and it seems difficult to find the best candidate.However, in this method, the probability depending on the context is estimated using a decision tree.
The efficiency of processing is improved by using a stack decoder algorithm described in detail below.

Next, the configuration and operation of the parsing system of FIG. 1 will be described. The decision tree learning device 10 uses the part-of-speech-added text data composed of the word string read from the memory 21 to determine the spelling characteristics of each word, the characteristics of the words used in the text, and the mutual information amount of the words. And has a tree structure of a binary tree format that is divided depending on the attribute value of each attribute using a plurality of attributes stored in the attribute list memory 22 including a hierarchical classification using A part-of-speech tree with frequency probability is generated by generating a part-of-speech decision tree for assignment and calculating and assigning frequency probabilities for a plurality of parts of speech to a leaf node which is an undivided node of the generated part-of-speech decision tree. Is generated and stored in the part of speech decision tree file memory 25. Next, the decision tree learning device 1
0 is the number of words to be processed, the part of speech of the head word of the processing target, the part of speech of the word immediately before the processing target, the part of speech of the word, Using a plurality of attributes stored in the attribute list memory 22 including a hierarchical classification using mutual information, a tree structure in a binary tree format that is divided depending on the attribute value of each of the above attributes is formed. By generating a grammar rule decision tree for providing a grammar rule, calculating and assigning frequency probabilities for a plurality of grammar rules to a leaf node which is an undivided node of the generated grammar rule decision tree. Then, a grammar rule decision tree with frequency probability is generated and stored in the grammar rule decision tree file memory 26.

Further, the decision tree learning device 10 includes a memory 2
Based on the part-of-speech-attached text data consisting of the word string read from 1, the number of words to be processed, the part of speech of the subject head word to be processed, the part of speech of the word immediately before the processing object, and the mutual information of the words A plurality of attributes stored in the attribute list memory 22 including the used hierarchical classification and having a tree structure of a binary tree format that is divided depending on the attribute value of each attribute, Generate a processing direction decision tree for determining a processing direction in each purging state in the adding process, and calculate frequency probabilities for a plurality of processing directions for leaf nodes that are undivided nodes of the generated processing direction decision tree. Then, a processing direction decision tree with frequency probability is generated and stored in the processing direction decision tree file memory 27. Here, when the decision tree learning device 10 performs the division in the form of the binary tree, the entropy H indicating the priority of the validity of the attribute before the division by each attribute is used.
An attribute having the largest difference (H ₀ −H) between ₀ and the entropy H after the division is selected as an attribute of a division candidate, and when a predetermined division continuation criterion is satisfied, the attribute is divided into a binary tree and divided into a decision tree. To update.

Next, the syntax information providing device 11 uses the part-of-speech decision tree with frequency probability generated by the decision tree learning device 10 based on the text data composed of the word string to be processed input from the text data memory 30. hand,
A part-of-speech sequence having the maximum combination probability in the word sequence of the text data, selecting the top plural n frequency probabilities from among the frequency probabilities assigned to the leaf nodes and assigning them to each word of the text data. Is determined as a correct part-of-speech sequence, and then, using a predetermined stack decoder algorithm, after selecting a purging state in which the connection probability for the word sequence in each purging state in the grammar rule assignment process has the maximum connection probability, Using the processing direction decision tree with frequency probability generated by the decision tree learning device 10, the processing direction in the word string to be processed is determined, and the parsing state in the determined processing direction is generated by the decision tree learning device 10. By adding grammatical rules to the word string to be processed according to the grammar rule decision tree with frequency probability By applying a sentence analysis information to output the parsed text data.

Here, the syntax information providing apparatus 11 selects a plurality of upper n frequency probabilities from the frequency probabilities assigned to the leaf nodes and assigns them to each word of the text data. The stack decoder algorithm is used to limit the part-of-speech candidates, leaving only the part-of-speech candidates whose connection probability to the word string of the text data being processed is equal to or greater than the predetermined connection probability. The part-of-speech sequence having the maximum connection probability in the sequence is determined as the correct part-of-speech sequence.

In the present embodiment, a part-of-speech decision tree having a tree structure of a binary tree format and having a frequency probability for giving part-of-speech is obtained by using a part-of-speech decision tree learning process using knowledge obtained from text data to which part-of-speech has been added. Generate and give part of speech. The attributes used in the part-of-speech decision tree with frequency probability use linguistic features and statistical features obtained from a corpus. In the conventional part-of-speech assignment, a method is generally used in which part-of-speech candidates are limited by drawing a dictionary, and the most appropriate part-of-speech is selected from the candidates, taking into account the relationship with words that appear before and after. However, there is a problem of the cost for creating and maintaining the dictionary. In addition, special processing is required for words that are not included in dictionary items (unknown words) and words that are used as parts of speech that are not included in dictionary part-of-speech candidates. In the method using the part-of-speech decision tree with frequency probability according to the present embodiment, a dictionary is not used in order to determine the part of speech of a word, so that the cost of creating and maintaining the dictionary does not matter. The part-of-speech decision tree with frequency probabilities is constructed by learning using the part-of-speech-added text. Therefore, if there is text data with the part of speech added, it is possible to flexibly cope with the part of speech system. In addition, the priority of the part of speech sequence can be automatically determined using the frequency probability. The part-of-speech decision tree is a tree structure model that classifies a target into an appropriate class from a plurality of attributes and their attribute values. In the part of speech, the object corresponds to each word and the class corresponds to the part of speech. As the attribute, a spelling feature of each word, a feature according to a usage in a sentence, a hierarchical classification using mutual information of words, and the like are used.

In the grammar assignment process in the syntax information assignment device 11, a grammar rule is assigned to a word string to be processed using a grammar rule decision tree and a processing direction decision tree. Here, as attributes of the grammar rule decision tree and the processing direction decision tree, the number of words of the processing target, the part of speech of the head word of the processing target, the part of speech of the word immediately before the processing target, and the mutual information of the words were used. Use hierarchical classification. In the method using the grammar rule decision tree and the processing direction decision tree, a dictionary is not used in order to determine the addition of the grammar rule, so that the cost of creating and maintaining the dictionary does not matter. A grammar rule decision tree with a frequency probability and a processing direction decision tree are constructed by learning using a parsed text. Therefore, if there is parsed text data, it is possible to flexibly cope with the system of sentence rules. Hereinafter, the syntax analysis system of the present embodiment will be described in detail.

In FIG. 1, a decision tree learning device 10 stores an attribute list stored in an attribute list memory 22 and a part-of-speech list memory 23 based on text data with a part of speech stored in a text memory 21 with a part of speech. By referring to the stored part-of-speech list and performing learning by performing a part-of-speech decision tree learning process described in detail later, a part-of-speech decision tree with frequency probability is generated and the part-of-speech decision tree file memory 25 is generated.
Then, based on the part-of-speech-added text data stored in the part-of-speech-added text memory 21, the attribute list stored in the attribute list memory 22 and the grammar rule list stored in the grammar rule list memory 24 With reference to, a grammar rule decision tree learning process, which will be described in detail later, is executed to perform learning, thereby generating a grammar rule decision tree with frequency probability and storing it in the grammar rule decision tree file memory 26,
Further, based on the part-of-speech-added text data stored in the part-of-speech-added text memory 21, reference is made to the attribute list stored in the attribute list memory 22 and the grammar rule list stored in the grammar rule list memory 24. By executing and learning a processing direction decision tree learning process, which will be described in detail later, a processing direction decision tree with frequency probability is generated and stored in the processing direction decision tree file memory 27. Then
A stack memory 12 is connected to the syntax information providing device 11. The syntax information providing device 11 is stored in a part-of-speech decision tree with frequency probability stored in a part-of-speech decision tree file memory 25 and stored in a grammar rule decision tree file memory 26. Grammar rule decision tree with frequency probability and processing direction decision tree file memory 2
7, the attribute list stored in the attribute list memory 22 using the processing direction decision tree with frequency probability stored in
Referring to the part-of-speech list stored in the part-of-speech list memory 23 and the grammar rule list stored in the grammar rule list memory 24, the part-of-speech assignment described in detail later is performed on the text data stored and input in the text data memory 30. By executing the syntax information providing process including the process (FIG. 7) and the grammar rule providing process (FIG. 8), the POS is given and the grammatical rule is provided, the parsed text data is generated, and the parsed text data is generated. It is stored in the text data 31. Here, the generated parsed text data may be output to an output device such as a CRT display or a printer.

Here, the decision tree learning device 10 and the syntax information providing device 11 are, for example, CPs for executing respective processes.
U, a ROM (read only memory) for storing a program for each process and data necessary for executing the program, and C
It comprises a digital computer having a RAM (random access memory) used as a working memory of the PU. Further, the memories 12, 21 to 27, 30, 31
Is composed of, for example, a hard disk memory. Further, the syntax information providing device 11 includes a stack decoder
A stack memory 12 for executing part-of-speech assignment processing and grammar rule assignment processing using an algorithm is connected.

Table 1 shows an example of the part-of-speech list stored in the part-of-speech list memory 23. The attribute list memory 22
Tables 2 and 3 show examples of the attribute list stored in.
Table 4 shows an example of the grammar rules stored in the grammar rule list memory 24.

[0026]

[Table 1] Part of speech list ─────────────────────────────────── Part of speech tag meaning ────── ───────────────────────────── NN1INTER-ACT Singular common noun, Interaction NP1CITYNM Proper noun, City name IIIN Preposition INJJVGINTER-ACT Adjective Present participle of the target usage, mutual action VVGCONSUME present participle, consumption VVGRECEIVE present participle, acceptance …………………………………………… ─────────────────────

[0027]

[Table 2] Attribute List for Part of Speech ─────────────────────────────────── Attribute Attribute Value ─── Classification code using mutual information of words Hierarchical classification code Word of ~ ing "Yes, No Word of target word is" ~ ed "Yes, No Length of target word Numerical value of word length (for example," word "is 4) Value of part of speech attribute of previous word Part of speech attribute Value The value of the part-of-speech attribute of the current word The value of the part-of-speech attribute The end of the sentence is “?” Yes, No ……………………… ──────────────────────────

[0028]

[Table 3] Attribute list for assigning grammar rules ─────────────────────────────────── Attribute Attribute value ──相互 Mutual information of subject words to be processed Hierarchical word classification based on classification code Code (predetermined bit) Processing target is only one word Yes, No Part of speech of head word to be processed is noun Yes, No Part of speech of word immediately before processing is noun Yes, No ……………………………… …………………… ───────────────────────────────────

[0029]

[Table 4] Grammar rule list ────────────────────── nbarq4: N '→ N'I Noun phrase nbar1: N' → N1 Noun phrase n1a: N1 → N * noun element i1e: I → P preposition phrase p1: P → II * N 'preposition element ………………………… ─────────────── ─────────────── ───────

In the grammar rule list in Table 4, for example,
The first line means that the noun phrase is composed of a noun phrase and a prepositional phrase, and the third line means that the noun element is composed of a word having a noun part of speech. The line indicates that the preposition element is composed of a preposition and a noun element. The processing direction is not shown as a list, but is any one of “right”, “left”, and “up” in the present embodiment.

Here, the part of speech attribute is an attribute roughly classifying the part of speech into 32 types, and the value of the part of speech attribute is, for example,
v (verb), n (noun), determine (article),
punct (symbol). A hierarchical classification code using mutual information of words is described in, for example, Japanese Patent Application No. 8-02 / 98.
7809 Patent Application and Prior Art Document 4 “Akira Ushiod
a, “Hierarchical Clustering of Words”, Proceedings
of COLING'96, The 16th International Conference on
Computational Linguistics, Vol. 2, pp. 1159-1162, 1996
8 is a hierarchical classification code classified using the word classification method disclosed in "August". In this word classification method, words having relatively low frequency of occurrence in words in text data are classified based on a criterion of assigning words having a high percentage of adjacent words to the same word to the same class, and then the word classification result is classified into an intermediate layer. , An upper layer, and a lower layer, and using a predetermined average mutual information that is a global (overall) cost function for all words in the text data, , Upper layer, and lower layer. In the clustering method using mutual information, it is assumed that a text having the number of words T, a vocabulary having the number of words V, and a vocabulary division function π exist. This is a mapping function representing a division mapping (mapping) to a word class set C in the middle. The likelihood L (π) of the bigram class model that generates text data composed of a plurality of words is obtained by the following equation.

[0032]

## EQU1 ## L (π) =-Hm + I

Here, Hm is the entropy of the word distribution of the monogram, and I is two adjacent words in the text data.
Average mutual information about _two classes C ₁ and C ₂ (Aver
ageMutual Information; Hereinafter, the average mutual information,
Notated as AMI. ) And can be calculated by the following equation.

[0034]

(Equation 2)

Here, Pr (C ₁ ) is the probability of occurrence of a word of the first class C ₁ , and Pr (C ₂ ) is the second class C ₂
Pr (C ₁ | C ₂ ) is the conditional probability that a word of the first class C ₁ will appear after a word of the second class C ₂ has appeared, and Pr (C ₁ | C ₂ ) C ₁ , C ₂ ) is the probability that a word of the _first class C _{1 and} a word of the second class C ₂ appear adjacent to each other. Therefore, AMI is different first mutually Class C ₁ word and the second represented by the number 2
The relative frequency of the word class C ₂ is the probability of occurrence adjacent, divided by the product of the above first occurrence probabilities of the words in the classes C ₁ and a second class of C ₂ probability of occurrence of words Represents the ratio of Since the entropy H is a value independent of the mapping function π, the mapping function that maximizes the AMI also maximizes the likelihood L (π) of the text. Therefore, the AMI can be used as an objective function in the word class configuration.

The above-described word classification method can generate a tree structure having a balanced binary tree form in that words having similar meanings or syntactic characteristics are arranged at close positions. At the end of the process, the path from the root node (root node) to the leaf node (leaf node) is traced, and 0 or 1 representing a leftward branch or a rightward branch, respectively. By assigning one bit to each branch, a bit string (word bit) can be assigned to each word in the vocabulary.

Next, an algorithm of a decision tree learning process for constructing a part-of-speech decision tree, a grammar rule decision tree, and a processing direction decision tree, and an algorithm of a syntax information adding process will be described.

In each decision tree learning process, the validity of each attribute is calculated independently of the other attributes, and the classification order based on the efficient attributes for determining the class is determined by dividing the structure divided in the form of a binary tree. Construct as a tree structure having The validity of the attribute is evaluated based on the entropy H after division and classification according to the attribute. The entropy here indicates the priority of the validity of the attribute. That is, when dividing into a node N ₁ and a node N ₂ with a certain attribute B, the entropy H ₀ before the division,
And entropy H after division, the entropy H ₁ for node N _1, and the entropy H ₂ to node N ₂ is expressed by the following equation.

[0039]

(Equation 3)

H = p ₁ H ₁ + (1−p ₁ ) H ₂ where:

(Equation 5)

(Equation 6)

Here, p (tagall) is all part-of-speech tags before division (for a part-of-speech decision tree; a grammar rule tag for a grammar rule decision tree; a processing direction tag for a processing direction decision tree, ie, , “Up,” “left,” and “right.”), The frequency probability or appearance probability of the number of events, and Σ for tagall indicates the sum of all part-of-speech tags before division. P ₁ is the sum of the frequency probabilities of the number of events of the part-of-speech tag included when the node is divided into nodes N ₁ . Furthermore, p (tagN ₁₎ is the number of frequency probability events for all parts of speech tags of the node N _1, the Σ of tagn _1, indicates the sum for all parts of speech tags of the node N _1. p (ta
gN ₂₎ is the number of frequency probability of the event for all of the part-of-speech tag of the node N _2, for tagN ₂ of Σ
Indicates the sum for all parts of speech tags of the node N _2.

For the purpose of calculating the validity, event information (event: hereinafter referred to as an event) consisting of a set of "attributes, their attribute values, and parts of speech" is preliminarily extracted from the text data for learning. Specifically, an attribute that minimizes the entropy H after classification is obtained for a set of all events, and is assigned to the first node. The set of events is divided according to the attribute value of this attribute, and corresponding child nodes are created. A tree structure is constructed by repeating the same process at each child node. The condition for stopping the division is that the number of events included in each node is equal to or less than a certain number, or the effectiveness of the division is equal to or less than a certain reference (here, a predetermined amount having a difference between entropy H after division and entropy H ₀ before division). Is not exceeded.) Here, a node that is not divided is called a leaf. At the leaf of the learned decision tree, the frequency probability of each part of speech is calculated from a given set of events.

Here, in the syntax information adding system according to the present embodiment, the related art document 5 “LEBaum,“ Aninequality a
nd associated maximization technique in statistica
l estimation for probabilistic functions of a Mark
ov process ", Inequalities, Vol. 3, pp. 1-8, 1972", based on the learning data for smoothing using the Forward-Backward algorithm.
Smoothing is performed so that the difference between the probability obtained from the learning data for smoothing and the probability obtained from the decision tree is minimized, and the last frequency probability distribution to which the part of speech and syntax information are to be added is corrected. Further, in the system of the present embodiment, a two-stage decision tree is created according to the algorithm of the decision tree learning process. The first row shows the parts of speech roughly classified (hereinafter referred to as GPOS (Global Part Of).
Speech). ) (Here, it corresponds to one of the attributes of the actual part of speech, and is classified into, for example, a verb, a noun, an article, etc.). First, a decision tree for determining an actual part of speech (part of speech tag level shown in Table 1) is created. In the present embodiment, a more detailed part-of-speech level name is called a part-of-speech tag. That is, by generating the decision tree by dividing it into two stages, the storage capacity of the storage device required in one process is greatly reduced.

In the part-of-speech giving process, the text data of the input sentence is processed from left to right, and a part-of-speech sequence that maximizes the connection probability is output. The input sentence is composed of a plurality of N words such as w ₁ , w ₂ ,..., W _N , and the part-of-speech sequence {t ₁ , t ₂ ,.
Assuming that _N ｝ (where t _i is the part of speech of the i-th word) is obtained, the connection probability P is represented by the following equation. In addition,
In the present embodiment, the appearance of the part of speech is not treated as a Markov information source, but as an information source depending on the words and parts of speech that have appeared so far. Thus, in a sufficiently long sentence, it is in principle possible to derive the part of speech of the last word depending on the first word of the sentence and its part of speech.

[0044]

(7) P≡p (t ₁ , t ₂ ,..., T _N │w ₁ , w ₂ ,..., W _N )

(Equation 8)

The right side of the above equation (7) represents input sentences w ₁ , w ₂ ,.
When w _N is input, it means the connection probability that a part of speech sequence t ₁ , t ₂ ,..., t _N is given, and the right side of the above equation 8 indicates the input sentence w ₁ , w ₂ , w ₃ ,. w _n , and i given a part-of-speech sequence t ₁ , t ₂ ,..., t _i-1 up to the (i−1) th word
This means the probability obtained by multiplying the probability of the part of speech by i from 1 to n. Here, the symbol of Π represents i as 2
From N to N. Then, using a context-dependent attribute, the leaf leaf of the decision tree
(L) is derived, the frequency probability distribution associated with L is represented by p _L , and approximated as follows using the conditional distribution of the decision tree.

[0046]

L _i ≡ context w ₁ , w ₂ ,..., W _N , t ₁ , t ₂ ,.
Leaf led at t _i-1

P (t _i | w ₁ , w ₂ ,..., W _N , t ₁ , t ₂ ,
..., t _i-1 ) ≒ p _Li (t _i )

The contexts w ₁ , w ₂ ,...
w _N , t ₁ , t ₂ ,..., t _i-1 mean the context of the i-th word w _i . Further, the left side of Expression 10 is the context w ₁ ,
w ₂ ,..., w _N , t ₁ , t ₂ ,..., t _{i -1} represent the frequency probability or occurrence probability of the word t _i , which is the right-hand side of Equation 10 in the context L _i This means that the probability of taking the part of speech t _i can be approximated. Therefore, the coupling probability P to be maximized is as follows.

[0048]

[Equation 11]

As is apparent from the above equation 11, the connection probability P is represented by the product of the probabilities of the parts of speech t _i obtained depending on the context of each word of the input sentence. Further, in the part of speech processing for each word of the input sentence, the following two-stage processing is performed. (A) Calculate the frequency probability of each part of speech of the GPOS. (B) Using the decision tree corresponding to each part of speech of the GPOS, the frequency probability of the part of speech is calculated.

In the calculation of the frequency probability of each word, it is necessary to consider all the part-of-speech sequences that may have been obtained so far. When dealing with a fine part-of-speech system, the search range becomes enormous. Therefore, in this system, the related art document 6 “F. Jelinek,
“A fast sequential decodingalgorithm using a stac
k ", IBM Journal of Research and Development, No. 13, p
p.675-685, 1969 "and prior art document 7" D. Paul, "Alg
orithms for an optimal a * search and linearizing t
he search in the stack decoder ", Proceedingsof the
June 1990 DARPA Speech and Natural Language Work s
hop, 1990 ".
The part-of-speech sequence with the maximum frequency probability or appearance probability is searched for using an algorithm. This algorithm is a kind of graph search algorithm, in which the search range is temporarily limited by a threshold value, and the best evaluation value can be searched. In other words, selecting a part of speech sequence with the highest frequency probability from a plurality of parts of speech that may be given to each word is performed by using a graph that connects each part of speech as a node and nodes attached to adjacent words. The stack decoder algorithm treats a plurality of nodes collectively as a stack structure in a tree-structured path divided in a binary tree format, and searches the stack structure for the optimum path. By changing the range, an optimal route can be efficiently found.

FIG. 2 is a flowchart showing a part-of-speech decision tree learning process executed by the decision tree learning device of FIG. In FIG. 2, first, the parsed (part of speech added) text data stored in the parsed text data memory 21 is read out in step S 1 and written to the RAM in the decision tree learning device 10. Next, in step S2, the frequency probabilities of the combination of each attribute and the part-of-speech tag (the above p (tagall), p (tagN ₁ ), p (t
agN ₂ ). ) To calculate the decision tree learning device 1
Write to RAM in 0. Further, a part-of-speech decision tree with frequency probability is generated by executing a decision tree creation process in step S3, and the part-of-speech decision tree with probability created in step S4 is output to the memory 24 and stored.

FIG. 3 shows a grammar rule decision tree learning process (steps S11-S) executed by the decision tree learning device of FIG.
14 is a flowchart showing 14), which is executed similarly to the part-of-speech decision tree learning process of FIG. 2. FIG. 4 is a flowchart showing a processing direction decision tree learning process (steps S21 to S24) executed by the decision tree learning device of FIG.
It is executed in the same manner as the part-of-speech decision tree learning process of FIG. Here, the processing direction is a direction to be processed in each purging state in the grammar rule providing process, and limits how to change a processing target within a range to which the grammar rule is provided. Here, the parsing state, as shown in FIG. 13, refers to a state of being partially analyzed by the syntax information providing apparatus 11, and information on a node or word to be processed at present (specifically, A word, its part of speech information, and which one is to be processed). The goal state is a state in which the final result of parsing is input, and is a parsing state united by a grammar rule that combines one sentence as a sentence.

FIG. 5 shows a decision tree creation process (steps S3, S13, S2) which is a subroutine of FIG. 2, FIG. 3, and FIG.
It is a flowchart which shows 3). First, step S3
1 and entropy H after division by all attributes,
The entropy H ₀ before the division is calculated using Equations 4 and 3, respectively. Next, in step S32, the attribute having the largest entropy difference (H ₀ −H) is selected as the attribute of the division candidate, and it is determined whether or not the attribute selected in step S33 satisfies the division continuation determination criterion. here,
The division continuation criterion is defined as (I) a difference (H ₀ −H) in entropy at the time of division based on the selected attribute is equal to or greater than a predetermined entropy threshold Hth, and (II)
The number of events after division based on the selected attribute is equal to or greater than a predetermined event number threshold Dth. Step S3
If it is determined in step S3 that the division continuation criterion is satisfied, step S3
In step 4, two nodes divided by the attribute value of the selected attribute are created, that is, divided in the form of a binary tree to update the decision tree. Then, in step S35, the processing returns to step S31, and the processing from step S31 is repeated with each of the created nodes as a processing target. On the other hand, if the division continuation criterion is not satisfied in step S33, the process returns to the original main routine.

An example of the part-of-speech decision tree, the grammar rule decision tree, and the processing direction decision tree created in these decision tree learning processes will be described. Here, when "meeting in London" is used as the text data to be input, "[nbarq4 [nbar
1 [n1a meeting_NN1INTER-A
CT n1a] nbar1] [i1e [p1 i
n_IIIN [nbar1 [n1a London
_NP1CITYNM n1a] nbar1] p
1] i1e] nbarq4] ”is output.

FIG. 10 shows an example of the part-of-speech decision tree with frequency probability created in the above example. As shown in FIG.
The part-of-speech decision tree with frequency probabilities is represented by each of the attributes 101 to 10
5 has a tree structure divided in the form of a binary tree, and a frequency probability for each part of speech tag is assigned to the last leaf. In this example, the input sentence is "meeting inL
Ondon ", the part-of-speech tag NN1INTER-ACT (singular common noun, reciprocity) is attached to the word" meeting ", while the word" Londo "
n ”for the part of speech tag NP1CITYNM (proper noun,
City name).

FIG. 11 shows an example of a grammar rule decision tree with frequency probability created in the above example. As shown in FIG. 11, the grammar rule decision tree with frequency probabilities
Has a tree structure divided in the form of a binary tree by
In the last leaf, the frequency probability for each grammar rule tag is given. In this example, the input sentence is "meeti
ng in London ", a grammar rule tag nbarq4 (meaning a noun phrase composed of a noun phrase and a preposition element) is attached to the leaf node 401, while a grammar rule tag nbar1 is assigned to the leaf node 403. (Meaning a noun phrase composed of one noun element).

FIG. 12 shows an example of the processing direction decision tree with frequency probability created in the above example. As shown in FIG. 12, the processing direction decision tree with the frequency probability has the attribute 501
Has a tree structure divided in the form of a binary tree by
In the last leaf, a frequency probability for each processing direction tag is given. In this example, the input sentence is "meeti
In the case of “ng in London”, in the leaf node 601, the processing direction tag “up” is added and the current processing target “meeting” remains, and the processing of the grammar assignment decision tree for this processing target is continued. On the other hand, in the leaf node 603, the processing direction tag “left” is added, and the current processing target “[[in [[Londo
n]]]] ”and“ [meeting] ”on the left of the new object“ [meeting] [[in
[[London]]]] ”is performed, and the processing of the grammar assignment decision tree is continued.

FIG. 6 is a flowchart showing a syntax information adding process executed by the syntax information adding apparatus of FIG. In FIG. 6, first, in step S41, the part-of-speech decision tree file with frequency probability stored in the probability-part-of-speech decision tree file memory 25 is read and written into the RAM in the syntax information providing device 11, and the grammatical rule decision tree with probability is read. The grammar rule decision tree file with frequency probability stored in the file memory 26 is read out, written to the RAM in the syntax information providing device 11, and the processing direction decision tree with frequency probability stored in the probability direction processing tree file memory 27 is stored. The file is read and written to the RAM in the syntax information providing device 11. Next, in step S42, the text data to be analyzed stored in the text data memory 30 is read and written to the RAM in the syntax information providing device 11. Further, in step S43, a part-of-speech providing process is executed to generate part-of-speech-added text data.
At 44, parsed text data is generated by executing a grammar rule adding process for adding a grammar rule tag to the part-of-speech-added text data generated at step S43. Then, the parsed text data generated in step S45 is output to the parsed text data memory 31 and written.

FIG. 7 is a flowchart showing the part of speech giving processing (step S43) which is a subroutine of FIG. First, in step S51, a word at the beginning of the text data to be analyzed is set as a target word. Next, step S5
In 2, the root node at the top of the decision tree is set as the current node to be processed. Then, in step S53, it is determined whether the current node is a leaf node.
If NO, the corresponding child node is set as the current node based on the attribute value of the current node in step S58, and the process returns to step S53. On the other hand, in step S53, Y
If it is ES, the top n frequency probabilities in the frequency probability list assigned to the leaf node in step S54 (where n is a plurality, for example, preferably 3 to 6, more preferably Is 4.) and gives it to the target word. Then, in step S55, the part-of-speech tag candidates are limited according to the above-described stack decoder algorithm while leaving the connection probability P equal to or higher than the predetermined connection probability.
Further, it is determined in step S56 whether or not there is a next processing word. If there is, the next word is set as a target word in step S57, and the process returns to step S52 to repeat the above processing. On the other hand, if there is no next word in step S56, the part-of-speech tag string having the maximum joint probability P is determined as the correct part-of-speech string in step S59, and the process returns to the original main routine.

FIGS. 8 and 9 are flowcharts showing a grammar rule assignment process (step S44) which is a subroutine of FIG. First, in step S61 in FIG. 8, a parsing state for the word at the beginning of the sentence is generated. Then
In step S62, the purging state generated immediately before is added to the stack in the stack memory 12 determined by the number of processing direction determinations and the number of grammar rule determinations.
The determined stack means a data structure for recording each purging state. And step S
At 63, select the purging state with the largest joint probability according to the stack decoder algorithm described above;
In step S64, the processing direction is determined using the processing direction determination tree. Here, when the processing direction is “right”, a purging state in which the next word is to be processed is generated in step S67 via step S65, and then in step S62.
Return to When the processing direction is “up” in step S64, a purging state in which a grammar rule tag is added to the node to be processed according to the grammar rule decision tree in step S68 via steps S65 and S66, Proceed to step S70. Here, the processing direction is “up”.
In step S68, processing according to the grammar rule decision tree is performed on the current processing target. Further, when the processing direction is "left" in step S64, the parsing in which the range of the node to be processed is extended to the left in step S69 and the grammar rule tag is added according to the grammar rule decision tree via steps S65 and S66. After generating the state, the process proceeds to step S70.

In step S70, it is determined whether there is no unprocessed word and the grammatical rule is satisfied as one sentence. If NO, the process returns to step S62. If YES, the process returns to step S71 in FIG. move on. In step S71, the current purging state is added to the goal state, and a predetermined number of predetermined purging states (for example, N when obtaining the top N results) are determined as the goal state in step S72. It is determined whether or not
If the determination is NO, the process returns to step S62. If the determination is YES, the grammar rule assignment process ends, and the process returns to the original main routine.

In the above embodiment, the learning text data that has been parsed by one predetermined grammatical rule system is used. However, the present invention is not limited to this, and it is analyzed by another grammatical rule system G1. When there is a certain amount of text data obtained, information of another grammar rule system G1 may be used. At this time, the accuracy of syntax analysis can be improved. Here, the grammar rule system to be used is G0. In order to use the information of the grammar rule system G1, a part of the text analyzed by the grammar rule system G1 and a text analyzed by the grammar rule system G0 are created. Grammar rules G0 and G1
Using the same text data analyzed on both sides, a decision tree in which the characteristics of the grammar rule system G1 are reflected in the attributes to be used is learned. By inputting the text analyzed by the grammar rule system G1 using the decision tree reflecting the grammatical features of the grammar rule system G1, analysis using abundant input information becomes possible, and the accuracy of the syntax analysis is improved. Can be improved.

[0063]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present inventor conducted the following experiment using the parsing system configured as described above. It is not uncommon for automatic evaluation to have a problem with a plurality of correct data. Even when the specifications of the tag are determined in great detail, there are often a plurality of correct answers. However, it is not realistic to add a plurality of correct answers on the corpus for that purpose. Also, correct answers can be narrowed down in paragraph units, but in many cases it is not possible to narrow down in one sentence. Therefore, it is thought that this problem can be solved by setting test data in advance and adding a plurality of correct answers only to the data.
Evaluation data was created.

In this experiment, first, a part-of-speech tag is attached, and then syntax analysis is performed. At present, the accuracy of only the part-of-speech tagging is 74% with respect to one correct answer (in the case of evaluation using a plurality of correct answers, an accuracy improvement of about 10% is expected). Thus, Table 5 shows the experimental results of parsing with the part-of-speech correctly assigned, and Table 6 shows the experimental results of processing from text alone.

[0065]

[Table 5] 数 Number of words (length) Number of sentences Top 1 Rank Up to the top 20 Cross Constituent / Sentence ─────────────────────────────────── 1-10 1044 81 8.8% 95.0% 89.1% 7.6 11-15 248 30.2% 72.6% 43.1% 23.9 16-23 201 17.4% 48.3% 28.4% 34 .2───────────────────────────────────

[0066]

[Table 6] 数 Number of words (length) Perfect match Syntactic perfect Match ───────────── ─────────────── Top 1 Top 10 Top Top 1 Top 10 ───── １− 1-10 34.5% 40.1% 50.4% 62.3% 11 -15 1.2% 3.6% 11.3% 25.6% ──────────────────────────────── ───

"Cross" in Table 5 is the ratio of sentences in which the brackets of the first analysis result and the correct analysis candidate do not intersect at all. "Exact match" in Table 6 is the ratio of a perfect match literally, and "syntactic perfect match" is the ratio of the case where the bracket position, grammar rule name, and syntactic tag match completely. is there.

As is evident from the experimental results, very high accuracy is obtained when the number of words is relatively small, but the accuracy becomes very poor as the number of words in one sentence increases. This problem is not caused by the decision tree model, but is considered to be a problem of a method of searching for a correct answer. That is,
If the number of words in one sentence increases, it is considered that the structure of the correct answer has leaked from the analysis candidate during the processing. Therefore, it is necessary to improve a stack decoder algorithm or a search method, which is a design problem. Further, it is necessary to improve the accuracy of tagging.

As described above, according to the present embodiment, the connection relation between parts of speech, the relation between words and parts of speech, and the dependency between distant words or parts of speech are statistically processed.
A part of speech can be automatically and uniquely assigned with high accuracy, and a grammar rule can be assigned with high accuracy, so that a high-precision parsing system can be provided. In addition, since a part of speech label is assigned to a word without using a dictionary, a special process for an unknown word, which is a problem of the related art, is unnecessary. Furthermore, since the learning is performed using the parsed text data to which the parts of speech are added, it is possible to flexibly cope with many parts of speech systems. Furthermore, since detailed syntax information can be automatically added, the added syntax information can be used for a translation system, a speech recognition system, or an information search system. In addition, since data with a syntax structure including detailed information can be automatically generated, a large amount of data with syntax information can be stored.

[0070]

As described above in detail, according to the parsing apparatus of the first aspect of the present invention, the spelling characteristics of each word and the spelling of each word are determined based on the parsed text data composed of word strings. Tree that is divided depending on the attribute value of each attribute, using a plurality of attributes including features based on usage of words and hierarchical classification using mutual information of words By generating a part-of-speech decision tree having a structure and giving part-of-speech, calculating and assigning frequency probabilities for a plurality of parts of speech to a leaf node which is an undivided node of the generated part-of-speech decision tree, Based on a part-of-speech decision tree learning means for generating a part-of-speech decision tree with probabilities, and based on parsed text data consisting of word strings, the number of words to be processed, the part of speech of the subject head word to be processed, and the word immediately before the processing target Part of speech, word Using a plurality of attributes including a hierarchical classification using mutual information, and having a tree structure of a binary tree format that is divided depending on the attribute value of each attribute, the A grammar rule decision tree with a frequency probability is generated by generating a grammar rule decision tree and calculating and assigning the frequency probabilities for a plurality of grammar rules to a leaf node which is an undivided node of the generated grammar rule decision tree. Based on the grammar rule decision tree learning means that generates the word, the number of words to be processed, the part of speech of the head word to be processed, the part of speech of the word immediately before the processing, and the word Using a plurality of attributes including a hierarchical classification using mutual information, a tree structure of a binary tree format that is divided depending on the attribute value of each of the above attributes is used in the grammar rule assignment process. Each purging Generating a processing direction decision tree for determining a processing direction in a state, and calculating and assigning frequency probabilities for a plurality of processing directions to a leaf node which is an undivided node of the generated processing direction decision tree. And a processing target decision tree learning means for generating a processing direction decision tree with frequency probability, and a word string to be processed, which is input using the part of speech decision tree with frequency probability generated by the part of speech decision tree learning means. Based on the text data, a plurality of top n frequency probabilities are selected from among the frequency probabilities assigned to the leaf nodes and assigned to each word of the text data, and the largest one in the word string of the text data is selected. The part-of-speech sequence having the joint probability is determined as the correct part-of-speech sequence, and then, using a predetermined stack decoder algorithm, After selecting the purging state in which the connection probability with respect to the word string in the purging state has the maximum connection probability, using the processing direction decision tree with frequency probability generated by the processing direction decision tree learning means, the processing target word string is used. In the parsing state in the determined processing direction, and adding a grammar rule to the word string to be processed according to the grammar rule decision tree with frequency probability generated by the grammar rule decision tree learning means. A syntactic information adding means for adding the analysis information and outputting the parsed text data. Therefore, since the part-of-speech connection relation, the relation between words and part-of-speech, and the dependency relation between distant words or parts of speech are statistically processed, parts of speech can be automatically and uniquely assigned with high precision, and grammar with high precision Rules can be given, and a highly accurate parsing system can be provided. In addition, since a part of speech label is assigned to a word without using a dictionary, a special process for an unknown word, which is a problem of the related art, is unnecessary. Furthermore, since the learning is performed using the parsed text data to which the parts of speech are added, it is possible to flexibly cope with many parts of speech systems. Furthermore, since detailed syntax information can be automatically added, the added syntax information can be used for a translation system, a speech recognition system, or an information search system. In addition, since data with a syntax structure including detailed information can be automatically generated, a large amount of data with syntax information can be stored.

According to a second aspect of the present invention, in the parsing apparatus according to the first aspect, each of the decision tree learning means may be configured to divide the binary data in the form of the binary tree.
The attribute having the largest difference (H ₀ −H) between the entropy H ₀ indicating the priority of the validity of the attribute before the division by each attribute and the entropy H after the division is selected as the attribute of the division candidate, and the predetermined division is performed. When the continuation criterion is satisfied, the decision tree is updated by dividing in the form of a binary tree. Therefore, since the part-of-speech connection relation, the relation between words and part-of-speech, and the dependency relation between distant words or parts of speech are statistically processed, parts of speech can be automatically and uniquely assigned with high precision, and grammar with high precision Rules can be given, and a highly accurate parsing system can be provided. In addition, since a part of speech label is assigned to a word without using a dictionary, a special process for an unknown word, which is a problem of the related art, is unnecessary. Furthermore, since the learning is performed using the parsed text data to which the parts of speech are added, it is possible to flexibly cope with many parts of speech systems. Furthermore, since detailed syntax information can be automatically added, the added syntax information can be used for a translation system, a speech recognition system, or an information search system. In addition, since data with a syntax structure including detailed information can be automatically generated, a large amount of data with syntax information can be stored.

Further, in the parsing apparatus according to the third aspect, in the parsing apparatus according to the second aspect, the division continuation criterion may include: (I) a difference in entropy when the division is performed based on the selected attribute ( (H ₀ −H) is equal to or greater than a predetermined entropy threshold Hth, and (II) the number of events of a set of the attribute after division based on the selected attribute and its attribute value and part of speech is a predetermined event number threshold Dth or more. Therefore, since the part-of-speech connection relation, the relation between words and part-of-speech, and the dependency relation between distant words or parts of speech are statistically processed, parts of speech can be automatically and uniquely assigned with high precision, and grammar with high precision Rules can be given, and a highly accurate parsing system can be provided. In addition, since a part of speech label is assigned to a word without using a dictionary, a special process for an unknown word, which is a problem of the related art, is unnecessary. Furthermore, since the learning is performed using the parsed text data to which the parts of speech are added, it is possible to flexibly cope with many parts of speech systems. Furthermore, since detailed syntax information can be automatically added, the added syntax information can be used for a translation system, a speech recognition system, or an information search system. In addition, since data with a syntax structure including detailed information can be automatically generated, a large amount of data with syntax information can be stored.

Further, in the parsing apparatus according to the fourth aspect, in the parsing apparatus according to the first, second or third aspect, the part-of-speech assigning means is a higher-ranking one among the frequency probabilities assigned to the leaf nodes. After selecting a plurality of n frequency probabilities and assigning them to each word of the text data, using the predetermined stack decoder algorithm, the connection probability for the word string of the text data being processed is determined by the predetermined connection probability. The part-of-speech candidates are limited while leaving only the part-of-speech candidates described above, and the part-of-speech string having the maximum connection probability in the word string of the text data at the end of the process is determined as the correct part-of-speech string. Therefore, the connection between parts of speech, the relation between words and parts of speech,
In addition, since the dependence on distant words or parts of speech is statistically processed, parts of speech can be automatically and uniquely assigned with high precision, and grammatical rules can be assigned with high precision. A system can be provided. In addition, since a part of speech label is assigned to a word without using a dictionary, a special process for an unknown word, which is a problem of the related art, is unnecessary. Furthermore, since the learning is performed using the parsed text data to which the parts of speech are added, it is possible to flexibly cope with many parts of speech systems. Furthermore, since detailed syntax information can be automatically added, the added syntax information can be used for a translation system, a speech recognition system, or an information search system. In addition, since data with a syntax structure including detailed information can be automatically generated, a large amount of data with syntax information can be stored.

[Brief description of the drawings]

FIG. 1 is a block diagram of a syntax analysis system including a decision tree learning device and a syntax information providing device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a part-of-speech decision tree learning process executed by the decision tree learning device of FIG. 1;

FIG. 3 is a flowchart showing a grammar rule decision tree learning process executed by the decision tree learning device of FIG. 1;

FIG. 4 is a flowchart showing processing direction decision tree learning processing executed by the decision tree learning device of FIG. 1;

FIG. 5 is a flowchart showing a decision tree creation process (steps S3, S13, S23) which is a subroutine of FIGS. 2, 3 and 4;

FIG. 6 is a flowchart showing a syntax information providing process executed by the syntax information providing apparatus of FIG. 1;

FIG. 7 is a flowchart showing a part of speech giving process (step S43) which is a subroutine of FIG.

FIG. 8 is a flowchart showing a first part of a grammar rule assignment process (step S44), which is a subroutine of FIG.

FIG. 9 is a flowchart showing a second part of the grammar rule assignment process (step S44), which is a subroutine of FIG.

FIG. 10 is a diagram showing an example of a part-of-speech decision tree in a part-of-speech decision tree file with frequency probability created by the decision tree learning device of FIG. 1;

11 is a diagram showing an example of a part of speech decision tree in a grammar rule decision tree file with frequency probability created by the decision tree learning device of FIG.

12 is a diagram showing an example of a part-of-speech decision tree in a processing direction decision tree file with frequency probability created by the decision tree learning device of FIG. 1;

13 is a flowchart showing an example of a parsing state and a processing direction during processing in the syntax information providing apparatus of FIG. 1;

[Explanation of symbols]

 10: Decision tree learning device, 11: Syntax information adding device, 21: Parsed text data memory, 22: Attribute list memory, 23: Part of speech list memory, 24: Grammar rule list memory, 25: Part of speech decision tree file memory, 26: grammar rule decision tree file memory, 27: processing direction decision tree file memory, 30: text data memory, 31: parsed text data memory.

────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Stephane G. Eubank 5 Shiratani Sanai-ya, Seika-cho, Soraku-gun, Kyoto Pref. Ezra W .; Black, Stephen G .; Eubank, "Morphological Analysis by Decision Tree Learning", SIG-SL UD-9603-4, p. 19-p. 24 (Jan. 1997) (58) Fields surveyed (Int. Cl. ⁷ , DB name) G06F 17/20-17/28 JICST file (JOIS)

Claims (4)

(57) [Claims] 1. A spelling feature of each word, a feature according to how the word is used in a sentence, and a hierarchical classification using mutual information of the words, based on the parsed text data composed of the word strings. Using a plurality of attributes including a part-of-speech tree having a tree structure of a binary tree format that is divided depending on the attribute value of each attribute, A part-of-speech decision tree learning means for generating a part-of-speech decision tree with frequency probabilities by calculating and assigning frequency probabilities for a plurality of parts of speech to a leaf node which is an undivided node of the tree; Based on the text data, multiple attributes including the number of words to be processed, the part of speech of the subject word to be processed, the part of speech of the word immediately before the processing object, and hierarchical classification using mutual information of words Use each of the above Is a node that has a tree structure in the form of a binary tree that is divided depending on the attribute value of gender, generates a grammar rule decision tree for assigning grammar rules, and is an undivided node of the generated grammar rule decision tree. Grammar rule decision tree learning means for generating grammar rule decision trees with frequency probabilities by calculating and assigning frequency probabilities for a plurality of grammar rules to leaf nodes, based on parsed text data consisting of word strings By using a plurality of attributes including the number of words of the word to be processed, the part of speech of the subject word to be processed, the part of speech of the word immediately before the object of processing, and a hierarchical classification using mutual information of words, Generating a processing direction determination tree for determining a processing direction in each parsing state in a grammar rule assignment process having a tree structure of a binary tree format that is divided depending on an attribute value of each attribute; A processing direction decision tree learning means for generating a processing direction decision tree with frequency probability by calculating and assigning frequency probabilities for a plurality of processing directions to a leaf node which is a node that is not divided by the processing direction decision tree; Using the part-of-speech decision tree with frequency probabilities generated by the part-of-speech decision tree learning means, based on the text data consisting of the input word string to be processed, the top plural n in the frequency probabilities assigned to the leaf nodes Number of frequency probabilities are selected and given to each word of the text data, the part-of-speech sequence having the maximum connection probability in the word sequence of the text data is determined as the correct part-of-speech sequence, and then a predetermined stack decoder -Using the algorithm, the connection probability for the word string in each purging state in the grammar rule assignment process has the maximum connection probability After selecting the paging state, the processing direction in the word string to be processed is determined using the processing direction decision tree with frequency probability generated by the processing direction decision tree learning means, and the parsing state in the determined processing direction is determined. Syntactic information for adding parsing information by adding a grammar rule to the word string to be processed according to the grammar rule decision tree with frequency probability generated by the grammar rule decision tree learning means and outputting parsed text data A syntactic analysis device comprising: an assigning unit. 2. The decision tree learning means, when dividing in the form of the binary tree, entropy H ₀ indicating the priority of the validity of the attribute before the division by each attribute, and entropy H after the division. the difference (H ₀ -H) selects the greatest attribute as an attribute of the candidate dividing, when satisfying the predetermined division continue criterion, and updates the decision tree is divided in the form of a binary tree of The syntax analysis device according to claim 1. 3. The division continuation criterion includes: (I) a difference (H _{0) in} entropy at the time of division based on a selected attribute.
-H) is equal to or greater than a predetermined entropy threshold Hth, and (II) the number of events of the set of the attribute after division based on the selected attribute and its attribute value and part of speech is equal to or greater than a predetermined event number threshold Dth 3. The method according to claim 2, wherein
The parser described. 4. The syntax information providing means selects a plurality of upper n frequency probabilities from among the frequency probabilities assigned to the leaf nodes and assigns the selected frequency probabilities to each word of the text data. By using a stack decoder algorithm, only the part-of-speech candidates whose connection probability to the word string of the text data being processed is equal to or greater than the predetermined connection probability are limited, and the part-of-speech candidates are limited. 4. The parsing apparatus according to claim 1, wherein a part-of-speech sequence having a maximum connection probability in the word sequence is determined as a correct part-of-speech sequence.