JP3908410B2 - Language analysis apparatus and method, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a language analysis device, and more particularly to a language analysis device based on lexicalized grammar that holds grammar rules in a dictionary of words.
[0002]
[Prior art]
A grammar is lexicalized when all the basic elements that make up the grammar, typically the parse tree, are all associated with any word. In other words, associating a grammar rule with a word is called lexicalization of grammar, and lexicalized grammar is called lexicalization grammar.
[0003]
As a natural language analysis device based on lexicalized grammar, for example, Japanese Patent Application Laid-Open No. 7-56921 (hereinafter referred to as “Literature 1”) describes a context free grammar as a “lexicalized context free grammar (LCFG)”. Describes a sentence analysis system that analyzes sentences after converting them into a lexicalized format.
[0004]
Also published in August 1996, Proceedings of the Force International Conference on Spoken Language Processing, Philadelphia, Pennsylvania, 1996 )) Entitled “Head Automata and Bilingual Tiling: Translation with Minimal Representations” (hereinafter referred to as “Reference 2”) ) Describes an analysis device that performs dependency grammar analysis and syntactic conversion between languages using an automaton associated with a word.
[0005]
There are also known ways to use lexicalized automata to streamline the analysis of lexicalized grammars, such as publications (Procedures of the First Workshop on April 1998).・ Glamour Compaction and Computing Sharing in as described in Tabulation in Parsing and Deduction (hosted by Proceedings of the First Tabulation in Parsing and Deduction, Paris, France, INRIA) The paper entitled “Grammar Compaction and Computation Sharing in Automaton-based Parsing” (hereinafter referred to as “Document 3”) is a finite automaton that accepts a sequence of non-terminal symbols in the leaves of a syntax tree associated with a word. To merge and minimize these finite automata associated with the same word. Accordingly, it is shown technique for efficient analysis.
[0006]
In this method, as described in section 3.4 of the above-mentioned document 3, a flat non-terminal symbol string (non-) in which a sequence of leafs is extracted and arranged in a line, not the syntax tree itself. terminal symbols).
[0007]
By creating one finite automaton that accepts this sequence for each syntax tree, merging and minimizing, common analysis processing is shared by a plurality of trees to improve processing efficiency.
[0008]
[Problems to be solved by the invention]
However, the lexicalized grammar analysis device described in each of the above-mentioned documents has a problem that the grammar format that can be handled is fixed throughout the system, and that it is difficult to make a flexible grammar description.
[0009]
For example, in the analysis device described in Document 1, the grammar format that can be analyzed is limited to context free grammar. This is clear from the configuration in which the grammar is converted to the lexical context-free grammar before the analysis proceeds.
[0010]
By the way, as is well known, there is a problem in context-free grammar that the arbitraryness of word order cannot be expressed concisely.
[0011]
For example, in Japanese, "I ate dinner slowly" and "I ate dinner slowly" are correct sentences and must be successfully analyzed.
[0012]
In Japanese, the order of verb case elements such as “ga”, “to”, and optional modifiers such as “slow” are generally free and may appear in any order.
[0013]
However, in the context-free grammar, the word order of the components on the right side of the rule is fixed, so this freedom (arbitraryness) of the word order cannot be expressed succinctly. It must be developed as a rule.
[0014]
Similarly, in the analysis apparatus described in the above-mentioned document 2, the grammar format that can be analyzed is limited to the dependent grammar.
[0015]
Further, in the analysis device described in Document 3, although there is a description that the target grammar format is not particularly limited, however, according to the configuration of configuring a finite automaton that accepts a sequence of non-terminal symbols in the leaves of the syntax tree, It is clear that the grammatical format to which the technique of the above-mentioned document 3 can be applied is limited. This is because a restriction is provided for a sequence of non-terminal symbols. Specifically, since this sequence is configured to be accepted by a finite automaton, after the syntax analysis is completed, what is permitted as a non-terminal symbol sequence of leaves of a syntax tree with a certain word as a head is Almost limited to grammatical forms that can be accepted by finite automata.
[0016]
In syntactic analysis, it has been common practice to share partial decomposition and improve analysis efficiency by controlling the analysis order based on dynamic programming. In the configurations of the above-mentioned documents 2 and 3 having an automaton to be controlled, there is no description on how to integrate the control of these two analysis orders to enable efficient analysis based on dynamic programming. It is.
[0017]
Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide a language analysis apparatus and method capable of handling lexicalized grammars in which the grammar classes accompanying the words are substantially different. There is to do.
[0018]
Another object of the present invention is to enable description and acceptance of general lexicalized grammars without being restricted by specific grammatical forms, and to enable efficient overall control of analysis based on dynamic programming. To provide a language analysis apparatus and method. Other objects, features, etc. of the present invention other than this will be readily apparent from the following description.
[0019]
[Means for Solving the Problems]
In order to achieve the above object, the language analysis apparatus of the present invention has a syntax in which a word in a dictionary accepts a syntax tree sequence formed by a set of syntax trees associated with the word and a sequence of syntax trees belonging to the syntax tree set. A dictionary configured to include a tree sequence accepting unit; and an analysis unit that advances an analysis of an input sentence using the syntax tree sequence accepting unit.
[0020]
According to the method of the present invention, a word in a dictionary is a set of syntax trees associated with the word, and a syntax tree sequence receiving unit that receives a syntax tree sequence created by a sequence of syntax trees belonging to the set of syntax trees. When analyzing a sentence input from the input means, (a) for each word in the dictionary, checking the formation of a syntax tree associated with the word;
(B) receiving, for each word in the dictionary, a syntactically permitted one from the syntax tree sequence created by the syntax tree established in step (a);
(C) repeating the steps (a) and (b) based on control by dynamic programming.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described. In an embodiment of the present invention, a word in the dictionary includes a syntax tree sequence receiving unit that receives a syntax tree set associated with the word and a syntax tree sequence created by a sequence of syntax trees belonging to the syntax tree set. The analysis unit that includes the word dictionary having the word information and advances the analysis of the input sentence executes the analysis using the syntax tree sequence reception unit corresponding to the word.
[0022]
As described above, in the embodiment of the present invention, not only a syntax tree but also a syntax tree sequence receiving unit that controls a combination of syntax trees is provided for each word, so that it is most suitable for grouping around words. You can select a grammatical form and use it.
[0023]
For example, for the incorporation of Japanese verb case elements and free modifier elements, a grammatical form that allows simple description of word order freedom is adopted and implemented as a verb syntax tree sequence acceptance part, while prefix modification to nouns. As shown in the above, it is easy to adopt a simple context-free grammar and implement it as a syntax tree sequence accepting unit when the word order is fixed.
[0024]
Further, in the embodiment of the present invention, the processing to be combined into one syntax tree and the combination processing between syntax trees are configured to be controlled by another automaton, and the analysis order is based on dynamic programming. By controlling this, it is possible to efficiently proceed with the analysis and obtain an optimal solution as a whole solution from partial decomposition.
[0025]
Specifically, at the stage where one parse tree is gathered, by registering the state of the parse tree sequence accepting part in the chart as a partial decomposition with the index, and packing (packing) the partial decompositions with the same index Thus, sharing of partial decomposition and analysis order control based on dynamic programming can proceed without contradiction.
[0026]
Further, in the embodiment of the present invention, the syntax tree associated with the words in the dictionary is configured as an automaton that accepts the final state sequence of the syntax tree sequence accepting unit, and the syntax tree sequence accepting unit includes the automaton. Is configured as an automaton that accepts a final state sequence of the syntax tree.
[0027]
With this configuration, the syntax tree can be configured as an arbitrary automaton, and flexible grammar description can be performed.
[0028]
Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.
[0029]
FIG. 1 is a diagram showing the configuration of the first exemplary embodiment of the present invention. Referring to FIG. 1, this natural language analyzing apparatus includes an input unit 1 for inputting an original sentence, a dictionary 2, an analysis part 3 that advances an analysis of the original sentence while referring to the dictionary 2, and a chart 4 for registering a partial analysis result. And an output unit 5 for outputting the analysis result.
[0030]
FIG. 2 is a diagram illustrating an example of the contents of the dictionary 2. Referring to FIG. 2, words in the dictionary 2 include a syntax tree pool 21 that stores a set of syntax trees, a syntax tree sequence reception unit 22 that receives a sequence of syntax trees allowed by the grammar, and a spelling of words. And word information 23 such as syntactic properties.
[0031]
FIG. 3 is a flowchart for explaining the operation of the first exemplary embodiment of the present invention. The operation of the first embodiment of the present invention will be described with reference to FIGS.
[0032]
When the original text input to the input unit 1 is sent to the analysis unit 3 (step 3-1), the analysis unit 3 refers to the dictionary 2 and divides the original text into words, and the contents of the dictionary of each word are stored in the dictionary 2. Is loaded and registered in the chart 4 (step 3-2).
[0033]
The analysis of the original text is based on the control of the analysis unit 3, and on the chart 4, according to the dynamic programming method, a linear order is assigned between all the partial sections according to the inclusion relation, and all partial analysis results for the partial sections are obtained. The process is repeated while moving the section of interest from a small section to a large section.
[0034]
Specifically, one unprocessed section is selected according to the linear order based on the inclusion relation (step 3-3). If such a section remains (Yes in Step 3-4), all partial decompositions of this section are obtained (Step 3-5), and the process returns to the section selection process (Step 3-3).
[0035]
When partial decomposition for all sections is obtained and there are no unprocessed sections (No in step 3-4), the partial decomposition over the entire section is sent to the output unit 5 as an analysis result (step 3-6).
[0036]
The output unit 5 outputs the sent analysis result to the outside (step 3-7).
[0037]
FIG. 4 is a flowchart showing details of the processing in step 3-5 in FIG. With reference to FIG. 4, the details of the process (step 3-5) for obtaining all partial decompositions for a certain section will be described.
[0038]
First, focus on words that are not included in the processed section (step 4-1). If such a word cannot be selected (No in step 4-2), the process ends.
[0039]
When a word that is not included in the processed section can be selected (Yes in step 4-2), processing of the word is started.
[0040]
First, the state of the word syntax tree sequence accepting unit is checked, and then one acceptable syntax tree is selected (step 4-3).
[0041]
Next, when an acceptable syntax tree can be selected (Yes in step 4-3), it is checked whether the syntax tree is established.
[0042]
The leaves of the syntax tree correspond to the final state of the syntax tree sequence accepting unit for words, and the syntax tree itself is implemented as an automaton that accepts the final state sequence registered on the chart 4.
[0043]
Whether or not the syntax tree is established is determined by whether or not this automaton reaches the final state.
[0044]
In step 4-5, the automaton of the selected syntax tree is activated to try to accept the final state sequence corresponding to the leaf.
[0045]
When the automaton reaches the final state (Yes in step 4-6), the syntax tree sequence accepting unit accepts the syntax tree and newly registers an edge having a new state in the syntax tree sequence accepting unit in Chart 4. (Step 4-8), the process returns to the process of selecting another syntax tree that can be accepted in the original word (Step 4-3). This is because, in general, there is not always one syntax tree that can be accepted in a certain state.
[0046]
On the other hand, if it was not accepted (No in step 4-6), analyze the reason why it was not accepted, and if the analysis progresses in the future and a new edge is created, whether or not it can be accepted. Determine (Step 4-7).
[0047]
When it is determined that there is a possibility that it can be accepted in the future (Yes in Step 4-7), the syntax tree automaton whose state transition has progressed halfway is registered in the chart 4 as an active edge (Step 4-9).
[0048]
On the other hand, when it is determined that there is no possibility of establishment in the future (No in step 4-7), the process returns to the process of selecting another acceptable syntax tree (step 4-3).
[0049]
In this way, the syntax tree string accepting unit selects the next acceptable syntax tree, and analyzes the automaton of the syntax tree itself to check whether or not the syntax tree is actually established, and repeats the analysis. Processing proceeds.
[0050]
Syntax tree string acceptor 22 is configured as a finite automaton in a simple case, but is not limited thereto, and may be a push-down automaton or a more complicated automaton. For example, an automaton provided with an auxiliary table for managing the number of transitions can be used for a set of state transitions so that the free word order can be handled.
[0051]
Since the syntax tree of a certain word and the syntax tree of another word are involved only via the final state of the syntax tree sequence receiving unit 22, the internal processing may be determined independently for the word.
[0052]
Therefore, a syntax tree sequence accepting unit is constructed using a pushdown automaton for a certain word, a finite automaton is used for another word, and a different form automaton is used for another word. The tree row receiving unit 22 can be configured. This is a major feature and advantage of the configuration of the present invention in which the syntax tree sequence accepting unit 22 is made independent of the process of accepting the syntax tree itself, and both are lexicalized and stored for each word.
[0053]
Furthermore, as is clear from the above description, the processing of accepting a syntax tree by an automaton for each word and the processing for checking whether a syntax tree is established are separated as separate processes. Regardless of the specific configuration of the column receiving unit 22, the entire analysis can be managed by the dynamic programming processing procedure (algorithm) using the chart 4, and the analysis can be efficiently performed while sharing the partial results. it can.
[0054]
This is made possible for the first time by the configuration of the present invention in which the syntax tree sequence accepting unit 22 is made independent of the process of accepting the syntax tree itself, and is a major feature and advantage of the present invention.
[0055]
FIG. 5 is a diagram for explaining a second embodiment of the present invention. Referring to FIG. 5, the third embodiment of the present invention records an input device 101, a data processing device 102 composed of a computer, an output device 103, a storage device 104, and a natural language analysis program. A storage medium 105. The recording medium 105 includes a magnetic disk, a magnetic tape, an optical disk, a semiconductor memory, and other recording media.
[0056]
The natural language analysis program is read from the recording medium 105 into the main storage device of the data processing device 102 and controls the operation of the data processing device 102. The data processing device 102 performs the following processing under the control of a natural language analysis program.
[0057]
When an input sentence is read from the input device 101, the analysis unit 103 is activated. The analysis unit 103 refers to the dictionary 102, recognizes a word in the sentence, loads the corresponding word dictionary, and registers it in the chart 4.
[0058]
When the dictionary of all words in the input sentence is registered in the chart 4, the analysis unit 103 starts analysis according to control based on dynamic programming. The analysis unit 103 determines one word of interest according to the control strategy, and activates the syntax tree sequence reception unit 22 in the word.
[0059]
The syntax tree sequence accepting unit 22 determines one acceptable syntax tree and starts checking whether the syntax tree is established on the chart 4. Since the leaves of the syntax tree are expressed as the final state of the syntax tree sequence accepting unit of the word, if the final states corresponding to the leaves of the syntax tree of interest are arranged in that order, the syntax tree is established on the chart 4. .
[0060]
If it is established, the syntax tree sequence accepting unit 22 accepts the syntax tree.
[0061]
When the syntax tree is accepted, the substructure of the accepted syntax tree is structured and the word is registered in the corresponding section of the chart 4.
[0062]
Furthermore, the next acceptable syntax tree is selected, and the condition check for the syntax tree is started. If the conditions for establishing the syntax tree are not satisfied, the processing is divided into two depending on the situation. If the analysis process proceeds and the syntactic tree sequence accepting unit 22 for another word reaches a final state, there is a possibility that the satisfaction condition may be satisfied. sign up. If there is no such possibility, the condition check fails.
[0063]
Analyzing this way, with the edge that stretches the whole, Syntax tree string acceptor If what is in the final state is obtained, the edge is output from the output unit 103 as the analysis result.
[0064]
【Example】
Next, embodiments of the present invention will be described with reference to the drawings.
[0065]
FIG. 6 is a diagram showing a configuration of a first embodiment in which the present invention is applied to an English (English) syntax analysis apparatus. Referring to FIG. 6, the CPU 7 that performs arithmetic processing, the storage device 8 that stores a dictionary, the keyboard 9 that inputs the original text, the input unit 1 that loads the original text from the keyboard 9 to the CPU 7, and the storage device. A dictionary 2, a chart 4 for recording the results of the analysis, a dictionary loader 6 for referencing the dictionary 2 to retrieve an input sentence and registering it in the chart 4, and an analysis control unit 3 for controlling the entire analysis process The output unit 5 outputs the analysis result to the outside of the CPU 7 and the display device 10 such as a CRT or LCD for displaying the output result.
[0066]
Further, as shown in FIG. 6, the dictionary of each word in the dictionary 2 includes a syntax tree pool 21, a syntax tree sequence accepting unit 22 that accepts a grammatically correct syntax tree sequence, and a syntactic property of the words. It comprises word information 23 which is various kinds of information including the beginning. Each component is connected via a data line / control line (communication line).
[0067]
FIG. 7 is a diagram showing a part of the contents of the dictionary 2 of the word “eats” in the first embodiment of the present invention. In the syntax tree pool 21 of the verb “eats” of the third person singular present tense, as a syntax tree directly related to itself, a syntax tree 7a that directly captures an object, a syntax tree 7b that represents backward free modification by an adverb or a preposition phrase. A syntax tree 7c representing subject import is registered. The syntax trees 7a, 7b, and 7c associated with the word “eats” included in the syntax tree pool 21 are each composed of a syntax tree (element) of depth (level) 1.
[0068]
In these syntax trees, the root node and one of its child nodes are specially marked as self.
[0069]
When two syntax trees are given, self nodes (one is a root node and the other is a leaf node) in the two syntax trees are overlapped with each other to connect the two syntax trees.
[0070]
Alternatively, the word itself may appear at the position of the self node in the syntax tree.
[0071]
When these syntax trees 7a, 7b, and 7c are concatenated into a large syntax tree, the path that follows self from the root self node and ends in the word itself is a syntax that uses that word as the head. Note that it is the spine of the tree.
[0072]
Symbols such as subject and dob that appear in other leaf nodes of the syntax tree Syntax tree string acceptor Corresponds to the final state.
[0073]
The syntax tree can be established when words having a final state designated on the left and right of self (that is, edges on the chart) appear.
[0074]
For example, the syntax tree 7a reaches the final state dob (direct object meaning) immediately after itself. Syntax tree string acceptor This is true only if an edge with a appears.
[0075]
The syntax tree 7b satisfies the satisfaction condition when the final state postmod (meaning postfix optional modifier) appears immediately after self.
[0076]
These syntax trees are implemented as an automaton that performs state transitions with the final state of the syntax tree sequence accepting unit as an input.
[0077]
Also, Syntax tree string acceptor 22 is a sequence of these three syntax trees.
Row 7a, 7b *, 7c
Configured to accept only.
[0078]
Here, the symbol “·” represents a concatenation of syntax trees, and “*” represents an arbitrary number of repetitions.
[0079]
That is, accept columns 7a, 7b *, 7c Syntax tree string acceptor 22 is configured to first accept the syntax tree 7a, then accept the syntax tree 7b by repeating it zero or more times, and finally accept the syntax tree 7c to reach the final state.
[0080]
A syntax tree sequence is identified as a large syntax tree by concatenating with the above self as a medium.
[0081]
For example, if a syntax tree corresponding to the columns 7a, 7b *, and 7c is represented, the syntax tree shown in FIG. 8 is obtained.
[0082]
In other words, “eats” Syntax tree string acceptor 22 is configured to accept the syntax tree shown in FIG. 8, and these two expressions are equivalent.
[0083]
FIG. 9 is a diagram showing a part of the dictionary contents of the word “he” in the first embodiment of the present invention. The syntax tree pool of “he” is empty, and the syntax tree sequence receiving unit 22 transitions from the start state to the end state subject with an input of empty (ε). In the word information, spelling: “he”, prototype: “he”, part of speech: pronoun is stored.
[0084]
FIG. 10 is a diagram showing a part of the dictionary contents of the word “dinner” in the first embodiment of the present invention. The syntax tree pool of “dinner” consists of Td1 which represents adjective prefix modification, Td2 which represents noun prefix modification, Td3 which represents qualifier modification, and Td4 which represents postfix modification by preposition phrase. Is
Syntax tree sequence (Td1 | Td2) * Td3 * {Td4}
Is configured to accept.
[0085]
Here, the symbol “|” indicates “OR”, and {} indicates that it can be omitted.
[0086]
That is, Syntax tree string acceptor Accepts an adjective prefix modification syntax tree Td1 or a noun prefix modification syntax tree Td2 any number of times, then accepts a qualifier phrase modification tree Td3 any number of times, and finally a postfix modification syntax with a preposition phrase Accept tree Td4 0 to 1 time to reach final state.
[0087]
The final state is a superposition of subject, dob, and prepobj.
[0088]
Next, the analysis processing of the first embodiment of the present invention will be specifically described.
[0089]
When the sentence “He eats dinner” is input from the keyboard 9, the CPU 7 is activated and input to the input unit 1 via the communication line.
[0090]
The input unit 1 converts the input from the keyboard 9 into a format that can be understood by the CPU 7 and sends it to the dictionary loader 6 via a communication line.
[0091]
The dictionary loader 6 refers to the dictionary 2 stored in the storage device 8, recognizes word boundaries in the input sentence, loads the dictionary contents of each word, and sends them to the chart 4 via a communication line.
[0092]
The chart 4 registers the dictionary contents on the chart 4 in units of words.
[0093]
FIG. 11 is a diagram illustrating a state of the chart 4 immediately after loading the dictionary.
[0094]
Chart 4 starts at position 0, ends at position 3, “he” from position 0 to 1 (e1), “eats” from position 1 to 2 (e2), “dinner” from position 2 to 3 ( e3) Registered. Each word is in its starting state, as shown in FIG. Syntax tree string acceptor Is attached. Hereinafter, the double line surrounding the state of the automaton indicates that the state is the current state. In FIG. 11, for simplicity, the syntax tree pool and word information are omitted. Syntax tree string acceptor Is attached to the word as well.
[0095]
When registration to the chart 4 is completed, the analysis unit 3 is activated.
[0096]
The analysis processing is basically performed in the same manner as the bottom-up chart method of context-free grammar from left to right under the control of the analysis unit 3.
[0097]
The order of selecting partial sections to be analyzed and the analysis using the active and inactive edges are the same as in the context-free grammar. However, in the case of context-free grammar, the partial analysis results registered in the partial section of the chart are distinguished by non-terminal symbols, whereas in the present invention, they are distinguished by the automaton state of the word syntax tree sequence acceptance unit. The
[0098]
In addition, for partial analysis results (inactive edges), the next applicable grammar rule is not selected from the global pool of grammar rules subject to non-terminal symbol matches, but the syntax associated with that edge. It is determined by the automaton of the tree row acceptance unit.
[0099]
First, for the first unprocessed section [0, 1], all partial decompositions of this section are obtained.
[0100]
Therefore, paying attention to the word “he” existing in this section, the syntax tree string accepting unit for the word “he” is activated.
[0101]
As described above, the syntax tree sequence accepting unit of “he” transitions to the final state with an empty (ε) syntax tree input, so the automaton reaches the final state subject without checking whether the syntax tree is satisfied. To do.
[0102]
Syntax tree string acceptor The final state is named according to what grammar function the accepted edge can have, and subject indicates that the edge having this final state can perform the grammatical function as the subject.
[0103]
In this way, since the acceptance of the syntax tree sequence has been completed, the syntax tree sequence acceptance unit newly creates an edge in the final state and registers it in the interval [0, 1] (e4).
[0104]
This new edge has the same information as “he” immediately after loading the dictionary, except that the syntax tree sequence accepting unit is in the final state.
[0105]
Since the analysis does not proceed any more in the interval [0, 1], the analysis unit 3 moves the attention interval to [1, 2] and starts a process for obtaining the total decomposition of this interval.
[0106]
Since there is only one word “eats” (more precisely, edge e2) in the interval [1,2], the syntax tree sequence accepting unit for the word “eats” is first activated.
[0107]
The word “eats” Syntax tree string acceptor Is configured to accept the syntax tree sequence 7a, 7b *, and 7c, and since there is only 7a that can be accepted immediately after the start state, it first checks whether the syntax tree 7a holds. .
[0108]
A condition for establishing the syntax tree 7a is that an edge having a final state dob exists to the right of self (self). However, since such an edge is not registered at this time, the syntax tree 7a is not established. However, if the analysis proceeds to the right in the future, such an edge may be newly registered. Therefore, the analysis unit 3 activates that the final state to be matched next from the syntax tree 7a is dob. Create an edge and register it in the interval [1,2].
[0109]
Next, the analysis unit 3 sets the attention interval to [0, 2] and tries to obtain all partial analysis results existing in this interval, but since there is no newly established syntax tree in this interval, Go to step.
[0110]
The analysis unit 3 sets the attention interval to [2, 3], and obtains all partial analysis results existing in this interval. Focus on the only word “dinner” in this interval (exactly edge e3) and start accepting the parse tree.
[0111]
The e3 syntax tree sequence accepting unit is a finite automaton configured to accept (Td1 | Td2) * Td3 * {Td4}.
[0112]
Td1, Td2, Td3, and Td4 are all accepted as the first acceptable syntax tree, but none of them is in a situation that can be established.
[0113]
on the other hand, Syntax tree string acceptor Can also accept an empty syntax tree and enter the final state, which transitions to the final state (dob | sub | prepobj).
[0114]
This final state has a state as a superposition of sub, dob, and prepobj, and expresses that it has a grammatical function as a direct object, subject, and preposition object. The edge having this final state is registered in the interval [2, 3] as the edge e5.
[0115]
When the processing of section [2, 3] is completed, the partial analysis result is newly registered, so the process proceeds to completion processing.
[0116]
In the completion process, it is checked whether or not the newly registered edge can be matched with the previously registered active edge. If matching is possible, a new edge is created.
[0117]
Here, since the active edge created from the syntax tree 7a is registered in the interval [1,2] and waiting for the final state of dob, the analysis unit 3 restarts the automaton of the syntax tree of 7a. . This automaton reaches the final state of the syntax tree itself by accepting an edge e5 having a final state of dob.
[0118]
As a result, the syntax tree 7a is accepted, and the analysis unit 3 determines that “eats” Syntax tree string acceptor The edge e6 that is advanced by one state is registered in the chart 4. The content is basically the same as the word “eats”, and the state of the syntax tree sequence acceptance unit is the state immediately after acceptance of the syntax tree 7a, and the next syntax tree to be accepted is 7b or 7c. Is different. FIG. 12 is a diagram showing the state of the chart at this time.
[0119]
Next, the analysis unit 3 enters the process of section [1,3]. Since the edge e6 registered in the previous completion process is registered in this section, the focus is shifted to this and the syntax tree sequence accepting unit is activated to check whether the next syntax tree can be accepted. To start.
[0120]
In e6, the next acceptable syntax tree is 7b or 7c. The syntax tree 7b satisfies the establishment condition if there is a word having the final state postmod, which is a free modification element, on the right side, and 7c satisfies the establishment condition if there is a word that reaches the final state of subj on the left side. Here, since there is an edge e4 having a final state subject immediately before, the condition for establishing the syntax tree 7c is satisfied, and this is accepted, and a new edge e7 is registered in the interval [0, 3].
[0121]
The e7 syntax tree string acceptor is in the final state. FIG. 13 is a diagram showing a chart in this state.
[0122]
The syntax tree shown in FIG. 13 is assembled from the syntax tree sequence received by the syntax tree sequence reception unit attached to “eats” in the process from e2 to e7.
[0123]
Thus, since all edges of the chart are extended and the syntax tree sequence receiving unit reaches the final state, the syntax analysis is successful. The solution is a syntactic tree sequence accepted by this edge.
[0124]
As described above, the syntax tree sequence can be compiled as a single syntax tree by superimposing self nodes, and the syntax tree thus created is equivalent to a solution. FIG. 13 shows the latter form.
[0125]
The output unit 5 receives this syntax tree and converts the data format into a form that can be displayed on the screen of the display device 10. When the conversion result is sent to the display device 10 via the communication line, the result received by the display device 10 is displayed to the user, and the analysis process ends.
[0126]
Although not shown in the above example, if the same edge is registered in the same section with the information related to the subsequent analysis processing, the efficiency that takes advantage of the dynamic programming method is packed by packing them. Analysis can proceed.
[0127]
Here, “the information related to the subsequent analysis processing is the same” means that the syntax tree sequence acceptance automaton associated with the edge is exactly the same including the current state. That is, when registering an edge, if you have a syntax tree acceptance automaton that is equivalent to you and an edge whose state is equivalent to you is already registered in the same section, stop registering it in the chart. Efficient analysis based on dynamic programming such as chart method and CKY method well known in the case of context-free grammar by packing in registered edges and registering them, and sharing the subsequent analysis processing The algorithm can be used as it is. The only difference is that in the context-free grammar, this condition is determined by a match of a non-terminal symbol, whereas in the present invention, this condition is determined by a match of the state of a syntax tree acceptance automaton. This is a major feature of the present invention, considering that the language analysis apparatus of the present invention can accept not only context-free grammar but a much wider range of grammar formats.
[0128]
FIG. 14 is a diagram showing the contents of the word dictionary in the second embodiment in which the present invention is implemented to constitute a Japanese analyzer. The configuration of the second embodiment of the present invention is the same as that of the first embodiment. Referring to FIG. 6, the CPU 7 that performs arithmetic processing, the storage device 8 that stores a dictionary, etc., and the original text are input. A keyboard 9, an input unit 1 that captures an original sentence from the keyboard 9 to the CPU 7, a dictionary 2 that is stored in the storage device, a chart 4 that records a result of the analysis, and a dictionary that is used to refer to the input sentence in the dictionary 2. The dictionary loader 6 registered in the chart 4, the analysis unit 3 that performs overall control of the analysis process, the output unit 5 that outputs the analysis result to the outside of the CPU 7, and the display device 10 that displays the output result are provided. Configured.
[0129]
Furthermore, the dictionary of each word in the dictionary 2 includes a syntax tree pool 21, a syntax tree sequence accepting unit 22 that accepts a grammatically correct syntax tree sequence, and various types of information including the syntactic nature of words. It is configured with certain word information 23. Each component is connected via a communication line.
[0130]
In the second embodiment of the present invention, the word dictionary contents are changed to Japanese as shown in FIG. 14 in order to implement the analysis device as a Japanese analysis device. This is different from the embodiment. In addition to simply changing the language, the structure of the automaton in the syntactic tree accepting unit is changed to suit the analysis of Japanese. The second embodiment of the present invention will be described below with reference to FIG.
[0131]
FIG. 14 shows a part of the dictionary contents of the verb “eat”. The syntax tree pool consists of only one Td, which is a syntax tree that fetches the preceding consecutive elements.
[0132]
The fact that “eating” has an essential case pattern of “to eat” is expressed not in the syntax tree but in an auxiliary table (acceptance count constraint table) provided in the syntax tree sequence receiving unit.
[0133]
The syntax tree sequence acceptance unit is an automaton whose state is the product of the finite automaton that forms the skeleton and the state of the acceptance number constraint table. Each time Td is accepted on the finite automaton side, the corresponding “number of times” column of the acceptance number restriction table is incremented.
[0134]
If the “number of times” column does not satisfy the constraints of the “number of times constraint” column, the process ends as an acceptance failure.
[0135]
With this mechanism, the structural constraint that the word “eat” can capture the particle “ga” once, the particle “ha” once, and any other optional modifiers any number of times in any order is simple. It can be expressed in form.
[0136]
On the other hand, for the description of grammatical phenomena in which the word order is fixed, such as the concatenation of nouns and particles, and the phenomenon that auxiliary verbs take in verbs, the order of acceptance of multiple syntax trees is limited as in English. It is natural to express with an automaton, and this is followed in the second embodiment of the present invention. Thus, it is a great feature of the present invention that the most appropriate grammar format can be adopted for each word according to the nature of the grammatical phenomenon related to the word.
[0137]
Conventionally, several formats have been proposed as lexicalized grammar formats that allow freedom of word order. For example, in 1994, a Ph.D. dissertation “Formal and Computational Aspects of Natural Language Syntax, PhD dissertation” submitted to the Department of Computer Science at Pennsylvania State University in 1994. to Universtiy of Pennsylvania, USA. 1994) (referred to as “Reference 5”).
[0138]
UVG-DL (Unordered Vector Grammar with Dominance Links), which is listed as an example, is a vector grammar whose unit is a group of multiple syntax trees. (link) is given.
[0139]
In this grammar, a syntax tree in a pair is used the same number of times, and a syntax tree connected by a dominating relationship link does not contradict the ancestor / descendant relationship (on the syntax tree). Must satisfy the constraint that it must be
[0140]
If this form is slightly modified, for example, it is easy to express the free word order of Japanese verb case elements concisely. However, such an expression format has been determined to be compatible with the description of the behavior of case elements in Japanese and German, and is not necessarily appropriate for the description of the behavior of other words. I can't say that. Of course, if the entire grammar form is chosen to facilitate the description of a particular grammatical phenomenon, it will naturally become inappropriate for the description of other phenomena.
[0141]
On the other hand, in the present invention, by having a syntax tree sequence accepting unit for each word, the substantial grammatical form followed by the word can be flexibly and easily changed for each word.
[0142]
In the second embodiment of the present invention, the grammar of the verb is implemented by a finite state automaton expanded by the application number constraint table, but this may be implemented by, for example, UVG-DL. Since the relationship with another word occurs only through the final state symbol of the syntax tree sequence accepting unit, no matter what processing is performed before reaching the final state, the influence is closed to that word.
[0143]
Further, as described in the first embodiment, syntax tree sequence acceptance is separated by separating the syntax tree formation check (final state acceptance processing by the syntax tree automaton) and syntax tree sequence acceptance processing. It is a feature of the present invention that an efficient analysis based on a dynamic programming method using a chart or the like can be performed regardless of the implementation of the part.
[0144]
A third embodiment of the present invention will be described. FIG. 15 is a diagram for explaining a third embodiment of the present invention. A case where analysis equivalent to the D-Tree Grammar shown in FIG. 3 is performed will be described.
[0145]
For more information on D-Tree Grammars, see “Parsing D-Tree Grammars” (1995) presented at International Workshop on Parsing Technologies (Parsing D-Tree Grammars) The description of “Document 5” is referred to.
[0146]
In the syntax tree shown in FIG. 15, links between S and VP are represented by dotted lines. This expresses that this link is not a fixed link having a length of 1, and that any number of syntax trees of 0 or more can be inserted between them. However, at the two nodes where the split syntax tree is connected to this syntax tree, the part of speech must be S or VP (depending on the split location).
[0147]
Such an expression is useful for expressing a long distance-dependent phenomenon such as WH movement and a response such as “so that”.
[0148]
In the present invention, the same can be realized by slightly expanding the syntax tree pool and storing and propagating the syntax tree in the pushdown stack.
[0149]
FIG. 16 shows the contents of the word dictionary for realizing FIG. This content is described in the word dictionary corresponding to V at the bottom in FIG.
[0150]
In the syntax tree pool, only the lowermost tree T1 in FIG. 15 is included in the syntax tree pool, and the other two trees above the broken line are stored in the pushdown stack. The syntax tree sequence accepting unit is configured to transition to a final state when T1 is accepted only once.
[0151]
When the syntax tree sequence accepting unit reaches the final state, the pushdown stack is extracted from the syntax tree pool and associated with the final state. In this state, the syntax tree string accepting unit may take out the syntax tree from the stack and continue accepting it, or accept itself as a child. Syntax tree string acceptor May be passed to However, in any case, the condition is that the non-terminal symbol of the extracted syntax tree matches the symbol in the final state. In the latter case, the stack propagates upward, and at some point, an attempt is made to accept the syntax tree in the stack.
[0152]
Fig. 17 schematically shows this situation. After the bottom VP grew as a child of another syntax tree (thick chain line), the top syntax tree in the stack popped. It shows how it is accepted.
[0153]
In this way, in the configuration of the present invention, the syntax tree is propagated in the stack, and when the acceptance processing of the syntax tree sequence is completed in the subsequent processing, the syntax tree is popped and the acceptance processing is performed. It is possible to easily handle the dominant relationship at a long distance as expressed by.
[0154]
Here, although a single syntax tree is pushed onto the stack, in general, it can be extended to push onto the stack in units of a syntax tree pool and a syntax tree string accepting unit.
[0155]
FIG. 18 is a diagram for explaining the processing of the response relationship of “so (adjective) that (sentence)”. Original text
“She is so kind that everybody likes her.”
At this time, the dictionary of “so” has a syntax tree (FIG. 18B) that summarizes the following “that” clause, and this is propagated as indicated by an arrow on the syntax tree of FIG. By starting acceptance when the whole is gathered as a sentence, it is possible to correctly summarize the sections below that.
[0156]
The responsive relationship between “So” and “that” clause is clear from the origin of the “that” clause syntax tree.
[0157]
It should be noted that, in general, when a context-free grammar is given, the analysis apparatus according to the present invention can be configured so as to be strongly equivalent to the original grammar by the following procedure.
[0158]
First, for each word, a syntax tree pool attached to the word is constructed as follows.
[0159]
When you fix a word (FIX), all the rules that lead that word from the preterminal are put into the syntax tree pool. At that time, the symbol self is given to the rule route in accordance with the pre-terminal symbol.
[0160]
Next, for a non-terminal symbol that is the root of a rule in the syntax tree pool, all rules having the symbol as a leaf are collected and added to the syntax tree pool. However, the symbol self is given to the noticed leaf and root. Even with the same rule, if the noticed leaf is different, the position of self is different, so it is added as another rule.
[0161]
This additional operation is repeated for the nonterminal symbol that is the root of the rule in the syntax tree pool until the rule addition stops. This completes the creation of the syntax tree pool.
[0162]
The syntax tree sequence accepting unit is a nondeterministic finite state automaton configured as follows. Each nonterminal symbol that is the root of a rule in the syntax tree pool is a state. From the start state, it is possible to make a transition to a state corresponding to the pre-terminal, and in each transition, a rule for generating a word from the pre-terminal of the transition destination is accepted.
[0163]
Furthermore, in the transition from state 1 to state 2, when there is a syntax tree in the syntax tree pool with state 2 as the root and state 1 as the leaf with the symbol self, this syntax tree is accepted and transition is made. To be configured. In addition, an empty syntax tree is accepted from all states to a final state, and transition is possible.
[0164]
With the above configuration, the analysis apparatus according to the present invention that accepts the same syntax tree set as the original context-free grammar can be configured.
[0165]
The present invention can be implemented with various modifications other than those described above.
[0166]
In each of the above-described embodiments, a phrase structure grammar (PSG) that exclusively uses a syntax tree as a grammar described in a word is adopted. And the contents of Syntax tree string acceptor It is also easy to configure. You may change this form for every word.
[0167]
Also, Syntax tree string acceptor As a matter of course, it is easy to adopt a push-down automaton instead of a finite automaton. Moreover, Syntax tree string acceptor In addition, by adopting a general automaton, it is easy to configure to accept a grammatical form with higher descriptive ability.
[0168]
Of course, the language analysis apparatus according to the present invention can be incorporated and applied to a translation apparatus, an information retrieval apparatus, voice recognition, and the like. Further, the present invention is not limited to natural language analysis, and it is obvious that the present invention can be easily applied to an artificial language (formal language) such as a computer programming language.
[0169]
As described above, the present invention includes various modifications within the scope according to the principle of the present invention.
[0170]
【The invention's effect】
As described above, according to the present invention, since the syntax tree sequence receiving unit has information about how to combine syntax trees that substantially control the grammatical form as the syntax tree sequence receiving unit, the word follows. The substantial grammatical form can be changed flexibly for each word, and therefore, there is an effect that a flexible and concise grammar description can be made according to the behavior of each word.
[0171]
In general, a grammatical form with high descriptive capacity requires a large amount of calculation for acceptance and has many problems from a practical point of view, but according to the present invention, such an acceptance part is placed and processing is actually performed. Stays in a few words that really need that descriptive ability.
[0172]
For many common words, it is sufficient to perform an accepting process with a light process, and according to the present invention, such a configuration can actually be adopted. Compared to the configuration that unifies everything in one grammatical form, it can be expected to be much reduced.
[0173]
Furthermore, according to the present invention, flexible grammar description independent of this specific grammatical format is possible, and as with context-free grammar, efficient analysis control based on dynamic programming is enabled. There is an effect.
[0174]
The effect of the present invention is due to the separation of the check of the establishment of the syntax tree itself and the check of the acceptance of the syntax tree sequence, and the effect is not obtained by the conventional system.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a natural language processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of a dictionary of the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 3 is a flowchart showing processing of the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 4 is a flowchart showing processing of the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 5 is a block diagram showing a configuration of a natural language processing apparatus according to a second embodiment of the present invention.
FIG. 6 is a block diagram showing a configuration of a natural language processing apparatus according to the first embodiment of the present invention.
FIG. 7 is a block diagram showing a configuration of a word dictionary in the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 8 is a diagram illustrating an example of a syntax tree accepted by the natural language processing apparatus according to the first embodiment of this invention.
FIG. 9 is a block diagram showing a configuration of a word dictionary in the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 10 is a diagram showing the contents of a chart in the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 11 is a diagram showing the contents of a chart in the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 12 is a diagram showing the contents of a chart in the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 13 is a diagram showing the contents of a chart in the natural language processing apparatus according to the first embodiment of the present invention.
FIG. 14 is a block diagram showing a configuration of a word dictionary in the natural language processing apparatus according to the second embodiment of the present invention.
FIG. 15 is a diagram illustrating an example of a syntax tree that can be processed by the natural language processing apparatus according to the third embodiment of the present invention;
FIG. 16 is a diagram showing an example of dictionary contents in the natural language processing apparatus according to the third embodiment of the present invention.
FIG. 17 is a diagram for explaining processing of the natural language processing apparatus according to the third embodiment of the present invention;
FIG. 18 is a diagram for explaining processing of the natural language processing apparatus according to the third embodiment of the present invention;
[Explanation of symbols]
1 Input section
2 Dictionary
3 Analysis Department
4 chart
5 Output section
6 Dictionary loader
7 CPU
8 Storage device
9 Keyboard
10 CRT
21 Syntax tree pool
22 Syntax tree string acceptance section
23 Word information
101 Input device
102 Data processing device
103 Output device
104 Storage device
105 Recording medium

Claims (3)

A syntax tree pool storing a set of syntax trees related to a word, and a syntax tree sequence receiving unit describing a sequence of syntax trees belonging to the syntax tree pool in an automaton;
A word dictionary having word information including word spelling and syntactic properties is stored in the storage means for each word;
In proceeding with the analysis of the original text input from the input means,
(A) dividing the original sentence into words, loading the contents of the word dictionary of the divided words from the dictionary, and registering them as an edge in a chart;
(B) determining a target edge from the edges registered in the chart;
(C) The state of the automaton corresponding to the target edge determined in the step (b) is checked with reference to the syntax tree sequence accepting unit in the target edge, and the current state of this automaton is changed to the next state. Selecting an acceptable syntax tree by transition to
(D) determining whether the syntax tree selected in step (c) holds on the chart;
(E) If it is determined in the step (d) that a syntax tree is established, the syntax tree is accepted, the state transition of the automaton is performed, a new edge corresponding to the automaton after the state transition is created, and the chart Registering with
(F) outputting the language analysis result of the original text by analyzing the contents of the chart;
A language analysis method comprising: Input means for inputting original text, dictionary storing word information, analysis means for analyzing the input original text while referring to the dictionary, and storage means for holding a chart for registering partial analysis results by the analysis means And an output means for outputting the analysis result,
In the storage means, a syntax tree pool storing a set of syntax trees related to words, a syntax tree sequence receiving unit describing a sequence of syntax trees belonging to the syntax tree pool in an automaton,
A dictionary having a word dictionary for each word with word information including word spelling and syntactic properties,
When the analysis unit proceeds with the analysis process of the original text input from the input unit,
(A) a function of dividing the original text into words, loading the contents of the word dictionary of the divided words from the dictionary, and registering them as an edge in a chart;
(B) a function for determining an edge of interest from among the edges registered in the chart;
(C) The state of the automaton corresponding to the target edge determined by the function of (b) is checked with reference to the syntax tree sequence accepting unit in the target edge, and the current state of this automaton is changed to the next state. A function to select an acceptable syntax tree by transition to
(D) a function of determining whether or not the syntax tree selected by the function of (c) is established on the chart;
(E) When it is determined that the syntax tree is established by the function of (d), the syntax tree is accepted, the state transition of the automaton is performed, a new edge corresponding to the automaton after the state transition is created, and the chart With the ability to register with
(F) a function of outputting a language analysis result of the original text by analyzing the contents of the chart;
A language analyzer characterized by comprising: A syntax tree pool storing a set of syntax trees related to words, and a syntax tree sequence accepting unit describing a sequence of syntax trees belonging to the syntax tree pool in an automaton;
A dictionary having a word dictionary for each word having word information including spelling and syntactic properties of the word is stored in the storage means of the computer,
When proceeding with the computer to analyze the original text input from the computer input means ,
(A) dividing the original sentence into words, loading the contents of the word dictionary of the divided words from the dictionary, and registering them as an edge in a chart;
(B) determining a target edge from the edges registered in the chart;
(C) The state of the automaton corresponding to the target edge determined in the step (b) is checked with reference to the syntax tree sequence accepting unit in the target edge, and the current state of this automaton is changed to the next state. Selecting an acceptable syntax tree by transition to
(D) determining whether the syntax tree selected in step (c) holds on the chart;
(E) If it is determined in the step (d) that a syntax tree is established, the syntax tree is accepted, the state transition of the automaton is performed, a new edge corresponding to the automaton after the state transition is created, and the chart Registering with
(F) outputting the language analysis result of the original text by analyzing the contents of the chart;
A recording medium that records a program for executing the program on a computer.

Images