JP2796690B2 - Example-driven natural language analyzer - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an example-driven natural language analyzing apparatus, and more particularly to a simple method for use in a machine translation system, an information retrieval system, a question answering system, etc., which can optimize an optimal structure for ambiguous sentences. The present invention relates to an example-driven natural language analyzer that can be selected with higher accuracy.

[0002]

2. Description of the Related Art The demand for computer-based language processing systems has been increasing, and research and development have been actively conducted. Recently, machine translation systems have been commercialized in various fields such as information retrieval systems and question answering systems.

[0003]

The language analyzer, which is an essential element of a computer-based language processing system, has conventionally had a large common problem. That is, it is not possible to select an optimal structure with high accuracy for an ambiguous sentence. For example, the involvement of a prepositional phrase is a typical example of causing structural ambiguity.

[0004] I present a paper at the conferenc
e. The preposition phrase "at the conference" in this example sentence can modify both the verb "present" and the noun "a paper".
In general, it is difficult to uniquely determine the destination of a prepositional phrase using a conventional analyzer based on grammar rules. In the computer processing, even when a plurality of destinations of a given preposition are possible, a human can usually determine the likelihood of the destination. In this example, the former is a more natural participant.

[0005] A method widely used at present is a method of designating priority information of a destination using a semantic marker or the like in a grammar rule or a dictionary. However, this method cannot deal with the case of the verb being free (for example, the above-mentioned preposition phrase "at the conference") or the case where the preposition phrase relates to a noun, aside from the essential case of the verb.

On the other hand, in recent years, stochastic grammars have been actively studied. Since this method optimizes the application of grammar rules, it is necessary to provide fine grammar rules at a level close to a word in order to resolve the ambiguity of prepositional phrases. It is not easy to describe such detailed grammar rules. Furthermore, even if it can be described, the learning time for optimization is enormous (the amount of computation is O if the size of the grammar is G).
(G ³ )).

There is a method using world knowledge and a discourse model for resolving structural ambiguity. However, this method has problems such as difficulty in creating knowledge and weight of processing.
Nothing can handle real text.

SUMMARY OF THE INVENTION Therefore, a main object of the present invention is to provide an example-driven natural language analyzer capable of selecting an optimum structure for an ambiguous sentence with high accuracy by a simple method.

[0009]

The invention according to claim 1 is
Input means for inputting a sentence of a natural language, analysis means for mapping the input sentence to an input substructure, and example storage for examining the entire input natural language sentence and storing frequently occurring example substructures Means, a thesaurus for prearranging words in the form of a tree based on the similarity of meaning, and obtaining a meaning distance between words in accordance with the meaning concept of the word, and an input partial structure and an example based on the mapped input structure Find the semantic distance between substructures based on the semantic distance calculated according to the thesaurus of words included in the substructures, search for an example substructure that minimizes the semantic distance between the substructures, and minimize the semantic distance between the substructures The likelihood calculating means for calculating the frequency of appearance of the example substructures and outputting the calculated semantic distance between the substructures and the frequency as the partial likelihood, and the calculated distance between the substructures and the frequency In accordance with the divided likelihood configured to include selecting means for selecting the input structure of the maximum likelihood indicating the most certainty, output means for outputting the structure of the maximum likelihood chosen.

[0010]

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

In the example-driven natural language analysis apparatus according to the present invention, a sentence of a natural language is input from input means, and the input sentence is mapped to an input structure by the analysis means. By examining the entire input natural language sentence and storing the substructures of frequently occurring examples, the inter-word semantic distance is obtained from the thesaurus based on the mapped input structure, and the inter-word semantic distance is minimized. Search for the example substructure, calculate the frequency of occurrence of the example substructure that minimizes the semantic distance between the substructures, find the likelihood for determining the likelihood of the structure, and determine the likelihood that indicates the most certainty according to the likelihood. By selecting and outputting the input structure of the likelihood, it is possible to select an optimum structure for an ambiguous sentence with high accuracy.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description, two embodiments in which the present invention is applied to the analysis of English will be described. Here, in particular, we deal with the dependency of prepositions. However, the present invention is effective in collecting other structural ambiguities, for example, to-infinitives, relative clauses, subordinate clauses, etc. is there.

For simplicity, the problem of deciding whether the plausible part of the preposition phrase "at the conference" in the above example sentence is the verb "present" or the noun "a paper" will be described. Treat as

FIG. 1 is a schematic block diagram of one embodiment of the present invention, and FIG. 2 is a diagram showing a specific example of the likelihood calculating section shown in FIG.

First, the input unit 1 includes a keyboard, a character recognition device, a voice recognition device, and the like. The input unit 1 inputs a sentence, divides the sentence into words at the same time, and gives information such as part of speech to each word according to a dictionary. Give to. The analysis unit 2 generates a possible structure for the input (input structure) by a conventional syntax analysis or a method based on pattern matching. Likelihood calculator 3
Includes an example storage unit 6 and a thesaurus 7, as shown in FIG. 2, and calculates the likelihood of each of the input structures. The likelihood calculating section 3 is a feature of the present invention, and will be described later in detail. The selection unit 4 selects the maximum likelihood structure from the likelihood calculated by the likelihood calculation unit 3 and the tree structure, and outputs the selected maximum likelihood structure to the output unit 5. The output unit 5 includes a display device, a printing device, and the like. Note that the likelihood, composite likelihood, and maximum likelihood used in this embodiment are defined for determining the certainty of the structure.

The above-mentioned likelihood calculating section 3 calculates the overall likelihood based on the partial likelihood calculated for each partial structure of the input structure. The partial likelihood is obtained by storing the example partial structure in which the meaningful distance (real value) between the partial structures with the input partial structure is the smallest, in the example storage unit 6.
, And the number (frequency) of the searched example partial structures. The inter-substructure semantic distance is determined based on the inter-word semantic distance calculated according to the thesaurus 7 of the words included in the input sub-structure and the example sub-structure. Here, the partial likelihood determination will be described in detail.

FIG. 3 shows an example sentence 2 corresponding to an ambiguous destination.
One input structure is shown here, and for the sake of simplicity, it is represented by what is called a dependent structure expression. FIG. 3A corresponds to the case related to the verb “present”, and FIG. 3B corresponds to the case related to “noun“ paper ”, and the dependency is indicated by a thick line. The likelihood calculation unit 3 uses these two possibilities, namely, “present, at, the conf”
erence "," a paper, at, the conference ". At this time, the example storage unit 6 and the thesaurus 7 shown in FIG. 2 are referred to. The example storage unit 6 is configured to be able to access using the partial structure as a key. In the case of the preposition described here, the partial structure is a triple set consisting of a verb (or a noun), a preposition, and a noun.

FIG. 4 is a diagram showing an example of the example storage unit.
In particular, FIG. 4A shows a case relating to a verb, and FIG.
Indicates a case relating to a noun. FIG. 5 is a diagram for explaining the calculation of the semantic distance between a part of the thesaurus and a word.

The thesaurus 7 is a dictionary in which words are systematized according to a common semantic concept. A part of this is shown in FIG. The meaning distance between words is calculated using the thesaurus 7. In this method, the inter-word semantic distance is calculated using the thesaurus 7
It is defined by the above semantic distance between concepts. The concept meaning distance is set to a value from 0 to 1 according to the position of the smallest common superordinate concept in the thesaurus 7. A value of 0 means that the two concepts are the same, and a value of 1 means that they are irrelevant. If the number of layers is (n + 1), from the bottom, 0, 1 / n, 2 /
.., 1 are assigned as distances. Thesaurus 7 in FIG.
Assigns 0, 1/3, 2/3, and 1 in the (3 + 1) hierarchy. For example, the smallest common superordinate concept between the two concepts "human" and "monkey" is "animal", and the distance is 1/3.
Therefore, the distance between the word "she" having the concept "human" and the word "chimpanzee" having the concept "monkey" is 1/3. In FIG. 5, d (x, y) represents a semantic distance between words (or concepts) x and y.

The semantic distance between partial structures is calculated as the sum of weights of distances between words. Input partial structure I
Assuming that the partial structure E is composed of columns of n words I _k and E _k , respectively, the meaning distance d (I, E) between the partial structures is calculated by the following equation.

[0026]

(Equation 1)

Although it is necessary to adjust the weight w _k so as to increase the accuracy rate, it is assumed here that w _k = 1 for simplicity.

Next, the likelihood of the entire input structure will be described. Here, first, considering a simple case where there is only one preposition phrase in the input sentence, the likelihood of the entire structure and the partial likelihood match. The selector 4 makes this comparison.

Next, specific processing will be described. The number of candidate candidates is n, the candidate (verb or noun) is x _i (1 <= i <= n), the preposition is p,
The object (noun) of p is y. Problem input "x _1,
, X _n , p, y ”is to select the maximum likelihood destination“ x _k ”of“ p, y ”. In the example sentence, the input is "pres
ent, a paper, at, the conference ”, and the correct answer is“ present ”. [Processing Procedure] (Step 1) The following processing is repeated (1 <= i <=
n).

[0030] The partial structure between the mean distance between the "x _{i p} y" calculated in accordance with equation (1) described above, to search for example partial structure of the minimum distance from the example storage unit 6. Storing the minimum distance d _i and frequency f _i of the example the partial structure of the same distance.

[0031] (Step 2) d _i is minimized x _i is 1
If only One does not exist, to end returns the x _i as the dependency destination.

(Step 3) If there is only one x _i with the largest f _i among the x _{i with} the smallest d _i , the process returns with x _i as the destination and ends.

(Step 4) Return all x _i in which d _i is the minimum as the destination and end. Specific example 1 When ending in step 2.

[0034] I present a paper at the conferenc
e. The shortest distance of "present, at, the conference" is 0.0
0, and the shortest distance of “a paper, at, the conference” is 0.33. Therefore, the former is decided. Specific example 2 When ending in step 3.

[0035] We have the next conference at the
hotel. In this case, "have, at, the hotel""the next confe"
The shortest distance of both candidates "rence, at, the hotel" was 0.17. An example of the former shortest distance is "hold,
"at, hotel" has a frequency of 92, and the latter example of the shortest distance is "meeting, at, hotel", and its frequency is 15; , "Have".

The procedure described above, the composite likelihood can be calculated from a partial example between mean distance d _i and frequency f _i of the example of the same distance,
For example d _i -f _i / 10 m (where ^m is dependent on the size of the example storage unit 6, the empirical is sufficient m = 6) is approximately equivalent to minimizing the.

According to the sum of the composite likelihoods of all prepositions, the maximum likelihood is determined. Next, a second embodiment will be described. The second embodiment is basically the same as the first embodiment. The difference is that the calculation of the likelihood and the partial likelihood is incorporated in the analysis unit 2. At the time of applying a grammar rule, a candidate for a destination is extracted, and at the same time, a partial likelihood calculation is performed.

The following grammar rules are prepared. (1) VP PP → VP (2) NP PP → NP The input sentence is assumed to be the following example sentence.

[0039] I present a paper at the conferenc
e. When the grammar rule (1) is applied, the substructure "present
, At, and the conference "can be extracted, and partial likelihood calculation of the partial structure is performed. When the grammar rule (2) is applied, a partial structure “a paper, at, the conference” can be extracted, and a partial likelihood calculation of the partial structure is performed. At the end of the analysis, the likelihood of the entire structure is obtained. Other points are the first
This is the same as the embodiment.

[0040]

As described above, according to the present invention, a sentence in a natural language is input and mapped to an input structure, and when a plurality of input structures are obtained, the likelihood of each structure is calculated. Since the maximum likelihood input structure is selected and output according to the degree, the maximum likelihood structure can be output by referring to an example for an ambiguous input sentence. In addition, approximate matching can be realized by using a thesaurus, and the number of examples can be reduced.
Furthermore, if an example is prepared for each field, it can be easily adjusted for each field. This method can be applied to any language, and can be easily incorporated into many conventional analysis methods (CFG, pattern matching, etc.).

[Brief description of the drawings]

FIG. 1 is a first example of an example-driven natural language analyzer according to the present invention.
It is a block diagram showing an outline of an example.

FIG. 2 is a block diagram illustrating an outline of a likelihood calculation unit illustrated in FIG. 1; ,

[Figure 3] Ambiguous input sentence "I present a paper at the con
FIG. 7 shows a possible input structure for "ference."

FIG. 4 is a diagram showing a part of an example storage unit.
(A) shows a case related to a verb, and (b) shows a case related to a noun.

FIG. 5 is a diagram for explaining calculation of a part of a thesaurus and a semantic distance between words.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Input part 2 Analysis part 3 Likelihood calculation part 4 Selection part 5 Output part 6 Example storage part 7 Thesaurus

────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Jin Iida Kyoto, Soraku-gun, Seika-cho, 5F, Inani, 5F, Sanpiraya Inside AT / R Automatic Translation and Telephone Research Institute, Inc. (56) References JP-A-4- 47364 (JP, A) JP-A-3-276367 (JP, A) JP-A-63-91776 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G06F 17/20-17 / 28 JICST file (JOIS)

Claims (1)

(57) [Claims] An input unit for inputting a sentence of a natural language; an analyzing unit for mapping an input sentence input by the input unit onto an input substructure; and a whole natural language sentence input from the input unit is examined. An example storage means for storing an example partial structure that frequently appears in the form of a word; and a thesaurus for prearranging the words in a tree form based on the similarity of the meaning and obtaining a meaning distance between words according to the concept of the meaning of the word; Based on the input structure mapped by the analysis means,
Calculating a semantic distance between partial structures based on a semantic distance calculated according to the thesaurus of words included in the input partial structure and the example partial structure; and storing the example partial structure having the minimum semantic distance between the partial structures as the example storage means. A likelihood calculating unit that calculates the frequency of occurrence of the example substructures in which the semantic distance between the partial structures is minimized, and outputs the calculated semantic distance between the substructures and the frequency as a partial likelihood; Selecting means for selecting the maximum likelihood input structure indicating the most certainty, according to the partial likelihood consisting of the distance between the partial structures and the frequency calculated by the calculating means, and outputting the maximum likelihood structure selected by the selecting means An example-driven natural language analysis device, comprising: