JPH08171563A - Natural language processing system - Google Patents

Abstract

PURPOSE: To reduce and shorten the throughput and processing time through the efficient search processing of a syntax tree when a probable syntax tree is selected by using an evaluation point for the generation of the syntax tree. CONSTITUTION: The processing is advanced preferentially from a generated syntax tree which has evaluation points in a higher range through processing A (S11-S18) which is repeated up to the end of a sentence while only syntax trees given full evaluation points are regarded as objects of modification search processing and ended while a completed syntax tree is regarded as a completed syntax tree when there is the completed syntax tree by the advance to the end of the sentence and repetitive processing B (S19-S31) which is repeated up to the end of the sentence while only syntax trees having evaluation points above certain points are regarded as objects of search processing when there is no completed syntax tree in the processing A, ended while a syntax tree is regarded as a completed syntax tree when a 'sentence' is completed with the syntax tree with evaluation points, and performed for a next syntax tree having lower points unless the completed syntax tree is obtained; and syntax trees in the same evaluation point range are processed by a lateral search.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a natural language processing system, and more particularly to a syntax tree selection algorithm for parsing and semantic analysis.

[0002]

2. Description of the Related Art Natural language processing using a computer such as a word processor, machine translation, document database, and hypertext has been put into practical use.

In natural language analysis for this purpose, a sentence to be analyzed is first divided into morpheme units (minimum units of word structure), and morpheme analysis is performed to clarify the properties of each morpheme. After this, a syntactic analysis that analyzes from the syntactic rules of natural language, and then a semantic analysis that removes ambiguity and vagueness,
Perform contextual analysis.

In the syntactic analysis, it is determined whether or not the morphologically analyzed sentence is a correct sentence by using a grammar, and when the sentence is correct, a tree structure (a parse tree) is obtained as the syntactic analysis result.
The parsing algorithm that creates this tree structure includes a top-down type that searches from a sentence to a word and a bottom-up type that searches from a word to a sentence.

Further, when there is a plurality of possibilities in a subtree structure to be created next at a certain stage of parsing, a horizontal type (parallel type) for simultaneously making each of these structures in parallel and a search by selecting only one There is a vertical type (serial type).

In general, in parsing processing, since attention is paid only to grammatical conformity, syntactic ambiguity occurs,
Many parse trees will be generated. A semantic analysis process is performed to select the correct parse tree from among these.

In the semantic analysis process, not only the grammatical category of a word (corresponding to a part of speech) but also its semantic information is used.
The method is to remove the incorrect syntax tree.

However, when a natural language sentence is targeted, it is difficult to select only one correct syntax tree by such an alternative process. Therefore, the evaluation point method is used. .

According to this method, an evaluation point is given to the syntax tree, and the evaluation point of the syntax tree that is subjected to the syntactic or semantic check is deducted. Is the method of selecting.

However, if a syntax tree with a low evaluation point is also analyzed, the search space becomes too wide, and the analysis efficiency drops. Therefore, a threshold is set for determining the semantic compatibility of the syntax tree. The method of deleting the parse tree which became is adopted.

FIG. 3 shows a process in which the evaluation points are added to the syntax analysis algorithm by the bottom-up-horizontal search, and in the process of reading a word (S1) and then generating a syntax tree (S2). When the word dependency occurs, the semantic check process (S3) including the evaluation point deduction process is executed, and these processes are repeated until all the words of one sentence are read (S4), and the evaluation score is also the largest. A syntax tree is selected (S5).

Here, in the meaning check process (S3), it is judged whether the dependency of the word is semantically compatible (S3 ₁ ), and if it is judged as compatible by this judgment. The syntax tree is grown as it is (S3 ₂ ), and if it is judged that the dependency is semantically inappropriate, the evaluation point is deducted according to the type of semantic check (S3 ₃ ), and When the evaluation point of the syntax tree becomes lower than the threshold value (S
In 3 ₄ ), the syntax tree is deleted (S _{3 5} ) to prevent the syntax tree from becoming large. The threshold value is calculated by checking the number of words resulting from the morphological analysis (S6).

[0013]

Generally, it is difficult to analyze a long sentence as a problem of the syntax / semantic analysis processing. This is because when the number of words increases, so-called combinatorial explosion occurs, the efficiency drops extremely, and it becomes impossible to perform analysis within a realistic time. This problem is unavoidable even in the bottom-up method / horizontal exhaustive search method.

A case where the analysis by this method is applied to a simple example sentence "Taro ran." Will be described below. FIG.
The generation order of syntax trees is indicated by English characters, and these syntax trees are generated as shown in the following table by the following analysis procedure. The codes corresponding to the procedures are also shown in the table.

[0015]

[Table 1]

(1) Look at "Taro" at the beginning of the sentence.

(2) Generate all syntax trees that can be generated only by "Taro". This syntax tree becomes a and b in the table below.

(3) Go to the next word "ga".

(4) Generate all syntax trees that can be generated from the word "ga". This syntax tree becomes c and d in the table.

(5) All of the syntactic trees of the word "ga" and the word "Taro" that can be combined are combined. This syntax tree becomes e which is the combination of b and d in the table.

(6) The above (3) to (5) are repeated.
The syntax tree generated at this time is "run" from f to i in the table.
Is generated, j is generated from e and g, and k is generated from j.

(7) When the last word "ta" is seen, the syntax analysis is terminated. In the syntax tree generated at this time, l and m are generated from "ta", and n to u are generated from the combination with other words.

(8) From the generated syntax tree, a word that can generate a "sentence" using all the words is selected. The syntax tree selected at this time becomes s.

As in the above example, a large number of syntax trees are generated even for the analysis of a simple sentence, and the number of syntax trees becomes enormous in the case of a long sentence, which makes the analysis difficult.

The purpose of the semantic analysis processing is to delete incompatible syntax trees and select one correct parse tree as the final result. In the deletion of this syntax tree, strengthening the semantic check is effective in reducing the number of remaining syntax trees, but in some cases, all of the syntax trees fall below the threshold value, and the analysis result shows nothing. It may happen that you do not get it.

On the contrary, loosen the semantic check,
If the threshold value is too low, many high-scoring parse trees remain until the end, making it difficult to select the correct parse tree or suppressing the combinatorial explosion and reducing the efficiency. As described above, the conventional method that relies only on the threshold value has a problem.

An object of the present invention is to provide a natural language processing system capable of reducing the processing amount and the processing time by an efficient search process of a syntax tree in the processing of selecting a likely syntax tree by using an evaluation point for generating the syntax tree. To provide.

[0028]

In order to solve the above problems, the present invention parses an input sentence for each word by a bottom-up type / horizontal type search mechanism to generate a syntax tree. Each time it is generated, a semantic check process is performed to deduct the evaluation point of the syntax tree according to whether or not the dependency of the word is semantically compatible.
In the natural language processing system that performs a semantic evaluation, the semantic check processing is repeated only until the end of the sentence for the dependency search processing targeting only the syntax tree with a perfect evaluation score, and the deducted syntax tree data is stored in the storage area. Processing A which stores the syntax tree and ends the processing as a completed syntax tree when the syntax tree is completed up to the "sentence" by the processing up to the end of the sentence
When the "sentence" is not completed in the process A, the process is repeated until the end of the sentence with only the syntax tree having the evaluation point having a certain score or more out of the syntax tree data having the deduction point, and the evaluation point When a "sentence" is completed with a syntax tree that has, the processing ends as the completed syntax tree, and when the completed syntax tree cannot be obtained, the syntax tree with a lower score or higher is processed next. It is characterized in that it is provided with repetitive processing B.

[0029]

The processing is preferentially proceeded from the syntax tree having a higher evaluation point in the generated syntax tree, and the horizontal search is performed on the syntax trees having the same evaluation point range.

As a result, the syntax tree to be searched does not run out, the wasteful search of the syntax tree is reduced, and the processing time is shortened.

[0031]

1 is a flow chart showing an embodiment of the present invention. The present embodiment is roughly divided into two stages, a process A of S11 to S18 and a process B of S19 to S31.

In the process A, the analysis is carried out in the same manner as the conventional algorithm, but only the evaluation points having a perfect score are targeted, and the deducted syntax tree data is stored in the storage area. When the processing proceeds to the end of the sentence and the syntax tree is completed up to the “sentence”, it means that a syntax tree with an evaluation score of 100 has been obtained, and this is output and the processing ends.

When the "sentence" is not completed in the process A, the process proceeds to the process B. The process B proceeds from the beginning of the input sentence to the analysis again, but proceeds while calling the syntax tree data saved in the process A. At this time, only the syntax trees having a certain score "point of interest" or more are processed. Proceed to the end of the sentence, and if the sentence is completed with the syntax tree having the evaluation score of the point of interest, output and end. If not, the "point of interest" is set to the next lower value and the process B is repeated from the beginning of the input sentence.

Therefore, from the highest evaluation point,
This is a method in which a syntax tree with equal evaluation points is searched, and this is called "contour point search". It can be said that the part having higher evaluation points is prioritized in the vertical type search, while the parts having the same evaluation points are processed in the horizontal type because they are processed simultaneously.

The processes A and B will be described in detail below. The process A becomes the following (1) to (6), and the process B becomes (7) to (11).

(1) The next candidate point is set as undetermined, and the analysis is started by the bottom-up method as in the conventional method (S11 and S12).

(2) Generate a syntax tree that is formed by the word W of interest alone (S13). Since these syntax trees have no dependency, the evaluation points are all 100 points.

(3) The evaluation point is calculated by performing a dependency search process between the syntax tree generated by the analysis up to the word immediately before the target word and the syntax tree generated in the process S13. The syntax tree whose evaluation points have been deducted is paired with that evaluation point and where it was deducted (that is, the position in the sentence of the word W that has undergone the modification) and is temporarily stored as candidate data (S
14).

(4) Only the syntax tree (that is, 100 points) whose evaluation points have not been deducted is passed to the dependency search processing for the next word W '(S15). If the next candidate point is undecided, the value of the next evaluation point in the syntax tree data saved in step S14 is set as the next candidate point. If the next candidate point is not yet determined, the larger one is set as a new next candidate point by comparison and passed to the next process.

(5) Until the processing reaches the end of the sentence, the word W'is set as a new word of interest and the process returns to step S12 (S1).
6).

(6) If there are syntax trees completed by the processing up to the end of the sentence, those are output and the processing is terminated (S1).
7 and S18).

(7) The next candidate point delivered from the process S17 or S29 is set as the "point of interest" and the next candidate point is set again as undecided (S19).

(8) Regarding the syntax tree to be the next candidate point, it is checked whether or not there is a generated syntax tree saved by the search processing from the beginning of the sentence to the next word (S20 to S22). Dependency search processing is performed with the syntax tree formed by the focused word W alone. Of the syntax trees generated by the dependency search processing, those with the number of points of interest or more are left, and other than that are temporarily stored as new candidate data in the same manner as in processing A (S23 and S24). If there is no delivered syntax tree data, it is saved as no new candidate data (S2).
5). The processing of the next candidate point is performed in the same manner as the processing A.

(9) With respect to the candidate data corresponding to the word of interest W (old and saved last time), candidate data having the number of points of interest are selected, and the candidate data of the word of interest are merged to generate them. Is passed to the processing of the next focused word W '(S26).

(10) If it is not at the end of the sentence, the noticed word W'is set as a new noticed word and the process returns to step S21. When the process up to the end of the sentence is completed, it is checked whether the syntax tree is completed (S27 and S27). S28).

(11) If the syntax tree is completed, it is output (S29). If the syntax tree is not completed, the process S is performed.
Return to 19 to see new candidate data (S30). If there is no candidate data, the process ends as a syntax analysis failure (S31).

Therefore, in this embodiment, as shown in the syntax tree generation example in FIG. 2, in the process A, the syntax tree consisting of one word and the syntax tree generated by the dependency search have a high evaluation score. This is a contour point search only for the syntax tree of.

If there is a syntax tree completed by this process A, the analysis ends at that point, and the number of syntax trees generated by limiting the horizontal search to those with a perfect score is reduced,
The number of search processes can be significantly reduced.

Then, when the syntax tree completed by the process A cannot be obtained, the horizontal search is performed only on the syntax trees having a certain number of points of interest among the syntax trees deducted, and when there is a completed syntax tree. When the analysis is completed, and when there is no parse tree, find a parse tree that is completed by iterating only parse trees that are equal to or more than the next number of points of interest.

Also in this process B, the horizontal search is performed on the syntax trees in the order of the higher evaluation points, so that the number of generated syntax trees can be reduced and the number of search processes can be greatly reduced.

Moreover, in the conventional threshold value setting method, the processing cannot be performed when the syntax tree above the threshold value is lost, and the useless search increases when the threshold value is lowered. In the example, the threshold is equivalently lowered stepwise from the highest point to the lowest point, and the most probable syntax tree can be obtained without causing processing failure and unnecessary search.

[0052]

As described above, according to the present invention, an evaluation point is set in the generated syntax tree, and the evaluation point is deducted every time it is determined as nonconformity in the semantic check processing, and the evaluation point is related. In the process of selecting the parse tree to be the target of the search process, only the parse tree with the highest evaluation score is involved and the process is repeated until the end of the sentence as the target of the search process. The process A for obtaining the syntax tree is performed, and when the "sentence" is not completed in this process A, the process B for obtaining the completed syntax tree by the horizontal search is preferentially repeated from the syntax tree having a high evaluation point. Therefore, the syntax tree to be searched is not lost, and the wasteful search of the syntax tree can be reduced and the processing time can be shortened.

[Brief description of drawings]

FIG. 1 is a flowchart showing an embodiment of the present invention.

FIG. 2 is an example of syntax tree generation in the embodiment.

FIG. 3 is a flowchart showing a conventional syntax / semantic analysis algorithm.

FIG. 4 shows an example of a conventional horizontal exhaustive search syntax tree generation.

Claims (1)

[Claims] 1. A bottom-up type / horizontal type search mechanism parses an input sentence for each word to generate a syntax tree, and each time the syntax tree is generated, the dependency of words is semantically compatible. Semantic check processing is performed to deduct the evaluation point of the syntax tree according to whether or not it has, and the syntax of each syntax tree from the evaluation point.
In a natural language processing system that performs a semantic evaluation, the semantic check processing is a syntax search tree with a full evaluation score, and the processing is repeated until the end of the sentence with a dependency search processing target.
Store the deducted syntax tree data in the storage area,
When the syntax tree is completed up to the "sentence" by the processing up to the end of the sentence, the processing A that terminates the processing as the completed syntax tree
And when the "sentence" is not completed in the process A, the process is repeated until the end of the sentence with only the syntax tree having the evaluation score of a certain score or more among the scored tree data that has been deducted,
When the "sentence" is completed with the syntax tree having the evaluation point, the processing is terminated as the completed syntax tree, and when the completed syntax tree is not obtained, the syntax tree with the next lower score or more A natural language processing system comprising: a repetitive process B for performing a process.