JPH0269869A - Language analyzing device - Google Patents

Abstract

PURPOSE:To eliminate the failure of analysis due to the mistake of the punctuation of a sentence by providing the inputting means of language to be analyzed, a means to register a sentence end pattern and a sentence dividing means to divide the inputted objective language to be analyzed into sentences by using the registered sentence end pattern. CONSTITUTION:An input file 1, a manual inputting part 2 such as a keyboard, an inputting device 3, a sentence end pattern file 4, a sentence end pattern registering part 5, a sentence end pattern buffer 6, a sentence end recognizing part 7, a sentence end deciding part 8, a language analyzing part 9 and a like of character collating table 10 are provided with a control part 11 and the output terminal 12 of an analyzed result. Then, the inputted objective language to be analyzed is divided into the sentences by using the registered sentence end pattern. Accordingly, the pattern of the end of the sentence different according to the sentence can be dealt with dynamically. Thus, in the case of natural language analysis, it can be easily and correctly divided into the sentence which is the unit of the language analysis according to the format or the like of the sentence.

Description

DETAILED DESCRIPTION OF THE INVENTION The present invention relates to a language analysis device, more specifically, to an analysis section of a natural language analysis device 1 machine translation device, and is applicable to, for example, language analysis in a question answering system, etc. It is.

Previous language analysis devices only recognize sentence ending patterns that are set in advance by the system manufacturer as sentence endings. For example, the pattern established by the system creator, such as making sure that after a semicolon is at the end of a sentence,
It always becomes a sentence break, and other times it does not become a sentence break. However, it is expected that there will be differences in the usage of punctuation depending on the text. For example, some people treat semicolons as sentence breaks, while others use them to explain appositions. In this case, the end-of-sentence pattern set in advance by the system manufacturer is at best compatible with either document.

Another drawback is that it cannot handle sentence ending patterns that the system creator has not anticipated.

In order to solve these shortcomings, the prior art.

For example, if you want to use the cross-editing function to edit the text itself so that the system recognizes it as the end of a sentence, or if you do not want to cut the sentence, you need to transform it into a shape that the system does not recognize as a break in the sentence. It was very troublesome. For this purpose, the user needs to know the standard pattern at the end of a sentence. Furthermore, since it is necessary to change punctuation marks, etc. in the sentence to be analyzed, there is also the disadvantage that the image of the original sentence is spoiled.

Furthermore, at present, most of the so-called natural language analysis systems that analyze natural language sentences are systems that analyze sentences in units of sentences. Therefore, if the recognition of sentence breaks fails, the analysis will not go well. As described above, in the prior art, only sentence breaks initially set by the system creator are allowed. In other cases, for example, it is necessary to insert a series of regular symbols that indicate breaks in sentences, but this requires a great deal of effort. By the way, it can be said that the rules for sentence breaks are fixed for each sentence. Therefore, even in the same English sentence, the text that should be used as a sentence break is different between a letter and a manual.

- Kazuma The present invention was made in view of the above-mentioned circumstances,
In particular, by allowing the user to freely set a series of symbols that indicate breaks in a sentence according to the type of sentence, it is possible to eliminate failures in analysis due to mistakes in the breaks in a sentence. It was done for a purpose.

(ii) In order to achieve the above object, the present invention provides, in a sentence segmentation device, an input means for a language to be analyzed, a means for registering a sentence ending pattern, and an input language to be analyzed using the registered sentence ending pattern. or a language analysis means for analyzing an input sentence having a sentence-end code, and the sentence division means performs the analysis after dividing the input language into sentences. The language analysis device is characterized by the following. Hereinafter, the present invention will be explained based on examples.

FIG. 1 is an overall configuration diagram for explaining one embodiment of the present invention, and the figure shows an example in which this analysis device is applied to an English language analysis device. In FIG. 1, 1 is an input file; 2 is a manual input section such as a keyboard.

3 is an input device, 4 is a sentence-end pattern file, 5 is a sentence-end pattern registration unit, 6 is a sentence-end pattern buffer, 7 is a sentence-end recognition unit, 8 is a sentence-end determination unit, 9 is a language analysis unit, 10 is a class matching table, 11 is a control section, 12 is an output end of the analysis result, and the area within the dotted line is a sentence division section.

2 to 4 are flowcharts for explaining the operation of FIG. 1, and the following description will be made according to this flowchart.

When the language analysis device is activated (FIG. 2, 5TART), the sentence-end pattern file 4 is accessed and registered in the sentence-end pattern buffer 6 in the language analysis system.

The structure is such that the name of the sentence-final pattern file can be given as a parameter when starting the language analysis system, so that a desired sentence-final pattern file can be used. For example, when using a translation editor, this can be given as a parameter when starting up, or a sentence-end pattern file can be given as a function in the translation editor.

The end-of-sentence pattern file 4 is configured as a set of records each pairing an end-of-sentence pattern and a corresponding end-of-sentence position designation. Therefore, in a language analysis system,
This allows one record of the sentence-end pattern file to be used as a - pattern, and as many patterns as there are records to be registered.

In the case of English, the sentence ending pattern is expressed as a character or a sequence of consecutive characters that can represent alphabets, punctuation marks, spaces, line breaks, uppercase letters, lowercase letters, etc. For example, make promises as shown in Table 1. The end-of-sentence position designation is for specifying after which character in the end-of-sentence pattern the end of a sentence is to be set. When setting the end of a sentence immediately before the end of sentence pattern, the value is O.

It is assumed that there is a tab code between the first table sentence end pattern and the first sentence end position specification. Therefore, the loop code of the sentence-end pattern file is written in the format of ``sentence-end pattern (TAB) sentence-end position designation (line feed)'', as shown in FIG. 5, for example. In the figure, mu indicates a tab code.

When the language analysis system is started (Fig. 2 5TART)
.

The sentence ending pattern file specified by the parameter is opened (FIG. 2, step 2). Next, read each - line from the sentence ending pattern file (Figure 2, step 2),
If it is not the end of the file (Fig. 2, step 2), register the part up to the tab as the end-of-text pattern.
p4), and the corresponding sentence end position designation is registered as a numerical value in the sentence end pattern buffer 6 (FIG. 2, step 5).

At this time, at the end of the sentence-end pattern, there is a null
The code shall be entered. Repeat this until the end of the sentence-end pattern file. FIG. 6 is an example of a sentence-end pattern buffer created in this manner. In the figure, r(
) J means a null code. Conventionally, these patterns have been fixedly incorporated into a system and processed, or fixedly processed using a program.

When the end-of-sentence determining unit 8 receives input from the input means 3 into the input buffer, it stores the current character position in the input buffer (Fig. 3, 5tepl), and determines whether the character is equal to the first character of the end-of-sentence pattern or not. (or whether they belong to the same class) (Figure 3, tap 2). The details of step 2 in Figure 3 are shown in Figure 4. If the characters in the sentence ending pattern are (Null), the matching is considered successful, and if it is not Null (Figure 4, 5te
pl), check whether the currently set character in the sentence ending pattern is ¥ (Figure 4, step 2). If it is ¥, take the next character (Fig. 4, step 5), and type matching table 10.
(FIG. 4, 5 step 6) The o-class matching table describes contents belonging to symbols accompanying ¥, as shown in FIG. 7. If it is a character in the class, add two characters for the sentence ending pattern and the input character (
Advance the buffer) by one character and return to 5tepl. If the first character does not match any sentence ending pattern, that character remains as is.

It is sent to the language analysis department and proceeds to the next character. -The letter is
For patterns that do not match, their positions on the input buffer are stored, and the second and subsequent characters are similarly checked, and when the pattern reaches a null code, the end of one sentence is recognized. If all the patterns fail, send the character as is to the language analysis section and proceed to the next character (Fig. 3, 5).
step 3).

The end-of-sentence recognition section 7 extracts characters corresponding to the end-of-sentence position indication area from the stored characters on the input buffer to the language analysis section 9.
send to Then, the end-of-sentence code, which is determined in advance by the language analysis section, is sent. (In this system, it is a series of three characters \ql.) The next character in the input buffer is unconditionally sent to the language analysis section, and the process of the end-of-sentence determination section 8 is performed again from the next character. Continue 9 Even when an input is received, the end-of-sentence code determined by the language analysis section is sent and the processing of the end-of-sentence determination section and end-of-sentence recognition section is completed.

The language analysis unit 9 repeatedly analyzes sentences, with the beginning of the input and immediately after the immediately preceding sentence end code up to the first sentence end code as one sentence. For example, JP-A-82-2678
The language analysis device shown in Publication No. 72-3 can be used. Furthermore, it is also very easy to configure the sentence segmentation device as a morphological analysis section in the language analysis device. The sentence-end code is configured so that it only serves to complete the sentence and is excluded from the analysis target. This can be achieved very easily with an analysis system that analyzes sentences in units of sentences. As for the analysis method, if it is a five-sentence analysis method, it will probably use a one-context-free grammar, but it may also use a case grammar or an extended transition network.

This system t1 is configured as a simple sentence analysis system, but the language analysis unit can be of any type as long as it is configured to recognize the sentence-final code as an abstract sentence-final position. It is.

For example, when the sentence ending pattern shown in Figure 5 is S&B, I have a book; But I don't
't have a book; specially
a good book.

Suppose that the following sentence is input.

The ; after the first book is recognized as the end of a sentence because the next character is an uppercase letter, and the next book; is recognized as not the end of a sentence because the next character is a lowercase character.

Therefore, the syntax parsing unit does the following: I have a book;\q] But I d
on't have a book; special
ly a good book, ￥q].

I have a book; BuL
I don't have a book;espe
cially a good book.

By analyzing these two sentences, it is possible to obtain correct results.

In particular, it is very effective in analyzing colons, semicolons, etc., which have different usage depending on the author and the type of document.

If this is implemented as a function in the translation editor, it can be easily achieved by clearing the previous sentence-end buffer and registering a new one in the sentence-end buffer every time a sentence-end pattern file is specified. realizable.

In this embodiment, the end-of-sentence pattern file is composed of a text file, and changes to the file are assumed to be made using a general editor or the like. However, in translation editors, etc., it is possible to register or change the sentence ending pattern.
Since the sentence ending pattern file is configured as a text file, it can be easily implemented.

In addition, you can configure the sentence ending pattern file in a format that is easy to use on the language analysis system, and use a translation editor (or a sentence ending pattern editor that matches the structure of the sentence ending pattern, etc.).
It is also possible to configure it so that it can be changed directly.

This example takes an English language analysis device as an example, but
By appropriately setting the classes etc. that can be registered in the sentence ending pattern, it can be easily applied to Japanese analysis and other language analysis devices.

As is clear from the above explanation, according to the present invention, since it is possible to dynamically deal with sentence ending patterns that vary depending on one sentence, during natural language analysis, the sentence format etc. Correct division can be easily achieved according to the following.

Therefore, the efficiency of sentence analysis is greatly improved, and there is no need to edit sentences by using pre-editing to match the sentence ending pattern required by the system, reducing the amount of pre-processing required for natural language analysis and machine translation. Can be done. In addition, since the sentence ending pattern created twice is effective for sentences of the same type, it is useful for machine translation of a large number of sentences of the same type, continuous translation, etc.
It is even more effective.

[Brief explanation of the drawing]

Figure 1 is an overall configuration diagram for explaining one embodiment of the present invention, Figures 2 to 4 are flowcharts for explaining the operation of the embodiment of Figure 1, and Figure 5 is the end of the first sentence. FIG. 6 is a diagram for explaining an example of a pattern file, FIG. 6 is a diagram for explaining an example of a sentence-end pattern buffer, and FIG. 7 is a diagram for explaining an example of a class matching table. 3... Input device, 4... Sentence end pattern file, 6
...Sentence end pattern buffer, 7...Sentence end recognition section, 8
... Sentence end determination section, 9... Language analysis section. ] Fig. Fig. Fig. Section

Claims (1)

[Claims] 1. It has a means for inputting a language to be analyzed, a means for registering a sentence ending pattern, and a sentence dividing means for dividing the input language to be analyzed into sentences using the registered sentence ending pattern. A language analysis device featuring: 2. An input means for the language to be analyzed, a means for registering a sentence ending pattern, a sentence dividing means for dividing the input language to be analyzed into sentences using the registered sentence ending pattern, and an input sentence having a sentence ending code. 1. A language analysis device comprising: language analysis means for dividing an input language into sentences, and performing analysis after dividing an input language into sentences in said sentence division means.