JPH02253370A - Morpheme analyzing device - Google Patents

Abstract

PURPOSE:To prevent the failure of analysis of the morpheme due to the mistake of the break of a word by registering a position where a word is not broken as a pattern and using this pattern to perform the division into the word units. CONSTITUTION:When a language analysis system is started, a morpheme analyzing part 2 containing a word dividing function gives an access to a file 3 for patterns (non-word terminal patterns) where the words are not broken and registers these patterns into a non-wrd terminal pattern buffer 4 and performs the division into the word units based on the registered patterns. Thus it is possible to dynamically cope with the non-word terminal patterns which are different by the sentences. As a result, a natural language can be easily and correctly divided into the words serving as the units for analysis of the language in accordance with the types of sentences, etc., when the natural language is analyzed. Then the sentence analyzing efficiency is extremely improved.

Description

[Detailed description of the invention] Separation of skills The present invention relates to a morphological analysis device for English, etc.

Japanese Patent Application Laid-Open No. 63-89975 discloses that in the extraction process of dictionary lookup units, the dictionary lookup delimiter can be used for alphabetical characters, numbers,
Word breaks were limited to characters other than apostrophes and periods, which were preset by the system creator and recognized as word breaks.

In the above example, 1 For example, when the left parenthesis " In sentences such as manuals, () may be treated as part of a word, such as read() (the name of a function, which is a noun for the entire function). This cannot be done with the word break pattern set by the person, and it is expected that such patterns will differ depending on the field of interest, and it is impossible for the system creator to anticipate all of them and incorporate them into the system. It is.

In order to solve these shortcomings, in the conventional technology, for example, there is no other way than to use a pre-editing function to give information such as that this part should be treated as one word to each word. There's nothing I can do about it, but
It was very tedious to provide that information in all cases.

Currently, most so-called natural language analysis systems that analyze natural language sentences are systems that perform analysis in units of words. Therefore, failure to recognize word breaks means that the analysis will not be successful.In the conventional technology, only the word breaks initially set by the system creator are allowed. In other cases, for example, a certain number of symbols must be inserted so that they do not break between words, but this requires a great deal of effort. By the way, it can be said that the rules for word breaks are fixed for each sentence.

Even in the same English sentence, word breaks are usually different between letters and manuals.

The present invention was made in view of the above-mentioned circumstances, and
By allowing the user to freely set a series of symbols that indicate non-word breaks, depending on the type of text, analysis will fail due to the wrong word break in cases such as the above. This was done with the purpose of providing a morphological analysis device that eliminates the need to do so.

Configuration In order to achieve the above object, the present invention provides a morphological analysis device having at least a function of dividing an input sentence in a language into word units, which is capable of registering positions that do not form word breaks as patterns. , the pattern is used to perform division into word units.

FIG. 1, which will be explained below based on an embodiment of the present invention, shows the following:
1 is a configuration diagram for explaining an embodiment of an English-Japanese machine translation device including a morphological analysis device according to the present invention, in which 1 is an input device;
3 is a non-word ending pattern file, 4 is a non-word ending pattern buffer, 5 is a language analysis unit, 6 is a conversion/generation unit, 7 is a dictionary grammar, 8 is an output unit, 9 1 is a morphological analysis section, 10 is a language processing section, and 11 is a display section.

First, when the language analysis system is started, it accesses the file 3 of patterns that do not form word breaks (hereinafter referred to as non-single request patterns) and registers them in the non-word ending pattern buffer 4 in the language analysis system. The configuration is such that the name of the non-word ending pattern file can be given as a parameter when starting the system, so that a desired non-word ending pattern file can be used. For example, when using something like a translation editor, it is also possible to give it as a parameter when starting it. Also, as a function in the translation editor, a non-word ending pattern file can be provided. The purpose of preparing a non-word ending pattern buffer and registering it there is to improve speed. Of course, it is also possible to configure the system so that a file is always accessed during pattern checking without providing a buffer.

The non-word ending pattern file 3 is configured as a collection of records whose codes are non-word ending patterns. Therefore, there are 4 patterns so that as many patterns as the number of records can be registered, with one code being one pattern.

In English, non-word ending patterns are expressed as characters or sequences of consecutive characters that can represent alphabets, punctuation marks, spaces, line breaks, uppercase letters, lowercase letters, etc.

For symbols that can be seen as a series of characters, use the symbols as they are. Periods (,) For similar cases such as question marks (?) or invisible codes, combine them with \,
We will include one character that indicates the type of symbol.

¥S: Spaces, ¥C: Uppercase letters, ¥S: Lowercase letters, ¥t
:Tab, ¥n: Line feed, etc. However, when representing ¥, use ¥¥.

The code for a non-word ending pattern file has the following format:

Non-word ending pattern (line break) Incidentally, an example of a non-word ending pattern is shown in FIG.

FIG. 2 shows a flowchart of word division processing, and each step will be explained below.

! Technique: When the language analysis system is activated, it reads line by line from the non-word-end pattern file and registers the part up to the line break as a non-single request pattern. At this time, a null code shall be inserted at the end of the non-single request pattern. This is repeated until the end of the non-single billing pattern file.

The morphological analysis unit 2 has standard patterns that serve as word breaks for dividing one sentence into word units. This pattern can be considered as a general English pattern.The word-end pattern of the system of the present invention is such that when a character that is neither an alphabetic nor a numeric character comes, the immediately preceding character is regarded as a word break.

Therefore, the non-single billing pattern will act as a restraint on them.

Step 2 3: In the word division process, when an input is received from the input means 1 into the input buffer, the current character position in the input buffer is stored,
Extract each character (, Is it equal to the word ending pattern (
whether they belong to the same class).

A pattern in which the character matches the end-of-word pattern in A"Li"L is processed according to the flow of checking non-word-end patterns shown in FIG.

1 time: If it does not match all the non-single-single patterns, the check for non-single-single patterns was unsuccessful, and a code indicating the end of the word is sent to the language processing unit.

s ta 6 7 ; If the word ending pattern is not a blank space after that,
It sends that character to the language processor, and then sends a code indicating a single request again.

1↓Miridan; If the character in step 2 does not match the word ending pattern, that character is sent as is to the language analysis department,
Advance to next character.

Next, a flowchart for checking this non-single request pattern is shown in FIG.

ste 1-7; Is the character equal to the first character of the non-single claim pattern? (
or whether they belong to the same class).

ste 8 10; If they are equal, the next character of the non-single request pattern and the next character in the input buffer are taken out and compared in the same way.

Once checked, one non-single request pattern is checked until it reaches a null code, and step 11: If the check is successful, the characters in the input buffer up to that part are sent to the language analysis section.

Step 12; Thereafter, move the character pointer to the position sent to the analysis section. If it fails, a similar check is performed for all patterns, including the first non-single request pattern.

In this way, the pointer indicating the processed character is advanced by one, and the same process is repeated until the input is exhausted.

In the analysis system of the present invention, a code indicating the end of a word is set as a line break code. Therefore, the language processing unit 1
0, one line is considered to be one word, and processing such as dictionary search and syntax analysis is performed. This is to be decided as appropriate between the device of the present invention and the language analysis unit 5 that follows it, and is not limited to this.

For example, when a non-single billing pattern as shown in Figure 4 is registered.

The function read() uses
the read buf[] buffer.

Suppose that the following sentence is input.

The 0 part immediately after read, the - in readJuf, and the [] part immediately after it match the end-of-word pattern, but they also match a non-claim pattern, so they are not word breaks. It will be recognized. Therefore, in the language analysis section 5.

he function read () uses he readjuf[] b　u　　f　f　e　r.

and so on, and each is recognized as a - word. After this, dictionary searches, syntax analysis, etc. are performed.

[The function readO uses the read buf[] buffer. ”, the correct translation will be obtained.

Conventionally, for example, the word "read" is broken after the word "read", so read is searched in the dictionary as (the past tense of) the verb.

This resulted in incorrect analysis and translation such as "The function read 0...".

If implemented as a function in the translation editor, every time a non-word ending pattern file is specified, the previous non-word ending buffer will be cleared and a new registration will be made to the non-word ending buffer. This can also be easily achieved by doing this.

In the embodiment of the present invention, the non-word-end pattern file is composed of a text file, and changes to the file are assumed to be made using a general editor or the like. However, since the non-word ending pattern file is configured as a text file, it is easily possible to register and change the non-word ending pattern using a translation editor or the like.

In addition, the non-word-final pattern file is configured in a format that is easy to use on the language analysis system, and configured to be directly modified with a translation editor (or a non-word-final pattern editor that matches the configuration of the non-word-final pattern). It is also possible to do so.

In addition, although the embodiment of the present invention takes an English language analysis device as an example, by appropriately setting classes etc. that can be registered in non-word ending patterns, character order can be separated by spaces etc. in French and other languages. It can be easily applied to the analysis of various types of languages. Furthermore, it is very easy to apply this method to the processing of parentheses, etc. in Japanese language analysis.

Hiro - As is clear from the above explanation, according to the present invention, since it is possible to dynamically deal with non-word ending patterns that vary depending on the sentence, when performing natural language analysis, the words, which are the unit of linguistic analysis, are Correct division can be easily performed depending on the type, etc.

Therefore, the efficiency of text analysis is greatly improved.

Furthermore, since it is no longer necessary to edit sentences using pre-editing or the like in order to change the word pattern to the one required by the system, it is possible to reduce the amount of pre-processing required for natural language analysis and machine translation. Furthermore, since the sentence ending pattern created twice is effective for sentences of the same type, it is even more effective in machine translation of a large number of sentences of the same type, continuous translation, etc.

[Brief explanation of drawings]

FIG. 1 is a block diagram for explaining an embodiment of an English-Japanese machine translation device including a morphological analysis device according to the present invention, FIG. 2 is a flowchart of word segmentation processing, and FIG.
FIG. 4, which is a flowchart of checking a non-word ending pattern, is a diagram showing an example of a non-word ending pattern. 1... Input device, 2... Morphological analysis unit having word division function, 3... Non-word ending pattern file, 4...
・Non-word ending pattern buffer, 5...Language analysis unit, 6
...Conversion/generation unit, 7...Dictionary grammar, 8...Output unit, 9...Morphological analysis unit, 1o...Language processing unit, 1
1...Display section.

Claims (1)

[Claims] 1. At least, in a morphological analysis device that has the function of dividing an input sentence in the language into word units, positions that do not break between words can be registered as a pattern, and the pattern can be used to divide into word units. A morphological analysis device characterized by performing segmentation.