JPH0519185B2 - - Google Patents

Description

[Detailed Description of the Invention] [Object of the Invention] (Industrial Application Field) The present invention provides an unknown word processing device for morphological analysis of sentences that performs processing when unknown words are detected in morphological analysis of natural language sentences. Regarding.

(Prior Art) In text processing devices such as automatic translation machines, it is important to accurately morphologically analyze natural language sentences. Conventionally, this morphological analysis process has been performed by checking connections between parts of speech using a word dictionary, its part of speech classification, and a connection table for each part of speech based on the part of speech classification.

However, for example, if you look at the three example sentences with the verb "I came late": verb + auxiliary verb Verb "I came late": adjective Verb "I came late": adverb The word preceding the verb modifies the degree of action of the verb "come," and in this respect the three sentences are syntactically identical. Therefore, in morphological analysis, in the above example, "conjunctive modifier + predicate"
It is enough to know that it is. Despite this, conventional morphological analysis relies exclusively on form-centered analysis of two-dimensional connectivity in the form of connection tables by part of speech, which requires setting detailed part-of-speech classifications and registering words in dictionaries. The procedures are complicated;
Moreover, there was a problem in that there was no way to deal with unregistered words.

(Problems to be Solved by the Invention) As described above, in conventional morphological analysis processing, processing is performed using a connection table for each part of speech, so dictionary registration is complicated, and furthermore, it is impossible to deal with unregistered words. It was hot.

The present invention solves the above-mentioned conventional problems, makes it easy to register unknown words in a dictionary for morphological analysis, and allows unknown words to be assigned a classification name at the morphological level. The purpose is to provide processing equipment.

[Structure of the Invention] (Means for Solving the Problems) An unknown word processing device according to the present invention includes an analysis failure character string detection means for detecting a character string in which morphological analysis of an input sentence has failed, and affix and auxiliary analysis means for analyzing the failed character string from the reverse direction to obtain affixes and auxiliaries included in the failed character string; and removing the found affix and auxiliary from the failed character string. An unknown word detection means for detecting an unknown word, and an analysis result that predicts the word base name of the unknown word from the connection relationship information of the affixes and auxiliary words obtained, and outputs this predicted word base name as the form name of the unknown word. and an output means.

Here, a word base specifies a morpheme and is defined as the "minimum meaningful linguistic form" (Ken Morioka, "Formation of Vocabulary", Meiji Shoin, 1987). "Significant" corresponds to a part where a certain character string indicates a concept or idea. The word base has nothing to do with how the character string functions syntactically; for example, it is a unit that can be considered with difficulty, such as becoming a subject, conjugating, or modifying a verb. be.

On the other hand, word stems are recognized only in so-called conjugated words, have a clearly defined syntactic function, and are more general than word stems.

For example, ``taka-'' is considered to be the stem of an adjective, and in this classification, analyzing ``taka-i'' and ``taka-katsuta'' makes comparison even if the adjective is treated as an independent word and a conjugated word. It may be simple, but if you consider the structure of vocabulary such as ``to rise (verb)'', ``takata (adverb)'', and ``height (noun)'', there is no conjugation for ``takata'' and ``height'', and this Analysis of the lexical structure of the case shows that ``taka-'' was originally a conjugated word of an independent word, but the function of the stem part changed, and furthermore, by adding specific suffixes such as ``-sa'' and ``-maru,'' Since the vocabulary is considered to change, the analysis procedure becomes complicated in machine processing.

On the other hand, according to the word base, ``高-'' is classified as an adjective base, but this classification depends on what kind of concept or idea it expresses, and it is classified regardless of conjugation, part of speech, etc. Therefore, by connecting a suffix to a word base, we can define (1) things that make up an adjective, (2) things that make up a verb, (3) things that make up an adverb, and (4) things that make up a noun. I can do it. Therefore, the generality of regularization is higher than that of word stems. For example, the above-mentioned ``taka-ai (adjective),'' ``taka-maru (verb),'' and ``taka-dai (adverb)'' can be analyzed based on the word base, regardless of the presence or absence of conjugation, so machine processing The complexity of analysis procedures is reduced.

(Operation) The present invention abolishes the conventional part-of-speech classification of words, and uses a false base, which is an idea word that makes up a word, and an affix, which is a function word that promotes the independence of the base in a sentence.
Morphological analysis is performed from the standpoint of word structure based on adjuncts. If the parsing fails, the parsing failed string is parsed in the reverse direction. In other words, a single clause is usually formed by adding adjuncts such as affixes and auxiliary words to the central word base. Therefore, by analyzing the clauses for which morphological analysis has failed in the reverse direction, affixes and auxiliary words can be analyzed first. Since affixes and auxiliary words are adjunctive words, their types are limited compared to base words, and it is easy to comprehensively register them all. Moreover, prediction information for determining the classification name of an unknown word can be obtained from these affixes and auxiliaries. Therefore, even if the analysis fails, by analyzing the affixes and auxiliaries contained in the string, it is possible to predict the classification name of the unknown word base from the connection relationship of the affixes and auxiliaries, and use this predicted word base name as the unknown word base name. By outputting the result as a morphological analysis result of a word, the morphological analysis result can be obtained even if the unknown word is not registered in the dictionary.

Therefore, according to the present invention, dictionary registration is easy;
Furthermore, it is possible to perform elementary analysis even on unknown words.

(Example) Hereinafter, an example of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing the configuration of a morphological analysis device including an unknown word processing device according to an embodiment of the present invention.

This morphological analysis device includes a morphological analysis unit 1, an analysis failure character string detection unit 2, a character string conversion unit 3, an affix/auxiliary analysis unit 4, an analysis result comparison unit 5, a predicted word base registration unit 6, and an unknown word replacement unit 7. and an analysis result output section 8.

The morphological analysis unit 1 performs morphological analysis of an input sentence based on a word dictionary, a word base dictionary, and an affix/auxiliary dictionary (not shown). The word base dictionary is generated based on a word dictionary, and is a dictionary that stores word bases (generally word stems) as idea words that are the core of words. Types of words include predicates, conjunctions, joint words, independent words, etc., and are elements that function in syntactic analysis. A sentence is treated as a chain of words. Each word base has different part-of-speech characteristics such as kinship base, personal base, honorific base, verb base, adjective base, and body language base. The affix and auxiliary dictionary was also created based on the word dictionary, and works as a function word that combines with a word base to derive, inflect, or make a word independent. Affixes and auxiliaries are also classified based on their properties, such as case affixes and auxiliaries, honorific affixes and auxiliaries, verb-type conjugation affixes and auxiliaries, and adjective-type conjugation affixes and auxiliaries.

The analysis failure character string detection section 2 detects character strings, specifically phrases, for which analysis has failed in the morphological analysis section 1. The character string converter 3 converts the failed character string detected by the failed analysis character string detector 2 in reverse order. The affix/auxiliary analysis unit 4 analyzes the failed character string converted in reverse order from the beginning, and analyzes the adjunct part of the failed character string. The analysis result comparison section 5 compares the result of this affix/auxiliary analysis with the reversed character string from the character string conversion section 3 to detect unknown words.
The predicted word base registration unit 6 registers the base of the unknown word predicted from the analysis result of the affix/auxiliary analysis unit 4 together with the detected unknown word. Unknown word replacement part 7
replaces the unknown word in reverse order and outputs it to the analysis result output unit 8 together with the predicted word base. Analysis result output section 8
adds the predicted word base of the unknown word to the analysis result in the morphological analysis unit 1, and outputs this as the analysis result.

Next, the operation of the apparatus configured as described above will be explained based on the flowchart of FIG. 2.

First, an input character string is morphologically analyzed by the morphological analysis device 1 (S1). For example, if the sentence ``Taro-san and Hanako-san ate Indian food today, Hanako-san, who doesn't like spicy food, was in a lot of trouble.''
In the analysis, the body language base (taro), affix (san), and auxiliary word (ga) as shown in Figure 3 are extracted. Here, what is written to the right is the connection information for each, and the derivation in this connection information indicates that the morphemes are connected, and the value in parentheses is the nature of the connection. It shows whether it is connected to a suffix. In addition, inflection indicates that the morphological connection ends there, and that the morphological connection is connected to the morphologies indicated by the values in parentheses to form one character string unit. Therefore, the indicators in parentheses are used to check the connection of strings, and the derivations and inflections indicate the construction path of words. ~
is the analysis path. In other words, "Taro-san"
First, the honorific affix "san" is added from "Taro", as in -Taro-derivative (honorific title)-san -inflection (case)-ga-
, forming a new derived base ``Taro-san'', and then, as in normal nominal language, the nuclear auxiliary ``ga'' is added and inflected.

In this example, "Taro" is a registered word. However, if "Taro" is not registered in the dictionary, the morphological analysis unit 1 will write "Taro-san ga" as "Taro-san ga".
Morphological analysis of the phrase fails.

The analysis failure character string detection unit 2 detects such an analysis failure character string (S2). In this example, the phrase "Taro-san ga" is detected as a character string that fails to be analyzed, and is sent to the character string converter 3.

In the character string conversion unit 3, the analysis failure character string detection unit 2
When a failed character string is detected (S3), the above-mentioned failed character string "Taro-san ga" is converted into "Gansaro-ta" in reverse order (S4), and the affix/auxiliary analysis unit 4 and the analysis result comparison unit Output to 5.

Among the rules prepared in the morphological analysis unit 1, the affix and auxiliary analysis unit 4 includes analysis rules for only affixes and auxiliaries excluding the word base, including all character strings and connection relations. Rules are provided in reverse order for . This analysis rule is a small-scale analysis rule composed only of adjunct words, and can cover everything. The affix/auxiliary analysis unit 4 performs affix/auxiliary analysis by applying the above-mentioned failed analysis character string in reverse order to this rule (S5).
This analysis is performed on the path of ~, as in - ga - inflection (case) - nsa - derivation (honorific title) -, and as a result, the affix ``gansa (derivation [honorific title]) /X'', You can obtain analysis results for auxiliary strings.

Next, the analysis result comparison unit 5 converts the character string
An unknown word is detected by comparing the reverse order of the failed analysis character string output by the affix and auxiliary character string with the affix and auxiliary strings removed from the affix and auxiliary strings analyzed by the affix and auxiliary analysis unit 4 (S6). In the above example, "rota" is detected as a character string of an unknown word.

The predicted word base registration unit 6 predicts the word base of the unknown word X based on the affix and auxiliary analysis results (S7). For example, in the above example, "derivation (honorific title)" is obtained from the analysis result of the affix and auxiliary sequence, but the word base that can be derived and connected with the "honorific title" affix afterward is a body language base. Therefore, it can be predicted that ``rota'' is a body language base. This predicted word base is registered together with the unknown word.

The unknown word replacement unit 7 replaces the unknown word "Rota" with "Taro".
and the analysis result output unit 8 along with the predicted word base.
send to In the analysis result output section 8, the morphological analysis section 1
The analysis result for the unknown word is added to the analysis result for the above, and this is output as the morphological analysis result (S8).

Next, in the example mentioned above, if we consider ``spicy no ga'', this includes the adjective base (shin), the i-based inflectional auxiliary (ii), (katsuta), and the functional auxiliary (no) as shown in Figure 5. ), the case particle (ga) is extracted. here,
What is written to the right is connection information, similar to FIG. 3. In this case as well, morphological analysis is performed along the route .about. in FIG. 5.

In this example, if "spicy" is not registered in the dictionary, the phrase "spicy" will fail in morphological analysis. In this case, similar to the process described above, the affix/auxiliary analysis unit 4 obtains an analysis result of "ganoi (derived [i-kei-a])". As a backward connection indicator, the base that is derived [i-kei-a] is
Since it is an adjective base, it can be predicted that the classification name of the unknown character string "spicy" is an adjective base.

Similarly, if "Taro no ga" fails, the unknown word analysis result will be "Gano (derived [neutral])"
Therefore, it can be seen that "Taro", which was detected as an unknown word, is a body language base. The same is true for "Taro-san no ga"; as a result of affix and auxiliary analysis, "ganonsa (derived [neutral])" is obtained, and the classification of "Taro" detected as an unknown word string is Becomes the basis of body language.

Such predictions are possible because basic morphological analysis is based on word structure. In other words, the word structure of this device has the advantage of providing predictive information for determining unknown words compared to part-of-speech connection tests, and this predictive information can be used to accurately predict the classification name of unknown words. can be done.

In this way, according to this device, "Taro" and "Spicy"
etc. is an unknown word, the classification name of the unknown word is predicted from the information of the affixes and auxiliary words added to it,
This can be output as an analysis result.

Note that the present invention is not limited to the embodiments described above. For example, by storing the above-described classification name analysis results for unknown words, the stored information may be used to predict the classification name when the same unknown word is input.

Further, the rules for the affix and auxiliary analysis unit 4 can be obtained from the rules provided in the morphological analysis unit 1, for example, as shown in FIG. That is, the rules stored in the morphological analysis rule storage unit 9 are given to the conversion device 10, and the conversion device 10 extracts rules for attached words (S11) and converts surface character strings (S12), as shown in FIG. ), conversion of conjunctive relations (S13), all character strings in the closed vocabulary are reversed, and the conjunctive relations are reversed both forward and backward, and the resulting affixes are used for unknown word analysis. - Obtain as an affix rule and store it in the affix/auxiliary rule storage unit 11.
By such processing, affix and auxiliary rules can be easily obtained.

[Effects of the Invention] As described above, according to the present invention, unlike conventional classification by parts of speech, analysis is performed according to the function of morphemes, and character strings that fail to be analyzed are
Since the classification name of an unknown word is predicted by analyzing the attached word part from the reverse direction, there is no need for a strict dictionary, dictionary registration is easy, and unknown words can also be analyzed. Elementary analysis can be performed. Particularly, this method has the effect of improving analysis efficiency for words that have conventionally been considered to have multiple parts of speech.

[Brief explanation of the drawing]

Fig. 1 is a block diagram showing the configuration of a morphological analysis device according to an embodiment of the present invention, Fig. 2 is a flowchart showing a morphological analysis procedure in the same device, and Fig. 3 shows an example of morphological analysis rules in the same device. Figure, 4th
The figure shows an example of the affix and auxiliary analysis rules in the same device, FIG. 5 shows another example of the morphological analysis rules in the same device, and FIG. 6 shows the conversion means for the affix and auxiliary analysis rules in the same device. Block diagram shown,
FIG. 7 is a flowchart showing the processing procedure of the conversion means. 1... Morphological analysis unit, 2... Analysis failure character string detection unit, 3... Character string conversion unit, 4... Affix auxiliary analysis unit, 5... Analysis result comparison unit, 6... Predicted word base registration unit, 7... Unknown word replacement section, 8... Analysis result output section.

Claims (1)

[Claims] 1. An analysis failure character string detection means for detecting a character string for which morphological analysis of an input sentence has failed, and affixes and auxiliary words included in the analysis failure character string by analyzing the detected analysis failure character string from the reverse direction. affix and auxiliary analysis means for determining the affix and auxiliary, unknown word detection means for detecting unknown words by removing the affix and auxiliary from the character string that has failed to be analyzed; 1. An unknown word processing device for morphological analysis, comprising: an analysis result output means for predicting a word base name of an unknown word and outputting the predicted word base name as a morphological name of the unknown word.