JPH09223141A - Device and method for analyzing japanese sentence - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide Japanese sentence analyzing device and method capable of accurately detecting only a character string forming an unregistered word out of a character string including the unregistered word and registering the word consisting of the detected character string as an unregistered word. SOLUTION: A morphem analyzing part 3 divides an inputted character string into words by using a dictionary file 4 for registering data indicating the character string of a word and the attribute of the word. When a word of which character string length consists of one character, a word of a prescribed part of speech or a word having less possibility of forming a composite word is included in the divided words, a word candidate detection part 5 detects a character string connecting the word to a preceding or succeeding adjacent word as an unregistered word (word candidate) and temporarily records the word in the dictionary file 4. Then a word candidate verifying part 7 verifies the validity of the temporarily registered word candidate. The word candidate of which validity is verified is formally registered in a word candidate formal registration part 10.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects a word not included in a dictionary included in an input Japanese sentence, particularly a word written in katakana, and registers the detected word in a Japanese sentence analyzer. Regarding Japanese sentence analysis method.

[0002]

2. Description of the Related Art A process called morphological analysis is performed as a preprocess for machine translation. The morphological analysis is simply a process of performing segmentation, word segmentation, etc. on an input Japanese sentence using a dictionary. Here, the problem is that if there is a word (hereinafter referred to as an unregistered word) that is not included in the dictionary in the input Japanese sentence, the morphological analysis cannot be performed accurately. Therefore, machine translation also fails. Therefore, it is necessary to accurately detect the unregistered word in order to give a warning in advance that the translation is likely to fail in the preprocessing of the machine translation.

Unregistered words are often new words used as keywords in documents. Therefore, in a technique such as automatic keyword creation (index creation) for document search, it is necessary to register an unregistered word as a keyword. That is, in order to register an unregistered word that is not in the dictionary as a keyword in the automatic keyword creation for document retrieval, the unregistered word must be detected accurately.

Conventionally, in most unregistered word detection processes, unregistered words are katakana character strings. Therefore, the entire character string in which the same character type (katakana) is concatenated to the character string for which dictionary lookup has failed is performed. It was to detect it as an unregistered word. For example, if "In" and "Tara" are registered words registered in the dictionary and "Puta" is an unregistered word for the character string "Interrupter", "Interrupter" is regarded as an unregistered word. Method of detection (Proceedings of the 36th National Convention of the Information Processing Society of Japan (above 1988) 12
Pages 31-1232 "Pre-editing system for Japanese-English machine translation
(2) -Method of detecting ambiguity of morpheme- ") and the information such as" New "," News "and" Hotel "that are looked up in the dictionary for the character string" New Station Hotel "are ignored, Method to detect "New Station Hotel", which is the entire katakana character string, as an unknown word (unregistered word referred to in this application) (Information Processing Society of Japan 47th (late 1993))
The National Convention Proceedings, pp. 3-159 to 3-160, "Unknown Word Estimation Mechanism in Japanese Morphological Analysis Introducing Selective Dictionary Lookup Mechanism").

[0005]

However, in the method of extracting unregistered words described above, if the character string of katakana is a compound word composed of a plurality of words, this compound word is detected as an unregistered word. There is a problem. For example, if there is a word in the character string "Financial System" (a word in which "Financial" is an unregistered word and is connected to "System" to form a compound word), the registered words "Phi" and " Ignoring the information such as "system" that is drawn in the dictionary ("phi" and "system" are words registered in the dictionary file), and the entire "financial system" is detected as an unregistered word. Further, a compound word including an unregistered word "Financial", "Financial Banking", "Financial Center", "Financial Advisor", "Financial Planner", etc. are also detected as another unregistered word ("Banking", "Center,""advisor,""planner," etc. are words registered in the dictionary.) As a result, the amount of processing in the preprocessing of machine translation is increased, and redundant keywords (compound words) are created as keywords for document retrieval.

Here, a method of detecting only a portion that does not match the registered word as an unregistered word is conceivable.
In this method, if a part of the unregistered word to be detected matches the registered word, an unregistered word of an inappropriate character string in which the matched part is cut off is detected. For the character string "Financial system" in the above example, a character string that is not valid as the word "National" excluding the part that matches the registered words "Phi" and "System" is regarded as an unregistered word. I will detect it.
Further, when the character string of the unregistered word happens to match the character string in which a plurality of words are concatenated, there is a problem that the unregistered word is not detected. For example, "Kalimantan"
"Kari" and "Man" for the unregistered word of the character string
If there are three registered words "tan", it means that the unregistered word is not detected.

An object of the present invention is to detect only a character string forming an unregistered word accurately from a katakana character string containing an unregistered word and register the detected unregistered word. And to provide a Japanese sentence analysis method.

The present invention also provides a Japanese sentence analysis apparatus and a Japanese sentence analysis method capable of detecting the validity of a detected unregistered word and canceling the unregistered word that is erroneously detected and registered. The purpose is to

[0009]

A Japanese sentence analysis apparatus according to claim 1 of the present invention inputs a character string of a word and a dictionary file in which data indicating an attribute of the word is registered and the dictionary file. A morpheme analysis means for performing morpheme analysis of dividing the character string into words, and when the concatenated katakana character string in the input character string is divided into a plurality of words by the morpheme analysis, If a word having a character string length of 1 character is included in the generated word, an unregistered word detection that detects a character string that connects the word and an adjacent word before or after the word as an unregistered word Means and an unregistered word registration means for registering the detected unregistered word.

In this configuration, the morpheme analysis means performs the morpheme analysis by dividing the character string input using the dictionary file into words. As a result of morphological analysis, when the concatenated katakana character string is divided into a plurality of words, if the divided words include a word with a character string length of 1 character, the unregistered word detection means A character string that connects a one-character word and a word adjacent to the word before or after the word is detected as an unregistered word. Then, the unregistered word registration means registers the detected unregistered word.

According to a second aspect of the present invention, a Japanese sentence analysis apparatus uses a dictionary file in which character strings of words and data indicating attributes of the words are registered, and a character string input using the dictionary file. A morpheme analysis means for performing morpheme analysis of dividing into concatenated katakana character strings in the input character string into a plurality of words by the morpheme analysis. If the word of the part of speech is included, an unregistered word detecting unit that detects a character string that connects the word and an adjacent word before or after the word as an unregistered word, and the detected unregistered word And an unregistered word registration unit for registering a word.

In this configuration, the morpheme analysis means performs the morpheme analysis of dividing the character string input using the dictionary file into words. As a result of morphological analysis, when concatenated katakana character strings are divided into a plurality of words, if a word with a predetermined part of speech is included in the divided words, the word and the word before or after the word A character string connecting adjacent words is detected as an unregistered word. Then, the unregistered word registration means registers the detected unregistered word.

According to a third aspect of the present invention, a Japanese sentence analysis apparatus uses a dictionary file in which character strings of words and data indicating attributes of the words are registered and character strings input using the dictionary file. Morphological analysis means for performing morphological analysis to divide into, and the data indicating the attribute includes data indicating whether or not the corresponding word is less likely to form a compound word, the input When concatenated katakana character strings in a character string are divided into a plurality of words by the morphological analysis, if the divided words include a word that is unlikely to form a compound word, the word and the word An unregistered word detecting means for detecting a character string connecting adjacent words before or after the word as an unregistered word, and an unregistered word registering means for registering the detected unregistered word. And it said that there were pictures.

In this configuration, the morpheme analysis means performs morpheme analysis by dividing the character string input using the dictionary file into words. As a result of morphological analysis, when the concatenated katakana character string is divided into a plurality of words, if the divided words include a word that is unlikely to form a compound word, an unregistered word detection unit. Detects a character string connecting the word and a word adjacent to the word before or after the word as an unregistered word. Then, the unregistered word registration means registers the detected unregistered word.

In the Japanese sentence analyzing apparatus according to claim 4 of the present invention, the unregistered word detecting means includes a word having a predetermined part of speech included in a word divided into a plurality of words by the morphological analysis. For example, it is characterized by including means for detecting a character string connecting the word and a word adjacent to the word before or after the word as an unregistered word.

This configuration is a combination of claim 1 and claim 2.

In the Japanese sentence analysis apparatus according to claim 5 of the present invention, the data indicating the attribute includes data indicating whether or not the corresponding word is less likely to form a compound word. The unregistered word detection means, if a word that is unlikely to form a compound word is included in the words divided into a plurality of words by the morphological analysis, the word and the word adjacent to the word before or after the word. It is characterized in that it includes means for detecting a character string connecting the and as an unregistered word.

This structure is a combination of any one of claims 1, 2, and 3.

In the Japanese sentence analysis device according to claim 6 of the present invention, the word of the character string that matches the unregistered word detected by the unregistered word detecting means is divided into a plurality of words by the morphological analysis. When it is included in a word, it is characterized by including a legitimacy verifying unit that verifies whether or not the unregistered word is valid as a word by comparing with a predetermined rule.

In this configuration, the correctness verifying means detects the word of the character string which matches the registered unregistered word detected by the unregistered word detecting means in the word divided into the plurality of words by the morphological analysis. Is included, it is verified whether the unregistered word is valid as a word.

The Japanese sentence analysis method according to the present invention described in claims 7 to 12 has a configuration in which the Japanese sentence analysis device according to any one of claims 1 to 6 is considered as a method.

[0022]

1 is a block diagram showing the functions of a Japanese sentence analysis apparatus according to an embodiment of the present invention.
The Japanese sentence analysis device 1 includes a text data storage unit 2, a morpheme analysis unit 3, a dictionary file 4, and a word candidate detection unit 5.
1, a word candidate registration unit 6, a word candidate verification unit 7, a word candidate deletion unit 8, a registered word verification unit 9, and a word candidate formal registration unit 10. The text data storage unit 2 stores text data to be processed. The morpheme analysis unit 3 performs morpheme analysis on the text data stored in the text data storage unit 2. The dictionary file 4 stores the character strings of words and the attributes (parts of speech, etc.) of the words in association with each other. The word candidate detection unit 5 detects an unregistered word that is not registered in the dictionary file 4 as a word candidate based on the result of the morpheme analysis performed by the morpheme analysis unit 3. The word candidate registration unit 6 provisionally registers the word candidates detected by the word candidate detection unit 5 in the dictionary file 4. The word candidate verification unit 7 verifies the validity of the word candidates temporarily registered in the dictionary file 4. Word candidate deletion unit 8
Deletes a word whose validity is not verified among the word candidates temporarily registered in the dictionary file 4. The registered word verification unit 9 verifies the validity of the word registered in the dictionary file 4 detected from the character string that may include the unregistered word. The word candidate formal registration unit 10 officially registers this word candidate when the validity of the temporarily registered word candidate is verified.

FIG. 2 is a flow chart showing the processing of the Japanese group analysis device according to the embodiment of the present invention. Here, a series of processes of the Japanese sentence analysis device 1 will be briefly described. The Japanese sentence analysis device 1 fetches and stores the text data to be processed in the text data storage unit 2 (n1). The morphological analysis unit 3 takes in text data for each character string unit delimited by punctuation marks (n2). When the morphological analysis unit 3 takes in the text data delimited by punctuation, it performs morphological analysis using the dictionary file 4 (n
3). By this morphological analysis, the text data delimited by the punctuation marks captured in n2 is divided into morphemes. The Japanese sentence analysis device 1 determines whether or not the result of the morphological analysis by the morphological analysis unit 3 in n3 includes a word of a katakana character string (n4). When the word of the katakana character string is not included, it is determined whether all the text data stored in the text data storage unit 2 in n1 has been processed (whether unprocessed text data remains) (n11 ), If unprocessed text data remains, the process returns to n2.

When the word of the katakana character string is included, the word candidate detecting unit 5 determines whether or not the word of the katakana character string includes an unregistered word (n.
5). If the unregistered word is included, the word candidate detection unit 5 detects the word candidate from the entire katakana character string including the word of the katakana character string concatenated to the unregistered word, and the detected word candidate is a dictionary file. The first word candidate to be temporarily registered in No. 4 is detected and registered (n6). When it is determined that the unregistered word is not included in n5, or the above n
When the process of 6 is completed, it is determined whether or not there is a place where registered words of a plurality of katakana character strings registered in the dictionary file 4 are connected (n7). If there is a portion where the registered words of the katakana character string are connected, a word candidate is detected from the entire katakana character string that connects the registered words, and the detected word candidate is temporarily registered in the dictionary file 4 Second word candidate detection , Registration processing is performed (n8).

Further, the word candidate verification unit 7 determines whether, among the words divided by the morphological analysis, there is a word having the same katakana character string as the word candidate temporarily registered in the dictionary file 4 at n6 or n8. Whether or not (whether the word candidate temporarily registered in the dictionary file 4 at n6 or n8 has reappeared from another character string) is determined (n9). When the word candidate reappears, the word candidate verification unit 7 executes a word candidate correctness verification process for verifying the correctness of the reappearing word candidate temporarily registered in the dictionary file 4 (n.
10).

Then, it is determined in n11 whether or not unprocessed text data remains, and if unprocessed text data remains, the above-described processes of n2 to n10 are repeated. If there is no unprocessed text data, all unnecessary word candidates temporarily registered in the dictionary file 4 (word candidates whose validity has not been verified) are deleted and the process is completed (n12).

The above processing will be described in detail below. n
1, a series of text data (file unit, record unit, directory unit, hard disk unit, time unit, 10
The data amount unit such as 0 MB is taken in and stored.

At n2 and n3, the morpheme analysis unit 3 extracts from the series of text data stored in the text data storage unit 2 in units of character strings sequentially delimited by punctuation and performs morpheme analysis. Then, the morpheme analysis unit 3 divides the text data in character string units delimited by the punctuation marks on which the morpheme analysis has been performed into words and outputs the words. For example, text data for morphological analysis has a "financial system".
It is assumed that the katakana character string is included. In addition, as shown in FIG.
It is assumed that the character string “system” is registered as a word, and the words of the character strings “nuncial”, “financial”, and “financial system” are not registered. Here, when the morpheme analysis unit 3 performs the morpheme analysis on the katakana character string called "financial system", as shown in FIG.
"N", "Sha", and "Le" failed to look up the dictionary, and "Phi"
And "system" are detected as words whose part of speech is a noun. Then, the morphological analysis unit 3 considers the katakana character string “Nanchal”, which is a combination of “Na”, “n”, “sha”, and “ru” that have failed to look up the dictionary, as one unregistered word,
"Financial system" as shown in FIG.
"Phi" as a result of morphological analysis of the character string
Outputs the three words "nuclear" and "system". At this time, the part of speech of "NANCIAL" is output as an unregistered word.

At n4, it is determined whether or not the morpheme analysis result output by the morpheme analysis unit 3 includes a word of katakana character string. Here, if the morpheme analysis result does not include the word of the katakana character string, the processes of n5 to n10 are not performed, and the presence or absence of unprocessed text data is determined in n11. On the other hand, if the morpheme analysis result includes words in the katakana character string, the processes of n5 to n10 are performed.

At n5, the word candidate detecting unit 5 determines whether or not the unregistered word is included in the morpheme analysis result. In this embodiment, when the morphological analysis result includes a word whose part of speech is an unregistered word, it is determined that the unregistered word is included. In the above-described example, the word “NANCIAL” whose part-of-speech is an unregistered word is included, so it is determined that the unregistered word is included in n5.

When the word candidate detection unit 5 determines that the word n5 contains an unregistered word, it detects the word candidate from the entire katakana character string connected to the unregistered word, and the detected word candidate is stored in the dictionary file. The first word candidate detection and registration process to be temporarily registered in No. 4 is executed. Here, the entire katakana character string that is connected to an unregistered word is a katakana character that includes a word of a katakana character string that is continuous before or / and after an unregistered word (a word that is connected to an unregistered word). It is a character string. In the above example, the “financial system” is the entire katakana character string that is connected to unregistered words. It is assumed that the words in the katakana character string are not concatenated before “Phi” and after “System”.

Here, the first word candidate detection / registration process will be described in detail with reference to FIG. FIG. 4 is a flow chart showing the flow of the first word candidate detection and registration processing. The entire katakana character string connected to the unregistered word of this katakana character string is set as one word, and the part of speech is registered as a noun in the dictionary file 4 (n21). In the example described above, “Financial system” is registered in the dictionary file 4 as a word (part of speech is a noun). Next, it is determined whether the unregistered word included in the katakana character string of the word registered in the dictionary file 4 in n21 is a one-character word (n22). If the word is not a one-character word, this unregistered word is detected as a constituent element of the word candidate (n2).
3). If the unregistered word is one character, and the registered word of the katakana character string is concatenated in front of this unregistered word, the word consisting of the character string connecting this registered word and the unregistered word is a constituent element of the word candidate. If a registered word of a katakana character string is concatenated before or after this unregistered word, a word consisting of a character string connecting this registered word and the unregistered word is used as a component of the word candidate. Detect (n2
4). For example, for the string "Iriomote",
Morphological analysis result "i" is unregistered word, "Rio""Mote"
Is a registered word, "i" is one character not registered yet, so that "irio" connected to the registered word "rio" behind is detected as a component of the word candidate. It is assumed that the word detected before the unregistered word “i” in the morphological analysis unit 3 is not a katakana character string. If a registered word of a katakana character string is connected both before and after this one-character unregistered word, the registered word that is connected in front and the component of the word candidate that is connected and the registered word that is connected behind The components of the connected word candidates are detected.

Then, in the morphological analysis, the correctness verification process of the registered word which is the target of the correctness of the registered word (the word registered in the dictionary file 4) detected from the entire katakana character string including the unregistered word. (N25).
Here, it is assumed that the registered word to be verified is valid if it is a word that can form a compound word, and is not valid if it is a word that cannot form a compound word. In the example of the character string "Financial system" described above, the two registered words "Phi" and "System" are to be verified for validity. The registered word verification unit 9 verifies the correctness of the word. The correctness of the word is verified based on the following rules of. Rule based on word string length In this rule, the possibility that a character string that is not a compound word contains a character string that matches the registered word decreases as the character string length of the registered word decreases. Therefore, in this embodiment, (1) a word having a character string length of 4 characters or more is a valid word, and (2) a word having a character string length of 2 or 3 is valid. A word that cannot be determined, (3) A word that is not valid if it consists of one character. Rule by word part of speech In this rule, the correctness is verified based on the function of the part of speech shown below. Since the verb does not have the property of modifying other words or being modified by other words, it is not a constituent word of a compound word. An adverb does not become a constituent word of a compound word because it does not have the property of modifying other words or being modified by other words. Verbs other than syugaku-inflection are not constituent words of compound words. The prefix never ends the compound word. The suffix does not start the compound word. Rendaku does not come at the beginning of a compound word. For the above reason, in the present embodiment, if the word (1) is any of a verb, an adverb, or a verb other than the syllable inflection, the word is not valid, and the (2) word is a prefix, And if the katakana character string is not concatenated after the word, the word is not valid, (3) the word is suffix, rendaku, and the katakana character string is not concatenated before the word. , Unjustified words,
(4) Unless any of the above (1), (2) and (3) is satisfied, it is assumed that the word cannot be judged to be valid. Rules based on word characteristics In this rule, the characteristics of each word are: a word that is likely to form a compound word, a word that is unlikely to form a compound word, a word that is neither No word, say.), Or (1)
Words that are likely to form a compound word that are legitimate, and (2) Words that are unlikely to form a compound word are not legitimate. ,
(3) If the word has no useful property,
It is assumed that it is a word whose validity cannot be determined. As a method of classifying the properties for each word,
Number of compound words formed for each word from multiple documents, etc., independent words containing matching character strings (words that are not compound words)
It suffices to take statistics such as the number of and to set the property of the word based on this statistics. Further, this statistic may be obtained manually, or the statistic may be automatically set to set the property of the word. The process of automatically taking this statistic and setting the property of the word will be described later.

FIG. 5 is a flowchart of the registered word validity verification process in n25. This process first verifies the correctness of the word from the rule based on the character string length. The character string length of the registered word that verifies the legitimacy is 4 characters or more, 2
Alternatively, it is determined whether it is three characters or one character (n41, n42). Here, if the character string length is 4 characters or more, the word is determined to be valid in n48. If the character string length is one character, the word is determined to be invalid in n49. If the character string length is 2 or 3 characters, it is not possible to verify the correctness of the word from the rule based on the character string length of the word, and the correctness is verified by the part of speech of the word.

Here, whether the part of speech of the word is a verb, an adverb, or a verb other than the syllable inflection, whether the part of speech of the word is a prefix and is not followed by a katakana character string, or the part of speech of the word is It is determined whether or not the suffix or rendaku is followed by a katakana character string (n43 to n45), and if the word corresponds to any of the above, it is determined to be an invalid word in n49. Further, if the word does not correspond to any of the above items, it is considered that the validity of the word cannot be verified from the rule of the part of speech of the word, and the validity is verified by the following properties of the word.

As described above, for each word, one of the properties of a word that is likely to form a compound word, a word that is unlikely to form a compound word, or a word that has no useful property is set. ing. It is determined which of the above-described properties of the word to be verified is set (n46, n47), and if the word is likely to form a compound word, it is determined to be valid in n48. If the word is unlikely to form a compound word, the word is determined to be invalid in n49. If the word has no useful property, it is determined that the word cannot be verified for validity (n50). As described above, in this processing, the registered word is determined to be either a valid word, a non-valid word, or a word whose validity cannot be verified. In the above embodiment,
The legitimacy of a word is verified by three rules: the rule based on the character string length of the word, the rule based on the word part of speech, and the rule based on the character of the word. The validity may be verified.

The word candidate detection unit 5 does not detect a word judged to be valid as a constituent element of the word candidate in the validity verification process of the registered word (n26 → n30). If the word is judged to be invalid, if a word in the katakana character string is concatenated before, a word consisting of the character string connecting this word is detected as a constituent element of the word candidate ( n29). If a word of a katakana character string is concatenated behind the word, a word composed of the character string connected to this word is detected as a constituent element of the word candidate (n29). If the word is not validated, the word is detected as a constituent element of the word candidate (n28).

For example, as shown in FIG. 6 (A), for a character string “Financial system”, the result of morphological analysis that “Phi” and “System” are registered words and “National” is an unregistered word In this case, since the character string length of the unregistered word “NANCIAL” is not one character, it is detected as a constituent element of the word candidate. Since the registered word "Phi" is a word whose validity is not verified from the character string length, the word part of speech, and its property, it is detected as a component of a word candidate ("Phi" is a useful property. It is a word that does not have.) In addition, since the registered word “system” has a character string length of 4 characters, it is determined as a valid word according to the rule based on the character string length.
Not detected as a component of a word candidate. Therefore,
In this example, "phi" and "nuclear" are detected as the constituent elements of the word candidate. In addition, as shown in FIG. 6 (B), for the character string “inflator”, the result of the morphological analysis is “in” and “flare” as registered words,
If “Tar” is an unregistered word, the character string length of the unregistered word “Tar” is not one character, so that it is detected as a component of a word candidate. "In" is a word whose validity is not verified from the character string length, word part of speech, and its property, so it is detected as a component of a word candidate ("In" is a word that does not have a useful property. There is.). In addition, since the touching verb "Flae" is determined to be an unjust word according to the rules based on the part-of-speech of the word, the word "In" that is connected to the word "In" that is adjacent to the front and the word "Tar" that is adjacent to the word behind is connected. "Flater" is detected as a component of the word candidate. Therefore, in this example, four "in", "inflation", "flater", and "tar" are detected as the constituent elements of the word candidate. Further, as shown in FIG. 6C, for the character string “Iriomote”, if the result of the morphological analysis is that “i” is an unregistered word and “Rio” and “Mote” are registered words. For example, since the unregistered word "i" has a character string length of one character, it is connected to the word "Rio" adjacent to the character string. In addition, the lower one-stage verb "Mote" is determined to be an invalid word according to the rule based on the word part of speech, and is connected to the adjacent word "Rio" in front. Here, since "I" is already connected to "Rio", "Iriomote" is detected as a component of the word candidate. Further, as shown in FIG. 6 (D), if the result of the morphological analysis is "In", "Tara", and the registered word "Puta" is an unregistered word for the character string "interrupter", The character string length of the unregistered word "Puta" is detected as a component of a word candidate that is not one character. Since “in” and “cod” are words whose legitimacy is not verified from the character string length, the part of speech of the word, and their properties, the word is detected as a component of the word candidate (“in”).
"Cod" is a word that has no useful properties. ). Therefore, in this example, "in""cod"
Three "Puta" are detected as the constituent elements of the word candidate.

A combination of the constituent elements of the word candidate thus detected and the constituent elements of the word candidate which are concatenated is created as a word candidate (n31). For example,
In the example shown in FIG. 6 (A), "Phi" and "National"
Three of "Financial" are created as word candidates. Further, in the example shown in FIG. 6B, "in", "inflation", "flater", "ter", and "inflator" are created as word candidates. In the example shown in FIG. 6C, “Iriomote” is created as a word candidate. In the example shown in FIG. 6D, “IN”, “Tara”, “Puta”, “Interla”, “Trapta”, and “Interrupter” are created as word candidates. In addition, the word candidate "imputa" which connects the words "in" and "puta" which are not connected is not created. Then, the word candidates of the character string created in n31 and not registered in the dictionary file 4 are provisionally registered in the dictionary file 4 (n32). The part of speech of the temporarily registered word candidate is set to “candidate”. In the temporary registration of word candidates, the data (pointer) indicating the position in the dictionary file 4 in which the word of the original character string (word registered in n21) from which this word candidate is cut is registered at the same time. . FIG. 7 shows an example of the dictionary file 4 in which word candidates are registered. In the example shown in FIG. 6A, since “Phi” is already registered in the dictionary, two of “National” and “Financial” are registered as word candidates, and the part of speech is set as a candidate. Further, these word candidates are registered in the dictionary file 4 with data (pointer 8) indicating the position in the dictionary file 4 in which the word “financial system” of the original character string that has been cut out is registered.

The reason why the part of speech of the word registered in n21 is used as a noun will be briefly described below. The sources of unregistered katakana words are roughly classified into the following three categories. When the verbs, adjectives, and nouns of foreign words are written in Japanese with katakana notation (see FIG. 8A) Is used as a noun. Also, foreign words adjectives become Japanese adjectives.
In addition, foreign words that have both adjective and noun properties are often written in katakana. For these reasons, it can be said that the unregistered word in katakana generated from this source is very likely to be a noun. When difficult kanji or words to be emphasized are written in katakana in Japanese (see Fig. 8 (B)) The katakana unregistered words generated from this source are verbs, adjective verbs, and nouns that are used as a safing as described above. In addition, it can be said that most proper nouns are treated grammatically in the same way as "nouns". Therefore, it can be said that the unregistered word in katakana generated from this source is very likely to be a noun. When an onomatopoeia or mimetic word of a foreign word is diverted, a foreign word is shortened, or a word is generated from a harmony katakana word (see FIG. 8C) In this case, there are various parts of speech, and which part of speech is the most. However, statistically speaking, the frequency of occurrence of unregistered katakana words from such sources is extremely low. For the reasons (1) to (3) above, it is considered optimal to use the part-of-speech of the unregistered katakana word as a noun.

At n7, it is judged from the result of the morphological analysis whether or not there is a portion where the registered words of a plurality of katakana character strings are connected. Here, if there is a portion where the registered words of a plurality of katakana character strings are concatenated, the second word candidate detection and registration processing is executed in n8. FIG. 9 is a flowchart showing the flow of the second word candidate detection / registration process.
In the morphological analysis result, there is a character string such as "Kalimantan" or "card system" as an example in which registered words of a plurality of katakana character strings are connected. The morphological analysis result of the character string "Kalimantan" is shown in FIG. 10 (A), and the morphological analysis result of the character string "card system" is shown in FIG.
It is shown in 0 (B). The character string “Kalimantan” is determined by morphological analysis to be a character string in which three registered words “Kali”, “Man”, and “Tan” are concatenated. The character string “card system” is determined by morphological analysis to be a character string in which two registered words “card” and “system” are concatenated.

The registered word verification section 9
The validity of the word is verified (n51, n52). The validity of this word is verified by the processing shown in FIG. 5 described above. Then, whether an unjustified word is detected (n5
3) Alternatively, it is determined whether or not words whose validity cannot be verified are concatenated and detected (n54). Here, if no unjustified word is detected and no unverified word is concatenated, it is determined that there is no possibility that an unregistered word is included, and the process is completed. When an unjustified word is detected, or when unverifiable words are concatenated and detected, the following processing is performed.

The entire katakana character string is registered as one word and the part of speech is registered as a noun in the dictionary file 4 (n5).
5). A word whose validity cannot be verified is detected as a constituent element of a word candidate (n56). In addition, the word that is determined to have no validity, if the registered word of the katakana character string is concatenated before, detects the word consisting of the character string connected to this registered word as a constituent element of the word candidate, When the registered words of the katakana character string are concatenated at the back, the word consisting of the character string connecting the registered words is detected as a constituent element of the word candidate (n57). Then, the constituent elements of the detected word candidates are combined to create word candidates (n58),
The word candidates of the character strings that have been created and are not registered in the dictionary file 4 are provisionally registered in the dictionary file 4 (n59). The part of speech of the temporarily registered word candidate is set as the candidate. Further, the position in the dictionary file 4 in which the word of the original character string (word registered in n55) from which this word candidate is cut out is also stored.

For example, it is assumed that all three registered words "Kari", "Man", and "Tan" that are the result of morphological analysis of the character string "Kalimantan" are words whose validity cannot be verified. In this case, the part of speech of "Kalimantan" is registered as a noun in the dictionary file 4 at n55. Also, "Kari"
“Man” and “Tan” are detected as constituent elements of the word candidate, and “Kaliman” and “Mantan” are provisionally registered as word candidates. In addition, the word "Karitan" which connects the words "Kari" and "Tan" which are not connected is not created as a word candidate. Further, "system", which is the result of the morphological analysis of the character string "card system", is determined to be a valid word from the rule based on the character string. Therefore, since an unjustified word is not detected and a word whose authenticity cannot be verified is not concatenated, it is determined that there is no possibility that an unregistered word is included, and the process after n55 is not performed.

That is, in this embodiment, when the result of the morphological analysis includes an unjustified word, or when a word whose authenticity cannot be verified is concatenated,
It is determined that the katakana character string may include an unregistered word, and in other cases, it is determined that the katakana character string may not include an unregistered word. When it is determined that the katakana character string may include an unregistered word, a word candidate is created and temporarily registered in the dictionary file 4.

In the second word candidate detection and the verification of the correctness of the words in the registration process, the above-mentioned rule is too strict and the correct words are judged to be the non-valid words. Cases are also envisioned. In such a case, there is a possibility that a compound word composed of a plurality of words registered in the dictionary file 4 may be registered as an unregistered word. Therefore, this second
In the word candidate detection / registration process, the rule based on the character string length of the word for verifying the validity of the word may be changed as follows.

(1) If the character string length is 3 characters or more, the word is valid, (2) If the character string length is 2 characters, the word whose validity cannot be determined, (3) It is assumed that one character is a word that is not valid. As described above, the possibility of registering a compound word composed of a plurality of registered words in the dictionary file 4 can be reduced by making the change.

At n9, it is determined whether the result of the morphological analysis includes a word candidate temporarily registered in the dictionary file 4 (whether the word candidate has reappeared). Here, when it is determined that the word candidate has reappeared, the validity verification process of the word candidate of n10 is executed. FIG. 11 is a flowchart showing the validity verification process of word candidates. First, it is determined whether or not the entire katakana character string connected to the re-appearing word candidate matches the character string cut out when the word candidate is provisionally registered in the dictionary file 4 (n61). That is, "Financial system"
If the word “Financial” cut out from the character string is provisionally registered in the dictionary file 4, it is determined again whether the word candidate “Financial” is cut out from the same character string. If it is determined in n61 that the word candidate is the same as the cut out character string,
The processing is completed because the correctness of the word candidate cannot be accurately verified.

When it is determined in n61 that the character strings are not the same, the word candidate is 2 in the result of the morphological analysis of this character string.
It is determined whether one or more is included (n62). n
If two or more word candidates are included in 62, it is determined that the validity of the word candidate cannot be verified, and the process is completed. On the other hand, when only one word candidate is included in this katakana character string, the validity verification process shown in FIG. 5 is performed on each registered word (n63, n6).
4). Then, if all the registered words are not determined as valid words (n65), the validity of the word candidates cannot be verified and the process is completed. If the validity of all registered words is verified, it is determined that the word candidate is valid, and the word candidate is formally registered in the dictionary file 4 (n66). The process of officially registering a word candidate in the dictionary file 4 is a process of changing the part of speech from a candidate to a noun. The word candidate formal registration unit 10 officially registers the temporarily registered word candidates.

For example, the character string “Financial system” is detected, and “Financial” and “National” are provisionally registered in the dictionary file 4 as word candidates. Here, the result of the morphological analysis of the character string "Financial Advisor" is as shown in FIG. Although it can be considered that the result of the morphological analysis of the character string “Financial Advisor” is as shown in FIG. 12 (B), the longest matching method (the longest word is given priority to the longest word is a general method of morphological analysis. Yes.)
Alternatively, using the minimum clause number method (minimizing the number of words to be divided) does not produce such a result. Then, the "advisor" is determined to be a valid word based on the character string length. As a result, the character string “Financial” is also determined to be a valid word candidate, and the part of speech of “Financial” in the dictionary file 4 is changed from a candidate to a noun. This makes it a “financial”
Is officially registered in the dictionary file 4.

When the above-mentioned processing is completed for the entire series of text data stored in n1, processing for deleting unnecessary words temporarily registered in the dictionary file 4 is executed. FIG. 14 is a flowchart showing the flow of unnecessary word deletion processing for deleting unnecessary words. Of the words registered in the dictionary file 4, all the words whose parts of speech are candidates are detected and deleted (n71 to n73). As a result, all of the word candidates that have been provisionally registered as word candidates but the same character string does not appear thereafter, and the word candidates whose validity has not been verified are deleted. For example, in the case of the dictionary file 4 shown in FIG. 13, "native,""inflation,""flater," etc. are deleted (see FIG. 15 (A)). Then, if there is a word storing a pointer indicating the position of the original character string, the word registered at the position designated by this pointer is deleted, and this pointer is also deleted at the same time (n74 to n77). ). As a result, the correctness is verified by the above processing, and the words in the katakana character string obtained by cutting out the words registered in the dictionary file 4 as the official words are deleted. Note that this Katakana character string is a compound word, and there is no problem even if it is not registered in the dictionary file 4. For example, in the dictionary file 4 shown in FIG. 15A, “Financial system” is deleted (see FIG. 15B), but “Financial” and “System” are registered as words, so “Financial” The removal of the "system" does not cause any problems.

As described above, in the present invention, the unregistered word is accurately detected from the compound word of the katakana character string formed by concatenating the unregistered word and the registered word and registered in the dictionary file 4. be able to. Further, even when the character string of the unregistered word and the katakana character string formed by connecting a plurality of registered words match, the unregistered word is accurately detected and registered in the dictionary file 4. You can Furthermore, since the incorrectly registered words are finally deleted, unnecessary words are not registered in the dictionary file 4.

Next, statistics are automatically taken and the characteristics of each word are classified into words that often form compound words, words that rarely form compound words, and words that do not have useful characteristics. The process of setting one of them will be described. FIG.
It is a flowchart which shows the property determination process of this word.
Here, as shown in FIG. 17, the dictionary file 4 has an area for storing the property of each word. In the figure, the property of a word that often forms a compound word is 1, the property of a word that rarely forms a compound word is 2, and the property of a word that does not have a useful property is 3. In addition, statistical data in which compound words and words (including unregistered words that are not registered in the dictionary file 4) are registered as data for obtaining statistics is prepared (see FIG. 18). In the compound word, "*" is put between the words as a symbol indicating a word delimiter.

When a word for which statistics are taken (hereinafter referred to as a target word) is selected and input in n81, all the words including the word as a partial character string are detected (n8).
2). For example, if the target word is “ism”, FIG.
From the dictionary file 4 shown in, "ism" and "egoism"
"Dadaism", "heroism", and "hebrewism" are detected, and if the target word is "money", "money", "managed", "management", "manager", and "manager" are detected. Then, the words are extracted one by one from the statistical data (n83), and the following determination is performed. It should be noted that the compound words are extracted for each word delimited by. For example, "ism group", which is a compound word in statistical data
If is included, it is extracted as two words of “ism” and “group”.

It is judged whether or not the target word is included in the word extracted in n83 as a partial character string (n84),
If it is not included, the process returns to n83 to extract the next word. On the other hand, if the target word is included as a partial character string, it is determined whether the word extracted in n83 and the target word completely match (whether the character string length is the same) (n85),
If the character string lengths are the same, the counter a (not shown) is incremented by 1 (n86). If the character string lengths are not the same (if the character string length of the word extracted in n83 is longer than the character string length of the target word), is the same word as the word extracted in n83 detected in n82? It is determined (n87) whether or not a completely matching word is registered in the dictionary file 4 (n87). If it is determined that the completely matching word is registered in the dictionary file 4, the process returns to n83 and the next word is extracted. For example, “egoism” is registered in the dictionary file 4, and n8
This is a case where the word extracted in 3 is also "egoism". If it is determined that the completely matching word is not registered in the dictionary file 4, the counter b (not shown) is incremented by 1 (n88). When the process of n86 or n88 is completed, the process returns to n83 and the next word is extracted. The counters a and b are set to 0 when the target word is selected in n81. All the words in the statistical data are extracted and the above n83
When the subsequent process is completed (n89), the following process for determining the property of the word is performed (n90), and this property is registered in the dictionary file 4 as the property of the target word (n9).
1).

Using the dictionary file 4 shown in FIG. 17 and the statistical data shown in FIG. 18, when "ism" and "money" are the target words, the count values of the counter a and the counter b in the above processing are as follows. It looks like this: “Ism” a = 1, b = 6 “Money” a = 5, b = 0 As described above, words that often form compound words, words that rarely form compound words, and useful It is determined to be one of three words that do not have this property.
In this embodiment, if a / (a + b)> 0.8 holds, the property is likely to be an independent word in a compound word, and if b / (a + b)> 0.8 holds, the property becomes It is difficult to be an independent word in a compound word,
Otherwise, it is determined that the word has no useful property.

At n90, the above calculation is performed by using the count values of the counter a and the counter b obtained by the above processing, and the nature of the word is determined. Then, in n91, this property is registered in the dictionary file 4 as the property of the target word, and the process is completed.

As described above, in the present embodiment, since the word property is set based on the statistics, the word property can be set objectively.

[0059]

As described above, according to the present invention, when an unregistered word is included in a concatenated katakana character string, the range of the character string forming the unregistered word is accurately specified to identify the unregistered word. Can be detected, and the detected unregistered word can be registered. In addition, since the words that are erroneously detected and registered in the dictionary file are finally deleted, the capacity of the dictionary file does not become unnecessarily large.

[Brief description of drawings]

FIG. 1 is a block diagram showing functions of a Japanese sentence analysis device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a process of the Japanese sentence analysis device of this embodiment.

FIG. 3 is a diagram showing a concept of morphological analysis.

FIG. 4 is a flowchart showing a flow of first word candidate detection / registration processing.

FIG. 5 is a flowchart of a registered word validity verification process.

FIG. 6 is a diagram showing examples of detected word candidate components and created word candidates.

FIG. 7 is a diagram showing a dictionary file in which word candidates are registered.

FIG. 8 is a diagram illustrating a generation source of unregistered katakana words.

FIG. 9 is a flowchart showing a flow of second word candidate detection / registration processing.

FIG. 10 is a diagram showing examples of detected word candidate components and created word candidates.

FIG. 11 is a flowchart showing the validity verification process of word candidates.

FIG. 12 is a diagram showing a morpheme analysis result of a character string including word candidates.

FIG. 13 is a diagram showing a dictionary file when a temporarily registered word candidate is officially registered.

FIG. 14 is a flowchart showing a flow of unnecessary word deletion processing for deleting an unnecessary word.

FIG. 15 is a diagram showing a dictionary file when unnecessary words are deleted.

FIG. 16 is a flowchart showing a word property determination process.

FIG. 17 is a diagram showing a dictionary file that stores the properties of words.

FIG. 18 is a diagram showing statistical data.

[Explanation of symbols]

 1-Japanese sentence analysis device 2-Text data storage unit 3-Morpheme analysis unit 4-Dictionary file 5-Word candidate detection unit 6-Word candidate registration unit 7-Word candidate verification unit 8-Word candidate deletion unit 9-Registration word verification Department 10-Word candidate formal registration department

Claims (12)

[Claims] 1. A dictionary file in which a character string of a word and data indicating an attribute of the word are registered, and a morpheme analysis means for performing a morpheme analysis for dividing a character string input using the dictionary file into words. When the concatenated katakana character strings in the input character string are divided into a plurality of words by the morphological analysis, if the divided words include a word having a character string length of 1 character, An unregistered word detecting means for detecting a character string connecting the word and an adjacent word before or after the word as an unregistered word, and an unregistered word registering means for registering the detected unregistered word. A Japanese sentence analysis device characterized by being equipped with. 2. A dictionary file in which a character string of a word and data indicating an attribute of the word are registered, and a morphological analysis means for performing a morphological analysis for dividing the character string input using the dictionary file into words. Provided, when the concatenated katakana character string in the input character string is divided into a plurality of words by the morphological analysis, if a word of a predetermined part of speech is included in the divided words, the word And an unregistered word detecting means for detecting a character string connecting the adjacent word before or after the word as an unregistered word, and an unregistered word registering means for registering the detected unregistered word. Japanese sentence analysis device characterized by 3. A dictionary file in which a character string of a word and data indicating an attribute of the word are registered, and a morphological analysis means for performing a morphological analysis for dividing the character string input using the dictionary file into words. The data indicating the attribute includes data indicating whether or not the corresponding word is a word less likely to form a compound word, and the concatenated katakana character string in the input character string is the morphological analysis. When the word is divided into a plurality of words by the word, if the word that is unlikely to form a compound word is included in the divided word, the word and the word adjacent to the word before or after the word are connected. An unregistered word detecting means for detecting a character string as an unregistered word, and an unregistered word registering means for registering the detected unregistered word. 4. The unregistered word detecting means, if the word divided into a plurality of words by the morphological analysis includes a word having a predetermined part of speech, the unregistered word detecting means is adjacent to the word and the word before or after the word. 2. The Japanese sentence analysis apparatus according to claim 1, further comprising means for detecting a character string connecting a matching word as an unregistered word. 5. The data indicating the attribute includes data indicating whether or not the corresponding word is a word that is unlikely to form a compound word, and the unregistered word detection means detects a plurality of words by the morphological analysis. If a word divided into words contains a word that is unlikely to form a compound word, a character string connecting the word and the word adjacent to the word before or after the word is detected as an unregistered word. 3. Means including means.
Or the Japanese sentence analysis device described in any one of 4. 6. When a word of a character string that matches the unregistered word detected by the unregistered word detecting means is included in a word divided into a plurality of words by the morphological analysis, a predetermined rule is set. 6. The Japanese sentence analysis device according to claim 1, further comprising a legitimacy verifying unit that verifies whether or not the unregistered word is valid as a word by comparison. 7. A morphological analysis that divides an input character string into words using a dictionary file in which character strings of words and data indicating attributes of the words are registered, and concatenates in the input character strings. When a katakana character string is divided into a plurality of words by the morphological analysis, if a word having a character string length of 1 character is included in the divided words, the word and the word before or after the word are adjacent to each other. A Japanese sentence analysis method characterized by detecting a character string connecting a matching word as an unregistered word and registering the detected unregistered word. 8. A morphological analysis for dividing an input character string into words is performed using a dictionary file in which a character string of a word and data indicating attributes of the word are registered, and concatenation in the input character string is performed. When a katakana character string is divided into a plurality of words by the morphological analysis, if a word having a predetermined part-of-speech is included in the divided words, the word and a word adjacent to the word before or after the word A method for analyzing a Japanese sentence, characterized in that a character string connected to each other is detected as an unregistered word, and the detected unregistered word is registered. 9. A character string input using a dictionary file in which data indicating attributes including data indicating whether a character string of a word and a corresponding word are unlikely to form a compound word is registered. Performing morphological analysis to divide into words, and when the concatenated katakana character string in the input character string is divided into a plurality of words by the morphological analysis, a possibility of forming a compound word in the divided words If a word containing a small number of words is included, a character string connecting the word and an adjacent word before or after the word is detected as an unregistered word, and the detected unregistered word is registered. Japanese sentence analysis method. 10. If a word having a predetermined part-of-speech is included in a word divided into a plurality of words by the morphological analysis, a character string connecting the word and an adjacent word before or after the word. 9. The Japanese sentence analysis method according to claim 7, wherein is detected as an unregistered word. 11. The data indicating the attribute includes data indicating whether or not a corresponding word is a word that is unlikely to form a compound word, and is included in a word divided into a plurality of words by the morphological analysis. 8. If a word that is unlikely to form a compound word is included, a character string that connects the word and a word adjacent to the word before or after the word is detected as an unregistered word. , 8 or 10 Japanese sentence analysis method. 12. When a word of a character string that matches the detected unregistered word is included in a word divided into a plurality of words by the morphological analysis, the unregistered word is valid as a word. 12. The Japanese sentence analysis method according to claim 7, wherein the verification is performed.