JP2001265763A - Morpheme analytic method and recording medium with recorded morpheme analytic program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve word acknowledgement accuracy in morpheme analysis. SOLUTION: In word dictionary retrieving processing 2-1, a Japanese text is inputted, a word dictionary is retrieved while using a word dictionary 5 and a grammar rule 4, all connectable word streams are selected and a word chain candidate stream 2-2 is prepared. In word selecting processing 2-3, when word information is applied to an inputted Japanese text 1 or registered in setting information 6, a word, which satisfies designated word information, is selected while applying priority to the part-of-speech chain of the previously specified grammar rule 4 or when there is not such a word, it is newly generated. Thus, concerning a character string, with which the word information 5 is designated, a word satisfying the designated word information or having the most certain information concerning non-designated word information is acknowledged as a first solution, the word, to which the word information is applied, is registered in the setting information 6 corresponding to the designation and a Japanese word information stream 3 is outputted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a Japanese morphological analysis method for dividing a Japanese text into words and adding information such as readings and parts of speech.

[0002]

2. Description of the Related Art In word recognition in Japanese morphological analysis, a word that satisfies a connectable part-of-speech chain is extracted using a word dictionary and grammatical rules, priority is assigned to the part-of-speech chain, and narrowing down by the part-of-speech chain is performed. Is performed, if there is an isomorphous word, a method of narrowing down by word priority and finally extracting a word chain as a first solution is common.

As a specific method of assigning priorities, there is a minimum number of clauses method for giving priority to a chain having a small number of clauses (Yoshimura et al.,
“Morphological analysis of solid Japanese sentences using the minimum number of clauses method”, Jiho-Ken, Vol. 24, no. 1, 1983), a cost minimization method in which costs are assigned to adjacent rules of words, and the sum of the rules is given priority to the chain with the minimum sum (Hisamitsu et al., "Proposal of morphological analysis and evaluation of computational complexity by the minimum connection cost method," Techniques, NLC90-8, 1990) and techniques combining them have been established.

However, in these conventional methods, even when a plurality of readings are considered, the word having the most common reading is selected as the first solution. Correct word recognition may not be performed. For example, “Kenji Nakajima” is usually recognized as the most common reading word as “Kenji Nakajima” in normal morphological analysis,
In fact, the certification "Kenji Nakajima (Takeshi)" may be correct.

Although there are many proper nouns having the same notation as general nouns, it is difficult to determine which one is appropriate without considering the context (for example, "the mayor of a field (= place name)", " Nohara (= general noun) "), in ordinary morphological analysis, it is generally fixed to one of the parts of speech.

[0006]

In the conventional Japanese morphological analysis method, a word having an unusual reading cannot be correctly given a reading. For example, when the word is used in text-to-speech synthesis, a personal name is erroneously read. And so on.

[0007] In addition, a word for which it is difficult to distinguish a part of speech is fixed to one of the parts of speech and is recognized. For example, even when the description of a product is known, the product name is treated as a general noun. There was a problem.

An object of the present invention is to provide a morphological analysis method for improving word recognition accuracy.

[0009]

According to the present invention, a word dictionary is searched using a word dictionary and grammar rules, all connectable word strings are selected, and a word chain candidate string is created. Next, word information is added to the input Japanese text,
Alternatively, if the word information is registered in the setting information, the word information that satisfies the specified word information is selected while giving priority to the part-of-speech chain of the grammatical rules prescribed in advance. For the given string
For the word information that satisfies the specified word information, and for the word information that is not specified, the word with the most likely information is recognized as the first solution, the word given the word information is registered in the setting information according to the specification, and the Japanese Output a word information sequence.

[0010]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a flowchart showing a morphological analysis method according to one embodiment of the present invention. A Japanese text 1 is input, a word dictionary search process 2-1 is performed, and a word chain candidate sequence 2-2 is output. , Word selection process 2-3, and the Japanese word information sequence 3
Grammar rule 4, word dictionary 5, and setting information 6
This is performed using

Japanese text 1 is any Japanese text. However, word information may be added. The word information referred to here is information to be registered in the word dictionary 5, and generally includes a lot of information such as notation, part of speech, reading, semantic attributes, and accent type. Here, since the notation is naturally included in the Japanese text 1 to be analyzed, word information other than the notation is added.

The word information designation pattern for assigning word information can be arbitrarily designated by the word information designation pattern of the setting information 6. For example, in FIG. 2A, a character string to which word information is to be added is enclosed by “$”, its part of speech is enclosed by “<>”, and the reading is enclosed by “()”. When specifying word information, the target character string and one or more word information are indispensable, but it is not necessary to describe all word information registered in the word information specification pattern. For example, in the example of FIG. 2, the part of speech and the reading are specified for the target character string “Nakajima”, but only the reading is specified for “Kenji”.

As described above, when specifying word information using a character string pattern, if characters used in the character string pattern are to be subjected to morphological analysis, "@" is added immediately before. For example, in order to analyze the European Union (EU) using the word information designation pattern of FIG. 2A, the Japanese text of the European Union (EU) is used.

FIG. 2B shows the designation of word information by XML. Based on XML, set the target character string to <wor
This is a description that the <d> element and its reading are specified by the yomi attribute of the <word> element, and the part of speech is specified by the Hinshi attribute of the <word> element. In this case, as in FIG. 2A, one or more attributes may be specified in the <word> element.

Returning to FIG. 1, the morphological analysis 2 is performed for each sentence.

In the word dictionary search process 2-1, grammar rules 4
Is extracted from the word dictionary 5 and the word chain candidate sequence 2-2 is output.

The word chain candidate sequence 2-2 is composed of a part of speech chain and a word. For example, in FIG. 5, the part connected by a line is a part-of-speech chain (the part-of-speech name is in <>), and one or more words exist in each part-of-speech chain.

Next, the word selection process 2-3 will be described with reference to FIG.
This will be described in detail with reference to FIG.

In step 10, the word chain candidate sequence 2-2
, And the word chain with the highest priority (the word with the highest priority in the part-of-speech chain with the highest priority) is set as the first solution.

In step 11, the Japanese text 1 is
A character string given word information or a word registered in the registered word information of the setting information 6 (hereinafter, “target character string”
Abbreviated). If there is, go to step 12. Here, when there are a plurality of target character strings, the processing after step 12 is sequentially performed on each target character string. If not, the process ends.

In step 12, it is determined whether or not a word that completely matches the target character string exists in the part of speech chain with the highest priority given in step 10. If there is, go to step 16. Step 1 if not present
Move to 3.

Here, "perfect match" will be described more specifically.

"Exact match" means that the target character string exactly matches the word boundary and that the word that satisfies all the specified word information and the target character string and the word boundary are different but the specified word information Also includes words that satisfy all.

For example, "{Yamada} (Yamada) Taro" (using the designated pattern in FIG. 2A) is input as Japanese text 1 (ie, only the reading of "Yamada" is designated).
If the word "Yamada Taro (Yomi = Yamataro)" is included in the first part-of-speech chain, the word from the beginning matches "Yamada" and there is no other word information specified. Is treated as an exact match.

Conversely, "｛Taro Yamada" is input as Japanese text 1, and "Yamada (reading = Yamada)" + "Taro (reading = Is included in the first part-of-speech chain, the notation of the two words matches the target character string, and the reading matches, so this word chain is treated as a completely matching word (chain). .

In step 13, it is determined whether or not a word that matches the target character string and the word boundary and has a part of speech designation also exists in the words of the word chain candidate string 2-2. If it exists, the process proceeds to step 14, and if not, the process proceeds to step 15.

In step 14, if a part of speech is specified for the target character string, if there is a word that matches the part of speech and satisfies other specified word information, the word is selected. If other specified word information is different,
Duplicate the word including the word chain and overwrite other specified word information.

If no part of speech is specified for the target character string, the word of the highest priority word chain in step 13 is copied including the part of speech chain, and the other specified word information Overwrite.

In step 15, if the part of speech is specified for the target character string, of the part of speech chain having the boundary of the target word string as a word boundary, the part of the part of speech that can be chained to the part of speech specified at that boundary is selected. The part-of-speech chain before and after the high-priority chain is copied, and the target word string fills in the word information specified as one word. Unspecified word information is generated from the original part-of-speech word (if there is more than one, the word with the highest priority) (for example, all readings are connected). If there is no chainable thing, a chain with an unknown word is generated.

If the part of speech is not specified, the highest priority word chain among the part of speech chains having the boundary of the target word string as the word boundary is copied, and the target word string is the word specified as one word. Fill in information. Unspecified word information is generated from the information of the original word.

In step 16, the priority of the word (chain) handled in the immediately preceding process is raised, and the word (chain) is selected at the first place.

Returning to FIG. 1, grammar rule 4 describes a grammar for generating a part-of-speech chain and assigning a priority.

The word dictionary 5 registers words recognized by morphological analysis and their word information.

The setting information 6 is a word information specification pattern (see FIG. 2) describing a target character string to which word information is to be assigned and a method of specifying word information to be given in the Japanese text 1. The word information includes word storage information indicating whether or not to use the word information as registered word information, word information designated by the word information, and registered word information for storing the specified word information.

If the word storage information of the setting information 6 is on in step 17, the word whose word information is designated is registered in the registered word information in step 18, and if it is off, the word is not registered. Also, by deleting the word information from the registered word information, the designated word information is released.

The Japanese word information sequence 7 is a word chain sequence finally selected. There may be a case where only the first solution is narrowed down, or a case where the top several word chains are presented.

FIG. 4 shows an example of Japanese text 1 using the word information designation pattern shown in FIG.

Hereinafter, the word selection process 2-3 will be described with reference to the example of FIG. Here, it is assumed that the priority is assigned using the adjacent cost of the part of speech, and the word cost is used to select a plurality of words having the same adjacent cost. ).

It is also assumed that grammar rule 4 describes that proper nouns: place names and proper nouns: names cannot be connected, signs and proper nouns, and proper nouns and signs can be connected.
Further, connection to an unknown word is allowed only when there is no other connected word.

The word storage information of the setting information 6 is turned on.

As for the first sentence, since there is no designation of the target character string, a word is selected by the normal priority assignment in step 10, and the process is terminated.

In the second sentence, there are two designations of the target character string, namely, the reading "Nakashima" for "Nakajima" and the reading "Takeshi" for "Kenji". FIG. 5A shows the word chain candidate sequence 2-2 at this time. here,
The flow of FIG. 3 will be described in detail using “Nakajima” as an example.

In step 10, the tracing of the part-of-speech chain (first part-of-speech sequence) of the solid line with the small total adjacent cost is performed, and "Kenji"
For, select a word with low word cost and the reading is "Kenji", and give priority to the space symbol Nakajima proper noun: last name Kenji Nakajima proper noun: first name Kenji's personal name suffix as the first solution.

The process proceeds from step 11 to step 12, and the word "Nakajima" included in the first part-of-speech chain sequence has only the reading "Nakajima" (there is no "Nakashima" reading). Go to step 13.

In step 13, since the part of speech is not specified (satisfies the part of speech) in the target character string "Nakajima" and the word "Nakajima" exists, the process proceeds to step 14.

In step 14, the word chain with the highest priority is "Nakajima (reading = Nakajima, part of speech = proper noun:
Therefore, the word is duplicated, the pronunciation is replaced with the designated "Nakashima", and the process proceeds to step 16 (see FIG. 6A).

At step 16, Nakashima proper noun: surname Nakashima is set as the first solution, and the process proceeds to step 17.

The process proceeds from step 17 to step 18, and
“Nakajima (reading = Nakashima)” is registered in the registered word information.

As for “Kenji”, since a word that completely matches the given pronunciation “Takeshi” exists in the first part-of-speech chain in step 12 (“Kenji, Yomi = Takeshi”), it is selected. Then, the process proceeds to step S16.

In step 16, instead of "Kenji, Yomi = Kenji", "Kenji, Yomi = Takeshi" is selected as the first priority word.

Finally, a word chain of the space symbol Nakajima proper noun: last name Kenji Nakashima proper noun: first name Takeshi personal name suffix is output as the first solution.

In the third sentence, the word "Nakashima" at the beginning of the sentence matches the word registered in the registered word information, so that the word given the reading "Nakashima" in the same flow as the second sentence is the first solution. Become.

Hereinafter, the flow of FIG. 3 will be described in detail for the “interspace” in which the part of speech is specified. The word chain candidate sequence 2-2 of this part is shown in FIG.

In step 10, a solid part-of-speech chain (= word chain, since all are composed of one word) having a small adjacent cost is the first solution.

The process proceeds from step 11 to step 12, and since there is no word in the first part-of-speech chain that exactly matches the proper noun which is the part of speech given to the given word information "interspace", the process proceeds to step 13.

In step 13, "interspace"
Since there is no word of the proper noun which is the part of speech given to, the process proceeds to step S15.

In step 15, the front boundary of "inter (prefix)" and the rear boundary of "space (noun)" which are the word chains of the first solution coincide with the boundary before and after the target character string "inter space". , In grammar rule 4, the symbol (")
Since the chain of a proper noun and a proper noun and a symbol (") is permitted, the part-of-speech chain of the first solution is copied and connected to a new word" interspace "(FIG. 6B).
The pronunciation of "interspace" is a combination of the readings of "inter (prefix)" and "space (noun)", and is referred to as "inta space".

In step 16, the first part-of-speech chain up to this point is replaced with the part where the part-of-speech chain of bold lines occurs, and the interspace proper noun int-space is set as the first solution.

Step 17 is followed by step 18.
"Interspace (part of speech = proper noun)" is registered in the registered word information.

As for the fourth sentence, since there is no designation of the target character string, the word is selected by the normal priority assignment in step 10, and the process is terminated.

FIG. 7 shows the configuration when the document analysis method of the present invention is implemented on a computer such as a personal computer.

The input device 21 is an input device such as a keyboard for inputting Japanese text 1. Storage device 2
The grammar rules 4, the word dictionary 5,
The setting information 6 is stored. The storage device 25 is a hard disk. The output device 26 is an output device such as a printer or a display to which a Japanese word information sequence is output.
The recording medium 27 is a recording medium, such as a floppy disk, a CD-ROM, or a magneto-optical disk, on which a document analysis program including the word dictionary search 2-1 and the word selection 2-3 is recorded. The data processing device 28 includes a CPU and various interfaces, reads a document analysis program from the recording medium 27, and executes it.

As described above, according to the present invention,
Word information such as reading, part of speech, and accent type can be specified for some words in the input text, and words that satisfy the specified word information while giving priority to the part of speech chain defined in advance by grammar rules. If no selection is made, a new word is generated to satisfy the specified word information, and for unspecified word information, the word with the most probable information is recognized as the first solution, and the word once specified is retained. By performing morphological analysis for certifying words that satisfy the specified condition until the word is canceled, word recognition accuracy can be improved by positively providing known information.

[Brief description of the drawings]

FIG. 1 is a diagram showing a morphological analysis method according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example of a word information designation pattern and an example of a Japanese text corresponding thereto.

FIG. 3 is a flowchart illustrating a process of a word selection process 2-3.

FIG. 4 is a diagram showing an example of a Japanese text.

FIG. 5 is a diagram showing an example of a word chain candidate string and a priority given by normal priority assignment;

FIG. 6 is a diagram illustrating an example of adding a generated word to a word chain candidate sequence.

FIG. 7 is a block diagram of an apparatus for implementing the morphological search method of the present invention.

[Explanation of symbols]

 1 Japanese text 2 Morphological analysis 2-1 Word dictionary search process 2-2 Word chain candidate sequence 2-3 Word selection process 3 Japanese word information sequence 4 Grammar rules 5 Word dictionary 6 Setting information 10-18 Step 21 Input device 22 -25 storage device 26 output device 27 recording medium 28 data processing device

Claims (8)

[Claims] 1. A morphological analysis method for dividing a Japanese text into words and adding word information, the method comprising inputting the Japanese text, describing a grammatical rule for generating a part-of-speech chain and assigning a priority A word dictionary search step that extracts all connectable word chains from the word dictionary that has registered words and their word information that have been identified by morphological analysis, creates and outputs word chain candidate strings, and adds words to the input Japanese text. Character strings to which information is added or registered in setting information that describes a target character string to which the word information is to be added in Japanese text and a method of specifying the word information to be given are prescribed in advance. By giving priority to the part-of-speech chain of the grammar rules and selecting a word that satisfies the specified word information, and generating a new word if it is not, for the character string whose word information is specified, The specified word information is satisfied, and the unspecified word information is recognized as the first solution having the most probable information, and the word given the word information is registered in the setting information according to the specification, and A morphological analysis method including a word selecting step of outputting a word / word information sequence. 2. The method according to claim 1, wherein the word selecting step includes the step of saving and designating the character string in which the word information is once specified in the input Japanese text when the same designation is performed in the subsequent text. The method according to claim 1, wherein the information is deleted when canceling. 3. The method according to claim 1, wherein the word selecting step assigns a priority to the word chain candidate sequence, and sets a chain having the highest priority as a first solution. A second step of determining whether or not there is a target character string that is a character string given information or a word registered in the registered word information of the setting information; A third step of determining whether a word that exactly matches the character string exists in the part-of-speech chain having the highest priority, and if there is no word that completely matches the target character string,
A fourth step of determining whether a word whose word boundary matches the word boundary and which matches the part of speech when there is a part of speech designation is present in the word chain candidate sequence; and If there is a word whose boundary matches, if a part of speech is specified for the target character string, if there is a word that matches the part of speech and satisfies other specified word information, the word is selected. If the other specified word information is different, duplicate the word including word chain, overwrite the other specified word information, and if no part of speech is specified for the target character string A fifth step of duplicating the word of the highest priority word chain in the fourth step including the part of speech chain and overwriting other specified word information; If the part of speech is specified, Among the part-of-speech chain with the boundary of the elephant word sequence as a word boundary,
The part-of-speech chain before and after the highest-priority chain that can be chained to the part-of-speech specified at that boundary is copied, and the target word string fills in the word information specified as one word. If the original part-of-speech chain word (if there are multiple words, the word with the highest priority) is generated, and if there is no chainable word, a chain with an unknown word is generated. If not specified, the part-of-speech chain with the boundary of the target word string as the word boundary
Copy the word sequence with the highest priority, and set the target word string to 1
A sixth method in which word information specified as a word is filled, and word information not specified is generated from information of the original word.
The method according to claim 1, further comprising the steps of: (a) increasing the priority of the words handled in the third, fifth, and sixth steps, and selecting the words in the first place. 4. The method according to claim 3, wherein the word selecting step further comprises a step of registering a word for which the word information is specified in the registered word information when the word storage information of the setting information is on. 5. A morphological analysis program that divides a Japanese text into words and adds word information, the morphological analysis program being described in a grammar rule for inputting a Japanese text and generating a part of speech chain and assigning a priority. A word dictionary search process that extracts all connectable word chains from the word dictionary in which morphological analysis identifies words and their word information, creates and outputs word chain candidate strings, and outputs words to the input Japanese text Character strings to which information is added or registered in setting information that describes a target character string to which the word information is to be added in Japanese text and a method of specifying the word information to be given are prescribed in advance. A word that satisfies the specified word information is selected while giving priority to the part-of-speech chain of the grammar rules. For the word information that satisfies the specified word information, and for the unspecified word information, the word having the most probable information is recognized as the first solution, and the word given the word information is registered in the setting information according to the specification. A recording medium storing a morphological analysis program for causing a computer to execute a word selection process for outputting a word word information sequence. 6. The word selecting process includes the steps of, when a character string in which word information is once specified in an input Japanese text and a similar specification is performed in subsequent texts, the information is saved and specified. 6. The recording medium according to claim 5, wherein when canceling, the information is deleted. 7. The word selection process includes: a first procedure for assigning priorities to the word chain candidate strings, and setting a chain having the highest priority as a first solution; A second procedure for determining whether or not there is a target character string that is a character string to which information is given or a word registered in the registered word information of the setting information; A third step of determining whether a word that exactly matches the character string exists in the part-of-speech chain with the highest priority; and if there is no word that completely matches the target character string,
A fourth procedure for determining whether or not a word whose word boundary matches the target character string and which matches the part of speech when there is a part of speech designation is present in the word chain candidate string; and If there is a word whose boundary matches, if a part of speech is specified for the target character string, if there is a word that matches the part of speech and satisfies other specified word information, the word is selected. If the other specified word information is different, duplicate the word including word chain, overwrite the other specified word information, and if no part of speech is specified for the target character string The fifth step is to copy the word of the highest priority word chain in the fourth step, including the part of speech chain, and to overwrite other specified word information. If the part of speech is specified, the target word Among the part-of-speech chain with a boundary as a word boundary,
The part-of-speech chain before and after the highest-priority chain that can be chained to the part-of-speech specified at that boundary is copied, and the target word string fills in the word information specified as one word. If the original part-of-speech chain word (if there are multiple words, the word with the highest priority) is generated, and if there is no chainable word, a chain with an unknown word is generated. If not specified, the part-of-speech chain with the boundary of the target word string as the word boundary
Copy the word sequence with the highest priority, and set the target word string to 1
A sixth method in which word information specified as a word is filled, and word information not specified is generated from information of the original word.
6. The recording medium according to claim 5, further comprising: a procedure for increasing the priority of the words handled in the third, fifth, and sixth steps, and selecting the word in the first place. 8. The recording medium according to claim 7, wherein said word selection processing further includes a step of registering a word for which word information is designated in said registered word information when word storage information of said setting information is on. .