JPS6366672A - Unknown word processing system for morpheme analysis of mixture of kanji and kana - Google Patents

Abstract

PURPOSE:To accurately detect the unknown words by using a paragraph end propriety table showing whether words can be put at the end of paragraphs or not. CONSTITUTION:A word extraction processing part 2 uses a word dictionary 2a, a part-of-speech sorting table 2b, a conjugation inflection table 2c, a connection weight matrix table 2d and a paragraph end propriety table 2e to produce a candidate word list. A word selection processing part 3 adds the candidate words set right before the word list to this list as long as said list is not empty. If the word list is empty, a back track is produced. Then the unknown word processing is carried out in case no answer is obtained even after production of said back track. When two or three characters counted from the head are equal to KANJI (Chinese characters) and a postpositional word is detected right after the KANJI part, this KANJI part is entirely defined as unknown words. If no postpositional word is detected, a dictionary is retrieved at and after the back of the KANJI part and a part preceding a success area is defined as an unknown word.

Description

[Detailed Description of the Invention] The present invention relates to an unknown word processing method in morphological analysis of sentences containing kanji and kana, and is applicable to Japanese-English machine translation, OCR (optical character reader), voice word processing, etc. It is something.

Regarding the Japanese morphological analysis system, which performs morphological analysis of the main language sentence, uses a dictionary system to obtain the information necessary for Japanese syntactic analysis as the analysis result, and outputs it to the syntactic analysis system, please contact the Agency of Industrial Science and Technology. Although it has already been proposed by the Electronics Technology Research Institute (research on a breaking system for Japanese and British scientific and technical literature, Japanese morphological analysis system manual (Syr
nboljcs3600 version) Version, 1 January 1987), this system is a dictionary I-driven system that checks the connection relationships between character strings from the left end of the sentence, and outputs all possible solutions for each sentence as a unit.
A path method is used, and all characters from the first character of the unknown word part to the character immediately before the character type change are treated as unknown words.

Therefore, in morphological analysis of sentences containing kanji and kana, it is impossible to register all words in the morpheme M analysis dictionary V (word dictionary), and the occurrence of unknown words (unregistered words) is inevitable. In this case, it is important to accurately find unknown words, but the above system treats everything from the first character of the unknown word part to the character immediately before the character type change as unknown words, and it is difficult to find unknown words just by character type. Since the unknown words are being judged, it is not possible to accurately find the unknown words. For example, if the word "currently" is an unregistered word, and the input sentence is "currently speaking." In the word processing system, even ``present talk'' becomes an unknown word, and the unknown word ``present'' cannot be accurately identified.

Purpose The present invention was made in view of the above-mentioned circumstances.
In the above example, only "present" is found as an unknown word, and the purpose is to accurately find unknown words.

In order to achieve the above-mentioned object, the present invention is characterized in that it uses a phrase-unavailable table that satisfies the condition that unknown words are only independent words, and that unknown word processing is changed depending on the conditions. .

Hereinafter, the present invention will be explained based on examples.

FIG. 1 is a block diagram of morphological analysis processing to which the unknown opening determination method according to the present invention is applied. In the figure, 1 is a character string generation unit to be analyzed, 2 is a word extraction unit, and 3 is a word selection unit. , 4 is a confirmation processing unit, and the analysis target character string creation processing unit 1 removes the character string up to the word break selected by the word selection processing unit 3 from the input sentence, and sets the remaining character string as the “analysis target character string”. The word extraction processing unit 2 includes a morphological analysis dictionary (word dictionary) 2a, a part-of-speech classification table 2b, a conjugation ending table 2c, and a connection weight matrix table 2d.

A ``candidate word list'' is created using the ``candidate word list'' 20. If the ``candidate word list'' is not empty, the word selection processing unit 3 selects the immediately preceding word of the list as a ``word list.'' Pile it on. If it is empty, a backtrack is created, and if a solution is still not found, unknown word processing is performed. next.

An evaluation value is determined for each word in the "candidate word list" using an evaluation formula, and the word with the highest evaluation value is selected as the first candidate. The confirmation processing unit 4 creates a "confirmed word list" from the "word list".

FIG. 2 is a flowchart for explaining an example of unknown word processing according to the present invention, and a specific method thereof will be explained below.

i) Word extraction processing 3 yields a head pointer P that includes an unknown word, but at this time, the word before the pointer P is always the word that ends in a clause (use of clause availability table).

ii) Perform the following processing on the character string after P in the input character string.

(b) When the second to third characters from the beginning are kanji, if a particle is detected immediately after the kanji part, the entire kanji part is treated as an unknown word.

If it is not a particle, a dictionary search is performed from the end of the kanji part, and if successful, the previous part is treated as an unknown word.

(Example 1) If the character string after P is "Tokyo is big" (assuming that there is no "PaTokyo" in the dictionary), "is" is detected as a particle, the unknown word becomes "Tokyo", and this In this way, unknown words can be detected accurately.

(Example 2) If the words after P are "I ate yesterday" (assuming there is no 1'yesterday in the dictionary), the particle immediately after the kanji part cannot be detected, and the dictionary search is performed from the end of the kanji part. As a result, ``I ate'' can be detected, and the unknown word becomes ``yesterday,'' making it possible to accurately detect the unknown word.

(b) If there are four or more consecutive kanji characters from the beginning, first skip two characters from the beginning and perform a dictionary search, and if successful, use the skipped characters as an unknown word. Thereafter, the dictionary search is performed by skipping one character at a time until the dictionary search is successful.

(Example 3) The text after P is “research on artificial language” (“artificial” in the dictionary).
), the first two characters are skipped and a dictionary search is performed starting with "word". As a result, the "language" is detected, the unknown word becomes "artificial", and the unknown word can be detected accurately.

(c) When the character type of the first character is katakana or a number, the part up to where the different character type appears is considered an unknown word.

(Example 4) If the characters after P are "program writing" (assuming "program" is not registered in the dictionary), then the katakana sequence ゛'
``Program'' becomes an unknown word and can be detected accurately.

Effects As is clear from the above explanation, according to the present invention, as shown in Examples 1 to 4 above, words that are not registered in the dictionary and are not numbered 9 can be accurately found.

[Brief explanation of the drawing]

FIG. 1 is a block diagram of a morphological analysis process to which the present invention is applied, and FIG. 2 is a flowchart for explaining an example of an unknown part determination method according to the present invention. 1... Analysis target character string creation section, 2... Word extraction section,
3... Il1 word selection section, 4... Confirmation processing section /f
Fi No. 2 (Import of amendment to figure procedure) December 17, 1985 Commissioner of the Japan Patent Office Kuro 1) Relationship with the case of the person making the amendment Patent applicant Ota Ri Nakamagome Address 1 Nakamagome, Ota-ku, Tokyo Chome 3-6 representative
Hama 1) Hiro 5, date of amendment order ρ..., No' 7, content of amendment (1), and the scope of claims of the specification are amended as shown in the attached sheet. (2), “
``Search using the dictionary system,'' is corrected to ``Search using the dictionary system,''. (3) The "Bunsetsu Unavailability Table" written in the 4th line of page 4 is amended to the "Bunsetsu End Availability Table." (4) Create r-backtrack as described in the first line of page 5, and correct "by performing r-backtracking" to ". (5), "Next" written in the second line of page 5 is replaced with "This unknown word processing will be explained later. If the "candidate word list" is not empty, then ,” is corrected. (6), "(Use of clause-end availability table)" written on page 5, line 13 is corrected to "(Use of clause-end availability table)." Claim (1): An unknown word processing method in morphological analysis of kanji and kana mixtures, characterized in that unknown word processing is changed depending on conditions using a clause-finality table that indicates whether a word can appear at the end of a clause. (2) The unknown word processing method in morphological analysis containing kanji and kana as set forth in claim (1), wherein the condition is that the first two to three characters of the unknown word part are kanji. (3) The unknown word processing method in morphological analysis containing kanji and kana as set forth in claim (1), wherein the condition is that the first four or more characters of the unknown word part are kanji. (4) The condition is that the first character type of the unknown word portion is katakana or a number.
Unknown word processing method in morphological analysis of kanji and kana mixtures described in section 1).

Claims (4)

[Claims] (1) An unknown word processing method in morphological analysis of kanji and kana mixtures, which is characterized by changing the unknown word processing depending on the conditions, using a clause impossibility table that satisfies the condition that unknown words are only independent words. (2) The unknown word processing method in morphological analysis containing kanji and kana as set forth in claim (1), wherein the condition is that the first two to three characters of the unknown word part are kanji. (3) The unknown word processing method in morphological analysis containing kanji and kana as set forth in claim (1), wherein the condition is that the first four or more characters of the unknown word part are kanji. (4) The condition is that the first character type of the unknown word portion is katakana or a number.
Unknown word processing method in morphological analysis of kanji and kana mixtures described in section 1).