JP2794998B2 - Morphological analyzer and phrase dictionary generator - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analysis device for dividing a Japanese sentence into word strings and a phrase dictionary creating device for creating a phrase dictionary used in the device.

[0002]

2. Description of the Related Art There are roughly two types of morphological analysis methods for Japanese sentences: a method using a grammar and a method using an empirical rule.

A method using a grammar has a high correct answer rate so that it is safe to perform a full solution search. However, there is a problem that the production cost is large and the execution speed is slow. Safety is
It means that the correct answer is always included in the multiple output solutions.

The method using the rule of thumb has the advantages of low production cost and high execution speed, but is not perfect.
That is, there is a problem that the accuracy rate is low.

Methods using the rule of thumb include the minimum number of clauses method,
There are the longest matching method and the character type cutting method. The execution speed is the slowest in the number of clauses method, the longest match method is intermediate, and the character type cutting method is the fastest. Therefore, the character type cutting method is the method with the highest execution speed among the various methods of morphological analysis. On the other hand, the character type segmentation method has the worst answer rate, the longest match method is intermediate, and the minimum phrase number method is the best. Therefore, the character type segmentation method has the lowest accuracy rate among the various methods of morphological analysis.

Here, the character type cutting method will be described. FIG. 31 shows Makoto Nagao, Junichi Tsujii, Akira Yamagami, Shuji Takebe: "Storage of Japanese Language Dictionary and Automatic Division of Japanese Sentence", Information Processing, Vol.
19, no. 6, pp. 514-521 (1978) is a diagram showing a conventional morphological analyzer based on the character type segmentation method, in which 1 is an input unit, 2 is a Japanese sentence, and 3 is a Japanese sentence.
Is a phrase division unit by the character type cutting method, 4 is a phrase sequence, 10 is a word division unit by dictionary lookup, 11 is a word dictionary, 12 is a word sequence, and 13 is an output unit.

Next, the operation will be described. The Japanese sentence 2 created by the input unit 1 is input to the phrase division unit 3. The phrase division unit 3 divides the Japanese sentence 2 into phrase strings 4 in accordance with the rule of typology of character type. The word division unit 10 divides the phrase string 4 into word strings 12 by dictionary lookup. And the output unit 13
Outputs the word string 12 to a storage device or a display device.

Next, the empirical rule for character type separation used in the phrase division section 3 will be described. FIG. 32 is a diagram showing the operation of the phrase division unit 3. Punctuation, hiragana,
And three other character types, before and after punctuation,
And split at the point of change from Hiragana to other character types. This is output as a phrase string.

Finally, the word division section 12 will be described. There are several word division methods. The method of Nagao et al. Takes a method of first dividing words into words at the division points of the phrase string 4 divided by the phrase division unit 3. And
Only when a possible interpretation cannot be obtained, re-phrase division is performed assuming that the division of the phrase string 4 is incorrect.

[0010]

Since the conventional apparatus is configured as described above, there is a problem that the output word string has a low accuracy rate.

For example, FIG. 33 shows an example in which the phrase division section 3 outputs an incorrect phrase sequence. In the figure, “:” is a phrase division symbol, and indicates a phrase division point divided by the phrase division unit 3. As shown in this example, a word lined up with "Kanji + Hiragana + Kanji + Hiragana" causes excessive segmentation in the segment segmentation unit 3 which is divided into two segments. However, if the word division in each "kanji + hiragana" succeeds, it is not analyzed as a word. In the example of a phrase string, "Write: change:"
"Writing:" can be interpreted as a verb "writing", and "changing:" can be interpreted as a terminating or continuous form of the verb "changing", so it is not interpreted as one word "rewrite".

Also, in the example of a phrase string of error division, "Experiment: return:", "experiment ::" is a noun "experiment" +
Since the case particle "wo" + the verb "kuru" can be interpreted as the conjunctive form "return:", the verb "return" can be interpreted as an end form or union form, so it is not interpreted as a single word "return".

[0013] The last phrasal string example of "Last year's Japan:" is also correct by the method of Nagao et al.
Japanese: "and the phrase segmentation error can be corrected.

The present invention has been made to solve the above problems, and has as its object to provide a morphological analyzer having a high execution speed and a high accuracy rate.

[0015]

SUMMARY OF THE INVENTION The morphological analysis apparatus according to the first invention, (a) an input unit for inputting a Japanese sentence, phrase division unit that divides a Japanese sentence input (b) in clause column, ( c) a word dictionary for storing words, (d) inputting words in the word dictionary and dividing them into phrase strings,
Select the word to be registered from the words in the phrase
A sentence that is stored as a word that causes a division error in the phrase segmentation unit
Correction of error division in phrase segmentation section using clause dictionary, (e) phrase dictionary
Clause connecting portion for word processor having a tokenizer unit for correcting an error division of the word division unit by using the word dividing unit and the word dictionary is divided into word strings to (f) clause column, the (g) word sequence It has the components of a storage device or an output unit for outputting to a display device. In particular, by having a phrase linking unit, it is possible to correct a division error of the phrase division unit.

A phrase dictionary creating apparatus according to a second aspect of the present invention is capable of automatically creating a phrase dictionary used in the phrase connecting unit of the first aspect from a word dictionary. The words extracted from the words in the dictionary, in which the word itself is erroneously divided into two or more phrases by the phrase division unit, are extracted by the registered word selection unit. According to experiments, the number of words in the phrase dictionary was less than 3% of the number of words in the word dictionary.

[0017]

The phrase linking unit in the first invention compares the character string of the phrase dictionary with the phrase string, and deletes the phrase division point when a character string registered in the phrase dictionary is found.
This eliminates erroneous segmentation. Since the phrase dictionary has a word count of less than 3% of the word dictionary and is very small, the execution speed of the entire morphological analyzer does not substantially decrease even if the phrase connection unit is added.

A phrase dictionary creating apparatus according to a second aspect of the present invention is an apparatus for automatically creating a phrase dictionary from a word dictionary. The phrase dictionary creation device inputs each word of the word dictionary to the phrase division unit according to the first invention, extracts words whose words themselves are phrase-divided incorrectly, and registers the words in the phrase dictionary.

[0019]

【Example】

Embodiment 1 FIG. An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a morphological analyzer according to the first invention. In the figure, 1 is an input unit, 2 is a Japanese sentence, 3 is a phrase division unit by the character type cutting method, and 4 is a phrase division unit. A phrase string, 5 is a phrase connection part, 6 is a phrase dictionary, 7 is a phrase string output by the phrase connection part, 8 is a word division part by the character type cutting method, and 9 is a word output by the word division part by the character type separation method Column 10 is a word recognition unit by dictionary lookup, 11 is a word dictionary, 12 is a word string output by the word recognition unit by dictionary lookup, and 13 is an output unit.

Next, the general operation of the morphological analyzer according to the first invention will be described. The Japanese sentence 2 created by the input unit 1 is input to the phrase division unit 3. The phrase division unit 3 divides the Japanese sentence 2 into phrase strings 4 in accordance with the rule of typology of character type. The phrase linking unit 5 deletes error division points of the phrase sequence 4 using the phrase dictionary 6 and outputs a phrase sequence 7. The word division unit 8 divides the phrase string 7 into word strings 9 in accordance with the rule of typology. The word recognition unit 10 corrects an error division point of the word string 9 using the word dictionary 11, and outputs the word string 12. Then, the output unit 13 outputs the word string 12 to a storage device or a display device.

Next, the algorithm of the morphological analyzer will be described in detail for each of the sections 3, 5, 8, and 10.

The morphological analysis algorithm is based on
The text is divided into word strings by sequentially performing the following four steps corresponding to steps 8 and 10. Furthermore, the part of speech is added, and the adjunct form (= standard form) of the inflected word is added. In the following (1) and (3), a process based on an empirical rule is performed. On the other hand, in (2) and (4), the exceptions (1) and (3) are corrected using a dictionary.

(1) Clause division unit (by rule of thumb) A text is divided into clause units using rule of thumb. The rules of thumb are shown below. [1] Divide where Hiragana changes to Kanji. [2] Divide where Hiragana changes to Katakana. [3] Divide where hiragana changes to alphanumeric. [4] Divide where hiragana changes into a symbol. [5] Divide before and after punctuation. An example of a result based on the above empirical rules will be described. : Indicated by
This is a segmentation point. It can be seen that most of the clauses are correctly segmented, but the "embedding: by embedding:" part is incorrectly divided into two clauses. The above rule of thumb using the character types will always give such incorrect results. This is the incompleteness of the rule of thumb, and the biggest difficulty of the Japanese morphological analysis algorithm using the character type rule of thumb. "Substrate: opposite conductivity type: semiconductor layer: part: second
Of: groove: first: from the groove: deeper: digging:,: second:
By: Growing: Insulating film: Growing:,: Second: Groove: Semiconductor layer: Growing: Filling: Embedding:
Integration: Improve: Aim: :] However, the rule of thumb is that it does not fail for any text, and the original text can be reproduced by removing the segmentation point :.

(2) Clause Linking Unit (by Clause Dictionary) A clause dictionary is used to correct the exception of the above clause division (1). For example, in segmentation, a word containing both hiragana and kanji is cut off in the middle of the word as follows. Such words are stored in the phrase dictionary, and excessive cutting is restored. Embedding: Layer Embedding: Metal Embedding: Embedding Yield: Performance Thrust: Exiting Reading: Emerging Examples of results of phrase concatenation are shown. It can be seen that the “embedded by:” part was changed to two phrases in (1), but was correctly corrected to one phrase. This solves the problem of "over-cutting", and guarantees that the morphological analysis algorithm is complete with respect to segmentation. "Substrate: opposite conductivity type: semiconductor layer: part: second
Of: groove: first: from the groove: deeper: digging:,: second:
In the groove: on the side wall: growing the insulating film :, in the second: in the groove: growing the semiconductor layer: by embedding: embedding: improving the integration: aiming at: : "However, though not related to completeness, << Overcut >> may remain. That is when the phrase dictionary is not complete. However, phrase dictionaries are data, and data integrity is usually ignored when discussing algorithm integrity. Therefore, this problem does not impair the completeness of the Japanese morphological analysis algorithm at all. It has nothing to do with completeness, but the problem of "cutting too much" remains. This is a problem that, although it should originally be two or more phrases, it is analyzed as one phrase. This will be covered in a later process. Therefore, completeness is guaranteed even if there is a problem of "too short". This is because the definition of completeness is that there is no irrecoverable error caused by the algorithm. This process is characterized in that it does not fail for any text and that the original text can be reproduced by deleting the segmentation point :.

(3) Word segmentation unit (by rule of thumb) Using only the rule of thumb, the word is further divided into word units. The rules of thumb are shown below. [1] Divide where kanji changes to hiragana. [2] Divide where katakana changes to hiragana. [3] Divide where alphanumeric characters change to hiragana. [4] Divide where the symbol changes to Hiragana. An example of a result based on the above empirical rules will be described. / Is a word division symbol, and the portion indicated by / is a word division point. It can be seen that most of the words are correctly word-divided, but the part of "by embedding / embedding / by:" is incorrectly three words. The above rule of thumb using the character types will always give such incorrect results. As in (1), this is also an imperfect rule of thumb and a drawback of the Japanese morphological analysis algorithm using the character type rule of thumb. “Substrate / and: opposite conductivity type / of: semiconductor layer / of: part /
Of: 2 / of: groove /: of 1 / of: groove / more: deep / deep:
Digging / Digging: 、 Second / No: Groove / No: Sidewall / On: Insulating film /
: Growth / development,: second / of: groove / to: semiconductor layer /
By: growing / growing: by embedding / embedding / embedding: integration /
: Improvement /: Planning: :] However, this process does not fail for any text, and the original text can be reproduced by deleting the phrase division point: and the word division point /.

(4) Word recognition section (by word dictionary) Using the word dictionary, the result of the word division in the above (3) is checked and errors are corrected. In addition, attach the part of speech. (Part of speech) is added to words registered in the word dictionary, and <part of speech> is added to unknown words. Further, in the case of a conjugate word, a continuous form (= standard form) is added as [continuous form]. The results are shown below. Here, one word is represented by one line for easy viewing. Although the part indicated by * was three words in (3), it can be seen that it was correctly corrected to two words. with this,
The problem of "overcutting" has been solved, and this Japanese morphological analysis algorithm is guaranteed to be complete with respect to word segmentation. (Noun) substrate / <attached> and: (noun) opposite conductivity type / <attached>: (noun) semiconductor layer / <attached>: (noun) part / <attached>: (noun) second / <Attachment>: (Noun) Groove / <Attachment>: (Noun) 1 / <Attachment>: (Noun) Groove / <Attachment> More: (Adjective) Deep [Deep]: (Verb) Digging [ (Dig): (reading point),: (noun) 2nd / <attachment>: (noun) groove / <attachment>: (noun) sidewall / <attachment>: (noun) insulating film / <attachment> : (Verb) growth [grow] / <attached>: (reading point),: (noun) second / <attached>: (noun) groove / <attached>: (noun) semiconductor layer / <attached >: (Verb) Growth [grow] / <attached>: (verb) Embed [embed] / <attached> By: (noun) integrated / <attached>: (noun Improvement / a <Included>: (verb) reduce [FIG. Ru]: (punctuation). : However, it has nothing to do with completeness, but sometimes "overcutting" remains. This is a problem that, although it should originally be one word, it is analyzed as two or more words. It depends on the completeness of the word dictionary. However, word dictionaries are data, and data integrity is usually ignored when discussing algorithm integrity. Therefore, this problem does not impair the completeness of the Japanese morphological analysis algorithm at all. Similarly, << Insufficient Cutting >> may remain, but does not impair the completeness of the Japanese morphological analysis algorithm. This process does not fail for any text, and the original text can be reproduced by deleting the segmentation points: word division point /, (part of speech), <part of speech> and [continuous form]. .

If no word is found, the word before the word segmentation point is output as an independent word and the word after it is output as an adjunct. Perform unknown word processing with inference.

Unknown word processing (by empirical rule) Unknown word processing is processing for inferring the part of speech, inflection, and union form of a word that has not been registered in the word dictionary. Currently 1
I'm thinking of 81 rules, but that's not all. Details of the unknown word inference rule are shown in FIGS. In the figure, "Kankan", "Kan" and ":" have the following meanings. Kanhan = one or more non-hiragana characters kan = one-character non-hiragana: = phrase segmentation symbol In the figure, "inference conditions" and "inference results"
When there is a word "Kan-Kan-Sen", this is used as an inference condition to derive an inference result of "Kan-Kan-San". For example, the word "execute" will be inferred to the word "execute".

Next, the phrase dictionary 6 used for morphological analysis,
And the word dictionary 11 will be described. (1) Clause dictionary This is a dictionary for clause connection. The format is one word per line. The phrase dictionary can be automatically created from the word dictionary by a program based on the second invention described later, so that development work is unnecessary. An example is shown below. Embedding: Layer Embedding: Metal Embedding: Embedding Yield: Performance Thrust: Exit Reading: Eject

(2) Word dictionary This is a dictionary for word division. The format is one word per line. For inflected words, the stem and the end are separated by a single-byte space. An example is shown below. Semiconductors Semiconductor wafers Semiconductor wafers Semiconductor wafers Semiconductors Inexpensive Add Strong Strong (adverb) Threshold voltage (Particles) Penetrate Embed Embed metal In conventional word dictionaries, in addition to head words, , Utilization, forward connection information, backward connection information, etc., need to be added to all words. This word dictionary has almost no such additional information, and is in a format that allows the user to easily register words. That is, the word dictionary is composed of a headword and some information.
B / very small word. Nouns are registered as headwords only, and all are considered to be changed. Verbs, adjectives and adjective verbs are registered in adnominal form, and stems and endings are separated. For other parts of speech, "(part of speech)" is described.
Since the amount of description is small, it is easy to create a dictionary even if one is not an expert in natural language processing.

Next, the specification for registration in the word dictionary will be described. First, general specifications will be described.・ Use one word per line. Do not register two or more words on one line, or register one word over two or more lines.・ Use only full-width characters. Half-width characters must not be used.・ Use only double-byte spaces. Do not substitute two single-byte spaces. Next, the registration specifications for each part of speech are described in FIGS.

Embodiment 2 FIG. As described above, in the first embodiment, each unit 3
5, 8 and 10 and the respective dictionaries 6 and 11 have been described. Next, the features of the present invention will be described using examples.

The phrase dictionary 6 will be described with reference to FIG. FIG. 17 shows an example of the contents of a phrase dictionary. Of the words registered in the word dictionary, words that are erroneously divided into two or more phrases by the phrase division unit 3 are registered in the form of phrase strings. In the example, "" Melting: inclusive "," represents the stem of the verb "melting in."

Next, the phrase connection unit will be described. FIG.
FIG. 8 is a diagram showing the operation of the phrase connection unit 5. Clause 4 "Wrong: segmentation point: write: rewrite:,: correct: create phrase sequence: make:"": write: replace:" is correctly ":
Rewrite: "but incorrectly over-split. The operation starts by searching for the first segmentation point of the phrase sequence in (1). In the example, a phrase division point of "wrong:" is found. Next, the first word of the phrase dictionary 6 is extracted in (2). Then, in (3), the character strings are compared to determine whether the phrase string matches the word. This is performed for all words in the phrase dictionary 6, but none of them match. Therefore, it turns out that this is not a false division. Next, the next segmentation point is searched in (7). In the example, a phrase division point of "A phrase division point:" is found. This also does not match any of the words in the phrase dictionary 6. Therefore, it turns out that this is not an erroneous division. Further, the next segmentation point is searched in (7). In the example, the segmentation point of “writing:” is found. On the other hand, when the character strings are sequentially compared with the words in the phrase dictionary 6, ""
Rewrite: Matches with ",". String comparison is phrase string 4
And the first segmentation point of a word in the phrase dictionary 6 are aligned, and a partial character string equal to the word length is matched.
In the case of "rewrite: change", the phrase division point is deleted because they match. Thereafter, the processing is sequentially repeated for the segment division points, but the erroneous division ends without being found any more. The phrase sequence 7 to be output becomes a correct phrase sequence as follows: "Wrong: Rewrite the phrase segmentation point :::: Correct: Create phrase sequence ::".

Next, a word division unit based on the character type cutting method will be described. FIG. 19 is a diagram showing the operation of the word division unit 8. Each character in the phrase string 7 is classified into hiragana, a phrase division point, and three other character types, and is divided at a change point from the other character type to hiragana. This is output as a word string 9. The segmentation point remains unchanged.

Next, a word recognition unit based on dictionary lookup will be described. Several methods are conceivable as a method of realizing this part. In this embodiment, the realization using the longest match method will be described. FIG. 20 is a diagram showing the operation of the word recognition unit 10. A character string is extracted in the order of a phrase from the front of the word string 9 and words are determined by the longest match method. If the unparsed string is only a hiragana string, it is output as an adjunct. The division by the longest match method is prioritized over the word segmentation point. If no word is found, the word before the word segmentation point is output as an independent word and the word after the word segmentation point is output as an adjunct word. If you find a dictionary word and a conjugation word,
Perform inflection ending processing. In addition, unknown word processing involving part-of-speech inference is performed on unknown words that are independent words. As described above, the word string 12 in which the words are correctly divided and to which the parts of speech and the utilization information are added is completed.

Embodiment 3 FIG. The word division unit 8 and the word recognition unit 10 may be combined into one word processing unit. In this case, the correct answer rate and the execution speed do not particularly increase or decrease.

Embodiment 4 FIG. The word recognition unit 10 uses the longest match method in the first embodiment, but there is also a method using a grammar. In this case, the accuracy rate is further improved, but the execution speed is reduced.

Embodiment 5 FIG. There is also a method in which the grammar is used by the word recognition unit 10 only for the processing of hiragana columns, like Nagao et al. In this case, the accuracy rate of the attached word string is improved, but the execution speed is slightly reduced.

Embodiment 6 FIG. There is also a method in which the word recognition unit 10 takes into account the characteristics of the kanji string as in Nagao et al. In this case, the correct answer rate for a kanji string of an unknown word is particularly improved.

Embodiment 7 FIG. Next, a phrase dictionary creation device according to a second invention will be described. FIG. 21 is a diagram showing a phrase dictionary creation device. In the figure, 11 is a word dictionary, 14 is a word in the word dictionary, 3 is a phrase division unit by the character type cutting method, 4
Is a phrase string obtained by dividing words into phrases, 15 is a registered word selection unit,
16 is a phrase string of words to be registered, and 6 is a phrase dictionary.

Next, the operation will be described. Word dictionary 11
Each word 14 is taken out of the phrase, and is formed into a phrase string 4 by the phrase division unit 3. The registered word selecting unit 15 selects a phrase sequence 16 of words that are to be phrase-divided in the middle of the word. This is registered in the phrase dictionary 6. FIG. 22 shows a specific example in which a phrase dictionary is created from a word dictionary.
Shows a case where the data is divided into “threshold: value” and a phrase string, and is selected by the registered word selecting unit 15 and registered in the phrase dictionary 6. Although not shown, for example, words such as “substrate”, “desk”, and “name” remain as “substrate”, “desk”, and “name” even if they are formed into a phrase sequence by the phrase division unit 3. Words in the word dictionary having the same phrase string generated by the phrase division unit 3 are not selected by the registered word selection unit 15 and are not registered in the phrase dictionary 6.

Embodiment 8 FIG. In the above embodiment, the phrase dictionary 6 registers a phrase string, but a distinction between an independent word and an adjunct word may be registered in addition to the phrase string. Hereinafter, an embodiment in which the distinction between the independent word and the attached word is registered and used in association with the flags 0 and 1 will be described. As shown in FIG. 23, in this embodiment, a phrase dictionary is provided with a flag (0/1) of an independent word or an attached word. Since the words in the word dictionary describe the parts of speech of nouns, verbs, adjectives, and parts of speech other than adjective verbs, it can be determined whether the words are independent words or adjunct words. For example, in FIG. 23, since there is a description that the word “related” is a “particle”, this is determined as an adjunct and a flag 1 is set. Since other words are independent words such as nouns and verbs, the flag 0 is set.

Next, the flag (0 /
A method of using the phrase dictionary 6 having 1) will be described. For example, it is assumed that the phrase division result by the phrase division unit 3 is as shown in FIG. The phrase linking unit 5 corrects the error of the phrase division by using the phrase dictionary 6. If there is an error division, if it is an independent word, the division point in the word is deleted and the phrase division is performed before the word. If there is no point, add it. In the case of an accessory word, the division point in the word is deleted. FIG. 25 shows an example of the phrase recognition result. Although “embedment: embedding” has two phrases in FIG. 24, it is correctly corrected to one phrase. In addition, "threshold: voltage" is a correct phrase segmentation, and at the same time, is a self-sufficient word with flag = 0 according to FIG. 23, and is added because there is no division point before the word. ].
"Prevention of fluctuation" remains as it is. In this way, clause recognition is
Excessive division of a known word and an incorrect division position can be corrected to 100%, and the correct answer rate is improved. In the case of an independent word, a segmentation dividing point is forcibly inserted before the independent word to eliminate insufficient segmentation.

FIGS. 26 and 27 correspond to FIGS. 24 and 25, respectively.
7 shows the results of the word division unit 8 and the word recognition unit 10 in the example of FIG. Word recognition is performed by searching a word dictionary in a phrase by the longest match method. The word dictionary is loaded on the main memory in advance, and a binary search is performed on the main memory. However, since the attached words are unlikely to be important keywords, word recognition is not performed. Next, a part of speech is given. Words registered in the word dictionary are given "(part of speech)", and in the case of inflected words, continuous form (= standard form)
Is added in the form of [[contiguous]]. Unknown words are given any one of "<noun>, <verb>, <adjective>, <adjective>, <attachment>". In FIG. 27, "buried / embedded / n"
Is three words in FIG. 26, but becomes one word correctly. Also, "variation prevention" is recognized as two words.

Embodiment 9 FIG. In FIG. 18, when the phrase connection unit searches for a phrase string in the phrase dictionary, an algorithm is used in which loops (3) to (6) are performed and the phrase string and each word are compared one by one. At the time of creating a dictionary, as shown in FIG. 22, it is desirable to arrange phrase strings in the reverse order of the character codes of the word strings employed in the word dictionary. For example, consider a case where a phrase dictionary is generated by arranging phrase strings in the same order as character codes in the same order as a word dictionary as follows. Filling: Embedding Filling: Embedding: Metal Filling: Embedding: Structure Embedding: Embedding: Area If you search for the word "embedding structure" in this phrase dictionary, there is a phrase string "embedding structure" in the third line. Nevertheless, the result matches "embedding" in the first row, and the result "embedding: structure" is obtained. Next, let us consider a case where this is reversed in the following character code order. Filling: Embedding: Area Filling: Embedding: Structure Filling: Embedding: Metal Filling: Embedding In this case, searching for “embedded structure” will match on the second line, and will be correctly identified as one phrase column .

Embodiment 10 FIG. In the ninth embodiment, the phrase dictionary in which the phrase strings are registered in the reverse order of the character codes is described. However, the phrase dictionary registered in the normal order may be searched in the reverse order.

Experimental results. A program was created on a workstation and experiments were performed. FIG. 28 shows the hardware configuration and software configuration, and FIG.
FIG. 30 shows the morphological analysis time. As shown in FIG. 29, the dictionary load time linearly increases in proportion to the number of word dictionary words. CPU time is about 5 seconds for 80,000 words,
In the case of mass document processing, its proportion in the whole can be ignored.
Also, as shown in FIG. 30, the morphological analysis time is proportional to the logarithm of the number of word dictionary words. This is because a binary search is used. The analysis of 100,000 characters using an 80,000-word dictionary is just over 12 seconds, which is extremely fast.

In the above embodiment, a method has been described in which the character type cutting method is separated into phrase divisions, and each result is recognized by the dictionary. We believe that a practical method was obtained by a relatively simple method, and the characteristics are summarized in the following five points. (1) High-speed (2) Correction of segmentation errors with a small dictionary (3) Success rate 100% (always output a solution) (4) Easy creation of word dictionary for users (5) Mixed kana-kanji Dedicated to Japanese sentences (based on character type cutting)

[0050]

As described above, according to the first aspect, a morphological analyzer having a high execution speed and a high accuracy rate can be obtained. Further, according to the second aspect, the phrase dictionary used therein can be automatically created by the phrase dictionary creating device.

[Brief description of the drawings]

FIG. 1 is a diagram showing a morphological analyzer according to a first invention.

FIG. 2 is a diagram showing inference rules for unknown word processing.

FIG. 3 is a diagram showing inference rules for unknown word processing.

FIG. 4 is a diagram showing inference rules for unknown word processing.

FIG. 5 is a diagram showing inference rules for unknown word processing.

FIG. 6 is a diagram showing an inference rule of unknown word processing.

FIG. 7 is a diagram showing specifications registered in a word dictionary.

FIG. 8 is a diagram showing specifications for registration in a word dictionary.

FIG. 9 is a diagram showing specifications registered in a word dictionary.

FIG. 10 is a diagram showing specifications registered in a word dictionary.

FIG. 11 is a diagram showing registration specifications in a word dictionary.

FIG. 12 is a diagram showing specifications registered in a word dictionary.

FIG. 13 is a diagram showing specifications registered in a word dictionary.

FIG. 14 is a diagram showing specifications registered in a word dictionary.

FIG. 15 is a diagram showing registration specifications in a word dictionary.

FIG. 16 is a diagram showing specifications registered in a word dictionary.

FIG. 17 is a diagram showing an example of contents of a phrase dictionary according to the first invention.

FIG. 18 is a diagram showing the operation of the phrase connection unit according to the first invention.

FIG. 19 is a diagram showing the operation of the word division unit according to the first invention.

FIG. 20 is a diagram showing the operation of the word recognition unit according to the first invention.

FIG. 21 is a diagram showing a phrase dictionary creation device according to a second invention.

FIG. 22 is a diagram showing another apparatus for creating a phrase dictionary according to the second invention.

FIG. 23 is a diagram showing another apparatus for creating a phrase dictionary according to the second invention.

FIG. 24 is a diagram showing an example of a phrase sequence of a phrase division unit according to the first invention.

FIG. 25 is a diagram showing another example of the operation of the phrase connection unit according to the first invention.

FIG. 26 is a diagram showing another example of the operation of the word division unit according to the first invention.

FIG. 27 is a diagram showing another example of the operation of the word recognition unit according to the first invention.

FIG. 28 is a diagram showing a configuration of an experiment.

FIG. 29 is a view showing an experimental result.

FIG. 30 is a view showing an experimental result.

FIG. 31 is a diagram showing a morphological analysis device using a conventional character type cutting method.

FIG. 32 is a diagram illustrating the operation of a phrase division unit using the conventional character type cutting method.

FIG. 33 is a diagram showing an example of a malfunction of a segment division unit by the conventional character type cutting method.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Input part 2 Japanese sentence 3 Phrase division part by character type separation method 4 Phrase string output by phrase division part 5 Phrase connection part 6 Phrase dictionary 7 Phrase string output by phrase connection part 8 Word division part by character type division method Partial example of word processing unit) 9 Word string output by word division unit by character type cutting method 10 Word recognition unit by dictionary lookup (partial example of word processing unit) 11 Word dictionary 12 Words output by word division unit by dictionary lookup Column 13 Output unit 14 Words registered in the word dictionary 15 Registered word selection unit 16 for selecting words to be registered in the phrase dictionary 16 Phrase sequence of words registered in the phrase dictionary

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-1-295369 (JP, A) JP-A-2-28873 (JP, A) (58) Fields investigated (Int.Cl. ⁶ , DB name) G06F 17/20-17/28 JICST file (JOIS)

Claims (2)

(57) [Claims] 1. A morphological analyzer having the following elements: (a) an input unit for inputting a Japanese sentence; (b) a phrase dividing unit for dividing the input Japanese sentence into phrase strings; and (c) storing words. word dictionary to place, divided into clause column by typing the word of (d) the word dictionary,
Select the word to be registered from the words in the phrase
A sentence that is stored as a word that causes a division error in the phrase segmentation unit
Correction of error division in phrase segmentation section using clause dictionary, (e) phrase dictionary
Clause connecting portion for, (f) a word processor for dividing the clause string into word sequence using a word dictionary, an output unit that outputs to the storage device or the display device (g) word sequence. 2. A phrase dictionary creating apparatus that has the following elements and automatically creates a phrase dictionary from a word dictionary: (a) a phrase division unit that inputs words of a word dictionary and divides the words into phrase strings; A registered word selection unit that selects words to be registered in the phrase dictionary from the set words and stores them in the phrase dictionary as words that cause a division error in the phrase division unit.