JP3369127B2 - Morphological analyzer - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morpheme analysis method and apparatus and a storage medium storing a morpheme analysis program, and more particularly to a Japanese morpheme analysis technique for performing word segmentation and part-of-speech addition of Japanese text. The present invention relates to a morpheme analysis method and apparatus for performing high accuracy in unknown word identification and part-of-speech estimation by using the type of characters that make up a word and its change characteristics, and a storage medium storing a morpheme analysis program.

[0002]

2. Description of the Related Art Conventional Japanese morphological analysis techniques are based on heuristic priority rules (longest match method and minimum bunsetsu method),
And the method based on the connection cost (the method with the minimum connection cost) is the mainstream. The method based on the heuristic priority rule ranks the morphological analysis candidates using heuristics such as longest match and minimum clause number. This method has a problem that the basis of ranking is unclear and the analysis accuracy is low. On the other hand, the method based on the connection cost can obtain high analysis accuracy if the connection cost can be set appropriately.
However, there is no method for setting the connection cost, and the cost must be decided by trial and error.

Therefore, in recent years, a method of using a logarithmic value of probability obtained from a statistical language model learned from a large amount of text data as a connection cost is becoming mainstream. As a result, the rationale for the priority of morphological analysis candidates is clarified, and a high degree of accuracy is obtained experimentally (Nagata: "Probabilistic Japanese morphological analysis using forward DP backward A ^* algorithm" information IPSJ Research Report 94-N
L-101-10, pp.73-80, 1994).

Generally, the statistical language model used for Japanese morphological analysis is obtained from another morphological analysis program or the frequency of appearance of words in a Japanese text in which word segmentation and part-of-speech assignment are performed manually. At this time, when a word that does not appear in the learning text appears in the input text, how to assign the probability becomes an important issue.

The phenomenon that a word is unknown is <U
NK> represents the word w _iIs a character string of length k
c₁, ..., c_kProbability that it is an unknown word
P (w_i<UNK>), without loss of generality, unknown
Word is the length k probability P | and (k <UNK>), length k
The unknown word is c₁, ..., c_kProbability P (c₁，
…, C_k| K, can be expressed as the product of <UNK>).

[0006] _{P (w i | <UNK>} = P (c 1, ..., c
_{k | <UNK>) = P} (k | <UNK>) P (c 1,
..., c _k | k, a <UNK>) (1) below, the words the former length probability, called latter as word notation probability. In the case of English, Brown et al. Have proposed an unknown word model of the following equation (Brown et al, "An Estimate of an Upper Bound for
the Entropy of English ", Computational Linguistic
s, Vol.12, No.1, pp31-40, 1992).

[0007]

[Equation 1]

Here, λ is the average word length in the learning text, and p is the reciprocal of the number of characters included in the ASCII character set. That is, in the Brown 92 unknown word model, it is assumed that the word length distribution follows a Poisson distribution with the average word length λ as a parameter, and all characters appear with equal probability. The Brown92 model has a problem of assigning a probability to a word of length 0 and a problem of not reflecting the occurrence distribution of characters. Therefore, a Japanese unknown word model such as the following formula is proposed (Nagata: "Japanese character recognition error correction method using character similarity and statistical language model", IEICE Transactions D-II. , Vol.J81-D-11, No.11, p
p.1-12,1998, (hereinafter referred to as "Nagata 98").

[0009]

[Equation 2]

Here, <bow> and <eow> represent the beginning and end of a word, respectively. That is, the unknown word model of "Nagata 98" solves the problem of assigning the probability of a word of length 0 by moving the lower limit of the word length distribution from 0 to 1, and the word writing probability is set to the character "bigram". By approximating by the product of, the appearance distribution of characters can be reflected.

[0011]

However, in the unknown word model of "Nagata 98" described above, long unknown words, especially transliterated foreign words, are divided into words in the dictionary and other words, and a plurality of unknown words are divided. There is a problem in that over-division easily occurs. For example, if the word "Pennsylvania" is not registered in the dictionary and the word "pen" is registered in the dictionary, the prefix "pen" happens to match a word in the dictionary, so "Pennsylvania" and "Pennsylvania" The phenomenon that it is decomposed into "Sylvania" occurs. This is because the average word length of the entire Japanese word is about 2 characters, whereas the average word length of the foreign words written in katakana is about 4 characters.

Further, the unknown word model of "Nagata 98" is a model for word division (identification of unknown word), and there is a problem that the part of speech of the unknown word is not taken into consideration. this is,
This is because this unknown word model is designed on the assumption that it will be used for error correction in character recognition, but in morphological analysis for speech synthesis and information retrieval, it is necessary to estimate the part of speech of the unknown word. .

The present invention has been made in view of the above points, solves the problem of over-division of unknown words in Japanese morphological analysis using a conventional statistical language model, and further estimates the part of speech of unknown words. It is an object of the present invention to provide a morphological analysis method and device capable of performing, and a storage medium storing a morphological analysis program.

[0014]

FIG. 1 is a diagram for explaining the principle of the present invention. This onset Ming, the morphological analysis method for performing morphological analysis of Japanese, searching for words that match the substring of the input text from the word dictionary in the database to generate a word candidates (step 1), the word dictionary
Unknown character from the substring of the input text that does not match the calligraphy
What is possible is the type of characters that make up the word and the
Word types whose word types are defined based on
Refer to the table definition table and specify any of the word types
Classifies the character strings of and the unknown words for each classified word type
It was identified as a complement (step 2) , and the word type was output for each part of speech.
See word type frequency table where the current frequency is defined
Then, the word type appearance probability is calculated for each part of speech, and the part of speech and word
Average word length table that defines the average word length for each type
Bulls to approximate the average word length by Poisson distribution.
Depending on the part of speech and word type, the word length can be any length
Probability is calculated and character ngram frequency by part of speech and word type is
Refer to the defined character ngram frequency table and
Word table of arbitrary character strings according to words and word types and word lengths
Part-of-speech of an unknown word candidate using an unknown word model for finding probabilities
Estimate another word appearance probability (step 3) and apply it to dynamic programming
To all combinations of word candidates and unknown word candidates
About the word obtained by referring to the word ngram frequency table
Simultaneous probability using ngram probability and word appearance probability by part of speech
The word string that maximizes is obtained (step 4).

[0015] The present onset Ming, at the time of obtaining the word notation probability, statement
Refer to the character ngram frequency table to determine the lower part of speech and
Character ngram frequency of word string and homogeneity or lower
Ngram frequency without considering the part of speech and word difference
, Linearly interpolating character ngram probabilities by part-of-speech and word type
Method, or from character ngram frequency
Divide the given word notation probability into all strings of the same length.
Any of the methods to normalize with the sum of assigned word notation probabilities
Use or .

FIG . 2 is a block diagram showing the principle of the present invention. Starting
Akira is a morphological analyzer that analyzes Japanese morphemes.
To match the substring of the input text with the word
Word dictionary collation hand that searches from a book and generates it as a word candidate
Based on column 1 and the type of characters that make up a word and their variations
The word type definition table that defines the word type
Browse and classify any string of word type
A word type determination hand that determines the classified word type
Level 6 and word type appearance frequency for each part of speech are defined
Refer to the word type frequency table to see the word type by part of speech.
A word type probability estimating means 7 for obtaining the appearance probability, and a word
The average word length for each part of speech and word type is defined.
Poisson distribution of average word length with reference to uniform word length table
By approximating with
A word length probability estimating means 8 for obtaining a word length probability of a length,
Character ngram frequency by lyric and word type is defined
Refer to the character ngram frequency table to identify the part of speech and word
Find the word notation probability of an arbitrary character string by group and word length
An unknown word model consisting of word notation probability estimation means 9;
Using the word type determination means 6 of the unknown word model,
The input collating means 1 does not collate with the word dictionary.
From the substring of the string that may be an unknown word
The unknown word candidate identifying means 2 for selecting the unknown word candidates, non
Word type probability estimator 7 of the word model, word length probability estimation
Unknown word candidates using the definition means 8 and the word notation probability estimation means 9.
Unknown word candidate probability estimation means for estimating the part-of-speech word appearance probability of
3 and the word candidates obtained by the word dictionary matching means 1,
And an unknown word candidate obtained by the unknown word candidate identifying means 2.
The word ngram frequency is defined for all combinations of
Is found by referring to the ngram frequency table
The estimated word ngram probability and unknown word candidate probability estimation means 3
By using the word appearance probability of each part of speech obtained by
And an optimum word string search means 4 for finding the maximum word string.
To do.

The present invention resides in the word notation probability estimation means 9.
And refer to the character ngram frequency table
Character ngram frequency of words and word strings and the same or
Character ngram frequency that does not consider the difference of low-order parts of speech and words
From the degree, the character ngram probability for each part of speech and word type is linearly calculated.
Interpolation method or character ngram frequency
Find the word probabilities found in all strings of the same length
How to normalize by the sum of assigned word notation probabilities
Use the difference .

The present invention is a form for performing morphological analysis of Japanese.
A storage medium storing a prime analysis program, input Te
Words in the database that match substrings of kist
A word dictionary that is searched as a word candidate and generated as a word candidate
Matching steps and input text that does not match the word dictionary
Words that may be unknown words from substrings
Based on the type of characters that make up the word and its variation
Refer to the word type definition table in which the
Classified any string into one of the types and classified
Unknown word candidates identified for each word type as unknown word candidates
Constant steps and word type frequency by part of speech are defined.
Refer to the word type frequency table that has
The type appearance probability is calculated, and the average unit for each part of speech and word type is calculated.
Refer to the average word length table that defines the word length, and
By approximating the uniform word length by Poisson distribution,
And the word length probability of any length for each word type
And sentences with defined character ngram frequencies by word type
Part-of-speech and word type by referring to the character ngram frequency table
And the word notation probability of an arbitrary character string is calculated for each word length.
Probability of word occurrence by part-of-speech of an unknown word candidate by using an intelligent word model
And an unknown word candidate probability estimation step of estimating, dynamic programming
Allows all combinations of word candidates and unknown word candidates
About the word ngram frequency table
The word with the highest joint probability using the word appearance probability
An optimal word string search step for obtaining a string.

The present invention includes a word notation probability estimation step.
And refer to the character ngram frequency table for lower order
Part-of-speech and character string ngram frequency and the same or
Character ngram that does not consider lower part-of-speech and word differences
Character ngram probability by part of speech and word type is plotted from frequency
Shape interpolation method or character ngram frequency
The word notation probability obtained from all strings of the same length
Of the method of normalizing the sum of the word notation probabilities assigned to
Use either one.

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

FIG. 3 shows the configuration of a Japanese morphological analyzer according to the present invention. The Japanese morphological analyzer shown in FIG. 1 includes a word matching unit 1, an unknown word candidate identifying unit 2, an unknown word candidate probability estimating unit 3, an optimum word string searching unit 4, a word dictionary 5, a word type determining unit 6, and a word type. Probability estimation unit 7, word length probability estimation unit 8, word notation probability estimation unit 9, word bigram probability estimation unit 10, word type definition table 11, word type frequency table 12, average word length table 13, character bigram frequency table 14, Word bigram frequency table 15, word dictionary creation unit 16, word type frequency calculation unit 17, average word length calculation unit 18, character bigram calculation unit 19, word bigram calculation unit 2
It consists of zero.

Although ngram is used as the word segmentation model in the above claims, the following description will be made using the word bigram, but the invention is not limited to this. Of the above configuration, the word collation unit 1, the unknown word candidate identification unit 2, the unknown word candidate probability estimation unit 3, the optimum word string search unit 4, the word type determination unit 6, the word type probability estimation unit 7, the word length probability estimation. Part 8, word notation probability estimation part 9, word
The bigram probability estimation unit 10 is for morphological analysis of the input text.

The word dictionary 5, the word type definition table 11, the word type frequency table 12, the average word length table 13, the character bigram frequency table 14, and the word bigram frequency table 15 are statistical language models used in morphological analysis. is there. Further, the word dictionary creation unit 16, the word type frequency calculation unit 17, the average word length calculation unit 18, the character bigram calculation unit 19, and the word bigram calculation unit 20 estimate the parameters of the statistical language model from the learning text.

In the above structure, when performing morphological analysis on the input text, first, the word dictionary matching unit 1 searches the word dictionary 5 for a word that matches the partial character string of the input text. Next, the unknown word candidate identification unit 2 selects, as an unknown word candidate, one that can be an unknown word from the partial character strings of the input text that are not matched with the word dictionary 5. The word type determination unit 6 determines the word type of the unknown word candidate based on the word type definition table 11. The unknown word candidate probability estimation unit 3 includes a part-of-speech word type probability obtained by the word type probability estimation unit 7, a part-of-speech and word type-dependent word length probability obtained by the word length probability estimation unit 8, and a word writing probability estimation unit 9
The part-of-speech word appearance probability of the unknown word is obtained from the obtained part-of-speech, word type, and word length-specific word notation probability. On this occasion,
The word type probability estimation unit 7, the word length probability estimation unit 8, and the word notation probability estimation unit 9 use the word type frequency table 12, the average word length table 13, and the character bigram frequency table 14, respectively.

The optimum word string searching unit 4 is a word dictionary matching unit 1.
The word bigram probabilities obtained by the word bigram probability estimation part 10 using the word bigram frequency table 15 for all combinations of the word candidates obtained by the above and the unknown word candidates obtained by the unknown candidate identification part 2, and , Using the word appearance probability by part of speech of the unknown word obtained by the unknown word candidate probability estimation unit 3, the word string having the maximum joint probability is obtained,
This is output as a morphological analysis result. Note that the word bigr
In the am probability estimation unit 10, the word bigram frequency table 1
5 is used, but the word ngram is used as a word segmentation model.
It is possible to use language models based on various words, such as.

When estimating the parameters of the statistical language model from the learning text, the word dictionary creating unit 16, the word type frequency calculating unit 17, the average word length calculating unit 18, the character bigram.
The calculation unit 19 and the word bigram calculation unit 20 respectively include a word dictionary 5, a word type definition 11, and a word type frequency table 1.
2. The parameters are stored in the average word length table 13, the character bigram frequency table 14, and the word bigram frequency table 15.

Next, the operation of the above configuration will be described. FIG. 4 is a flow chart for explaining the morphological analysis processing of the present invention. Step 101) Input the text into the word dictionary matching unit 1. Step 102) The word dictionary matching unit 1 searches the word dictionary 5 for a word that matches the partial character string of the input text.

Step 103) Unknown word candidate identifying section 2
However, a word that can be an unknown word is selected as an unknown word candidate from the partial character strings of the input text that are not matched with the word dictionary 5. Step 104) The word type determination unit 6 refers to the word type definition table 11 and determines the word type of the unknown word candidate selected in step 103.

Step 105) The word type probability estimating unit 7 estimates the word type probability by part of speech by referring to the word type frequency table 12, and sends it to the unknown word candidate probability estimating unit 3. Step 106) The word length probability estimation unit 8 refers to the average word length table 13 to estimate the word length probability by part of speech and word type, and passes it to the unknown word candidate probability estimation unit 3. Step 107) The word notation probability estimation unit 9 uses the character bigr
The part-of-speech, the word type, and the word notation probability by word length are estimated with reference to the am frequency table 14, and are passed to the unknown word candidate probability estimation unit 3.

Step 108) The unknown word candidate probability estimation unit 3 uses the part-of-speech word type probability acquired from the word type probability estimation unit 7, the part-of-speech and word type-specific word length probability acquired from the word length probability estimation unit 8, and the word notation. The part-of-speech-specific word appearance probability of the unknown word is obtained from the part-of-speech, word type, and word length-specific word notation probability acquired from the probability estimation unit 9. Step 109) The word bigram probability estimation unit 10 uses the word
The word bigram probability is calculated using the bigram frequency table 15, and in the optimum word string search unit 4, all combinations of word candidates obtained by the word dictionary matching unit 1 and unknown word candidates acquired from the unknown word candidate identifying unit 2 About the word
Using the word bigram probability obtained by the bigram probability estimation unit 10 and the word appearance probability by part of speech of the unknown word obtained by the unknown word candidate probability estimation unit 3, a word string having the maximum joint probability is obtained, and this is subjected to morphological analysis. Output as a result.

[0035]

Embodiments of the present invention will be described below with reference to the drawings. In the following, the word dictionary 5 and the word dictionary matching method, the definition of the word type and the unknown word candidate identification method, and the appearance of the unknown word candidate by part-of-speech based on the configuration of FIG. The probability estimation method, the word bigram probability estimation method, and the optimum word string search method will be described in this order.

In the following description, the learning text is
It is assumed that word division and part-of-speech assignment have been performed in advance by a human or another morphological analysis program. (1) Word dictionary 5 and word dictionary matching process: The word dictionary creating unit 16 creates the word dictionary 5 from a list of words in which the appearance frequency exceeds a certain threshold in the learning text. Here, a word is defined to be composed of a notation and a part of speech, and even if the same notation has a different part of speech, it is considered as a different word. In the present embodiment, the threshold of the appearance frequency is 1.

The word dictionary matching unit 1 enumerates the words in the word dictionary that match the partial character strings of the input text. For this purpose, the word dictionary 5 uses a data structure called "try" that merges common prefixes of character strings. (2) Word type definition and unknown word candidate identification processing: In modern Japanese orthography, at least five character types (Kanji, Hiragana, Katakana,
Alphabet, Arabic numerals) are used. Kanji is used for Chinese foreign words (Chinese) and Japanese notation (with sending kana) that is semantically equivalent to Chinese. Hiragana is used for notation of functional words such as postpositions and inflection endings, and katakana is used for phonetic notation of foreign words in Western Europe. The alphabet is used to represent Western European words and acronyms, and the Arabic numeral is used to represent numbers.

FIG. 5 shows the results of an examination of the types of characters that make up words and their changes for words with an appearance frequency of 1 in the EDR corpus. The EDR corpus is a typical Japanese text corpus that collects newspapers, magazines, textbooks, and the like. Generally, it is said that the properties of words that appear only once in a learning text are close to the properties of unknown words. According to the figure, words consisting of one character type (Kanji, Katakana, Hiragana, numbers, alphabets)
It can be seen that occupies about 65% of the total.

Further, among words composed of two or more character types, "morpheme", that is, "minimum linguistic element having no meaning when further divided" is "Kanji-Hiragana" or The only pattern is "Hiragana-Kanji". The former corresponds to a combination of kanji such as "ru" and futana, and the latter corresponds to a word in hiragana that represents difficult kanji such as "ei".

Therefore, in this embodiment, in the Japanese orthography, a plurality of word types are set on the basis of the types of characters forming a word and the patterns of their changes, and these are set in the word type definition 11 using Backus notation. Naur Form, BN
F). FIG. 6 shows a word type definition according to one embodiment of the present invention. This figure shows an example in which Japanese unknown words are classified into nine word types. here,[…]
Indicates matching with any one character in the character set.
A character range is indicated by writing "-" between two characters. JIS-X-0208 is assumed for the character code. “*” Represents 0 or more repetitions, and “+” represents one or more repetitions.

<Sym>, <num>, <alpha>
>, <Hira>, <kara>, and <kan> are a symbol string, a number string, an alphabet string, and a hiragana string, respectively.
Represents a character string that consists of one character type, the Katakana string and the Kanji string. <Kan-hira>, <hira-ka
n> are each kanji string - Hiragana Gana column, hiragana strings - representing the character string composed of two character types that kanji string. A character string composed of a plurality of character types other than these is all set to <misc>.

FIG. 6 shows an example of the word type definition 11, and the word type definition can be freely changed as needed. The word type determination unit 6 obtains a word type when an arbitrary character string is regarded as a word based on the word type definition 11. The unknown word candidate identifying unit 2 extracts a partial character string of the input text that has not been matched with the word dictionary 5 and removes a partial character string that cannot be a word (for example, one that includes punctuation marks inside). The determination unit 6 determines the word type and creates an unknown word candidate.

(3) Part-of-Speech Appearance Probability Estimation Method of Unknown Word Candidate: If an event in which the part-of-speech of an unknown word is t is represented by <Ut>, the appearance probability of the unknown word by part-of-speech, that is, a character string. c
_The probability P that an unknown word composed of ₁ , ..., C _k is t
(C ₁ , ..., c _k | <Ut>) is the following without loss of generality, considering the process of first selecting the word type, then the length, and finally the notation. It can be decomposed into the product of probabilities.

P (c ₁ ... C _k | <U-t>) = P (<WT> | <U-t>) P (k | <WT>, <U-t>) P (c ₁ ... c _k | k, <WT>, <Ut>) (5) The unknown word candidate probability estimation unit 3 estimates the word appearance probability by part-of-speech of the unknown word candidate using Expression (5).

The first term on the right side of the equation (5) is obtained from the relative frequency of related events in the learning text.

[0046]

[Equation 3]

Here, C (·) represents the frequency of events in the learning text. Therefore, the word type frequency calculator 1
7 is the frequency C (<WT>, <Ut>) of the combination of the word type and the part of speech in the learning text and the frequency C (<U of the part of speech).
-T>) is calculated and stored in the word type frequency table 12. Further, the word type probability estimation unit 7 estimates the word type probability by part of speech using the formula (6).

The second term on the right side of the equation (5) is the average word length of the words whose word type is <WT> and whose part of speech is <Ut>.

[0049]

[Equation 4]

It is approximated by a Poisson distribution using as a parameter.

[0051]

[Equation 5]

Therefore, the average word length calculation unit 18 determines that the word type is <WT> and the part of speech is <U in the learning text.
-Average word length of words with t>

[0053]

[Equation 6]

Is calculated and stored in the average word length table 13. Further, the word length probability estimation unit 8 estimates the word length probability for each word type and part of speech using Expression (7). The third term on the right side of the equation (5) is obtained by a method of approximating the appearance probability of the character string by the product of the character bigram probabilities and adding a correction for limiting the length to this. First, P _b (c ₁ , ...,
c _k | <WT>, <Ut>) is defined as follows. This is a product of the character bigram probabilities for each word type and part of speech, and corresponds to an approximation of the word notation probabilities for each word type and part of speech.

[0055]

[Equation 7]

Next, P _b (k | <WT>, <Ut>)
Is defined as follows. This corresponds to the probability that a word of length k appears for each word type and part of speech approximated by a character unigram model for each word type and part of speech.

[0057]

[Equation 8] P _b (k | <WT>, <Ut>) to (1-P <eow> | <WT>, <Ut>)) ^k-1 P ( <eow> | <WT>, <Ut >) (9)

In the above equation, the probability of an event in which a word of length k appears is equal to the probability of an event in which a word end symbol appears after k-1 times other than the word end symbol <eow> appears. . Now, the third term on the right side of the equation (5), that is, word notation probability P (c ₁ ...
c _k | k, <WT>, <Ut>) is the word notation probability P _b (c ₁ ...) for each word type and part of speech defined in Expression (8).
c _k | k, <WT>, <Ut>), and the occurrence probability P _b (k | of the word defined in Expression (9) and the word having the part-of-speech length k.
It can be approximated by the ratio of <WT>, <Ut>).

[0059]

[Equation 9]

Here, the character bigram probability P (c _i | c _i−1 , <W for each word type and part of speech necessary for the calculation of the equation (8).
T>, <U-t>) estimation method becomes a problem. Basically, the relative frequency f (c _i |
It may be estimated from c _i-1 , <WT>, <Ut>).

[0061]

[Equation 10]

However, since there are more than 3000 types of characters in Japanese, dividing the character bigram by word type and part of speech causes a problem of insufficient data. Therefore, the character bigram probability for each word type and part of speech is obtained by the linear interpolation method shown in the following equation.

[0063]

[Equation 11]

Here, α _i is α ₁ + α ₂ + α ₃ + α ₄
The weight satisfies + α ₅ = 1 and can be automatically determined by the linear interpolation method. _{f (c i, <WT>} , <U-t>)
And f (c _i | c _i-1 , <WT>, <Ut>) represent the relative frequency of the character unigram and the character bigram for each word type and part of speech. f (c _i ) and f (c _i | c _i-1 ) are
It is the relative frequency of character unigram and character bigram. V is
It is the number of different characters in the learning text.

Therefore, the character bigram frequency calculation unit 19 calculates the character unigram frequency and the character bigr required for the calculation of the equation (12).
Calculate am frequency from training data. Further, the word notation probability estimation unit 9 uses the formula (8), the formula (9), the formula (10), and the formula (1
2) is used to estimate word writing probabilities by word type, part of speech, and length. FIG. 7 is an example of a character bigram according to part of speech and word type of the present invention.

The character bigram contains key information for estimating the part of speech of an unknown word. For example, in FIG. 7, the first example indicates that the katakana sequence with the long syllabary "-" at the end is likely to be a noun, and the third sequence has the end with "target".
It means that the Kanji string that is is likely to be an adjective verb. (4) Word bigram probability estimation process and optimum word string search process: When an input sentence composed of character strings C = c ₁ ... C _m is divided into word strings W = w ₁ ... W _n , mathematics In particular, Japanese morphological analysis requires conditional probability P (W
It can be defined as a problem for finding a word string W ′ that maximizes | C). Here, since the character string C is common to all word divisions, a word string that maximizes P (W) may be obtained. Note that P
(W) (word segmentation model) can be used as long as it is a language model based on words, such as a word ngram or a class-based ngram. The following example will be described using the word bigram.

W ′ = argmax _WP (W | C) = argmax _WP (W) (13) In the present embodiment, P (W) is approximated by a word bigram model such as the following equation.

[0068]

[Equation 12]

Here, <bos> and <eos> are special symbols representing the beginning and end of a sentence. Basically, the word bigr
The am probability can be obtained from the relative probability f (w _i | w _i-1 ) of the corresponding event in the learning text.

[0070]

[Equation 13]

However, in this embodiment, the part of speech of the unknown word is estimated.
In order to do this, expand the normal word bigram model as follows.
To stretch. An event in which the part of speech of an unknown word is t is called <Ut>
If you choose to represent it with a sign, the word w_iIs an unknown word whose part of speech is t
, Then the word w_i-1An unknown word of part of speech t appears next to
Probability P (<Ut> | w_i-1) And an unknown word of part of speech t
The notation is w_iProbability P (w_i| <U−t>) product,
Word bigram probability P (w _i| W_i-1) Is approximated.

[0072]

[Equation 14]

P (<Ut> | w _i-1 ) is a relative value after replacing a word whose appearance frequency is less than or equal to a threshold value in the learning text with the unknown word symbol <Ut> corresponding to the part of speech t. Calculate from frequency. In the present embodiment, the appearance frequency threshold is set to 1. P (w _i | <Ut>) is obtained by the unknown word candidate probability estimation unit 3 described in the previous section. Therefore, the word bi
The gram frequency estimation unit 20 obtains the word bigram frequency in the learning text in which words whose appearance frequency is less than or equal to the threshold value are replaced with the unknown word symbol <Ut> corresponding to the part of speech t, and the word bigram frequency is calculated.
Stored in the frequency table 15. Further, the word bigram probability estimation unit 10, if w _i is the known word sought word bigram probability using equation (15), the formula when w _i is the unknown words (1
6) is used to find the word bigram probability.

Then, the optimum word string searching unit 4 selects from among all combinations of the word candidates generated by the word matching unit 1 and the unknown word candidates generated by the unknown word candidate identifying unit 2 as shown in Expression (14). The word string that maximizes the joint probability is obtained using dynamic programming. Furthermore, in the present embodiment, an A ^* search is used to obtain an arbitrary number of morphological analysis candidates in descending order of probability.

FIG. 8 is an example of a word bigram including a part-of-speech unknown word symbol according to an embodiment of the present invention. By introducing unknown word symbols for each part of speech, the word bigram will carry important information about the context in which the unknown word appears. For example, in FIG. 8, the first example indicates that a noun is likely to appear immediately after "no", and the second example is that a verb (sa verb) may appear immediately before "shi". It is high.

A specific processing example is shown below. here,
"The University of Pennsylvania Celebrates 50th Anniversary of ENIAC"
"Pennsylvania" and "EN
Assume "IAC" is an unknown word.

FIG. 9 shows an example of unknown word candidate generation and optimum word string search according to an embodiment of the present invention. In the figure, the state of the optimum word string search at character position 4 (between "Pencil" and "Vania") of the sentence "The University of Pennsylvania celebrates 50th anniversary of ENIAC" is shown. In the optimum word string search, the probabilities of combinations of all word candidates ending at a character position and all word candidates starting at that character position are calculated.

Before and after the character position 4, the partial character string “pencil” is collated with the word dictionary 5 to generate word candidates. The partial character string that is not matched with the word dictionary 5 becomes an unknown word candidate, the word type is determined based on the word type definition table 11, and the word appearance probability by part of speech is obtained based on the unknown word model. For example, the word type of "vania" is <kana> (consisting only of katakana).
Is a noun (<U-noun>) unknown word candidate, and “Vania University” is a noun unknown word candidate with a word type of <misc> (other).

In this figure, only the case where the part of speech of the unknown word is a noun (<U-noun>) is shown for the sake of simplicity. In the present embodiment, in practice, all the part-of-speech word appearance probabilities are calculated for each partial character string, and only the top five parts-of-speech are generated as unknown word candidates. FIG. 10 shows an example of a morphological analysis result according to an embodiment of the present invention. In the figure, the top three morphological analysis candidates of the sentence "The University of Pennsylvania celebrates the 50th anniversary of ENIAC" are shown. First
In the candidates, "Pennsylvania" and "ENIAC" are identified as unknown words of the word types <kana> and <alpha>, and the parts of speech are both nouns <U-noun>.
It is estimated that The second candidate and the third candidate are different in that "Bania University" of the word type <misc> and "Bania" of the word type <kana> are identified as unknown words (nouns), respectively.

As described above, Japanese kanji, hiragana,
By classifying words based on the character set such as katakana and its variation, it becomes possible to better model Japanese words. Although the above embodiment has been described based on the configuration of FIG. 3, a disk device or a floppy (registered trademark) connected to a computer used as a morphological analyzer is constructed by building the components of FIG. 3 as a program. The present invention can be easily realized by storing it in a portable storage medium such as a disk or a CD-ROM and installing it when carrying out the present invention.

[0081]

As described above, according to the present invention, a word type is defined on the basis of the types of characters that make up a word and its change, and the word appearance probability of each word category of an unknown word candidate is estimated for each word type. However, since the word sequence that maximizes the joint probability is obtained by using the word occurrence probability by part of speech of this unknown word and the word ngram probability that includes the unknown word symbol by part of speech, it is possible to prevent the unknown word from being overdivided and It is possible to realize a morphological analysis process capable of estimating the part of speech of.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the principle of the present invention.

FIG. 2 is a principle configuration diagram of the present invention.

FIG. 3 is a block diagram of a Japanese morphological analyzer of the present invention.

FIG. 4 is a flowchart for explaining a morphological analysis process of the present invention.

FIG. 5 is a diagram showing types of characters constituting a word and changes thereof in the EDR corpus according to the exemplary embodiment of the present invention.

FIG. 6 is a definition of a word type according to an embodiment of the present invention.

FIG. 7 is an example of a character bigram for each part of speech and word type according to an embodiment of the present invention.

FIG. 8 is an example of a word bigram including an unknown word symbol for each part of speech according to an embodiment of the present invention.

FIG. 9 is an example of generation of an unknown word candidate and optimum word string search according to an embodiment of the present invention.

FIG. 10 is an example of a morphological analysis result according to an embodiment of the present invention.

[Explanation of symbols]

1 word dictionary matching means, word dictionary matching unit 2 Unknown word candidate identification means, unknown word candidate identification unit 3 Unknown word candidate probability estimation means, unknown word candidate probability estimation unit 4 Optimal word string search means, optimal word string search unit 5 word dictionary 6 word type determination means, word type determination unit 7 Word type probability estimation means, word type probability estimation unit 8 Word Length Probability Estimating Means, Word Length Probability Estimating Unit 9 word notation probability estimation means, word display probability estimation unit 10-word bigram probability estimator 11 Word type definition table 12 word type frequency table 13 Average word length table 14-character bigram frequency table 15-word bigram frequency table 16 word dictionary creation section 17 Word Type Probability Estimator 18 Average word length calculator 19-character bigram frequency calculator 20 word bigram frequency calculator

Continuation of the front page (56) References JP-A-9-288673 (JP, A) JP-A-8-315078 (JP, A) Masaaki Nagata, Automatic collection of unknown words based on expected word frequency, Information Processing Society of Japan Research Report, Japan, November 19, 1996, Vol. 96, No. 114, p. 13-p. 20 Masaaki Nagata, Japanese Character Recognition Error Qualitative Method Using Character Similarity and Statistical Language Model, IEICE Transactions, Japan, November 25, 1998, Vol. J81-D-I I, No. 11, p. 2624-p. 2634 (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06F 17/21-17/28 G06K 9/72

Claims (2)

(57) [Claims] 1. A morphological analyzer for performing morphological analysis of Japanese, comprising: a word dictionary matching means for searching a word dictionary for a word to be matched with a partial character string of input text and generating it as a word candidate; A word type determination unit that classifies any character string of any of the word types and refers to the word type definition table in which the word type is defined based on the type of character And the word type frequency table in which the word type appearance frequency is defined for each part of speech, the word type probability determining means for determining the word type appearance probability for each part of speech, and the average word length for each word part of speech and word type are defined. The average word length table is referenced, and the average word length is approximated by the Poisson distribution to determine the word length probability of an arbitrary length for each part of speech and word type. A word notation for determining a word notation probability of an arbitrary character string by part of speech, word type, and word length by referring to a word ngram frequency for which a word ngram frequency for each part of speech and word type is defined Probability estimation means,
An unknown word model consisting of, using the word type determination means of the unknown word model,
In the word dictionary matching unit, an unknown word candidate identifying unit that selects, as an unknown word candidate, a potentially unknown word from a partial character string of the input text that is not matched with the word dictionary, and the word of the unknown word model. An unknown word candidate probability estimating means for estimating a part-of-speech word appearance probability of an unknown word candidate using a type determining means, a word length probability estimating means, and a word notation probability estimating means, and the word candidate obtained by the word dictionary matching means,
And, for all combinations of the unknown word candidates obtained by the unknown word candidate matching means, the word ngram probability obtained by referring to the word ngram frequency table in which the word ngram frequency is defined, and the unknown word candidate A morpheme analysis apparatus comprising: an optimum word string search unit that finds a word string having a maximum joint probability using the part-of-speech word appearance probability obtained by the probability estimation unit. 2. The word notation probability estimation means is the character ng.
By referring to the ram frequency table, character ngram frequencies of lower-order parts of speech and word strings and character ngram frequencies that do not consider differences between homogeneous or lower-order parts of speech and words the method obtains the character ngram probability by linear interpolation or normalized word notation probabilities obtained from the character ngram frequency by the sum of all words notation assigned to a string probability <br/> rate of the same length, The morphological analysis apparatus according to claim 1, wherein any one of the following methods is used.