JP2000231572A - Method and device for registering unknown word with noun thesaurus and recording medium with unknown word registration program recorded therein - Google Patents

Abstract

PROBLEM TO BE SOLVED: To statistically strictly decide a node of a noun thesaurus having a multinomial distribution being close to the multinomial distribution of unknown words as an unknown work registration node by using a Bayesian estimator logically having guarantee under a limited sample instead of a cooccurrence frequency and Kullback-Leibler information quantity(KL information quantity) being an inter-distribution distance in probability distribution space instead of a cosine between vectors in vector space. SOLUTION: This device consists of a means 100 which calculates the cooccurrence frequency of an unknown word and each verb in document data in corpus 120 and the cooccurrence frequency of each node and each verb of a noun thesaurus 130 in the document data in the corpus, a means 200 which uses the cooccurrence frequency information and calculates the Bayesian estimator of a multinominal distribution in which the unknown word co-occurs with each verb and the Bayesian estimator of a multinomial distribution in which each node of the noun thesaurus co-occurs with each verb and a means 300 which uses the Bayesian estimators and outputs a node of the noun thesaurus having a multinomial distribution being the closest to the unknown word as an unknown work registration node with Kullback-Leibler information quantity as a standard.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for registering an unknown word in a noun thesaurus, and more particularly, to co-occurrence of an unknown word with each verb in a corpus and each node of the noun thesaurus and each verb in the corpus. Based on the assumption of a polynomial distribution for co-occurrence with the existing noun thesaurus, the node of the noun thesaurus having a polynomial distribution closest to the polynomial distribution of the unknown word is used as the unknown word registration node using the Kullback-Leibler information as a scale. The present invention relates to a method and an apparatus for registering an unknown word in a computer, and a recording medium on which the program is recorded.

[0002]

2. Description of the Related Art In the field of natural language processing, many noun thesauruses in which nouns are semantically classified into a tree structure have been constructed for use in information retrieval and document clustering (for example, the "NTT Thesaurus"). Ikehara et al., “Japanese vocabulary system”, Iwanami Shoten, 1997). Further, as a part of the management method of the existing noun thesaurus, a method of registering an unknown word in the existing noun thesaurus has been proposed.

Many of the conventional methods for registering unknown words in the noun thesaurus include the Harris distribution hypothesis (Harris, Zelig S.) that "the meaning of a word is characterized from the viewpoint of co-occurring with the word.""Mathematical Srtuctu
res of Language ", New York: Willey (1968))
, A node for registering an unknown word is determined based on the cosine between co-occurrence vectors indicating what word and how much co-occurrence.

Here, Uramoto "Corpus based thesaurus" (Transactions of Information Processing Society of Japan, Vol.
2182-2189, 1996) and the like, the outline of an unknown word registration method based on the cosine between co-occurrence vectors of each node of an unknown word and a noun thesaurus common to many conventional methods is as follows. (1) Given an unknown word, calculate the co-occurrence frequency of the unknown word and each verb in the document data in the corpus and the co-occurrence frequency of each node and each verb in the noun thesaurus in the document data in the corpus. (2) The cosine between co-occurrence vectors composed of co-occurrence frequencies is calculated, and the node of the noun thesaurus with the maximum cosine is determined as the unknown word registration node.

[0005] The co-occurrence vector of noun n _i is given by the following equation.

[0006]

(Equation 1)

However, n _i ∈N (N is a set of nouns), v _j ∈V
(V is a verb set), | · | is the density of the set, co (n _i ,
v _j ) indicates the co-occurrence frequency which is the total number of times a noun _ni and a verb v _j co-occur in the same sentence in a corpus which is a database of documents.

[0008] The co-occurrence vector of node node _i of the noun thesaurus is given by the following equation.

[0009]

(Equation 2)

[0010] However, node _i ∈NODE (NODE is the node set of noun _{thesaurus), co (node i, v} j) node of the noun thesaurus in the same sentence in the corpus is a database of document node _i and a verb v _j is Indicates the co-occurrence frequency, which is the total number of co-occurrences, and is calculated by the following equation.

[0011]

(Equation 3)

The thesaurus includes a classification thesaurus in which words are registered only in the leaves of the tree structure, and an upper / lower thesaurus in which words are also registered in leaves and intermediate nodes. Is not performed.

The co-occurrence vector of the unknown word unknown is given by the following equation.

[0014]

(Equation 4)

Here, co (unknown, v _j ) is an unknown word unknow in the same sentence in a corpus which is a document database.
Indicates the co-occurrence frequency, which is the total number of times n and the verb v _j co-occur.

An unknown word registration node dnode_vec (unknown) for registering an unknown word unknown is determined by the following equation.

[0017]

(Equation 5)

Here, cos is the cosine between the vectors, “•” is the inner product of the vectors, and {vec} is the norem of the vector vec.

[0019]

The above-mentioned conventional method has several problems. First, since it is based on the distribution hypothesis, some probability distribution should be assumed for the co-occurrence frequency, but in practice, the true distribution is calculated only by using the value of the co-occurrence frequency as it is. I do not try to estimate. In this case, even if the similarity or the distance between the unknown word and each node of the noun thesaurus is measured by any scale, it is not possible to make a strict magnitude judgment.

Second, since it is based on the distribution hypothesis, some probability distribution should be assumed for the co-occurrence frequency, but in practice, the cosine between co-occurrence vectors in the vector space is used as a measure. I have. The cosine between co-occurrence vectors in vector space cannot always be a measure in probability distribution space. Assuming a probability distribution,
Exact similarity or distance cannot be calculated unless some measure in the probability distribution space is used.

An object of the present invention is to provide a Bayesian estimator theoretically guaranteed under a finite sample instead of a co-occurrence frequency, and Kullback-Leibler information (K), which is the distance between distributions in the probability distribution space instead of the cosine
L information), the node of the noun thesaurus having a polynomial distribution close to the polynomial distribution of the unknown word is determined more strictly statistically than the conventional method as an unknown word registration node.

[0022]

According to the method of registering unknown words in a noun thesaurus according to the present invention, when an unknown word is given as input data, a polynomial distribution of the unknown word and each verb, each node of the noun thesaurus, and each verb A node of the noun thesaurus whose distance calculated using the Bayesian estimator between the polynomial distributions and the Kullback-Leibler information (KL information), that is, the co-occurrence method that is statistically most similar to the unknown word Is output as an unknown word registration node.

The KL information amount between the probability distributions p (x) and q (x) is expressed by the following equation (for example, Hirasawa "Information Theory")
(Information Mathematics Series B-1) pages 22-23, Baifukan, 1
996).

[0024]

(Equation 6)

The Bayesian estimator for the Kullback-Leibler information is expressed by the following equation (for example, “IEEE TRA”).
NSACTIONS ON INFORMATION THEORY ”VOL.37, NO.5, PP.
1288-1291, see SEPTEMBER 1991).

[0026]

(Equation 7)

Here, θ is a continuous parameter that controls the distribution, and p (θ) is a prior distribution for the parameter.

FIG. 1 is a diagram showing the principle of the present invention. That is, the unknown word registration device for the noun thesaurus of the present invention, when given an unknown word, the co-occurrence frequency of the unknown word and each verb in the document data in the corpus and each node of the noun thesaurus in the document data in the corpus. When the co-occurrence frequency of each verb is calculated and the unknown word and the co-occurrence frequency information are given,
A Bayesian estimator that calculates the Bayesian estimator of a polynomial distribution in which an unknown word co-occurs with each verb and a Bayesian estimator of a polynomial distribution in which each verb co-occurs with each node in the noun thesaurus, and outputs the unknown word and the Bayesian estimator Given the calculation unit 200 and the unknown word and the Bayesian estimator, unknown word registration that outputs a node of a noun thesaurus having a polynomial distribution closest to the unknown word on the basis of the Kullback-Leibler information amount as an unknown word registration node And a node determining unit 300.

FIG. 2 is a flowchart for explaining the principle configuration of the present invention shown in FIG. That is, in the method of registering an unknown word in the noun thesaurus of the present invention, the unknown word is input in step S10, the co-occurrence frequency of the unknown word and each verb in the document data in the corpus, and each of the noun thesaurus in the document data in the corpus. Calculating the co-occurrence frequency of the node and each verb and outputting the co-occurrence frequency information; and, in the case of a polynomial distribution using the co-occurrence frequency information, a Kullback live between the finite sample and the true distribution. Calculating and outputting a Bayesian estimator of each node of the unknown word and the noun thesaurus whose error information amount is guaranteed to be minimum under the Bayesian criterion, and a polynomial of the unknown word using the Bayesian estimator Calculating and outputting the Kullback-Leibler information amount between the distribution and the multinomial distribution of each node of the noun thesaurus; And a step S60 of outputting, as an unknown word registration node, a node of a noun thesaurus having a polynomial distribution that is statistically closest to the polynomial distribution of an unknown word, which is obtained in the step S50. .

The method of registering unknown words in a noun thesaurus according to the present invention firstly guarantees that the amount of Kullback-Leibler information with a true distribution for a finite sample is minimized under the Bayes criterion. Since the obtained Bayesian estimator is used, the statistical information of the co-occurrence frequency can be completely used, unlike the conventional method of using the co-occurrence frequency as it is. Secondly, since the Kullback-Leibler information amount, which is the distance in the probability distribution space, is used as a measure,
A node of a noun thesaurus having a polynomial distribution that is statistically strictly closest to the polynomial distribution of an unknown word can be output as an unknown word registration node.

[0031]

FIG. 3 is a block diagram of an embodiment according to the present invention. In FIG. 3, 100 is a co-occurrence frequency calculation unit, 200 is a Bayesian estimation amount calculation unit, and 300 is an unknown word registration node determination unit. The co-occurrence frequency calculator 100 includes a co-occurrence frequency calculator 110 and a corpus 120 of a document database.
And an existing noun thesaurus 130. The thesaurus includes a classified thesaurus in which words are registered only in the leaves of the tree structure, and an upper / lower thesaurus in which words are also registered in leaves and intermediate nodes. However, no distinction is made here. The Bayesian estimator calculator 200 includes a Bayesian estimator calculator 210 and a beta distribution parameter table 220.
Consists of The unknown word registration node determining unit 300 includes a KL information amount calculator 310 and an unknown word registration node determining unit 320. Hereinafter, the operations of the co-occurrence frequency calculation unit 100, the Bayesian estimation amount calculation unit 200, and the unknown word registration node determination unit 300 will be described.

FIG. 4 is an operation flowchart of the co-occurrence frequency calculating section 100. First, an unknown word is input to the co-occurrence frequency calculator 110 (Step 70). When an unknown word is input, the co-occurrence frequency calculator 110 calculates the co-occurrence frequency co (u) of the unknown word unknown and the verb v _j in the document data in the corpus 120.
nknown, v _j ) is calculated (step 72). Next, the co-occurrence frequency calculator 110 calculates the node node _i and the verb v _j of the noun thesaurus 130 in the document data in the corpus 120.
Of the co-occurrence frequency co (node _i , v _j ) (Step 7)
4). here,

[0033]

(Equation 8)

The corpus 12 of the noun _nk and the verb v _j
The co-occurrence frequency at 0 is shown.

After calculating the co-occurrence frequency, the co-occurrence frequency calculator 110
Outputs unknown word and respective co-occurrence frequency information (step 76).

FIG. 5 is a flowchart of the operation of the Bayesian estimation amount calculating section 200. First, the Bayesian estimator calculator 210
The unknown word and co-occurrence frequency information are input from the co-occurrence frequency calculator 110 (step 80). When the unknown word and the co-occurrence frequency information are input, the Bayesian estimator calculator 210 calculates the co-occurrence frequency information and the beta distribution parameters β (v _j | node _i ) and β (v _j ) of the beta distribution parameter table 220. ｜ unknown)
Is used to calculate a Bayesian estimator corresponding to each node of the noun thesaurus 130 and a Bayesian estimator corresponding to the unknown word (steps 82 and 84). Here, β (v _j | node _i ) and β (v _j | unknown) are the nodes of the noun thesaurus 130
A node _i and a verb v _j , or a prior distribution p (θ) of a parameter θ governing a polynomial distribution representing a co-occurrence of an unknown word unknown and a verb v _j . FIG. 6 is a conceptual diagram of the beta distribution parameter table 220. Each Bayesian estimator is calculated as follows.

Noun Thesaurus 13 in Corpus 120
Any noun _nk ( _{nk k} nod included in node _i of node 0
Under the condition that e _i ) exists, the Bayesian estimator p ′ (v _j | node _i ) for the Kullback-Leibler information of the conditional probability distribution in which the verb v _j exists is

[0038]

(Equation 9)

Is calculated (step 82). Where β
(V _j | node _i ) is a parameter of the beta distribution as shown in FIG. 6, and is a true multinomial distribution p (v _j | node _i , θ′node).
_i ) means the prior distribution p (θ) of the parameter θ that governs it.

Note that θ ′ (node _i ) is the polynomial distribution p (v
_j | node _i , θ ′ (node _i )), the true parameter θ ′ (node _i ) ∈Θ, and the estimator according to equation (7) are the true distribution and the culvert This is an estimated amount that is guaranteed that the amount of the information on the liver is minimized under the Bayes criterion, and the following equation holds.

[0041]

(Equation 10)

The unknown word unknow in the corpus 120
Under the condition that n exists, the Bayesian estimator p ′ (v _j | unknown) regarding the Kullback-Leibler information of the conditional probability distribution in which the verb v _j exists is

[0043]

[Equation 11]

Is calculated (step 84). This estimator calculates the true polynomial distribution p (v _j ) between the unknown word unknown and each verb.
| Unknown, θ ′ (unknown)).

After calculating the Bayesian estimator, the Bayesian estimator calculator 210 outputs the unknown word and the Bayesian estimator (step 86).

FIG. 7 is an operation flowchart of the unknown word registration node determining section 300. First, the KL information amount calculator 31
At 0, the unknown word and the Bayesian estimator are input from the Bayesian estimator calculator 210 (step 90). When an unknown word and Bayesian estimator is input, KL information amount calculator 310, Bayesian estimator of the unknown word _{unknown p '(v j | unknown} ) and Bayesian estimator of the nodes node _i noun thesaurus p' ( v _j ｜ nod
e _i ) and the amount of Kullback-Leibler information between
It is calculated by the expression 2) (step 92).

[0047]

(Equation 12)

Next, the unknown word registration node determiner 320
The Kullback library information amount D (p '(• unknown); p' (• | n calculated by the KL information amount calculator 310
ode _i )) are compared, and the node node _{i of the} noun thesaurus that minimizes the value of the Kullback-Leibler information amount
Is determined by the following equation (13), and is set as an unknown word registration node dnode_prob (unknown) for the unknown word unknown (step 94).

[0049]

(Equation 13)

Finally, unknown word registration node determiner 320
Outputs the unknown word registration node dnode_prob (unknown) (step 96).

FIG. 8 is a view for explaining a simulation result of registering an unknown word in a noun thesaurus according to the present invention. Approximately 1000 nouns already registered in the actual existing noun thesaurus are extracted, and a registration experiment is performed by assuming the 1000 words as unknown words, and a conventional noun thesaurus registration method to the noun thesaurus and a noun thesaurus according to the present invention are used. We compared the method of registering unknown words to Wikipedia. The noun thesaurus uses the NTT thesaurus (Nagai Ikehara, “Japanese vocabulary system”, Iwanami Shoten, 1997), and the corpus which is a database of sentences is the EDR corpus (Japan Electronic Dictionary Research Institute,
"EDR Electronic Dictionary User Manual Version 2.1", 19
94), the top 50 most frequent verbs in the EDR corpus
The co-occurrence frequency with 0 words was used.

In FIG. 8, the horizontal axis indicates that not only one node having the minimum Kullback-Leibler information amount but also the number of candidate ranks is considered. The vertical axis shows the original N in the view from the smallest node to the node of the candidate rank.
If there is a node that is the same as the TT thesaurus, the correct answer is determined, and the cumulative correct answer rate is shown as a percentage. The word "conventional" indicates the result of registration of an unknown word in a noun thesaurus using a cosine between conventional co-occurrence vectors, and the present invention indicates the result of registering an unknown word in a noun thesaurus according to the present invention. As shown in FIG. 8, the correct answer rate of the method for registering an unknown word in a noun thesaurus according to the present invention always achieves a correct rate of 20% or more higher than that of the conventional method of registering an unknown word in a noun thesaurus. .

Although the embodiment of the present invention has been described above, the configuration shown in FIG. 3 is actually constructed on a so-called computer. The processing procedure and algorithm of each unit in FIG. 3 are collectively described in a computer-executable format, and are provided by being recorded on a computer-readable recording medium such as a floppy disk or a compact disk (CD-ROM). It is possible to

[0054]

As described above, according to the present invention, a Bayesian estimation that guarantees that the amount of Kullback-Leibler information with a true distribution for a finite sample is minimized under the Bayesian criterion Since the unknown word registration node is determined using the quantity and the Kullback-Leibler information amount, which is the distance in the probability distribution space, the noun having a polynomial distribution that is statistically strictly closest to the polynomial distribution of the unknown word. It becomes possible to output the thesaurus nodes as unknown word registration nodes.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a flowchart illustrating the principle configuration of the present invention.

FIG. 3 is a configuration diagram of an embodiment according to the present invention.

FIG. 4 is an operation flowchart of a co-occurrence frequency calculation unit in FIG. 3;

FIG. 5 is an operation flowchart of a Bayesian estimation amount calculation unit in FIG. 3;

FIG. 6 is a conceptual diagram of a beta distribution parameter table of FIG. 3;

FIG. 7 is an operation flowchart of an unknown word registration node determining unit of FIG. 3;

FIG. 8 is an explanatory diagram of a simulation result of registering an unknown word in a noun thesaurus according to the present invention.

[Explanation of symbols]

 Reference Signs List 100 Co-occurrence frequency calculating unit 120 Corpus 130 Noun thesaurus 200 Bayesian estimator calculating unit 220 Beta distribution parameter table 300 Non-word registered node determining unit

Claims (3)

[Claims] In the method of registering an unknown word in an existing noun thesaurus, when an unknown word is given, the co-occurrence frequency of the unknown word and each verb in the document data in the corpus and each of the noun thesaurus in the document data in the corpus are provided. Calculating a co-occurrence frequency of a node and each verb; a Bayesian estimator of a polynomial distribution in which an unknown word co-occurs with each verb using the co-occurrence frequency; and a polynomial in which each node and each verb of the noun thesaurus co-occur. Calculating a Bayesian estimator of the distribution, and using the Bayesian estimator to determine, as an unknown word registration node, a node of a noun thesaurus having a polynomial distribution closest to the unknown word on the basis of the amount of Kullback-Leibler information And a method for registering unknown words in a noun thesaurus. 2. An apparatus for registering an unknown word in an existing noun thesaurus, wherein when an unknown word is given, the co-occurrence frequency of the unknown word and each verb in the document data in the corpus and the noun in the document data in the corpus. A Bayesian estimator and a noun for a polynomial distribution in which an unknown word co-occurs with each verb, given a co-occurrence frequency calculator that calculates the co-occurrence frequency of each node and each verb in the thesaurus, and given the unknown word and co-occurrence frequency information A Bayesian estimator calculator that calculates a Bayesian estimator of a polynomial distribution in which each node of the thesaurus and each verb co-occurs, and given an unknown word and a Bayesian estimator, the unknown word and the Kullback-Leibler information amount are used as scales. An unknown word registration node determining unit that outputs a node of the noun thesaurus having the closest polynomial distribution as an unknown word registration node; apparatus. 3. A computer-readable recording medium storing a program for registering an unknown word in an existing noun thesaurus, wherein when the unknown word is given, the unknown word and each verb in the document data in the corpus are provided. Processing for calculating the co-occurrence frequency of each node and the co-occurrence frequency of each node and each verb of the noun thesaurus in the document data in the corpus,
A process of calculating a Bayesian estimator of a polynomial distribution in which an unknown word co-occurs with each verb and a Bayesian estimator of a polynomial distribution in which each verb co-occurs with each node of the noun thesaurus using the co-occurrence frequency, A recording medium characterized by recording a process of determining a node of a noun thesaurus having a polynomial distribution closest to an unknown word as an unknown word registration node using the amount as a measure of the amount of Kullback-Leibler information.