JPH11250100A - Hierarchical document classifying device and machine-readable recording medium recording program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To classify a sentence to categorical hierarchies based on the distribution of words (keyword) appearing in the sentence (text and document). SOLUTION: This method grasps document classification as a statistical test problem. A categorical hierarchy storing part 1 stores category hierarchies. A probability model storing part 2 stores linear combination models. A learning part 3 refers to categorical hierarchies stored in the part 1, learns a linear combination model corresponding to each category from a document that is already classified to a category and stores the linear combination model in the part 2. A document classifying part 4 newly inputs a document, refers to each category in category hierarchies stored in the part 1, refers to a linear combination model corresponding to each category from the part 2 for the category, calculates the negative logarithmic likelihood of each linear combination model to the inputted document and classifies the inputted document to a category corresponding to a linear combination model that has the smallest negative logarithmic likelihood.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to automatic classification of Internet home pages, retrieval of documents in electronic libraries, retrieval of patent application information, automatic classification of digitized newspaper articles, and automatic classification of multimedia information. It is about classification and search.

[0002]

2. Description of the Related Art In the field of information classification and retrieval, development of a sentence classification (also referred to as document classification or text classification) apparatus is a major issue. Here, the sentence classification means that a human has provided a category in advance, further determines which category each sentence belongs to, classifies those sentences into the corresponding category, and stores them in the system. Knowledge is automatically acquired from the stored information.
Refers to automatically classifying newly input sentences based on the acquired knowledge.

[0003] Since sentences are classified into categories, when searching for sentences, only sentences in related categories need to be searched, and the search is efficient and accurate.

Heretofore, several text classification devices have been proposed. Above all, a sentence classification device proposed by Salton et al. Is well known (G. Salton and M.S.
J. McGill, Introduction to
Modern Information Retrie
val, New York: McGraw Hill
1, 1983). The sentence classification apparatus regards a cosine value between a frequency vector of a word appearing in a sentence and a frequency vector of a word in a category as a distance between the sentence and the category, and classifies the sentence into a category having the smallest distance. I have.

[0005]

However, most of the conventional methods classify sentences into several parallel categories, and there is no device for automatically classifying sentences into categories having a hierarchical structure. For example, the category of "politics" is further divided into subcategories of "parliament" and "political parties", and after categorizing sentences into the category of "politics", it is better to classify them into "diet" and "political parties" later. Searches are even faster.

An object of the present invention is not to classify sentences into parallel categories, but to automatically classify sentences into categories having a hierarchical structure.

It is another object of the present invention to realize highly reliable automatic classification of sentences.

[0008]

According to the present invention, a category is hierarchized, a probability model called a linear combination model or a set of probability models is made to correspond to each category, and when a new text is input, a linear model for the text is input. The negative log likelihood of the combined model or the stochastic complexity of the set of probabilistic models is calculated, and the new text is classified into the category having the smallest negative log likelihood or the smallest stochastic complexity.

That is, in the present invention, the sentences are classified into categories based on the distribution of words in the sentences. Especially,
A feature is that sentences are classified by a statistical test using a probabilistic model.

More specifically, the first hierarchical sentence classifying apparatus of the present invention displays a category hierarchy as a graph in which a node represents a category in which a sentence is classified and a link represents an upper / lower relationship of the category. For storing the category hierarchy storage unit, for each category of the category hierarchy stored in the category hierarchy storage unit, the weighted average of the probability model in the word space of the lower category as a linear combination model of the category, Probability model storage unit that stores a linear combination model of each category, based on sentences classified into each category of the category hierarchy stored in the category hierarchy storage unit, the linear combination model of each category, the lower category A learning unit that learns from the linear combination model of the above and stores the linear combination model of each category that has been learned in the probability model storage unit, and newly inputs a sentence , Regarding the input sentence as a data string of words, and for each category of the category hierarchy stored in the category hierarchy storage unit, the input sentence of the linear combination model of the category stored in the probability model storage unit And a sentence classification unit that calculates the negative log likelihood of the input sentence and classifies the input sentence into a category having the smallest calculated negative log likelihood.

[0011] In the first hierarchical sentence classifying apparatus configured as described above, the learning unit, for example, a sentence that is manually classified in advance into each category of the category hierarchy stored in the category hierarchy storage unit. Based on, the linear combination model of each category is learned from the linear combination model of the lower category, and the linear combination model of each category that has been trained is stored in the probability model storage unit. Is input, the sentence classification unit inputs the sentence,
This input sentence is regarded as a word data sequence, and for each category of the category hierarchy stored in the category hierarchy storage unit, the negative logarithm of the linear combination model of the category stored in the probability model storage unit for the input sentence is calculated. The likelihood is calculated, and the input sentence is classified into a category having the smallest calculated negative log likelihood.

Further, the second hierarchical sentence classification device of the present invention provides a graph in which a node represents a category in which a sentence is classified and a link represents an upper / lower relationship of the category.
A category hierarchy storage unit for storing the hierarchy of categories, and for each category of the category hierarchy stored in the category hierarchy storage unit, a set of probability models in the word space of lower categories, and a probability model of the category. And a set of
Probability model set storage unit that stores all elements of the set of probability models of each category, based on sentences classified into each category of the category hierarchy stored in the category hierarchy storage unit, the probability model of each category A learning unit that learns a set from a set of probability models on a word space of a lower category and stores all elements of a set of probability models of each category that can be learned in the probability model set storage unit. Input, regard the input sentence as a word data string,
For each category of the category hierarchy stored in the category hierarchy storage unit, calculate the stochastic complexity of the input sentence for the set of probability models of the category stored in the probability model set storage unit, and calculate A sentence classification unit for classifying the input sentence into a category having the smallest probabilistic complexity.

[0013] In the second hierarchical sentence classifying apparatus configured as described above, the learning unit, for example, manually classifies sentences classified in advance into each category of the category stored in the category hierarchical storage unit. Based on, the set of probability models of each category is learned from the set of probability models in the word space of lower categories, and all elements of the set of probability models of each category that have been learned are stored in the probability model set storage unit. Then, when a sentence to be automatically classified is input, the sentence classification unit inputs the sentence, regards the input sentence as a data string of words, and stores the category of the category stored in the category hierarchy storage unit. For each category of the hierarchy, the probabilistic complexity of the input sentence for the set of probabilistic models of the category stored in the probabilistic model set storage unit is calculated. To classify the input sentence is also a small category.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described in detail with reference to the drawings.

Referring to FIG. 1, the first embodiment of the present invention comprises a category hierarchy storage unit 1, a probability model storage unit 2, a learning unit 3, and a sentence classification unit 4.

The category hierarchy storage unit 1 stores the category hierarchy. The hierarchical structure of the categories is represented as a graph. In the graph, the nodes represent categories, and the links represent the upper and lower relationships of the categories. In addition, the category contains sentences that have already been classified. FIG. 2 shows an example of a category hierarchy. Here, the category hierarchy has a tree structure, but the category hierarchy generally has a more complicated graph structure.

The probability model storage unit 2 stores one probability model corresponding to one category in the category hierarchy. The present embodiment is characterized in that this probability model takes the form of a linear combination model. A linear combination model of a certain probability is defined as a weighted average of the probability models of lower categories. The following is an example of a linear combination model.

Example of linear combination model; In a tree structure of categories, nodes represent categories. The linear combination model of the category of the node c is defined as the linear combination model of the category of the child node and the linear combination of the probability model belonging to the node c itself as follows.

(Equation 1) In Equation 1, the random variable W is a set of words W = {w1, w
2,..., Ws}. P (W | c1), P (W |
c2),..., P (W | cn) are child nodes c1, c of c.
2, cn. P
(W | c ′) is a probability model belonging to the node c itself. That is, P (W | c ′) is a probability model that belongs to the category represented by c and does not belong to the category represented by c _1, ... _, C _n . P (c ′ | c), P (c1 | c),..., P (c
n | c) is the prior probability of c ′, c1,..., cn.

The learning unit 3 refers to the category hierarchy stored in the category hierarchy storage unit 1, learns the linear combination models of each category from the sentences already classified into the categories, and calculates It is stored in the model storage unit 2.

The sentence classification unit 4 inputs a new sentence, regards the sentence as a data string of words, and stores the sentence in the category hierarchy storage unit 1.
Is referred to, and for each category, the corresponding linear combination model of the category is referred from the probability model storage unit 2, and the negative log likelihood of each linear combination model for the sentence is referred to. The degree is calculated, and the sentence is classified into a category corresponding to the linear combination model having the smallest negative log likelihood.

The learning section 3 can learn the linear combination model by several methods. For example, a linear combination model of the lower category can be estimated as a histogram. Further, the weight coefficient can be learned by an algorithm called an EM algorithm.

Here, an example of the learning algorithm of the learning section 3 will be described. The graph representing the hierarchy has a tree structure.
The learning unit 3 refers to a category hierarchy having a tree structure and learns a linear combination model of the categories from the bottom up.
The learning algorithm is as follows, and its flowchart is shown in FIG.

The input is a node c. First, the root node of the tree structure is input. if node c is a leaf node. Then, a linear combination model of c is learned from the sentences classified into the category of the node c, and the process returns. Reference the linear combination model of the child node ci (i = 1, 2,..., n) of the else node c. The linear combination model of if node ci has not been learned yet. This algorithm is recursively applied to the then node ci. else Learn the linear combination model of node c from the linear combination model of node ci and the probability model of c itself, and return.

The sentence classification unit 4 classifies the sentence by a statistical hypothesis test of the sentence. Next, an example of the algorithm of the sentence classification unit 4 is shown, and its flowchart is shown in FIG.

Let d be an input sentence. The node c and the sentence d are input. First, the root node of the tree structure is input. if node c is a leaf node. Then, assume that the sentence d belongs to the category of the node c, and the processing ends. Else Calculate the negative log likelihood L (d | c) of the linear combination model of node c for sentence d. The negative log likelihood L (d | ci) of the child node ci (i = 1, 2,..., N) of the node c is also calculated. The minimum value of the calculated L (d | c) and L (d | ci) is obtained. The negative log likelihood of ci in the if child node is minimal. This algorithm is applied recursively to the then node ci. Else It is assumed that the sentence d belongs to the category of the node c, and the processing ends.

Next, the method of learning the linear combination model by the learning unit 3 and the method of calculating the negative log likelihood by the sentence classification unit 4 will be described through more specific examples. The category hierarchy is as shown in FIG. In FIG. 5, c1, c2, c
3 is a category, and d1, d2, and d3 are already classified sentences. FIG. 6 shows the appearance frequencies of the words w1, w2, and w3 in the sentences d1, d2, and d3. Word w
1, w2 and w3 are predetermined keywords.

Example of Learning Linear Combination Model Since c2 and c3 are leaf nodes, the linear combination model of the category of these nodes is learned as a histogram of words in a sentence as shown in FIG. 7A.

From the sentence d2 classified as c1, a probability model belonging to c1 itself is learned as a word histogram. This is represented as P (W | c1 ′). That is, it is c
The probability model belongs to category 1 and does not belong to categories c2 and c3, and is learned as shown in FIG.

On the other hand, the prior distribution of each model is learned as follows.

(Equation 2)

Here, f (ci) is the number of sentences to which the node ci and its dominant node belong, and N is the total number of sentences.
Therefore, the prior distribution of each model is learned as shown in FIG.

Further, according to the definition of the linear combination model,
The linear combination model at the node c1 can be learned as follows.

(Equation 3)

That is, the linear combination model at the node c1 is as shown in FIG.

Example of calculation of negative log likelihood The distribution of words in a new sentence d is shown in FIG. That is, it is assumed that the sentence classification unit 4 detects two words w1, one word w2, and one word w3 from the input sentence. The negative log likelihood of c1 for d is calculated as follows. Assume that the base of the logarithm is 2.

(Equation 4)

Similarly, the negative log likelihood of c2 and c3 is calculated.

(Equation 5)

(Equation 6)

Since the likelihood L (d | c1) is the smallest, d is classified as c1.

Referring to FIG. 9, the second embodiment of the present invention comprises a category hierarchy storage 1, a probability model set storage 5,
It comprises a learning unit 6 and a sentence classification unit 7.

The category hierarchy storage unit 1 stores the category hierarchy. In the category hierarchy, nodes represent categories, and links represent higher / lower relationships. FIG. 2 described above is an example of the category hierarchy.

The probability model set storage unit 5 stores a set of probability models. One set of probability models is defined and stored for each category in the category hierarchy. The following is an example of a set of probability models.

Example of Probability Model Set The set of the probability models of the node c is the probability model P (W |
c ′), P (W | c1),..., P (W | cn). P is a | | (cn W) is a child node c ₁ of _c, ... probability model (probability distribution) with a set of probability models of _{c n (W c1), ...} , P. P (W | c ′) is a probability model belonging to the node c itself. That is, it is a probability model that belongs to the category of c and does not belong to the categories of c1,. In addition, the prior probability P (c ′ |
c), P (c1 | c),..., P (cn | c) exist. Stochastic models P (W | c ′), P (W | c1),
.., P (W | cn) are expressed, for example, in the form of a histogram.

The set of probability models of each category includes a probability model in the word space based on sentences belonging to itself and a probability model in the word space based on sentences belonging to lower categories.

The learning unit 6 refers to the category hierarchy stored in the category hierarchy storage unit 1, learns the corresponding model set of each category from the sentences already classified into the category, and sets the set of probability models that can be learned. Is stored in the probability model set storage unit 5.

The sentence classification unit 7 inputs a new sentence, regards the sentence as a word data string, and stores the sentence in the category hierarchy storage unit 1.
Is referred to, and for each category, the corresponding probability model set of the category is referred from the probability model set storage unit 5, and the probability of the sentence for each referenced probability model set is referred to. The sentence is calculated into a category corresponding to a set of stochastic models having the smallest stochastic complexity.

The stochastic complexity is a quantity representing the minimum description length when data is described using a set of probabilistic models, and is proposed by Rissanen (Jorma Rissanen, Stockas).
tic Complexity in Statist
ical Inquiry, World Scientific
if publishing Co. , Singa
pore, 1989). In this embodiment, the probabilistic complexity is calculated as the negative logarithm of the weighted average of the likelihood for the data of the probabilistic model in the probabilistic model set.

Next, an example of the learning algorithm of the learning section 6 will be described. It is assumed that a graph representing a hierarchy has a tree structure. The learning unit 6 references a category hierarchy having a tree structure and learns a probability model set of the category from the bottom up. The algorithm is as follows, and its flowchart is shown in FIG.

The node c is input. First, the root node of the tree structure is input. if node c is a leaf node. Then, from the sentences classified into the category of the node c, learn all elements of the set of probability models of c and return. The probability model set of the child node ci (i = 1, 2,..., n) of the else node c is referred to. The set of probability models for if node ci has not been learned yet. This algorithm is recursively applied to the then node ci. Else The probability model set of node c is learned from the probability model set of node ci and the probability model of the text classified into c, and the process returns.

The sentence classifying unit 7 classifies sentences by a statistical hypothesis test. Next, an example of the algorithm of the sentence classification unit 7 will be described. FIG. 11 is a flowchart thereof.

Let d be an input sentence. The node c and the sentence d are input. First, the root node of the tree structure is input. if node c is a leaf node. Then, assume that the sentence d belongs to the category of the node c, and the processing ends. The probabilistic complexity SC (d | c) of the sentence d at the else node c is calculated. The probabilistic complexity SC (d | ci) of the child node ci (i = 1, 2,..., N) of the node c is also calculated. Find the minimum value of the calculated SC (d | c) and SC (d | ci). The stochastic complexity of ci in the if node is minimal. This algorithm is applied recursively to the then node ci. Else It is assumed that the sentence d belongs to the category of the node c, and the processing ends.

Next, an example of calculating the stochastic complexity will be described.

The hierarchy of categories is as shown in FIG.
It is also assumed that the frequency of appearance of a word (keyword) in a sentence is as shown in FIG.

Since nodes c2 and c3 are leaf nodes,
Each set of probability models contains one probability model. Further, those probability models are learned as a histogram as shown in FIG.

The probability model belonging to the node c1 itself is also learned as a histogram as shown in FIG.

Accordingly, the set of probability models of c1 includes the probability models P (W | c1), P (W | c2), and P (W | c3). Prior probabilities P (d
i | c) has a uniform distribution.

Assume that the frequency of appearance of a word in a new sentence d is as shown in FIG.

The stochastic complexity of d with respect to c1 is calculated as follows. Assume that the base of the logarithm is 2.

(Equation 7)

The stochastic complexity of d with respect to c2 and c3 is calculated as follows.

(Equation 8)

(Equation 9)

Since SC (d | c3) is the smallest,
d is classified into c3.

FIG. 13 is a block diagram of a third embodiment of the hierarchical sentence classification apparatus according to the present invention. The hierarchical sentence classification device of this example includes a computer 104 including a CPU 101, a main storage 102 and an auxiliary storage 103, a display device 105 connected to the computer 104, an input device 106, and an input / output device 108 including a file 107; And a recording medium 109 for recording a hierarchical sentence classification program. The recording medium 109 is a machine-readable recording medium such as a CD-ROM, a semiconductor memory, and the like.
4 by controlling the operation of the computer 104, the computer stores the category hierarchy storage unit 1, the probability model storage unit 2, the learning unit 3 and the sentence classification unit 4 shown in FIG. The illustrated category hierarchy storage unit 1, probability model set storage unit 5, learning unit 6, and sentence classification unit 7 are realized.

[0058]

As described above, according to the present invention,
Sentences can be automatically classified into categories having a hierarchical structure, and sentence classification with high statistical reliability based on the theory of likelihood ratio test can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram of a first embodiment of a hierarchical sentence classification device of the present invention.

FIG. 2 is a diagram illustrating an example of a category hierarchy.

FIG. 3 is a flowchart illustrating an example of a learning algorithm in the first embodiment of the hierarchical sentence classification device of the present invention.

FIG. 4 is a flowchart illustrating an example of a sentence classification algorithm in the first embodiment of the hierarchical sentence classification device of the present invention.

FIG. 5 is a diagram illustrating an example of a category hierarchy.

FIG. 6 is a diagram showing an example of word distribution in a sentence.

FIG. 7 is an explanatory diagram of a learning example of a linear combination model.

FIG. 8 is an explanatory diagram of a calculation example of negative log likelihood.

FIG. 9 is a block diagram of a second embodiment of the hierarchical sentence classification device of the present invention.

FIG. 10 is a flowchart illustrating an example of a learning algorithm in the second embodiment of the hierarchical sentence classification device of the present invention.

FIG. 11 is a flowchart showing an example of a sentence classification algorithm in the second embodiment of the hierarchical sentence classification device of the present invention.

FIG. 12 is an explanatory diagram of a calculation example of stochastic complexity.

FIG. 13 is a block diagram of a third embodiment of the hierarchical sentence classification device of the present invention.

[Explanation of symbols]

 1 Category Hierarchy Storage Unit 2 Probability Model Storage Unit 3 Learning Unit 4 Sentence Classification Unit 5 Probability Model Set Storage Unit 6 Learning Unit 7 Sentence Classification Unit

Claims (4)

[Claims] 1. A category hierarchy storage unit that stores a hierarchy of categories as a graph in which a node represents a category into which a sentence is classified and a link expresses an upper / lower relationship of the category, and is stored in the category hierarchy storage unit. For each category in the category hierarchy, a probability model storage unit that stores a weighted average of a probability model in a word space of a lower category as a linear combination model of the category, and stores a linear combination model of each category; Based on sentences classified into each category of the category hierarchy stored in the hierarchy storage unit, the linear combination model of each category is learned from the linear combination model of a lower category, and the linear combination of each category that can be learned is obtained. A learning unit that stores a model in the probability model storage unit, inputs a new sentence, regards the input sentence as a data string of words, For each category of the category hierarchy stored in the category hierarchy storage unit, the negative log likelihood for the input sentence of the linear combination model of the category stored in the probability model storage unit is calculated, and the calculated negative is calculated. A text classification unit that classifies the input text into a category having the smallest log likelihood. 2. A category hierarchy storage unit that stores a hierarchy of categories as a graph in which nodes represent categories into which sentences are classified and links express upper / lower relationships of the categories, and are stored in the category hierarchy storage unit. For each category in the category hierarchy, a set of probabilistic models in the word space of lower categories is set as a set of probabilistic models of the category, and a probability model that stores all elements of the set of probabilistic models of each category A set storage unit, based on sentences classified into each category of the category hierarchy stored in the category hierarchy storage unit, a set of probability models of each category, A learning unit that learns from and stores all elements of a set of probability models of each category that have been learned in the probability model set storage unit. A set of probability models of the category stored in the probability model set storage unit for each category of the category hierarchy stored in the category hierarchy storage unit. A sentence classification unit that calculates the stochastic complexity of the input sentence with respect to, and classifies the input sentence into a category having the smallest calculated stochastic complexity. 3. The computer of claim 1, wherein:
A machine-readable recording medium that records a program that functions as a category hierarchy storage unit, a probability model storage unit, a learning unit, and a sentence classification unit. 4. The computer of claim 2, wherein:
Category hierarchy storage unit, probability model set storage unit, learning unit,
And a machine-readable recording medium on which a program for functioning as a text classification unit is recorded.