JP2000035963A - Automatic sentence classification device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for automatically classifying sentences into the clusters of the sentences based on the appearing frequency of a word in the sentence. SOLUTION: A statistic processing part 1 inputs plural sentences and prepares a matrix composed of the appearing frequency of a word in the sentences and a sentence classification part 2 inputs the matrix from the statistic processing part 1, classifies the sentences into the clusters of the sentences based on data in the matrix and outputs the result. The automatic classification problem of the sentences is captured as the estimation problem of a probability model defined on the direct product of the division of a word set and a sentence set, it is assumed that the respective sentences are generated from the belonging cluster in certain probability and the respective words are generated from the belonging cluster in certain probability in the probability model and the probability mode is selected by using an information amount standard. Clustering is alternately performed to the sentence set and the word set in a bottom-up manner.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic text classification apparatus, and more particularly, to an automatic text classification apparatus and method for automatically organizing and classifying electronic texts (texts and documents) such as electronic mails.

[0002]

2. Description of the Related Art In multimedia information processing, it is considered that classification and retrieval of a large number of electronic sentences (texts and documents) are performed more frequently and familiarly than at present. In order to meet such needs, it is necessary to establish a technique for organizing and classifying sentences accurately and at high speed.
Among them, the so-called "sentence automatic classification" or "sentence classification" technology that automatically classifies sentences into sentence clusters (groups) based on the distribution pattern of the frequency of occurrence of words in the sentences. The research on is active.

As a conventional method, for example, reference 1 (G. Sal
tot, M. McGill, Introduction toModern Informatio
n Retrieval, New York, McGraw-Hill, 1983), there is a method of automatically classifying sentences based on the distribution pattern of the frequency of occurrence of words in sentences. In this method, a word appearance frequency vector in each sentence is obtained, and the sentence is classified using the magnitude of the angle (cosine value) between the word frequency vectors of the two sentences as the distance between the two sentences. This method makes it very easy to classify sentences.

[0004] However, this method is not based on the theory of mathematical statistics, and has a problem that there is no theoretical guarantee on the quality of the classification obtained as a result.

To solve the above problem, Japanese Patent Application Laid-Open No. Hei 8-263510 (Patent Application No. 07-006) has been proposed.
No. 5722) proposes an automatic sentence classification system for automatically classifying sentences using stochastic methods such as mathematical statistics and information theory. In this sentence automatic classification system, a sentence is regarded as a word frequency vector, and MDL is used.
We take sentence classification as a stochastic model estimation problem using criteria. Specifically, it is assumed that a word frequency vector of a sentence to be classified is generated from a certain probability model, and MD
Based on the L criterion, one probabilistic model that can best explain the observed data (text features) is selected from the possible model classes. The grouping of the sentences in the selected model is set as the result of the sentence classification.

The MDL standard (The Minimum Description
Length Principle (also called the principle of minimum description length, also called MDL principle) is proposed in mathematical statistics and information theory. Based on input data, an optimal model is selected from a plurality of probability models. It is a standard to do.

Specifically, the MDL criterion asserts that a model that minimizes the sum of a quantity called “model description length” and a quantity called “data description length” is the optimal model. For example, reference (1) (J. Rissanen, Modeling
by Shortest Data Description, Automatica, Wo
l.14, 1978), Reference (2) (J. Rissanen, UniversalCo
ding, Information, Prediction, and estimation, IE
EE Trans. On IT, Vol. IT-30, 1984).

[0008] It has been demonstrated that the choice of a probabilistic model according to the MDL criterion has many desirable properties. The automatic sentence classification system described in Japanese Patent Application Laid-Open No. 8-263510 is noted as the first application of the MDL standard for automatic sentence classification.

[0009]

However, the automatic text classification system proposed in Japanese Patent Application Laid-Open No. 8-263510 has the following two problems.

A first problem is that there is a limit to using the word frequency information as it is for sentence classification.

[0011] Words include synonyms, synonyms, and related words. Many superficially different words have substantially the same meaning. Preferably, synonyms, synonyms, and related words are grouped together. It is desirable to classify sentences based on the appearance frequency of so-called “clusters of words”.

In the automatic sentence classification system proposed in Japanese Patent Laid-Open No. 8-263510, this problem is solved by referring to a dictionary prepared in advance. However, such word dictionaries are different in each field,
It is practically very difficult to prepare all such dictionaries comprehensively.

As a second problem, the above-mentioned Japanese Patent Application Laid-Open
The automatic sentence classification system proposed in Japanese Unexamined Patent Publication No. -263510 uses an annealing method, and has a problem in terms of processing efficiency.

Accordingly, the present invention has been made in view of the above-mentioned problems, and an object of the present invention is to automatically classify sentences into sentence clusters based on the frequency of occurrence of words in the sentence, thereby improving accuracy. Another object of the present invention is to provide a sentence automatic classification device which improves the processing efficiency and the processing efficiency.

[0015]

According to the present invention for achieving the above object, a plurality of sentences are input, the frequency of occurrence of words in each of the sentences is counted, and a matrix composed of the frequency of word appearance in each of the sentences is created. Statistical processing means and the matrix are input from the statistical processing means, and operations of classifying sentences and words into respective clusters (hereinafter referred to as “sentence clusters” and “word clusters”) based on the matrices are performed alternately. And a sentence classifying unit that outputs a cluster of the classified sentences.In the sentence classifying unit, the data in the matrix including the frequency of appearance of the word in each sentence is obtained from a simultaneous probability model of the sentence and the word. And the joint probability model calculates the joint probability of the sentence cluster and the word cluster as Is multiplied by the conditional occurrence probability from the sentence cluster to which the word belongs and the conditional occurrence probability from the word cluster to which the word belongs, and the classification operation of the sentence and the word is performed as an estimation problem of the probability model. The probability model is estimated using a criterion.

[0016]

Embodiments of the present invention will be described below. The present invention removes the assumption that the appearance frequency vector of a word in a sentence arises from a certain probability model, and simultaneously defines a sentence and a word, which are defined on the direct product of a sentence set division and a word set division. A sentence classification problem is considered as an estimation problem of a stochastic model.

More specifically, the joint probability of a sentence and a word is
Assume that the simultaneous probability of a sentence cluster and a word cluster is multiplied by a conditional occurrence probability from a word cluster to which a word belongs and a conditional occurrence probability from a sentence cluster to which the sentence belongs.

As a result, words are organized into clusters, and sentences can be classified based on the frequency of occurrence of word clusters, and it is expected that the accuracy of sentence classification is improved.

Second, by categorizing words and sentences into their clusters bottom-up and alternately,
Improve the efficiency of sentence classification.

In a preferred embodiment, the automatic sentence classification apparatus of the present invention receives a plurality of sentences (texts and documents), statistics the frequency of occurrence of words in each sentence, and comprises the frequency of occurrence of words in each sentence. A statistical processing unit for creating a matrix, and the matrix is input from the statistical processing unit, based on the input matrix,
A sentence classifying unit that alternately performs an operation of classifying a sentence and a word into respective clusters and outputs a cluster of the classified sentences.

It is assumed that the data in the matrix consisting of the frequency of appearance of the word in each sentence is generated from a joint probability model of the sentence and the word. And the conditional occurrence probability from the sentence cluster to which the word belongs, multiplied by the conditional occurrence probability from the word cluster to which the word belongs, and the classification operation of the sentence and the word is performed by the estimation problem of the probability model. And the probability model is estimated using the information amount criterion. Note that, in the embodiment of the present invention, the functions of the statistical processing unit and the sentence classifying unit can be realized by a program executed on a computer. Is loaded and executed.

[0022]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a configuration of an embodiment of an automatic sentence classification device of the present invention. Referring to FIG. 1, the automatic sentence classification device includes a statistical processing unit 1 and a sentence classification unit 2.
The statistical processing unit 1 receives a plurality of sentences as input, statistically calculates the frequency of occurrence of a word in each sentence, and creates a matrix including the frequency of occurrence of a word in each sentence. afterwards,
The sentence classification unit 2 receives a matrix from the statistical processing unit 1.

The sentence classification unit 2 assumes that each sentence forms one sentence cluster and each word forms one word cluster as an initial process.

Next, an operation of selecting two sentence clusters and combining (merging) them into one sentence cluster using the information amount criterion and an operation of selecting two word clusters and merging them into one word cluster are described. Repeat alternately.

Finally, the history of the sentence cluster created by the repeated operation is traced to output a sentence classification tree of the sentence cluster.

FIG. 2 shows an example of a matrix formed by the statistical processing section 1 and composed of the frequencies of appearance of words in each sentence. In the example shown in FIG. 2, there are four sentences (text 1
4) There are five words. The rows of the matrix correspond to words, and the columns correspond to sentences. Text
1 and 2 are sentences related to tennis, and 3 and 4 are sentences related to soccer.

In the former two sentences (texts 1 and 2), the appearance frequency patterns of words are similar. Also,
In the latter two sentences (texts 3 and 4), the appearance frequency patterns of words are similar. The sentence classification unit classifies the sentences based on the similarity of the appearance frequency patterns.

In this automatic sentence classification apparatus, sentences are recursively grouped into sentence clusters, so that a sentence classification tree composed of hierarchical sentence clusters is ultimately created. FIG. 3 shows a sentence classification tree created based on the data shown in FIG. Branch from the root to two nodes, text 1 and 2 branch from one node, text 3,
4 branches from the second node, and each text 1 to 4 constitutes a leaf.

The sentence classifying unit 2 classifies sentences into sentence clusters and also classifies words into word clusters.

For example, the words "shot" and "racquet" are words related to tennis and often appear in tennis-related sentences. The words "kick" and "goal" are words related to soccer and often appear in soccer-related sentences. Therefore, the words can be classified based on the distribution pattern of the appearance frequency in the text.

FIG. 4 shows a word classification tree created based on the data shown in FIG. Synonyms, synonyms, and related words are assigned to the same word cluster. As a result, the appearance frequency pattern of a word cluster composed of synonyms, synonyms, and related words becomes clearer, and the sentence can be classified with higher accuracy based on the pattern.

Hereinafter, the processing of the sentence classification unit 2 will be described in more detail.

The classification of sentences in a sentence set and the classification of words in a word set are performed recursively as follows.

As an initial setting, each sentence in a sentence set forms one sentence cluster, and similarly, each word in a word set forms one word cluster.

First, clustering is performed once for a sentence cluster in a sentence set. Here, selecting the two clusters that are the most similar and combining the two clusters into one cluster is called “clustering”.

Next, while the obtained sentence cluster is fixed, clustering is performed once on the word clusters in the word set.

As described above, the clustering is alternately performed on the sentence set and the word set, and the processing is performed to a point where the clustering cannot be further performed.

Finally, a sentence classification tree and a word classification tree are obtained. Clustering is, from the perspective of a classification tree,
It is done from the bottom up.

At the time of performing cluster limber, two clusters are selected from a sentence or a word cluster at the present stage, and are combined into one cluster. Which cluster is most similar to each other, and these are regarded as one cluster Determining whether to combine is most important in clustering.

Hereinafter, a method of combining two clusters will be described.

In this embodiment, it is assumed that the appearance frequency data of a word in a sentence is generated from a probability model defined using a specific division of a sentence set and a direct product of the specific division of the word set. .

For a given sentence set and word set,
In general, since many divisions are possible, one probability model corresponds to one division of a sentence set and one division of a word set.

In the present embodiment, one frequency model that can best explain the frequency data of words in a sentence and that is moderately simple is selected, and the sentence grouping and word grouping in the model are classified into sentences and This is the result of word classification.

In bottom-up clustering, this is a determination of whether more current clusters should be combined and, if so, which cluster is best.

The above probability model is defined below. This probability model is defined using the direct product of a sentence set division and a word set division, and the joint probability of a sentence and a word is calculated as the joint probability of the sentence cluster and the word cluster to which they belong. And the conditional occurrence probability from the sentence cluster to which the sentence belongs and the conditional occurrence probability from the word cluster to which the word belongs.

In this model, sentences belonging to the same sentence cluster do not occur uniformly but generally occur with different probabilities. Also, words belonging to the same word cluster do not occur uniformly but generally occur with different probabilities.

The above probability model is given by the following equation (1).

P (t, w) = P (t | Ct) × P (w | Cw) × P (Ct, Cw) (1)

This model is based on the appearance frequency of words in a sentence, that is, the occurrence probability P (t, w) of a sentence word pair (t, w).
9 is a stochastic model for w).

Ct and Cw represent a sentence cluster and a word cluster to which t and w belong, respectively.

P (t | Ct) represents a conditional occurrence probability of the sentence t from the sentence cluster Ct to which it belongs, and P (w |
Cw) represents the conditional occurrence probability of the word w from the word cluster Cw to which it belongs, and P (Ct, Cw) represents the joint probability of the sentence cluster Ct and the word cluster Cw.

Referring to FIG. 5, an example of the probability model will be described. In this probability model, for example, text 1 and text 2 are classified into the same cluster. The word "racquet" and the word "shoot" are classified into the same cluster. FIG. 5 also shows conditional occurrence probabilities of words, conditional occurrence probabilities of sentences, and simultaneous probabilities of sentence clusters and word clusters.

Next, the criterion for selecting the probability model becomes a problem. In the present embodiment, the information criterion (I
nformation Criteria)
Choose the above probability model.

The MDL standard claims that the model with the smallest sum of the model description length and the data description length (total description length) is the best model. In general, the model with the smallest total description length is a simple model that can explain the data well. MD
Model selection based on the L criterion can be realized by calculating the total description length and selecting a model having the minimum total description length.

In bottom-up clustering, for example, when deciding which two sentence clusters should be combined, two sentence clusters that create a model having the smallest total description length are selected and combined. It is equivalent to

When one probability model is given, the total description length of the model with respect to input data is calculated as in the following equation (2).

L = −ΣlogP (t, w) + k / 2 × logS, t∈T, w∈W (2)

The first term of the above equation (2) is the data description length, and the second term is the model description length. Also, t represents a sentence, and T
Represents a sentence set. w represents a word, and W represents a word set. k is the number of free parameters in the probabilistic model,
S represents the number of data.

In the bottom-up clustering, for example, first, the total description length of the current model is calculated, and then,
Calculate the total description length of the probability model obtained by merging all possible two sentence clusters, calculate the decrease of the total description length of these probability models, and select the model with the largest total description length reduction. Further, if the total description length of the probabilistic model obtained by merging these two sentence clusters does not decrease, further clustering would be preferable from the viewpoint of the MDL standard. To end.

The grouping of sentences in the finally selected model is used as the result of the sentence classification.

Next, an actual text classification algorithm will be described. This algorithm is called “clustering”. In clustering, a sentence set and a word set are alternately classified.

Clustering (T, W) T represents a sentence set, and W represents a word set.

Step 1. A set of sentence clusters CT and a set of word clusters CW are initialized as follows.

CT = {t} | t belongs to T {CW = {w} | w belongs to W} Step2. Repeat the following steps.

Step 2.1: Sentence Merge (CT, C
W) to update the CT.

Step 2.2: Word merge (CW, C
The CW is updated by T).

Step 2.3: If both CT and CW are unchanged, go to procedure 3.

Step 3. The history of the past CT is converted into a tree structure, and this is output as a sentence classification tree.

One clustering is performed by the following algorithm called “merge”. First, a case where word clusters are fixed and sentence clusters are merged will be described.

Merge (CT, CW) Step 1. When all pairs of sentence clusters in the set CT of sentence clusters are merged, the decrease in the total description length of the probability model is calculated, and the sentence cluster pairs are sorted in descending order of the decrease in the total description length.

Step 2. The largest pair of the total description length reduction is selected, and the most merge of the total description length reduction is performed.

Step 3. Current sentence cluster set C
Outputs T and ends.

Next, a case in which sentence clusters are fixed and word clusters are merged will be described.

Merge (CW, CT) Step 1. The reduction of the total description length of the probability model when all two word clusters in the word cluster set CW are merged is calculated, and the word cluster pairs are sorted in descending order of the total description length reduction.

Step 2. The largest pair of the total description length reduction is selected, and the most merge of the total description length reduction is performed.

Step 3. Current word cluster set C
Outputs W and ends.

[0077]

As described above, according to the present invention,
This has the effect of enabling highly accurate and efficient classification of sentences. The reason is,

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an automatic sentence classification apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining one embodiment of the present invention;
FIG. 9 is a diagram illustrating an example of data of the frequency of occurrence of words in a sentence.

FIG. 3 is a diagram for explaining one embodiment of the present invention;
It is a figure showing an example of a classification tree of a sentence.

FIG. 4 is a diagram for explaining one embodiment of the present invention;
It is a figure showing an example of a classification tree of a word.

FIG. 5 is a diagram for explaining one embodiment of the present invention;
It is a figure showing an example of a probability model.

[Explanation of symbols]

 1 Statistical processing unit 2 Text classification unit

Claims (8)

[Claims] And a statistical processing means for inputting a plurality of texts, statistically calculating the frequency of occurrence of words in each of the texts, and creating a matrix including the frequency of word appearance in each of the texts. By inputting and alternately classifying a sentence and a word into respective clusters (referred to as “sentence cluster” and “word cluster”, respectively) based on the matrix, a cluster of the classified sentences is output. The sentence classifying means, wherein the sentence classifying means considers that the data in the matrix composed of the frequency of appearance of words in each sentence is generated from a simultaneous probability model of sentences and words, The probabilistic model calculates the simultaneous probability of a sentence cluster and a word cluster by using the condition from the sentence cluster to which the sentence belongs. The occurrence probability and the conditional occurrence probability of a word from the word cluster to which the word belongs are calculated by multiplication, and the classification operation of the sentence and the word is performed by using the information amount criterion as an estimation problem of the probability model. An automatic sentence classification apparatus for estimating a sentence. 2. The automatic sentence classification apparatus according to claim 1, wherein an MDL criterion is used as said information amount criterion. 3. A statistical processing means for inputting a plurality of sentences, statistic of the frequency of occurrence of words in each of the sentences, and creating a matrix composed of the word occurrence frequencies in each of the sentences, By inputting and alternately classifying a sentence and a word into respective clusters (referred to as “sentence cluster” and “word cluster”, respectively) based on the matrix, a cluster of the classified sentences is output. An automatic sentence classification device, comprising: sentence classification means. 4. The sentence classifying means, as an initial processing, assumes that each sentence forms one sentence cluster and each word forms one word cluster. Then, two sentence clusters are selected and an information amount reference is made. The operation of merging into one sentence cluster using, and the operation of selecting two word clusters and merging into one word cluster are alternately repeated, and the history of the sentence cluster created by the repeated operation is traced to the sentence. 4. The automatic sentence classification device according to claim 3, wherein a classification tree of the sentence composed of the clusters is output. 5. The sentence classification means, wherein the word appearance frequency data in the sentence is generated from a probability model defined by using a specific division of a sentence set and a direct product of the specific division of the word set. Assuming that the probability model, the joint probability of a sentence and a word, the joint probability of the sentence cluster and the word cluster to which they belong, the conditional occurrence probability from the sentence cluster to which the sentence belongs, and the word cluster to which the word belongs Multiplied by the conditional occurrence probability of
When one probability model is given, two sentence clusters which create a model having a minimum total description length are selected for the total description length of input data of the model, and these are combined. 3. An automatic sentence classification device according to 3. 6. A statistical processing means for: (a) inputting a plurality of sentences, statistically calculating the frequency of occurrence of a word in each of the sentences, and creating a matrix composed of the frequency of occurrence of the words in each of the sentences; By inputting the matrix from the processing means and alternately classifying sentences and words into respective clusters (hereinafter referred to as “sentence clusters” and “word clusters”) based on the matrix,
A sentence classifying means for outputting a cluster of classified sentences, comprising: as initial processing, each sentence forms one sentence cluster, and each word forms one word cluster. It is created by repeating the operation of selecting a sentence cluster and merging it into one sentence cluster using the information amount criterion, and the operation of selecting two word clusters and merging it into one word cluster. A text classification means for tracing the history of a text cluster and outputting a text classification tree composed of text clusters; a statistical processing means; and a recording medium recording a program for causing the text classification means to function on a computer. 7. A method comprising the steps of: (a) inputting a plurality of sentences, statistically calculating the frequency of occurrence of words in each of the sentences, and creating a matrix including the frequency of occurrence of words in each of the sentences; and (b) generating a matrix based on the matrix. Outputting a cluster of classified sentences by alternately performing an operation of classifying sentences and words into respective clusters (referred to as “sentence clusters” and “word clusters”, respectively); In step (b), as an initial process, each sentence forms one sentence cluster and each word forms one word cluster. Then, two sentence clusters are selected and one sentence is selected using an information amount criterion. An operation of merging into a sentence cluster and an operation of selecting two word clusters and merging them into one word cluster are alternately repeated. And outputs the classification tree of a sentence consisting of a cluster of sentences following the cluster history text created by the operation, the sentence automatic classification method characterized by. 8. In the step (b), the appearance frequency data of a word in a sentence includes a specific division of a sentence set,
The probability model is generated from a probability model defined by using a direct product of a specific division of a word set. The probability is calculated by multiplying the conditional occurrence probability from the sentence cluster to which the sentence belongs and the conditional occurrence probability from the word cluster to which the word belongs.When determining whether two sentence clusters should be put together,
When one probability model is given, two sentence clusters which create a model having a minimum total description length are selected for the total description length of input data of the model, and these are combined. 7. Automatic sentence classification method described in 7.