JP3439494B2 - Context-sensitive automatic classifier - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention automatically classifies text types from statistical characteristics of the texts when storing texts composed of character strings having certain meanings in a document database or the like. The present invention relates to an automatic text classification method that supports efficient creation of a document database.

[0002]

2. Description of the Related Art In recent years, there has been an increasing need for more efficient document processing work in offices. When a new document is stored / registered in a document database or an electronic cabinet in a document processing operation, it is often the case that the document classification information is added so that the document can be easily searched later to effectively use the document.

At present, the classification work of this document is dealt with by two methods of manual classification or automatic classification by computer although it is still technically inexperienced. Among them, the manual classification can obtain the most accurate classification result at the current state of the art.

However, when creating a large-scale text database, a. Time efficiency of classification work b. Due to the need for objective classification criteria (the need to eliminate individual differences and subjectivity when classifying by multiple people), automatic classification of text by a computer is desired.

The present invention relates to an automatic text classification technique by this computer, and classifies texts having similar contents into similar text classes by using statistical characteristics of the text as a clue.

As a classification method using the statistical characteristics of the text as a clue, a method of supervised classification in the field of multivariate analysis is used to map the text onto the classification space,
A method of classifying a text into a text class having the shortest distance in a classification space by determining the similarity of the texts based on the distance between the texts is known from the following literature (i). Literature a. Shizuo Mizutani: Discrimination of Ume / Sakura by term, Japanese Language Measurements Vol. 12, No. 1, PP. 1-13, 1979.

It is known from the following document B that the paragraph structure of a text group consisting of a plurality of paragraphs becomes a state transition network. However, when automatically classifying the paragraph structure of a document, this state transition probability Should be considered. Literature b. Tadashi Hoshiai, et al .: Support for drafting paragraph structures for business document preparation, Proc. Of the 43rd National Convention of IPSJ, 2F-1, PP. 3-315-316, 1991.

[0008]

However, the "document structure database construction processing method" (patent application on September 20, 1991) previously submitted by the applicant of the present application did not consider the above points. However, there is a problem that the optimal classification process cannot be performed.

In view of the above problems of the prior art, the present invention provides automatic classification of text having a paragraph structure,
The state transition that expresses the paragraph structure that is the context by considering the information (probability and transition probability) related to the probability distribution of the text class itself, which is the teacher information, and reflecting these in the weighting during classification processing. The purpose is to achieve appropriate classification that is suitable for the network.

[0010]

According to the invention, the above mentioned objects are achieved by means of the patent claims.

[0011] That is, the invention of claim 1, a text input unit for entering text classification target, input text
The comparison data used to classify the strikes into text classes.
Text that is a database that can store and retrieve data
Lath feature data section and input text according to its content
Texts for classifying into any of the above text classes
In an apparatus having an automatic classification section,
The dynamic classification unit determines the position of the input text in the classification space,
The distance from the position of the center of gravity of each of the above text classes
Find the simple occurrence of the text class itself for that distance
Weighted using probabilities, based on weighted distance
Input text into one of the text classes above
It is a context-dependent automatic classification device.

[0012]

[0013]

The invention of claim 2 is a text class which is a database for storing and retrieving a text input section for inputting text to be classified and comparison data used when classifying the input text into text classes. In a device having a feature data section and an automatic text classification section for classifying an input text into one of the text classes according to its content, the automatic text classification section is provided with a position of the input text in a classification space, The distance from the position of the center of gravity of the text class is obtained, the distance is weighted in consideration of the state transition probability between the text classes, and the input text is classified into one of the text classes based on the weighted distance. It is a context-dependent automatic classification device.

The invention according to claim 3 is the context-dependent automatic classification device, wherein the state transition probability between the text classes also considers the state transition probability to itself.

[0016]

[0017]

[0018]

The invention according to claim 4 is the context-dependent automatic classification device having a classification result output unit for outputting the classification result of the input text by the automatic text classification unit.

[0020]

According to the present invention, in automatic classification of text having a paragraph structure, information regarding probability distribution of a text class (occurrence probability, transition probability, etc.) is taken into consideration and reflected in weighting at the time of classification. By this, it becomes possible to realize the classification suitable for the state transition network expressing the paragraph structure which is the context.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The flow of automatic classification processing according to the present invention is divided into five steps, Steps [1] to [5], and these steps are based on the block diagram of the automatic classification apparatus shown in FIG. And will be explained in order.

Step [1]. The user inputs the text data T to be classified by himself / herself from the input / output device 7.
In this case, for example, if you want to input a document file in a floppy disk, you can select a floppy disk device as the input device, and if you want to input a received e-mail, consider the modem and communication software as the input device. The hardware used as the input device may be appropriately selected depending on the information form of the input document.

Step [2]. The text input unit 4 sequentially sends the text data (paragraph sentence) T in the document input to the system to the automatic text classifying unit 1.

Step [3]. Next, the automatic classification unit 1
The statistical characteristics of the text data T are analyzed, the distance between the classification target and the classification destination is obtained, and then the classification is performed by weighting the context information. Then, the text data T is
Classify into text classes with close statistical characteristics and high expected values of appearance.

That is, the statistical features of the keywords in the input text data are extracted, and the group of the input text data T and the text as the teacher information (in the present specification, "text class" is extracted by the method of multivariate analysis). ).) In the classification space.

As a method of classifying texts by multivariate analysis, there is, for example, the document B (supra) by the method of quantification theory. Further, as the same supervised classification method, various methods such as one based on Mahalanobis's general distance in multiple discriminant analysis of multivariate analysis have been proposed, but in the present invention, the concept of distance is used as a classification criterion. Any method can be used as long as it uses.

By the way, the present invention only uses such a classification method, does not cover the contents of the distance calculation method, and the use of the distance concept is not affected by the difference in the method, so that the statistical method is used. The definition of the method and the explanation of the calculation method are minimized. The outline of the distance calculation is as follows.

First, the text automatic classification unit 1 uses the text class C _j as a comparison target of classification (that is, teacher information).
Feature data of (j = 1, 2, ..., M) is extracted and stored in the text class feature data unit 3.

Next, the automatic text classification unit 1 extracts the characteristic data of the input text T to be classified, and the characteristic data of the text class C _j (j = 1, 2, ..., M) as the teacher information. Compare all of.

Here, the input text on the classification space
Position vector of T is X (T), text class C_j In
Each text T_j1・ ・ ・, T_jNj Position vector of X
(T _j1), ..., X (T_jNj ), The text
Russ C_j The center of gravity of

[0031]

[Equation 1]

It becomes

Therefore, the distance between the input text T and the text class C _j in the classification space is

[0034]

[Equation 2]

Can be given by After the text,
This is the focus of the present invention. Therefore, the distances from all the text classes C _j are calculated, weighting described later is performed, and the optimum text class is set as the classification destination.

Step [4]. The weighting by the context information is calculated for the distance obtained in step [3]. Here, a method of weighting with is roughly can be classified and weighted by a simple probability, the two methods of the weighting by the state transition probability.

The two types will be described below.
In the first method, as shown in FIG. 2, the simple occurrence probability of the text class itself is used as the distance weighting. In FIG. 2, some text classes are displayed on the classification space. The text class may be, for example, a visiting text class, a thank-you text class, or a selling text class.

The position of the input text on the classification space and each text class Cj (j = 1, 2, ...
. , M; where M is the number of text classes) and the distance r _{tj to} the position of the center of gravity in the classification space is compared, the weighting is performed in consideration of the simple occurrence probabilities of the text class itself, and Perform classification suitable for the probability distribution of teacher information.

Simple occurrence probability p (C of each text class
_i ) is

[0040]

[Equation 3]

According to the above, it is obtained from the teacher information. However, f
(C _i ) is the frequency of instance data in the text class C _i (that is, the number of paragraph sentences belonging to C _i ).

The text class C _i has a normal distribution N (0,
The occurrence probability conversion radius r _i of the text class C _i when obeying 1) is

[0043]

[Equation 4]

It is determined by

The distance r standardized by this conversion radius
_ti ^* is obtained by ^{_{r ti * = r t i /}} r i. In the classification using the simple occurrence probability, the text class C _min that minimizes this normalized distance is obtained.

In the second method, as shown in FIG. 3, the state transition probability between text classes is used as the weighting of the distance. In FIG. 3, the text class that is a node of the state transition network is, for example, a greeting text class or a history text class.
It may be the text class of description or the text class of invitation.

Text data (paragraph text) in the input document
Of the text classes that became the classification class immediately before in C
_i and the next text data is being processed. However, when the first paragraph of the document is being processed, C _i = C ₀ = start point node.

The position of the input text in the classification space and each text class C _j (j = 1, 2, ...
., M; where M is the number of text classes) and the distance r _{tj from} the position of the center of gravity in the classification space is compared to the text classes C _i to C _j (j = 1, 2,
, M) are weighted in consideration of the state transition probabilities to perform the classification suitable for the probability distribution of the teacher information as the classification context.

The state transition probability p (C _j | C _i ) from the text class C _i to C _j is

[0050]

[Equation 5]

Is obtained from the teacher information. However, f
(C _i ) is the frequency of instance data in the text class C _i (that is, the number of paragraph sentences belonging to C _i ).

When the state transition from the text class C _i to C _j follows the normal distribution N (0,1), the state transition conversion radius r _i is

[0053]

[Equation 6]

It is determined by

The distance r _tj ^* standardized by the state transition to the text class C _j is obtained by r _tj ^* = r _tj / r _j . In the classification using the state transition probability, the text class C _min that minimizes this normalized distance is obtained.

Step [5]. The classification result output unit 5 outputs the examination result for the classification result to the input / output device 7. That is, the text class name C _jmin of the classification destination is output as the classification result for the input text T.

[0057]

As described above, according to the present invention,
In the automatic classification of texts with a paragraph structure, a state transition network that expresses a contextual paragraph structure by considering the information about the probability distribution of text classes (occurrence probabilities, transition probabilities, etc.) and reflecting it in the weighting during classification. It becomes possible to realize an appropriate classification that conforms to, and it greatly contributes to the efficiency of the document processing work.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a configuration of a text classification device.

FIG. 2 is an explanatory diagram regarding weighting based on a simple occurrence probability.

FIG. 3 is an explanatory diagram regarding weighting based on a state transition probability.

[Explanation of symbols]

1 Text automatic classification section 2 Text classification result examination department 3 Text class feature data section 4 Text input section 5 Classification result output section 6 New Class Proposal Department 7 I / O device

Continuation of front page (56) References JP-A-2-285419 (JP, A) En Sato et al., Automatic packaging of net news articles, Journal of Information Processing Society of Japan, Japan, Information Processing Society of Japan, 1997 6 Mon, VOL38 No6, pages 1225 to 1234 Atsio Kawai, automatic document classification based on learning results of semantic attributes, IPSJ Journal, Japan, IPSJ, September 15, 1991, VOL33 No9, No. 1114 No. 1122 Tadashi Hoshiai et al., Support for drafting paragraph structure for business document creation, Proc. Of the 43rd National IPSJ Conference (Late 1991) (3), Japan, Information Processing Society, October 22, 1991, pp. 3-315, pp. 3-316 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06F 17/30 JISST file (JOIS)

Claims (4)

(57) [Claims] 1. A text input section for inputting a text to be classified, a text class characteristic data section which is a database capable of storing and retrieving comparison data used when classifying the input text into a text class, and an input text. An apparatus having a text automatic classification unit for classifying the text class into any of the text classes according to the content thereof, wherein the text automatic classification unit is the position of the input text in the classification space and the position of the center of gravity of each of the text classes. Context-dependent, characterized in that the distance between the text and the text class is weighted using the simple occurrence probability of the text class itself, and the input text is classified into one of the text classes based on the weighted distance. Automatic classifier. 2. A text input section for inputting a text to be classified, a text class characteristic data section which is a database capable of storing and searching comparison data used when classifying the input text into a text class, and an input text. An apparatus having a text automatic classification unit for classifying the text class into any of the text classes according to the content thereof, wherein the text automatic classification unit is the position of the input text in the classification space and the position of the center of gravity of each of the text classes. A context characterized in that the distance between the text and the text class is weighted in consideration of the state transition probability between the text classes, and the input text is classified into one of the text classes based on the weighted distance. Dependent automatic classifier. 3. The context-dependent automatic classification device according to claim 2, wherein the state transition probability between the text classes also considers the state transition probability to itself. 4. A context-sensitive automatic classification apparatus according to any one of claims 1 to 3 having a classification result output unit for outputting a classification result of the input text by the text automatic classification section.