JPH1139313A - Automatic document classification system, document classification oriented knowledge base creating method and record medium recording its program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To dissolve the polysemy of a semantic category and to improve precision more by selecting and utilizing only an optimum semantic category that a word has at the time of generating learning data. SOLUTION: A characteristic extraction mechanism 10 analyzes a classification tagged document set of a file 20, examines the semantic category of a word and outputs an extracted characteristic vector and data of a classification category to which its text belongs to a learning data file 22 before polysemy dissolution. A polysemy dissolution mechanism 12 is inputted from the file 22 and an association degree data file 23 and outputs a polysemy dissolution result to a learning data file 24 after polysemy dissolution. A document classification oriented knowledge base creation mechanism 13 is inputted from the file 24 and outputs the weight between a characteristic and a classification to a document classification oriented knowledge base 25. A classification processing mechanism 14 is inputted from a thesaurus dictionary 21 and the base 25 and outputs a classification tag that corresponds to a new text inputted by a user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the technical field of natural language processing, and more particularly, to an automatic document classification system, a method of generating a knowledge base for document classification from a set of documents with classification tags, and a recording medium recording the program thereof. It is about.

[0002]

2. Description of the Related Art Document classification means that a text is classified into predetermined classification categories (for example,
Politics, sports, economy, etc.), and now that a large amount of digitized texts are in circulation,
It is an important issue in the efficient search / use of text. An automatic document classification system realizes automation of the document classification work. This system creates an appropriate classifier (knowledge base for document classification) using text that has been categorized in advance (here, this is referred to as a “document set with a classification tag”), and based on the classifier. To categorize the newly input text.

A conventional automatic document classification system extracts a word which is a key of classification from a set of documents with classification tags,
A classifier is generated by learning the degree of association, which is the strength of association between those words and the pre-assigned classification category. Since words that characterize the field frequently appear in texts having the same classification category, the degree of association between the word and the specified classification category increases,
The degree of association between the words that are not so and the category is reduced. The classifier includes a word, a classification category, and a link indicating the degree of association between the word and the classification category.

FIG. 2 shows an example of a classifier. As shown in FIG. 2, the feature node is composed of words frequently appearing in the text, and the classification node is composed of the classification category of the document set. The numerical value on the link between the characteristic node and the classification node indicates the degree of association. Various methods such as a chi-square calculation method can be considered as a learning method of the degree of association.

When a new text is input, first, words appearing on the text are extracted, and then, the relevance between the text and each classification category is determined from the relevance between the words on the classifier and the classification tags. Is calculated. Those whose values exceed a predetermined threshold value or have the largest value are determined as the category of the text.

[0006]

According to the above-mentioned prior art, a classifier can be automatically generated from a set of documents with a classification tag. Since the classification target text newly input includes an unknown word not included in the feature node, the classification accuracy of the text is deteriorated. As a method for solving this problem, a method of generating feature nodes using not only words but also semantic categories of words obtained by using a thesaurus is conceivable. A semantic category is a more general concept than a word. Generally, a plurality of words have a common semantic category. Therefore, generation of a feature node based on this semantic category can solve the problem of an unknown word.

However, on the other hand, the number of meaning categories of each word obtained by using a thesaurus is usually not limited to one. For example, the word "route" is used in the political field to mean "policy", and in the transportation field it is used to mean "path". Such polysemous words have a plurality of semantic categories on the thesaurus. In the learning step of generating a classifier, when generating a feature node using a thesaurus, if all of a plurality of semantic categories obtained from polysemous words are used without being narrowed down according to the context, a feature node, a classification node, Is incorrectly calculated,
Classification accuracy may be degraded. When a semantic category on a thesaurus is used for a feature node, it is necessary to specify what the semantic category is accurately according to the context, that is, to eliminate polysemy.

An object of the present invention is to provide an automatic document classification system which eliminates the ambiguity of a semantic category and has higher accuracy, a method of generating a knowledge base for the document classification, and a recording medium on which a program thereof is recorded. .

[0009]

According to the present invention, there is provided an automatic document classification system which inputs a document set with a classification tag and a thesaurus dictionary, and learns a word characterizing a document, its semantic category, and a classification category of the document. Means for calculating the relevance between the classification category and the feature from the generated learning data; inputting the calculated relevance and the learning data; and selecting an incorrect semantic category from the learning data. Means for removing and generating new learning data, means for receiving the generated new learning data as input, calculating relevance between a classification category and a feature to generate a knowledge base for document classification, And a means for inputting the above document and outputting a corresponding classification category based on the knowledge base.

According to the method for generating a knowledge base for document classification of the present invention, a document set with a classification tag and a thesaurus dictionary are input to generate learning data composed of words characterizing a document, their meaning categories, and the category of the document. Calculating, for the learning data, a relevance between a classification category and a feature; removing inappropriate features from the learning data based on the relevance to generate new learning data; By calculating the weight between the classification category and the feature, a knowledge base for document classification is generated.

The computer-readable recording medium of the present invention reads a document set with a classification tag, analyzes the document set with a classification tag, extracts words characterizing the document, and refers to a thesaurus dictionary. Examining the semantic category of the word, generating learning data composed of a feature vector composed of the word and the semantic category and the classification category of the document, and calculating the degree of association between the classification category and the feature for the learning data And generating new learning data by removing inappropriate features from the learning data based on the degree of association.
Calculating a weight between the classification category and the feature based on the new learning data to generate a knowledge base for document classification.

The disambiguation of the semantic category is based on the property that "the semantic category is also generated from another word related to the classification category to which the target text belongs", and the classification category information of the text is preliminarily obtained. This can be done by using the obtained feature and the degree of association between the corresponding category. For example, the polysemy word “route” has two meaning categories “S traffic route” and “S form” on a certain thesaurus. Here, a letter S is added to the head of the semantic category in order to distinguish the semantic category from the word. “S form” is a semantic category that is also generated from the words “trend” and “emergency” that frequently appear in the category “politics”, and “S traffic road” frequently appears in the category “traffic”. It is also generated from the words "Shinkansen" and "Tokaido".
Therefore, it can be seen from the text having a category having the classification category of “politics” and “traffic” that “S traffic route” is strongly related to “traffic” and “S morphology” is strongly related to “politics”. Should be. By using this degree of relevance, when the word “Route” appears in the target text, if the classification category of the text is “Politics”, select “S form” as the semantic category and select “Transportation”. For example, "S
It is possible to select and use "traffic route".

As described above, in the automatic document classification system using the thesaurus, by incorporating a processing program for selecting an optimal semantic category and eliminating polysemy, noise in learning data is reduced and a more accurate document classification is performed. It is possible to generate a knowledge base.

[0014]

An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of an automatic document classification system according to an embodiment of the present invention. The automatic document classification system includes a feature extracting mechanism 10 and a relevance calculating mechanism 1
1, the ambiguity eliminating mechanism 12, the knowledge base generating mechanism 13 for document classification, and the classification processing mechanism 14 that outputs a classification label to an unclassified document, the apparatus main body 1, a document file 20 with a classification tag, a thesaurus dictionary 21, a polysemy. It is composed of a group of external storage devices for storing a learning data file 22 before resolving sex, a relevance data file 23, a learning data file 24 after resolving ambiguity, a knowledge base 25 for document classification, and the like.
The apparatus main body 1 is a so-called computer including a CPU, a RAM, a built-in hard disk, and the like.

In the configuration shown in FIG.
Receives as input the classification tag-added document file 20 and the thesaurus dictionary 21 and outputs a document representing a document and its classification category information to a learning data file 22 before disambiguation. The relevance calculating mechanism 11 receives the learning data file 22 before disambiguation as an input and outputs relevance information between the feature and the classification category to the relevance data file 23. The disambiguation mechanism 12 receives the learning data file 22 before disambiguation and the relevance data file 23 as input, and outputs the disambiguation result to the learning data file 24 after disambiguation. Knowledge base generation mechanism for document classification 1
Reference numeral 3 receives, as an input, the learning data file 24 after disambiguation, and outputs the recalculated feature and the degree of association between the classification categories to the knowledge base 25 for document classification. Classification processing mechanism 1
Numeral 4 receives the knowledge base 25 for document classification and the new text input and input by the user, and outputs a classification category corresponding to the input text. Hereinafter, the configuration and operation of each mechanism 10, 11, 12, 13, 14 will be described in detail.

<Feature extraction mechanism 10> Feature extraction mechanism 10
Analyzes a document set with a classification tag given as an input to the file 20, extracts words whose part of speech is a noun or a fixed noun, and then refers to the thesaurus dictionary 21 to check the meaning category of those words. Then, a feature vector composed of words and semantic categories included in the text is extracted from one text, and the data composed of the feature vector and the classification category to which the text belongs is output to the learning data file 22 before disambiguation. I do.

FIG. 3 is a block diagram showing one embodiment of the feature extracting mechanism 10. As shown in FIG. The feature extraction mechanism 10 includes a morphological analyzer 10
1, a word extraction unit 102 and a meaning category assignment unit 103.

The morphological analysis unit 101 performs a morphological analysis on the text input from the file 20, and performs word extraction and its part of speech. The word extracting unit 102 selects words of necessary parts of speech (nouns, proper nouns). The semantic category assigning unit 103 examines the thesaurus dictionary 21 to be used for the word selected by the word extracting unit 102, assigns the semantic category, and assigns the feature vector including the word and the semantic category to the classification category to which the text belongs. (Learning data before disambiguation) is output to the file 22.

FIG. 4 shows a specific processing example of the feature extracting mechanism 10. Here, the character "S" is added to the head of the semantic category to distinguish it from a word. For example,
If the thesaurus 21 has two meaning categories of "traffic road" and "form" in the thesaurus dictionary 21,
"S traffic route" and "S form" are added. "S party", "S
The same applies to "organizations."

<Relationship Calculation Mechanism 11> Relevance Calculation Mechanism 1
1 receives the learning data file 22 before disambiguation as an input and outputs the relevance information between the feature and the classification category to the relevance data file 23.

FIG. 5 shows a configuration diagram of an embodiment of the relevance calculating mechanism 11. The relevance calculating mechanism 11 includes a case counter module 111, a chi-square calculating module 11
2. It is composed of an association degree storage module 113.

The case number counter module 111 receives the learning data file 22 before disambiguation as an input and counts the following four values for each feature for each classification category. That is, in the text set in which the corresponding feature appears, the number of texts having the corresponding category “Nr +”
The number of texts that do not have “Nn +”, the number of texts that have the corresponding category in the text set where the feature does not appear “Nr−”, the number of texts that do not have “Nn−”
-”Is required.

The chi-square calculation module 112 calculates
Using these two values, a chi-square value (degree of association) is calculated based on the following equation. Here, N represents the total number of cases.

[0024]

(Equation 1)

The relevance storage module 113 stores the calculation results of the chi-square calculation module 112 (the chi-square values for each classification category and for each feature) in the relevance data file 2.
3 is stored.

<Semantic elimination mechanism 12> Ambiguity elimination mechanism 1
2 receives the learning data file 22 before the disambiguation and the relevance data file 23 as input and outputs the disambiguation result to the learning data file 24 after the disambiguation.

FIG. 6 is a block diagram showing an embodiment of the disambiguation mechanism 12. The polysemy elimination mechanism 12 includes a polysemy word selection module 121, a relevance reference module 122, and an optimal meaning category selection module 123.

The polysemy word selection module 121 uses the learning data file 22 before ambiguity resolution as an input, searches for polysemy words to which a plurality of semantic categories are assigned, and outputs the semantic category and the classification category to which the text belongs. . The relevance reference module 122 receives the relevance data file 23 and outputs the relevance corresponding to the semantic category and the classification category obtained by the module 121. The optimal meaning category selection module 123 selects the one having the largest value of the degree of association as the optimal meaning category based on the association destination obtained in the module 122, and outputs it to the learning data file 24 after disambiguation.

For example, in the case of FIG. 4, of the two semantic categories "S traffic road" and "S form" corresponding to the polysemy word "route", "S form" is selected as the optimal meaning category.

<Knowledge Base Generating Mechanism 13 for Document Classification>
The document classification knowledge base generation mechanism 13 receives the learning data file 24 after disambiguation as an input and outputs the weight between the feature and the classification tag to the document classification knowledge base 25.
As an example of the embodiment, it is assumed here that document classification is performed based on a linear classification model, and an error-driven learning algorithm is used for learning weights in the linear model. The linear classification model is composed of a feature set node and a classification category node, and determines whether an input case belongs to the classification category by determining the sum of the weight between the feature node of the input case and the classification category. Is greater than a predetermined threshold. FIG. 7 shows an example of the linear classification model. The linear classification model and the error-driven learning algorithm are described in, for example, "N. Littlestone,"
ning quickly when irrelevant attributes abound: A
new linearthreshold algorithm ”, Machine Learni
ng, No. 2, pp 285-318, 1988. ".

FIG. 8 is a block diagram showing an embodiment of the knowledge base generating mechanism 13 for document classification using an error-driven learning algorithm. Knowledge base generation mechanism for document classification 1
Reference numeral 3 includes a weight initial value setting module 131, a weight summation module 132, a correct answer determination module 133, and a weight update module 134.

The weight initial value setting module 131 receives the learning data file 24 after disambiguation as an input, initializes features appearing in the learning data and classification categories to a certain value, and outputs the values to the knowledge base 25 for document classification. . The weight summation module 132 receives as input the learning data file 24 after disambiguation and the knowledge base 25 for document classification, and calculates, for each input case, a score for each classification category as a sum of weights from features that appear in the case. I do. The correct answer determination module 133 determines whether or not the score calculated in the previous module exceeds a predetermined threshold, and assigns a classification category. The weight update module 134 updates the weight between the feature and the classification category only when the classification category assigned by the previous module is different from the correct classification category, and updates the knowledge base 2 for document classification.
5 is output.

<Classification processing mechanism 14> Classification processing mechanism 14
Is a thesaurus dictionary 21 and a knowledge base 25 for document classification.
And a new text input by the user as an input, and outputs a classification tag corresponding to the input text.

FIG. 9 is a block diagram showing an embodiment of the classification processing mechanism 14. The classification processing mechanism 14 includes a feature extraction module 141, a weight summation module 142, and a classification category generation module 143.

The feature extraction module 141 performs morphological analysis on a new unclassified text input by the user, selects words that are nouns and fixed nouns, and assigns semantic categories of the words with reference to the thesaurus dictionary 21. , Generate a feature vector. This module 141 is basically the same as the feature extraction mechanism 11. The weight summation module 142 calculates a score for each classification category by using the feature vector generated in the module 141 and the knowledge base 25 for document classification as inputs. The classification category generation module 143 outputs a classification category in which the score calculated in the previous module is equal to or more than a predetermined threshold.

The automatic document classification system according to one embodiment of the present invention has been described above. In FIG. 1, a feature extraction mechanism 1 related to generation of a knowledge base for document classification is described.
0, the relevance calculating mechanism 11, the disambiguation mechanism 12, and the processing program of the document classification knowledge base generating mechanism 13 may be integrated into one. FIG. 10 shows the processing flow, and processings 1010 to 1013 correspond to each mechanism 10 shown in FIG.
~ 13. The programs of the processes 1010 to 1013 are recorded in advance on a recording medium such as a CD-ROM, and the same processes as those of the mechanisms 10 to 13 of FIG. I do.

[0037]

As described above, according to the present invention,
When creating learning data to be used to generate a classifier (a knowledge base for document classification) from a set of documents with classification tags and a thesaurus, only the most appropriate semantic category is selected from the semantic categories of the words. A classifier with higher accuracy can be generated as compared with the case where the ambiguity is not eliminated by the conventional method.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram of an automatic document classification system according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a classifier.

FIG. 3 is a configuration diagram of one embodiment of a feature extraction mechanism of FIG. 1;

FIG. 4 is a diagram showing a specific processing example of a feature extraction mechanism.

FIG. 5 is a configuration diagram of an embodiment of an association degree calculation mechanism of FIG. 1;

FIG. 6 is a configuration diagram of an embodiment of the disambiguation mechanism of FIG. 1;

FIG. 7 is a diagram illustrating an example of a linear classification model.

8 is a configuration diagram of an embodiment of a knowledge base generation mechanism for document classification in FIG. 1;

FIG. 9 is a configuration diagram of an embodiment of the classification processing mechanism of FIG. 1;

FIG. 10 is a schematic processing flowchart of a method for generating a knowledge base for document classification according to the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Document automatic classification apparatus main body 2 External storage device group 10 Feature extraction mechanism 11 Relevance calculation mechanism 12 Ambiguity elimination mechanism 13 Knowledge base generation mechanism for document classification 14 Classification processing mechanism 20 Document file with classification tag 21 Thesaurus dictionary 22 Ambiguity elimination Previous learning data file 23 Relevance data file 24 Learning data file after disambiguation 25 Knowledge base for document classification

Claims (3)

[Claims] 1. An automatic document classification system using a thesaurus dictionary, which receives a document set with a classification tag and a thesaurus dictionary as input, and generates learning data composed of words characterizing a document, their semantic categories, and the category of the document. Means for inputting the generated learning data as input, and calculating the degree of association between the classification category and the feature, and inputting the calculated degree of association and the learning data as input, and extracting an inappropriate feature from the learning data. Means for removing and generating new learning data, means for receiving the generated new learning data as input, calculating weights between classification categories and features, and generating a knowledge base for document classification, Means for inputting a document and outputting a corresponding classification category based on the knowledge base. M 2. A method of generating a knowledge base for document classification in an automatic document classification system using a thesaurus dictionary, comprising the steps of: inputting a document set with a classification tag and a thesaurus dictionary; Generating learning data composed of the classification categories of the document; calculating the relevance between the classification categories and the features with respect to the learning data; removing inappropriate features from the learning data based on the relevance; A method for generating a knowledge base for document classification, comprising: generating new learning data; calculating a weight between a classification category and a feature based on the new learning data; and generating a knowledge base for document classification. 3. A computer-readable recording medium recording a program for generating a document classification knowledge base in an automatic document classification system using a thesaurus dictionary, comprising: a process of reading a document set with a classification tag; Analyzing the tagged document set, extracting words characterizing the document, examining the semantic category of the word with reference to a thesaurus dictionary, and comprising a feature vector composed of the word and the semantic category and a classification category of the document Generating learning data, calculating the relevance between a classification category and a feature for the learning data, and removing inappropriate features from the learning data based on the relevance to obtain new learning data. And the weight between the classification category and the feature is calculated based on the new learning data. To, computer-readable recording medium storing a program that includes a process of generating a document classification for the knowledge base, a.