JPH11167581A - Information sorting method, device and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information sorting device which can sort the texts with high accuracy. SOLUTION: An information sorting device 1 includes a text input part 11, a word processing part 12, a vector processing part 13, a learning feature vector set file 14, a similarity processing part 15, a category decision part 16 and an external or internal document data base 17. The part 12 calculates the importance of category of every word that is extracted from a learning text based on both number of appearance and category frequencies of the word. The part 15 calculates the similarity of words based on the learning feature vector, the learning feature vector set and the sorting object text feature vector which are calculated based on the importance of words calculated at the part 12. The part 16 decides a prescribed number of corresponding categories as the categories of the sorting object texts based on the similarity having the largest calculation value. Then the sorting object texts sorted in each category are stored in the" data base 17.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information classification method for efficiently classifying digitized text groups in the field of natural language processing and information retrieval technology.

[0002]

2. Description of the Related Art As a method of classifying digitized information, a method of determining a category by comparing the characteristics of a vector-represented category with unclassified digitized information is known. Hereinafter, an outline of this method will be described. As a precondition, it is assumed that N types of categories C1, C2,..., CN are respectively provided in advance to a group of learning electronic information (hereinafter, learning text) used for extracting a feature of a category. .

First, in order to create a feature vector pi representing a feature of a category Ci (1 ≦ i ≦ N), a category C
A word is extracted from the learning text group to which i is assigned. Then, the importance of each word in the learning text is determined for each category. As a method of determining the importance, the TF / IDF method proposed in the field of information retrieval is widely known (“Introduction to Modern Information Retriev”).
al: by G. Sallton, McGraw-Hill "). This TF ・ I
The importance of a word in the DF method is defined using the appearance frequency tf and the reciprocal idf of the number of appearances df. Specifically, the importance level W (tk, tk) of the word tk in the category Ci
Ci) is calculated by the following equation (1). W (tk, Ci) = tf (tk, Ci) log (Li / df (tk, Ci) +1) (1) where tf (tk, Ci) is the frequency of occurrence of word tk in category Ci. And df (tk, Ci) is
It represents the number of occurrences of the word tk in the category Ci. On the other hand, Li represents the total number of texts in the category Ci.

Next, the importance in the category Ci is calculated for each of the words t1, t2,..., And tM appearing in the learning text set by the above equation (1), and a vector having the calculated importance as an element is calculated. Is the feature vector pi of the category Ci. The feature vector q is similarly calculated for the unclassified text T. In this case, the appearance frequency tf is mainly used as the importance of the word as an element of the feature vector. To determine the category of the uncategorized text T, the similarity d (p) between the feature vector pi (1 ≦ i ≦ N) of each category and the feature vector q of the uncategorized text T is determined.
i, q) are used. As a typical example of the similarity calculation, a method of calculating an inner product of both vectors and a method of using a set-theoretic measure are known, and are described in "Information Search: Tetsuro Ito, Shokodo". It has been described.

As described above, the above-described similarity d
By calculating and using (pi, q), a plurality of categories having similar degrees of similarity to the unclassified text T are selected, and the category to be classified is determined.

[0006]

As described above, TF.
The IDF method is, for example, a method used for creating a vector for comparing a search word with a text in a search database, and the reciprocal id of the number of appearances increases as the appearance frequency tf increases.
The larger the value of f, that is, the smaller the number of occurrences df, the higher the importance.

However, in the classification of texts, a group of texts for which a vector is to be created usually belongs to the same category, and the characteristics of the text are different from those of the information retrieval which does not consider the categories. Therefore, when focusing on texts belonging to the same category, an important word that becomes a feature of the category (hereinafter, characteristic word) may appear in many texts, that is, the number of occurrences df may be large. This means that i using the reciprocal of the number of occurrences df
In the TF / IDF method using df, it means that the feature word may be given low importance. As a result, when the importance of a word is determined by using the TF / IDF method, there is a problem that it is difficult to create a feature vector that clearly expresses a feature of a category, and the classification accuracy is reduced.

On the other hand, even when the number of occurrences df is used,
The word group having a large number of occurrences df includes general words (hereinafter, general words) that appear without depending on the category, and words having a large number of occurrences df are not necessarily feature words. For this reason, the number of occurrences df itself is rarely used for the importance of the characteristic word.

It is an object of the present invention to provide a new information classification method which enables highly accurate classification by considering the importance of words which are characteristic words of a category in a learning text. Another object of the present invention is to provide an information classification device and an information classification system suitable for implementing the information classification method.

[0010]

In order to solve the above-mentioned problems, the present invention extracts words from learning texts to which categories to which they belong are known, and for each extracted word, the number of occurrences and the number of appearing categories are reduced. A process of generating a learning feature vector representing a feature for each category using the calculated importance as an element, and a process for each word in the classification target text for which the category is unknown. A process of calculating importance based on the frequency of appearance and generating a classification target feature vector representing a feature for each text using the calculated importance as an element; and a similarity between the classification target feature vector and the learning feature vector for each category. And determining whether the similarity with the text to be classified is within a predetermined range or a learning feature vector or a similarity. And Category candidates to be given to the classified text category corresponding to a predetermined number or more training feature vectors from providing information classification methods.

In this information classification method, the step of generating the learning feature vector does not depend on, for example, a feature word serving as an index indicating a feature of a category and a category by focusing on the appearance tendency of a word in the learning text. Determining a common word and reducing the importance of the common word based on the number of categories in which the word appears, thereby generating a learning feature vector in which the importance of the characteristic word is reflected relatively high. And

According to the information classification apparatus of the present invention for solving the above-mentioned other problems, a category to be assigned to a classification target text whose category is unknown is determined in accordance with a classification system of a learning text to which one or more categories are assigned. This is an apparatus for performing a classification process by determining, and includes the following elements. (1) Word processing means for extracting words from each of the learning text and the classification target text, and calculating the importance of each extracted word. This word processing means, for example,
Means for calculating a category frequency coefficient based on division of the total number of categories in the learning text by the number of categories in which a specific word appears, wherein the number of occurrences of words appearing in a specific category and the category frequency coefficient The importance of each word in the learning text is calculated by multiplication, and the importance of words having a relatively large number of occurrences and relatively little dependence on categories is reduced. Further, the importance of each word in the learning text is calculated by multiplying the calculated value obtained by multiplying the number of occurrences of the word appearing in the specific category by the category frequency coefficient by the appearance frequency of the word. It is configured as follows. Alternatively, there is provided a means for measuring the frequency of appearance of the words in the text to be classified, so that the lower the frequency of appearance, the higher the importance in the text to be classified. (2) Vector processing for generating a learning feature vector expressing the features of the learning text for each category and a classification target feature vector expressing the characteristics of the classification target text for each text, using the importance of each word as an element. means. (3) Similarity processing means for determining the similarity of the classification target feature vector to the learning feature vector for each category based on the feature difference between each classification target feature vector and the learning feature vector. The similarity processing means calculates a cosine of both vectors based on, for example, an inner product between each learning feature vector and a classification target feature vector, and arranges the calculated values of the cosine in a predetermined order to obtain a characteristic difference between the two vectors. Is configured to quantify (4) Based on the determination result by the similarity processing means,
Category determining means for determining a category to be assigned to the text to be classified.

Preferably, the apparatus further comprises presentation means for visually recognizing and presenting a category corresponding to one or a plurality of learning feature vectors whose similarity to the classification target text falls within a predetermined range. In this case, the category determination unit is configured to determine a category specified in correspondence with the presentation by the presentation unit as a category to be assigned to the classification target text.

According to another aspect of the present invention, there is provided an information classification system, comprising: an information classification device according to the invention; and a text input unit for inputting the classification target text distributed via a communication line into the information classification device. It is characterized by having. It is preferable that the text input unit is configured to input the classification target text to the information classification device through an agent function.

[0015]

Embodiments of the present invention will be described below in detail with reference to the drawings. (First Embodiment) FIG. 1 is a functional block diagram showing an embodiment of an information classification device according to the present embodiment. The information classification device 1 of the present embodiment includes a document database 17 built in a stand-alone computer device or in an external storage device, and a text input unit formed by reading and executing a predetermined program by the computer device. 11, word processing unit 12, vector processing unit 1
3, a learning feature vector set file 14, a similarity processing unit 15, and a category determination unit 16.

The program is usually stored in an internal storage device or an external storage device of the computer, and is read and executed at any time.
A recording medium that can be separated from the computer device, for example, a CD
-Stored in a portable recording medium such as a ROM or FD, or a program server connected to the computer device and a private network, and installed in the internal storage device or the external storage device at the time of use and provided for execution at any time; It may be.

The document database 17 stores a plurality of digitized document data (hereinafter, text). The text group includes a learning text group (hereinafter, learning text) stored in advance, and a classification result of one or more texts (hereinafter, classification target texts) to be newly classified with respect to the learning text. Is configured to be stored.

Further, it is assumed that one or a plurality of N types of categories C1, C2,..., CN are previously assigned to the learning text for each text. The learning text with the category is input to the word processing unit 12.

The text input unit 11 receives an input of a text to be classified and inputs the text to the word processing unit 12 by input means (not shown). Word processing unit 12
Is to perform a predetermined morphological analysis on an input text to extract words, and to assign importance to each of a plurality of extracted words. The word group to which the importance is assigned is input to the feature vector processing unit 13. How to assign importance will be described later.

The vector processing unit 13 extracts a feature vector or a set of feature vectors for each category using the importance given by the word processing unit 12 as an element. A feature vector set extracted from the learning text (hereinafter, learning feature vector set) is input and held in the learning feature vector set file 14, and the feature vector extracted from the classification target text is sent to the similarity processing unit 15. Is to be entered.

The similarity processing unit 15 calculates the similarity of the classification target text to the learning text for each category based on the feature vector corresponding to the classification target text and the feature vector set corresponding to the learning feature vector set file 14. It is to be calculated. The calculated similarity is input to the category determination unit 16. The similarity calculation processing will be described later.

The category determining section 16 determines a category to be assigned to the text to be classified based on the calculated similarity for each category. This category determination unit 1
6 is, for example, sequentially presented to the user through a display device or the like (not shown) from the one having the highest similarity, and the category specified by the user is determined as the category to be assigned to the text to be classified based on the presentation. It is composed of In this way, even when the user or the like has only a vague image of the information required, the necessary information can be searched in order from the highest similarity to the lowest similarity. It can be easily obtained.
The category determining unit 16 is configured to add the determined category to the text to be categorized and send it to the document database 17. Thereby, the document database 17 can accumulate the classification target text for each category.

Next, an information classification method using the information classification apparatus 1 of the present embodiment will be described focusing on processing of assigning importance to the learning text and the text to be classified, creating a feature vector, and determining the similarity. I do. In the word processing unit 12, first, a plurality of words tk (1
≦ k ≦ M), and the number of occurrences df (t (t) of the word tk in the learning text belonging to the category Ci (1 ≦ i ≦ N)
k, Ci) are calculated. The number of occurrences is calculated by calculating the number of occurrences df (t1, Ci), df (t2, Ci),..., D corresponding to all the extracted words t1, t2,.
f (tM, Ci) are each calculated.

Here, a word group having a large number of occurrences df is
There is a problem that characteristic words and general words are mixed as described above, not necessarily only important words in a category. Specifically, it is considered that a characteristic word represents a high number of occurrences only in a specific category, while a general word represents a high number of occurrences commonly in many categories. Therefore, the word processing unit 12 defines the category frequency cf in order to determine the generality of the word. For example, a category frequency cf (t) when a specific word tk appears in n categories in all category numbers N
k) is represented by n (n ≦ N). That is, the number of categories in which a specific word appears can be defined as the category frequency of the word. The larger the category frequency cf (tk) is, the more the word tk can be specified as a general word having little dependence on the category.

Next, the importance W (tk, Ci) of the word tk in the category Ci is calculated by, for example, the number d of occurrences of the word.
f and the value icf using the reciprocal of the category frequency cf
Using (category frequency coefficient), it is defined as in the following equations (2) and (3). W (tk, Ci) = df (tk, Ci) × icf (tk) ・ ・ ・ (2) icf (tk) = log (N / cf (tk)) ・ ・ ・ (3) Number of occurrences df and category frequency By using the above expression (2) based on cf, it is possible to reduce the importance of general words in a word group having a high number of occurrences df, and to increase the importance of words serving as characteristic words. Can be granted. FIG. 2 is a conceptual diagram illustrating calculation of the importance of a word.

In addition to the above equation (2), the importance of a word is defined so as to be calculated by fusing it with a parameter used by a conventional method, for example, by further multiplying the appearance frequency tf of the word. You can also.

FIG. 3 is an explanatory diagram of a procedure for extracting a feature vector corresponding to a learning text. Specifically, the feature vector pi of the category Ci in the learning text can be calculated by the following equation (4) using the importance of the word defined by the above equation (2) as each element. pi = (W (t1, Ci), W (t2, Ci),…, W (tM, Ci)) ・ ・ ・ (4)

The vector processing unit 13 determines the feature vectors p1, p2,..., PN for all the categories C1, C2,..., CN based on the above equation (4) based on the number of occurrences df and the category frequency cf. Are calculated (steps S101 to S102). A set composed of these category-specific feature vectors, that is, a learning feature vector set is
It is stored in the learning feature vector set file 17 (step S103).

On the other hand, a feature vector q in an unclassified text, ie, a classification target text T to which no category is assigned, is
q = (W ′ (t1), W ′ (t2),..., W ′ (t
M)). Here, W ′ (tk) is the importance of the word tk in the classification target text T, and is, for example, the frequency of occurrence tf of the word in the classification target text T
It is calculated based on the above.

The feature vector q of the text T to be classified
, The similarity processing unit 15 calculates the similarity of the classification target text T with respect to the category of the learning text.
This similarity can be calculated, for example, by the following equation (5) using the inner product between known vectors adopted in the conventional method.

[0031]

(Equation 1)

"D (pi, q)" in the above equation (5)
Represents the cosine of the angle formed by both feature vectors, and its value is in the range of “−1 ≦ d (pi, q) ≦ 1”. The larger the cosine d (pi, q), the closer the directions pointed to by both feature vectors, in other words, the higher the possibility that the classification target text T belongs to the category Ci. The cosine d (pi, q) is the similarity, and the category determination unit 16 determines the category to be classified in a predetermined order from the category determined to have a high similarity with the text T to be classified. .

FIG. 4 is an explanatory diagram of the procedure of the classification process of the text to be classified. Here, it is assumed that the learning feature vector set in the learning text has already been extracted and is stored in the learning feature vector set file 14.

The text to be classified is input to the text input unit 11.
The word is then input to the word processing unit 12 and a word is extracted.
Then, the appearance of each extracted word in the text
The frequency and the importance based on the appearance frequency are calculated. Be
The vector processing unit 13 requires the calculated importance of each word.
Extract the feature vector q of the text to be classified as a prime
(Step S201). Note that if the text to be classified
In the case of a number, a feature vector q is extracted for each text.
You. The similarity processing unit 15 calculates the feature vector of the classification target text.
Torq and each feature in learning feature vector set file 14
Similarity Di to vector pi (= d (Vector pi,
Vector q)) for each category
(Steps S202 to S203).

After the similarity Di is calculated, the category determining unit 16 sorts the similarities in descending order of the calculated value (step S204), and selects a predetermined number from the one having the largest calculated value. Then, a category group related to the calculated value is determined as a category candidate to which the classification target text belongs. A category group in which the calculated value falls within a predetermined range may be set as a category candidate to be assigned to the text to be classified. As a result, the text to be classified is classified by the category (step S205), and the document database 1 is classified.
7 is stored. Note that the category determination in steps S204 to S205 focuses on the magnitude of the calculated value of the similarity. However, without being limited to this example, the category determination threshold should be set as appropriate and determined. It is also possible to narrow down the categories.

As described above, the information classification device 1 of the present embodiment
In the above, the number of occurrences and the category frequency (or category frequency coefficient) are used when determining the importance of a word in the learning text, so that a candidate word to be a characteristic word of the category can be easily selected. Was.

Further, since the ratio of words appearing in all categories is reflected in the importance, the importance of common words in a word group having a high number of occurrences is reduced, and the importance is higher than that of common words. Can be added to words. As a result, the quality and the classification accuracy of the learning feature vector are greatly improved.

(Second Embodiment) The present invention functions as a system for automatically classifying a large amount of computerized information distributed through a public network such as the Internet, for example, as the above-mentioned information classifying apparatus. It is also possible to implement the present invention in the form of an information classification system in which the information classification server and the client functioning as an information acquisition device are deployed. The information classification server in this case is positioned, for example, as a search engine for a plurality of large-scale databases on the Internet environment.

This information classification server, like the information classification device 1 of the first embodiment, builds the same database as the document database 17 in the internal or external storage device of the computer device, and communicates with the client via the public network. A communication control unit for performing communication, and a functional block similar to that of the information classification device 1; a text input unit 11;
It comprises a word processing unit 12, a feature vector processing unit 13, a learning feature vector set file 14, a similarity processing unit 15, and a category determination unit 16 (codes are in accordance with FIG. 1).

This information classification server is the information classification device 1
This is different from the first embodiment in that a known communication control unit for performing communication control is provided. The computerized information group distributed through the communication control unit is input to the text input unit 11 and a classification request from the client is input. It is configured to receive. For this classification request, for example, information for identifying digitized information to be classified may be used. Similarly, the classification result can be replaced by transmitting the classification result to the client via the communication control unit, and the same effect as that of the information classification device 1 can be obtained. As a classification result in this case, for example, a category to which the target text belongs may be used.

Further, by using an agent function under the Internet environment as a text means to the information classification server, a system construction capable of automatically performing information classification and management for a large amount of electronic information information distributed. Becomes possible. Therefore, for example, even when the user on the client side has only a vague image of the text required, the classification processing is sequentially performed from the upper level related to the text classification to the lower level, and the progress is performed. , It is possible to easily obtain necessary information.

[0042]

As is clear from the above description, according to the present invention, the learning feature vector can be clearly expressed.
High-precision classification becomes possible. Further, by automatically performing the classification process of the present invention in accordance with the existing classification system in the learning text, it is possible to easily search and utilize information required by a user or the like. Further, when the present invention is adapted to an information search system or the like, it becomes possible to provide a system in which the efficiency and practicality of search processing are significantly improved.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of an information classification device according to an embodiment of the present invention.

FIG. 2 is a conceptual diagram illustrating calculation of importance of a word.

FIG. 3 is a processing procedure diagram for creating a learning feature vector set.

FIG. 4 is a processing procedure diagram in classification processing.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 information classification device 11 text input unit 12 word processing unit 13 feature vector processing unit 14 learning feature vector set file 15 similarity processing unit 16 category determination unit 17 document database

Claims (11)

[Claims] 1. A method for extracting words from a learning text to which a category to which a category belongs is known, calculating, for each extracted word, the number of occurrences and the importance based on the number of appearing categories, and calculating the calculated importance as an element. A process of generating a learning feature vector representing a feature for each category, and calculating importance based on the frequency of appearance of each word in the classification target text for a category unknown text, and calculating the calculated importance. Generating a classification target feature vector representing a feature of each text using as an element, and determining a similarity between the classification target feature vector and the learning feature vector for each category, wherein the similarity to the classification target text is included. A category corresponding to a learning feature vector having a degree within a predetermined range is set as a category candidate to be assigned to the text to be classified. Information classification method. 2. A method according to claim 1, wherein a category corresponding to a predetermined number or more of learning feature vectors from a higher rank when arranged in descending order of similarity with the classification target text is set as a category candidate to be assigned to the classification target text. 2. The information classification method according to claim 1, wherein: 3. The process of generating the learning feature vector includes: determining a feature word serving as an index representing a feature of a category and a general word that does not depend on the category by focusing on an appearance tendency of a word in the learning text; 2. The learning feature vector in which the importance of the feature word is reflected relatively high by reducing the importance of the common word based on the number of categories in which the word appears. Information classification method. 4. An apparatus for performing a classification process by determining a category to be assigned to a classification target text whose category is unknown according to a classification system of a learning text to which one or a plurality of categories have been assigned, Word processing means for extracting a word from each of the learning text and the classification target text and calculating the importance of each extracted word; and using the importance of each word as an element, the features of the learning text are classified for each category. Vector processing means for generating a classified learning feature vector and a classification target feature vector expressing the characteristics of the classification target text for each text; and for each category based on a feature difference between each classification target feature vector and the learning feature vector. Similarity processing means for determining the similarity of the classification target feature vector to the learning feature vector of According determination result based, information classification device characterized by and a category determining means for determining the category to be assigned to the classified text. 5. The word processing means has means for calculating a category frequency coefficient based on division of the total number of categories in the learning text by the number of categories in which a specific word appears, and appears in a specific category. The importance of each word in the learning text is calculated by multiplying the number of occurrences of the word by the category frequency coefficient, and the importance of the word having a relatively large number of occurrences and relatively little dependence on the category is calculated. 5. The information classification device according to claim 4, wherein the information classification device is configured to reduce the number of times. 6. The learning text in the learning text by multiplying a value calculated by multiplying the number of occurrences of a word appearing in a specific category by the category frequency coefficient by an appearance frequency of the word. 5. The information classification device according to claim 4, wherein the information classification device is configured to calculate the importance of each word. 7. The word processing means includes means for measuring the frequency of appearance of a word in the text to be classified, and is configured such that a word having a lower frequency of occurrence has a higher importance in the text to be classified. 5. The method according to claim 4, wherein
Described information classification device. 8. The similarity processing means calculates a cosine of both vectors based on an inner product between each of the learning feature vectors and the classification target feature vector, and arranges the calculated values of the cosine in a predetermined order to obtain the two vectors. 5. The information classification device according to claim 4, wherein the information classification device is configured to quantify a characteristic difference of the information. 9. A presentation unit for visually recognizing and presenting a category corresponding to one or a plurality of learning feature vectors whose similarity to the classification target text is within a predetermined range, wherein the category determination unit includes: 5. The information classification apparatus according to claim 4, wherein a category specified in response to the presentation by the means is determined as a category to be assigned to the text to be classified. 10. An information classifying apparatus according to claim 4, further comprising: a text input unit that takes in the classification target text distributed via a communication line into the information classifying apparatus. An information classification system characterized by the following. 11. An information classification system according to claim 11, wherein said text input means is configured to input said classification target text to said information classification device through an agent function.