JP2002108894A - Device and method for sorting document and recording medium for executing the method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document sorting device which can supply information effective for analyzing a partial document set and also can extract many analysis information from a document set. SOLUTION: A document analysis part 102 extracts word information from a document that is inputted from a document input part 101 and a document sorting part 103 sorts the document into a partial document set on the basis of the word information. A keyword extraction part 104 extracts a keyword set from the partial document set and a relative word extraction part 105 extracts a relative word set of the partial document set by using a relative word dictionary. A partial document set information generation part 106 generates information on each partial document set and the relative information on these partial document sets on the basis of information on the relative word set, the keyword set and the document set of partial document set. Then a sorting result preservation part 107 preserves the sorting result of the part 103 and information generated at the part 106.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document classification apparatus, a document classification method, and a recording medium for executing the method, and relates to a document classification technique applicable to information classification, information analysis, information retrieval, and the like.

[0002]

2. Description of the Related Art With the spread of the Internet and the like, access to a large amount of document information becomes possible, and intellectual work such as classifying a large amount of collected document information into meaningful groups and grasping the structure of a document set is performed. It is beginning to take place. When analyzing a large document set, the document set is first classified into several topics, and various tasks are performed on the obtained partial document set (a plurality of documents collected according to a certain standard). It is considered that work can be performed efficiently. If users manually classify large amounts of document information,
Since human / time costs are enormous, a device that can automatically classify a document set according to the content of the document is desired.

Conventionally, inventions for obtaining high-quality classification results from a huge document set have been widely performed. For example, the invention described in Japanese Patent Application Laid-Open No. 7-36897 uses a document feature vector having words included in a set of documents to be classified as feature amounts, and performs classification by applying a clustering method to the document feature vector. Things. The above invention also suggests that the user designates an initial center-of-gravity vector for clustering in order to perform classification reflecting the user's intention.

The invention described in Japanese Patent Application Laid-Open No. H11-296552 discloses a method of sharing words between documents by applying singular value decomposition to an inner product matrix between documents in order to consider the ambiguity / synonymity of words. Generate a latent semantic space based on the origin,
Documents and words are projected into a potential semantic space, and document classification is performed in the potential semantic space using a clustering method or the like. Various inventions have been proposed to obtain high-quality classification results from such a huge document set. However, in order to analyze the document set, it is not enough to classify the document set, and An important issue is how to extract effective information from the partial document set, but there are few inventions on this point.

[0005] In addition, it is possible to spatially express a document as a word frequency vector (document feature vector) by extracting words constituting the document from the document using natural language processing such as morphological analysis. Is called a document vector space model and is widely used. The invention of Japanese Patent Laid-Open No. 7-36897 described above classifies documents in such a document vector space by applying a clustering method.

When performing a document classification process or a document search process using a statistical method in the document vector space as described above, it is considered that the quality of the obtained result is different if the document vector space is different. Whether to generate a document vector space is an important issue in obtaining high-quality processing results.

As described above, since each axis of the normal document vector space is configured based on words extracted as a result of applying morphological analysis to the document data to be classified, for example, Japanese Patent Laid-Open No. 11-110408, JP-A-11-25948
The invention typified by Japanese Patent Publication No. 7-203, discloses a method in which morphological analysis is applied to a search query word and a search target document, and compound words are generated from words extracted as a result under appropriate conditions. Is also used to generate the document vector space, thereby improving the accuracy of document search performed in the document vector space. Therefore, even when the document classification process is performed in the document vector space, it is expected that a high-quality classification result is obtained by generating the document vector space in consideration of the compound words.

By the way, when a compound word is considered, including the above-mentioned prior application, the part of speech is a noun or something similar thereto, but not only the noun but also other possible parts of speech are appropriately combined. It is considered that this makes it possible to configure a higher-quality document vector space. In other words, words that have not been treated so much in the prior invention, etc., are generated by combining words with prefixes, suffixes, classifiers, and parts of speech similar to them with words before and after them with appropriate criteria. Consider replacing words with appropriate words, and replacing parts of speech with appropriate ones.

For example, when a morphological analysis is applied to a character string "Car made in Italy" and the result is "[Particle], car in Italy [common noun], made [prefix]," Attention is paid to the word "made" which is a prefix, and this is combined with the common noun "Italy" extracted immediately before this to generate the word "made in Italy", which has the part of speech of the common noun The word "made"
Replace with the word And in addition to this string,
Consider generating a vector space consisting of the strings "Italian features" and "Italian dishes".

[0010] Considering that a space is generated only by a noun, if the above-mentioned combination / replacement processing is not performed, the words constituting the vector space are "Italy, car, spot color, plate".
And the character string is (1, 0, 1, 0), (1, 0, 0, 0), when the appearance frequency of the word is considered as a coordinate value.
(1, 0, 0, 1). In this case, when the mutual similarity of the three character strings is calculated by the inner product between the vectors, the mutual similarity of the three character strings becomes the same. On the other hand, if a space is generated with a noun after performing the above-mentioned combination / replacement processing, the words constituting the vector space are "Italy, made in Italy, car, special feature, plate". Similarly, the vectors of the three documents in this space are (1,1,0,
(1,0), (1,0,0,0,0), (1,1,0,
0, 1). In this case, there is a difference in the similarity between the three character strings, and the first character string and the last character string have a higher similarity than the second character string. That is, it is considered that the feature dimension for measuring the meaning limited by the vector space can be added by the combination / replacement process, and the quality of the document classification performed in the vector space can be improved.

Further, a morphological analysis is applied to a character string “goal of 2000”, and a result such as “2000 [numerical], year [classifier], [case particle], goal [common noun]” is obtained. It is assumed that it has been obtained. At this time, attention is paid to the word "year" which is a classifier, and this is combined with the numeral "2000" extracted immediately before this to form "2000".
The word “year” is generated, and this word is replaced with the word “year” as a word having the part of speech of a common noun.
000 ", which is more semantically limited to" 2000 "instead of the words" year "and" 2000 ", which have very vague meanings. Therefore, it is expected that the document vector space can be configured based on more important words as variables.

As described above, the spread of the Internet and the like makes it possible to access a large amount of document data.
As a result, it became possible to easily and easily collect a large amount of document data describing information of interest, but on the other hand, due to the large amount of collected document data, it was possible to collect The task of reading sensitive information has become extremely difficult. For this reason,
Research and development on document search and automatic document classification have been actively conducted for the purpose of easily extracting valid information automatically or semi-automatically from a large amount of document data. In particular, the document classification method is very effective as a method for grasping the structure of the entire document data, considering that a plurality of generated partial document data sets indicate a plurality of topics included in the document data. .

A typical technique developed for the above-mentioned purpose is the Scatter / Gather method (D. Cutting et.
l., Scater / Gather: A Cluster-based Approach to Br
owsing Large Document Collections., Proc. ACM SIGI
R '92). In the Scatter / Gather method, the topic of a document data set is represented by a representative document and a representative word, and sequential clustering is applied to a document set with an unclear topic to divide it into multiple partial document data sets. By understanding the various topics included in the document set,
In order to understand the structure of the document set, it is of course necessary to understand each of the partial document sets included in the document set, but in addition, it is considered that information on the relationship between the partial document sets is also necessary. However, since the Scatter / Gather method only presents information on each partial document set, it is considered difficult to grasp the structure of the document set only by the Scatter / Gather method.

In general, in a document classification method,
The number of partial document sets to be generated is required before execution, but it is extremely difficult to predict the optimal number of partial document sets. Moreover, on the other hand, if the number of generated partial document sets is different, the structure of the generated partial document set also changes. Therefore, in order to obtain necessary information, it is necessary to repeatedly perform document classification while changing the number of partial document sets to be generated. The Scatter / Gather method also offers a solution in this regard, in which the sequential clustering is applied only to the sub-document set for which the user wants to know a more detailed structure, and a new sub-document set is generated. It can be considered that desired information can be obtained by analyzing them in detail, and that this operation makes it easier to understand the structure of the entire document set.

That is, what the user wants to do is grasp the structure of the document set, and it is considered that the act of generating the partial document set does not need to be performed by the user.
Then, the user generates a various number of partial document sets from the document set in advance and calculates the relationship between the generated multiple partial document sets, so that the user can grasp the structure from the beginning. You can concentrate on However, as described above, the Scatter / Gather method does not consider the relation between partial documents.

[0016]

SUMMARY OF THE INVENTION Claims 1 to 5 of the present invention
In the inventions of Nos. 4, 14 to 17, and 27 to 30, the document classification is performed, and the partial document sets generated based on the related word information of the words and the related information thereof are further generated, whereby the partial document set is generated. The purpose is to provide useful information for analysis. Furthermore, by focusing on the antonym as a related word, if the generated partial document set does not include a partial document set that includes the opposite word of the representative word set of each partial document set, a new partial document that includes the opposite word An object of the present invention is to provide a document classification device that can extract more analysis information from a document set by generating the set.

Therefore, the invention of claims 1, 14 and 27 is
By extracting the representative word set of each of the generated partial document sets, further obtaining related words for each of the representative words, and generating the relevant information between each of the partial document sets and the partial document sets based on this information. It is an object of the present invention to provide a document classification device, method, or recording medium that provides information effective for analyzing a partial document set.

The invention according to claims 2, 15 and 28 is a document classification device which mainly provides information on similarity by using at least one combination of synonyms, synonyms and antonyms as related words. It is intended to provide a method or a recording medium.

Claims 3, 4, 16, 17, 29 and 30
Uses an antonym as a related word of the set of representative words of each sub-document set. An object of the present invention is to provide a document classification device, a method, or a recording medium that extracts more analysis information from a document set by extracting the set from a set and making it a new partial document set.

According to a fifth aspect of the present invention, there is provided a document classification apparatus for performing a morphological analysis on a document to be classified and classifying the document to be classified into several document sets based on the obtained analysis result. A high-quality document is obtained by replacing words having the specified part of speech among the words obtained as a result of the analysis with words that are appropriately combined with the words before and after the words, and also replacing the parts of speech with appropriate words. An object of the present invention is to provide a document classification device, a method, or a recording medium capable of forming a vector space, and performing high-quality document classification results by performing document classification using statistical processing in the document vector space.

According to the inventions of claims 6, 19 and 32, a clustering method is used as a statistical method for performing document classification, whereby a high-quality document classification result can be obtained easily and easily. It is intended to provide a recording medium.

According to the seventh, twentieth and thirty-third inventions, among words extracted by applying morphological analysis to a document to be classified, in particular, a part of speech is a prefix, a suffix, a classifier, and the like. An object of the present invention is to provide a document classification device, a method, or a recording medium that can obtain a high-quality document vector space by performing an appropriate combining process on a word that is a part of speech.

According to the eighth, twenty-first and thirty-fourth aspects of the present invention, a high-quality document vector space is generated by generating new words by continuing to combine words until a specific part of speech word appears in the word combining process. It is an object of the present invention to provide a document classification device, method, or recording medium that can obtain the following.

According to a ninth, twenty-second, and thirty-fifth aspect of the present invention, in a word combining process, when a word having a part of speech is a numerical suffix or a classifier, a plurality of words to be combined are deleted and a document vector space is generated. An object of the present invention is to provide a document classification device, method, or recording medium that can obtain a high-quality document vector space by not using information of those words.

[0025] Further, claims 10 to 13, 23 to 2 of the present invention.
In the inventions 6 and 36 to 39, various numbers of partial document sets are generated in advance from the document sets, and the relationship between the generated multiple partial document sets is calculated, so that the user can start the structure of the document set from the beginning. The purpose is to provide information that allows the user to concentrate on the task of grasping the situation.

Therefore, according to the tenth, twenty-third, and thirty-sixth aspects of the present invention, a large number of partial document sets are generated by repeatedly performing document classification processing using the number of partial document sets to be generated as a parameter using a vector space model of the document. In addition, the present invention provides a document classification device, a method, or a recording medium that provides a document classification device that generates information that can assist in understanding the structure of a document set by calculating a mutual relationship between a large number of generated document sets. The purpose is to:

According to the eleventh, twenty-fourth, and thirty-seventh aspects of the present invention, a non-hierarchical clustering technique is used as a statistical technique for classifying documents, thereby providing a document classification apparatus, method, or recording medium for easily generating a large number of partial document sets. The purpose is to provide.

According to the twelfth, twenty-fifth, and thirty-eighth aspects of the present invention, information that can easily grasp the structure of a document set can be obtained by calculating a similarity relation and an inclusion relation as a mutual relation between a large number of generated document sets. An object of the present invention is to provide a document classification device, method, or recording medium to be provided.

According to the inventions of claims 13, 26 and 39, the mutual relationship is calculated by using only information on words among the information of a large number of generated document sets, so that versatility and reusability are improved. A document classification device that calculates high related information,
It is intended to provide a method or a recording medium.

[0030]

According to the first aspect of the present invention, there is provided a document classification apparatus for classifying a set of documents in accordance with the contents thereof, wherein a document input unit for inputting a plurality of documents and a document input unit for inputting the plurality of documents. A document analysis unit for extracting word information constituting each document from each document, and a document set including the plurality of documents based on the word information of each document extracted by the document analysis unit. A document classification unit for classifying the document into a set of documents;
A representative word extracting unit for extracting a representative word set from each of the partial document sets classified by the document classifying unit, and a representative word extracting unit using a related word dictionary in which related words of arbitrary words are described. A related word extracting unit for extracting a related word set for each of the representative word sets of each partial document set extracted in step 2, a related word set extracted by the related word extracting unit, and a representative word set extracted by the representative word extracting unit A partial document set information generation unit that generates individual partial document sets and related information between the partial document sets based on information about documents belonging to each partial document set, and a classification result in the document classification unit. And a classification result storage unit that stores the information together with the information generated by the partial document set information generation unit.

According to a second aspect of the present invention, in the first aspect of the present invention, the related word set extracted by the related word extracting unit is:
It is a combination of at least one of synonyms, synonyms, and antonyms.

According to a third aspect of the present invention, in the first aspect, the related word set extracted by the related word extracting section includes at least an opposite word, and the opposite word set extracted from the representative word set of a certain partial document set. If the word set does not match the representative word set of any other sub-document set including itself, the document including the inconsistent opposite word set is extracted from the document set, and the process of generating a new sub-document set is entirely performed. It is characterized in that it further includes a reciprocal partial document set generation unit that recursively repeats the document set.

According to a fourth aspect of the present invention, in the first aspect of the present invention, the related words extracted by the related word extracting section include at least an opposite word, and the opposite word extracted from a representative word set of a certain partial document set. If the set does not match the representative word set of any other sub-document set, including the self, the document including the unmatched opposite word set and the word set obtained by removing the representative word corresponding to the opposite word set from the document set is converted to a document. It is characterized in that it further includes a reciprocal partial document set generation unit that extracts from the set and generates a new partial document set recursively for all the partial document sets.

According to a fifth aspect of the present invention, there is provided a document classifying apparatus for classifying a document according to the content of the document, comprising: a document input unit for inputting document data; applying a morphological analysis to the document data; A document analysis unit that extracts constituent words together with their part of speech information and the like, and a document vector space for expressing the document data in a multidimensional vector space is generated from the analysis information of the document data extracted by the document analysis unit. A document vector space generating unit, and a document classifying unit that classifies document data by using a statistical method in the document vector space generated by the document vector space generating unit, Is generated by combining a word having a part of speech with one or more words extracted before and after the specific part of speech based on the part of speech information of the specific part of speech. That replaces a word, and the word class information of a specific part of speech is also that is characterized by appropriately replacing.

According to a sixth aspect of the present invention, in the fifth aspect of the invention, the document classifying section classifies the document data by using a clustering method as a statistical method.

According to a seventh aspect of the present invention, in the invention of the fifth or sixth aspect, the document analysis unit replaces the words and the parts of speech with respect to the words whose parts of speech are prefixes, suffixes, classifiers, and similar parts of speech. It is characterized by performing.

According to an eighth aspect of the present invention, in any one of the fifth to seventh aspects of the present invention, the combination of words is continued until a word of a specific part of speech appears in the document analysis unit. .

According to a ninth aspect of the present invention, in the invention according to any one of the fifth to eighth aspects, the part of speech is combined with the word of the numeral suffix or the classifier in the document analysis unit. The document classification unit does not use information on the deleted words.

According to a tenth aspect of the present invention, there is provided a document classification device for classifying a document data set according to the contents of a document, wherein the document input unit for inputting the document data set and a morphological analysis are applied to all the document data. A document analysis unit that extracts words constituting the document data together with their part of speech information, a document analysis result storage unit that stores analysis results of the document data extracted by the document analysis unit, A document vector space generation unit that generates a vector space for expressing the document data in a multidimensional vector space from the extracted analysis information of the document data; and a document vector space generated by the document vector space generation unit. A document vector data storage unit that stores the vector data of the document data; and a classification number determination unit that determines the classification number of the document data set based on a designated condition. A document classifying unit for classifying document data into sub-document sets of the specified number of classifications by using a statistical method in the document vector space generated by the document vector space generating unit; and a classification generated by the document classifying unit. A classification result storage unit that stores results, a repetition determination unit that determines whether to repeat the processing from the classification number determination unit to the classification result storage unit, the document vector data storage unit, and the classification result storage A partial document set relation calculating unit that calculates relation information between all partial document sets generated using information stored in the partial document set, and a partial document set generated by the partial document set relation calculating unit. And a partial document set relation storage unit for storing the relation information of the partial document set.

According to an eleventh aspect, in the tenth aspect, the statistical method used in the document classifying unit is a non-hierarchical clustering method.

The twelfth aspect of the present invention provides the tenth or the first aspect.
In the invention according to the first aspect, the relationship calculated by the partial document set relationship calculating unit is a similarity relationship and an inclusion relationship.

According to a thirteenth aspect, in the twelfth aspect, the relation between the partial document sets is calculated using only word information extracted from each partial document set. .

According to a fourteenth aspect of the present invention, there is provided a document classification method for classifying a set of documents in accordance with the contents thereof, comprising: a document inputting step of inputting a plurality of documents; A document analysis step for extracting word information constituting the document, and a document for classifying a document set of the plurality of documents into several partial document sets based on the word information of each document extracted in the document analysis step A classification step, a representative word extraction step of extracting a representative word set from each of the partial document sets classified in the document classification step, and using a related word dictionary in which related words of arbitrary words are described. A related word extracting step of extracting a related word set for each representative word set of each partial document set extracted in the representative word extracting step; Based on the extracted related word set, the representative word set extracted in the representative word extraction step, and information on documents belonging to each partial document set, individual partial document sets and related information between the partial document sets are generated. A partial document set information generating step; and a classification result storing step of storing a classification result in the document classifying step together with information generated by the partial document set information generating unit. .

According to a fifteenth aspect, in the fourteenth aspect, the related word set extracted in the related word extracting step is a combination of at least one of a synonym, a synonym, and an antonym. It is characterized by having.

According to a sixteenth aspect, in the fourteenth aspect, the related word set extracted in the related word extracting step includes at least an opposite word, and the opposite word set extracted from a representative word set of a certain partial document set. If the word set does not match the representative word set of any other sub-document set including itself, the document including the inconsistent opposite word set is extracted from the document set, and the process of generating a new sub-document set is entirely performed. The method further comprises a step of generating a reciprocal partial document set recursively with respect to the document set.

According to a seventeenth aspect, in the fourteenth aspect, the related word extracted in the related word extracting step includes at least an opposite word, and the opposite word extracted from a representative word set of a certain partial document set. If the set does not match the set of terms in any other sub-documents, including you,
A document including a word set obtained by removing a non-matching opposite word set and a representative word corresponding to the opposite word set from the representative word set is extracted from the document set, and a process of generating a new partial document set is performed on all the partial document sets. The method further includes a step of generating a recursive partial document set that is recursively repeated.

The invention according to claim 18 is a document classification method for classifying a document according to the contents of the document, comprising the steps of: inputting document data; applying morphological analysis to the document data; Generating a document vector space for expressing the document data in a multidimensional vector space from the analysis information of the document data extracted in the document analysis step; A document vector space generating step, and a document classifying step of classifying document data by using a statistical method in the document vector space generated in the document vector space generating step, wherein the identification extracted in the document analyzing step Are extracted before and after the specific part of speech based on the part of speech information of the specific part of speech. That replaces one or more of the words and the words that are generated by combining and is obtained by said replacing also appropriate part of speech information of the particular part of speech.

The invention of claim 19 is characterized in that, in the invention of claim 18, the document data is classified by using a clustering method as a statistical method in the document classification step.

The twentieth aspect of the present invention is the twelfth aspect of the invention.
A ninth aspect of the present invention is characterized in that in the document analyzing step, the words and the parts of speech are replaced for words whose parts of speech are prefixes, suffixes, classifiers, and similar parts of speech.

The invention of claim 21 is the invention of claims 18 to 2
0, wherein the combination of words is continued until a word of a specific part of speech appears in the document analysis step.

The invention of claim 22 is the invention of claims 18 to 2
In the invention according to any one of the first to third aspects, for the word whose part of speech is a numeral suffix or a classifier in the document analysis step, a plurality of words combined with the numeral suffix or the classifier word are deleted, and in the document classification step, the plurality of words are deleted. This feature is characterized by not using the information of the word.

According to a twenty-third aspect of the present invention, there is provided a document classification method for classifying a document data set according to the contents of a document, wherein a document input step of inputting the document data set and a morphological analysis are applied to all the document data. A document analysis step of extracting words constituting the document data together with their part of speech information and the like; a document analysis result storing step of storing an analysis result of the document data extracted in the document analysis step; A document vector space generating step of generating a vector space for expressing the document data in a multidimensional vector space from the analysis information of the extracted document data; and a document vector space generated in the document vector space generating step. A document vector data storing step of storing vector data of the document data;
A classification number determining step of determining a classification number of a document data set from a specified condition; and a document vector space generated by the document vector space generation step by using a statistical method to convert the document data into the specified classification number. A document classification step of classifying the documents into partial document sets, a classification result storage step of storing the classification results generated in the document classification step, and whether to repeat the processing from the classification number determination step to the classification result storage step And a relation between partial document sets for calculating relation information between all partial document sets generated using the information stored in the document vector data storing step and the information stored in the classification result storing step. Calculating step and the partial document set generated in the partial document set relation calculating step It is obtained by comprising a partial document set between relationship storage step of storing the relationship information.

According to a twenty-fourth aspect, in the twenty-third aspect, the statistical technique used in the document classification step is a non-hierarchical clustering technique.

The invention of claim 25 is the invention of claim 23 or 2
The invention according to claim 4, characterized in that the relations calculated in the partial document set relation calculation step are similarity relations and inclusion relations.

The invention of claim 26 is the invention of claim 25, wherein the relation between the partial document sets is calculated using only word information extracted from each partial document set. .

According to a twenty-seventh aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing a document classification method for classifying a document set according to the contents thereof, and a document input step of inputting a plurality of documents. A document analysis step of extracting word information constituting each document from each document input in the document input step; and the plurality of document information based on the word information of each document extracted in the document analysis step. A document classification step of classifying a set of documents into several partial document sets; a representative word extraction step of extracting a set of representative words from each of the partial document sets classified in the document classification step; For each representative word set of each partial document set extracted in the representative word extraction step using the related word dictionary in which the related words are described. A related word extraction step of extracting a related word set, a related word set extracted in the related word extraction step, a representative word set extracted in the representative word extraction step, and information on documents belonging to each partial document set. A partial document set information generating step for generating individual partial document sets and related information between the partial document sets based on the information generated by the partial document set information generating unit; And a computer-readable recording medium on which a program for executing a document classification method including a classification result storage step of storing together is stored.

According to a twenty-eighth aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing the document classification method according to the twenty-seventh aspect, wherein a related word set extracted in the related word extracting step is included. , A computer-readable storage medium storing a program for executing a document classification method that is a combination of at least one of synonyms, synonyms, and antonyms.

According to a twenty-ninth aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing the document classification method according to the twenty-seventh aspect, wherein a related word set extracted in the related word extracting step is included. A document containing at least an antonym and extracted from a set of representative words of a partial document set and not including a set of non-matching opposite words, if the set of the opposite words does not match the representative word set of any other partial document set including itself. Computer for recording a program for executing a document classification method further including a reciprocal partial document set generating step of recursively repeating a process of generating a new partial document set recursively for all the partial document sets. It is a readable recording medium.

According to a thirtieth aspect of the present invention, in a computer-readable recording medium storing a program for executing the document classification method according to the twenty-seventh aspect, at least a related word extracted in the related word extracting step is included. If the set of the opposite words including the antonym and extracted from the set of the representative words of a partial document set does not match the representative word set of any of the other partial document sets including the same, the incompatible opposite word set and the representative word set An anonymous sub-document generation that recursively repeats the process of extracting from the document set a document containing the word set excluding the representative word corresponding to the antonym set from the document set and generating a new sub-document set A computer-readable storage medium storing a program for executing a document classification method further including steps.

According to a thirty-first aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing a document classification method for classifying documents according to the contents of the document. A document analysis step of applying morphological analysis to the document data to extract words constituting the document data together with their part of speech information and the like, and converting the document data from the analysis information of the document data extracted in the document analysis step. A document vector space generating step for generating a document vector space for expressing in a multidimensional vector space, and a document classification for classifying document data by using a statistical method in the document vector space generated in the document vector space generating step And a specific part of speech extracted in the document analysis step. That a word, on the basis of the part of speech information of the particular part of speech, replaced with words that are generated by combining one or more words to be extracted before and after the specific part of speech,
In addition, the present invention is a computer-readable recording medium on which a program for executing a document classification method for appropriately replacing the part of speech information of the specific part of speech is recorded.

According to a thirty-second aspect of the present invention, there is provided a computer-readable recording medium storing a program for executing the document classification method according to the thirty-first aspect, wherein a clustering method is used as a statistical method in the document classification step. This is a computer-readable recording medium on which a program for executing a document classification method for classifying document data is recorded.

The invention of claim 33 is the invention of claim 31 or 3
2. In a computer-readable recording medium on which a program for executing the document classification method described in 2 is recorded, in the document analyzing step, the words of which the parts of speech are prefixes, suffixes, classifiers, and parts of speech similar thereto And a computer-readable recording medium on which a program for executing a document classification method for replacing part of speech is recorded.

The invention of claim 34 is the invention of claims 31 to 3
3. In a computer-readable recording medium having recorded thereon a program for executing the document classification method according to any one of 3., a document classification method that continues combining words until a word of a specific part of speech appears in the document analysis step Is a computer-readable recording medium on which a program for executing the program is recorded.

The invention of claim 35 is the invention of claims 31 to 3
4. In a computer-readable recording medium storing a program for executing the document classification method according to any one of 4), in the document analysis step, for the words whose part of speech is a numeral suffix or a classifier, the numeral suffix or a classifier And a computer-readable recording medium on which a program for executing a document classification method that deletes a plurality of words combined with the word and does not use information of the deleted words in the document classification step is recorded.

According to a thirty-sixth aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing a document classification method for classifying a document data set according to the contents of a document. Step and apply morphological analysis to all document data,
A document analysis step of extracting words constituting the document data together with their part of speech information and the like; a document analysis result storage step of storing an analysis result of the document data extracted in the document analysis step; A document vector space generating step of generating a vector space for expressing the document data in a multidimensional vector space from the analysis information of the extracted document data, and a document vector space generated in the document vector space generating step. A document vector data storing step of storing vector data of each document data, a classification number determining step of determining a classification number of a document data set from designated conditions, and a document vector space generated in the document vector space generation step. Classification of the document data into the specified classification by using a statistical method A document classification step of classifying the document into a partial document set, a classification result storage step of storing the classification result generated in the document classification step, and whether to repeat the processing from the classification number determination step to the classification result storage step Between the partial document sets for calculating relation information between all the partial document sets generated using the information stored in the document vector data storing step and the classification result storing step. A program for executing a document classification method including a relation calculating step and a partial document set relation storing step of storing relation information between partial document sets generated in the partial document set relation calculating step is recorded. It is a computer-readable recording medium.

According to a thirty-seventh aspect of the present invention, in the computer-readable recording medium storing a program for executing the document classification method according to the thirty-sixth aspect, the statistical method used in the document classification step is a non-hierarchical clustering. This is a computer-readable recording medium on which a program for executing a document classification method, which is a technique, is recorded.

The invention of claim 38 is the invention of claim 36 or 3
7. In a computer-readable recording medium on which a program for executing the document classification method described in 7 is recorded, a relation calculated in the step of calculating a relation between partial document sets is a similar relation and an inclusion relation. Is a computer-readable recording medium on which a program for executing the program is recorded.

According to a thirty-ninth aspect of the present invention, in a computer-readable recording medium storing a program for executing the document classification method according to the thirty-eighth aspect, the relation between the partial document sets is extracted from each partial document set. The computer-readable recording medium stores a program for executing a document classification method calculated using only word information to be processed.

[0069]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the description of the embodiments of the present invention, a group of one or more sentences described in a natural language, which is to be classified, is called a document. Also,
It is assumed that the end of one document is provided with a document end symbol that can be identified. To give a concrete example, a patent publication and a specific newspaper article are also documents, and even if a claim or a specific one sentence is extracted therefrom, it is regarded as a document.

FIG. 1 shows a first embodiment of the present invention.
FIG. 6 is a block diagram of a document classification device for describing an embodiment corresponding to the inventions of Nos. 5, 27 and 28. In the document input unit 101, input means such as a keyboard, an OCR device, and an auxiliary storage device such as a hard disk are connected to the document classification apparatus 100 directly or via a network, and a document or a document group is obtained from such an input means. An interface for inputting document data. FIG. 2 is a flowchart illustrating an example of a process for inputting document data.

The document analysis unit 102 in FIG. 1 performs a natural language analysis on each of the input documents, and extracts words and their parts of speech. Furthermore, the order in which words appear in the document, and meta information of the document such as the creator and date of creation of the document can also be included. After that, a unique word ID is assigned to the word that has appeared in the document group, and the number of occurrences of the word in the document is counted. As an example, by applying morphological analysis to a document, word expressions and parts of speech in the document are extracted, and based on the results, unique word expressions, parts of speech, and identification numbers that appear in the document group are extracted. Further, an example in which each document is represented by the extracted unique word identification number and its frequency is shown, and a flowchart thereof is shown in FIG.

For example, document 1 and document 2 shown in FIG.
On the other hand, when morphological analysis is applied, a result as shown in FIG. 4B is obtained. In FIG. 4B, the numerical value below each cut-out word indicates those parts of speech, and the correspondence table is shown in FIG. 4C. Assuming that the document group is composed of only the two documents shown in FIG. 4A, the notation, part of speech, and identification number of a unique word appearing in the document group, and each document are represented by a word identification number and its frequency. The results are shown in FIGS.
(C). However, for the sake of simplicity, only nouns and unregistered words are used as parts of speech.

In the document classifying unit 103, the document analyzing unit 102
Classify documents based on the information generated in. In the present invention, although the classification method is not particularly limited, here, as an example, each document is represented by a vector of the frequency of occurrence of a unique word in the document group, inheriting the embodiment in the document analysis unit 102, and An example in which document classification is performed using the k-means method, which is one of clustering methods based on vectors, will be described, and a flowchart thereof is shown in FIG. Here, it is assumed that the similarity between the vectors is a real number between 0 and 1 and the maximum similarity is 1.

FIG. 7B shows the result of classifying the 15 documents shown in FIG. 7A into three partial document sets based on the algorithm shown in FIGS. Here, only the noun and the unregistered word are adopted as the parts of speech, the similarity measure in the k means method is a cosine measure, and the repetition stop condition is five repetitions. The representative word extraction unit 104 extracts a representative word set in each partial document set based on the word information of each document generated by the document analysis unit 102 and the information on the partial document group generated by the document classification unit.

In the present invention, the method of extracting the representative words is not particularly limited. Here, as an example, the embodiment in the above-described document classifying unit is inherited, and the documents belonging to them in each partial document set are regarded as one virtual document. FIG. 8 shows an example in which a word having a frequency of occurrence of a unique word in a document group that is equal to or higher than a specified threshold when the document is regarded as a document is set as a representative word set of those partial document sets. Show.

FIG. 9 shows a representative word set obtained according to the above-described flowchart for each partial document set of the embodiment in the document classification unit. Here, the threshold value of the appearance frequency is 2.

In the related word extracting unit 105, the representative word extracting unit 1
For each of the representative words of each partial document set extracted in 04, related words are extracted using a related word dictionary, and these are set as a related word set of each partial document set. As the related term dictionary, a synonym dictionary, a broad term dictionary, a narrow term dictionary, a synonym dictionary, an antonym dictionary, a sibling dictionary, a high-level concept dictionary, a low-level concept dictionary, etc. can be used. An example in which the embodiment in the representative word extracting unit is inherited and a related word set of each partial document set is obtained using an arbitrary one related word dictionary is shown, and its flowchart is shown in FIG. If a plurality of dictionaries are used, the above process may be repeated for each dictionary. For the sake of simplicity, it is assumed that only the synonyms are handled as related words. If the synonyms of each representative word obtained by the embodiment of the representative word extracting unit are as shown in FIG. The set is shown as in FIG.

The partial document set information generation unit 106 includes word information of each document generated by the document analysis unit 102, information on the partial document group generated by the document classification unit 103, and each partial document set extracted by the representative word extraction unit 104. Based on the representative word set and the related word set of each partial document set generated by the related word extraction unit 105, individual partial document sets and related information between the partial document sets are generated.

The information unique to each partial document set includes a set of representative word sets, a set of related word sets, and, when each partial document set is generated by a classification method that allows multiple classification, representative words and / or Information such as a subset of documents belonging to a partial document set that includes a related word or more than a specified threshold number can be used. The related information between the partial document sets includes the intersection, union, and difference set of the representative word set between the partial document sets, the intersection, union, and difference set of the set of the related word sets, and each partial document. If the set is generated by a classification method that allows multiple classification, documents belonging to a partial document set that contains many intersections, unions, difference sets, representative words, and / or related words of documents belonging to the partial document set Information such as the intersection, union, and difference between the subsets can be used.

Here, as an example, assuming that the document subset is generated by a classification method that allows multiple classifications, inheriting the embodiment in the related word extraction unit, the representative word set information is used as partial document set information. An example of generating a set, a set of related word sets, a product set, union and difference set of a representative word set set between partial document sets, and a product set, union set and difference set of a related word set set between partial document sets The flowchart is shown in FIG. With such information, it is possible to grasp the similarity, relevance, inclusion relationship, and the like among the partial document sets.

In the classification result storage unit 107, the document analysis unit 1
02, information on the partial document group generated by the document classifying unit 103, a representative word set of each partial document set extracted by the representative word extracting unit 104, and each part generated by the related word extracting unit 105 The related word set of the document set, the individual partial document sets generated by the partial document set information generation unit 106, and the related information between the partial document sets are stored in an appropriate format. The stored related information is output to the output unit 108.
Is output to a predetermined output unit as needed in response to a user request or according to a predetermined condition.

FIG. 13 shows a third embodiment of the present invention.
FIG. 7 is a block diagram of a document classification device 200 for describing an embodiment corresponding to 7, 29, and 30. FIG.
Portions having the same functions as in FIG. 1 are given the same numbers as in FIG. When at least an antonym is extracted as a related word generated by the related word extracting unit 105 in the antonymous partial document set generation unit 201, the antonym of any document subset is changed to any other part including itself. If the word does not match the representative word of the document set, a document containing this opposite word is extracted from the document group, and the process of making it a new partial document set is performed recursively for all the partial document sets.

Here, as an example, it is assumed that only the antonym is extracted by the related word extraction unit 105 by inheriting the above embodiment, and the antonym set included in each partial document set is included in another part including itself. Judge whether or not it matches the representative word of the document set, and if the antonym does not match any of the representative words, extract documents containing the antonym from the document group using a search method and make them a new sub-document set An example is shown, and the flowchart is shown in FIG.

For example, let us focus on the representative word set of the partial document set 3 in FIG. 7B obtained as a result of classifying the document group shown in FIG. 7A. In this case, it is assumed that the word “free, free” is obtained as an opposite word of the representative word “commercial”. In this case, these words do not match any of the representative words, and the result of searching the document group for the word “free” is 0. However, when searching for the word “free”, the document 4,
Document 5 and document 12 are searched. If this is a new partial document set, "Linux, free, distribution" can be obtained as a representative word.

Thus, even if a partial document set having the opposite meaning to an arbitrary partial document set is not generated from the document group by the document classifying unit, a partial document set having the opposite meaning can be generated. Therefore, it is possible to extract a wider topic from the document group.

According to the inventions of claims 4, 17 and 30, when a new partial document set is obtained from an antonym, a partial document set is also obtained together with a representative word of a partial document set other than the representative word that generated the antonym. This makes it possible to generate a partial document set having a meaning opposite to that of the target partial document set, but the basic processing can be obtained by the same processing as in the above embodiment. That is, for example, in the flowchart shown in FIG. 14, the step of searching for a document group using an antonym may be performed using a logical expression that combines an antonym and a representative word of a partial document set other than the representative word that generated the antonym. .

FIG. 15 is a cross-sectional view of the present invention.
FIG. 22 is a block configuration diagram of a document classification device for describing an embodiment corresponding to Nos. 22 and 31 to 35. Document input unit 30
An input unit 1 such as a keyboard, an OCR device, an auxiliary storage device such as a hard disk is connected to the document classifying device 300 directly or via a network, and a document or a document group is obtained from such an input unit. An interface for inputting data. At this time, in order to uniquely identify each document data, an identifier such as a unique number is assigned to each document.

The document analysis unit 302 applies morphological analysis to each of the input documents, and extracts words constituting each document together with part of speech information and the like. At this time, in order to identify the extracted words, unique identifiers are attached to the extracted words having a unique notation. Further, a word having a specified part of speech out of the words obtained as a result of the morphological analysis is replaced with a word appropriately combined with the preceding and following words, and the part of speech is also replaced with an appropriate word. As an example, a description will be given of an operation of performing the above-described combination and replacement processing on a word whose part of speech is a general prefix, a general suffix, and a classifier.

First, in the present example, the part-of-speech is described as “1. 1. general prefixes; 2. general suffixes other than numeric suffixes; In the case of a numerical suffix or a number particle, it is performed according to the following rules. However, the rules of the combination and replacement processing in the present invention are not limited to these.

○ Prefix in general If {the part of speech of the target word is a prefix}｝ Counting variable: assign 1 to i Repeat ｛Word extracted at the beginning of the target word i times before the target word If {the part of speech of the word extracted i times before is the part of speech used for classification}, {exit the repetition loop も な く otherwise ｛increase i by 1｝ を change the part of speech of the target word ｝

○ General suffixes other than numerical suffixes If {the part of speech of the target word is a suffix other than the numerical suffix}, {counting variable: assign 1 to i repeat} target at the end of the target word Combine words extracted i times after the word If {the part of speech of the word extracted i times before is the part of speech used during classification}, {exit the repetition loop も な く otherwise, increase i by 1 ｝｝ Change the part of speech of the target word｝

○ Numerical suffix or classifier If {the part of speech of the target word is a numerical suffix or classifier} repeat {if the part of speech of the word extracted immediately before the target word is a numeric part}結合 Join the word extracted just before the target word to the beginning of the target word Delete the word extracted i times before the target word も な く Otherwise, go out of the repetition loop｝ 変 更 Change the part of speech of the target word ｝

The six document data shown in FIG. 16 are classified as document data, and morphological analysis is applied to the document data to extract words and their parts of speech as shown in FIG.
Shown in However, the morphological analysis system is not specified in the present invention. FIG. 18 shows the analysis result of the document data when the part of speech used at the time of classification is a common noun, a paranoun, a proper noun, a number, an adjective, a general prefix, a general suffix, and a classifier bet.
Shown in

FIG. 19 shows the result of performing the combining / replacing process on the word whose POS is a general prefix, a suffix, or a number particle in the result shown in FIG. 18 in accordance with the above rules. For example, the character string {Chiba [ordinary noun], Mr. [proper noun suffix]} in document 1 is expressed as {Chiba [ordinary noun], Mr. Chiba [property noun] using the general suffix rules other than the numerical suffix. Noun]} and {1 [numerical], 9 [numerical], 5 [numerical], 0 [numerical],
The character string “year [number classifier]} becomes a character string“ {1950 [common noun]} ”using the rule of a numerical suffix or classifier.

The document vector space generation unit 303 generates a space for expressing the document data in a vector based on the word information of each document data extracted by the document analysis unit. As an example, an operation of generating a vector representation of each document data in the case where a document vector space is generated based on the frequency of unique words in the entire document data based on the example in the document analysis unit will be described. However, in the present invention,
The vector space generation method is not limited to this,
For example, a vector space can be generated by linear conversion of all words.

A unique word is extracted from the document analysis results shown in FIG. 18 and FIG. 19, the frequency of the corresponding word in each document is counted, and the results are converted into the word in the column direction and the document data in the row direction. FIG. 20 and FIG. 21 show the matrix representation by the attachment. In these matrices, the column vector is the vector data of each document data.

The document classifying unit 304 classifies the document data vectors generated by the document vector space generating unit into several sets by using a statistical method. Output unit 30
In 5, a set of document data vectors classified by the document classification unit 304 is output to a predetermined output unit as needed in accordance with a user request or according to predetermined conditions. Although various types of statistical processing can be used in the document classifying unit 304, the invention of claim 5 is limited to using a clustering method for reasons such as the simplicity of the algorithm and the presence or absence of parameters. As an example, an operation of classifying document vectors by using a clustering method based on the example in the document vector space generation unit will be described.

Here, the Warrd method, which is one of the clustering techniques, is used, and the similarity measure uses a standardized Euclidean distance measure. As for the clustering method, "Introduction to Multivariate Analysis (Morikita Publishing)"
Familiar with. FIGS. 22 and 23 show the results of applying the Warrd method to the document data shown in FIGS.
Shown in Here, FIG. 20 shows data constituting a document vector space as a result of applying the combining / replacement processing, and FIG. 21 shows data constituting a document vector space as a result of not applying the combining / replacement processing. . The numerical values in FIGS. 22 and 23 are the distances between the clusters.

When comparing the results of FIGS. 22 and 23,
The difference in the position of the document 4 is very distinctive, and when the combining / replacement process is applied, the document 4 becomes the document 2 or the document 5
When the combining / replacement process is not applied, the document 4 is determined to be similar to the document 1 or the document 6. Judging from the degree of conformity of the subjective vocabulary, Document 4 is more likely to be {Document 1, Document 5} than {Document 2, Document 5}.
It seems that it is more appropriate to be included in the set of Document 3, Document 6 #. Therefore, from this result, a higher-quality document vector space can be configured by applying the combining / replacement process, and a high-quality document classification result can be obtained by performing the classification process in this document vector space. it can.

FIG. 24 is a cross-sectional view of the present invention.
FIG. 27 is a block configuration diagram of a document classification device for describing embodiments corresponding to -26 and -39. Document input unit 4
Reference numeral 01 denotes an input unit using an auxiliary storage device such as a keyboard, an OCR device, or a hard disk, which is connected to the document classifying device 400 directly or via a network. Interface for inputting At this time, in order to uniquely identify each document data, for example, a unique number,
Assign an identifier to each document.

The document analysis unit 402 applies morphological analysis to each of the input documents, and extracts words constituting each document together with part of speech information and the like. At this time, in order to identify the extracted word, a unique identifier is attached to the extracted word having a unique notation, similarly to the document data. As an example, morphological analysis is applied to document data to identify words whose notation and part of speech are unique in the entire document data, assign unique identification numbers to them, and assign each document data to the words that compose it. FIG. 25 shows a pseudo code for expressing the identification number and its appearance frequency. In the present invention, the morphological analysis system may be any system that can extract necessary information.

The document analysis result storage unit 403 stores the result of the morphological analysis of the document data extracted by the document analysis unit 402 in an appropriate format. Document vector space generation unit 404
Then, based on the word information of each document data extracted by the document analysis unit 402, a space for expressing the document data in a vector is generated. As an example, a pseudo code for generating a vector representation of each document data in the case where a document vector space is generated based on the normalized frequency of unique words in the entire document data based on the example in the document analysis unit 402. FIG.
Shown in However, in the present invention, the vector space generation method is not limited to this. For example, a vector space can be generated by linear transformation of all words using singular value decomposition or the like.

The document vector data storage unit 405 stores the document data vector generated by the document vector space generation unit 404 in an appropriate format. Classification number determination unit 406
Then, the number of classifications at the time of repeatedly classifying documents is determined (a pseudo code in the case where the number of classifications is a constant × the number of repetitions is included in FIG. 27). The document classification unit 407 classifies the document data vectors generated by the document vector space generation unit 404 into a classification number determination unit set by using a statistical method.

Although various types of statistical processing can be used, the invention of claim 11 uses a non-hierarchical clustering method because of the simplicity of the algorithm and the characteristic that the classification structure dynamically changes due to the change in the number of clusters. Limited. As an example, FIG. 27 shows a pseudo code for classifying document vectors by using a clustering method as the number of clusters multiplied by the number of repetitions and a constant N. Here, the k mean method, which is one of the clustering methods, is partially modified, and a cosine measure is used as the similarity measure. The clustering method is described in detail in "Introduction to Multivariate Analysis (Morikita Publishing)".

The document classification result storage unit 408 stores the document classification result generated by the document classification unit 407 in an appropriate format. The repetition determination unit 409 determines whether or not to continue repetition when performing repetitive document classification (FIG. 27 includes a pseudo code when the repetition determination is limited to a specified maximum number). In the partial document set relation calculating section 410, the relation information between a plurality of partial document sets stored in the document classification result storage section 408 is stored in the document analysis result storage section 403 and the document vector data storage section 405. It is calculated using information on various types of document data. As an example, an operation of calculating a similarity relation and an inclusion relation between partial document sets by using document data and / or word information constituting the document data will be described.

First, a formulation for expressing the similarity relation and the inclusion relation between the partial document sets by the document data is performed. It is assumed that a plurality of partial document sets stored in the document classification result storage unit 408 have unique identification numbers.
The characteristic vector Vm of the m-th partial document set is defined as follows.

The number of dimensions of Vm is equal to the total number of document data. Each element of Vm corresponds to one document data.
There is no overlap. When the similarity between the document data corresponding to the element i and the partial document set is equal to or larger than the threshold, the element i is 1. When the similarity between the document data corresponding to the element i and the partial document set is less than the threshold value, the element i becomes 0.

Using the above definitions, the relationship between the m-th partial document set and the n-th partial document set: Rmn and Rnm are defined as follows. (1) Rmn = <Vm, Vn> / <Vm, Vm> (2) Rnm = <Vm, Vn> / <Vn, Vn> where <,> indicates an inner product.

From the values of Rmn and Rnm, it is possible to calculate the similarity and the inclusive relation between the partial document sets. FIG. 28 shows a geometrical interpretation based on the values of Rmn and Rnm. That is, when Rmn is close to 1, it can be said that the partial document set m is included in the partial document set n. Further, it can be said that the partial document set m and the partial document set n are more similar as Rmn and Rmn are both closer to 1. Further, it can be read that, as (Rmn, Rnm) is closer to the straight line of Rmn = Rnm, the document data are mutually included at the same rate.

Next, a formula for expressing the similarity relation and the inclusion relation between the partial document sets by the appearance frequency information of the words constituting the document data is formulated. The characteristic vector Wm of the m-th partial document set is defined as follows.

The number of dimensions of Wm is equal to the number of unique words in all document data. Each element of Wm corresponds to a unique word, and the I-th element value of Wm without duplication is represented by wm
(I). The appearance frequency (the number of appearances) of the word corresponding to the element i in all the documents whose similarity with the partial document set is equal to or larger than the threshold value is set as the element value of the element i.

Using the above definition, the relationship between the m-th partial document set and the n-th partial document set: R′mn and R′nm are defined as follows. (3) R'mn = Σf (wm (k), wn (k)) / Σf (wm (k), wm (k)) (4) R'mn = Σf (wm (k), wn (k) ) / Σf (wn (k), wn (k)) (5) f (wm (k), wn (k)) = 0 for wm (k) × wn (k) = 0 = wm
(k) × (a + b / | wn (k)-wn (k) | +1) for forwm (k)
× wn (k)! = 0 where a and b are constants, and a + b = 1, a, b> = 0

Even if the values of R'mn and R'nm are used, the same interpretation as the relationship between Rmn and Rmn shown in FIG. 28 can be made. Therefore, it is possible to calculate the similarity relationship and the inclusion relationship between the partial document sets. Becomes possible. Furthermore, R'mn and R '
In the case where the relationship of the partial document set is defined using nm, it is possible to obtain a relationship that cannot be obtained at the document data level. For example, even if the contents match, the document data to be analyzed is different. It is also possible to calculate the relationship between the partial document sets even when there is. Further, the partial document set relation storage unit 411 stores the relation information between the partial document sets generated by the partial document set relation calculation unit 410 in an appropriate format. Further, the output unit 412 outputs the relation information stored in the partial document set relation storage unit 411 to the output unit in response to a user request or according to a predetermined condition.

[0114]

According to the inventions of claims 1, 14 and 27, a representative word set of each of the generated partial document sets is extracted, and a related word is obtained for each of the representative words. By generating the partial document sets and the related information between the partial document sets, it is possible to provide information effective for analyzing the partial document sets.

According to the invention of claims 2, 15 and 28,
The use of at least one combination of synonyms, synonyms, and antonyms as related words can provide information mainly on similarity.

Claims 3, 4, 16, 17, 29 and 30
According to the invention, when an antonym is used as a related word of the representative word set of each partial document set, and the opposite word does not match the representative word set of any other partial document set including itself, the document including the opposite word is used. Is extracted from a set of documents, and is used as a new partial document set, whereby more analysis information can be extracted from the set of documents.

According to the invention of claims 5, 18 and 31,
In a document classification device that performs a morphological analysis on a classification target document and classifies the classification target document into several document sets based on the obtained analysis result, a word having a specified part of speech among words obtained as a result of the morphological analysis , A high-quality document vector space is constructed by performing a process of replacing the word with the word before and after the word and a word appropriately combined, and replacing the part of speech with an appropriate word. By performing document classification, high-quality document classification results can be obtained.

According to the invention of claims 6, 19 and 32,
By using a clustering method as a statistical method for performing document classification, a high-quality document classification result can be easily obtained.

According to the seventh, twentieth and thirty-third inventions,
Among words extracted by applying morphological analysis to documents to be classified, in particular, parts of speech are prefixes, suffixes, classifiers,
By performing appropriate combining processing on words that are part of speech and similar words, a high-quality document vector space can be obtained.

According to the invention of claims 8, 21 and 34,
By generating new words by continuing to combine words until a specific part of speech word appears in the word combining process, a high-quality document vector space can be obtained.

According to the ninth, twenty-second and thirty-fifth aspects,
In the word combining process, for a word whose part of speech is a numerical suffix or a classifier, delete a plurality of combined words,
When generating the document vector space, information of those words is not used, so that a high-quality document vector space can be obtained.

According to the tenth, twenty-third, and thirty-sixth aspects, a large number of partial document sets are generated by repeatedly performing document classification processing using the number of partial document sets to be generated as a parameter using a vector space model of the document. Further, by calculating the mutual relationship between a large number of generated document sets, it is possible to provide a document classification device that generates information that can assist in understanding the structure of the document set.

According to the eleventh, twenty-fourth, and thirty-seventh aspects, in addition to the above objects, a non-hierarchical clustering technique is used as a statistical technique for classifying documents, so that a large number of partial document sets can be easily generated. it can.

According to the twelfth, twenty-fifth, and thirty-eighth aspects of the present invention, in addition to the above objects, similarity relations and inclusion relations are calculated as a mutual relation between a large number of generated document sets, so that the document sets can be easily obtained. The information which can grasp a structure can be provided.

According to the thirteenth, twenty-sixth, and thirty-ninth aspects of the present invention, in addition to the above objects, the mutual relationship is calculated by using only information relating to words among the information of a large number of generated document sets. It is possible to calculate related information having high versatility and reusability.

[Brief description of the drawings]

FIG. 1 is a block diagram of a document classification device for explaining an embodiment corresponding to the inventions of claims 1, 2, 14, 15, 27 and 28 of the present invention.

FIG. 2 is a flowchart illustrating an example of a process for inputting document data.

FIG. 3 is a flowchart illustrating an example of processing for applying morphological analysis to a document.

FIG. 4 is a diagram for describing an application example of morphological analysis.

FIG. 5 is a diagram illustrating an example of a result of applying morphological analysis.

FIG. 6 is a flowchart illustrating an example of a process of classifying a document group based on information generated by a document analysis unit.

FIG. 7 is a diagram illustrating classification of documents into partial document sets.

FIG. 8 is a flowchart illustrating an example of representative word extraction processing.

9 is a diagram showing an example of a representative word set obtained according to the flowchart shown in FIG.

FIG. 10 is a flowchart illustrating an example of a process of extracting a related word using a related word dictionary for each of the representative word sets of each partial document set.

FIG. 11 is a diagram illustrating an example of a synonym of each representative word obtained by the representative word extraction unit and a related word set of each partial document set.

FIG. 12 is a flowchart illustrating an example of a process of generating a partial document set and related information between the partial document sets together with the extracted or generated representative word set and related word set.

FIG. 13 of the present invention.
30 is a block diagram of a document classification device for explaining an embodiment corresponding to FIGS.

FIG. 14 is a flowchart illustrating an example of a process of extracting a document including an antonym and forming a new partial document set.

FIG. 15 of the present invention.
FIG. 1 is a block diagram of a document classification device for describing an embodiment corresponding to 1 to 35.

FIG. 16 is a diagram illustrating an example of classification target document data.

17 is a diagram showing an example in which words and parts of speech are extracted by applying morphological analysis to the document data shown in FIG. 16;

FIG. 18 is a diagram illustrating an example of an analysis result of document data.

FIG. 19 is a diagram illustrating another example of the analysis result of the document data.

FIG. 20 is a diagram illustrating an example in which document data is represented in a matrix by being positioned in a row direction.

FIG. 21 is a diagram illustrating another example in which document data is represented in a matrix by being positioned in the row direction.

FIG. 22 is a diagram illustrating a result of applying the Warrd method to the document data of FIG. 20;

FIG. 23 is a diagram showing a result of applying the Warrd method to the document data of FIG. 21;

FIG. 24 is a block diagram of a document classification device for explaining an embodiment corresponding to claims 10 to 13, 23 to 26 and 36 to 39 of the present invention.

FIG. 25 is a diagram showing an example of a pseudo code for expressing an identification number of a word of document data and its appearance frequency.

FIG. 26 is a diagram showing an example of a pseudo code for generating a vector representation of each document data.

FIG. 27 is a diagram showing an example of pseudo code for classifying document vectors using a clustering technique.

FIG. 28 shows a geometric interpretation based on values of Rmn and Rnm.

[Explanation of symbols]

100, 200, 300, 400 ... document classification device, 10
1, 301, 401... Document input unit, 102, 302, 4
02: Document analysis unit, 103, 304, 407: Document classification unit, 104: Representative word extraction unit, 105: Related word extraction unit, 1
06: partial document set information generation unit, 107: classification result storage unit, 108, 305, 412: output unit, 201: unintended partial document set generation unit, 303, 404: document vector space generation unit, 403: document analysis result Storage unit, 405: Document vector data storage unit, 406: Classification number determination unit, 408:
Document classification result storage unit, 409 ... repetition determination unit, 410
... A partial document set relation calculating section 411... A partial document set relation storage section.

Claims (39)

[Claims] 1. A document classification apparatus for classifying a set of documents according to their contents, comprising: a document input unit for inputting a plurality of documents; and a word constituting each document from each of the documents input by the document input unit. A document analysis unit for extracting information; a document classification unit for classifying a document set of the plurality of documents into several partial document sets based on word information of each document extracted by the document analysis unit; A representative word extracting unit that extracts a set of representative words from each partial document set classified by the document classifying unit, and a relevant word dictionary that describes a related word of an arbitrary word. A related word extraction unit for extracting a related word set for each of the representative word sets of each partial document set extracted, and a related word set extracted by the related word extraction unit and a representative word set extracted by the representative word extraction unit. Each partial document set A partial document set information generating unit that generates individual partial document sets and related information between the partial document sets based on information about documents belonging to And a classification result storage unit that stores the information together with the information generated by the unit. 2. The document classification device according to claim 1, wherein the related word set extracted by the related word extracting unit is:
A document classification device characterized by a combination of at least one of synonyms, synonyms, and antonyms. 3. The document classification apparatus according to claim 1, wherein the related word set extracted by the related word extracting unit includes at least an opposite word, and an opposite word extracted from a representative word set of a certain partial document set. If the set does not match the representative word set of any other sub-document set including itself, the process of extracting a document including the inconsistent opposite word set from the document set and generating a new sub-document set is performed for all sub-documents. A document classification apparatus, further comprising a reciprocal partial document set generation unit that recursively repeats a set. 4. The document classification apparatus according to claim 1, wherein the related words extracted by the related word extracting unit include at least an opposite word, and an opposite word set extracted from a representative word set of a certain partial document set. If the word set does not match the representative word set of any other sub-document set including itself, the document set including the word set obtained by removing the non-matching opposite word set and the representative word corresponding to the opposite word set from the representative word set. A document classification apparatus further comprising: a reciprocal partial document set generation unit that recursively repeats a process of generating a new partial document set by extracting the partial document sets from all the partial document sets. 5. A document classification apparatus for classifying documents according to the contents of the document, comprising: a document input unit for inputting document data; And a document vector space generating unit for generating a document vector space for expressing the document data in a multidimensional vector space from the analysis information of the document data extracted by the document analyzing unit. And a document classifying unit that classifies document data by using a statistical method in the document vector space generated by the document vector space generating unit, and a word having a specific part of speech extracted by the document analyzing unit. Is replaced with a word generated by combining one or more words extracted before and after the specific part of speech based on the part of speech information of the specific part of speech. And a part of speech information of the specific part of speech is appropriately replaced. 6. The document classification device according to claim 5, wherein the document classification unit classifies the document data by using a clustering method as a statistical method. 7. The document classification device according to claim 5, wherein the document analysis unit replaces a word and a part of speech with respect to a word whose part of speech is a prefix, a suffix, a classifier, or a part of speech similar thereto. Document classification apparatus characterized by the above-mentioned. 8. The document classification device according to claim 5, wherein the document analysis unit continues combining words until a word of a specific part of speech appears in the document analysis unit. 9. The document classification device according to claim 5, wherein the part of speech is combined with the word of the numeral suffix or the classifier in the document analysis unit with respect to the word of the numeral suffix or the classifier. A document classification device, wherein a plurality of words are deleted, and the information of the deleted words is not used in the document classification unit. 10. A document classification device for classifying a document data set according to the contents of a document, comprising: a document input unit for inputting a document data set; and applying morphological analysis to all the document data to form the document data. A document analysis unit that extracts words together with their part-of-speech information, a document analysis result storage unit that stores an analysis result of the document data extracted by the document analysis unit, and a document data extracted by the document analysis unit A document vector space generating unit for generating a vector space for expressing the document data in a multidimensional vector space from the analysis information of the document vector space, and vector data of each document data in the document vector space generated by the document vector space generating unit A document vector data storage unit that stores the document vector, a classification number determination unit that determines the classification number of the document data set based on designated conditions, A document classifying unit that classifies document data into partial document sets of the specified number of classes by using a statistical method in a document vector space generated by a space generating unit, and stores a classification result generated by the document classifying unit. A classification result storage unit, a repetition determination unit that determines whether to repeat the processing from the classification number determination unit to the classification result storage unit, and stores the document vector data storage unit and the classification result storage unit. And a relation information between the partial document sets generated by the partial document set relation calculation unit for calculating relation information between all the partial document sets generated using the generated information. And a partial document set relation storage unit for storing a document classification set. 11. The document classification device according to claim 10, wherein the statistical method used in the document classification unit is a non-hierarchical clustering method. 12. The document classification device according to claim 10, wherein the relationship calculated by the partial document set relationship calculation unit is a similarity relationship and an inclusive relationship. 13. The document classification device according to claim 12, wherein the relationship between the partial document sets is calculated using only word information extracted from each partial document set. 14. A document classification method for classifying a set of documents in accordance with the contents thereof, comprising: a document input step of inputting a plurality of documents; A document analysis step of extracting information; a document classification step of classifying a document set of the plurality of documents into several partial document sets based on word information of each document extracted in the document analysis step; A representative word extraction step of extracting those representative word sets from each partial document set classified in the document classification step,
A related word extracting step of extracting a related word set for each representative word set of each partial document set extracted in the representative word extracting step using a related word dictionary in which the related word is described for an arbitrary word; Based on the related word set extracted in the word extraction step, the representative word set extracted in the representative word extraction step, and information on documents belonging to each partial document set, the individual partial document sets and the relation between the partial document sets A partial document set information generating step of generating information; and a classification result storing step of storing the classification result in the document classification step together with the information generated by the partial document set information generating unit. Document classification method to be used. 15. The document classification method according to claim 14, wherein the related word set extracted in the related word extracting step is a combination of at least one of a synonym, a synonym, and an antonym. A document classification method, characterized in that: 16. The document classification method according to claim 14, wherein the related word set extracted in the related word extracting step includes at least an opposite word, and an opposite word extracted from a representative word set of a certain partial document set. If the set does not match the representative word set of any other sub-document set including itself, the process of extracting a document including the inconsistent opposite word set from the document set and generating a new sub-document set is performed for all sub-documents. A document classification method, further comprising a step of generating a reciprocal partial document set recursively repeating the set. 17. The document classification method according to claim 14, wherein the related words extracted in the related word extracting step include at least an opposite word, and an opposite word set extracted from a representative word set of a certain partial document set. If the word set does not match the representative word set of any other sub-document set including itself, the document set including the word set obtained by removing the non-matching opposite word set and the representative word corresponding to the opposite word set from the representative word set. And generating a new partial document set, and recursively repeating the process of generating a new partial document set for all the partial document sets. 18. A document classification method for classifying a document according to the contents of the document, comprising: a document inputting step of inputting document data; And a document vector space generating step of generating a document vector space for expressing the document data in a multidimensional vector space from the analysis information of the document data extracted in the document analyzing step. And a document classification step of classifying the document data by using a statistical method in the document vector space generated in the document vector space generation step, and a word having a specific part of speech extracted in the document analysis step. Based on the part-of-speech information of the specific part of speech, one or more words extracted before and after the specific part of speech A document classification apparatus characterized in that the word classification is replaced with a word generated by combining the word class and the part of speech information of the specific part of speech. 19. The document classification method according to claim 18, wherein in the document classification step, the document data is classified by using a clustering method as a statistical method. 20. The document classification method according to claim 18 or 19, wherein in the document analyzing step, the words and the parts of speech are replaced for words whose parts of speech are prefixes, suffixes, classifiers, and similar parts of speech. A document classification method, characterized in that: 21. The document classification method according to claim 18, wherein the combining of the words is continued until a word of a specific part of speech appears in the document analysis step. 22. The document classification method according to claim 18, wherein in the document analysis step, a part of speech is combined with a word of a numeral suffix or a classifier with a word of the numeral suffix or a classifier. A document classification method, wherein a plurality of words are deleted, and the information of the deleted words is not used in the document classification step. 23. A document classification method for classifying a document data set according to the contents of the document, the document input step of inputting the document data set, and applying morphological analysis to all the document data to form the document data A document analysis step of extracting words together with their part-of-speech information, a document analysis result storage step of storing analysis results of the document data extracted in the document analysis step, and a document data extracted in the document analysis step A document vector space generating step of generating a vector space for expressing the document data in a multidimensional vector space from the analysis information of
A document vector data storing step of storing vector data of each document data in the document vector space generated in the document vector space generating step, a classification number determining step of determining a classification number of the document data set from designated conditions; A document classification step of classifying document data into partial document sets of the specified classification number by using a statistical method in the document vector space generated in the document vector space generation step; A classification result storing step of storing a classification result; a repetition determining step of determining whether to repeat the processing from the classification number determining step to the classification result storing step; a document vector data storing step and the classification result The source generated using the information stored in the storage step Calculating the relation information between the partial document sets, and storing the relation information between the partial document sets generated in the calculating step. And a document classification method. 24. The document classification method according to claim 23, wherein the statistical method used in the document classification step is a non-hierarchical clustering method. 25. The document classification method according to claim 23, wherein the relation calculated in the partial document set relation calculation step is a similarity relation and an inclusion relation. 26. The document classification method according to claim 25, wherein the relationship between the partial document sets is calculated using only word information extracted from each partial document set. 27. A computer-readable recording medium on which a program for executing a document classification method for classifying a document set according to its contents is recorded, the document inputting step of inputting a plurality of documents, and the document inputting step. A document analysis step of extracting word information constituting each document from each of the documents input in the step; and a document set of the plurality of documents based on the word information of each document extracted in the document analysis step. A document classification step of classifying the document into several partial document sets, a representative word extraction step of extracting a representative word set from each of the partial document sets classified in the document classification step, A related word set for each representative word set of each partial document set extracted in the representative word extracting step using the described related word dictionary Related word extraction step of extracting
Based on the related word set extracted in the related word extracting step, the representative word set extracted in the representative word extracting step, and information on documents belonging to each partial document set, each partial document set and each partial document set And a classification result storing step of storing the classification result in the document classification step together with the information generated by the partial document collection information generation unit. A computer-readable recording medium on which a program for executing the method is recorded. 28. A computer-readable recording medium recording a program for executing the document classification method according to claim 27, wherein a related word set extracted in the related word extracting step is a synonym or a synonym. And a computer-readable recording medium recording a program for executing a document classification method that is a combination of at least one of the opposite words. 29. A computer-readable recording medium storing a program for executing the document classification method according to claim 27, wherein a related word set extracted in the related word extracting step includes at least an opposite word. ,
If the opposite word set extracted from the representative word set of a certain partial document set does not match the representative word set of any other partial document set including itself, the document including the inconsistent opposite word set is extracted from the document set. A computer-readable recording medium storing a program for executing a document classification method further including a step of generating a recursive partial document set, which recursively repeats a process of generating a new partial document set for all partial document sets. 30. A computer-readable recording medium recording a program for executing the document classification method according to claim 27, wherein the related words extracted in the related word extracting step include at least an opposite word, If the antonym set extracted from the representative word set of a partial document set does not match the representative word set of any other partial document set including itself, the non-matching opposite word set and the representative word set are changed to the opposite word set. A document further including an anonymous sub-document set generation step of extracting a document including the word set excluding the corresponding representative word from the document set and recursively repeating the process of generating a new sub-document set for all the sub-document sets A computer-readable recording medium recording a program for executing the classification method. 31. A computer-readable recording medium on which a program for executing a document classification method for classifying documents according to the content of the document is recorded, wherein a document input step of inputting document data; A document analysis step of applying morphological analysis to extract the words constituting the document data together with their part of speech information and the like; and converting the document data in a multidimensional vector space from the analysis information of the document data extracted in the document analysis step. A document vector space generating step of generating a document vector space for expression, and a document classifying step of classifying document data by using a statistical method in the document vector space generated in the document vector space generating step, A word having a specific part of speech extracted in the document analysis step is identified. A document classification method for replacing a word generated by combining with one or more words extracted before and after the specific part of speech based on the part of speech information of the specific part of speech and appropriately replacing the part of speech information of the specific part of speech Computer-readable recording medium on which a program for executing the program is recorded. 32. A computer-readable recording medium on which a program for executing the document classification method according to claim 31 is recorded, wherein the classification of the document data is performed by using a clustering method as a statistical method in the document classification step. A computer-readable storage medium storing a program for executing a document classification method to be performed. 33. A computer-readable recording medium recording a program for executing the document classification method according to claim 31 or 32, wherein the part of speech is a prefix, a suffix, a classifier, and a part thereof in the document analysis step. A computer-readable storage medium storing a program for executing a document classification method for replacing words and parts of speech for words that are parts of speech similar to. 34. A computer-readable recording medium on which a program for executing the document classification method according to claim 31 is recorded, wherein a word of a specific part of speech appears in the document analysis step. A computer-readable storage medium storing a program for executing a document classification method that continues to combine words up to a word. 35. A computer-readable recording medium on which a program for executing the document classification method according to claim 31 is recorded, wherein the part of speech is a numerical suffix or a classifier in the document analyzing step. A computer-readable recording program for deleting a plurality of words associated with the word of the number suffix or classifier for the word, and executing a document classification method that does not use information of the deleted word in the document classification step. Recording medium. 36. A computer-readable recording medium recording a program for executing a document classification method for classifying a document data set according to the contents of a document,
A document input step of inputting a document data set, a document analysis step of applying morphological analysis to all the document data, and extracting words constituting the document data together with their part of speech information, and the like; Generating a vector space for expressing the document data in a multi-dimensional vector space from the analysis information of the document data extracted in the document analysis step; A document vector space generating step; a document vector data storing step of storing vector data of each document data in the document vector space generated in the document vector space generating step; The number of classifications to be determined and the document vector space generation step A document classification step of classifying document data into partial document sets of the specified number of classifications by using a statistical method in the document vector space, and a classification result storage step of storing a classification result generated in the document classification step. A repetition determining step of determining whether or not to repeat the processing from the classification number determination step to the classification result storage step; and using the information stored in the document vector data storage step and the classification result storage step. Calculating the relation information between all the partial document sets generated by the partial document sets, and storing the relation information between the partial document sets generated in the calculating step between the partial document sets. Computer storing a program for executing a document classification method including a set relation storage step. Over data-readable recording medium. 37. A computer-readable recording medium having recorded thereon a program for executing the document classification method according to claim 36, wherein the statistical method used in the document classification step is a non-hierarchical clustering method. A computer-readable recording medium on which a program for executing the method is recorded. 38. A computer-readable recording medium on which a program for executing the document classification method according to claim 36 or 37 is recorded, wherein the relation calculated in the partial document set relation calculation step is similar. A computer-readable storage medium storing a program for executing a document classification method having a relation and an inclusion relation. 39. A computer-readable recording medium recording a program for executing the document classification method according to claim 38, wherein the relation between the partial document sets is only word information extracted from each partial document set. A computer-readable recording medium on which a program for executing a document classification method calculated by using a computer is recorded.