JPH06348755A - Method and system for classifying document - Google Patents

Abstract

PURPOSE:To save huge human labor for classifying document data by preparing any specified dictionary for classification and classifying non-classified documents while utilizing this dictionary for classification. CONSTITUTION:A document data word division part 1 divides the document data into words while referring to classified document data (a) and registers the classified result on a classified word division table (b). A dictionary for classification preparation part 2 detects the appearance frequency of words while referring to the classified word division table (b) and registers the result on a dictionary (c) for classification. After the dictionary (c) is generated, the word division part 1 divides the document data into words while referring to classification target document data (d) and registers the divided result on a classification target word division table (e). A document classification part 3 classifies the document data (e) while referring to the division table (e) and the dictionary (c) for classification and registers them on a document classified result (f). Thus, since the system is provided with the dictionary for classification preparing function and the document classifying function, the document data can be automatically classified.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for automatically classifying document data, and more particularly to a method and system for classifying a document suitable for automatically classifying a large amount of document data.

[0002]

2. Description of the Related Art A conventional document classification method is described, for example, in Japanese Patent Laid-Open No. 158871/1990. The conceptual feature amount of each document is obtained from the frequency value of the keyword included in the document, and the document is classified according to this.

[0003]

In the conventional document classification method, the distance between documents is obtained according to the difference in the conceptual feature amount between documents, and the document is classified by this distance. In order to obtain it, a thesaurus for document classification and a keyword classification item are manually registered in advance and used. That is, there is a problem that the information for classification must be manually defined in order to classify the documents.

In order to solve the above problem, an object of the present invention is to use the calligraphy data classified into a plurality and the frequency of appearance of each word in the classification to automatically extract keywords for each classification, (EN) A method and system for creating a classification dictionary and classifying unclassified documents by using this classification dictionary.

[0005]

In order to achieve the above object, the document classification method of the present invention extracts a word as a keyword for each classification from classified document data in which one classification consists of one document data or more. A method for creating a classification dictionary, a method for classifying unclassified document data using the classification dictionary,
A method of newly detecting a keyword from the result of classification by the method of classifying unclassified document data and registering it in the classification dictionary is provided. The document classification system of the present invention is a natural language processing system including a program for executing the above document classification method.

[0006]

The document classification method and system of the present invention first obtains classified document data. Next, if the creation range of the classification dictionary is specified by the user by the items that make up the document data, the document data of only the specified items is processed, and if not specified, the entire document data is processed. Create a classification dictionary. Next, a word existing in only one classification is detected for each classification obtained from the classified document data, and this word is registered in the classification dictionary as a keyword of the corresponding classification. When the document data is composed of a plurality of items, the correspondence between the keywords and the items is also registered in the classification dictionary.

After the classification dictionary is created, unclassified document data is acquired. Next, if the user specifies the classification range of the document data by the items that compose the document data, the document data of only the specified items is processed, and if not specified, the entire document data is processed. Classify. The word of the unclassified document data is detected and compared with the keyword in the classification dictionary to find the number of matches for each classification.

Next, the classification with the largest number of keyword matches is taken as the classification result of the classification target document data. In addition, priority is given to the order of the classification having the largest number of coincidences, and the result is classified. In addition, if the user is inquired whether or not the result of classification by the method for classifying unclassified document data is correct, and if the result is correct, the words in the document data classified by the method for classifying unclassified document data are used for classification. If the word does not exist in the dictionary, this word is registered in the classification dictionary as a keyword representing the classification. Moreover, when registering a keyword in the classification dictionary, only the keyword designated by the user is registered in the classification dictionary.

In addition, if the user inquires whether or not the result of classification by the method of classifying unclassified document data is correct, the matched keyword is presented to the user,
When an incorrect keyword exists, the user deletes the keyword from the classification dictionary according to the instruction. As described above, unclassified document data can be automatically classified, and at the same time, the classification dictionary can be self-proliferated.

[0010]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing an embodiment of the document classification method of the present invention. Document data word division unit 1
Refers to the classified document data a, divides the document data into words, and registers the division result in the classified word division table b. The classification dictionary creating unit 2 uses the classified word division table b.
, The appearance frequency of the word is detected and registered in the classification dictionary c. After the classification dictionary is generated, the document data word division unit 1 refers to the classification target document data b, divides the document data into words, and divides the division result into the classification target word division table e.
Register with. The document classification unit 3 refers to the classification target word division table e and the classification dictionary, classifies the classification target document data d, and registers it in the document classification result f.

By thus providing the classification dictionary creating function and the document classification function, automatic classification of document data is realized.

As is clear from the figure, the document data word division unit 1, the classification dictionary creation unit 2, and the document classification unit 3 indicate the processing, and the classified document data a, the classified word division table b,
The classification dictionary c, the classification target document data d, the classification target document word division table e, and the document classification result f are files (also called tables). Thus, according to this embodiment,
Each functional block is configured by program logic. Therefore, it is possible to implement an LSI for each functional block unit, and the processing speed can be increased as a document classification device.

FIG. 2 is a block diagram showing the overall hardware configuration of the document classification device shown in FIG. The input / output device 4 inputs data and displays various information. The processor 5 executes the processing in FIG. 1 based on the program. The storage device 6 stores various document data a, various programs, and the like shown in FIG. further,
The storage device 6 is a memory for executing each process of the processor 5, a working area ae, a document data word dividing section storage area 10, a classification dictionary creating section storage area 20, a document classification section storage area 30, a document data storage area ad. The storage unit has a word division table storage area be, a classification dictionary storage area c, and a document classification result storage area f.

Each program stored in the storage device 6 is
It is executed in the processor 5. To do that,
The input / output device 4 is used as necessary.

FIG. 3 is a PAD diagram (Problem Analysis Diagr) showing the processing procedure of the document data word division unit in FIG.
am). Processing for obtaining document data from the classified document data a and performing word division and storing in the classified word division table b, or for obtaining document data from the classification target document data d and performing word division, classification target word division table It shows the processing up to storing in e.

This process will be described below with reference to the PAD diagram. By referring to the classified document data a, the following processing is performed from the first document data to the last document data (step 1
1). First, data for one document is acquired from the classified document data a (step 12), the document data is divided into words, and the headline character string and the part of speech are stored in the headline character string b1 and the part of speech b2 of the word division table b, respectively ( Step 13).

Next, it is determined whether the document data is divided into items (step 14). If the document data is divided into items, the item corresponding to each word is stored in the item b4 of the classified word division table b. (Step 15). Next, the field name and document number of the corresponding document data are stored in the classification name b5 and the document Nob6 of the classified word division table b (step 16). Next, the processing target is moved to the next document data (step 17).

When the classification target document data d is input in steps 11 to 17, the word division result is stored in the classification target word division table e. By steps 11 to 17, the classified document data a shown in FIG. 4 is acquired, word division is performed, and the word data is stored in the classified word division table b shown in FIG. Also, step 11
Through step 17, the classification target document data d shown in FIG. 9 is acquired and word division is performed and stored in the classification target word division table e shown in FIG.

FIG. 4 is an example of the classified document data a,
It is an example of classified document data a with a classification name and a document number. Classified document data is "Title""Summary""Purpose"
It consists of items such as "main contents" and "future tasks".

FIG. 5 is an example of a word division table, which is composed of headline character string b1, part of speech b2, item b3, classification name b4, and document Nob5. The headline character string b1 is a character string that constitutes a divided word, the part of speech C2 is the part of speech of the word, the item b3 is the item name of the item corresponding to the corresponding word, the classification name b4 is the classification name of the corresponding document data, and the document Nob5. Represents the identification number of the document data.

FIG. 6 shows a classification dictionary creating section 2 in FIG.
FIG. 6 is a PAD diagram showing the process of FIG. 4 and shows the process of creating a dictionary for classifying unclassified documents. Hereinafter, this processing will be described with reference to the PAD diagram.

Referring to the classified word division table b, the following processing is performed from the first record to the end record of the classified word division table (step 201). First, referring to the classified word division table b, information for one record is acquired (step 202), and the headline character string b1 and the part of speech b are acquired.
Another record having the same classification name B4 as 2 is searched from the classified word division table b, held, and stored in the work table (step 203).

Next, it is determined whether or not there is an item constituting the corresponding document data (step 204), and if the item exists, the number of appearances of each word for each item is obtained and stored in the work table (step 205). ). Next, the number of appearances of each word in the entire document data is obtained and stored in the work table (step 206).

Next, the record to be processed is moved to the next record (step 207). Step 201
After the above-mentioned repeated processing is completed, the following processing is performed from the first record to the last record of the work table stored in 201 to 206 (step 208).

First, the information of one record sentence is acquired from the work table and held (step 209). Next, it is determined whether or not another record having the same part of speech as the headline character string exists in the work table (step 210), and if it does not exist, the held record is stored in the classification dictionary 3 (step 211).

Next, the record to be processed is moved to the next record (step 212).

By steps 201 to 207,
The work table shown in FIG. 7 is generated from the classified word division table b shown in FIG.
12, the classification dictionary c shown in FIG. 8 is generated from the work table shown in FIG.

FIG. 7 is an example of the work table in FIG. 6, and is composed of items of a headline character string W1, a part of speech W2, a title W3 to a future task W7, a total W8, and a classification name W9. The headline character string W1 is a character string that constitutes a divided word, the part-of-speech W2 is the part-of-speech of the word, the title W3 to the future task W7 is the number of appearances of the word that appears in the items that form the document data, and the total W8 is the document data. The number of appearances of words appearing in the category name W9 represents the category name of the corresponding document data.

FIG. 8 is an example of the classification dictionary in FIG. 1, and is composed of items of a headline character string C1, a part of speech C2, a title C3 to a future task C7, a total C8, and a classification name C9. The headline character string C1 is a character string forming a divided word, the part-of-speech C2 is the part-of-speech of the word, the title C3 to the future task C.
Reference numeral 7 represents the number of appearances of words that appear in the items forming the document data, total C8 represents the number of appearances of words that appear in the document data, and classification name C9 represents the classification name of the corresponding document data. Unlike the work table shown in FIG. 7, this classification dictionary has no overlapping of the index character strings, and one index character string corresponds to one classification name.

FIG. 9 is an example of the classification target document data a, and is an example of the classification target document data d with a document number.
The classification target document data is composed of items such as “title”, “summary”, “purpose”, “main content”, and “future task”.

FIG. 10 shows an example of a classification target document word division table d, which includes a headline character string e1, a part of speech e2, and an item e.
3, the classification name e4, and the document Noe5. The headline character string e1 is a character string that constitutes a divided word, the part of speech e2 is the part of speech of the word, the item e3 is the item name of the corresponding item, the classification name e4 is the classification name of the corresponding document data,
Document Noe5 represents the identification number of the document data.

FIG. 11 is a PAD diagram showing the processing of the document classification unit 3 in FIG. This process is a process of classifying unclassified documents using a classification dictionary. Further, as a premise of this processing, it is assumed that the classification target document data d in FIG. 1 is word-divided by the document data word division unit 1 and the word division result is stored in the classification target document word division table e.

This process will be described below with reference to the PAD diagram. With reference to the classification target document word division table b, the following processing is performed from the first record to the last record of this table (step 301). First, it is determined whether or not the user has designated an item to be classified as a document (step 3
02), if specified, the information corresponding to one record is acquired only in the record corresponding to the corresponding item (step 3
03), if not specified, information for one record is acquired (step 304).

Next, the acquired record is set to a character string variable MI.
The data is stored in DASHI1 (step 305). Next, referring to the classification dictionary c, the following processing is performed from the first record to the last record of the classification dictionary (step 30).
6). First, it is determined whether or not the user has designated an item to be a document classification target (step 307). If so, only the record corresponding to the relevant item is limited to one record from the classification dictionary c. Obtain information (step 30
8) If not specified, information for one record is acquired (step 309).

Next, the acquired record is set to a character string variable MI.
It is stored in DASHI2 (step 310). Then M
The headline character string and part-of-speech are acquired from the information stored in IDASHI1 and MIDASHI2, and it is determined whether the headline character string and part-of-speech match (step 311). If they match, the number of matches is counted for each classification name and held ( In step 312), the matching headline character string and part of speech are held (step 313).

Next, the target of this iterative process is moved to the next record in the classification dictionary (step 314). Next, after the repetition processing of step 306 is completed, step 3
The target of the iterative processing of 01 is moved to the next record in the classification target document word division table (step 315). Next, after the iterative process of step 301 is completed, the classification name having the largest number of matches held above is stored in the document classification result f as the classification result of this document (step 316).
For the classification name having the number of times of matching of 1 or more, rearrange in order of the number of times of matching according to the classification name and store it as the classification candidate in the document classification result f (step 317). The result is stored in the classification result f (step 318).

By steps 301 to 318,
Documents can be classified based on the classification target document word division table e and the classification dictionary c shown in FIG. 10, and the document classification result f shown in FIG. 12 can be obtained. By showing the document classification result f to the user, the classification result can be confirmed.

FIG. 12 is an example of the document classification result f in FIG. In this example, the classification results of three documents are shown. The first document is the document with the document number “101” shown in FIG. 9, and as a result is classified into “language processing”. This means that the classification name “language processing” has the highest number of matching of words in the classification dictionary. In addition, the matched words are “dictionary”, “natural language”, “LR analysis”, etc., and the numerical value in parentheses indicates the number of times the words match. Also,
The category candidate having the largest number of word matches next to "language processing" is "communication".

The second document is the document No. "1" shown in FIG.
02 ”, and as a result, it is classified into“ electric circuit ”. This means that the classification name “electrical circuit” has the highest number of matching of words in the classification dictionary. In addition, the matched words are “AD conversion”, “analog”, “digital”, etc., and the numerical value in the parentheses indicates the number of times the words match. It also means that there was no classification candidate having the largest number of word matches next to “electrical circuit”.

The third document is the document No. "1" shown in FIG.
No. 03 ”document, and as a result, is classified as“ communication ”. This means that the classification name “communication” has the highest number of matching of words in the classification dictionary. The matched words are “network”, “protocol”, “LAN”, etc., and the numerical value in parentheses represents the number of times the words match.
Also, the category candidate having the highest number of word matches after "communication" is "electrical circuit".

As described above, according to this embodiment,
Conventionally, it is possible to automatically classify document data that has been manually classified, and it is possible to eliminate an enormous amount of work that is required to manually classify document data. In addition, a news service (bulletin board) such as a personal computer or Unix has a large number of newsgroups to be posted depending on the content of the document, and it may not be possible to determine which newsgroup to post, but according to this embodiment, , It becomes possible to automatically classify by creating a classification dictionary from documents classified by news group, and it is possible to automatically know the news group to be posted even if the news group of the document to be posted is not specified. Also, you can automatically classify and post.

Next, an extended example of the document classification method of the present invention will be described. FIG. 13 is a functional block diagram for realizing confirmation of the result of classification by the document classification method of the present invention with the user, and registering a new keyword in the classification dictionary based on the confirmed result. The document classification confirmation unit 4 refers to the document classification result f, presents the classification result of the classified document and the matching keyword to the user, prompts the user to confirm, and if there is a matching keyword that is not suitable, Delete the keyword from the classification dictionary by the user's designation,
Determine if there are new keywords in the classified documents,
If it exists, a new keyword is registered in the classification dictionary, and the classification dictionary is self-propagated.

[Screen 1] in FIG. 14 is an example of confirmation of the classification result of the classification target document. The presentation contents include a document number, a title, a classification result, and a classification candidate. In addition to the classification result, the keywords that match the keywords in the classification dictionary and the number of times the keywords match are also shown. Based on these information, the user determines whether the classification result is correct or not, and inputs whether the classification result is correct or not.

In [Screen 2] of FIG. 14, after the user judges that the classification result is correct in [Screen 1], a keyword that matches the keyword of the classification in the classification dictionary that is the basis of classification is used. In this example, the keyword is presented to the user and the user is prompted to specify any keyword that is not suitable. If specified, the keyword is deleted from the classification dictionary to improve the accuracy of the classification dictionary. In this example, the user responds that there is no such keyword.

[Screen 3] of FIG. 15 shows that when the classification result is correct, it does not exist in the classification dictionary, and when the keyword exists only in the document to be classified, this keyword is newly added to the classification. Whether to automatically register as a keyword,
Further, in this example, after confirming the keyword, the user prompts the user to respond by asking whether to select and register various keywords. In this example, the user responds to automatically register the newly appearing keyword without confirming it.

As described above, according to this extended example,
If an incorrect keyword is registered as an arbitrary classification in the classification dictionary, the user can check the keyword that matches the classification target document and delete the incorrect keyword to improve the accuracy of the classification dictionary. It can be easily realized. In addition, when a new keyword appears in the classification target document, the keyword is automatically registered in the classification dictionary, or presented to the user and only the necessary keyword is registered in the classification dictionary. It is possible to easily register a keyword in the classification dictionary or self-proliferate without spending time in registering the dictionary.

Next, another extended example of the document classification method of the present invention will be described. FIG. 16 shows that when a classification dictionary is created or when classification target documents are classified, if the document data is classified by certain items, the entire document data is created or classified. Allows users to specify items in the document instead of
It is a functional block diagram which implement | achieves making only the designated item the process target. Document classification target item specification unit 6
Is to classify the document data by recognizing the item designated by the user and referring to the classified document data a when creating the classification dictionary and acquiring the document data of the corresponding item from the document data. The document data of the corresponding item is acquired from the document data by referring to the classification target document data d, and only the document data acquired in the subsequent process is processed.

In addition, it is possible to specify and recognize a logical relationship between arbitrary items, rather than simply specifying the items. For example, words appearing in the item "main contents" or the item "future tasks" It is possible to easily implement the words that appear in both the item “main contents” and the item “future tasks” as processing targets.

As described above, according to this extended example,
By limiting the items without processing the entire document data, it is possible to expect an improvement in classification dictionary accuracy and classification accuracy depending on the character of the document data, and at the same time when a huge amount of document data or the document data itself is large. Also has the effect that the practical processing speed can be maintained and the processing speed is improved.

[0050]

As described above, according to the present invention, it is possible to automatically classify document data which has been conventionally classified manually, and it is possible to omit an enormous amount of work for manual document data classification work. Has the effect of becoming.
In addition, a news service (bulletin board) such as a personal computer or Unix has a large number of newsgroups to be posted depending on the content of the document, and it may not be possible to determine which newsgroup to post. , It becomes possible to automatically classify by creating a classification dictionary from documents classified by news group, and it is possible to automatically know the news group to be posted even if the news group of the document to be posted is not specified. Also, you can automatically classify and post. Further, there is an effect that the keyword can be easily registered in the classification dictionary, and at the same time, the classification dictionary can be self-propagated.

[Brief description of drawings]

FIG. 1 is a functional block diagram showing an embodiment of a document classification method according to the present invention.

FIG. 2 is a block diagram showing the overall hardware configuration of the document classification method in FIG.

FIG. 3 is a PAD diagram of the document data word division program in FIG.

FIG. 4 is an example of classified document data in FIG.

5 is an example of a classified word division table in FIG.

6 is a PA of the classification dictionary creating program in FIG.
Figure D.

7 is an example of a work table for creating the classification dictionary in FIG.

8 is an example of a classification dictionary in FIG.

FIG. 9 is an example of document data to be classified in FIG.

10 is an example of a classification target document word division table in FIG.

11 is a PAD of the document classification program in FIG.
Fig.

FIG. 12 is an example of a document classification result in FIG.

FIG. 13 is a functional block diagram showing a document classification result confirmation method and a classification dictionary learning method.

14 is an example of a dialogue with a user in the confirmation of the classification result and the learning of the dictionary for classification in FIG.

FIG. 15 is an example of a dialogue with a user in the confirmation of the classification result and the learning of the dictionary for classification in FIG.

FIG. 16 is a functional block diagram showing an item specifying method in a document to be processed in the document classification method.

[Explanation of symbols]

1 ... Document data word division unit, 2 ... Classification dictionary creation unit, 3
Document classification unit, a ... Classified document data, b ... Classified word division table, c ... Classification dictionary, d ... Classification target document data, e ... Classification target document word division table.

Claims (12)

[Claims] 1. A processing system for classifying document data, and means for creating a classification dictionary by extracting words that are keywords for each classification from document data that has already been classified into one document data. A document classification system having means for classifying uncategorized document data using a dictionary for use. 2. The document classification system according to claim 1, wherein the means for creating the classification dictionary detects words in the document data by using the classified document data, and detects words that appear in only one classification. A document classification system, which detects and registers the word as a keyword representing the classification in a classification dictionary. 3. The document classification system according to claim 1, wherein the means for creating the classification dictionary detects the items constituting the document data by using the classified document data and detects the word for each item. A document classification system, wherein a word that appears only in the classification is detected, and the word represents the classification and is registered in the classification dictionary as a keyword that appears in the item. 4. The document classification system according to claim 1, wherein the means for creating a classification dictionary has an item that constitutes the classified document data, and a range in which the user creates the classification dictionary is the item. The document classification system, wherein the classification dictionary is created only for the document data corresponding to the specified item in the classified document data when specified by. 5. The document classification system according to claim 1, wherein the means for classifying the unclassified document data detects a word in the unclassified document data and finds the number of matches with a keyword registered in the classification dictionary. The document classification system, wherein the category having the largest number of matches among the detected and matched categories is used as the classification result of the uncategorized document data. 6. The document classification system according to claim 1, wherein the means for classifying the unclassified document data detects a word in the unclassified document data and matches the number of keywords with the keywords registered in the classification dictionary. The document classification system characterized in that the unclassified document data is classified as a category having a high priority in the order of the number of coincidences among the matched categories. 7. The document classification system according to claim 1, wherein the means for classifying the unclassified document data has an item that constitutes the unclassified document data, and a range for classifying the unclassified document data by a user. Is designated by the item, the document classification system is characterized in that only the document data corresponding to the designated item in the uncategorized document data is processed. 8. A processing system for classifying document data, and means for creating a classification dictionary by extracting words that are keywords for each classification from document data that has been classified with one classification being one or more document data, Document classification system having means for classifying uncategorized document data using a dictionary for use, and means for newly detecting a keyword from the result of classification by the means for classifying the uncategorized document data and registering it in the classification dictionary . 9. The document classification system according to claim 8, wherein the means for newly detecting a keyword and registering it in the classification dictionary is a word in the document data classified by the means for classifying the unclassified document data. A document classification system, wherein if the word does not exist in the classification dictionary, the word is registered in the classification dictionary as a keyword representing the classification. 10. The document classification system according to claim 8, wherein the means for newly detecting a keyword and registering the keyword in the classification dictionary determines whether or not the classification result by the means for classifying the unclassified document data is correct. If the word in the document data classified by the means for classifying the unclassified document data is not a word in the classification dictionary, the word is used as a keyword representing the classification. A document classification system characterized by being registered in the classification dictionary. 11. The document classification system according to claim 8, wherein the means for newly detecting a keyword and registering the keyword in the classification dictionary determines whether or not the classification result by the means for classifying the unclassified document data is correct. If the word in the document data classified by the means for classifying the unclassified document data is not a word in the classification dictionary, the word is used as a keyword representing the classification. A document classification system characterized by presenting to a user and registering only keywords designated by the user in the classification dictionary. 12. The document classification system according to claim 8, wherein the means for newly detecting a keyword and registering the keyword in the classification dictionary determines whether or not the classification result by the means for classifying the unclassified document data is correct. The document classification is characterized by presenting the matching keyword to the user when it is instructed to be correct, and deleting the keyword from the classification dictionary according to the user's instruction when the incorrect keyword exists. system.