JPH0298778A - Automatic document classification device - Google Patents

Abstract

PURPOSE:To decide a classification destination based on a bylaw and an illustration by using a generation rule describing the bylaw and the illustration at the time of classifying based on relation between classification fields and cogenerating relation between appearing words. CONSTITUTION:A classification candidate selection means 200 extracts the key word of an input sentence, and collates it with the key word appearing partially in a document at every prescribed classification stored in a key word storage means 100, and selects at least one classification candidate. A generation rule storage means 300 stores the generation rule which regulates a set including plural pieces of classification based on the bylaw and the illustration in the classification and one piece of classification corresponding to the set. A classification candidate decision means 400 collates at least one classification candidate selected at the classification candidate selection means 200 with the generation rule stored in the generation rule storage means 300, and decides one classification candidate. Thereby, it is possible to decide the classification destination based on the bylaw and the illustration.

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to an apparatus for automatically classifying the words of documents stored in a database in order to create a document database.

[Conventional technology]

When creating a database containing a large number of documents such as newspaper articles, patent application specifications, and technical papers, it is necessary to assign a classification code to each document when inputting the database. Conventionally, for this purpose, a method has been used that focuses on words that tend to appear biasedly in a certain classification field.

In this method, words in documents that have already been classified are statistically processed to determine words that appear biasedly in each field (referred to as field identification words). Next, this is a method of determining the classification destination of a document using field identification words in unclassified documents as clues. For example, when setting classification fields such as sports, world, etc., if the word "Olympic" appears biasedly in the field "sports" and the word "diplomat" appears in the field "world", then Olympics, diplomacy, etc. Let ``kan'' be the field identification word.

However, with this method, if multiple field identification words appear in a document, and each field identification word is from a different field, the final classification destination is only statistical information such as the frequency of field identification words. Determine. For example, ``Professional baseball players form a labor union.

In this article, the field identification words for the field of sports and the field identification words for the field of labor appear, and the classification destination is determined only by the frequency of the field identification words for both fields.

[Problem to be solved by the invention]

However, in actual classification work, when there are competing classification candidates, the final classification target is determined by referring to the classification rules and legend, and the decision is not made simply based on word frequency. isn't it. These detailed rules and legends are written with a focus on statistical data, that is, relationships between classification fields, co-occurrence relationships between words, etc., rather than the frequency of words in a document.

In other words, conventional statistical methods alone cannot describe detailed rules and legends during classification, and therefore, when there are competing candidates for classification, the detailed rules and legends have the disadvantage that it is not easy to determine the classification destination.

Therefore, the present invention solves the problems of the prior art described above, and uses production rules that describe detailed rules and legends during classification based on the relationship between classification fields and co-occurrence relationships between words. The purpose is to enable classification destination determination based on

[Means to solve the problem]

FIG. 1 is a functional block diagram of the present invention. J2] In the figure, the keyword storage means 100 describes keywords that appear in documents in a biased manner for each predetermined classification.

The classification candidate selection means 200 extracts keywords from the input text, collates them with keywords for each classification, and selects at least one keyword.
Select one classification candidate.

The generation rule storage unit 300 stores in advance a generation rule that defines a set including a plurality of categories and one category corresponding to the set based on the detailed rules and legends of the classification.

The classification candidate determining means 400 determines one classification candidate by comparing at least one classification candidate selected by the classification candidate selecting means with the generation rule stored in the generation rule storage means.

[Effect]

The classification candidate selection means 200 extracts keywords from the input text and compares them with keywords that appear in the document in a biased manner for each predetermined classification stored in the keyword storage means 100. For example, the bias of keywords in the input text is classified as A.
and B, the classifications A and 8 are selected as classification candidates. Then, the classification candidates determining means 400 compares the classifications A and B with the generation rules stored in the generation rule storage means 300. The production rules are based on detailed rules and legends, and for example, a set consisting of categories A and 8 is ultimately specified as category B. The classification candidate determining means 400 searches for a generation rule consisting of candidate classifications A and B,
Based on this production rule, classification B is finally determined as one classification candidate.

In this way, classification candidates are determined by considering not only the frequency of appearance of keywords, but also the detailed rules and legends used during classification, which solves the conventional problems.

Embodiment C] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 2 shows an embodiment in which the present invention is implemented by hardware. Reference numeral 1 denotes a sample document file, which is sample data used to extract statistical characteristics of word omissions in documents for each field. 2 is a field feature extraction device that extracts document features for each field. The characteristics of documents for each field (bias in the frequency of appearance of keywords for each field) are as follows:
Record in the 7-point table (memory). 4 is a classification code production rule dictionary that describes detailed rules and legends used for classification in the form of production rules. 5 is an input document file that stores unclassified documents. A classification light identification device 6 automatically assigns a classification code to an unclassified document and outputs the result to an output document file 7.

Here, the relationship with the functional block diagram of the present invention shown in FIG. 1 will be explained. The keyword storage means 100 corresponds to the sample document file 11, the field feature extraction device 2, and the field identification word score table 3. The classification candidate selection means 200 corresponds to the input document file 5 and the classification light identification device 6. The generation rule storage means 300 corresponds to the classification code generation rule dictionary 4. The classification candidate determining means 400 corresponds to the classification light identification device 6. That is, the classification light identification device 6 uses the classification candidate selection means 200
and classification candidate determining means 400.

The output document file 7 is generated by the classification candidate determining means 4. The final classification candidates (classification light candidates) determined in OO are stored in the form of a hood.

FIG. 3 is a detailed block diagram of the field feature extraction device 2. As shown in the figure, the field feature extraction device 2 has an input device Ji.
o, Keyword extraction unit 111 Field frequency meter Brightness unit 12. Score sheet calculation section 13. It is configured to include a keyword table 14 and a keyword field frequency table 15. Input device?
Keywords are extracted by the keyword extraction unit 11 from the classification coded document (sample data) read from the sample document file 1 via Z10, and stored in the keyword table 14. During this operation, the keyword extraction unit 11 uses the thesaurus 9 to unify different notations, paraphrase substitute expressions, generate higher-level concepts from lower-level concepts, and unify terms expressing the same concept. . Next, the field frequency calculation unit 12 calculates the frequency of appearance of the keywords in the keyword table 14 using the classification code
The keywords are sequentially added to the frequency table 15 by keyword field based on the code indicating the classification. The above operations are performed for the number of sample documents.

(The Q table calculation unit 13 creates a field identification word score table 3 by deleting unnecessary keywords and converting frequencies to scores. Specifically, from the keyword field frequency table 15, Keywords that appear in a small number of times when doing so, (i) Keywords that appear uniformly across all fields and have little bias in the fields in which they appear are deleted.Next, the frequency of each keyword in each field is converted into a score.

FIG. 4 is a detailed block diagram of the classification light identification device 6.

The classification light identification device 6 includes an input device 16. Keyword extraction part 1
71 field score calculation unit 18, one production rule application unit, output device i20. Bungata memory 21゜keyword table 221
Classification code candidate table 23 and classification code table 2
4.

Unclassified input documents stored in the input text file 5 are read by the input device 16 and provided to the keyword extracting section 17 . The keyword extraction unit 17 extracts keywords from the input document and stores them in the keyword table 22. Next, field score i! ', the brightening unit 18 breaks down the scores of the keywords listed in the field identification + 1 word score table 3 from the keyword table 22 for each classification code. Based on the scores of each classification code (here), the first
The classification code with the highest score and the classification code with the score within a predetermined threshold from that score are taken as candidate classification codes. Then, this classification code is written into the classification code candidate table 23.

The generation rule application unit 19 uses the classification code candidate table 23
When a plurality of classification codes exist, the classification code generation rule dictionary 4 is used to finally narrow down the classification codes to one. The classification code generation rule consists of the condition part, classification light,
The rules consist of three points (see Figure 5). The condition part consists of a string of one or more keywords or classification codes.

Each keyword or classification code is C1 (1≦1≦n)
Expressed as It also represents the certainty of the rule of score α. The condition section refers to the keyword table 22 and the classification code candidate table 23 and checks whether each term of C1 to Qn exists in the table.

Then, when all terms C1 to Cn are in the table,
- Generate a classification code corresponding to the conditional part and give a score. The final classification code is determined by the following 0) to ■, depending on the number of applied production rules.

('1) When the number of applied production rules is 0, the classification code with the highest score in the field score calculation section is used as the final classification code.

(2) When only one production rule is applied, the classification code written in that production rule is used as the final classification destination.

■When two or more production rules are applied, for each classification code, add a certain score to the production rule that generated it, and use the classification code with Rtl'F1 point as the final classification destination. .

The final classification code determined in this manner is stored in the classification code table 24. Finally, the output device 20
outputs the finally determined classification code together with the document to the output text file 7 on the external storage device.

FIG. 5 is an example of classification code generation rules. In the figure, []
indicates the classification code, <> indicates the keyword, and number ■ indicates the rule number of the classification code generation rule. #1 has the legend that labor issues and employment situations in the transportation and communications fields are classified as articles in the 'labor' category, and #2 has a legend that states that articles on labor issues and employment situations outside of Japan are classified as articles in the 'world' category. It represents the legend that it is classified as an article. Also, #4
represents the legend that if deliberations in the field of science and technology are held in the Diet, the article will be classified as an article in the field of "Politics".

FIG. 6 shows an example of a cheekiness table by keyword field. In this example, the classification codes are political, economic, labor,...
, sports, and 10 fields from around the world. It shows how many times each keyword appears in documents in each field. The keyword "labor union" appears 5 times in articles in the political field and 56 times in articles in the labor field.

FIG. 7 is an example of field identification word score table 3, which was created based on the table in FIG. 6. In the figure, *marks indicate words to be deleted. In other words, the overall appearance frequency of "ZENDENTSU""OFFSIDE" is small, 3 times and 1 time, respectively, and even if it is registered as field identification Ill, there is a low probability that it will appear in other documents, so it is Delete from the table in the figure.Also,
1 Tokyo" and "Newspaper article" appear on average in documents in each classification field, so conversely, it is difficult to use that keyword as a clue to identify the field. Therefore, it is inappropriate as a field identification word. , from the table in Figure 6.Next, the frequency xj of the field identification word j selected in this way
In this operation example, k is converted into a score Xjk using equation 1). In this way, the table shown in FIG. 7 is obtained.

Yjk"(xJk-Mjk"Mjk...(Formula 1)Mj
k: The theoretical frequency of word j in the field, which is obtained by (Equation 2).

(■-Total number of fields) FIG. 8 is an example of the operation of the classification destination identification device shown in FIG. 4. The keyword automatic extraction unit 17 extracts a keyword table 26 from the input document 25 read from the input device. Next, from the keyword table, field identification I
Add up the scores for the keywords "'Professional baseball, Yakult, night game, labor union" listed in the one-word score table (Figure 7) for each classification field. Here, if a word appears multiple times, the number of appearances is added. This result is shown in Figure 8 (C).
The scores for each classification code are shown in 27. Here, the sports and labor classification codes with the first and second scores are written as candidates in the classification code candidate table 23. Therefore, this determination is performed by the generation rule application section. The production rule applying unit 19 applies the production rule 28 shown in FIG. 8 (D>) and sets the final classification code 29 as labor.

The present invention has been described in detail above. In the above embodiment, classification code generation rule No. 4 is used, which consists of a classification code and a keyword, but in principle, it is also possible to use classification code generation rules consisting only of classification codes, such as #1 to #3. .

〔Effect of the invention〕

As explained above, according to the present invention, by using classification code generation rules, relationships between classification fields and co-occurrence relationships between words, which could not be expressed using conventional techniques, can be expressed. It is possible to classify input sentences by taking into account production rules and legends.

[Brief explanation of drawings]

Fig. 1 is a functional block diagram of the present invention, Fig. 2 is a block diagram of an embodiment of the present invention, and Fig. 3 is a functional block diagram of the present invention.
Figure 4 is a block diagram of the classification optical identification device shown in Figure 2, Figure 5 is a diagram showing an example of classification code generation rules, Figure 6 is keyword frequency by field. FIG. 7 is a diagram showing a table, FIG. 7 is a diagram showing field identification words, and FIG. 8 is a diagram showing an example of the operation of the classification light identification device. 1... Sample document file, 2... Field feature extraction device, 3... Field identification intermediate score table, 4... Classification code generation rule dictionary, 5... Input document file, 6... classification light identification device, 7... output document file,
8...Japanese dictionary, 9...Thesaurus, 10...
Input device, 11...Keyword extraction unit, 12...Field-based gradient calculation unit, 13...Score table needle n part, 14...
Keyword table, 15... Frequency table by keyword field, 16... Input bag d, 17... Keyword extraction section, 18... Field score 81 part, 19... Production rule application section, 20. ... Output device η, 21... Document memory, 22... Keyword table, 23... Classification code candidate table, 24... Classification code table, 25... Input document,
26... Keyword table, 27... Score of each classification code, 28... Applied production rule, 29... Final classification code, 100... Keyword storage means, 200... Classification candidate Selection means, 300... Generation rule storage means, 400... Classification candidate determination means. Patent applicant Nippon Telegraph and Telephone Corporation Invention 5') Functional block diagram Figure 1 Figure 2 Figure 3 Yang Offside A diagram showing the keyword frequency table by field. ,,,,,,,,
,, ,,, ,,,,,,,,,,,,,,,,,,,,,
, , , , Y Figure 7 showing the Ik field identification word score table Labor], [Sports] = [Science and Technology], <Parliament> → Classification destination [Labor] [World Co. [Labor Co. [Politics] Diagram showing an example of classification code generation rules Figure 5 Operation of keyword classification optical identification device Figure 8 showing an example

Claims (1)

[Scope of Claim] An automatic document classification device that determines the classification of an input text, comprising: a keyword description means that describes keywords that appear biasedly in a document for each predetermined classification; Classification candidate selection means that selects at least one classification candidate by comparing keywords for each classification, and predefining a set containing multiple classifications and one classification corresponding to the set based on detailed rules and legends for classification. 1 by comparing at least one classification candidate selected by the production rule storage means that stores the production rules that have been selected and the classification candidate selection means with the production rules stored in the production rule storage means.
An automatic document classification device comprising: classification candidate determining means for determining two classification candidates;