JP3464055B2 - Keyword extraction device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a keyword extracting device for extracting a keyword from a document, and can be applied to a document registering device, a document searching device and the like.

[0002]

2. Description of the Related Art Extracting a keyword from a document is an important technique for searching and classifying a document. Usually, in order to extract a keyword from a document, words are divided and part-of-speech is applied by a morphological analysis technique, and it is evaluated whether or not a word having a specific part-of-speech (particularly noun) is used as the keyword. Since this is a technical term in the target field, a method of extracting by paying attention to the keyword property of the word (Japanese Patent Laid-Open No. 62-287337) or a compound word when extracting a keyword of the compound word Judgment Method Based on Keyword Characters of Words Constituting Words and Roles in Compound Word Construction (“text database system based on short unit keywords”; Page 47). However, information to be used for such determination, the basically assumed to store the information in the dictionary, the easy to configure and maintenance of such information
Not . Therefore, a method for discriminating keyword candidates by character type without the need for information for such keywords (Japanese Patent Laid-Open No. 64-28770) or an extraction device for calculating keyword characteristics based on the length of a word and its frequency of use (JP-A-63-244259) has been proposed.

[0003]

Considering that a keyword characterizes an important part of the document, it can be said that a word having a higher degree of specialization / speciality as such characteristic is more likely to exhibit the characteristic. . Further, it can be said that a compound word in which a plurality of words are compounded is likely to show such specialization and peculiarity. From such a viewpoint, the object of the present invention is to use the degree of complexity of the words of the keyword candidates in order to solve the conventional problems, thereby avoiding having information for the keywords in the dictionary, and more accurate keywords. An object of the present invention is to provide a keyword extraction device that evaluates sex. Furthermore, when a keyword candidate is easily extracted without using a morphological analysis system, a keyword extraction device that easily obtains the degree of word complexity from the length of the keyword candidate and easily evaluates the keyword as a whole is provided. The purpose is to do. Furthermore, in the conventional evaluation of keyword candidates considering the appearance frequency of words, the same word candidates are counted respectively, but when similar words appear, they are counted separately, so There was a drawback that the appearance frequency would be small. As described above, in the present invention, when there is similarity between keyword candidates, it is an object to provide a keyword extraction device that improves the evaluation of keyword property by temporarily increasing the frequency of both keyword candidates based on the similarity. And Furthermore, since similarity between keywords usually requires a synonym dictionary, etc., and its development and maintenance are not easy. Therefore, by substituting the similarity between keyword candidates by the degree of duplication of character strings, It is an object of the present invention to provide a keyword extracting device that easily determines similarity. Furthermore, another object of the present invention is to provide a keyword extracting device that accurately displays the characteristics of a document by displaying keywords (candidates) arranged in the descending order of the keyword evaluation obtained by the present invention.

[0004]

The [Summary of invention of claim 1 wherein, with respect to electronic documents, to extract keywords candidate
And keyword candidate extraction unit that, in the keyword extracting device and a keyword candidate evaluating means for evaluating the keyword candidates said extraction, wherein the keyword candidate evaluating means frequency of occurrence of the keyword candidate during the document
And the degree of compounding of the words that form the keyword candidates.
It is characterized in that the keyword candidates are evaluated based on the evaluation value obtained based on the above.

According to a second aspect of the present invention, in the keyword extracting device of the first aspect, the keyword candidate extracting means extracts the character types of the electronic document according to the arrangement, and the keyword candidate evaluating means combines the words constituting the keyword candidates. The feature is that the degree is evaluated according to the type and length of the character string of the keyword candidate.

According to a third aspect of the invention, in the keyword extracting device according to the first or second aspect, the keyword candidate extracting means counts the appearance frequency of each keyword candidate,
The keyword candidate evaluation means evaluates the keyword candidates in accordance with the frequency of appearance in the electronic document, and adjusts the frequency of appearance of the keyword candidates according to the degree of similarity with other keyword candidates and the frequency thereof. Characterize.

According to a fourth aspect of the present invention, in the keyword extracting device according to the third aspect, the keyword candidate evaluation means uses the degree of overlap of characters of the character string of the keyword candidate instead of the similarity between the keyword candidates. To do.

The invention of claim 5 is the first to fourth aspects of the invention.
In the keyword extraction device, the means for displaying the extracted keywords arranged in order from the highest evaluation by the keyword candidate evaluation means is added.

[0009]

According to the present invention, the keyword candidates are extracted by the keyword candidate extraction means from the digitized document that has already been input or already existed as a file, and the keyword candidates are extracted and the appearance of each keyword candidate appears. Frequency is also counted. Next, the keyword candidate evaluation means obtains the degree of complexity according to the type and length of the character string of the word of the keyword candidate, and the appearance of the keyword candidate adjusted by the degree of similarity with other keyword candidates and its frequency. Evaluate keyword candidates for frequency. Alternatively, instead of the similarity between the keyword candidates, the degree of overlap of characters in the character string of the keyword candidates is used for evaluation.
By displaying the keyword candidates extracted in this way in order from the highest evaluation by the keyword candidate evaluation means, the characteristics of the document contents can be easily recognized.

According to the present invention, it is possible to perform a more accurate evaluation of the keyword property while avoiding having the information for the keyword in the dictionary by adding the specialty / specificity depending on the degree of compounding of the word. Moreover, since a morphological analysis system is not used, keyword extraction can be performed easily and quickly. In addition, by increasing the appearance frequencies of similar keyword candidates together, it is possible to improve the keyword property of a keyword candidate that is similar even if the appearance frequency is small by itself, and improve its evaluation.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to FIGS. FIG. 1 is a block diagram for explaining the keyword extracting device of the present invention. In FIG. 1, a computerized document 1 is a file input from the outside or a computerized document that already exists as a file. The keyword candidate extraction means 2 extracts keyword candidates from the character type list of this electronic document. The appearance frequency counting means 5 counts the appearance frequency of each of the extracted keyword candidates. The word complexity evaluation means 6 calculates the complexity according to the type and length of the character string of the keyword candidate word. The appearance frequency evaluation unit 7 obtains the appearance frequency of the keyword candidate adjusted based on the similarity with other keyword candidates and the frequency thereof, or instead of the similarity between keyword candidates, In some cases, the degree of overlap is used for the calculation. The keyword candidate evaluation means 3 is
The keyword candidates are evaluated according to the degree of complexity and the appearance frequency obtained by the word degree of complexity evaluation unit 6 and the appearance frequency evaluation unit 7. The keyword candidate display means 4 displays the keyword candidates extracted in this way in order from the highest evaluation by the keyword candidate evaluation means 3.

In this embodiment, an electronic document having the contents shown in FIG. 2 will be described as an example. FIG. 3 shows the computerized document in which the keyword candidates are extracted by the keyword candidate extraction means 2, the appearance frequencies are counted by the appearance frequency counting means 5, and the appearance frequencies are shown for each appearance frequency. The frequency of occurrence is shown in the '[]' at the beginning of each list.
It should be noted that the keyword candidates may be detected by an arrangement of characters of the same character type such as kanji, katakana, and alphanumeric characters, a morphological analysis system that performs word division and part-of-speech assignment, or may be extracted using an existing method.

(1) Invention of Claim 1 In general, a kanji compound word can be divided into a plurality of words. For example, the compound word "Export Trade Control Order" can be divided into the words "Export", "Trade", and "Control Order".
Depending on the word, the prefix (word) and the suffix (word) can be divided. The above "management order" can be divided into a word group "management" and a suffix "order". Here, the symbols representing the dividing points are defined as follows. The division point of the word by "|", the prefix (word) by ".", The division point of the suffix (word) and the word base "Export Trade Control Order" is expressed as "Export | Trade | Control. Order" . Such a division is obtained as a result of the morphological analysis system. When extracting keyword candidates by character type, prefix (word),
It can be easily divided by a word dictionary including suffixes (words). In the present invention, the word complexity is evaluated by the word complexity evaluation means 6 to the keyword candidates thus extracted. That is, the degree of complexity of the "Export Trade Control Ordinance" is 3, and the degree of complexity of "Export Control" is 2.
FIG. 4 shows each of the keyword candidates extracted according to the degree of compounding of the word, divided into a prefix (word), a suffix (word) and a word. The number in '[]' at the beginning of each list indicates the degree of word complexity.

There are various methods of adding the degree of word complexity to the evaluation of keyword property. Here, it is assumed that evaluation is performed together with the frequency of appearance in a document. For example, the product of the two is used as an index value indicating the keyword property, and FIG. The table shown in is obtained.
Here, the number in "[]" at the head of each list represents the index value of keyword property. Each number in '{}' represents the frequency of occurrence and the degree of word complexity. The appearance frequency may be set when the keyword candidates are extracted. For example, if the threshold is set to 1/2 or more of the maximum index value, the next 10 words are adopted as keywords. [6]: {3,2}: Export restrictions [4]: {2,2}: Industrial products {2,2}: Specific areas {1,4}: Export control committee for communist areas [3]: { 3,1}: Iraq {3,1}: Export {1,3}: Democratic Republic of Korea {1,3}: Seven major advanced countries {1,3}: Export trade control order {1,3}: Normal Weapons related

(2) Invention of Claim 2 The invention of Claim 2 is such that the degree of word complexity in the word degree-of-complexity evaluation means 6 is simply determined without using a morphological analysis system or word dictionary. is there. It is known that a general kanji compound word consists of a concatenation of two-letter words , sometimes with a one-letter prefix or suffix added to it (see “Kanji Kanji Memory and Japanese Sentences”). Automatic division of information ”; Nagao et al .; Information processing Vol.19 No.6, 197
8, pages 518-520). Therefore, the length of the Kanji character string of the keyword candidate and the degree of compounding of the word can be approximated as follows, for example. -------------------------------------------------- ------------- Number of Kanji characters: 1-3 4-5 6-8 9-11 12-14 ・ ・ ・ Word complexity: 1 2 3 4 5 ・ ・ ・ ---- -------------------------------------------------- --------- For katakana and alphanumeric character strings, it is assumed that there are word breaks (delimiters such as midpoints, diagonal lines and spaces) as in "Computer Graphics". The degree of compositeness shall be measured by the break. However, since such a break may not be inserted in some cases, it may be necessary to correspond to one word in units of 7 or 8 characters. In any case, the length is made to correspond to the approximate value of the degree of compounding of the word, depending on the character type of the character string. Note that the approximate value is L / L for the length L, even if it is not an integer as above.
It may be a real number such as 3. FIG. 6 shows keyword candidates according to the approximate complexity based on the above criteria. As can be seen by comparing with Figure 4, there is no difference except that the compositeness of the "Democratic Republic of Korea" has increased by one. Therefore, even when the product of the appearance frequency and the approximate degree of complexity of the word is used as the evaluation value of the keyword property, it is possible to obtain a result that hardly changes.

(3) Inventions of Claims 3 and 4 In the appearance frequency counting means 5 of the above description, the "processing machine" and "machine tool" or "export control" and "export control order" are used. Even for similar keyword candidates, the frequency of appearance is counted independently. However, the appearance of similar terms in this way reinforces each other in terms of expression,
It can be considered that the keyword nature is also increasing. Therefore, the appearance frequency evaluation unit 7 determines the degree of similarity between the two keyword candidates and increases the appearance frequency of other similar keyword candidates according to the value of the keyword candidate that appears.
Try to enhance the keyword. For example, it is assumed that the similarity is set to α (A, B) for the two keyword candidates A and B. Here, the value of α (A, B) takes a value between 0 and 1. Also, the appearance frequencies of the keyword candidates A and B are F (A) and F (B), respectively. As a method of increasing the appearance frequency, for example, the product of the appearance frequency of the opponent and the similarity is taken, and F (B) × α (A, B) for A is F (A) × α ( A, B) is increased. Since this increase is performed with all the keyword candidates other than itself, if the set of keyword candidates is K, the increase amount of A is ΣF (X) × α (A, X) (where X ΕK, X ≠ A). The appearance frequency of itself is added to this, and the changed appearance frequency (hereinafter, referred to as “pseudo appearance frequency”). α
Since (A, A) = 1, the pseudo appearance frequency is as follows. ΣF (X) × α (A, X) X ∈ K

There are various methods of determining the degree of similarity, and for example, a method based on the difference between the codes given in the classification vocabulary table can be considered. However, this method requires setting the classification vocabulary table code in the word dictionary. In the present invention, in order to calculate the similarity simply without having such special information, the similarity is used instead of the degree of overlap of character strings between character strings of words. As the degree of overlap, for example, the ratio of the partial character strings that are common to both keyword candidate character strings is used. That is, if the character string lengths of the two keyword candidates A and B are len (A) and len (B) and the common character string length is Len (A, B), the following common characters are obtained. The proportion of the row portion can be the degree of overlap. 2 × Len (A, B) / (len (A) + len
(B)) For example, the “export control” (4 character string) and the “export trade control order” (7 character string) have the same 2 character string “export”, so the degree of overlap is 2 × 2 / It becomes (4 + 7). The increase in the appearance frequency (the product with the appearance frequency of the other party) is as follows. Contribution to the “Export Control” [3] by the “Export Trade Control Order” [1]: + 1 × 4/11 = 0.36 According to the “Export Control” [3] to the “Export Trade Control Order” [1] Contribution to increase: + 3 × 4/11 = 1.09 Other keywords related to “export control” with a degree of similarity over 0 are “export”, “regulated object”, “regulated item”, and “regulated item”. Yes, the contributions to the increase in the frequency of each occurrence, including the “Export Trade Control Order”, are as follows. Contribution to “Export Control” [3] by “Export Control Committee for Communist Areas” [1]: + 1 × 6/15 = 0.40 Contribution to “Export” [3]: + 3 × 4 / 6 = 2.00 Contribution to increase by “Export Trade Control Ordinance” [1]: + 1 × 4/11 = 0.36 Contribution to increase by “Regulation target” [1]: + 1 × 4/8 = 0.50 “Regulated item” [ 1] increase contribution: +1 × 4/8 = 0.50 “Regulated products” Increase contribution by [1]: +1 × 4/7 = 0.57 The total of these increase contributions is 4.33. The original appearance frequency of 3 is added, and the pseudo appearance frequency is
It will be 7.33. FIG. 7 shows the pseudo appearance frequency of each keyword candidate as a result of applying it to the target document. Candidates such as country names that did not appear in duplicate dropped relative rank,
Candidates including "exports" are on the rise. Note that here, with respect to "export restrictions", duplicates of only one character ("import", "limit") such as "embargo" and "economic sanction" are excluded. This is a duplication of two or more characters,
This is because there is a high possibility that there is similarity, but in the case of one character, there is often not always similarity. The index value thus obtained is used as the keyword candidate evaluation means 3
Calculates the index value of the keyword property by the product of this pseudo appearance frequency and the approximate composite degree obtained in (2) (as in FIG. 8), and sets the appropriate threshold value as described in (1) above. Is set and selected from the keyword candidates and adopted as the keyword.

(4) The invention keyword candidate display means 4 of claim 5 displays the evaluation results of the keyword candidates obtained as described above in descending order of the keyword index value. (For example, it is displayed as shown in FIGS. 6 and 8.) By displaying in this way, it is possible to show the characteristics of the document contents and to accurately display the characteristics.

[0019]

As is apparent from the above description, according to the invention described in claim 1, by adding the specialization / specificity depending on the degree of complexity of words, the dictionary has information for keywords. While avoiding, it is possible to more accurately evaluate the keyword property.

According to the second aspect of the present invention, a word is extracted by a character type without using a morphological analysis system, and the degree of complexity of the word is simply determined by the character type and length of the word. Extraction is simple and fast.

According to the invention described in claim 3, by increasing the appearance frequency of similar keyword candidates together, it is possible to enhance the keyword property of the keyword candidates that are similar even if the appearance frequency is small,

According to the invention described in claim 4, the similarity between keyword candidates can be easily obtained by duplication of character strings without requiring a synonym dictionary or the like.

According to the invention described in claim 5, by displaying the keywords in the order of the keyword property, the characteristics of the document contents can be easily recognized.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a keyword extracting device according to an embodiment of the present invention.

FIG. 2 is a content of a digitized document for explaining an embodiment of the present invention.

FIG. 3 is a diagram showing keyword candidates for each appearance frequency in an electronic document according to an embodiment.

FIG. 4 is a diagram showing keyword candidates (with division marks) for each degree of complexity in the digitized document of the embodiment.

FIG. 5 is a diagram showing an example of evaluation of keyword candidates based on a product of an appearance frequency and a word compound word in the electronic document according to the embodiment.

FIG. 6 is a diagram showing keyword candidates for each approximate complexity degree in the electronic document according to the embodiment.

FIG. 7 is a diagram showing a pseudo appearance frequency according to a character string duplication degree in an electronic document according to an embodiment.

FIG. 8 is a diagram showing an example of evaluation of a keyword candidate based on a product of a pseudo appearance frequency and an approximate compound word in a digitized document according to the embodiment.

[Explanation of symbols]

1 …… Electronic document, 2 …… Keyword candidate extraction means, 3 …… Keyword extraction evaluation means, 4 …… Keyword candidate display means, 5 ... appearance frequency counting means, 6 ... Word complexity evaluation means, 7: Appearance frequency evaluation means.

Continuation of front page (56) Reference JP-A 1-217623 (JP, A) JP-A 64-28770 (JP, A) JP-A 63-244259 (JP, A) JP-A 63-254522 (JP , A) JP 60-33665 (JP, A) JP 3-116375 (JP, A) JP 3-116377 (JP, A) JP 63-201824 (JP, A) JP 3-125264 (JP, A) JP-A-3-127176 (JP, A) JP-A-3-286372 (JP, A) JP-A-5-257982 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 17/30 JISST file (JOIS)

Claims (5)

(57) [Claims] 1. A keyword extraction device having keyword candidate extraction means for extracting keyword candidates from a computerized document, and keyword candidate evaluation means for evaluating the extracted keyword candidates. Is the keyword candidate in the document
Combination of appearance frequency and words that make up the keyword candidates
A keyword extracting device characterized in that a keyword candidate is evaluated by an evaluation value obtained based on the degree . 2. The keyword candidate extraction means extracts the character types of the digitized document, and the keyword candidate evaluation means determines the degree of complexity of the words constituting the keyword candidates as the type of the character string of the keyword candidates. The keyword extraction device according to claim 1, wherein the keyword extraction device evaluates according to the length. 3. The keyword candidate extraction means counts the appearance frequency of each keyword candidate, and the keyword candidate evaluation means evaluates the keyword candidate according to the appearance frequency in the digitized document, and the keyword candidate The keyword extraction device according to claim 1 or 2, wherein the appearance frequency of the candidate is adjusted according to the degree of similarity with other keyword candidates and the frequency thereof. 4. The keyword extraction device according to claim 3, wherein the keyword candidate evaluation means uses the degree of overlap of characters in the character string of the keyword candidates instead of the similarity between the keyword candidates. 5. The keyword extracting device according to claim 1, further comprising a device for displaying the keywords extracted by the keyword extracting device in a descending order of evaluation by the keyword candidate evaluating device.