JP2812511B2 - Keyword extraction device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a keyword extracting device.

2. Description of the Related Art Conventionally, as a technique for automatically extracting a keyword from a document, there is a method focusing on a language phenomenon. This is considered to be able to reflect the contents of the document as compared with a statistical method based on frequency or the like. As one of the methods focusing on linguistic phenomena, there is a system for determining whether a keyword is a keyword based on the arrangement of parts of speech in a sentence. This is the US Department of Defense DDC
"Machine assisted search system MAI", "Trends in Automatic Indexing Research" (Information Processing, Sep.1984.Vol.25No.9)
Has been introduced. In this system, a word is recognized to detect a compound word, and the part of speech of the compound word is determined as a keyword if there is a corresponding pattern using a format dictionary. That is, the frequency of the part-of-speech arrangement is measured in advance, and a pattern with a high frequency is used as a keyword.

However, according to such a conventional method, in the case of a compound word having a large number of connections, it is necessary to increase the part-of-speech string pattern to be registered in the dictionary. Will not be keywords. For example, in the case of a compound word “automatic index generation system”, a part of speech of noun, adverb, sa-variant noun, and noun is determined, but if the word is not registered in the dictionary in this order, the keyword is not determined.

In addition, even if the words are structurally similar, they become different patterns, so that the dictionary becomes complicated, and the same evaluation value cannot be given even though the words have the same structure. For example, in the "index automatic generation system" and the "index generation system", the latter is a part-of-speech list of nouns, sa-variable nouns, and nouns, whereas the former is only a modifier for the sa-various noun in the same part-of-speech list. The difference is
Since the part-of-speech sequence pattern is different, the determination as to whether the keyword is a keyword also differs.

Therefore, it cannot necessarily be said that the keyword extraction method has high accuracy.

Means for Solving the Problems Morphological analysis means for recognizing a word unit and part of speech for a Japanese document, compound word extracting means for extracting compound words from the Japanese document as keyword candidates, and an arrangement of part of speech pairs ,
A part-of-speech sequence evaluation value table holding rewriting parts of speech representing this sequence and an evaluation value for the sequence, and whether or not the compound word is a keyword by analyzing the evaluation value of the part-of-speech sequence structure in the extracted compound word And keyword determining means for determining the

Action Based on the evaluation of the part-of-speech sequence structure inside the compound word with reference to the part-of-speech sequence evaluation value table, it is determined whether or not the keyword candidate compound is a keyword. Since the structural analysis based on the combination of pairs is sufficient, keyword extraction with less omission can be performed without registering the part-of-speech sequence pattern of each compound word. In this case, even in the case of compound words that are not the same but are structurally similar, the internal part word sequence structure is analyzed according to the same rule, so that the same evaluation is obtained and the same keyword determination can be performed.

Embodiment An embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a keyword extraction system according to the present embodiment. A morphological analysis unit 2 for certifying a word unit and a part of speech for a Japanese document using a word dictionary 1, and using the certifying result. A compound word extracting means 3 for extracting compound words from Japanese documents as keyword candidates and a part-of-speech string evaluation value table 4 holding a list of part-of-speech and an evaluation value for the part-sequence are referred to. A keyword determining means for calculating an evaluation value according to the part-of-speech string structure and determining whether or not the compound word is a keyword;

Here, the word dictionary 1 used for the morphological analysis is the second dictionary.
As illustrated in the figure, a pair of a notation of each word and a part of speech is held.

FIG. 3 shows an example of the part-of-speech sequence evaluation value table 4. The keyword determining means 5 analyzes the internal structure of the compound word with reference to the part-of-speech sequence evaluation value table 4 and gives an evaluation value. For this reason, the part-of-speech sequence evaluation value table 4 holds a part-of-speech arrangement (before and after), a part-of-speech (re-writable) representative of the part-of-speech arrangement based on the relation between the parts of speech, and an evaluation value for the part-of-speech arrangement. It is. The part-of-speech sequence rewriting rules are arranged in the order in which the relationship between the parts of speech (modification-modification) is clear.

In such a configuration, the determination process for keyword extraction is performed according to the flowchart shown in FIG. First, the part-of-speech sequence is rewritten and the evaluation value is calculated until the part-of-speech sequence becomes 1 for the extracted compound candidate keyword. That is, when the part-of-speech sequence is greater than 1, the part-of-speech sequence is rewritten with reference to the part-of-speech sequence evaluation value table 4 and an evaluation value is assigned. The rewriting of the part-of-speech sequence and the addition of the evaluation value are repeated until the part-of-speech sequence becomes 1. When the part-of-speech sequence becomes 1, an evaluation value is added by referring to the part-of-speech sequence evaluation value table 4. If the evaluation value is larger than a preset threshold, the compound word is determined as a keyword. However, if the evaluation value is smaller than the threshold, it is determined that the keyword is not a keyword.

 Specific examples of such processing will be described.

Specific Example 1 "Automatic Index Generation System" First, words are divided into "index | automatic | generation | system" by morphological analysis, and parts of speech are recognized as nouns, modified nouns, sa-variable nouns, and nouns. This is because the part-of-speech sequence is greater than 1,
Referring to the part-of-speech sequence evaluation value table 4, the corresponding part-of-speech sequence is rewritten and an evaluation value is given. In this case, the sequence of the modified noun / sa-variable noun is rewritten to the sa-variant noun, and its evaluation value is set to 1. Therefore, it becomes a part-of-speech sequence of noun, sa-variant noun, and noun,
In the part-of-speech sequence, the arrangement of nouns / sa-variable nouns is rewritten to sa-variable nouns, and the evaluation value is added with 1 to become 2. Then, the part-of-speech sequence of the sa noun / noun is rewritten into a noun, and its evaluation value 2 is added to make a total of four. here,
Since the length of the part-of-speech sequence is 1 for only nouns, the evaluation value 2 of the noun is added, and the final evaluation value is 6. Assuming that a value obtained by dividing the evaluation value by the number of words is 1 or more, in this case, 6/4 = 1.5, and a compound word “automatic index generation system” is determined as a keyword.

Example 2 “System that can be mass-transferred” Words are divided into “mass | migration | possible | possible | system |” by morphological analysis, and the compound part “mass-migration possible” is recognized as a part-of-speech as a qualified noun / sa-variable noun / adjective verb. . Since the part-of-speech sequence is greater than 1, the part-of-speech sequence evaluation value table 4 is referred to, the corresponding part-of-speech sequence is rewritten, and an evaluation value is assigned. First, the sequence of the modified noun / a-sa noun is rewritten to the sa-noun, and its evaluation value is set to one. Therefore, the part-of-speech sequence of sa-variant noun / adjective verb is obtained. This part-of-speech arrangement is rewritten to an adjective verb, and the evaluation value is added with 0 to become 1. Here, since the length of the part of speech example is 1, the evaluation value 0 of the adjective verb is added,
The final evaluation value is 1. Assuming that the threshold value is equal to or greater than 1 obtained by dividing the evaluation value by the number of words, in this case, 1/3
= 0.33, and it is determined that the compound word “migration is possible” is not a keyword.

Specific Example 3 “Index generation system” Words are divided into “index | generation | system” by morphological analysis, and nouns / sa-variable nouns / nouns are recognized as parts of speech. Since the part-of-speech sequence is larger than 1, the part-of-speech sequence evaluation value table 4 is referred to, the corresponding part-of-speech sequence is rewritten, and an evaluation value is assigned. Here, in the part-of-speech sequence consisting of noun / sa-variable noun / noun, the sequence of the noun / sa-variant noun is rewritten into a sa-variable noun, and an evaluation value of 1 is assigned. As a result, the part-of-speech sequence of sa-variant noun and noun becomes
This is rewritten into a noun, the evaluation value 2 is added, and the evaluation value becomes 3. At this time, since the part-of-speech sequence becomes 1, the evaluation value 1 of the noun is added, and the evaluation value becomes 4. Assuming that the value obtained by dividing the threshold by the evaluation value by the number of words is 1 or more, 4/3 = 1.3
It becomes 3. As a result, the compound word “index generation system” structurally similar to the “index automatic generation system” of the first example is evaluated with a small difference in the presence or absence of the modifier “automatic”, and an equivalent evaluation value is obtained. Is determined as a keyword.

Advantageous Effects of the Invention As described above, the present invention provides a morphological analysis unit that recognizes a word unit and a part of speech for a Japanese document, a compound word extraction unit that extracts a compound word, and an arrangement of a part of speech part. A part-of-speech string evaluation value table holding representative part-of-speech rewriting parts and evaluation values for their arrangement, and an evaluation value analysis of the part-of-speech string structure in the extracted compound word determine whether the compound word is a keyword. A keyword determining means,
Based on an evaluation value analysis of the part-of-speech sequence structure inside the compound word with reference to the part-of-speech sequence evaluation value table, it is determined whether or not the compound word of the keyword candidate is a keyword. Structural analysis based on the combination of part-of-speech pairs is sufficient, and keyword extraction with less omission can be performed without registering the part-of-speech sequence pattern of each compound word. In the case of compound words that are similar to each other, their internal part-segment string structure is analyzed by the same rule, so that the same evaluation can be obtained and the same keyword judgment result can be obtained. It is not necessary to keep it.

[Brief description of the drawings]

The drawings show an embodiment of the present invention, and FIG. 1 is a block diagram,
FIG. 2 is an explanatory diagram showing the contents of the word dictionary, FIG. 3 is an explanatory diagram showing the contents of the part-of-speech sequence evaluation value table, and FIG. 4 is a flowchart. 2 ... morphological analysis means, 3 ... compound word extraction means, 4 ...
Part-of-speech sequence evaluation value table, 5 ... keyword determination means

Continuation of the front page (58) Field surveyed (Int.Cl. ⁶ , DB name) G06F 17/30 JICST scientific and technical literature file

Claims (1)

(57) [Claims] 1. A morphological analysis means for recognizing a word unit and a part of speech with respect to a Japanese document, a compound word extracting means for extracting a compound word from the Japanese document as a keyword candidate, A part-of-speech sequence evaluation value table holding rewriting parts-of-speech representing the list and evaluation values for the sequence, and an evaluation value analysis of the part-of-speech sequence structure inside the extracted compound word determines whether the compound word is a keyword. A keyword extracting device comprising a keyword determining means for determining.