JPH07230468A - Method and device for automatically extracting keyword - Google Patents

Abstract

PURPOSE:To improve the quality or keywords and the retrieval precision of an information retrieval system. CONSTITUTION:A character string 6 is divided into character string elements (KANJI (Chinese character) character strings, KATAKANA (square form of Japanese syllabary) character strings, numeric character strings, etc.,) each of which consists of continuous characters of the same character type to generate a character string element list 7, and character strings such as numeric character strings and KATAKANA character strings which are not subjected to morpheme analysis out of these character string elements are substituted with alternative characters * or the like having corresponding grammatical information to generate a text for morpheme analysis, and this text is subjected to morpheme analysis to generate a morpheme analysis list a where at least KANJI character strings out of character string elements are divided into morphemes again. A keyword list 9 which consists or nouns or the like in character string elements and morphemes and compound words of nouns which are adjacent based on prescribed grammatical relations of nouns or the like is generated based on grammatical information in the morpheme analysis list 8.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a keyword automatic extraction device and a keyword for executing a morphological analysis of only a predetermined character type portion of a text (input sentence), for example, a kanji character string, when obtaining a keyword of the text (input sentence). It relates to an automatic extraction method.

In general, in an information retrieval system for a large number of documents, a document (input sentence) is divided into morphemes by referring to a dictionary created in advance, and nouns are formed among these morphemes and morphemes. Automatic extraction of matching keywords and the like has been carried out, and the present invention responds to the demand for improving the quality of such keywords.

The term "morpheme" used in the present specification means one word registered in a dictionary, and is not necessarily "the smallest significant unit constituting a sentence", for example, an idiomatic expression, a prefix, Suffixes, inflectional endings, compound verbs, collocations, etc. are also included, and "letter" includes numbers and symbols.

[0004]

2. Description of the Related Art In the conventional automatic keyword extraction, a morpheme analysis is performed on the entire character string constituting an input sentence, and keywords are extracted from the morphemes obtained by this morpheme analysis and their collocations.

Morphological analysis is a general method of Japanese analysis. In this analysis, a series of processes such as: dictionary lookup, adjacency check, and maximum likelihood evaluation are executed.

Here, the "dictionary lookup" refers to a morpheme that has a notation at the beginning of a character string to be processed by referring to a dictionary prepared in advance, and a morpheme that has a notation in which a succeeding character is sequentially added. This is a process that seeks all.

For example, as shown in FIG. 10 (a), "I" and "ha" for the input "I bought a vase in Tokyo."
In the state in which the two morphemes are analyzed, first, the character part having the notation of "east" at the beginning of the next processing target character string "I bought a vase in Tokyo." The character part with the notation of "Tokyo" added is subtracted from the dictionary, and the character part with the addition of the character "Tokyo de" is added from the dictionary.

The above dictionary lookup ends because the character portion having the notation "in Tokyo" cannot be retrieved from the dictionary, that is, it is not registered in the dictionary, and ends in "east" and "Tokyo".
The two character portions of are listed as morpheme candidates [see FIG. 10 (b)]. As can be seen from this list, the dictionary also registers grammatical attributes of each morpheme, left and right adjacent information, frequency, and the like.

Next, any one of these "east" and "Tokyo" morpheme candidates will be selected. At this time, the above-mentioned "adjacent check" and "maximum likelihood evaluation" are selected. Used.

The adjacency check determines whether adjacency between each of the morpheme candidates and the immediately preceding analyzed morpheme "ha" is recognized (adjacent information of the analyzed morpheme "ha" and left of the morpheme candidate). This is a process of making a determination based on (adjacent information).

That is, the adjacency of "case particle" which is the right adjacency information of "ha" and "noun" which is the left adjacency information of "east" and "noun" which is the left adjacency information of "Tokyo". Whether or not it is possible is judged by referring to the adjacency matrix [see FIG. 10 (c)].

The adjacency matrix indicates whether adjacency between the two is admitted depending on the value of the matrix element selected by the row of the right adjacency information and the column of the left adjacency information. A value of "1" is set if there is a possibility, and a value of "0" is set if there is no possibility of being adjacent.

According to this adjacency check, both “East” and “Tokyo” are the analyzed morphemes “ha” immediately before that.
The following maximum likelihood evaluation is required to determine which of the two is to be selected.

That is, an evaluation value is calculated for each morpheme candidate based on its word length and frequency, and the one having the highest evaluation value is selected as the morpheme following "ha". The evaluation value increases as the word length increases and the frequency increases. In this case, the evaluation value for "Tokyo" is "east".
Higher than that.

According to this maximum likelihood evaluation, "Tokyo" is selected as the next morpheme, and then the morpheme analysis of the next character string "I bought a vase." Is sequentially executed, and finally, FIG.
A morpheme list as shown in (e) is obtained, and morphemes having a predetermined grammatical attribute (for example, a noun) or their collocations are extracted as keywords of the input sentence.

[0016]

As described above, in the conventional keyword extraction method, morphological analysis is performed on the entire character string of the input sentence, that is, the character string is divided by the morphemes registered in the dictionary. However, there is a problem that a part of a katakana character string or a numeric character string, a prefix and a suffix in a kanji character string are extracted as inappropriate keywords.

Examples of this inappropriate keyword extraction include: -extracting the keyword "squirrel" from the katakana character string of "restructuring" -extracting the keyword "re" from the kanji character string of "financial re-easing" This includes, for example, extracting the keyword “multiplication table” from the numeric character string “1994”.

Therefore, in the present invention, the character string of the input sentence is first divided into consecutive character string elements (Kanji character string, Katakana character string, number string, etc.) of the same character type, and at least the Kanji character string among them is divided. Is morphologically analyzed and subdivided into morphemes.For example, without performing morphological analysis for other Katakana character strings or number strings, after that, a keyword including a compound word is specified from these character string elements and morphemes. The purpose is to improve the quality of keywords and the search accuracy of the information search system by selecting the rules.

[0019]

FIG. 1 is a diagram for explaining the principle of the present invention. In FIG. 1, 1 is a processor including a CPU and a memory, 2 is a main control unit that performs overall control related to keyword extraction, and 3 is a main control unit that divides an input character string into continuous character string elements of the same character type. Furthermore, for character string elements that are not subjected to morpheme analysis, for example, for numeric character strings, katakana character strings, alphabetic character strings, etc., the character type classification unit 4 that replaces with a symbol accompanied by the grammatical information is used. A morpheme analysis unit 5 that subdivides a specific character type including a string into morphemes is a keyword extraction unit 6 that selects a keyword of an input sentence from character string elements and words or their collocations according to a predetermined rule. The character string (text) 7 is a processing result of the character type classification unit 3, and the character string element list 8 is a processing of the morphological analysis unit 4. Morphological analysis list 9 is a fruit shows a keyword list as the processing result of the keyword extracting unit 5, respectively.

FIG. 1 shows a case in which only the Kanji character string and the Hiragana character string in the text are subjected to the morphological analysis. The processing procedure of the keyword extraction at this time is as follows ( See FIG. 2).

The character strings in the text (as a business policy in 1994 ... costdown by reducing personnel costs) are based on the code of each character: Katakana, numbers, letters / symbols, Kanji The character string element list 7 is created by dividing the character string elements into consecutive character string elements for each character type of hiragana and stored in the work area. Katakana character strings and alphabetic / symbol character strings are the alternative character "*" for general nouns, and numeric character strings are the alternative character "*" for numeric characters.
Replace each with, and leave the kanji character string and hiragana character string as they are. For the character string element replaced with this substitute character, the position in the original text and the character string element length are stored in the replacement table. Morphological analysis is performed on the new text after replacing the katakana character string, the alphabet / symbol character string, and the numeric character string with alternative characters to create a morpheme analysis list 8, and this is stored in the work area. The entire morphological analysis list 8 is shown in FIG. From the morphological analysis list 8, notations having nouns (general nouns, proper nouns, sahen nouns, etc.) as grammatical information are extracted to create a keyword list 9, which is stored in the work area. Note that the sahenun is, for example, "develop" in "develop"
And so on. In the notation of this morphological analysis list 8, nouns + nouns, noun prefixes + nouns, nouns + noun suffixes, numerical + noun suffixes, numerical nouns, etc. 9 is created and stored in the work area. The collocation corresponding to "numerical + noun" in this rule is, for example, "100" + "data", "100 data".

[0022]

As described above, according to the present invention, the text to be extracted as a keyword is first divided into character string elements for each character type, and then morphological analysis of a specific character string element including a Kanji character string is executed. By selecting a keyword from a character string element, a morpheme and a collocation thereof according to a predetermined rule, the quality of the keyword and the search accuracy of the information search system are improved.

The morphological analysis used in the present invention requires that the Kanji character string or the like is simply divided into morphemes registered in the dictionary. Whether or not to use the method of adjacency check or maximum likelihood evaluation is arbitrary. It should be noted that the morpheme analysis list 8 may also include adjacency information and frequency of each morpheme (registered in the dictionary).

Further, as shown in FIG. 1, whether or not to perform morpheme analysis on a hiragana character string and whether or not to perform morpheme analysis on any character type other than kanji is arbitrary. As shown in FIG. When performing a morphological analysis on a Hiragana character string, the original keyword can be selected from the text in which the noun originally written in Kanji is written in Hiragana.

When the katakana character string and the alphabetic character string have the ending "suru", "sahen noun" is selected as the grammatical information when the character type classifying unit 3 replaces these with the alternative character "*".

Further, in the character type classification unit 3, without replacing the Katakana character string, the alphabetic character string or the numeric character string with the substitute character "*" accompanied by the grammatical information, a code indicating each character string is simply set in that portion. Alternatively, the morphological analysis unit 4 may determine that the codes of the katakana character string and the alphabetic character string are nouns, and the codes of the numeric character strings are numeric.

Further, as a method of extracting a character string element that matches the above-mentioned collocation extraction rule, for example, "business policy" itself, select the character string element "business policy" from the work area as it is. Whichever is selected as a collocation of “business” and “policy”. In the former case, the character type classification unit 3 may add a "noun" to the kanji character string as its grammatical information.

The collocation rule is so-called recursive. For example, when there are morphemes "Heisei", "Six", and "FY", first, the number of "Six" and the suffix of "FY" are added. The "6th year" collocation (noun) is created, followed by the "Heisei" noun and the conjunctive word "1994". The keyword of “organizational reorganization” is extracted by the same process.

[0029]

Embodiments of the present invention will be described with reference to FIGS. FIG. 4 is an explanatory diagram showing how the text 13 is extracted from one document 12 stored in the document file 11. In this extraction process, the beginning of the document 12 and the spaces in the sentence are padded, and before and after the line feed code. I am aware of the connection of sentences in.

As a result, an unnecessary blank portion of the text 13 in executing the keyword extraction process is eliminated, so that the keyword extraction process such as character type classification and morphological analysis for the text 13 can be efficiently performed.

FIG. 5 is an explanatory diagram showing how the morphological analysis text 14 is edited from the text 13. In this editing process, the text 13 is converted into numeric character strings and kanji character strings by referring to the editing rule file 15. , Character / symbol character string, hiragana character string, and katakana character string.

That is, one sentence of "development plan for 1994 year" is "1994 / year / no / development plan"-"FAILS-SV has the following functions" 1
The sentence is divided into "FAIRS-SV / ha / below / of / function / with / with / without", and "1994" and "FAIR"
Each character string element of "S-SV" is treated as one number and alphabet / symbol that are not subjected to morphological analysis in the subsequent processing.

FIG. 6 is an explanatory diagram showing how the morphological analysis list 16 is created from the morphological analysis text 14, and in this creating process, the Kanji of the morphological analysis list 16 is referred to by referring to the morphological analysis dictionary file 17. The hiragana part is divided into morphemes. The illustrated morphological analysis list is for one sentence of "1994 / year / no / development plan".

FIG. 7 is an explanatory diagram showing how the keyword list 18 is created from the morphological analysis list 16. In this creating process, the grammar information is referred to as a noun (general noun, sa “Development” and “plan” which are nouns, proper nouns, etc.) and “1992 fiscal year” and “development plan” that correspond to the above-mentioned compound word creation rule are extracted. The illustrated keyword list is for one sentence of "1994 / year / no / development plan".

FIG. 8 is an explanatory diagram showing a procedure for extracting a document from a document file and extracting its keyword.
The procedure is as follows. (1) Documents subject to keyword extraction processing from the document file 11
Read all 12 and go to the next step. (2) Extract the text 13 of the first document and proceed to the next step (see FIG. 4). (3) Edit the text 13 into the text 14 for morphological analysis,
Proceed to the next step (see FIG. 5). (4) The first sentence of the morphological analysis text 14 is taken out, and the process proceeds to the next step. (5) A morphological analysis list 16 is created by dividing this one sentence into morphemes, and the process proceeds to the next step (see FIG. 6). (6) From the morphological analysis list 16, the grammatical information having a noun is extracted, and the process proceeds to the next step. (7) From this morphological analysis list 16, the part of the collocation that satisfies the collocation extraction rule (noun + noun, number + numerical suffix, etc.) is extracted, and the process proceeds to the next step. (8) Store the morphemes and collocations found in steps (6) and (7) in a file and proceed to the next step. (9) For example, it is determined whether or not the keyword extraction processing has been completed for all the sentences of the morphological analysis text 14 based on the presence or absence of a sentence end code.
Proceed to (11), and if "NO", proceed to the next step. (10) Take out the next sentence from the morphological analysis text 14,
Return to step (5). (11) For example, by counting the number of documents for which the keyword extraction processing has been executed, it is determined whether or not the keyword extraction processing has been completed for all the documents 12 read in step (1), and if YES, the step Go to (13)
If "NO", proceed to the next step. (12) Extract the text 13 of the next document and return to step (3). (13) Output the keywords stored in the file (see FIG. 7).

[0036]

As described above, according to the present invention, in the text,
Character string elements (numerical character strings, katakana character strings, alphabetic character strings, etc.) that have a high degree of extraction of inappropriate keywords by morphological analysis are left as they are without morphological analysis, and the character string elements and morpheme It is possible to improve the quality of keywords and the search accuracy of the information retrieval system by extracting specific ones (nouns and collocations that match certain rules) as keywords from morphemes such as Kanji character strings to be analyzed. .

[Brief description of drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is an explanatory diagram showing an example of a procedure of keyword extraction processing according to the present invention.

FIG. 3 is an explanatory diagram showing the entire morphological analysis list of FIG. 1 according to the present invention.

FIG. 4 is an explanatory diagram showing how text is extracted from a document according to the present invention.

FIG. 5 is an explanatory diagram showing a state of editing a morphological analysis text from text according to the present invention.

FIG. 6 is an explanatory diagram showing a state in which a morphological analysis list is created from a morphological analysis text according to the present invention.

FIG. 7 is an explanatory diagram showing how a keyword list is created from a morphological analysis list according to the present invention.

FIG. 8 is an explanatory diagram showing a procedure of extracting a document from a document file and extracting a keyword thereof according to the present invention (No. 1).
Is.

FIG. 9 is an explanatory view showing a procedure of extracting a document from a document file and extracting a keyword thereof according to the present invention (No. 2).
Is.

FIG. 10 is an explanatory diagram showing a general morphological analysis.

[Explanation of symbols]

 In FIG. 1, 1 ... Processor 2 ... Main control unit 3 ... Character type classification unit 4 ... Morphological analysis unit 5 ... Keyword extraction unit 6 ... Keyword extraction target character string (text) 7: Character string element list 8: Morphological analysis list 9: Keyword list

Claims (4)

[Claims] 1. A keyword automatic extraction device configured to obtain a keyword of a character string by executing a morphological analysis only on a specific character type in the character string, wherein A character type division unit that divides into character string elements consisting of character types, a morpheme analysis unit that executes the morpheme analysis to divide at least a Kanji character string in the character string elements into morphemes, the character string element and the morpheme, respectively. Based on the grammatical information of the above, a keyword extracting unit for selecting a predetermined one such as a noun and these collocations that were originally adjacent due to a predetermined grammatical relationship such as nouns as the keyword. An automatic keyword extraction device characterized in that 2. The character type classification unit replaces a character string element, such as a numeric character string or a Katakana character string, which is not subjected to the morphological analysis, with an alternative character having corresponding grammatical information. The automatic keyword extraction device according to claim 1, wherein a morphological analysis text is created, and the morphological analysis unit is configured to execute the morphological analysis on the morphological analysis text. 3. A keyword automatic extraction method for obtaining a keyword of a character string by executing a morphological analysis only for a specific character type in the character string, wherein the character string is continuously Divide into character string elements having the same character type, then execute the morphological analysis to divide at least the Chinese character string in the character string elements into morphemes, and then divide the character string elements and the morphemes, respectively. Based on the grammatical information of the above, a predetermined one such as a noun or the like, and those compound words which are originally adjacent to each other due to a predetermined grammatical relationship such as nouns are selected as the keyword. Automatic keyword extraction method. 4. A morpheme in which a portion of the character string element that is not subjected to the morphological analysis, such as a numeric character string or a katakana character string, is replaced with a substitute character having corresponding grammatical information. 4. The automatic keyword extraction method according to claim 3, wherein an analysis text is created and the morphological analysis is performed on the morphological analysis text.