JPH06301722A - Morpheme analyzing device and keyword extracting device - Google Patents

Abstract

PURPOSE:To provide the morpheme analyzing device which can execute efficiently a word processing with respect to a character-string in which a KATAKANA (square form of Japanese syllabary) and the Roman alphabet are continued, and the keyword extracting device which can extract efficiently a keyword of a word and a composite word. CONSTITUTION:This device is provided with a morpheme analyzing means 2 for extracting a character-string of a KATAKANA or the Roman alphabet which is continued, in a Japanese language document inputted by a document input means 1, as a noun equivalent word, and extracting a noun word by using a dictionary 5 as for other kind of character, a composite word generating means 3 for generating all composite words which can generate those words as constitution words from the continued noun words or noun equivalent words, and a keyword evaluating means 4 for calculating an evaluation value of keyword likelihood, based on appearance frequency of these words and composite words and general occurrence frequency.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analysis apparatus and a keyword extraction apparatus which require the creation of a dictionary used in a word processor, a Japanese document analysis apparatus, etc., and the extraction processing of a character string having a meaning such as a keyword. It is a thing.

[0002]

2. Description of the Related Art In a Japanese analysis processing system including a keyword extraction device, morpheme analysis is performed by using a word registered in a dictionary in an input sentence as a clue to extract a document summary or a keyword word. ing. However, if an unknown word that is not registered in the dictionary exists in the sentence, it is difficult to perform a correct morphological analysis including the unknown word portion and a compound word.

Conventionally, there is a document analysis method disclosed in Japanese Patent Application Laid-Open No. 62-90760 regarding the processing of unknown words in katakana among the unknown words. This is a process of treating a katakana notation not registered in the dictionary as an independent word when the katakana notation exists in the sentence.

FIG. 6 shows a block diagram of a document analysis device for explaining the processing of the above-mentioned katakana unknown word. In FIG. 6, reference numeral 65 is a dictionary search unit that searches a word in a sentence using a dictionary, and 61 is a katakana that is processed as one word when a term in katakana notation in the sentence is not registered as a word in the dictionary. An unknown word processing unit, 62 is a DP matching unit that performs DP matching, 63 is a post-processing unit that integrates the results processed by the dictionary searching unit 65, the DP matching unit 62, and the katakana unknown word processing unit 61, and 64 is an input sentence. , A document analysis management unit that integrates pre-processed and post-processed data divided by punctuation marks or symbols.

FIG. 7 is a flow chart showing the operation flow of the above document analysis apparatus. The document analysis method for Katakana characters will be described below with reference to this flowchart. (A) When the character string of the input sentence is input to the document analysis management unit 64, it is divided into analysis units and sent (step S
1). Here, the analysis unit is a character string delimited by punctuation marks or symbols. (B) With respect to the above analysis unit, the dictionary search unit 65 performs a dictionary search, and if there is a match, it is sent to the DP matching unit 62, and if there is no match, it is sent to the katakana unknown word processing unit 61. It is sent (step S2). (C) If they match as a result of the dictionary search, the DP matching unit 62 determines the optimum word combination that matches the extracted word with the analysis unit (step S3), and if it is determined to be inappropriate, the dictionary search unit 65. Return it to the post-processing unit 63 if appropriate.
Send to. (D) The dictionary search unit 65 sends the analysis unit returned from the DP collation unit 62 to the katakana unknown word processing unit 61. (E) The katakana unknown word processing unit 61 processes the katakana part of the analysis unit sent as a result of a mismatch in the dictionary search or a non-matchable result as one word (step S4),
It is sent to the post-processing unit 63. (F) The post-processing unit 63 integrates data such as words sent from the DP collation unit 62 and the Katakana unknown word processing unit 61, and returns the data to the sentence analysis management unit 64 (step S6).

By the above procedure, the document analysis regarding the word in katakana notation is sequentially advanced.

A technical term extraction system disclosed in Japanese Patent Laid-Open No. 3-116374, which particularly creates a keyword of a compound word from frequency information, will be described below. This is to extract all compound words from a sentence, extract a specific constituent word from a compound word as a technical term characteristic word base, and include that word base from the appearance frequency and the number of word combinations of the compound words of the compound word. All compound words are used as keywords.

FIG. 8 is a block diagram showing the configuration of the conventional technical term extraction system described above. The conventional technical term extraction system uses a morphological analysis unit 8 that receives an input document 81 as input.
2. A compound word extraction unit 83 that extracts all compound words in the input document 81, a technical term characteristic word base extraction unit 84 that extracts a specific constituent word from the extracted compound words, and the extracted specific structure. It is configured by a technical term extraction unit 85 that extracts a keyword based on a word, and processing by them is sequentially performed.

First, the morpheme analysis unit 82 performs a morpheme analysis on the input document 81 in an arbitrary field to identify a word unit and a part of speech. The compound word extraction unit 83 performs compound word extraction on the result of morphological analysis to extract a sequence of nouns, adjective verbs and affixes as compound words, and extracts all compound words in the input document 81. The technical term feature word base extraction unit 84 judges whether or not the subject base, which is a constituent word, of the extracted compound word is a base word that constitutes a technical term, using an evaluation value, and determines a base word that constitutes a technical term. A technical term feature word base extraction process for extracting only After that, the technical term extraction unit 85 performs a process of extracting, from the extracted compound words, only those having a word determined to be a technical term characteristic word base as a technical term.

The outline of the processing by the technical term characteristic word base extraction unit 84 will be described with reference to FIG. With respect to the compound words in the compound word list 86 extracted by the compound word extracting unit 83 described above, first, the frequency of the subject base located at the end of the word is calculated (step S11). Next, the average number of combinations of each subject base in the input document 81 is calculated (step S1).
2). This is to calculate the number of modifiers that precedes a certain word base when it is used as a base. Next, the evaluation value is calculated from the frequency and the average number of bonds of the subject group thus obtained, and if it is equal to or more than a certain threshold value, the word is determined as a technical term feature word group (step S14).

[0011]

However, the method of collating the katakana character string with the dictionary as in the former case of the conventional example has the following problems. The English katakana notation and Japanese katakana character strings in Japanese documents often have slightly different notations even though they have the same etymology.
Therefore, there are many katakana words and English letters that are regarded as unregistered words in the dictionary. For example, “Fuzz” that represents ambiguity
Although the English word "y" is written in katakana as "fuzzy", "fuzzy", "fuzzy", etc., it is rare that all these expressions are registered in the dictionary. Therefore, it is wasteful to register all such subtle differences in notation in the dictionary and to collate a character string consisting of Katakana or alphabetic characters in the dictionary.

Further, in the latter keyword extraction method of the conventional example, the extracted words of the technical term characteristic word base are
There is a problem that it is not necessarily a word that truly represents the characteristics of the field. This is because there are a large number of words in fields that are completely different from the commonly used words or target fields in the constituent words of the compound words that are technical terms. Those words alone have little meaning as keywords.

Further, since all compound words including a specific common constituent word as the subject base are used as keywords, if there is an error in the selection of the constituent word as the subject base, it will consist of many unnecessary words. There is a problem that the subject base cannot be created when an inappropriate keyword is selected, when there is no compound word in the sentence, or when there is no common compound word.

Further, humans often select a compound word containing many unknown words such as a new word or a coined word as a keyword or a technical term in a specific field, but in the conventional compound word creating means, even one compound word is included in the compound word. If an unknown word is included, it is difficult to identify it as one compound word.

In consideration of such problems in the conventional document processing, the present invention provides a morphological analyzer capable of efficiently word-processing a character string in which katakana or English letters are continuous, and
An object of the present invention is to provide a keyword extraction device that can efficiently extract keywords of words and compound words.

[0016]

The present invention according to claim 1 uses a dictionary in which at least grammatical information necessary for analysis is stored and at least a noun word and a noun-equivalent word from an input Japanese sentence. In the morpheme analysis device provided with the morpheme analysis means for extracting, the morpheme analysis means is a morpheme analysis device for extracting a character string consisting of continuous katakana or English characters in the input Japanese sentence as a noun equivalent word.

The present invention according to claim 2 is a dictionary storing at least grammatical information necessary for analysis, and a morphological analysis means for extracting at least a noun word and a noun-equivalent word from an input Japanese sentence using the dictionary. , Compound word candidate creation which enumerates compound words formed by combining a plurality of respective words or equivalent words forming a compound word in which noun words or noun-equivalent words extracted by the morphological analysis means are combined as keyword candidates Part and the compound word keyword candidates listed by the compound word candidate creation part, considering the mutual overlapping frequency from the cumulative frequency in the input Japanese sentence,
A keyword extracting device comprising a compound word evaluating means for determining an evaluation value of each compound word keyword candidate and a keyword determining means for determining a keyword based on the compound word evaluating means.

[0018]

According to the present invention, the dictionary stores at least grammatical information necessary for analysis, and the morpheme analysis means extracts at least a noun word from the input Japanese sentence using the dictionary,
Also, a character string consisting of continuous katakana or English characters in the input Japanese sentence is extracted as a noun equivalent word.

Further, in the present invention, the dictionary stores at least grammatical information necessary for analysis, and the morphological analysis means extracts at least a noun word and a noun-equivalent word from the input Japanese sentence by using the dictionary, The compound word keyword candidate creation unit enumerates, as keyword candidates, compound words formed by combining a plurality of words or equivalent words forming a compound word in which the extracted noun words or noun-equivalent words are continuous,
The compound word evaluation means determines the evaluation value of each compound word keyword candidate by considering the mutual overlapping frequency from the cumulative frequency of the listed compound word keyword candidates, and the keyword determination means,
The keyword is determined based on it.

[0020]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings showing its embodiments.

FIG. 1 is a block diagram showing a keyword extracting device according to an embodiment of the present invention. That is, the keyword extraction device is provided with a morpheme analysis device as another embodiment according to the present invention as the morpheme analysis means 2, and the morpheme analysis means 2 extracts words from the character string input from the document input means 1. It has a function of extracting, recognizing the part of speech, and extracting a noun. Further, the morpheme analyzing means 2 includes a compound word creating means 3 for creating a compound word by combining the extracted words of the noun, a keyword evaluating means 4 for evaluating the word and the compound word extracted from the document together as a keyword, and A dictionary 5 used for word extraction, part-of-speech recognition, etc. is connected. This dictionary 5 is, for example, as shown in FIG.
In addition to word information such as hiragana / kanji word strings and part-of-speech information, a storage unit 51 that stores the occurrence frequency of a noun word in a general sentence (hereinafter, referred to as general occurrence frequency), the occurrence frequency in a specific field, It is configured by a storage unit 52 for storing and a selection unit 53 for selecting the storage units 51, 52 according to an input document.

The compound word creating means 3 constitutes a compound word keyword candidate creating part, and the keyword evaluating means 4 constitutes a compound word evaluating means and a keyword determining means.

Here, the outline of the processing by the morphological analysis means 2 will be described with reference to FIG. In FIG. 2, the morpheme analysis unit 2 is configured to sequentially perform the processes of the unknown word extraction unit 21 and the noun extraction unit 22. Regarding the character string input by the character input means 1,
The unknown word extraction unit 21 determines the character type, extracts a continuous katakana or English character string as a noun-equivalent word, and registers it. Further, the character string of kanji or hiragana is sent to the noun extraction unit 22 as it is, and other punctuation marks and symbols are excluded.

Next, the noun extraction unit 22 identifies the word and the part of speech by inquiring the sent character strings of kanji and hiragana with the character strings registered in the dictionary 5. At this time, only the word whose part of speech is determined to be a noun is registered at the same time as its general occurrence frequency. The dictionary 5 stores a character string of hiragana or kanji words, its part-of-speech information, and a general occurrence frequency of the word in the storage unit 51 (see FIG. 5). "General occurrence frequency" here means
The probability that the phrase appears in a typical document.

The noun word and the noun-equivalent word registered as described above are counted as the noun word keyword candidates by counting the appearance frequency in the input document. The continuous nouns and noun-equivalent words are sent to the compound word creating means 3.

By separately processing character strings of a specific character type in this way, not only are character strings such as katakana and English characters that are likely to be unknown words used as keyword candidates, but the probability that they are registered in the dictionary is low. By avoiding the inquiry to the dictionary 5 of the character string consisting of the character types, it becomes possible to efficiently extract the words.

Next, the outline of the processing by the compound word creating means 3 will be described with reference to FIG. As shown in FIG. 3, the compound word creating means 3 is configured to sequentially perform the processes of the compound word candidate creating unit 31 and the compound word evaluating unit 32. The compound word candidate creation unit 31 enumerates all compound word candidates that can be created using the continuous noun words extracted by the morpheme analysis means 2 as constituent words, and counts the co-occurrence frequency of each compound word candidate. . Here, the co-occurrence frequency is the frequency with which a certain word appears side by side with another specific word, that is, the appearance frequency of a compound word. For example, the nouns "A", "B", and "C" are converted into a sentence "~ ABC ~".
"AB ~ AB ~", the compound word candidate character string created and its co-occurrence frequency are "AB
“C” is 1, “AB” is 3, and “BC” is 1. By the processing in the compound word candidate creating unit 31, it is possible to create and evaluate a compound word composed of constituent words commonly used among compound word candidates. Here, even when the noun "A" in the dictionary is a compound word composed of the words "a1" and "a2" also in the dictionary, the compound word candidate may be created as described above, but the constituent word is " a1 ”“ a2 ”“ B ”
It is also possible to create compound word candidates as four “C”.

Next, when the compound word evaluation unit 32 compares the compound word candidates created by the compound word candidate creating unit 31, for example, a certain compound word X is a part of the constituent words of another compound word Y. , The value obtained by subtracting the co-occurrence frequency of the compound word Y from the co-occurrence frequency of the compound word X is a threshold value (≧
If 0 or less, a process of deleting the compound word candidate X from the compound word candidate is performed. For example, the above sentence example "~ ABC ~ A
In the case of “B-AB-”, the compound word candidate creation unit 31 displays “AB
"C", "AB", and "BC" are created, but since "AB" and "BC" are included in "ABC", the co-occurrence frequency 3 of each is 1 to the co-occurrence frequency 1 of "ABC". The subtracted values are 2 and 0, respectively. If the threshold value at this time is 0, the candidate “BC” is excluded from the compound word candidates. this is,
This is because if the frequency is the same, it is considered that “ABC” is a keyword that reflects the content more than “BC”.

If the appearance frequency of the compound word candidate is equal to or higher than the set threshold value, the candidate is recognized as the keyword candidate of the compound word. These thresholds can be appropriately set depending on the application content.

Next, the outline of the processing by the keyword evaluation means 4 will be described with reference to FIG. In FIG. 4, the keyword evaluation means 4 is provided with an evaluation value calculation unit 41,
The evaluation value calculation unit 41 inputs a character string of a keyword candidate of a noun word, an appearance frequency, and a general occurrence frequency from the morpheme analysis unit 2, and a keyword candidate of a compound word and a co-occurrence frequency from the compound word creating unit 3, respectively. It is configured to be.

First, depending on the application, the evaluation value calculation unit 41
In, when the keyword candidate of the noun word is a constituent word of the keyword candidate of the compound word, the co-occurrence frequency of the compound word is subtracted from the appearance frequency of the noun word. As a result, a process of excluding, from the keyword candidates, words whose appearance frequency is less than or equal to a certain threshold value is performed. By this processing, words that are difficult to be used alone in the document are excluded from the keyword candidates.

Next, in the evaluation value calculation unit 41, the evaluation value of the likelihood of becoming a keyword is calculated from the frequency information of words and compound words. Basically, as shown in (Equation 1), the evaluation value E of keyword-likeness is a value obtained by subtracting a value obtained by multiplying a general occurrence frequency F by a constant b from a value obtained by multiplying an appearance frequency f in a sentence by a constant a. To do. The constant a and the constant b, which are weighting factors for the appearance frequency and the general occurrence frequency, are appropriately set depending on the application. In the case of a compound word, the general occurrence frequency is 0. The appearance frequency is normalized according to the length of the applied text. In the above example, the occurrence frequency of the compound word is set to 0, but it may be set based on the length of the keyword character string, the occurrence frequency of the constituent word, the character type, the number, and the like.

[0033]

[Equation 1] E = fa-Fb (where a> 0, b ≧ 0) The evaluation value calculated by the keyword evaluation means 4 in this way has a higher appearance frequency in the document than a general occurrence frequency. The more likely it is to become a keyword. As for the keywords, a certain number from the top may be recognized, or all of the evaluation values having a certain threshold value or more may be selected. Such a threshold value is also appropriately set according to the scale and purpose of the document to be applied.

As described above, the keyword candidates of the word and the compound word are integrated and evaluated, and the keyword of an appropriate length is extracted, so that the document content can be reflected with a smaller number of keywords.

Further, at this time, if a keyword of a word relating to a specific field is extracted, as shown in FIG. 5, a noun occurrence frequency selection unit 53 provided in the dictionary 5 stores a occurrence frequency storage unit in the specific field. 52 may be selected. By selecting the storage unit 52 in this manner, the occurrence frequency of a noun in a specific field can be used instead of the occurrence frequency of a general noun, and it is possible to extract a keyword that more reflects the document content. Become.

Generally, the occurrence frequency is the probability that a word appears in sentences of various fields, but the occurrence frequency of a specific field is obtained by analyzing the appearance frequency of a noun word from a document in a specific field of interest in advance. It can be easily collected. For example, if keyword extraction is performed for a specific article in a newspaper, the frequency of occurrence of nouns in that newspaper may be calculated. For politics, only past political articles may be extracted and statistics may be extracted. Very good.

When the occurrence frequency information is used or collected, a person may decide which field the occurrence frequency is to be used depending on the intended use, but the words extracted from the keyword extraction or the sentence for which the occurrence frequency is collected are extracted. It is also possible to automate by checking with a thesaurus dictionary and estimating the field.

Further, by registering the obtained keyword information as a keyword dictionary, it can be used in a more advanced Japanese document analysis system.

As described above, the morpheme analysis unit 2 extracts the katakana and English character strings as they are as noun-equivalent words, so that words composed of those character types that are likely to be unknown words and compound words including the words are extracted. It becomes possible to efficiently make a keyword candidate, and further, by the simple keyword evaluation means 4 which evaluates a keyword candidate from the appearance frequency of a word in a sentence, a general occurrence frequency, and a co-occurrence frequency of a compound word,
By evaluating and extracting keywords of words and compound words at the same time, a large-capacity dictionary that registers all the subtle differences in notation such as katakana words and technical term characteristic word base extraction means It is possible to automatically create and extract a keyword including a word including an unknown word in katakana or an alphabet and a compound word consisting of valid constituent words from an input document without requiring detailed analysis of the relationship.

In the above embodiment, the morpheme analysis means 2 of the keyword extraction device is combined with the morpheme analysis device of the present invention, but the morpheme analysis means may be configured by a conventional method. Alternatively, the method other than the morphological analysis unit 2 may be configured by a conventional method.

The method for calculating the keyword evaluation value shown in the above embodiment is not limited to the above method as long as the evaluation value that more reflects the document content can be calculated.

In the above embodiment, the occurrence frequency storage section in a specific field is one field, but the storage section is not limited to this.
Storage may be provided for various specific fields.

Further, in the above-mentioned embodiment, each processing means is constituted by dedicated hardware, but not limited to this, the same function may be realized by software using a computer.

[0044]

As is apparent from the above description, in the present invention, the morpheme analysis means extracts a continuous katakana or a character string consisting of English characters in the input Japanese sentence as a noun equivalent word. It has the advantage that it can efficiently process words in a character string of consecutive letters.

The present invention also provides a compound word evaluation unit for determining an evaluation value of each compound word candidate, taking into consideration the mutual overlapping frequency of the compound word keyword candidates in the input Japanese sentence. Further, since it is provided with the keyword determining means for determining the keyword based on it, there is an advantage that the keyword can be efficiently extracted.

[Brief description of drawings]

FIG. 1 is a block diagram showing a keyword extracting device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an outline of a morpheme analysis means of the embodiment.

FIG. 3 is a block diagram showing an outline of a compound word creating means of the embodiment.

FIG. 4 is a block diagram showing an outline of a keyword evaluation unit of the embodiment.

FIG. 5 is a block diagram showing an outline of a dictionary of the embodiment.

FIG. 6 is a block diagram for explaining processing of a conventional katakana unknown word.

FIG. 7 is a flowchart showing the document analysis order of the conventional example in FIG.

FIG. 8 is a block diagram for explaining a conventional technical term extraction system.

9 is a flowchart showing a technical term characteristic word base process of the conventional example in FIG.

[Explanation of symbols]

 1 Document Input Means 2 Morphological Analysis Means 3 Compound Word Creating Means 4 Keyword Evaluating Means 5 Dictionaries 21 Unknown Word Extractors 22 Noun Extractors 31 Compound Word Candidate Creators 32 Compound Word Evaluators 41 Evaluation Value Calculators 53 Selectors

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Toshio Niwa 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd.

Claims (5)

[Claims] 1. A morphological analysis comprising: a dictionary storing at least grammatical information necessary for analysis; and a morphological analysis means for extracting at least a noun word and a noun-equivalent word from an input Japanese sentence using the dictionary. In the apparatus, the morpheme analysis means extracts a character string consisting of continuous katakana or English characters in the input Japanese sentence as a noun-equivalent word. 2. A dictionary that stores at least grammatical information necessary for analysis, a morpheme analysis unit that extracts at least a noun word and a noun-equivalent word from an input Japanese sentence using the dictionary, and the morpheme analysis unit. A noun word or a noun-equivalent word that forms a compound word consisting of a series of compound words, or a compound word keyword candidate creation unit that enumerates compound words formed by combining a plurality of words or equivalent words as keyword candidates, and its compound Compound word evaluation means for determining the evaluation value of each compound word keyword candidate in consideration of the mutual overlapping frequency of the compound word keyword candidates listed by the word keyword candidate creating unit in the input Japanese sentence. And a keyword determining device for determining a keyword based on the keyword extracting device. 3. The keyword extraction according to claim 2, wherein the mutual overlapping frequency is taken into consideration, and the appearance frequency of one compound word keyword candidate included in another one compound word keyword candidate is subtracted by one. apparatus. 4. The compound word evaluation means, the occurrence frequency of the noun word in a general sentence, the appearance frequency of the noun word and the noun equivalent word in the input Japanese sentence, and the compound word keyword candidate. Based on the frequency of occurrence of
A method of calculating an evaluation value of keyword-likeness by crossing the noun word and the noun-equivalent word with the compound word keyword candidate, and at that time, reducing the evaluation of the constituent words included in the compound word keyword candidate. 3. The keyword extracting device according to claim 2, which is characterized in that. 5. The dictionary stores a frequency of occurrence of noun words in a sentence in a predetermined specific field, and a selection for switching between the frequency of occurrence in the general sentence and the frequency of occurrence in the specific field sentence according to the use. The keyword extracting device according to claim 4, further comprising a section.