JPH0785101A - Keyword extract processing unit - Google Patents

Abstract

PURPOSE:To obtain a keyword extract processing unit by adding an attribute of a meaning of a word to an extract condition so as to apply overall evaluation to a sentence together with an item relating to apparent configuration of the sentence with respect to keyword extraction. CONSTITUTION:A syntax analysis section 1 executes a predetermined syntax analysis for a given document, a weighting section 2 allocates each importance accumulation value to each word resulting from the syntax analysis, a meaning processing section 4 adds a predetermined weighting value based on an attribute of the meaning of each word to the importance accumulation value, a syntax processing section 5 adds a predetermined weighting value to the importance accumulation value based on a part of speech and a case of each word, an appearance state processing section 6 adds a predetermined weighting value to the importance accumulation value based on a predetermined appearance state of each word in the document, and an extract processing section 3 provides the output of the word with the importance accumulation value larger than a threshold value as a keyword based on the result of the processing by the weighting section 2.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a keyword extraction processing apparatus for extracting, from a document, a word or phrase having a high relevance to the content represented by the sentence as a keyword.

[0002]

2. Description of the Related Art When a person detects a keyword or phrase in a sentence, an operator reads the sentence, understands the written contents, and intuition based on the understanding. Extract words or phrases that are judged to be highly relevant to the content, either experimentally or empirically.

Therefore, in this case, in order to extract high-quality keywords, it is necessary for the operator to have high skill, it is difficult to maintain the quality, and the cost and time are likely to increase.
From the above, automation of keyword extraction has been considered in order to efficiently extract keywords from a large amount of document data.

For example, Japanese Unexamined Patent Publication No. 03-127176, “Keyword Extraction Device” describes a method of extracting a keyword of a compound word by paying attention to an affix, and Japanese Unexamined Patent Publication No. 03-135669 “keyword automatic extraction system”. Describes a method of extracting a keyword based on the number of appearances of a subject and an object in a syntax.

Further, in Japanese Unexamined Patent Publication No. 02-32469, "Information Retrieval Method", paying attention to syntactic dependency structure, the depth of the dependency structure is used as an element expressing importance in keyword extraction. Is stated.

As described above, all of these methods use only keywords relating to the outward appearance of a sentence, such as the position of the phrase from the syntactical aspect of the sentence and the number of appearances, which are not directly related to the meaning of the phrase. It is used as an element of extraction.

Therefore, unlike the case where a person understands the content of a sentence and considers the relation between the content and the meaning content of a phrase, omission or unnecessary extraction is likely to occur in terms of meaning. An object of the present invention is to provide a keyword extraction processing device that automatically processes a keyword extraction by using the semantic attribute of a word as one of extraction conditions and comprehensively evaluating it and matters relating to the outward appearance of a sentence.

[0008]

FIG. 1 is a block diagram showing the configuration of the present invention. The figure shows the configuration of a keyword extraction processing device, which includes a syntax analysis unit 1, a weighting unit 2, and an extraction processing unit 3.

The syntactic analysis unit 1 executes syntactic analysis on a given document to convert a character string constituting the document into words,
Divide into words including compound words and phrases, and for each of the words,
Determine part of speech and case.

The weighting unit 2 has a meaning processing unit 4, a syntax processing unit 5, and an appearance state processing unit 6, and assigns an initial value to each importance cumulative value to each word of the syntax analysis result, The meaning processing unit 4 adds, for each of the words and phrases, a weight value determined in advance based on the meaning attribute of the word and phrase to the importance level cumulative value.

The syntax processing unit 5 adds, for each of the words and phrases, a predetermined weight value to the importance level cumulative value based on the part of speech and the case of the word and phrase. The appearance state processing unit 6 adds, for each of the words and phrases, a weight value determined in advance based on the appearance state in the document to the importance degree cumulative value.

The extraction processing unit 3 selects, from the result of the processing of the weighting unit 2, the importance cumulative value having a value larger than a predetermined threshold value, and the word or phrase corresponding to the selected importance cumulative value is used as a keyword. Output.

In the second invention, the appearance state processing section 6 is provided.
Is, for each of the words and phrases, a weight value predetermined based on the appearance position in the document, and a weight value predetermined based on the appearance frequency of the word with the same notation in the document. To calculate the cumulative value.

In the third invention, the appearance state processing unit 6 of the first or second invention uses a predetermined symbol as an emphasis symbol, and the appearance position of the emphasis symbol in the document and the position of the word / phrase. A weight value determined in advance based on the relationship is calculated as the relevant importance level cumulative value.

In a fourth aspect of the invention, the first, second or third aspect is provided.
The meaning processing unit 4 of the invention has a predetermined weight value for each predetermined field based on the semantic attribute of the phrase, and calculates the weight cumulative value according to the field specified for the document. Select a value.

[0016]

With the keyword extraction processing device of the present invention, the document from which the keyword is to be extracted is syntactically analyzed, and the importance of each word or phrase, that is, the word, the compound word and the phrase, is evaluated from the syntactical role. To do.

At the same time, the semantic attribute of each word or phrase is evaluated, if necessary, according to the criteria for each field of the document content, and a weight value is provided for each evaluation item in order to synthesize the evaluation by the syntax and the semantic attribute. The weight of the evaluation result is cumulatively added for each word to obtain a cumulative importance value, and a keyword with a large cumulative importance value is extracted as a keyword.

Therefore, according to the present invention, it becomes possible to evaluate the importance of a word or phrase in consideration of the meaning and content of a document when automatically extracting a keyword.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS As an example of a keyword extraction processing device of the present invention, an example of a device for extracting a keyword from a Japanese document will be described below.

In this case, the syntactic analysis unit 1 in FIG. 1 performs syntactic analysis processing on the input Japanese document. This syntactic analysis process is similar to that performed by normal translation process, etc.
This is a process of analyzing the grammatical structure of each sentence of a document by referring to a built-in word dictionary or the like.

As a result of such a syntactic analysis process, the syntactic analysis unit 1 decomposes each sentence into words, determines a part of speech of each word, a compound word composed of a sequence of words, and a phrase, and also determines a necessary word. Determine syntactic case for words, compounds, and phrases.

The weighting unit 2 receives the analysis processing result of the syntax analysis unit 1 and first assigns the importance degree cumulative value to each of necessary words such as nouns. The initial value of the cumulative value of importance is 0
In the following process, the weight value determined for each evaluation is cumulatively added to each applicable importance degree cumulative value.

The weighting unit 2 first evaluates the importance level by the meaning processing unit 4 based on the meaning attribute of the word. For that purpose, the meaning processing unit 4 has a word table showing semantic attributes as shown in FIG. 2A and a meaning attribute table showing weight values of the respective semantic attributes as shown in FIG. 2B for necessary words such as nouns. have.

The meaning attribute table is preferably provided for each appropriate field. If the meaning attribute table of FIG. 2 (b) is a table showing the weight value of the foreign affairs field, for example, the information industry field is shown in (c). Prepare a table of such weight values.

The meaning processing unit 4 searches the word table for each word for which a syntactic analysis result is required and obtains each meaning attribute,
When the meaning attribute table is searched with the meaning attribute to obtain the weight value, the weight value is added to the importance value cumulative value of the corresponding word.

Next, the weighting unit 2 evaluates the degree of importance based on the part of speech and the case of the phrase by the syntax processing unit 5, and obtains the accumulated value of the significance of the compound word or phrase. Therefore, the syntax processing unit 5
Is a part-of-speech table showing the part-of-speech name and its weight value as shown in FIG. 3 (a), a phrase table showing the structure of compound words and phrases and their weight values as shown in (b), and (c) , And a case table showing the syntactic case and its weight value.

The syntax processing unit 5 first searches the part-of-speech table for each word by the part-of-speech name indicated by the syntax analysis result, obtains a weight value, and adds the weight value to the importance cumulative value of the word.

Next, the syntactic processing unit 5 adds the weight value determined from the phrase table to the largest value of the cumulative value of importance of the words forming each compound word or phrase for the compound word and the phrase of the syntactic analysis result. The value is set as the importance cumulative value of the compound word or phrase.

After that, the syntax processing unit 5 adds the weight value determined from the case table, to the case shown in the syntax analysis result, to the importance cumulative value of the corresponding word. Further, the weighting unit 2 causes the appearance state processing unit 6 to
The importance is evaluated based on the appearance state in the document.

Therefore, the appearance state processing unit 6 sets the weight value to 2, for example, in three cases, for example, if the word to be processed is in the title, in the first half if it is in the body, for example in 200 characters, or after it. It is assumed to be 1, 0, and the position of the word in the document is identified, and the weight value is added to each importance degree cumulative value.

Further, the appearance state processing unit 6 emphasizes parenthesis symbols, underline symbols, special character font designations and the like defined as emphasis symbols as shown in FIG. 3 (d) and their weight values. When a symbol table is provided and there is an emphasis symbol shown in the table, the weight value obtained from the table is added to the accumulated importance value of the words emphasized by the emphasis symbol.

When a word is emphasized with an emphasizing symbol, the whole word in parentheses is the object of emphasizing when enclosed in parenthesis, and the marked word is the object of emphasizing when the character font is bold or underlined. Becomes

Next, the appearance state processing unit 6 totals the appearance frequencies of all the words of the same notation for all the words and phrases of the syntactic analysis result, and adds a weight value according to the appearance frequency to each importance degree cumulative value. The weight value according to the appearance frequency is obtained as follows, for example.

When N is the total number of characters in the document, if appearance frequency ≦ (5/200) × N, weight value = 0 (5/200) × N <appearance frequency ≦ (5/100) × N weight value = 1 ( 5/100) × N <frequency of occurrence = weight value = 2 FIG. 4 is a diagram for explaining a state in which the above processing is performed for the example sentence “US / SO agreed on“ middle-range nuclear ban ””. As a result of the syntax analysis by the analysis unit 1, words are divided into words as shown in the word dividing line in the figure. In this document, "diplomacy field" is specified as the field.

Further, the syntactic analysis unit 1 determines the part of speech of each word as shown in the line of speech part of the figure, and shows that three compound words are formed as shown in the line of compound word / phrase, Further, as shown in the case line, the case is identified with respect to necessary words and phrases, and the analysis results are passed to the weighting unit 2.

Therefore, the weighting unit 2 determines the meaning attribute of each word by referring to the word table (FIG. 2 (a)) as shown in the row of the meaning attribute in the figure, and the meaning attribute (foreign field) With reference to the table (FIG. 2 (b)) and the part-of-speech table (FIG. 3 (a)), weight values are determined as indicated by parenthesized numbers in the figure and added to each importance cumulative value.

Regarding the appearance positions of the words and phrases, assuming that this example sentence is placed in the title, the weight value relating to the appearance positions of all the words is set to "2". Next, with respect to the importance cumulative value obtained by adding the above weight values, the process of determining the importance cumulative value of the compound word / phrase as described above is performed, and the maximum importance cumulative value of each of the three compound words is accumulated. Since a value is taken and all are nouns in this case, the value added with the weight value "1" determined by the phrase table (Fig. 3 (b)) is set as shown in the compound word / phrase line of Fig. 4. Set as a cumulative value of importance.

In addition, a case table for the case of the syntactic analysis result
From Fig. 3 (c), the emphasis symbol table for emphasis symbols
From FIG. 3D, the weight values shown in the row of case and emphasis symbol weight in FIG. 4 are added.

In this example, since "rice" and "so" are connected by the parallel sign ".", They are identified as having the same degree of importance, and the weight value of the nominative in this case is Applied to both words.

Regarding the frequency of appearance, this example sentence is a sentence in a document of 300 characters, and the results of examining the frequency of appearance of the entire document by grouping the words and phrases in the same notation are shown by the numerical values shown in the line of frequency of occurrence in FIG. Suppose that

When the above weight value determination condition is applied to this appearance frequency, the frequency of (5/200) × 300 = 7.5 or less exceeds the weight value “0”, 7.5 (5/100) × 300 = 15 or less. The frequency is the weight value “1”, and the frequency exceeding 15 is the weight value “2”, and each weight value in parentheses in the figure is determined.

The result of adding the above weight values to each importance cumulative value is shown in the bottom row of the importance cumulative value in FIG. Of these accumulated values of importance, the extraction processing unit 3 selects a predetermined threshold value, for example, a value larger than “7”,
Output corresponding words as keywords.

Therefore, in the case of this example, the extraction processing unit 3
“US”, “SO”, “nuclear”, “prohibition”, “agreement”, “intermediate range nuclear” and “intermediate range nuclear prohibition” are extracted as keywords. FIG. 5 is a diagram showing an example of the processing flow of the present invention.
First, the parsing unit 1 receives a designation of a document and a field in a processing step 10.

Next, the syntactic analysis unit 1 reads one sentence of the designated document in the processing step 11, identifies the one in the processing step 12 and reads the one sentence instead of the end of the document, and proceeds to the processing step 13. Then, the syntax analysis process is performed as described above, and the analysis result and field are passed to the weighting unit 2.

The weighting unit 2 initializes and assigns the importance degree cumulative value to each word in the processing step 14, and the processing step 15
Then, the meaning processing unit 4 performs the weight value addition according to the meaning attribute of each word in the designated field as described above, and the processing step
At 16, the appearance state processing unit 6 performs weighting value addition according to the appearance position of each word as described above.

Next, the syntax processing section 5 performs processing step 17
Then, add the weight value according to the part of speech of each word as described above,
In processing step 18, the importance accumulation value of the compound word / phrase is set as described above, and in processing step 19, weight value addition by case is performed for the corresponding word / phrase as described above.

After that, in processing step 20, the appearance state processing unit 6 performs the weight value addition by the emphasizing symbol for the necessary words and phrases as described above. Next, the weighting unit 2 totals the number of characters of the sentence in processing step 21 and accumulates them as character numerical values, and then returns to the processing step 11 to make the syntax analysis unit 1 analyze the next sentence.

By repeating the processing for each sentence in this way,
Finally, when the syntactic analysis unit 1 identifies that the processing of all the sentences of the document has been completed in the processing step 12, the weighting unit 2 is notified of the end of the document.

Therefore, the weighting unit 2 collects the processing results of all the words and phrases in processing step 22 for each word having the same notation, and uses the character numerical value previously counted in processing step 23 as the total character number N of the document, As described above, the weight value determination condition based on the appearance frequency is calculated.

The following process is repeated for each word group having the same notation while identifying the end of the process at the processing step 24, and the appearance state processing unit 6 makes the group having the same notation at the processing step 25 into the group. The number of contained words is counted as the frequency of appearance.

Next, in processing step 26, a weight value according to the appearance frequency is determined from the calculation result, and in processing step 27, the maximum cumulative value of importance of the same notation group is taken out as the cumulative value of importance of the notation, and The weight value according to the appearance frequency is added, and the notation and the importance degree cumulative value are passed to the extraction processing unit 3.

In the processing step 28, the extraction processing unit 3 compares the received cumulative importance value with a predetermined threshold value, and only in the case where the cumulative importance value is larger than the threshold value, in the processing step 29, the word in the received notation is used as a keyword. Then, the processing returns to the post-processing step 24 of the above processing, and the weighting unit 2 processes the next same notation group.

[0053]

As is apparent from the above description, according to the present invention, in keyword extraction processing, syntax analysis is performed, and the importance of these words and phrases is evaluated from the syntactic role of each word and phrase. , The semantic attribute of each word is evaluated according to the criteria for each field of the document content, and these evaluations are integrated as the cumulative value of importance, so when automatically extracting keywords, the importance of the word is determined by taking into account the semantic content of the document. It is possible to evaluate, and there is a remarkable industrial effect that an appropriate keyword can be automatically extracted.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of the present invention.

FIG. 2 is a diagram for explaining a semantic attribute table and the like.

FIG. 3 is a diagram illustrating a part-of-speech table and the like.

FIG. 4 is a diagram illustrating an example of processing of the present invention.

FIG. 5 is a process flow chart of the present invention.

[Explanation of symbols]

 1 Syntax Analysis Section 2 Weighting Section 3 Extraction Processing Section 4 Semantic Processing Section 5 Syntax Processing Section 6 Appearance State Processing Section 10-29 Processing Steps

Claims (4)

[Claims] 1. A syntactic analysis unit (1), a weighting unit (2), and an extraction processing unit (3), the syntactic analysis unit (1) performing syntactic analysis on a given document. Then, the character string constituting the document is divided into words and phrases including words, compound words, and phrases, and the part of speech and case are determined for each of the words and phrases, and the weighting unit (2) determines the meaning processing unit (4). ) And the syntax processor (5)
And an appearance state processing unit (6), assigning an initial value to each importance accumulated value for each word and phrase of the syntactic analysis result, and for each word and phrase by the meaning processing unit (4), A weight value determined in advance based on the meaning attribute of the word is added to the importance cumulative value, and the syntax processing unit (5) determines the weight determined in advance for each word based on the part of speech and the case of the word. A value is added to the importance cumulative value, and by the appearance state processing unit (6),
A weighting value determined in advance based on the appearance state in the document is added to the importance degree cumulative value, and the extraction processing unit (3) is determined to be larger than a predetermined threshold value from the processing result of the weighting unit (2). A keyword extraction processing device configured to select the importance cumulative value having a value and output the word corresponding to the selected importance cumulative value as a keyword. 2. The appearance state processing unit (6) determines, for each of the words and phrases, a weight value determined in advance based on an appearance position in the document, and an appearance frequency of the words and phrases in the same notation in the document. The keyword extraction processing device according to claim 1, wherein a weight value determined in advance is calculated as the corresponding cumulative importance value. 3. The appearance state processing unit (6) uses a predetermined symbol as an emphasis symbol, and applies a predetermined weight value based on a positional relationship between the appearance position of the emphasis symbol in the document and the word / phrase. The keyword extraction processing device according to claim 1, wherein the keyword extraction processing device calculates the importance level accumulated value. 4. The meaning processing unit (4) has a weight value predetermined for each predetermined field based on the semantic attribute of the word and phrase, and the accumulated importance value according to the field specified for the document. 4. The keyword extraction processing device according to claim 1, claim 2 or claim 3, wherein the weight value to be calculated is selected.