JP3500698B2 - Keyword extraction device and keyword extraction method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a keyword extracting device and a keyword extracting method for automatically extracting a keyword for searching text information from a text in a document database or the like.

[0002]

2. Description of the Related Art Conventionally, as a method for searching desired information from a large amount of information stored in a database or the like, a keyword is assigned to each data in advance, and a keyword matching a search key input by a user is used. Devices have been developed to search for and output the information that they have.

In the keyword search of text information, an expert called an indexer generally assigns an appropriate keyword to the text information in advance.
However, since the addition of this keyword requires a huge amount of work, many studies have been conducted on a technique for automatically extracting the keyword. For example, Japanese Unexamined Patent Publication No. H11-11233
In the keyword importance automatic evaluation device described in Japanese Patent Publication No. 1, a noun is extracted from a document as a keyword, and statistical, syntactic and semantic importance evaluations are added.

However, in such a conventional keyword extraction method, since the extraction is generally performed on a word-by-word basis, the search results inevitably include many unrelated ones to be originally obtained. That is, there is a problem in that the precision rate decreases. This is because the concept corresponding to the search request desired by the user does not necessarily match the word-level expression, so when searching with word-level keywords, it is used in a meaning different from the meaning assumed by the user. This is because the text will also be searched. Also, regarding the evaluation of importance, even when a word is used for various meanings in one document, the importance of a word is evaluated without considering such meaning. Since it has been done, the importance is not always evaluated correctly.

On the other hand, it is possible to extract keywords in units of compound words, verb phrases, noun phrases and the like.
For example, in the keyword extracting device described in Japanese Patent Publication No. 58-33993, a method using a compound word has been proposed. According to this method, there is no restriction that the concept is extracted word by word. Also, even if it is not a compound word, an expression having the same meaning as the compound word, for example, “method of forming insulating film” for “insulating film forming method”
If such an expression is present in the text, it can be extracted as a keyword, and the keyword can be extracted regardless of the surface expression.

However, this method requires that the compound word to be extracted is registered in the keyword table in advance, and does not freely extract a word group having a relationship with each other from the text. Also, when extracting expressions that are similar to compound words, if words have a dependency relationship, they can be compound words. Therefore, for example, "search for a document" and "search from a document" are both "document / search"
Will be extracted as. For this reason, the search results often included unsuitable ones. Moreover, since the relationships between words are ignored in this way, they could not be used for the evaluation of importance.

[0007]

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and a keyword extracting device and a keyword extracting method capable of extracting an appropriate keyword capable of performing a search with a high matching rate. The purpose is to provide.

[0008]

According to a first aspect of the present invention, in a keyword extraction device for extracting a keyword from a text, an expression extraction for extracting an expression consisting of a plurality of words and phrases related to each other from the text. Means, a relation extracting means for estimating a relation between the words and phrases constituting the expressions extracted by the expression extracting means, and outputting relation information indicating the relationship and a relation expression composed of a plurality of words, and output by the relation extracting means. A relation expression selecting means for extracting a relation expression composed of the same phrase from the relation information and the relation expression which are not contradictory to each other and selects the relation expression having the relation information having the lowest abstraction as a keyword candidate. Characterized by having .

According to the second aspect of the invention,
In a keyword extracting device for extracting a keyword from text, an expression extracting means for extracting an expression consisting of a plurality of words and phrases related to each other from the text, and estimating a relationship between the words and phrases forming the expression extracted by the expression extracting means. A relation extracting means for outputting relation information representing a relation and a relation expression consisting of a plurality of words, a frequency counting means for counting the appearance frequency of the relation expression outputted by the relation extracting means, and a relation extracting means for outputting the relation expression. From the relation information and the relation expression, a relation expression composed of the same words and having relations with each other that are not in conflict with each other is extracted, and the relation expression having the relation information with the lowest abstraction is selected as a keyword candidate and is not selected. Oh And a relation expression evaluation means for evaluating the importance of the relation expression selected by using the appearance frequency of the relation expression selected as a candidate by the frequency coefficient means, and a keyword based on the evaluation result of the relation expression evaluation means. It is characterized by having a keyword selecting means for selecting.

Further, in the invention according to claim 3, in the keyword extracting method for extracting a keyword from text, a step of extracting an expression composed of a plurality of mutually related terms from the text by the expression extracting means, and a relation extracting means A step of estimating a relation between the words and phrases constituting the expressions extracted by the expression extracting means, outputting relation information representing the relationship and a relation expression consisting of a plurality of words, and a relation expression selecting means outputting the relation expression. It has a step of extracting from the relation information and the relation expression a relation expression composed of the same words and phrases in which the relation information does not contradict each other, and selecting the relation expression having the relation information with the lowest abstraction as a keyword candidate. It is a characteristic.

[0011]

According to the present invention, in the inventions of claims 1 and 3, the expression extracting means extracts an expression consisting of a plurality of words and phrases related to each other from the text, and the relationship extracting means extracts the expression. The relationship between the words and phrases that compose the expression extracted in 1. is estimated, and the relational information that represents the relationship and the relational expression composed of a plurality of words and phrases are output. The relation expression selecting means extracts a relation expression composed of the same words and phrases from the relation information and the relation expression output by the relation extracting means, and the relation information is consistent with each other, and has the relation information having the lowest abstraction among them. Extract expressions as keywords. In this way, not only the words in the text but also the relationships between the words can be extracted as keywords, and appropriate keywords can be extracted from similar expressions with different degrees of abstraction of the relationships.

Further, in the invention described in claim 2,
Furthermore, a frequency counting means for counting the appearance frequency of the relational expressions output from the relation extracting means is provided, and the relational information and the relational expressions output from the relational extracting means are composed of the same words and the relational expressions are not contradictory to each other. The relation expression having the relation information having the lowest abstraction is extracted as a keyword candidate, and the appearance frequency counted by the frequency coefficient means of the relation expression not selected and the relation expression selected as a candidate is used. The relation expression evaluation means evaluates the importance of the selected relation expression. By selecting a keyword by the keyword selection means based on this evaluation result, the importance can be evaluated more accurately and the keyword can be extracted accurately by using the appearance frequency of the keyword candidates that have not been selected. it can.

[0013]

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a first keyword extraction device according to the present invention.
3 is a block diagram showing the overall configuration of the embodiment of FIG. 1 in the figure
Is a keyword extracting device, 2 is a data input unit, 3 is an expression extracting unit, 4 is a relation extracting unit, 5 is a relation expression selecting unit,
Reference numeral 6 is a storage unit, 11 is a morphological analysis unit, and 12 is a word group extraction unit.

The data input section 2 is a magnetic disk or OC.
A Japanese text from which a keyword is extracted is read from R, MT, or the like. Of course, the configuration may be such that the input device is used to directly input. The expression extraction unit 3
It includes a morphological analysis unit 11 and a word group extraction unit 12. The expression extraction unit 3 performs a morpheme analysis on the text read by the data input unit 2 by a morpheme analysis unit 11, and based on the result, an expression that may be a keyword candidate according to the expression candidate extraction rule in the word group extraction unit 12 together with a part of speech. Extract. The relation extracting unit 4 analyzes the relation between the words constituting the expression extracted by the expression extracting unit 3, converts the relation into a relation symbol representing the relation, and generates a relation expression including a set of a plurality of terms and the relation symbol. To do. Relation expression selection unit 5
Out of the keyword candidates output by the relationship extracting unit 4,
A relation expression that is composed of the same words and whose relation symbols do not contradict each other is extracted, and only the relation expression having the relation symbol with the lowest abstraction is selected as a keyword candidate. Based on the selected keyword candidate, the extracted keyword is recorded in the storage unit 6 together with the input document.

The expression extraction unit 3 is provided with a syntactic analysis unit, a semantic analysis unit, etc. in addition to the morpheme analysis unit 11, or an analysis unit integrating them, for deeper analysis. More accurate extraction may be performed based on this.

Next, an example of the operation in the first embodiment of the present invention will be described. First, the document data is electronically read from the data input unit 2 and sent to the expression extraction unit 3. As described above, the expression extraction unit 3 reads the document data one sentence at a time, that is, up to a punctuation point, and the morphological analysis unit 11 performs morphological analysis to divide the data into words. For morphological analysis,
It is widely known as a basic technique of natural language processing, and various known methods such as a Japanese morphological analysis method described in JP-A-60-20234 can be used.

Next, the word group extraction unit 12 extracts a word group that can be a keyword candidate according to an expression extraction rule based on a combination of a part of speech and a surface expression. In this example, a method of extracting a word group from only the morphological analysis result is used, and two independent word groups that are directly connected via an auxiliary word such as a particle are used as the extraction unit. Of course, the word group may be extracted with reference to other information, such as using the semantic analysis result. Also, three or more independent word groups may be used as the extraction unit. In the following description, it is assumed that the extracted expression pattern is “pre-independent word group + adjunct word group + post-independent word group”, and in order to extract this pattern, the pre-independent word group stream JF and the adjunct word group are extracted, respectively. The group stream FZ and the post independent word group stream JB are used. The expression extraction rule describes a possible relationship between the pre-independent word group, the adjunct word group, and the post-independent word group of the expression to be extracted.

Although the preceding independent word group, the adjunct word group, and the subsequent independent word group in the above-described extracted expression patterns are directly connected in the following description, they are directly connected. It is not necessary and can be configured to extract the closest word group that matches the expression extraction rule. In addition to this pattern, other patterns may be extracted.

FIG. 2 is a flowchart showing an example of the operation of the expression extraction unit 3. The expression extracting unit 3 takes out one sentence from the document data in S22, and in S23, the morphological analyzing unit 1
Morphological analysis is performed in 1 to divide into words. S
In step 24, the pre-independent word group stream JF, the adjunct word group stream FZ, and the post-independent word group stream JB are cleared, and in S25, the independent word group in the front independent word group stream JF, in this case, the sa verb verb stem, noun, and adjective verb stem Read the sequence of. In the following description and drawings, the sa verb verb stem may be abbreviated as Sa, and the adjective verb stem may be abbreviated as adjective verb. In S26, the adjunct word group following the previous independent word group is read into the adjunct word stream FZ.

In S27, the pre-independent word group stream J
F, it is determined whether or not the combination of the pre-independent word group and the adjunct word group read in the adjunct word group stream FZ matches the expression extraction rule, and if they do not match, it is not a pattern to be extracted. Return and discard the extracted pre-independent word group and attached word group. If it matches the expression extraction rule, in S28, the independent word group following the attached word group is read into the subsequent independent word group stream JB. Then, in S29, it is determined whether or not it matches the expression extraction rule using the front independent word group stream JF, the auxiliary word group stream FZ, and the rear independent word group stream JB. If they match, in S31, the contents read in the preceding independent word group stream JF, the attached word group stream FZ, and the subsequent independent word group stream JB are output to the output expression stream EX.
If there is no match, in S30, it is determined whether or not a compound word is read in the previous independent word group stream JF. If a compound word is read, in S31, the previous independent word group stream JF is determined. The content of is output to the output expression stream EX.

In S32, the rear independent word group stream JB among the currently read front independent word group stream JF, adjunct word group stream FZ, and rear independent word group stream JB is read.
The independent word group read in can be the previous independent word group of the succeeding word string. Therefore, the independent word group read in the rear independent word group stream JB is stored in the front independent word group stream JF, and the adjunct word group stream FZ and the rear independent word group stream JB are cleared.

In S33, it is determined whether or not the sentence has been read to the end, and if it is still in the middle of the sentence, the process returns to S25 and the above-described pattern extraction processing is continued. When the end of the sentence is reached, the process returns to S21, and it is determined whether or not the document is the end. If the sentence still remains, the processes from S22 are repeated. When the processing is performed up to the end of the document, the processing of the expression extracting unit 3 ends. At this time, the expression output to the expression output stream EX is the expression extracted by the expression extracting unit 3. The extracted expression is sent to the relation extracting unit 4.

The above operation will be described using a specific example. FIG. 3 is an explanatory diagram of an example of a morpheme analysis result, FIG. 4 is an explanatory diagram of an example of an extracted expression, and FIG. 5 is an explanatory diagram of an example of an expression extraction rule. For example, in S22, it is assumed that the sentence "The system realized by us realizes high-speed search of documents by applying this method." S2
3, the morphological analysis unit 11 performs morphological analysis,
The analysis result as shown in FIG. 3 is obtained.

Next, the word group extraction unit 12 extracts a word group that can be a keyword candidate from the divided word strings according to the expression extraction rule. An example of the expression extraction rule is shown in FIG. For example, in Rule 1, the pre-independence word group is composed of a noun, a sa verb verb stem, or an adjective verb stem, or a sequence of these, followed by "ni" as an adjunct group, and further behind it. It is shown that the expression is extracted when the syllable stem is present as a word group. The same applies to other rules.

In the above-mentioned example sentence, from the morphological analysis result shown in FIG. 3, first, in S25, "we" (pronoun) is in the pre-independent word group stream JF, and in S26, "ga" (case particle) is an adjunct word group. Each is read into the stream FZ, and matching is performed in S27 with the expression extraction rule shown in FIG. Since the expression extraction rule shown in FIG. 5 does not have a rule starting with a pronoun, the following process is not performed, and the process returns to S24 to clear each stream.

Then, "realization" (sa-hen) is added to the pre-independent word group stream JF, and "shi" (sa verb ending / adjunct ending (hereinafter sometimes abbreviated as sa-ending) is added to the adjunct word stream FZ. Each is read and matched with the expression extraction rule. In this case, since it matches rule 7 of the expression extraction rule shown in FIG. 5, the contents of each stream are retained as they are, and in S28, the independent word "system" that follows is maintained.
(Noun) is read into the post-independent word group stream JB.
Subsequently, the user tries to read "ha" (adjective particle), but since it is not an independent word, it is not read, and matching with the expression extraction rule is performed in S29. Since this matches the rule 7 of FIG. 5, the expression “realized system” is output to the expression output stream EX together with the part-of-speech information in S31.

Next, in S32, the independent word group stream JB
Copy the contents of the previous independent word group stream JF, S2
Read the following words at 5,26. That is, "system" is copied to the pre-independent word group stream JF, and "ha"
(Sub particle) is read into the attached word group stream FZ.
In this case, since there is no matching expression extraction rule, S24
Return to and read the next word. here,","
(Symbol) and "book" (prefix) are not independent words and are ignored.

Next, when the "method" (noun) is read into the pre-independent word group stream JF, the subsequent "application" (sa-hen) is also an independent word, so that it continues as a compound word in the pre-independent word group stream JF. take in. In this way, an independent word group is treated as a sequence of independent words without an inflection ending, that is, a case in which any of nouns / sahen / adjective verbs continues, and the part of speech of the independent word group is treated as a noun. Then, "by" (a case particle equivalent) is attached to the adjunct word stream FZ.
Read in. In this case, there is a rule (rule 5) that matches the expression extraction rule shown in FIG.
(Noun) is read into the post-independent word group stream JB.
Then, when matching the expression extraction rule, there is no matching rule. That is, the pre-independent word group and the adjunct word group are
It matches the rule 5 of the expression extraction rules shown in FIG. 5, but it does not match the rule 5 because the post-independent word group is not a sa-adjective or adjective verb. Therefore, the “method application” read in the pre-independent word group stream JF as a compound word in S30 is output to the expression output stream EX together with the information of the part of speech as an expression that can be a keyword candidate. Then, the "document" read in the rear independent word group stream JB is copied to the front independent word group stream JF, and extraction of expressions is continued in the same manner.

The expressions extracted from the above-mentioned example sentences as described above are shown in FIG. As described above, when a compound word forms a front independent word group or a rear independent word group, the compound word is treated as one noun, and therefore each word forming the compound word is shown by connecting with a'- '. There is. Other words are shown separated by '/'. Each expression is extracted together with part-of-speech information.

When the expression extraction unit 3 detects in S33 that the expression extraction up to the end of the sentence is completed, the next one sentence is extracted in S22, and similarly, the morphological analysis and the word group extraction are performed.
In this way, when it is detected in S21 that the expression extraction up to the end of the document has been completed, the relationship extracting unit 4 performs the relationship extraction operation on the expressions output to the expression output stream EX.

FIG. 6 is a flow chart showing an example of the relation extracting operation of the relation extracting unit 4. The relationship extracting unit 4 receives the expressions extracted by the expression extracting unit 3, analyzes the relationship between the words forming each expression using the relationship estimation rule, converts the expressions into relation symbols representing the relationships, and converts the expressions into a plurality of words. And a relational expression consisting of a pair of relational symbols and relational symbols.

In S41, the expression output stream EX
Take one of the expressions from. A relation estimation rule that matches the expression extracted in S42 is searched. A relation symbol corresponding to the matched relation estimation rule is obtained, and the relation symbol is added to the expression in S43.

Then, a relation expression of the form {relation symbol expression 1 expression 2} is generated. Basically, the pre-independent word group becomes expression 1 and the post-independent word group becomes expression 2. However, only when the paraphrased expressions in which the word order is exchanged can be performed, the pre-independent word group and the back-independent word group are used if necessary. Swap the independent words. For example, there are cases such as "realization of system" and "realized system", and in this case, the word order of "realized system" is exchanged. As for which to replace, generally, it is basically the order of the nouns and the nouns. In S44, it is determined whether or not the word order needs to be exchanged, and if so, a relation expression in which the items of expression 1 and expression 2 are exchanged is generated in S45. If there is no need for replacement, the relation expression is generated in S46 in that order. The generated relation expression is S
At 47, the relation expression group stream RL is output.

In S48, it is determined whether or not all expressions in the expression output stream EX have been processed. If any unprocessed expressions remain, the process returns to S41 to repeatedly generate relation expressions.

An example of the operation of the above-mentioned relation extracting unit 4 will be described based on a concrete example. FIG. 7 is an explanatory diagram of an example of the relation estimation rule, and FIG. 8 is an explanatory diagram of an example of the relation expression.
Here, as a specific example, a case will be described in which the expressions shown in FIG. 4 are extracted by the expression extraction unit 3, stored in the expression output stream EX, and passed to the relationship extraction unit 4.

As shown in FIG. 7, the relation estimation rule is described as a correspondence table of a set of a front independent word group, a rear independent word group, an adjunct word group, and a relation symbol. For example, the relation symbol [no] corresponds to an expression in which the anteroposterior word group is noun or sahen, the adjunct word group is “no”, and the posterior independent word group is sahen. The same applies to other combinations.

First, in S41, one expression is extracted from the expression output stream EX. It is assumed that the expression "realized system" is retrieved. The relation extracting unit 4 estimates the relation between “realization” and “system”. In S42, the relation estimation rule shown in FIG. 7 is searched. In this example, the previous independent word group "realization" is Sahen, and the subsequent independent word group "System"
Is a noun and the adjunct word group is “do”, so the relation estimation rule shown in the bottom column of the relation estimation rules shown in FIG. 7 matches. Take the corresponding relation symbol [thru] and give this relation to the representation.

Next, in S44, it is determined whether the replacement is necessary. In this example, the words are arranged in the order of idioms-syllables, and an expression in which the word order is exchanged is also possible, so it is determined that the exchange is necessary. The determination at this time is, for example,
It can be configured such that matching is performed with a word order changing rule that represents a special case where the word order can be changed, and if there is a corresponding rule, the word order is changed. In the case of the above-mentioned "realized system", the word order is changed in S45, and the relation expression of the form {[sul] system realization} is generated. The generated relational expressions are sequentially output to the relational expression group stream RL in S47.

In the case of a compound word containing only independent words,
The relation symbol "direct connection" is added. Here, in the case of a compound word consisting of two words, the two words are expression 1 and expression 2, respectively. In the case of a compound word of three or more words, expression 1 and expression 2 are not specified. For example, "high-speed search function"
In the case of the compound word, {[direct connection] high speed-search-function}
Is generated. At this time, the terms high speed and search-function or high speed-search and function are not used.

In this way, the relation expression {[sul] system realization} was generated from the first expression "realized system". Hereinafter, with respect to “application of method”, “high-speed search function of document”, and “realization of high-speed search function”, the relation processing as shown in FIG. 8 is generated by similar processing. When all the expressions extracted by the expression extracting unit 3 are converted into relation expressions in the relation extracting unit 4, the relation expression group output to the relation expression group stream RL is passed to the relation expression selecting unit 5.

FIG. 9 is a flowchart showing the operation of the relation expression selecting unit 5. The relation expression selecting unit 5 extracts one relation expression from the relation expression group stream RL in S52 and copies it to the relation expression stream RE. Subsequently, in S53, all relation expressions composed of the same words as the relation expression stream RE are set to the relation expression group stream RL.
Search inside and copy to stream SI. At this time,
Relation expression stream RE itself is also stream SI
To copy.

Next, the relation symbols of the relation expressions in the stream SI are compared. The relation symbol is given in advance information on the degree of abstraction and the relation symbol that does not contradict the relation symbol. Based on this information, in S54, the relation expression inconsistent with the relation expression stream RE is deleted from the stream SI. In S55, relational expressions in the stream SI are deleted from the relational expression group stream RL so that selection processing is not performed redundantly for relational expressions that do not contradict each other. And S56
At, the one having the lowest degree of abstraction of each relation symbol in the stream SI is selected and output to the result output stream SO.

Returning to S51, until there is no relation expression in the relation expression group stream RL,
The above process is repeated. As a result, the result output stream SO contains the keyword candidates from which expressions having a similar relationship are excluded. The keywords written in the result output stream SO are registered in the storage unit 6 together with the input document.

An example of the operation of the above-mentioned relation expression selecting unit 5 will be described based on a concrete example. FIG. 10 is an explanatory diagram of an example of a relation expression composed of the same words, and FIG.
FIG. 4 is an explanatory diagram of an example of information included in a relation symbol.

In S52, the relation expression group stream R
If, for example, the expression {[sul] system realization} is retrieved from L, this expression is copied to the relational expression stream RE. Then, the content of the relation expression stream RE and the relation expression having the same words as this expression, that is, "system" and "realization", are copied from the relation expression group stream RL to the stream SI. The relational expression copied to stream SI is shown in FIG. As shown in FIG. 10, even relation expressions having the same word have different relation symbols. For example, "realized system", "realized system",
There are expressions such as “system realization” and “system realization”. These are distinguished by adding different relation symbols in the relation extracting unit 4.

Next, the relation symbols of the relation expressions in SI are compared. As described above, the relation symbol is previously provided with the degree of abstraction and the relation symbol information that does not conflict with the relation symbol. This example is shown in FIG. As shown in FIG. 11, the degree of abstraction is represented by, for example, a number from 1 to 7, and the greater the number, the greater the degree of abstraction. In FIG. 11, for example, the abstraction degree of the relation symbol [wo] is 1 and the abstraction degree of the relation symbol [no] is 3.
Is. This means that, for example, the expression "realization of the system" may mean "realization of the system", but this is not always the case. In this way, the degree of abstraction expresses that certainty differs depending on the expression. The degree of abstraction depends on the type of relation symbol described above, and if the type of relation symbol changes, the way in which the degree of abstraction is given also changes.

Using such information, first, relation symbols are compared with each other, and keyword candidates having inconsistent relation symbols are deleted from SI. Generally, relation symbols having the same degree of abstraction conflict with each other, and some relation symbols having different degrees of abstraction do not. In FIG. 11, relational symbols having a higher degree of abstraction than themselves that do not conflict are given as information of relational symbols that do not conflict. That is, the relation symbol [no] is given the relation symbols [thru] and [direct connection] that are not inconsistent among the relation symbols [noun connection], [thru], and [direct connection] having a higher degree of abstraction.

For example, "realize a system" can be rephrased as "realize a system", but "exhibit at an exhibition" cannot be rephrased as "exhibit at an exhibition". . Therefore, the relation symbols [wo] and [no] are inconsistent, but the relation symbols [d] and [no] are inconsistent.

The relation symbols are compared by selecting the relation symbol in the relation expression stream RE and the stream SI having the lowest abstraction level, and the other relation symbols having the consistent relation symbols of the selected relation symbols. If included in the information of
Judge that there is no contradiction. In the example shown in FIG. 10, the relation symbol [wo] having the lowest abstraction is selected, and the relation symbols [no], [thru], and [direct connection] that do not contradict this are selected.
And the relation symbols of other relation expressions. When a relational expression having an inconsistent relational symbol is found, the inconsistent candidate is removed according to a predetermined rule such as giving priority to a low abstraction. In the example shown in FIG. 10, since all candidates are consistent, no removal operation is performed. In this way, the stream SI in which the inconsistent relation expressions are deleted
The relation expressions in each have similar semantic relationships. Therefore, one of these relation expressions may be extracted as a keyword candidate.

Since the keyword candidates are extracted from the relation expressions remaining in the stream SI by the selection processing, the relation expressions in the stream SI are related to each other in order not to extract the keywords from these relation expressions. Erase from stream RL.

Next, keyword candidates are extracted from the similar relation expressions. The relation symbols of the relation expressions in the stream SI are compared, the one with the lowest abstraction level is selected, and the relation expression is written as a keyword in the result output stream SO. The reason why the relation symbol having a low degree of abstraction is selected is that the lower the degree of abstraction, the more reliable the relationship between words and the more effective the function as a keyword. Here, the relation symbol [wo] is selected, and the relation expression {[wo] system realization} is used as a keyword to output the result output stream SO.
Written in.

In this way, the selection operation for one relational expression RE is completed. The relation expression selecting unit 6 repeats the same operation thereafter. When the selection of all relation expressions of the relation expression group stream RL is completed, the relation expression selection operation ends.

As described above, according to the first embodiment, by selecting the relation expression having the lowest abstraction degree of the relation symbol as the keyword from the relation expressions composed of the same independent word, the relation is confirmed. The relation expression can be used as a keyword, and the accuracy of keyword extraction can be improved.

FIG. 12 is a block diagram showing the overall construction of the second embodiment of the present invention. In the figure, the same parts as those in FIG. 7 is a frequency counter,
Reference numeral 8 is a relation expression selection / evaluation unit, and 9 is a keyword selection unit. In the second embodiment, in addition to the configuration of the first embodiment, a frequency counting unit 7 is provided after the relationship extracting unit 4. Further, a relation expression selection / evaluation unit 8 is provided in place of the relation expression selection unit 5 of FIG.
After that, the keyword selection unit 9 is provided.

The data input unit 2, the expression extraction unit 3, and the relation extraction unit 4 are the same as those in the first embodiment. The frequency counting unit 7 counts the appearance frequency of each expression by removing duplication from the relation expression output from the relationship extraction unit 4, and outputs the relation expression together with the appearance frequency. The relation expression selecting / evaluating unit 8 firstly, like the first embodiment, the relation extracting unit 4
From the keyword candidates output from, the relation expressions that are composed of the same words and have relation symbols that do not contradict each other are extracted, and only the relation expression having the relation symbol with the lowest abstraction is selected as the keyword candidate. Furthermore, in this second embodiment,
The importance of the selected relational expression is evaluated by using the appearance frequency of the relational expression that is not selected. The keyword selection unit 9 selects a keyword based on the evaluation result of the relation expression selection / evaluation unit 8.

Next, the second keyword extraction device of the present invention will be described.
An example of the operation in this embodiment will be described. Relationship extraction unit 4
The operations up to are the same as those in the first embodiment. When all the expressions extracted by the expression extracting unit 3 are converted into relation expressions in the relation extracting unit 4, the frequency counting unit 5 removes overlapping relation expressions and gives appearance frequencies. As a result, the relation expression is converted into, for example, the form of {relation symbol expression 1 expression 2 count value}. Specifically, for example, the format is {[Sul] system expression 2}.

The relation expression group given the frequency is
The relation expression selection / evaluation unit 8 selects / evaluates. FIG. 13 is a flowchart showing an example of the operation of the relation expression selecting / evaluating unit 8. From the relational expressions composed of the same independent words, the one that has the lowest abstraction degree of the relational symbol is completely the same as in the first embodiment. That is, in steps S61 to S65 of FIG. 13, the same processing as that of steps S51 to S55 of FIG. 9 is performed.

At S66, in the second embodiment,
Write the selected relational expression to stream CO. Then, the importance of the relational expression written in the stream CO in S67 is calculated. The importance can be calculated by simply adding the appearance frequencies of all relation expressions in the stream SI as the importance. In addition to this, various methods such as weighting the frequency of appearance of each relational expression according to the degree of abstraction of the relational symbol and adding them can be used.

In S68, the relational expression in the stream CO and the importance calculated in S67 are output to the result output stream SO, and the selection / evaluation operation for one relational expression read in the relational expression stream RE ends. To do. The relation expression selecting / evaluating unit 8 repeats the same operation thereafter. When the selection / evaluation of all relation expressions of the relation expression group stream RL is completed, the relation expression selection / evaluation unit 8
Operation ends.

FIG. 14 is an explanatory diagram of another example of the relation expression composed of the same words. As a specific example,
As in the embodiment of FIG.
It is assumed that the relation expression shown in (1) is read in the relation expression stream RE and stream SI.
Here, the appearance frequency of each relational expression read into the relational expression stream RE and the stream SI is counted by the frequency counting unit 7, and a count value is given. In S66, the relation expression {[wo] system realization 2} having the lowest abstraction degree is selected from these relation expressions and written in the stream CO. At S67, the importance of this relational expression is calculated. As described above, the importance of the relation expression {[wo] system realization 2} is 8 if the importance is simply added with the appearance frequencies of all the relation expressions in the stream SI. In this way, the selected relation expression and its importance are output to the result output stream SO in the form of {[wo] system realization 8}.

The keyword selection unit 9 uses the importance calculated by the relation expression selection / evaluation unit 8 to calculate, for example,
A keyword having a predetermined value or more is selected as a keyword and registered in the storage unit 9 together with the input document. For the value given in advance, for example, give the calculation method,
It is configured to calculate at the time of keyword selection based on the number of keyword candidates or the distribution of importance, or the user gives it to the system at the start of keyword extraction operation, presents the importance evaluation result to the user and inputs a threshold value, etc. Various methods are conceivable, and any method may be used.

As described above, according to the second embodiment, the importance of the selected relation expression is evaluated more accurately by using the appearance frequency of the relation expression not selected as the keyword candidate. It is possible to provide a keyword extracting device that can evaluate various importance levels and can extract keywords with high accuracy.

In the second embodiment described above, the frequency counting unit 7 counts the appearance frequency of the same relational expression, and the keyword selecting unit 9 selects the relational expression selecting / evaluating unit 8.
The importance is evaluated from the appearance frequency of the relation expressions not selected as the keyword candidates and the appearance frequency of the selected relation expressions. However, in the above method, for example, “document processing” and “document processing system” are extracted as different relation expressions. Then, since the word groups are not the same, the relation expression selection / evaluation unit 8 does not extract the relation expressions having the same word, so that they are output to the keyword selection unit 9 as different keyword candidates. In a document, for example, "document processing" is twice, and "document processing system" is 3 times.
If the keyword selection unit 9 selects a relation expression three or more times, the document processing system
Is extracted as a keyword, and “document processing” is no longer selected as a keyword. However,
“Document processing” and “document processing system” are not completely different concepts, but if we focus on the concept of “document processing”,
It is reasonable to think that it has appeared once. Therefore, in reality, “document processing” may be more important.
As described above, if the evaluation and selection are performed only with respect to the same word group, accurate evaluation of the importance cannot be performed, which may cause a decrease in the matching rate at the time of search.

In order to solve this, at the time of evaluation by the relation expression selecting / evaluating unit 8 or at the time of selection by the keyword selecting unit 9, there is a second keyword candidate having a certain first keyword candidate as a part. When
The importance of the keyword candidate may be evaluated based on at least the appearance frequency of the first keyword candidate and the appearance frequency of the second keyword candidate. As a result, even when a certain keyword candidate is included in another keyword candidate, it is possible to add the degree of importance commensurate with the actual appearance frequency. For example, when the importance calculated by the relation expression selection / evaluation unit 8 is “document processing” is 2 and the “document processing system” is 3, the importance of “document processing” is evaluated as 5 can do.

The evaluation method described above can be applied to devices other than the keyword extracting device using the relational expression. FIG. 15 is a block diagram showing a third embodiment of the keyword extracting device of the present invention. In the figure, the same parts as those in FIG. 1 and FIG. Reference numeral 10 is an importance evaluation unit, and 13 is a word group extraction unit. The data input unit 2 and the storage unit 6 are the same as those in the first and second embodiments. The keyword selection unit 9 is also the same as in the second embodiment.

The expression extraction unit 3 is composed of a morpheme analysis unit 11 and a word group extraction unit 13. The expression extracting unit 3 uses the text read by the data input unit 2 as a morphological analysis unit 11
The morpheme analysis is carried out at, and the word group extraction unit 13 extracts a keyword candidate from the result according to the expression extraction rule. In the third embodiment, the word group extraction unit 13 will be described below as extracting a compound word. However, the present invention is not limited to this, and various patterns such as the same patterns as those of the first and second embodiments described above may be extracted.

The importance evaluation unit 10 calculates the importance based on the appearance frequency of the expression output from the expression extraction unit 3 and counted by the frequency counting unit 7. At this time, when a certain keyword candidate is included in another keyword candidate, the importance degree of the keyword candidate is calculated from the appearance frequencies of both.

Next, the third embodiment of the keyword extracting device of the present invention will be described.
An example of the operation in this embodiment will be described based on a specific example. As a specific example, consider a case where the example sentence “The system realized by us realizes high-speed search of documents by applying this method” used in the above-described first and second embodiments is input. This example sentence is morphologically analyzed by the morphological analysis unit 11 in the expression extraction unit 3, and the morphological analysis result as shown in FIG. 3 is obtained. The word group extraction unit 13 extracts, from the word string divided by the morpheme analysis unit 11, a word group that can be a keyword candidate according to the expression extraction rule. Here, as an expression extraction rule, it is assumed that an independent word without an inflection ending is extracted, that is, when any of a noun, a verb, and an adjective verb is continuous. FIG. 16 is an explanatory diagram of an example of keyword candidates extracted by the word group extraction unit 13. In the above example sentence, two keyword candidates as shown in FIG. 16 are extracted.

Morphological analysis unit 11 and word group extraction unit 13
The process is performed one sentence at a time, and thereafter, this is repeated until the end of the document data. When the extraction of the word groups for all the document data is completed, the frequency counting unit 7 searches the extracted word groups for overlapping word groups, counts the appearance frequencies of those word groups, and eliminates the duplication. . The set of the word group and the appearance frequency thus obtained is passed to the importance evaluation unit 10.

FIG. 17 is a flow chart showing an example of the operation of the importance evaluation section 10 in the third embodiment of the keyword extraction system of the present invention. It is assumed that the set of word groups and appearance frequencies passed from the frequency counting unit 7 has been input to the keyword candidate group stream KE.

In step S71, the importance evaluation section 10 first sorts the keyword candidate group stream KE in forward matching order (dictionary order) and the forward matching order keyword candidate group stream KF and the keyword candidate group stream KE in backward matching order (S71). A backward matching order keyword candidate group stream KB prepared by sorting (from the end of the character string in the dictionary order) is prepared.

Subsequently, in S72, it is confirmed that there is a keyword candidate in the prefix matching order keyword candidate group stream KF, and in S73, one keyword candidate is extracted from the prefix matching order keyword candidate group stream KF and read into the variable KY. .

Next, in S74, it is determined whether or not there is a keyword candidate that prefix-matches the variable KY in the prefix matching order keyword candidate group stream KF. If there is, a variable KY is determined in S75. All keyword candidates that prefix match with are copied to the prefix match candidate stream FO. Further, in S76, it is determined whether or not there is a keyword candidate that back-matches the variable KY in the backward-matching order keyword candidate group stream KB, and if there is, a backward-matching keyword candidate is found in S77. All are copied to the trailing match candidate stream BA. Here, the forward match and the backward match are determined not on a character-by-character basis but on a word-by-word basis, and when all the words forming the variable KY are included, it is determined that the prefix match and the backward match are made.

FIG. 18 is an explanatory diagram of an example of keyword candidates obtained by prefix matching and suffix matching. It is assumed that {high speed-search 3} is read in the variable KY. Here, "high speed-search" is a keyword candidate, and "3"
Is the frequency of occurrence. In the prefix match candidate stream FO, prefix match with this keyword candidate, for example, "high speed-search-system", "high speed-search-function", or the like is written. Also, in the trailing match candidate stream BA,
For example, "document-high speed-search" or the like is written.

Returning to FIG. 17, in S78,
The importance of the variable KY is calculated using the information of the candidates in the prefix match candidate stream FO and the suffix match candidate stream BA. Here, the degree of importance is calculated by adding the appearance frequencies of all the candidates in the forward matching candidate stream FO and the backward matching candidate stream BA to the appearance frequency of the variable KY. In the example shown in FIG. 18, the keyword candidate “high speed-search” has an importance level of 15. The keyword candidates to which the degree of importance is given are written in the candidate-group stream IM with degree of importance in S79. The importance degree evaluation unit 10 repeats the above processing until there are no keyword candidates in the front matching order keyword candidate group stream KF.

The importance evaluation unit 10 calculates the importance in addition to the method of simply adding the appearance frequencies as described above. For example, the candidates in the forward match candidate stream FO and the backward match candidate stream BA may be calculated. There are various possibilities such as weighting and adding frequencies according to the number of overlapping words of the keyword candidate KY, the ratio of overlapping words, forward matching or backward matching, and the like. Any method may be used as long as it uses frequency information.

The keyword candidate group to which the importance level is assigned by the importance level evaluation section 10 is passed to the keyword selection section 9, and a keyword is selected from the keyword candidates to which the importance level is assigned based on the importance level. . For example, a candidate having a degree of importance greater than a predetermined threshold is selected as a keyword and registered in the storage unit 6 together with the input document.
A predetermined threshold value for selecting a keyword to be used by the keyword selecting unit 9 is calculated, for example, by a calculation method, and is calculated at the time of keyword selection based on the number of keyword candidates or distribution of importance, or at the time of starting keyword extraction operation. Various methods can be used, such as the user giving the result to the system, or presenting the importance evaluation result to the user to input the threshold value.

In the third embodiment, the expression extracting unit 3
Although the case where a compound word is extracted as a keyword candidate is shown, a specific part-of-speech pattern may be extracted based on a rule even if there is an adjunct word between them or they are not continuous, and an adjunct word is also extracted. May be. If you also want to extract attached words,
Only independent words can be used to judge forward match and backward match. Of course, as in the first and second embodiments, the relation expression can be extracted.

[0080]

As is clear from the above description, according to the present invention, not only the notation of words in a text but also the relation between them can be extracted to extract keywords.
Appropriate keywords can be extracted from similar expressions with different degrees of relation abstraction. Further, by using the appearance frequency of the keyword candidates that have not been selected, it is possible to evaluate the importance more accurately, and thus it is possible to extract the keywords with high accuracy.

[0081]

[Brief description of drawings]

FIG. 1 is a block diagram showing the overall configuration of a first embodiment of a keyword extraction device of the present invention.

FIG. 2 is a flowchart showing an example of the operation of the expression extraction unit 3.

FIG. 3 is an explanatory diagram of an example of a morphological analysis result.

FIG. 4 is an explanatory diagram of an example of extracted expressions.

FIG. 5 is an explanatory diagram of an example of an expression extraction rule.

FIG. 6 is a flowchart showing an example of a relation extracting operation of a relation extracting unit 4.

FIG. 7 is an explanatory diagram of an example of a relationship estimation rule.

FIG. 8 is an explanatory diagram of an example of a relation expression.

FIG. 9 is a flowchart showing an operation of the relation expression selection unit 5.

FIG. 10 is an explanatory diagram of an example of a relational expression composed of the same words.

FIG. 11 is an explanatory diagram of an example of information included in a relation symbol.

FIG. 12 is a block diagram showing an overall configuration of a second exemplary embodiment of the present invention.

FIG. 13 is a flowchart showing an example of the operation of the relation expression selection / evaluation unit 8.

FIG. 14 is an explanatory diagram of another example of relation expressions composed of the same word.

FIG. 15 is a block diagram showing a third embodiment of the keyword extracting device of the present invention.

16 is an explanatory diagram of an example of keyword candidates extracted by the word group extraction unit 13. FIG.

FIG. 17 is a flowchart showing an example of an operation of the importance evaluation section 10 in the third exemplary embodiment of the keyword extracting device of the present invention.

FIG. 18 is an explanatory diagram of an example of keyword candidates obtained by prefix matching and suffix matching.

[Explanation of symbols]

1 ... Keyword extraction device, 2 ... Data input unit, 3 ... Expression extraction unit, 4 ... Relationship extraction unit, 5 ... Relation expression selection unit, 6 ... Storage unit, 7 ... Frequency counting unit, 8 ... Relation expression selection / evaluation unit , 9 ... Keyword selection section, 10 ... Importance evaluation section, 11 ... Morphological analysis section, 12 ... Word group extraction section, 1
3 ... Word group extraction unit.

─────────────────────────────────────────────────── ─── Continued Front Page (72) Inventor Juhei Nakagaki, 134, Kobe-cho, Hodogaya-ku, Yokohama-shi, Kanagawa Yokohama Business Park East Tower, Fuji Xerox Co., Ltd. (56) Reference JP-A-2-158872 (JP, A) JP-A-7-244673 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G06F 17/30 210 -419 JISST file (JOIS)

Claims (3)

(57) [Claims] 1. A keyword extracting device for extracting a keyword from a text, between an expression extracting means for extracting an expression composed of a plurality of words and phrases related to each other from the text, and a word or phrase constituting the expression extracted by the expression extracting means. Relationship extracting means for estimating the relationship between the relation information and the relation information representing the relationship and outputting a relation expression consisting of a plurality of words and phrases, and the relation information composed of the same words from the relation information and the relation expression output by the relationship extracting means are mutually related. A keyword extracting device comprising relation expression selecting means for extracting a relation expression that does not contradict and selecting a relation expression having relation information having the lowest abstraction as a keyword candidate. 2. A keyword extracting device for extracting a keyword from a text, between an expression extracting means for extracting an expression composed of a plurality of words and phrases related to each other from the text, and a word or phrase constituting the expression extracted by the expression extracting means. Relationship extracting means for estimating the relationship of the relation expression and outputting the relation information representing the relationship and the relation expression consisting of a plurality of words, the frequency counting means for counting the appearance frequency of the relation expression output from the relation extracting means, and the relation extraction. From the relation information and the relation expression output from the means, a relation expression composed of the same words and phrases in which the relation information does not conflict with each other is extracted, and the relation expression having the relation information with the lowest abstraction is selected and selected as a keyword candidate. Not relayed Relation expression and means for evaluating the importance of the relation expression selected by using the frequency of appearance of the relation expression selected as a candidate and the relation expression evaluation means, and based on the evaluation result of the relation expression evaluation means. A keyword extracting device comprising a keyword selecting means for selecting a keyword. 3. A keyword extracting method for extracting a keyword from a text, wherein the expression extracting means extracts an expression consisting of a plurality of words and phrases related to each other from the text, and the relationship extracting means extracts the expression extracted by the expression extracting means. A step of estimating a relation between the constituent phrases and outputting relation information representing the relation and a relation expression consisting of a plurality of terms; the same phrase from the relation information and the relation expression output from the relation extracting means by the relation expression selecting means A keyword extraction method comprising the step of extracting a relational expression composed of the relational information that is not inconsistent with each other and selecting the relational expression having the relational information with the lowest abstraction as a keyword candidate.