JP6230190B2 - Important word extraction device and program - Google Patents

Description

  The present invention relates to a keyword extraction device and a program.

  The task of extracting important words in sentences is important. For example, an important word extracted from a long sentence can be used for estimating what the topic of the sentence is or estimating a clue when automatically summarizing the sentence.

  Conventionally widely used methods for extracting important words from sentences use weight settings for each word obtained by TF-IDF or okapi BM25. These methods give a high weight to "words that frequently appear in sentences where important words are to be extracted and do not appear so much in other sentences." Is widely used because

  In addition, the technology for extracting important words calculates the degree of versatility that expresses "how widely used" from the appearance frequency obtained by aggregating words that appeared within a certain period, and the degree of versatility is low. In addition, some words that appear frequently in the target sentence are important words (see, for example, Patent Document 1). As another technique for extracting an important word, there is a technique in which a word that appears in common in many utterances is used as a clue, and a word that is frequently used at a time close to the word is used as an important word (for example, Patent Document 2 reference).

JP 2011-70291 A JP 2011-248409 A

  In TF-IDF and okapi BM25, the word weight is set based only on the number of appearances in the sentence, and the context and the meaning of the whole sentence are not used. Therefore, it is impossible to extract important words that match the context. The technique of Patent Document 1 also extracts important words using only the appearance frequency, and does not consider the meaning of sentences or words. The technique of Patent Document 2 does not simply use the number of appearances, but still does not use the meaning of sentences or words.

  The present invention has been made in consideration of such circumstances, and provides an important word extraction device and a program that can extract an important word that matches a context from a sentence.

One aspect of the present invention includes a word extraction unit that extracts a word having a predetermined part of speech from sentence data, a word pair creation unit that creates a word pair composed of the words extracted by the word extraction unit, and the word pair creation unit For each of the word pairs created by, the similarity calculation unit that calculates the similarity between the words constituting the word pair, and the word that includes the word for each word extracted by the word extraction unit The word extraction unit extracts the similarity based on the similarity calculated by the similarity calculation unit for the pair and the score of each of the words calculated by the similarity calculation unit. An important word extraction apparatus comprising: an important word selection unit that selects an important word from the words.
According to this invention, the keyword extraction device extracts a word with a predetermined part of speech from sentence data, and calculates the similarity between words for each word pair created from the extracted word. The keyword extraction device calculates the score of each word based on the similarity of the word pair including the word, and selects the keyword according to the calculated score.
As a result, the important word extraction device extracts important words and topic words using the meaning of the context and sentence in order to extract words that are highly related to other words used in the whole sentence as important words. Can do.

One aspect of the present invention is the key word extraction device described above, wherein the word extraction unit extracts a word whose part of speech is a noun from the sentence data.
According to this invention, the keyword extraction device extracts nouns from the sentence data, and selects nouns that are highly related to other nouns in the sentence from the extracted nouns.
Thereby, the important word extraction device can extract words that are easy to understand as topic words as important words.

One aspect of the present invention is the above-described important word extraction device, wherein the word extraction unit extracts a word whose part of speech is a noun from the sentence data, and among the extracted words, adjacent words in the sentence data And the similarity totaling unit sets the word extracted by the word extracting unit and the compound noun as a score calculation target, and calculates the score of the word of the score calculation target or the compound noun. The target word, or any word that constitutes the compound noun that is the score calculation target, and any word that constitutes the other score calculation target or the other noun calculation target noun Each of the word pairs is calculated based on the similarity calculated by the similarity calculation unit, and the important word selection unit is based on the score calculated by the similarity totaling unit. There selecting key words among the words and the composite nouns, characterized in that.
According to the present invention, the important word extracting device extracts a noun word from a sentence and extracts a compound noun composed of continuous nouns. The keyword extraction device sets the extracted word and compound noun as score calculation targets, and the score calculation target word or compound noun constitutes the score calculation target word or the score calculation target compound noun. Calculated based on the similarity of a word pair consisting of any word and another score calculation target word or any other word constituting a compound noun for another score calculation target. Select a word.
Thereby, the important word extraction apparatus can extract a compound noun as an important word.

One aspect of the present invention is the above-described key word extraction device, wherein the word extraction unit extracts a word with a predetermined part of speech from sentence data, and includes the extracted word according to the number of appearances in the sentence data. The word pair creation unit creates a word pair using the words included in the word group created by the word extraction unit, and the similarity tabulation unit Each of the extracted words is set as a score calculation target, and the score of the word of the score calculation target is calculated by an average of the similarities calculated by the similarity calculating unit for the word pair including the word. And
According to this invention, the keyword extraction device extracts a word having a predetermined part of speech from sentence data, creates a word group including the extracted word according to the number of appearances in the sentence data, and is included in the word group. Calculate the similarity of word pairs created using words. The important word extraction device calculates the score of each word based on the average similarity calculated for the word pair including the word, and selects the important word according to the calculated score.
Thereby, the keyword extraction device can easily determine that a word that appears frequently in a sentence is a keyword.

  In one aspect of the present invention, the computer includes a word extracting unit that extracts a word having a predetermined part of speech from sentence data, a word pair creating unit that creates a word pair including the word extracted by the word extracting unit, and the word For each of the word pairs created by the pair creation means, similarity calculation means for calculating the similarity between the words constituting the word pair, and each of the words extracted by the word extraction means are score calculation targets, Similarity counting means for calculating the score of the word to be scored based on the similarity calculated by the similarity calculating means for the word pair including the word, and the similarity calculating means Important word selection means for selecting an important word from the words extracted by the word extraction means based on the score of each word. Is a program for functioning as a main word extraction device.

  According to the present invention, it is possible to extract important words that match a context from a sentence.

It is a block diagram which shows the structure of the keyword extraction device by the 1st Embodiment of this invention. It is a flowchart which shows the important word extraction process of the important word extraction apparatus by the embodiment. It is a figure which shows the example of the similarity of the word pair which the keyword extraction device by the same embodiment computed. It is a block diagram which shows the structure of the important word extraction apparatus by 2nd Embodiment. It is a flowchart which shows the important word extraction process of the important word extraction apparatus by the embodiment. It is a figure which shows the example of the similarity of the word pair which the keyword extraction device by the same embodiment computed. It is a figure which shows the simulation result of the important word extraction apparatus by 1st Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
An important word in a sentence is a word representing “the content of the entire sentence”. Therefore, the important word extraction device of the present embodiment selects a word close to the semantic center from words appearing in the sentence. In order to obtain a word close to the semantic center, the important word extraction apparatus of the present embodiment extracts all words having a predetermined part-of-speech that can be important words from sentences, and calculates the similarity between two words obtained by combining the extracted words. calculate. The important word extraction apparatus according to the present embodiment calculates an average of similarity for each word, and extracts a word representing the topic of the sentence that is closest to the content of the entire sentence based on the calculated average of the similarity. .

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of the keyword extraction device 1 according to the first embodiment of the present invention, and shows only functional blocks related to the present embodiment. The keyword extraction device 1 is realized by a computer device, and as shown in the figure, a similarity database 10, a sentence input unit 11, a word extraction unit 12, a word pair creation unit 13, a similarity calculation unit 14, and a similarity calculation A unit 15, a ranking unit 16, an important word selection unit 17, and an output unit 18 are configured.

  The similarity database 10 stores information indicating the similarity between words. The sentence input unit 11 receives input of sentence data from which important words are extracted. The word extraction unit 12 extracts a word with a predetermined part of speech from the sentence data. The word pair creation unit 13 creates word pairs for all combinations of words extracted by the word extraction unit 12. The similarity calculation unit 14 calculates the similarity between two words constituting a word pair for each word pair created by the word pair creation unit 13. The degree of similarity is a value that quantitatively represents the degree of similarity between two words. The similarity totaling unit 15 sets each word extracted by the word extracting unit 12 as a score calculation target that quantitatively represents the relevance with other words in the entire sentence. The similarity totaling unit 15 calculates a score of a word for which a score is to be calculated based on the similarity calculated by the similarity calculating unit 14 for a word pair including the word. The ranking unit 16 arranges the words extracted by the word extracting unit 12 in the order of the scores calculated by the similarity totaling unit 15. The important word selection unit 17 selects an important word from the words arranged by the ranking unit 16 according to a predetermined rule. The output unit 18 outputs the important word selected by the important word selection unit 17.

FIG. 2 is a flowchart showing the keyword extraction process of the keyword extraction device 1.
The sentence input unit 11 receives input of sentence data from which important words are extracted (step S11). The sentence indicated by the sentence data may be a single sentence or a sentence composed of a plurality of sentences. Further, the length of the text indicated by the text data is also arbitrary. For example, a sentence of about several hundred characters such as a program summary sentence or a news manuscript longer than that can be used as the sentence indicated by the sentence data.

  The word extraction unit 12 performs a morphological analysis on the text data input in step S11 using a conventional morphological analyzer. As a conventional morphological analyzer, for example, MeCab, ChaSen, or the like can be used. Based on the morphological analysis result, the word extracting unit 12 extracts a word whose part of speech is a noun from the sentence indicated by the sentence data (step S12). The nouns are classified into general nouns (common nouns and proper nouns), suffixes, numbers, etc., but the word extraction unit 12 excludes only general nouns, general nouns and nouns of a predetermined classification, or proper nouns. General nouns may be extracted. When the same noun appears multiple times in the sentence data, the word extraction unit 12 extracts the words in duplicate according to the number of appearances, and creates a word group including the extracted words. When the text data indicates “Tomorrow's weather. Tomorrow is a good weather nationwide.”, The word extraction unit 12 extracts the extracted noun words “Tomorrow”, “Weather”, “Tomorrow”, “Nationwide”. Create a word group consisting of “weather”.

The word pair creation unit 13 creates word pairs of all combinations using the words in the word group created by the word extraction unit 12 (step S13). That is, when a word group consists of words w ₁ , w ₂ ,..., W _n (n is an integer of 2 or more), the word pair creation unit 13 creates word pairs (w _i , w _j ) of all combinations. (I ≠ j, i is an integer from 1 to n, and j is an integer from 1 to n). For example, suppose that the word extraction part 12 created the word group which consists of words "Tomorrow", "Weather", "Tomorrow", "Nationwide", and "Weather" like the above example. The word pair creation unit 13 includes word pairs (tomorrow, weather), (tomorrow, tomorrow), (tomorrow, nationwide), (tomorrow, weather), (weather, tomorrow), (weather, tomorrow), (weather, nationwide). , (Weather, weather), (tomorrow, tomorrow), ... are created.

  The similarity calculation unit 14 calculates the similarity between two words constituting each word pair created by the word pair creation unit 13 (step S14). Here, the similarity calculation unit 14 reads the similarity between two words included in each word pair from the similarity database 10.

  In this embodiment, the context similarity based on Jensen-Shannon Divergence is used as the similarity between two words. The context similarity is a technique for calculating the similarity between words based on the idea that “words that are likely to appear in similar sentences are similar”. When obtaining the context similarity between the word A and the word B, a probability distribution is obtained for the context in which each word appears using learning data, and the difference in probability distribution between the word A and the word B is determined. The result calculated by Jensen-Shannon Divergence is the similarity between these words. This context similarity is in the range of 0 to 1, and the smaller the value, the more similar the words. For details on the context similarity, see, for example, the literature “Keiichi Kazama, Stijn De Saeger, Kentaro Torizawa, Masaki Murata,“ Creating a Large-Scale Similarity List Using Dependent Stochastic Clustering ”, Language Processing Society 15th Annual Convention 2009, p.84-87 ”.

  The similarity calculation unit 14 can use a website accessed via the Internet as the similarity database 10. Examples of available websites are “National Institute of Information and Communications Technology (NICT),“ ALAGIN Language Resource / Speech Resource Site ”, Advanced Language Information Fusion Forum, [online], Internet <URL: https: //alaginrc.nict .go.jp / resources / nictmastar / li-resource-info / li-resource-outline.html> ".

  In the case of the above text data, the similarity calculation unit 14, for example, (tomorrow, weather) similarity = 0.804, (tomorrow, tomorrow) similarity = 0, (tomorrow, nationwide) similarity = 0. .965, (tomorrow, weather) similarity = 0.804,...

The similarity totaling unit 15 totals the score of each word based on the similarity of each word pair calculated by the similarity calculation unit 14 (step S15). Specifically, the similarity totaling unit 15 calculates the score of a word w _i (i is an integer of 1 to n) as a word pair (w _i , w _j ) (j ≠ i and j are integers of 1 or more and n or less), and the similarity is calculated by the average of the similarities calculated by the similarity calculation unit 14. It should be noted that, if the same word appears more than once in the text, the word w _{1 ~} word w _n contain the same word. In this case, the similarity totaling unit 15, the word w _{1 ~} word w For duplicate words from the _n to remove, leaving only one, word w _{1 ~} word w word _n each w _i remaining result of the deletion The score may be calculated as

  In the case of the above sentence data, the similarity totaling unit 15 calculates the score of the word “Tomorrow” for each of the similarities of (Tomorrow, Weather), (Tomorrow, Tomorrow), (Tomorrow, National), (Tomorrow, Weather). An average of 0.643 is calculated. Similarly, the similarity totaling unit 15 calculates the score of the word “weather” as 0.646 and the score of the word “nationwide” as 0.979.

  The ranking unit 16 arranges the words extracted by the word extracting unit 12 in the order of the scores calculated by the similarity totaling unit 15 (step S16). In this embodiment, since the context similarity by Jensen-Shannon Divergence is used as the similarity, the similarity is higher as it is closer to 0. As the similarity, a value obtained by subtracting the context similarity by Jensen-Shannon Divergence from 1 may be used. In this case, the similarity is higher as it is closer to 1.

  The important word selection unit 17 selects an important word from the words arranged by the ranking unit 16 in accordance with a predetermined rule (step S17). The keyword selection unit 17 may select words having a predetermined rank or higher, may select words having a score higher than a predetermined level, and ranks a predetermined percentage of the words extracted from the words extracted by the word extraction unit 12. May be selected in descending order. For example, the important word selection unit 17 selects “words with the highest five scores” and “words from the best score to a score that is 1.2 times the best score”.

  The output unit 18 outputs the important word selected by the important word selection unit 17 (step S18). For example, the output unit 18 causes the important words to be displayed in descending order of score on a display provided in the important word extraction apparatus 1 or a display of a computer device connected to the important word extraction apparatus 1 via a network. Alternatively, the output unit 18 outputs and stores the sentence data or the identification information of the sentence data, the important word extracted from the sentence data, and the score thereof in a storage device provided inside or outside the keyword extraction apparatus 1. Also good.

A specific processing example of the keyword extraction device 1 will be shown.
The sentence input unit 11 receives an input of sentence data “the children experienced traditional mochi using a mortar and a pestle at a nursery school in Yamagata city”. The important word extraction device 1 of the present embodiment uses the similarity between nouns appearing in a sentence to extract the important words in the sentence. Therefore, the word extraction unit 12 extracts the noun words “Yamagata”, “city”, “nursery school”, “schoolchild”, “mortar”, “rice cake”, “mochitsuki”, “experience” from the sentence data. . The similarity calculation unit 14 calculates the similarity of each word pair created by the word pair creation unit 13 using these words.

FIG. 3 shows the similarity of each word pair calculated by the similarity calculator 14. In the figure, word pairs _{(w i, w j) w} i of the word of the vertical axis, _{w j} represents the words of the horizontal axis. In addition, the context similarity represented by Jensen-Shannon Divergence is used as the similarity.

  The similarity totaling unit 15 uses the similarity of each word pair calculated by the similarity calculating unit 14 and obtains the average of similarities with other words for each word. The average similarity is 0.961 for "Yamagata", 0.957 for "city", 0.910 for "nursery school", 0.928 for "children", 0.804 for "mortar", "杵" Is 0.827, “mochi” is 0.875, and “experience” is 0.932. The ranking unit 16 arranges the words in ascending order of the score represented by the average similarity. The order of the words is “Usu”, “Kashiwa”, “Matsutsuki”, “Nursery school”, “Kindergarten”, “Experience”, “City”, “Yamagata”. The important word selection unit 17 selects the three words “mortar”, “杵”, and “mochi” from the top as important words that become topics in the sentence. The output unit 18 outputs the important words “mortar”, “杵”, and “mochi” selected by the important word selection unit 17.

  In addition, when the same word appears twice or more in a sentence, the word extraction part 12 extracts the same word by the appearance number. Therefore, the number of words corresponding to the number of appearances is included on the vertical and horizontal axes in FIG. Since the Jensen-Shannon Divergence between the same words becomes 0, the average value of the similarity is reduced as a result. Therefore, when the same word appears multiple times, the rank of the word tends to increase.

  In the embodiment described above, the similarity calculation unit 14 uses the context similarity as the similarity, but may calculate the similarity between two words by any other similarity calculation method. For example, a similarity calculation method using statistical word co-occurrence can be used. However, in general, words that have the same meaning, such as words used for paraphrasing, rarely co-occur in a sentence, and the similarity calculation method using co-occurrence may not provide a high similarity. . From this point of view, it is preferable to use context similarity.

[Second Embodiment]
In the first embodiment described above, since a score based on the similarity between two words is calculated for each word, an important word can be obtained only in units of one word. Therefore, it is difficult to calculate a score for compound nouns such as “weather information”. Therefore, in this embodiment, compound nouns in sentences can be extracted as important words. Below, 2nd Embodiment is described centering on the difference with 1st Embodiment.

  FIG. 4 is a block diagram showing the configuration of the keyword extraction device 2 according to the second embodiment of the present invention, and shows only functional blocks related to the present embodiment. In this figure, the same parts as those of the keyword extraction device 1 according to the first embodiment shown in FIG. The keyword extraction device 2 is realized by a computer device, and as shown in the figure, a similarity database 10, a sentence input unit 11, a word extraction unit 22, a word pair creation unit 13, a similarity calculation unit 14, and a similarity calculation A unit 25, a ranking unit 26, an important word selection unit 27, and an output unit 18 are configured.

  The word extraction unit 22 extracts noun words from the sentence data. Further, the word extraction unit 22 creates a compound noun (noun phrase) from nouns that are continuous in the sentence. At that time, the word extraction unit 22 also holds information on the words constituting the compound noun. The similarity totaling unit 25 calculates the score of each noun and compound noun extracted from the sentence data based on the similarity between words calculated by the similarity calculation unit 14. The ranking unit 26 arranges the nouns and compound nouns extracted from the sentence data in the order of the scores calculated by the similarity totaling unit 25. The important word selection unit 27 selects an important word from the nouns and compound nouns arranged by the ranking unit 26 according to a predetermined rule.

FIG. 5 is a flowchart showing the keyword extraction process of the keyword extraction device 2.
The sentence input unit 11 receives input of sentence data (step S21). The word extraction unit 22 performs morphological analysis on the input sentence data and extracts a word whose part of speech is a noun (step S22). For example, the word extraction unit 22 extracts only general nouns, general nouns and nouns of a predetermined classification, or general nouns excluding proper nouns. When the same word appears in the sentence data a plurality of times, the word extraction unit 22 extracts the word in duplicate according to the number of appearances.

  Next, the word extraction unit 22 creates a compound noun from the nouns continuous in the sentence indicated by the sentence data based on the morphological analysis result in step S22 (step S23). Specifically, the word extraction unit 22 creates a compound noun in which a plurality of general nouns continue, or a compound noun in which a general noun, suffix, and number continue. When the same compound noun appears in the sentence data a plurality of times, the word extraction unit 22 creates the compound noun in duplicate according to the number of appearances. The word extraction unit 22 creates a word group composed of the words extracted in step S22 and the words not extracted in step S22 among the words constituting the compound noun created in step S23. Further, the word extraction unit 22 combines the word extracted in step S22 and the created compound noun as an important word candidate. In addition, the word extraction part 22 excludes the word which is not used independently in a sentence among the words which comprise a compound noun from an important word candidate.

  For example, if the sentence data indicates “The chef at the restaurant will teach you how to take the soup”, in step S22, the word extraction unit 22 will use the common noun words “dashi”, “restaurant”, “cooking”. ”,“ Transfer ”. In step S <b> 22, the word extraction unit 22 creates a compound noun “cooker” composed of the continuous general noun “cook” and the suffix “human”. The word extraction unit 22 extracts the word “dashi”, “restaurant”, “cooking”, “transmission” extracted in step S22 and the compound noun “cooker” created in step S23 in step S22. A word group consisting of the words “people” that were not created is created. Further, the word extraction unit 22 combines the word extracted in step S22 and the created compound noun as an important word candidate. However, among the words “dashi”, “restaurant”, “cooking”, and “transmission” extracted in step S22, the word “cooking” that constitutes the compound noun “cooker” is not used alone in the sentence. . Therefore, the word extraction unit 22 sets the word “dashi”, “restaurant”, “transmission”, and the compound noun “cooker” obtained by removing “cooking” from the word extracted in step S22 as important word candidates.

  The word pair creation unit 13 creates word pairs of all combinations using the words in the word group created by the word extraction unit 22 in step S23 (step S24). The similarity calculation unit 14 calculates the similarity between two words constituting each word pair created by the word pair creation unit 13 (step S25).

Based on the similarity of each word pair calculated by the similarity calculator 14, the similarity calculator 25 totals the scores of each important word candidate (step S26). When the key word candidates are x ₁ , x ₂ ,..., X _n (n is an integer equal to or greater than 1), the similarity tabulation unit 25 determines each key word candidate x _i (i is an integer equal to or greater than 1 and equal to or less than n) ) Is calculated as follows. When the same important word candidate appears multiple times in the sentence, the important word candidate x ₁ to the important word candidate x _n include the same word or compound noun. In this case, the similarity tabulation unit 25 deletes only one of the important word candidates x ₁ to important word candidates x _n from the important word candidates x ₁ and deletes them. The score may be calculated by using each of the important word candidates x _n as the important word candidate x _i .

The similarity totaling unit 25 calculates a score by averaging the similarities between the keyword candidate x _i and other keyword candidates x _j (j ≠ i, j is an integer of 1 to n). When either one or both of the _keyword candidate x _{i and the} keyword candidate x _j are compound nouns, the similarity tabulation unit 25 selects the words constituting the keyword candidate x _i and the words constituting the keyword candidate x _j of similarity of all combinations word pairs consisting of the best similarity and important word candidate x _i and similarity of important word candidate x _j.

For example, the similarity totaling unit 25 calculates the similarity between the important word candidate x _i “weather” and the important word candidate x _j “heavy rain warning” among the similarities of the word pair (weather, heavy rain) and (weather, warning). Be the better one. Further, for example, the similarity totaling unit 25 calculates the similarity between the keyword candidate x _i “weather information” and the keyword candidate x _j “snow” as the similarity between the word pair (weather, snow) and (information, snow). Let me be the better one. Further, for example, the similarity totaling unit 25 uses the word pair (weather, heavy rain), (weather, warning), (information) for the similarity between the key word candidate x _i “weather information” and the key word candidate x _j “heavy rain warning”. , Heavy rain), (information, warning) similarity is the best similarity.

Alternatively, when the keyword candidate x _i is a compound noun, the similarity tabulation unit 25 obtains an average of similarities with each of the other keyword candidates x _{j for} each word constituting the keyword candidate x _i . Similarity degree collection section 25, of the average degree of similarity calculated for each word constituting the important word candidate x _i, the best value score of important word candidate x _i.

For example, it is assumed that the important word candidate x _i is “weather information”. The similarity totaling unit 25 calculates an average of similarities between the word “weather” constituting the important word candidate x _i “weather information” and each of the important word candidates x _j . If the keyword candidate x _j is a single word, the similarity totaling unit 25 determines the similarity between the word pair composed of the word “weather” and the keyword candidate x _j as the word “weather” and the keyword candidate. Let x _{j be} the similarity. If the important word candidate x _j is a compound noun, the similarity totaling unit 25 is the best among the similarities of word pairs composed of the word “weather” and each word constituting the important word candidate x _j. the value and the word "weather" as the similarity of the important word candidate x _j. Similarly, the similarity totaling unit 25 calculates the average of the similarities between the other words “information” constituting the important word candidate x _i and each of the important word candidates x _j . The similarity totaling unit 25 sets the better one of the averages of similarities calculated for each of the word “weather” and the word “information” as the score of the keyword candidate x _i “weather information”.

  The ranking unit 26 arranges the important word candidates extracted by the word extraction unit 12 in the order of the scores calculated by the similarity totaling unit 25 (step S27). The important word selection unit 27 selects an important word from the important word candidates arranged by the ranking unit 26 in accordance with a predetermined rule (step S28). The output unit 18 outputs the important word selected by the important word selection unit 27 (step S29).

A specific processing example of the keyword extraction device 2 will be shown.
The sentence input unit 11 receives input of sentence data “Next is tomorrow's weather information.” The word extraction unit 22 extracts the words “next”, “tomorrow”, “weather”, and “information”. The word extraction unit 22 creates one compound noun “weather information” from “meteorology” and “information” consecutive in the sentence from these words. The word extraction unit 22 creates a word group composed of the words “next”, “tomorrow”, “weather”, and “information” extracted from the sentence data. Further, the word extraction unit 22 constitutes the compound noun “weather information” from the words extracted from the compound noun “weather information” and sentence data from the word group, and is used alone in the sentence. The words “next” and “tomorrow” excluding the words “meteorology” and “information” that are not used are important word candidates. The similarity calculation unit 14 calculates the similarity of each word pair created by the word pair creation unit 13 using words included in the word group.

  FIG. 6 shows the similarity of each word pair calculated by the similarity calculation unit 14. In the figure, the context calculated by the similarity calculation unit 14 for each word pair created by the word pair creation unit 13 using the words “next”, “tomorrow”, “weather”, and “information” included in the word group. The similarity is shown. In the figure, the degree of similarity is not calculated for word pairs constituting the same compound noun.

  The similarity totaling unit 25 obtains the score of the important word candidate “next”, the similarity between the important word candidate “next” and the important word candidate “Asu”, and the important word candidate “next” and the important word candidate “weather information”. The average of the similarity is calculated. The similarity totaling unit 25 sets the similarity between the important word candidate “next” and the important word candidate “ASU” as the similarity “0.676” of the word pair (next, tomorrow) calculated by the similarity calculating unit 14. . The similarity totaling unit 25 sets the similarity between the important word candidate “next” and the important word candidate “weather information” to the similarity “0.965” of the word pair (next, weather) calculated by the similarity calculating unit 14. The word pair (next, information) similarity “0.875” is the better one. The similarity totaling unit 25 sets the score of the important word candidate “next”, the similarity “0.676” between the important word candidate “next” and the important word candidate “Asu”, the important word candidate “next”, and the important word candidate. “0.776” is calculated from the average of the similarity “0.875” of “weather information”.

  The similarity totaling unit 25 calculates the score of the important word candidate “As”, the similarity between the important word candidate “As” and the important word candidate “Next”, and the important word candidate “As” and the important word candidate “Weather”. It is calculated by the average of the similarity of “information”. The similarity totaling unit 25 sets the similarity between the important word candidate “ASU” and the important word candidate “NEXT” as the similarity “0.676” of the word pair (ASU, NEXT) calculated by the similarity calculating unit 14. . The similarity totaling unit 25 calculates the similarity between the keyword candidate “ASU” and the keyword candidate “weather information” as the similarity “0.918” of the word pair (tomorrow, weather) calculated by the similarity calculation unit 14. , The better of the word pair (tomorrow, information) similarity “0.990”. The similarity totaling unit 25 sets the score of the important word candidate “As”, the similarity “0.676” between the important word candidate “As” and the important word candidate “Next”, and the important word candidate “As” and the important word candidate. “0.797” is calculated from the average of the similarity “0.918” of “weather information”.

  The similarity totaling unit 25 calculates the score of the important word candidate “weather information”, the similarity between the important word candidate “weather information” and the important word candidate “next”, and the important word candidate “weather information” and the important word. Calculated based on the average of the similarities of the candidate “Asu”. The similarity totaling unit 25 sets the similarity between the important word candidate “weather information” and the important word candidate “next” as the similarity “0.965” of the word pair (weather, next) calculated by the similarity calculating unit 14. , The better of the word pair (information, next) similarity “0.875”. The similarity totaling unit 25 calculates the similarity between the important word candidate “weather information” and the important word candidate “ASSU” as the similarity “0.918” of the word pair (weather, tomorrow) calculated by the similarity calculating unit 14. The word pair (information, tomorrow) similarity “0.990” is the better one. The similarity totaling unit 25 sets the score of the important word candidate “weather information”, the similarity “0.875” between the important word candidate “weather information” and the important word candidate “next”, and the important word candidate “weather information”. It is calculated as “0.897” from the average of the similarity “0.918” of the keyword candidate “ASU”.

  Alternatively, the similarity totaling unit 25 calculates the score of the important word candidate “weather information”, the average of the similarity of the word pair (weather, next) and the similarity of (weather, tomorrow), and the word pair (information, next). It is good also as the better one of the similarity of (information, tomorrow) and the average of similarity. The similarity totaling unit 25 calculates the average similarity of the word “weather” constituting the keyword candidate “weather information”, the similarity “0.965” of the word pair (weather, next), and the word pair (weather) , Tomorrow) is calculated as “0.942” from the average of the similarity “0.918”. The similarity totaling unit 25 calculates the average similarity between the words “information” constituting the keyword candidate “weather information”, the word pair (information, next) similarity “0.875”, and the word pair. From the average of the similarity (0.990) of (information, tomorrow), “0.933” is calculated. The similarity totaling unit 25 sets the score of the keyword candidate “weather information” to “0.942”, which is the better of the average similarity of the word “weather” and the average similarity of the word “information”. To do.

The ranking unit 26 arranges important word candidates in the order of “next”, “tomorrow”, and “weather information” based on the score calculated by the similarity totaling unit 25. The important word selection unit 27 selects an important word from the important word candidates arranged by the ranking unit 26, and the output unit 18 outputs the important word selected by the important word selection unit 27.
In the above description, short sentence data is input to explain the processing. However, it is considered that inputting a little longer sentence data improves the accuracy of extraction.

FIG. 7 is a diagram illustrating an evaluation experiment result by the keyword extraction device 1 according to the first embodiment. In the evaluation experiment, sentence data indicating a search word and a program summary sentence for each of 100 programs was used as evaluation data. For each program, three workers extracted keywords from the text data within five words, and the union of keywords selected by each of the three workers was used as correct data for important words. Note that the same words as the search words were excluded during the evaluation.
The figure shows the probability that the top n important words extracted by the keyword extraction device 1 and the top n important words extracted using the conventional okapi BM25 are included in the keywords of the correct data. Yes. As shown in the figure, the probability that an important word extracted by the important word extraction device 1 of the present embodiment is included in correct data is higher than that of the prior art, particularly for a word extracted at a higher level, and good results are obtained. It was.

  According to the embodiment described above, the important word extraction device extracts nouns that are highly related to other nouns used in the entire sentence as important words. A noun highly related to other nouns in a sentence is a semantically-centric word that well represents the meaning of the entire sentence. In other words, a noun that is highly related to other nouns in a sentence is a word that is highly connected to the semantic content in the flow of the sentence and is in context. Therefore, the important word extraction device can extract the important words and the topic words using the context and meaning in the sentence, not the probabilistic statistics of the appearance frequency of simple words. For example, in a conventional apparatus that performs a program search, when presenting program information obtained as a result of a search by a keyword input by a user, the important word extraction apparatus of the present embodiment extracts the important words extracted from the program outline. It is possible to present it together. By presenting this important word, the contents of the program obtained as a result of the search can be easily communicated to the user. Also, according to the embodiment described above, the keyword extraction device does not need to collect a large amount of documents of similar domains, unlike the case of weighting by TF-IDF.

  As described above, since the important word extraction apparatus according to the present embodiment extracts important words using the meaning of words used in a sentence, the context is improved in performance compared to the conventional technique using the appearance frequency of words. Can be extracted. In the case of TF-IDF that has been used in the past, it is necessary to collect statistics of similar styles, and in this embodiment, learning data for calculating the similarity between words is used. There is no need to “gather a lot of sentences with similar styles” for searching.

  The above-described important word extraction devices 1 and 2 have a computer system therein. The operation process of the keyword extraction devices 1 and 2 is stored in a computer-readable recording medium in the form of a program, and the above-described processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a portable dictionary such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a conceptual dictionary storage unit such as a hard disk built in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1, 2 Important word extraction apparatus 10 Similarity database 11 Input part 12, 22 Word extraction part 13 Word pair creation part 14 Similarity calculation part 15, 25 Similarity totaling part 16, 26 Ranking part 17, 27 Important word selection part 18 Output section

Claims (2)

Extracting a word of a noun from sentence data, and a word extracting unit in which the adjacent words in the sentence data are extracted as one compound noun among the extracted words;
A word pair creation unit that creates a word pair composed of the words extracted by the word extraction unit;
For each of the word pairs created by the word pair creation unit, a similarity calculation unit that calculates the similarity between the words constituting the word pair;
When each of the words extracted by the word extraction unit is a score calculation target and the same word appears multiple times in the document data, only one word is duplicated among the words of the score calculation target. A similarity totaling unit that calculates the score of the word to be score-calculated, which is left as a result of deletion, by calculating the average of the similarities calculated by the similarity calculating unit for the word pair including the word; ,
An important word selection unit that selects an important word from the words extracted by the word extraction unit based on the scores of the words calculated by the similarity totalization unit;
Equipped with a,
The similarity totaling unit, the word extracted by the word extraction unit and the compound noun as a score calculation target, the word of the score calculation target or the score of the compound noun, the word of the score calculation target, or For each of the word pairs consisting of any word that constitutes the compound noun that is the score calculation target, and any word that constitutes the other noun calculation target word or the other noun calculation target noun Calculated based on the similarity calculated by the similarity calculation unit,
The important word selection unit selects an important word from the word and the compound noun based on the score calculated by the similarity tabulation unit,
An important word extraction device characterized by that. A program for causing a computer to function as the important word extracting device according to claim 1 .

Images