JP2015088064A - Text summarization device, text summarization method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a summary in alignment with a meaning of an entire text to be output.SOLUTION: Analysis means 22 performs a sentence division and a morphological analysis on an input text. Input-text-concept-vector generation means 24 generates an input-text concept vector representing a meaning of the input text while referring to a word concept base 20. Partial-sentence extraction means 26 extracts a partial sentence set that is a set of clauses in a sentence from the input text. Partial-sentence ranking means 28 generates a partial-sentence concept vector representing a meaning of each of partial sentences extracted by the partial-sentence extraction means 26 while referring to the word concept base 20, calculates a similarity between the input-text concept vector and the partial-sentence concept vector, and outputs partial sentences high in the similarity as a summary.

Description

  The present invention relates to a text summarization apparatus, method, and program for outputting a summary sentence that accurately represents the contents of an entire text.

  As a text summarization technique, there is a technique described in Non-Patent Document 1. In the method of Non-Patent Document 1, importance is given to each sentence in the text, and a sentence group is made so that different words appear as much as possible so that the sum of importance is as high as possible, and so that readability is as high as possible. To present. Here, the importance of the sentence is the sum of the importance of the constituent words of the sentence, and the appearance frequency and IDF are assumed as the importance of the constituent words.

Hitoshi Nishikawa, Takaaki Hasegawa, Yoshihiro Matsuo, Genichiro Kikui, “Summary Model for Evaluation Documents that Simultaneously Select and Order Sentences”, 2013, Journal of the Japanese Society for Artificial Intelligence, Vol. 28, No. 1, H., p.88-99

  The first problem of the conventional method will be described. For example, assume that the text to be summarized is the content of FIG. In the conventional method, the first sentence “Mr. A sang while singing sanma in the tent” has the largest number of constituent words and the highest importance, so the first sentence is output as a summary. .

  However, the second sentence says that the sardines have been boiled, the third sentence says that the fish has been cooked, and the whole text says that the fish has been cooked. However, the output sentence states that the saury, which is a part of the fish, has been burned, and the contents deviate from the meaning of the whole text.

  In this way, in the conventional method, since the importance of the sentence is calculated by the sum of the importance of the constituent words, the extracted sentence does not necessarily conform to the meaning of the entire text to be summarized. There is a problem that it may be shifted from the meaning of the whole text.

  A first object of the present invention is to output a summary in accordance with the meaning of the entire text from the text to be summarized.

  The second problem of the conventional method will be described. For example, assume that the text to be summarized has the contents shown in FIG. In the conventional method, the first sentence “Sanma baked” has the highest importance, so the first sentence is output as a summary.

  However, the second sentence states that the sardines have been boiled, and the entire text describes the cooking of fish. However, the output sentence states that the saury that was part of the fish has been burned, and represents only a part of the content of the text.

  As described above, the conventional method is a method of extracting a sentence having high importance, but the extracted sentence represents only a part of the content of the text to be summarized, and the entire content of the text to be summarized is completely There is a problem that it is not covered.

  The second object of the present invention is to output a summary that covers the entire contents of the text to be summarized.

  A third problem of the conventional method will be described. For example, assume that the text to be summarized has the contents shown in FIG. In the conventional method, the first sentence “Mr. A sang while sang sanma in a tent” and the second sentence “Mr. B boiled autumn sardines.” Since the degree of importance increases, the first sentence and the second sentence are output as a summary.

  However, the 1st and 2nd sentences describe cooking fish, and the 3rd and 4th sentences describe growing fruit, and the text contains a plurality of different topics. However, the output sentence is only related to the topic of cooking fish, and only a part of the topic in the text is represented.

  Thus, in the conventional method, there is a problem that the extracted sentence may not cover all the topics in the summary target text.

  A third object of the present invention is to output a summary that covers all the topics in the text from the text to be summarized.

  In order to achieve the above object, a text summarizing apparatus according to a first invention comprises an analysis means for performing sentence division and morphological analysis on an input text, and a word and a word concept vector representing the meaning of the word. A word concept base that stores a set of pairs; an input text concept vector generating means that generates an input text concept vector representing the meaning of the input text by referring to the word concept base; and A partial sentence extracting means for extracting a set of partial sentences, which is a set of clauses, and the meaning of the partial sentence by referring to the word concept base for each partial sentence extracted by the partial sentence extracting means A partial sentence concept vector is generated, the similarity between the input text concept vector and the partial sentence concept vector is calculated, and the partial sentence with the high similarity is output as a summary sentence. And having a partial sentence ranking means, a.

  A text summarizing method according to a second invention comprises an analysis means, a word concept base storing a set of pairs of a word and a word concept vector representing the meaning of the word, an input text concept vector generation means, a partial sentence extraction means, and A text summarization method in a text summarizing apparatus including a partial sentence ranking means, wherein the analysis means performs sentence division and morphological analysis on an input text, and the input text concept vector generation means generates the word. A step of generating an input text concept vector representing the meaning of the input text by referring to the concept base, and a partial sentence set that is a set of clauses in the sentence is extracted from the input text by the partial sentence extraction means Extracted by the partial sentence extracting means by the partial sentence ranking means. For each partial sentence, a partial sentence concept vector representing the meaning of the partial sentence is generated by referring to the word concept base, and a similarity between the input text concept vector and the partial sentence concept vector is calculated, And outputting the partial sentence with high similarity as a summary sentence.

    A text summarization apparatus according to a third aspect of the present invention is a word concept that stores a set of pairs of an analysis unit that performs sentence division and morphological analysis on an input text, and a word concept vector that represents the meaning of the word. A base, and an input text concept vector generating means for generating an input text concept vector representing the meaning of the input text by referring to the word concept base; and a set of independent parts obtained from clauses in the input text is clustered A self-supporting part clustering means, and for each self-supporting part cluster obtained by the self-supporting part clustering means, a superordinate concept word acquiring means for acquiring a word that is a superordinate concept of the self-supporting part belonging to the self-supporting part cluster, and the input A partial sentence extracting means for extracting a set of partial sentences, which is a set of clauses in the sentence, from the text, and the partial sentence extracting means For each extracted partial sentence, a converted partial sentence obtained by converting each independent part in the partial sentence into a higher-order concept word of the independent part cluster to which the independent part belongs, obtained by the higher-order concept word acquisition unit. The generated partial sentence conversion means and the converted partial sentence concept vector representing the meaning of the converted partial sentence by referring to the word concept base for each converted partial sentence obtained by the partial sentence conversion means And a converted partial sentence ranking means for calculating a similarity between the input text concept vector and the converted partial sentence concept vector, and outputting the converted partial sentence having a high similarity as a summary sentence. It is characterized by.

  A text summarization method according to a fourth invention comprises an analysis means, a word concept base storing a set of pairs of words and word concept vectors representing the meaning of the word, input text concept vector generation means, self-supporting part clustering means, host A text summarization method in a text summarization apparatus including a concept word acquisition means, a partial sentence extraction means, a partial sentence conversion means, and a post-conversion partial sentence ranking means, wherein the analysis means divides a sentence into input text, and A step of performing morphological analysis; a step of generating an input text concept vector representing the meaning of the input text by referring to the word concept base by the input text concept vector generating unit; Clustering a set of freestanding parts obtained from clauses in the input text Obtaining a word that is a superordinate concept of an independent part belonging to the independent part cluster for each independent part cluster obtained by the independent part clustering means by the superordinate concept word obtaining means; A step of extracting a set of partial sentences, which is a set of clauses in the sentence, from the input text by the partial sentence extracting means; and for each partial sentence extracted by the partial sentence extracting means by the partial sentence converting means, Generating a converted partial sentence obtained by converting each independent part in the partial sentence into a high-order concept word of the independent part cluster to which the independent part belongs, obtained by the higher-order concept word acquisition unit; and the converted part The sentence ranking means refers to the word concept base for each converted partial sentence obtained by the partial sentence conversion means, thereby changing the change. A converted partial sentence concept vector representing the meaning of the subsequent partial sentence is generated, a similarity between the input text concept vector and the converted partial sentence concept vector is calculated, and the converted partial sentence having a high similarity is used as a summary sentence And a step of outputting.

  In the first and second inventions, the method further includes partial text clustering means for clustering a set of partial texts obtained from the input document, and for each cluster obtained by the partial text clustering means, the partial text constituting the cluster. As a set of the input text, a summary sentence can be output by performing each process by the analyzing means, the input text concept vector generating means, the partial sentence extracting means, and the partial sentence ranking means.

  In the third and fourth inventions, the method further comprises partial text clustering means for clustering a set of partial texts obtained from the input document, and for each cluster obtained by the partial text clustering means, the partial text constituting the cluster. As the input text, the analysis means, the input text concept vector generation means, the self-supporting part clustering means, the superordinate concept word acquisition means, the partial sentence extraction means, the partial sentence conversion means, and the post-conversion part A summary sentence can be output by performing each process by the sentence ranking means.

  The program according to the present invention is a program for causing a computer to function as each unit of the text summarizing device according to the present invention.

  According to the text summarizing apparatus, method, and program of the present invention, it is possible to output a summary in accordance with the meaning of the entire text.

It is a figure for demonstrating a high-order concept word. It is a block diagram which shows the example of 1 structure of the text summarizing apparatus based on the 1st Embodiment of this invention. It is a figure which shows an example of the set stored in a word concept base. It is a figure for demonstrating the dependency relation of a sentence. It is a flowchart which shows the content of the text summarization processing routine in the text summarizing apparatus based on the 1st Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the text summarizing apparatus based on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the text summarization processing routine in the text summarizing apparatus based on the 2nd Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the text summarizing apparatus based on the 3rd Embodiment of this invention. It is a flowchart which shows the content of the text summary processing routine in the text summarizing apparatus based on the 3rd Embodiment of this invention. It is a figure for demonstrating a prior art.

<Summary of invention>
The configuration of the present invention for solving the first problem of the conventional method has the following effects.

  Since the partial sentence concept vector representing the meaning of the partial sentence output by the present invention has a high similarity to the input text conceptual vector representing the meaning of the input text, the meaning of the partial sentence is the meaning of the input text. Get closer. Therefore, the output partial sentence is a summary in accordance with the meaning of the entire input text.

  Taking the text to be summarized in FIG. 10A as an example, the concept vector of the partial sentence “fish cooked” in the third sentence is output because the similarity to the concept vector of the text is the highest. The This partial sentence is a summary in line with the meaning of the entire text.

  The effect produced by the configuration of the present invention for solving the second problem of the conventional method will be described.

  Here, the basic idea that led to this configuration will be described first.

  Taking the text to be summarized in FIG. 10B as an example, the words “sanma” and “iwashi” in the text are semantically close, and the set of these words has a superordinate concept of “fish”. Words can be associated. If the broader concept word “fish” is used in the summary sentence, the degree of coverage of the summary sentence with respect to the content of the entire text is higher than when the lower order concept words “sanma” and “sardine” are used. Similarly, the words “baked” and “boiled” in the text are semantically close, and a set of these words can be associated with a high-level concept word “cooked”. If you use the broader concept word “cooked” in the summary sentence, the coverage of the summary text with respect to the entire text is higher than when you use the lower concept words “baked” or “boiled”. Become.

Therefore, if a summary sentence “cooked fish” using a high-level concept word in a summary target text is generated, the summary sentence covers the entire contents of the text.
Hereinafter, the effect produced according to the configuration of the present invention will be described.

  The self-supporting part clustering means clusters the self-supporting parts in the text to be summarized, and the superordinate concept word acquiring means associates the word corresponding to the superordinate concept of the self-supporting part belonging to each self-supporting part cluster. In FIG. 1A, as a result of clustering the independent parts in the text to be summarized and associating the higher-order concept words, the higher-order concept word “fish” is found in the cluster composed of independent parts of “Sanma”, “Iwashi”, and “mackerel”. Corresponding, the high-level concept word “cooked” is associated with a cluster consisting of independent groups of “baked” and “boiled”, and higher than the cluster consisting of independent families of “house”, “house” and “home” The concept word “home” corresponds, and the superordinate concept word “music” is associated with a cluster composed of independent groups “classic”, “jazz”, and “pops”.

  Next, it is considered that a high-order concept word of each independent cluster is connected with an attached word to generate a sentence to be a summary sentence. The partial sentence conversion means converts each independent part in the partial sentence in the summary target text into a broader concept word corresponding to the independent part cluster to which the independent part belongs. This is realized by converting the partial sentence. For example, for the partial sentence "Sanma baked at home" in the text to be summarized, the independent parts "Home", "Sanma", and "Baken" in it are changed to "Home", "Fish", and "Cooked" respectively. By converting, the partial sentence is converted into “cooked fish at home”.

  Then, the converted partial sentence ranking means selects a converted partial sentence whose partial sentence concept vector representing the meaning is highly similar to the input text concept vector representing the meaning of the input text. The selected partial sentence is a summary in accordance with the meaning of the entire text, and is a summary that covers the contents of the entire text.

  As described above, the configuration of the present invention can achieve the effect of solving the second problem, but the following configuration changes are also conceivable.

  In the independent part cluster obtained by the independent part clustering means, there is a case where the set of independent parts to which the independent part cluster belongs is a set of synonyms (for example, {sama, sanma, katana, saury}). For such a self-supporting part cluster, the high-order concept word acquisition means acquires the high-order concept word (eg “fish”) of the self-supporting part to which it belongs, and the partial sentence conversion means obtains it by converting it to this high-order concept word. The partial sentence covers the entire contents of the text to be summarized.

  However, it can be said that it is appropriate to use the words of the original self-supporting part as a summary sentence with a high degree of detail of information while maintaining the coverage of the contents. Therefore, for a self-supporting part cluster in which the set of free-standing parts to which it belongs is a set of synonyms, the high-order concept word acquisition means does not acquire the high-order concept word, and the partial sentence conversion means does not acquire the high-order concept word Thus, when the independent cluster to which it belongs is a set of synonyms, it is possible to adopt a configuration in which the conversion process to the broader concept word is not performed. By adopting such a configuration, it is possible to output a summary sentence that covers the entire contents of the text to be summarized and further increases the level of detail of the information as much as possible.

  Taking the text to be summarized in FIG. 1B as an example, for the self-supporting part cluster {sanma, sanma} obtained by the self-supporting part clustering means, since the set of synonyms, For the self-supporting portion cluster {baked, boiled}, the higher-level concept word “cooked” is acquired by the higher-level concept word acquisition unit. In the partial sentence conversion means, for the independent part "Sanma" in the partial sentence "Sanma baked", the independent part cluster to which it belongs is a set of synonyms, so conversion processing to a broader concept word is not performed, and the independent part " For `` baked '', it is converted to the broader concept word `` cooked '' in the independent part cluster to which it belongs, and the converted partial sentence `` cooked sama '' is obtained, and the converted partial sentence ranking means The partial sentence is output as a summary sentence. This converted partial sentence covers the entire contents of the text to be summarized shown in FIG. 1B, and further increases the level of detail of the information as much as possible.

  The effect produced by the configuration of the present invention for solving the third problem of the conventional method will be described.

  A partial text clustering unit clusters a set of partial texts constituting an input document into topic units. Each cluster obtained as a result is a partial text set made up of a single topic. With the partial text set constituting each cluster as an input, the analysis means, the input text concept of the first embodiment to be described later Processing by vector generation means, partial sentence extraction means, and partial sentence ranking means, or analysis means, input text concept vector generation means, independent part clustering means, superordinate concept word acquisition means of the second embodiment described later, A summary sentence for each topic can be output by performing processing by the partial sentence extraction means, the partial sentence conversion means, and the post-conversion partial sentence ranking means, so a summary that covers all the topics in the original input document is output. it can.

  Taking the text to be summarized in FIG. 10C as an example, it is a cluster related to cooking of fish composed of the first and second sentences by clustering a set of sentences in the text by the partial text clustering means. A partial text set, which is a cluster related to the cultivation of fruits composed of the third and fourth sentences, is obtained.

  With each partial text set as an input, the processing by the analysis means, input text concept vector generation means, partial sentence extraction means, and partial sentence ranking means of the first embodiment, or the second embodiment of the present invention It is related to cooking of fish by performing processing by analysis means, input text concept vector generation means, independent part clustering means, superordinate concept word acquisition means, partial sentence extraction means, partial sentence conversion means, and converted partial sentence ranking means. A summary and a summary of fruit cultivation are available.

[First embodiment]
<System configuration>
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 2 is a block diagram showing the text summarizing apparatus 100 according to the first embodiment of the present invention. The text summarization apparatus 100 outputs a summary sentence corresponding to the input text. The text summarization apparatus 100 is composed of a computer including a CPU, a RAM, and a ROM storing a program for executing a text summarization processing routine described later, and is functionally configured as follows. .

  As shown in FIG. 2, the text summarizing apparatus 100 according to the present embodiment includes an input unit 1, a calculation unit 2, and an output unit 3.

  The input unit 1 accepts input of summary input text.

  The computing unit 2 includes a word concept base 20, an analysis unit 22, an input text concept vector generation unit 24, a partial sentence extraction unit 26, and a partial sentence ranking unit 28.

  The word concept base 20 stores a set of pairs of words (content words such as nouns, verbs, and adjectives) and word concept vectors representing the meaning of the words. FIG. 3 shows an example of a set stored in the word concept base 20. The word concept base 20 is generated by, for example, the following reference technique.

References: Katsuto Bessho, Toshiro Uchiyama, Atsushi Uchiyama, Ryoji Kataoka, Masahiro Oku, “Corpus concept-based generation method based on co-occurrence between words and semantic attributes”, Dec. 2008, Transactions of Information Processing Society of Japan, Vol.49 , No.12, p.3997-4006

  The word is registered in the word end form, and when searching in the word concept base 20, the word end form is searched. The concept vector of each word is in the D-dimensional space, and the concept vectors of words that are semantically close are arranged nearby. The similarity between words is calculated by, for example, cosine between corresponding concept vectors.

  The analysis means 22 divides the input text received by the input means 1 and performs morphological analysis on each obtained sentence. When a partial tree is taken as a partial sentence extracted by the partial sentence extraction unit 26, the analysis unit 22 may further perform dependency analysis.

  As an example, when the input text has the contents shown in FIG. 10A, the input text is divided into sentences separated by punctuation by sentence division. By performing morphological analysis and dependency analysis on the second sentence “Mr. B has boiled autumn sardines” in FIG. 10A, as shown in FIG. 4, a morpheme string, a phrase string, and a phrase A dependency relationship between them is obtained. One morpheme includes notation and accompanying information such as part of speech and final form (in the case of a non-use word, the notation is regarded as final form). One clause includes one self-supporting part and 0 to 1 attachment part. Each of the one self-supporting portion and the one attachment portion includes one or more morphemes. The self-supporting part includes at least one self-supporting word, and the appendix consists of one or more appendices.

  In the morpheme constituting the self-supporting part, the morpheme other than the final morpheme may be described, and the final morpheme may be identified as the same self-supporting part by a character string concatenated in a terminal form.

  In this embodiment, in the case of notation of a free standing part, it means a character string written for all constituent morphemes, and in the case of a closing form of a free standing part, it means a character string that takes only the final form. To do. When two clauses are in a dependency relationship, a dependency relationship link is provided between the clauses.

  As shown in FIG. 4 above, between the phrase “Iwashi wo” and the phrase “boiled”, the phrase “Iwashi wo” is related to the phrase “boiled”. Add a dependency-related link with the phrase “Nibuta” as the link destination.

  The input text concept vector generation means 24 refers to the word concept base 20 to generate an input text concept vector representing the meaning of the input text. An example of the generation method will be described. For each document constituting the input text, the concept vector of the content word constituting the self-supporting part in the document is obtained from the word concept base 20, and the sum of the word concept vectors is normalized to length 1, The concept vector of the document is used. A concept vector of the input text is obtained by normalizing the sum of each document concept vector to length 1.

The partial sentence extracting unit 26 extracts a set of partial sentences, which is a set of clauses in the sentence, from the input text.
As an example of a set of partial sentences taken out from each sentence, there is a set consisting only of the sentences. In this case, the summary sentence unit is the sentence unit in the input text.
As another example of a set of partial sentences taken out from each sentence, there is a set of subtrees, which is a set of clauses connected by a dependency relationship, obtained from the result of dependency analysis performed by the analysis means 22. In the dependency analysis result of each sentence, a set of clauses of a connected graph formed by tracing a dependency relationship link an arbitrary number of times in the direction of the dependency origin with each clause as a base point is called a subtree. When the phrase “boiled” in FIG. 4 is used as a base point, the subtrees are “boiled”, “boiled sardines”, “boiled sardines in autumn”, “boiled Mr. B”, “B Mr. Boiled sardines "and" Mr. B boiled autumn sardines "are obtained. When the phrase “Iwashi” is used as a base point, “Iwashi” and “Autumn Iwashi” are obtained as subtrees.

  In the process of the partial sentence extracting unit 26, only partial sentences having the specified number of phrases may be extracted. Alternatively, the language likelihood of each partial sentence may be calculated using morpheme notation, part-of-speech N-gram, etc., and only the partial sentence having a high language likelihood may be extracted.

  The partial sentence ranking means 28 generates a partial sentence concept vector representing the meaning of the partial sentence by referring to the word concept base 20 for each partial sentence extracted by the partial sentence extraction means 26, and the input text concept. The similarity between the vector and the partial sentence concept vector is calculated, and a partial sentence with a high similarity is output as a summary sentence.

An example of the processing method will be described. For each partial sentence, the concept vector of the content word constituting the self-supporting part in the partial sentence is acquired from the word concept base 20, and the sum of the word concept vectors normalized to length 1 is A concept vector.
Another example of the processing method will be described. For each self-supporting part in each partial sentence, the concept vector of the content word constituting the self-supporting part is obtained from the word concept base 20, and the sum of the word concept vectors normalized to length 1 The concept vector. The concept vector of the partial sentence is obtained by normalizing the sum of the concept vectors of the independent parts in the partial sentence to length 1.
The inner product of the input text concept vector and the partial sentence concept vector is calculated as the similarity between the input text and the partial sentence.

  Then, a specified number of partial sentences from those having a high degree of similarity, or a partial sentence having a similarity equal to or higher than a specified threshold is output as a summary sentence. In this case, the sentence that is output after the second sentence is not adopted if it is the same character string as the partial sentence that has been output so far, or has a common clause, or appears in the same sentence. It may be said that.

  The output unit 3 outputs the summary sentence output by the partial sentence ranking unit 28.

<Operation of text summarization device>
Next, the operation of the text summarization apparatus 100 according to the present embodiment will be described. When the input text to be summarized is input to the text summarizing apparatus 100, the input text is received by the input means 1. Then, the text summarization processing routine shown in FIG.

  First, in step S100, the analysis unit 22 divides the input text received by the input unit 1 and performs morphological analysis on each obtained sentence. In some cases, dependency analysis is also performed.

  Next, in step S102, the input text concept vector generation means 24 refers to the word concept base 20 to generate an input text concept vector representing the meaning of the input text.

  In step S104, the partial sentence extraction unit 26 extracts a set of partial sentences, which is a set of clauses in the sentence, from the input text. In some cases, a set of subtrees, which is a set of clauses connected by the dependency relationship, is extracted from the result of the dependency analysis in step S100.

  In step S106, the partial sentence ranking unit 28 generates a partial sentence concept vector representing the meaning of the partial sentence by referring to the word concept base 20 for each partial sentence extracted in step S104. A similarity between the text concept vector and the partial sentence concept vector is calculated, and a partial sentence with a high similarity is output as a summary sentence.

  In step S108, the output means 3 outputs the summary sentence output in step S106 as a result, and the text summarization processing routine ends.

  As described above, according to the first embodiment, the meaning of the input text is expressed by referring to the word concept base that stores a set of pairs of a word and a word concept vector representing the meaning of the word. Generates an input text concept vector, extracts a set of partial sentences, which is a set of clauses in the sentence, from the input text, and refers to the word concept base for each extracted partial sentence, meaning of the partial sentence Is generated, a similarity between the input text concept vector and the partial sentence concept vector is calculated, and a partial sentence with a high similarity is output as a summary sentence.

  Since the partial sentence concept vector representing the meaning of the output partial sentence has a high similarity to the input text concept vector representing the meaning of the input text, the meaning of the partial sentence is close to the meaning of the input text. Therefore, the output partial sentence is a summary in accordance with the meaning of the entire input text.

[Second Embodiment]
Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the self-supporting part in the input text is replaced with a broader concept word. In addition, about the text summarizing apparatus 200 which concerns on 2nd Embodiment, about the same structure as the text summarizing apparatus 100 which concerns on 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

<System configuration>
The text summarization apparatus 200 according to the second embodiment is configured by a computer including a CPU, a RAM, and a ROM, like the text summarization apparatus 100 according to the first embodiment. Functionally, this computer can be represented by a configuration including an input unit 1, a calculation unit 2, and an output unit 3, as shown in FIG.

  The computing means 2 includes a word concept base 20, an analysis means 22, an input text concept vector generation means 24, a thesaurus database 220, a language model database 222, a self-supporting part clustering means 224, and a superordinate concept word acquisition means 226. , Partial sentence extraction means 26, partial sentence conversion means 228, and post-conversion partial sentence ranking means 230.

  The thesaurus database 220 stores the upper / lower relationship of words in advance.

  The language model database 222 stores N-gram data acquired from the language corpus.

  The independent part clustering means 224 clusters the set of independent parts obtained from the clauses in the input text. The elements of the set of self-supporting parts here may be considered as different elements if they have the same end form but different appearance positions in the text, and the end form is the same even if the appearance position in the text is different. There is also an idea to consider it as the same element.

  For example, for each independent part in the input text, the independent part clustering means 224 obtains the concept vector of the content word constituting the independent part from the word concept base 20 and sets the sum of the word concept vectors to the length 1 The concept vector of the self-supporting part is normalized to. Clustering of a set of independent parts in the input text is performed by clustering of a corresponding independent part concept vector set. Clustering is performed by, for example, the k-means method.

Clustering generates a set {C _i } of self-supporting part clusters C _i .

  The superordinate concept word acquisition unit 226 acquires, for each independent part cluster obtained by the independent part clustering means 224, a word that is a superordinate concept of the independent part belonging to the independent part cluster.

  Specifically, the high-order concept word acquisition unit 226 searches the thesaurus 220 for each self-supporting part cluster in the self-supporting part belonging to the self-supporting part cluster, and acquires the high-order concept word. In general, there are a plurality of higher-order concept words to be acquired, and a score can be given to each acquired higher-order concept word according to the number of corresponding independent parts. Moreover, you may acquire the common high-order concept word of the independent part which belongs to an independent part cluster. Also in this case, a score can be given to the higher-order concept word according to the distance between the higher-order concept word and each independent part on the thesaurus.

  In addition, as another method of acquiring the high-level concept word, a method using an arithmetic operation using a word concept vector in the word concept base 20 is also conceivable. Also in this case, there are generally a plurality of high-order concept words, and a score can be given to each acquired high-order concept word.

In this way, a set {(u _ij , s _ij )} of a set of a corresponding j-th higher-order concept word u _ij and its score s _ij is derived for the i-th independent cluster C _i .

  In addition, when the self-supporting part cluster is a set of synonyms, the high-order concept word of the self-supporting part cluster may not be acquired. As a method of determining whether or not the independent part cluster is a set of synonyms, a method of determining whether the independent words belonging to each other are synonymous with reference to information such as the thesaurus 220 or a word dictionary can be considered.

  As another method, in the process of the self-supporting part clustering means 224, the concept vector of the self-supporting part cluster (the center of gravity of the concept vector of the self-supporting part to which it belongs, or the center of gravity normalized to length 1) is obtained, and self-supporting The standard deviation, which is the square root of the mean square of the distance between the part cluster concept vector and the concept vector of the independent part to which it belongs, is calculated, and if the standard deviation is below a certain threshold, the concept vector of the independent part to which it belongs is Since they are very close together, a method of determining a set of synonyms is also conceivable.

  For each partial sentence extracted by the partial sentence extracting unit 26, the partial sentence converting unit 228 obtains each independent part in the partial sentence obtained by the high-order concept word acquiring unit 226, and the independent part to which the independent part belongs. Generates a converted partial sentence converted into a broader concept word of the cluster. In the present embodiment, a case will be described in which the partial sentence extraction unit 26 takes a set of partial trees.

Let {(w _k , h _k )} be the sequence of clauses in the target subtree. Here, w _k and h _k are a self-supporting part and an appending part of the k-th clause in the subtree, respectively. A set of dependency relationships is assumed to be {(k _r0 , k _r1 )}. Here, k _r0 and k _r1 are a source clause number and a destination clause number of the r-th dependency relationship, respectively.

The i-th independent cluster to which the independent part w _k belongs

Jth broader term

In which the notation is grammatically connectable to the appendix h _k and the set of scores

Take. Whether it can be connected grammatically is a high-level concept word

Of whether the last morpheme word class P ₀ and the part of speech P ₁ of the first morpheme appendages h _k can be connected, it can be determined by referring to the grammar rule to be used by the morphological analysis or the like. Alternatively, it is possible to adopt a method of determining that connection is possible when a value represented by the following expression acquired from a language corpus and stored in the language model database 222 is equal to or greater than a certain threshold value.

(The number of phrases in which the part of speech of the final morpheme of the self-supporting part is P ₀ and the part of speech of the _first morpheme of the appendix is P ₁ ) /
(The number of phrases in which the part of speech of the final morpheme of the self-supporting part is P ₀ )

Here, the notation different at appendage h _{k 'that} are equivalent to the appendages h _k,

If you can connect with the notation of

And the set of scores in advance, including the process to be described later in the process of partial text conversion unit 228, broad term free-standing unit w _k is

If it is decided, h _k may be converted to h _k ′ at the time of partial sentence conversion.

  Independent cluster

Is a set of synonyms, a set {(w _k , s _k )} consisting only of a set of the independent part w _k and its score s _k (the maximum value of the independent part can be taken) is taken. You may do it. Hereinafter, for this set as well, for convenience

It substitutes with the way of writing.

Next, for each independent part w _k

Then, the one having the maximum score V of the following subtree is obtained.

  here

Is the score of the r-th dependency relationship, the k _r0- th clause

To k _r1 th clause

It is a dependency relationship score. This score is the following value acquired from the language corpus stored in the language model database 222.

  A method is also conceivable in which the calculation is performed by replacing the independent part of the relation source or the relation destination with the semantic category (described in the word dictionary) to which the relation belongs.

  Score V of the entire subtree is maximized

Can be efficiently obtained by the Viterbi algorithm.
The language likelihood of the target subtree may be calculated using morpheme notation, part-of-speech N-gram, and the like, and the language likelihood may be added to the score V of the subtree.

  As described above, the target subtree is a phrase converted from a self-supporting part.

Is converted to
If the partial sentence is not a partial tree having a dependency structure, the expression for calculating the score V may be one in which the terms relating to [Equation 11] to [Equation 12] are eliminated.

  The post-conversion partial sentence ranking unit 230 refers to the word concept base 20 for each post-conversion partial sentence obtained by the partial sentence conversion unit 228, thereby expressing the post-conversion partial sentence concept. A vector is generated, the similarity between the input text concept vector and the converted partial sentence concept vector is calculated, and the converted partial sentence having a high similarity is output as a summary sentence. The processing content of the post-conversion partial sentence ranking unit 230 is the same as the processing content of the partial sentence ranking unit 28 of the first embodiment, except that the processing target is a post-conversion partial sentence.

<Operation of text summarization device>
Next, the operation of the text summarizing apparatus 200 according to this embodiment will be described. When the input text to be summarized is input to the text summarizing apparatus 200, the input means 1 receives the input text. Then, the text summarization apparatus 200 executes a text summarization processing routine shown in FIG. In addition, about the process similar to 1st Embodiment, the same code is attached and description is abbreviate | omitted.

  In step S <b> 200, the independent part clustering means 224 clusters the set of independent parts obtained from the clauses in the input text.

  Next, in step S202, the high-order concept word acquisition unit 226 searches for the thesaurus 220 for each self-supporting part cluster obtained in step S200 in the self-supporting part belonging to the self-supporting part cluster. get.

In step S104, the partial sentence extraction unit 26 extracts a set of partial sentences, which is a set of clauses in the sentence, from the input text. In some cases, a set of subtrees, which is a set of clauses connected by the dependency relationship, is extracted from the result of the dependency analysis in step S100.

  In step S204, for each partial sentence extracted in step S104 by the partial sentence conversion means 228, each independent part in the partial sentence is obtained as the independent part cluster to which the independent part belongs obtained in step S202. Generates a converted partial sentence converted to a broader concept word for.

  In addition, since the other structure and effect | action of the text summarizing apparatus 200 which concern on 2nd Embodiment are the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, according to the second embodiment, the set of independent parts obtained from the clauses in the input text is clustered, and the independent part belonging to the independent part cluster is obtained for each obtained independent part cluster. The word which becomes the superordinate concept of is obtained, the converted partial sentence converted into the superordinate concept word of the independent part cluster to which the independent part belongs is generated, and the word concept base is referred to for each obtained converted partial sentence As a result, a converted partial sentence concept vector representing the meaning of the converted partial sentence is generated, the similarity between the input text concept vector and the converted partial sentence concept vector is calculated, and the partial sentence having a high similarity is used as a summary sentence. By outputting, it is possible to output a summary that covers the entire contents of the text.

[Third embodiment]
Next, a third embodiment will be described. The third embodiment is different from the first and second embodiments in that a summary sentence is generated for each topic in the input document. In addition, about the text summarizing apparatus 300 which concerns on 3rd Embodiment, about the structure same as the text summarizing apparatus 100 which concerns on 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

<System configuration>
Similar to the text summarization device 100 according to the first embodiment, the text summarization device 300 according to the third embodiment is configured by a computer including a CPU, a RAM, and a ROM. Functionally, this computer can be represented by a configuration including an input means 1, an arithmetic means 2, and an output means 3, as shown in FIG.

  The computing means 2 includes a word concept base 20, a partial text clustering means 320, an analyzing means 22, an input text concept vector generating means 24, a partial sentence extracting means 26, a partial sentence ranking means 28, and a control means 322. It has.

The partial text clustering means 320 clusters a set of partial texts obtained from the input document.
Specifically, sentence division and morphological analysis are performed for each partial text constituting the input document. By referring to the word concept base 20 for each partial text, a partial text concept vector representing the meaning of the partial text is generated.

  An example of the generation method will be described. For each partial text, the concept vector of the content word constituting the self-supporting part in the partial text is obtained from the word concept base 20, and the sum of the word concept vectors normalized to length 1 is obtained as the partial text. The concept vector.

  Clustering of a set of partial texts in the input document is performed by clustering of a corresponding partial text concept vector set. Clustering is performed by, for example, the k-means method.

  For each cluster obtained by the partial text clustering means 320, an analysis means 22, an input text concept vector generation means 24, a partial sentence extraction means 26, and a partial sentence ranking means 28, using the partial text set constituting the cluster as an input. Perform each process.

  The control unit 322 determines whether or not there is a partial text set that forms an unprocessed cluster. If there is an unprocessed cluster, the control unit 322 analyzes the partial text set that forms the unprocessed cluster from the analysis unit 22 onward. Each process is performed.

  The output means 3 combines and outputs the summary sentences output for each cluster by the partial sentence ranking means 28.

  In the processing of the partial text clustering means 320, sentence division and morphological analysis of the input partial text set in the processing after the analysis means 22 are performed, so the processing of the analysis means 22 is not performed, or only dependency analysis is performed. It can also take the structure of.

  In addition, in the processing of the partial text clustering means 320, after clustering is finished, the concept vector of each cluster can be calculated as the sum of the concept vectors of the partial texts constituting the cluster normalized to length 1. For example, since this becomes a concept vector of the input partial text set for the processing after the analysis means 22, it is possible to adopt a configuration in which the processing of the input text concept vector generation means 24 is eliminated.

<Operation of text summarization device>
Next, the operation of the text summarizing apparatus 300 according to this embodiment will be described. When an input document to be summarized is input to the text summarization apparatus 300, the input unit 1 receives the input document. Then, the text summarization processing routine shown in FIG. In addition, about the process similar to 1st Embodiment, the same code is attached and description is abbreviate | omitted.

  First, in step S300, the partial text clustering means 320 obtains, from the word concept base, the concept vector of the content word constituting the self-supporting part in the partial text for each partial text obtained from the input document. A vector is obtained, and a set of partial texts is clustered based on the partial text concept vector.

  In step S302, a partial text set of the target cluster is set.

  In steps S100 to S106, the processing of the analysis unit 22, the input text concept vector generation unit 24, the partial sentence extraction unit 26, and the partial sentence ranking unit 28 is performed with the partial text set constituting the target cluster as an input. .

  In step S304, the control unit 322 determines whether there is a partial text set that constitutes an unprocessed cluster. If there is an unprocessed cluster, the process returns to step S302. On the other hand, if the processing of step S302 to step S106 has been executed for all clusters, the process proceeds to step S108.

  In step S108, the output unit 3 combines and outputs the summary sentence output for each cluster in step S106, and ends the text summarization processing routine.

  In addition, since the other structure and effect | action of the text summarizing apparatus 300 which concern on 3rd Embodiment are the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, according to the third embodiment, a set of partial texts obtained from an input document is clustered, a partial sentence having a high similarity is selected for each cluster, and each selected partial sentence is selected. Can be output as a summary sentence, so that a summary sentence for each topic can be output, so that a summary covering all the topics in the input document can be output.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention. For example, a technique for performing clustering of a set of partial texts in an input document, and performing processing for each cluster using a partial text set constituting the cluster as an input may be applied to the second embodiment. Good. In this case, for each cluster obtained by the partial text clustering unit 320, the partial text set constituting the cluster is used as an input, and the analysis unit 22, the input text concept vector generation unit 24, the self-supporting part clustering unit 224, the superordinate concept Each process of the word acquisition means 226, the partial sentence extraction means 26, the partial sentence conversion means 228, and the converted partial sentence ranking means 230 is performed.

  Further, at least one of the word concept base 20, the thesaurus database 220, and the language model database 222 may be provided externally and connected to the text summarization apparatus via a network.

  The above text summarization apparatus has a computer system inside, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

  The present invention is applicable to a technique for outputting a summary sentence that accurately represents the contents of the entire text.

DESCRIPTION OF SYMBOLS 1 Input means 2 Calculation means 3 Output means 20 Word concept base 22 Analysis means 24 Input text concept vector generation means 26 Partial sentence extraction means 28 Partial sentence ranking means 100,200,300 Text summarization apparatus 220 Thesaurus database 222 Language model database 224 Independent Partial clustering means 226 Higher concept word acquisition means 228 Partial sentence conversion means 230 After conversion partial sentence ranking means 320 Partial text clustering means 322 Control means

Claims (7)

An analysis means for performing sentence division and morphological analysis on the input text;
A word concept base storing a set of pairs of a word and a word concept vector representing the meaning of the word;
Input text concept vector generation means for generating an input text concept vector representing the meaning of the input text by referring to the word concept base;
Partial sentence extraction means for extracting a set of partial sentences, which is a set of clauses in the sentence, from the input text;
By referring to the word concept base for each partial sentence extracted by the partial sentence extracting means, a partial sentence concept vector representing the meaning of the partial sentence is generated, and an input text concept vector and the partial sentence concept vector are generated. A partial sentence ranking means for calculating a similarity with the above and outputting the partial sentence with a high similarity as a summary sentence;
A text summarization apparatus comprising: An analysis means for performing sentence division and morphological analysis on the input text;
A word concept base storing a set of pairs of a word and a word concept vector representing the meaning of the word;
Input text concept vector generation means for generating an input text concept vector representing the meaning of the input text by referring to the word concept base;
Self-supporting part clustering means for clustering a set of self-supporting parts obtained from clauses in the input text;
For each self-supporting part cluster obtained by the self-supporting part clustering means, a superordinate concept word acquiring means for acquiring a word that is a superordinate concept of the self-supporting part belonging to the self-supporting part cluster;
Partial sentence extraction means for extracting a set of partial sentences, which is a set of clauses in the sentence, from the input text;
For each partial sentence extracted by the partial sentence extracting means, each independent part in the partial sentence is converted into a higher-order concept word of the independent part cluster to which the independent part belongs, obtained by the higher-order concept word acquiring means. Partial sentence conversion means for generating the converted partial sentence,
By referring to the word concept base for each converted partial sentence obtained by the partial sentence converting means, a converted partial sentence concept vector representing the meaning of the converted partial sentence is generated, and an input text concept vector And the converted partial sentence concept vector, and the converted partial sentence ranking means for outputting the converted partial sentence having a high similarity as a summary sentence;
A text summarization apparatus comprising: Further comprising partial text clustering means for clustering a set of partial texts obtained from the input document;
For each cluster obtained by the partial text clustering means, a set of partial texts constituting the cluster is used as the input text, the analyzing means, the input text concept vector generating means, the partial sentence extracting means, and the partial sentence 2. The text summarization apparatus according to claim 1, wherein a summary sentence is output by performing each process by the ranking means. Further comprising partial text clustering means for clustering a set of partial texts obtained from the input document;
For each cluster obtained by the partial text clustering means, a set of partial texts constituting the cluster is used as the input text, and the analyzing means, the input text concept vector generating means, the self-supporting part clustering means, the superordinate concept words 3. The text summarization apparatus according to claim 2, wherein the summarization text is output by performing each processing by the acquisition means, the partial text extraction means, the partial text conversion means, and the post-conversion partial text ranking means. Text in text summarizing apparatus including analysis means, word concept base storing a set of pairs of a word and a word concept vector representing the meaning of the word, input text concept vector generation means, partial sentence extraction means, and partial sentence ranking means A summarization method,
Performing sentence division and morphological analysis on the input text by the analyzing means;
Generating an input text concept vector representing the meaning of the input text by referring to the word concept base by the input text concept vector generating means;
Extracting a set of partial sentences, which is a set of clauses in the sentence, from the input text by the partial sentence extracting means;
The partial sentence ranking unit generates a partial sentence concept vector representing the meaning of the partial sentence by referring to the word concept base for each partial sentence extracted by the partial sentence extracting unit, and the input text concept Calculating a similarity between a vector and the partial sentence concept vector, and outputting the partial sentence having a high similarity as a summary sentence;
A text summarization method characterized by comprising: Analysis means, word concept base storing a set of pairs of words and word concept vectors representing the meaning of the words, input text concept vector generation means, freestanding clustering means, superordinate concept word acquisition means, partial sentence extraction means, part A text summarizing method in a text summarizing device including a sentence converting means and a converted partial sentence ranking means,
Performing sentence division and morphological analysis on the input text by the analyzing means;
Generating an input text concept vector representing the meaning of the input text by referring to the word concept base by the input text concept vector generating means;
Clustering a set of free standing parts obtained from clauses in the input text by the free standing part clustering means;
Obtaining a word that is a superordinate concept of an independent part belonging to the independent part cluster for each independent part cluster obtained by the independent part clustering means by the superordinate concept word acquiring means;
Extracting a set of partial sentences, which is a set of clauses in the sentence, from the input text by the partial sentence extracting means;
For each partial sentence extracted by the partial sentence extraction means by the partial sentence conversion means, each independent part in the partial sentence is obtained by the broader concept word acquisition means, and the independent part to which the independent part belongs Generating a converted partial sentence converted into a broader concept word of the cluster;
The converted partial sentence concept representing the meaning of the converted partial sentence by referring to the word concept base for each converted partial sentence obtained by the converted partial sentence ranking means by the converted partial sentence ranking means. Generating a vector, calculating a similarity between the input text concept vector and the converted partial sentence concept vector, and outputting the converted partial sentence having a high similarity as a summary sentence;
A text summarization method characterized by comprising:   The program for functioning a computer as each means of the text summarization apparatus of any one of Claims 1-4.