JP5523929B2 - Text summarization apparatus, text summarization method, and text summarization program - Google Patents

Description

  The present invention relates to a technique for summarizing text (documents).

  By combining the important sentence extraction method that creates a summary by selecting a sentence from multiple sentences and the sentence shortening method that shortens the sentence itself, more original sentences can be obtained even within a limited summary length. It is known as the prior art (see Non-Patent Document 1) that the above information can be included in the summary.

  As a conventional sentence shortening method, Non-Patent Document 2 describes a method of shortening a sentence by extracting a set of important words in a sentence and joining them using an n-gram model. . Alternatively, Non-Patent Document 3 describes a method of taking out important subtrees by deleting insignificant subtrees from the whole sentence based on the dependency structure of clauses including those words and shortening the sentences. ing. Also, a method for shortening a sentence by selecting a phrase based on the strength of dependency between phrases learned from a corpus is proposed in Non-Patent Document 4.

Manabu Okumura, Hideaki Namba, “Automatic Text Summarization”, Ohmsha, 2005, p.55-56 Tomoori Hori and Sadaaki Furui, “Attempt for Automatic Summarization of Speech”, Proceedings of Spoken Language and Engineering Workshop, 2001, p.165-171 Takeshi Omori, Hidetaka Masuda, Hiroshi Nakagawa, “Summary of Web Newspaper Articles and Evaluation Based on Articles for Mobile Devices”, Information Processing Society of Japan, Information Processing Society of Japan, 2003, NL153-1, p.1-8 Kiwamu Yamagata, Satoshi Fukutomi, Kazuyuki Takagi, Kazuhiko Ozeki, "Sentence Compression Using Statistical Information about Dependency Path Length", Text, Speech and Dialogue, Springer Berlin / Heidelberg, 2006, Volume 4188, p. 127-134

  However, the prior art can have the problem that a low-readable summary is generated. The conventional sentence shortening technique is based on a statistic in a corpus such as tf * idf as a criterion for selecting an important word, or uses a criterion based on dependency strength between phrases learned from the corpus. By combining sentence shortening and important sentence extraction based on these criteria, it is possible to shorten the sentences selected by the important sentence extraction and arrange them, or to shorten each sentence and then select and arrange the shortened sentences. Become. At this time, a summary with low readability may be generated. For example, consider the case where the next adjacent sentence n and sentence n + 1 are summarized.

Sentence n: (important word 1 unimportant word 1 unimportant word 2)
Sentence n + 1: (Important word 2 Non-important word 1 Important word 3)
Words that exist in common between adjacent sentences (for example, non-important word 1) may be more readable when included in the summary, but are judged to be less important in the whole text and are deleted by sentence shortening. May be.

  Conversely, different words that are important in the entire text (for example, important word 1 included in sentence n and important word 2 and important word 3 included in sentence n + 1, such that important word 1 and important word 2 indicate different topics. If one important word (for example, the important word 2 included in the sentence n + 1) is included in the summary, the topic may be shifted and a summary with low readability may be created. Since the entire text is judged to be important, the important word may be included in the summary without being deleted by sentence shortening.

  Thus, the problem in the conventional combination of important sentence extraction and sentence shortening is that the important sentence extraction and sentence shortening processes are independent, and global information (tf * idf or corpus) is used as a measure for selecting words and phrases. Dependency strength obtained from For this reason, local information cannot be used in adjacent short sentences constituting the summary, and a summary with low readability may be generated.

  In order to solve the above problems, the text summarization technique of the present invention stores in advance a weight parameter for a feature that is a combination considering the order of two feature elements and a sentence element score for a sentence element constituting the sentence. Each input sentence is converted, one or more conversion sentences are generated for each input sentence, a feature element is extracted from the conversion sentence, and a sentence element score for each sentence element included in each conversion sentence is used to generate each conversion sentence. The content score is calculated, the concatenation score of the conversion sentence is calculated using the feature element extracted by the feature element extraction unit and the weight parameter, and the sum of the content score and the concatenation score becomes the maximum value, or the maximum value Search for permutations of conversion statements that are approximate values of.

  The text summarization technique according to the present invention generates one or more conversion sentences for each input sentence, obtains a content score of each conversion sentence and a concatenation score of the conversion sentences, and searches permutations of the conversion sentences within the summary length. Therefore, in addition to the global information obtained from the corpus, etc., it is possible to use local information related to the concatenation of the conversion sentences, and to generate a summary with high content coverage and readability. Play.

The figure which shows the structural example of the text summarization apparatus 100,200. The figure which shows the processing flow of the text summarization apparatus. The figure which shows the example of a data of the sentence 1 after morphological analysis. The figure which shows the example of data of the sentence 2 by which the morphological analysis was completed. The figure which shows the example of data of the sentence 3 by which the morphological analysis was completed. The figure which shows the example of data of the sentence 4 by which the morphological analysis was completed. The figure which shows the structural example of the conversion sentence production | generation part 112. FIG. The figure which shows the processing flow of the conversion sentence production | generation part 112. FIG. The figure which shows the example of data of a language model. The figure which shows the processing flow of the short sentence production | generation part 112a. The figure for demonstrating the production | generation method of a feature vector. The figure which shows the pseudo code example of the learning algorithm using an averaging perceptron. The figure which shows the example of a flowchart of FIG. The figure which shows the example of data of a weight parameter. The figure which shows the pseudo code example of Held and Karp Algorithm used when calculating | requiring a maximum value. The figure for demonstrating a dynamic programming and a beam search. The figure which shows the example of a processing flow of an important sentence permutation search part. FIG. 3 is a block diagram of the text summarization apparatus 100. The figure which shows the structural example of the conversion sentence production | generation part 212. FIG. The figure which shows the processing flow of the conversion sentence production | generation part 212. FIG. The figure which shows the data example of a weight parameter (conditional probability).

  Hereinafter, embodiments of the present invention will be described in detail.

<Text Summarization Device 100>
A text summarizing apparatus 100 according to the first embodiment will be described with reference to FIGS. 1 and 2. The text summarizing apparatus 100 includes a storage unit 103, an input unit 131, a converted sentence generation unit 112, a feature element extraction unit 113, a content score calculation unit 115, a concatenation score calculation unit 117, an important sentence permutation search unit 119, and an output unit 135. Have. For example, the feature element extraction unit 113, the content score calculation unit 115, the concatenation score calculation unit 117, and the important sentence permutation search unit 119 have the same configuration as the text summarization device described in the unpublished Japanese Patent Application No. 2010-010906.

  The text summarization apparatus 100 summarizes text data composed of one or more input sentences. Details will be described below. The text data is composed of one or more sentences, and the sentences express the thoughts and feelings of the speaker or the writer, and are composed of one or more sentences.

<Storage unit 103 and input unit 131>
The storage unit 103 stores in advance a weight parameter for a feature that is a combination considering the order of two feature elements, and a sentence element score for a sentence element constituting the sentence.

  A method for generating each data will be described later. Further, the storage unit 103 stores / reads each input / output data and each data of the calculation process one by one. Thereby, each calculation process is advanced. However, the data need not necessarily be stored in the storage unit 103, and data may be directly transferred between the units.

  The text summarization apparatus 100 receives text data to be summarized via the input unit 131. Furthermore, in this embodiment, a summary length value that defines the summary length is also input via the input unit 131. The summary length may be any number that defines the length of the summary, such as the number of characters, the number of bytes, the number of words, etc., and is appropriately set by the summary creator.

The text data to be input may have been subjected to morphological analysis, extracted unique expressions, and dependency analysis. The input unit 131 is a keyboard, a file, a communication line, or an input terminal thereof, and is not particularly defined. For example, FIG. 3 to FIG. 6 show data examples of text data including sentences 1 to 4 shown below.
Sentence 1: “I ate pizza here, but the image of pizza has changed a lot.”
Sentence 2: “Every dish by a chef trained in Italy is the best.”
Sentence 3: “The food is a creative dish based on Italian and very delicious.”
Sentence 4: “This store is a long-established Italian restaurant located 10 minutes on foot from Yokohama Station.”

<Conversion sentence generation unit 112>
The conversion sentence generation unit 112 receives an input sentence via the input unit 131, converts the input sentence, and generates one or more conversion sentences (hereinafter referred to as “variations of conversion sentences”) for each input sentence (s112). ) And transmitted to the feature element extraction unit 113. For example, the converted sentence generation unit 112 includes only the shortened sentence generation unit 112a illustrated in FIG.

(Short sentence generator 112a)
The abbreviated sentence generation unit 112a generates one or more abbreviated sentences (hereinafter referred to as “variations of abbreviated sentences”) for each input sentence using the input sentence (s112a, see FIG. 8), and uses this as a converted sentence. It transmits to the element extraction part 113.

The shortened sentence generation unit 112a generates a non-natural shortened sentence when the sentence is shortened. The method for generating a shortened sentence is not particularly specified, and an existing sentence shortening method may be used. For example, a variation of a shortened sentence may be generated using the method described in Reference 1, or a shortened sentence candidate that is simply a subtree may be generated as a shortened sentence variation using only the dependency structure. Good.
[Reference 1] Takaaki Hasegawa, Hitoshi Nishikawa, Kenji Imamura, Genichiro Kikui, Manabu Okumura, "Generation of Summary Text from Web Pages for Mobile Devices", Journal of the Japanese Society for Artificial Intelligence, 2010, Vol.25, No .1, p.133-143

  As an example, all subtrees that contain the root clause based on the dependency relationship are used as sentence shortening candidates, and the generation probability of each shortened sentence candidate is calculated using a language model based on the word n-gram or part-of-speech n-gram. A method for narrowing down to the top N short sentence candidates with high generation probabilities and outputting them as variations of the short sentence will be described. Note that a threshold may be provided to narrow down to short sentence candidates that have a generation probability equal to or higher than the threshold. Also, the original sentence, that is, the original input sentence that has not been shortened, may be included as one of the variations of the shortened sentence.

  In this case, the storage unit 103 stores a word connection probability in advance as a language model. For example, the language model stores n-gram probabilities of words or parts of speech based on the appearance frequency of a chain of words or parts of speech from a large number of text corpora (see FIG. 9). Hereinafter, a processing example of the shortened sentence generation unit 112a will be described with reference to FIG.

  The following processing is performed for all input sentences. First, a dependency analyzed sentence is input (s112a-1), and a pointer is set at the end of the sentence (s112a-2). If the pointer is not the beginning of a sentence (s112a-3), the clause in which the pointer is set is connected before all the sentence shortening candidates of the sentence (s112a-4). If the clause where the pointer is set is the root clause indicating the predicate, or there is a destination of the clause where the pointer is set (s112a-5), the root clause or the pointer is set. A sentence shortening candidate to which the phrase is connected is newly generated as a shortened sentence candidate, and the generation probability of the shortened sentence candidate is calculated (s112a-6). After that, the pointer is moved by one phrase toward the beginning of the sentence (s112a-7). If the clause of the pointer does not satisfy the previous condition (s112a-5), the pointer is moved forward by one clause without being newly generated as a shortened sentence candidate (s112a-7). The above process is repeated (until the beginning of the sentence) until there are no more phrases that can be processed (s112a-3). In this process, the search (beam search) may be performed while limiting the number of shortened sentence candidates to the highest N of the generation probabilities. In addition, an efficient method such as dynamic programming may be applied in the search process. When all the clauses have been processed (s112a-3), the top N (N-best) short sentence candidates of the short sentence generation probability, or the short sentence candidate having a generation probability equal to or higher than a predetermined threshold value are selected. Select (s112a-8) and output as a variation of the abbreviated sentence. For example, the generation probability is obtained by the following equation.

Here, W is a word sequence of abbreviated sentence candidates, w _i is the i-th word, and w ₀ and w _{n + 1} are a head symbol <s> and a sentence end symbol </ s>, respectively. The generation probability may be a geometric average based on the length of the word.

As an example, a variation of a short sentence generated from sentence 1 is shown below. It is an example in which the top 12 cases of the generation probability of the short sentence candidate are variations of the short sentence.
Short sentence 1-1 “changed drastically” (10 characters)
Short sentence 1-2 “Image has changed.” (12 characters)
Short sentence 1-3 "Image has changed a lot" (15 characters)
Short sentence 1-4 “The image of pizza has changed” (18 characters)
Short sentence 1-5 “The image of pizza has changed significantly” (21 characters)
Short sentence 1-6 “I ate pizza, but the image has changed.” (22 characters)
Short sentence 1-7 “I ate pizza, but the image changed a lot.” (25 characters)
Short sentence 1-8 “I ate pizza, but the image of pizza has changed.” (28 characters)
Short sentence 1-9 “I ate pizza, but the image of pizza has changed a lot.” (31 characters)
Short sentence 1-10 “I ate pizza here, but the image has changed a lot.” (28 characters)
Short sentence 1-11 “I ate pizza here, but the image of pizza has changed.” (31 characters)
Short sentence 1-12 “I ate pizza here, but the image of pizza has changed a lot.” (34 characters)

Similarly, the variation of the short sentence produced | generated from the sentence 2 is shown below.
Short sentence 2-1 “It ’s the best.” (5 characters)
Short sentence 2-2 "Everything is the best" (8 characters)
Short sentence 2-3 “Cooking is the best” (8 characters)
Short sentence 2-4 "Every dish is the best" (11 characters)
Short sentence 2-5 “Cook cooking is the best” (14 characters)
Short sentence 2-6 “Everything cooked by a chef is the best” (17 characters)
Short sentence 2-7 “Cooking by a chef who has trained is the best” (20 characters)
Short sentence 2-8 “Every dish by chefs who have trained is the best.” (23 characters)
Short sentence 2-9 “Cooking by a chef trained in Italy is the best” (25 characters)
Short sentence 2-10 “Everything cooked by a chef trained in Italy is the best” (28 characters)

Similarly, the variation of the short sentence produced | generated from the sentence 3 is shown below.
Short sentence 3-1 “Delicious.” (7 characters)
Short sentence 3-2 “Very delicious” (9 characters)
Short sentence 3-3 "Creative cuisine is delicious" (12 characters)
Short sentence 3-4 “It is very delicious with creative dishes” (14 characters)
Short sentence 3-5 "It is delicious with creative dishes based on Italian." (24 characters)
Short sentence 3-6 "It is very delicious with creative dishes based on Italian" (26 characters)
Short sentence 3-7 “Cooking is delicious” (12 characters)
Short sentence 3-8 “Cooking is very delicious” (14 characters)
Short sentence 3-9 “Cooking is delicious with creative dishes” (17 characters)
Short sentence 3-10 “Cooking is very delicious with creative dishes” (19 characters)
Short sentence 3-11 “Cooking is delicious with creative cuisine based on Italian” (29 characters)
Short sentence 3-12 “Cooking is an Italian-based creative dish that is very delicious.” (31 characters)

<Feature Element Extraction Unit 113>
The feature element extraction unit 113 receives the conversion sentence, extracts a feature element (for example, a content word (noun, verb, adjective)) from the conversion sentence (s113), and transmits it to the connection score calculation unit 117. Note that the conversion sentence generation unit 112 uses the input sentence (see FIGS. 3 to 6) that has been subjected to morphological analysis, extracted from the specific expression, and subjected to dependency analysis to generate conversion sentence variations. Feature elements can be extracted based on the morphological analysis data added to the sentence.

<Content score calculator 115>
The content score calculation unit 115 receives the sentence element included in each conversion sentence, obtains the content score of each conversion sentence using the sentence element score for the sentence element (s115), and searches the important sentence permutation for the content score. To the unit 119.

  For example, when a feature element is used as a sentence element (in this embodiment, a feature element is a content word), the output of the feature element extraction unit 113 is used as an input, and the content score calculation unit 115 receives the input content word. As a key, the sentence element score for the content word is searched and acquired from the storage unit 103, and the sum of the sentence element scores for all the content words included in the sentence is obtained. This sum is used as a content score, and can be expressed by the following formula. However, Content (s) represents the content score of the sentence s, and Weight (p) represents the sentence element score of the content word p included in the sentence s.

[Calculation method of sentence element score]
The sentence element score of each content word is stored in the storage unit 103 in advance. The text summarizing apparatus 100 has, for example, a sentence element score calculation unit (not shown), and obtains a sentence element score. As the sentence element score, for example, the number of times the word appears in the text to be summarized can be used.

  For example, the sentence element score calculation unit obtains a sentence element score from sentence elements included in the text set by using a text set for learning sentence element scores in advance. For example, the sentence element score calculation unit counts the number of sentences including sentence elements using a sentence element score learning text set in advance, and obtains a sentence element score using a database in which the number cnt is recorded.

  For example, when the sentence element score calculation unit is more important in the learning text set, the sentence element score is calculated so that the sentence element score increases as the number cnt increases. In this case, the sentence element score is the number cnt itself or the logarithm of cnt. In addition, when the number of sentence elements is not important in the learning text set, the sentence element score is calculated so that the sentence element score decreases as the number cnt increases. In this case, the sentence element score is a value obtained by dividing the number of texts included in the text set by the number cnt, the logarithm of the divided value, or the like. With such a configuration, an appropriate content score can be calculated for a set of sentences to be summarized.

  Of course, information extraction results such as evaluation information can also be used as sentence elements. In that case, a sentence element score is given to a word that evaluates an object such as “good image quality” or “dishes is delicious” instead of the above-described content word, and a content score is given to the sentence based on those words. be able to.

<Connection score calculation unit 117>
The concatenation score calculation unit 117 receives the feature element, obtains the concatenation score of the converted sentence using the weight parameter corresponding to the feature obtained from the feature element (s117), and transmits it to the important sentence permutation search unit 119. The connection score of two sentences is a value indicating the goodness of connection between the two sentences.

  For example, suppose there is a sentence “I ate rice yesterday” and a sentence “But it was n’t so delicious”. These two sentences are natural if they appear in the order of “I ate rice yesterday” or “But it was n’t so delicious”, but they were “but not so good” or “I ate rice yesterday” It is very unnatural when it appears at. This is because the sentence "but not so delicious" implicitly assumes that the topic about meals has appeared in the previous sentence.

  Similarly, when a summary is generated by connecting a plurality of sentences, the generated summary may be very difficult to read and unnatural if the sentences cannot be rearranged appropriately.

  For example, you can give a score to the goodness of the connection of sentences, but rather than the line of sentences "But it wasn't very delicious" and "I ate rice yesterday", "I ate rice yesterday" If you can give a high score to the line that says “There was no,” you can rearrange the sentences according to the score. That is, if two sentences si and sj are given, the order of si, sj and the order of sj, si are calculated, and the order with the highest score is adopted.

  First, a connection score is defined. In this embodiment, as an example, a connection score when a sentence sj appears after the sentence si is defined by the following function Connect (sj | si).

Connect (sj | si) = w ^T φ (si, sj) (2)
Here, w is the above-described weight parameter, φ (si, sj) is a binary vector (hereinafter referred to as “feature vector”) representing the connection between the sentence si and the sentence sj, and T represents transposition. w ^T φ (si, sj) is an inner product of w ^T and φ (si, sj). The weight parameter w is calculated in advance by a method described later as an example, stored in the storage unit 103, and called from the storage unit 103 when summarizing.

The connected score calculation unit 117 includes, for example, a feature vector generation unit 117a and a calculation unit 117b.
(Feature Vector Generation Unit 117a)
The feature vector generation unit 117a sets each element of the Cartesian product set of the feature elements included in the two sentences si and sj as the features of the two sentences, sets the dimension corresponding to the obtained feature to 1, and sets the other dimensions to 0. A feature vector φ (si, sj) is generated.

  A feature vector representing a connection between two sentences is given as an example of a Cartesian product set of content words (nouns, verbs, and adjectives) included in the two sentences. This will be described with reference to FIG. It is assumed that the sentence si is a sentence “I ate rice yesterday” and the sentence sj is a sentence “but not very delicious”. The sentence si includes content words “Yesterday”, “rice”, and “eat”, and the sentence sj includes content words “delicious” and “a”. As shown in FIG. 11, these Cartesian product sets are “Yesterday”, “Delicious”, “Yesterday”, “A”, “Rice”, “Delicious”, “Rice”, “A”, “Eat”, “Delicious”, “Eat”, “A” "Is a set of six words. The feature vector φ (si, sj) is a binary vector whose dimension corresponding to these six types of features is 1. If the feature vector is not trimmed, the dimension of the feature vector is the same as the number of features that appear in the text set used for learning. Therefore, although it is a much higher-dimensional vector in practice, in FIG. 11, it is a 6-dimensional vector corresponding to the six types of features shown in the figure for simplicity. In addition to the features shown above, as an example, dependency between words, specific expressions, and the like can be used. Note that the feature pruning is a process of deleting a feature that is not very important as a parameter indicating the goodness of a sentence when calculating a weight parameter, and reducing the number of features.

(Calculation unit 117b)
The calculation unit 117b obtains the inner product of the weight parameter w and the feature vector φ (si, sj) as the concatenation score Connect (sj | si) of the two sentences. That is, the calculation of Expression (2) is performed using the weight parameter and the feature vector.

(Calculation unit of connection score of 3 or more sentences)
The concatenated score calculation unit 117 calculates a concatenated score w ^T Φ (x, y) of a sequence of three or more sentences using the concatenated score Connect (sj | si) of the two sentences (s117). In this case, the connection score represents the goodness of the overall connection of a set of three or more sentences. x represents a given set of sentences, and y represents a sequence of sentences.

For example, when sentences s1, s2, and s3 are given, there are six ways of arranging them. Of the six ways of arrangement, the sequence with the highest connection score of the sequence of sentences is defined as the sequence of three given sentences. For this purpose, it is necessary to calculate the connection score of a sequence of three or more sentences, but here, the sequence of three or more sentences is decomposed into a sequence of two sentences, and the connection score of the two decomposed sentences is calculated. Let the sum be the concatenation score of a sequence of three or more sentences. A connection score w ^T Φ (x, y) of a sequence of three or more sentences is defined as follows.

y represents a sequence of sentences. In this example, y represents that a given sentence set x = {s1, s2, s3} is arranged in the order of s2, s3, and s1. In addition, s0 and s4 represent the beginning and end of the document, respectively, that is, the sentence s2 is at the beginning of the three sentences and the sentence s1 is at the end of the three sentences. By introducing s0 and s4, it is possible to consider a sentence that tends to be at the beginning of a document or a sentence that tends to be at the end.

[Calculation method of weight parameter]
A method for calculating the weight parameter w will be described. Here, it is assumed that the sentence parameter of the text written by a human is a correct answer and the weight parameter w that can reproduce the sentence sequence is a good parameter. In other words, when a set of sentences is given, the parameter w that allows them to be arranged in an easy-to-read manner can be restored to the original sequence when given a set of sentences contained in a certain handwritten text. Or put the assumption that they can be arranged side by side as close as possible. Under this assumption, the parameter w is estimated from a manually written text set. For example, the weight parameter w can be calculated from a text set by the algorithm shown in FIGS. 12 and 13 (see Reference 2).
[Reference 2] Michael Collins, “Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms”, In Proceedings of the 2002 Conference on Empirical Methods on Natural Language Processing (EMNLP), Association for Computational Linguistics, 2002, Volume 10, pp.1-8

The text summarization apparatus 100 has a weight parameter calculation unit (not shown), for example, and obtains a weight parameter. The weight parameter calculation unit receives training data τ including a set of Q x _q (a set of sentences) and a correct sentence sequence y _q for each sentence set, and the algorithm shown in FIG. 12 learns the weight parameter w. To do. However, q = 1, 2,..., Q. Specifically, taken one by one Q-number of training cases, arrange the statements in x _q using the current w (4 line of FIG. 12). Among the possible sentence sequences obtained using the current weight parameter w, the sentence sequence y ′ having the maximum concatenation score w ^T Φ (x, y) is obtained (argmax operation on the fourth line in FIG. 12). Details of the argmax operation will be described later. When y _q ≠ y ′ (that is, when the sequence of sentences having the largest concatenation score w ^T Φ (x, y) is different from the sequence of correct sentences), the current weight parameter w can reproduce the correct sequence. It was impossible. At that time, the weight parameter w is updated so that sentences can be rearranged correctly (line 5 in FIG. 12). If the correct sequence can be reproduced with the current parameter w, the weight parameter is not updated (line 6 in FIG. 12).

  This is performed on Q training data, and the weight parameter w is updated by repeating it N times, and the sum v of the weight parameter w at a certain moment is divided by N × Q, which is the number of times w is added. This is averaged, and this is set as the final weight parameter w (9th line in FIG. 12) and stored in the storage unit 103. Note that N varies depending on the nature of the weight parameter w to be obtained, and can be appropriately obtained by experiments or the like prior to the calculation of the weight parameter w. Further, the training data τ is an arbitrary text in which the beginning and end of the document and the sentence boundary are specified. Of course, it is also possible to devise such as learning the parameter w only from text of the same genre as the genre of text to be summarized. As an example, the estimated weight parameter is shown in FIG. The feature column shown in FIG. 14 corresponds to the feature shown in FIG. 11, and the weight column is a weight parameter of the feature. According to FIG. 14, when arranging sentences, it is easier to arrange sentences including sentences such as “vegetables”, “oil”, and “friends” after sentences including the word “cooking”. On the other hand, if a sentence including the words “night view” and “elegance” is arranged after a sentence including the word “cooking”, an error is likely to occur.

(Argmax operation)
argmax operation, articulation score of the list of possible sentences using the statements contained in the set x _q sentence obtaining the sequence y 'statement that maximizes. This is a so-called traveling salesman problem, and it is difficult to obtain an exact solution in a short time. For example, Q! It is necessary to obtain the sequence of sentences with the highest connection score from the sequence of street sentences, and the amount of calculation increases exponentially as the Q value increases.

Therefore, as an example, an approximate solution can be obtained using dynamic programming and beam search, and the argmax operation can be substituted. Specifically, as an example, Held and Karp Algorithm, which is a kind of dynamic programming, is used (see Reference 3). FIG. 15 shows a search for an approximate solution of a sentence sequence by Held and Karp Algorithm.
[Reference 3] Michael Held and Richard M. Karp, “A dynamic programming approach to sequencing problems”, In Journal of the Society for Industrial and Applied Mathematics (SIAM), 1962, Vol. 10, No. 1, pp.196 -210

S is a set of sentences to be rearranged, in which s0 indicates the head of the document and s (Q + 1) indicates the end of the document. That is, there is a problem of searching for the route having the highest score among all the routes starting from s0 and always passing through the sentences from s1 to sQ once and arriving at s (Q + 1). M is a matrix that stores connection scores between all sentences included in S. For example, M _{k, j} indicates a connection score of sentences sk and sj, that is, corresponds to w ^T φ (sk, sj). H _i (C, sk) is a hypothesis that has already passed C⊆S and added a sentence sk at time i and the score of the hypothesis. H ^* is the path with the highest connection score of the sentence sequence.

  When arranging the sentences, the Held and Karp Algorithm uses all the hypotheses except the one with the highest score if there are multiple hypotheses where the last sentence selected and the previously selected sentence are the same regardless of the order. The search is efficiently performed by discarding the hypothesis (line 5 in FIG. 15). For example, the sequence of sentences s1, s2, s3 and s2, s1, s3 shown in broken lines in FIG. 16 ends with s3, and the sentences selected so far are the same, so both of these two hypotheses are expanded. do not have to. For example, when the connection score of s2, s1, and s3 is high, only the hypotheses related to this sentence should be developed, and the hypotheses related to s1, s2, and s3 should be discarded.

  However, since the search space is still vast, only the top b hypotheses with a high connection score at the time point i are expanded to be the hypothesis at the time point i + 1 (beam search, line 4 in FIG. 15). That is, the hypotheses other than the top b are discarded. For example, if the sentence sequence s3, s1, and s2 represented by the alternate long and short dash line in FIG. 16 is not within the top b at i = 3, it is not necessary to develop this hypothesis after i = 4. As a result, the search space can be significantly reduced. Thereby, an approximate solution can be searched more efficiently.

  For example, when Q = 100 and b = 20 and the Held and Karp Algorithm is not used, only the top 20 hypotheses are developed out of the 100 hypotheses generated at time i = 1. Therefore, 99 hypotheses are developed for the top 20 cases. As a result, among the 1980 hypotheses generated, only the top 20 hypotheses need be developed. When the beam search is not performed, 99 hypotheses are developed for every 100 cases. For the 9900 hypotheses generated as a result, 98 hypotheses are developed. Since this process is repeated up to the time point i = Q, the amount of calculation is enormous as compared with the case of using a beam search.

  The argmax operation is performed in the process of obtaining the weight parameter w (line 4 in FIG. 12), and the concatenation score M is used during the operation (see FIG. 15). This concatenation score is used when the argmax operation is called. Using the weight parameter (during updating) of (2) and the like.

<Important sentence permutation search unit 119>
The important sentence permutation search unit 119 receives the content score and the concatenation score, searches for a permutation of the conversion sentence in which the sum of these becomes the maximum value or the approximate value of the maximum value (s119), and the output unit Send to 135.

  The important sentence permutation search unit 119 searches for a list of one or more conversion sentences obtained from the text data to be summarized within the summary length based on the connection score and the content score. At this time, a restriction may be added so as not to obtain a permutation between variations of conversion sentences generated from the same input sentence. In this case, it is not necessary for the connection score calculation unit to obtain a connection score between variations of the converted sentence generated from the same input sentence. For example, an input sentence identifier is added to a variation of a short sentence, and a connection score and a permutation between short sentences given the same input sentence identifier are not obtained.

S ^* is a sequence of sentences in which the sum of the content score and the connection score is the maximum value or an approximation of the maximum value, U is a sequence of sentences that can be constructed from the text to be summarized, and S is U One of the arbitrary sequences included in. When selecting a sentence, instead of simply selecting a sentence, the best sentence sequence S ^* is selected from U in consideration of the content score and the connection score. As an example, S ^* can be defined as follows.

λ is a parameter that controls which of the content score and the connection score is important. (Si, sj) ∈ S indicates that among the adjacent sentences si and sj in the sentence sequence S, si appears before sj. length (S) indicates the length of the sentence sequence S, and K indicates the summary length.

The best summary S ^* is the sum ΣContent (s) of the content score Content (s) of all sentences s included in S ^* and the connection score Connect (sj | si) between adjacent sentences among those sentences. The sum of the total ΣConnect (sj | si) is the largest. Note that the maximum value of the sum of the content score and the connection score is not only argmax [ΣContent (s) + ΣConnect (sj | si)] but also a value adjusted using λ argmax [ΣContent (s) + λΣConnect (sj | Si)].

However, it is difficult to obtain an exact solution in a short time for such a problem of obtaining S ^*, that is, a problem of performing the argmax operation of Equation (4). In order to cope with this, as in the case of learning the weight parameter w, a search is performed using dynamic programming and beam search to obtain an approximate solution. Specifically, H _i (C, sk) in FIG. 15 represents the sum of the concatenation score and the content score (only the concatenation score is represented when learning w), and only the hypothesis having a high sum is represented. Try to expand sequentially. On the other hand, unlike when learning w, there is a limitation on the summary size, so it is not necessary to arrange all sentences. Therefore, while performing a search using the Held and Karp Algorithm and beam search shown in FIG. 15, the hypothesis exceeding the summary size is stopped regardless of the addition of any sentence, and is separately stored as a summary candidate. Then, after all hypotheses are developed, an approximate solution can be obtained by selecting the one with the highest score from the stored summary candidates.

  Hereinafter, a processing example of the important sentence permutation searching unit 119 will be described with reference to FIG. Let S (i) = {S (i, 1), S (i, 2),..., S (i, H (i))} be the set of H (i) sentence sequences at time point i. The summarization source text includes Q sentences, and the set is represented as Z = {s1, s2,..., SQ}.

  First, initial setting is performed (s119a). It is determined whether or not S (i, h) of the sentence sequence at time i already covers the sentence sq (s119b). If not, the sentence sq is added to S (i, h). , A sentence sequence S (i + 1, k) is generated (s119c). It is determined whether or not the size of S (i + 1, k) is equal to or smaller than the summary size K (s119d). If it is larger, the sentence sequence S (i, h) before the sentence sq is added is saved (s119e). Thereafter, the hypothesis is not developed for this sentence sequence S (i + 1, k). For example, in FIG. 16, when S2 is added to the sentence sequence S3, S1 represented by the alternate long and short dash line, if the summary size K is exceeded, the sentence sequence S3, S1 is stored, and the sentence sequence is stored. The hypothesis is not developed for S3, S1, and S2.

  This processing is performed for all the sentence sequences S at time i (s119g, h), and is further performed for all sentences included in the text to be summarized (s119i, j).

  The content score of each sentence sequence included in the set S (i + 1) = {S (i + 1,1), S (i + 1,2),..., S (i + 1, k)} A sum sum of scores is obtained (s119k). If k ′ = 1, 2,..., k, the content score of each sentence sequence is Content (S (i + 1, k ′)), and the connection score is Connect (S (i + 1, k ′)),

It can be expressed as. Sum (S (i + 1, k ′)) corresponding to all k ′ is obtained, and whether or not there is a sequence of sentences in which the last added sentence is the same and the sentence set already covered is the same. If it is present, it is determined whether sum (S (i + 1, k ′)) is maximum in the sequence of existing sentences (s119m). If not, the corresponding hypothesis is discarded. (S119n). After that, it is determined whether or not the value of each sum belongs to the top b cases (s119p). If it does not belong, the corresponding hypothesis is discarded (s119n). Thereafter, the hypothesis is not expanded for the discarded sentence sequence.

i is updated (s119q), and the above processing (s119b to s119q) is repeated. Usually, since K is smaller than the size of the text of the summarizing source, all of the sentences included in the text of the summarizing source are arranged. Hypotheses are discarded or saved. Then, the sequence of sentences corresponding to the largest sum among the stored hypotheses is S ^* .

<Output unit 135>
The output unit 135 receives the list of converted sentences searched by the important sentence permutation searching unit 119, shapes the permutation into a sentence, and outputs the sentence.

For example, when text data composed of sentence 1, sentence 2, and sentence 3 (see FIGS. 3 to 5) and 45 characters are input as the summary length, the following output (summary) ) Can be obtained.
Sentence 2 “Cooking by a chef trained in Italy is the best.” (25 letters: 0.89 reduction rate)
↓
Sentence 1 “The image of pizza has changed.” (18 characters: shortening rate 0.53)
Summary “Cooking by a chef trained in Italy is the best. The image of pizza has changed.” (43 characters: summary rate 0.46)
This means that the shortened sentence of sentence 1 is connected after the shortened sentence of sentence 2, and the shortening rate of each sentence is different.

<Effect>
With such a configuration, a plausible conversion sentence variation is generated for each input sentence, the content score of each conversion sentence and the connection score of the conversion sentence are obtained, and a permutation that considers this simultaneously is searched. It is a permutation of important and natural translations. In addition to global information, local information can be used, and it is possible to generate a summary with high content coverage and readability. In other words, it is possible to generate an easy-to-read summary that covers many important contents of the original text.

  In particular, the shorter the summary length, the more advantageous the present invention. According to the present embodiment, since the ease of connection between adjacent shortened sentences is considered in the summary, sentence shortening that does not lack information essential for easy connection can be realized while sentence shortening is limited. An easy-to-read summary can be generated even with the summary length.

  Also, for the problem of generating a summary from multiple sentences, the summary content is covered by including another valuable information in the summary for the shortened length without sacrificing the readability of the summary even if the sentence is shortened. Can be improved.

  Further, according to the present embodiment, the optimum sentence shortening permutation uniquely determines the optimum sentence shortening shortening rate for each sentence. For this reason, there is an effect that a parameter relating to a sentence shortening rate in each sentence is no longer required.

<Hardware configuration>
FIG. 18 is a block diagram illustrating a hardware configuration of the text summarizing device 100 in the present embodiment. As illustrated in FIG. 18, the text summarization device 100 in this example includes a CPU (Central Processing Unit) 11, an input unit 12, an output unit 13, an auxiliary storage device 14, a ROM (Read Only Memory) 15, and a RAM (Random). Access Memory) 16 and a bus 17.

  The CPU 11 in this example includes a control unit 11a, a calculation unit 11b, and a register 11c, and executes various calculation processes according to various programs read into the register 11c. The input unit 12 is an input interface for inputting data, a keyboard, a mouse, and the like. The output unit 13 is an output interface for outputting data, a display, a printer, and the like. The auxiliary storage device 14 is, for example, a hard disk, a semiconductor memory, or the like, and stores programs and various data for causing the computer to function as the text summarization device 100. Further, the above program and various data are expanded in the RAM 16 and used by the CPU 11 or the like. The bus 17 connects the CPU 11, the input unit 12, the output unit 13, the auxiliary storage device 14, the ROM 15, and the RAM 16 so that they can communicate with each other. In addition, as a specific example of such hardware, a server apparatus, a workstation, etc. other than a personal computer can be illustrated, for example.

<Program structure>
As described above, each program for executing each process of the text summarizing apparatus 100 of this embodiment is stored in the auxiliary storage device 14. Each program constituting the text summary program may be described as a single program sequence, or at least a part of the program may be stored in the library as a separate module.

<Cooperation between hardware and program>
The CPU 11 expands the above-described program and various data stored in the auxiliary storage device 14 in the RAM 16 according to the read OS program. The address on the RAM 16 where the program and data are written is stored in the register 11c of the CPU 11. The control unit 11a of the CPU 11 sequentially reads these addresses stored in the register 11c, reads a program and data from the area on the RAM 16 indicated by the read address, causes the calculation unit 11b to sequentially execute the operation indicated by the program, The calculation result is stored in the register 11c.

  FIG. 1 is a block diagram illustrating a functional configuration of a text summarizing apparatus 100 configured by reading and executing the above-described program in the CPU 11 as described above.

  Here, the storage unit 103 corresponds to any one of the auxiliary storage device 14, the RAM 16, the register 11 c, other buffer memory and cache memory, or a storage area using these in combination. Moreover, the conversion sentence generation part 112, the feature element extraction part 113, the content score calculation part 115, the connection score calculation part 117, and the important sentence permutation search part 119 are comprised by making CPU11 run a text summary program. is there.

[Modification 1]
Only parts different from the first embodiment will be described with reference to FIGS.

  The conversion sentence generation unit 112 includes only the division unit 112b illustrated in FIG. The storage unit 103 stores a division rule for dividing a compound sentence and a heavy sentence into a single sentence.

  The division unit 112b receives the input sentence via the input unit 131, refers to the division rule of the storage unit 103, and divides the input sentence into a single sentence when the input sentence is a compound sentence or a heavy sentence (s112b), and two or more A simple sentence is transmitted to the feature element extraction unit 113 as a variation of the converted sentence.

  For example, the morpheme “de / determinant: combination” + “a / verb stem: R” + “ri / verb suffix: combination” + “, / reading” is changed to “de / determination: combination” + “a / verb A division rule of dividing into “stem: R” + “ru / verb suffix: end” + “./ phrase” is stored in the storage unit 103. The dividing unit 112b receives the morpheme-analyzed sentence 4 "This store is a long-established Italian restaurant and is located a 10-minute walk from Yokohama Station (see FIG. 6)", and the morpheme "de / determinant: The storage unit 103 is searched by using “continuous use” + “a / verb stem: R” + “ri / verb suffix: continuous use” + “, / read” as a key, and the corresponding division rule is obtained. 4-1 “This shop is a long-established Italian restaurant” and sentence 4-2 “Located 10 minutes on foot from Yokohama station” are divided into two sentences, and these are output as variations of conversion sentences. To do.

  According to this modification, one or more conversion sentences are generated for each input sentence, the content score of each conversion sentence and the concatenation score of the conversion sentence are obtained, and the permutation of conversion sentences within the summary length is searched. In addition to global information obtained from a corpus or the like, it is possible to use local information related to the connection between converted sentences, and it is possible to generate a summary with high content coverage and readability. In particular, since compound sentences and heavy sentences can be divided into simple sentences and converted into variations of sentences, easy-to-read (highly readable) summaries can be generated.

  In this modification, it is not necessary to add a restriction so as not to obtain a permutation between variations of the conversion sentence generated from the same input sentence. This is because the variations of the converted sentence are sentences having different contents.

[Modification 2]
Only parts different from the first embodiment will be described with reference to FIGS.

  The conversion sentence generation unit 112 includes only the paraphrase unit 112c illustrated in FIG. The storage unit 103 stores, as a paraphrase table, a rule for paraphrasing a certain expression into a shorter expression having the same meaning.

  The paraphrase unit 112c receives the input sentence via the input unit 131, refers to the paraphrase table in the storage unit 103, paraphrases the expression of the input sentence into another expression having the same meaning (s112c), and paraphrases. The sentence is transmitted to the feature element extraction unit 113 as a variation of the converted sentence.

  For example, in the case where a rule that paraphrases the expression “located at a place of X minutes on foot” to the expression “gets walking when X minutes” having the same meaning is stored in the storage unit 103 as a paraphrase table, 112c receives sentence 4 "This shop is a long-established Italian restaurant located 10 minutes on foot from Yokohama Station" and stores it using the expression "located 10 minutes on foot" as a key. Part 103 is searched, the above rule is obtained, and according to this rule, sentence 4 is rephrased as “This store is a long-established Italian restaurant, and it will arrive after walking 10 minutes from Yokohama station”. It transmits to the feature element extraction part 113 as a variation. Two or more paraphrase expressions may be prepared for one input sentence and stored in the paraphrase table. The paraphrase expression may be other than the character string pattern. For example, it may be a morpheme pattern or a dependency pattern, and is not particularly defined.

  According to the present modification, a summary with high content coverage and high readability can be generated as in the first embodiment and the first modification. In particular, a summary of the same content can be created with a shorter number of characters to paraphrase another expression having the same meaning.

[Modification 3]
Only parts different from the first embodiment will be described with reference to FIGS.

  The conversion sentence generation unit 112 includes a shortened sentence generation unit 112a, a division unit 112b, and a paraphrase unit 112c. In the first embodiment, the first modification, and the second modification, each unit performs the conversion process independently, but in this modification, each unit performs the conversion process in parallel. The abbreviated sentence generating unit 112a, the dividing unit 112b, and the paraphrase unit 112c obtain variations of the first converted sentence, second converted sentence, and third converted sentence, respectively (s112a, s112b, and s112c). And transmitted to the feature element extraction unit 113. The processing of each part is as described in the first embodiment, the first modification, and the second modification.

  By adopting such a configuration, the same effects as those of the first embodiment, the first modification, and the second modification can be obtained. Furthermore, it is possible to obtain more conversion sentence variations for the input sentence, and to search important sentence permutations based on more variations, thus generating a more comprehensive and readable summary. can do. In addition to combining important sentence extraction and sentence shortening, instead of sentence shortening, combining each sentence with a synonym for a shorter expression or shorter sentence division, or combining sentences with important sentence extraction To generate a summary that is easier to read and covers the original information.

  The converted sentence generation unit 112 may include any two of the abbreviated sentence generation unit 112a, the division unit 112b, and the paraphrase unit 112c.

[Modification 4]
Only parts different from the first embodiment will be described with reference to FIGS. 19 and 20.

  The conversion sentence generation unit 212 includes a shortened sentence generation unit 112a, a division unit 112b, and a paraphrase unit 112c. In the first embodiment, the first modification, and the second modification, each unit performs the conversion process independently, but in this modification, each unit performs the conversion process in a column (s212). For example, the abbreviated sentence generation unit 112a obtains a variation of the first converted sentence using the input sentence (s112a). The dividing unit 112b receives the variation of the first conversion sentence obtained by the shortened sentence generation unit 112a instead of the input sentence, and uses this to obtain the variation of the second conversion sentence (s112b). Further, the paraphrase unit 112c receives a variation of the second conversion sentence instead of the input sentence, and uses this to obtain a third conversion sentence variation (s112c). Then, the variation of the third conversion sentence is transmitted to the feature element extraction unit 113.

  By adopting such a configuration, the same effects as those of the first embodiment, the first modification, and the second modification can be obtained. Furthermore, the abbreviated sentence creating unit 112a can obtain many variations of the converted sentence for the input sentence, and the dividing unit 112b can obtain a highly readable variation of the converted sentence. A summary of the same content can be created with a shorter number of characters. Since important sentence permutations can be searched based on these sentences, it is possible to generate a summary with higher content coverage and readability.

  In addition, it is good also as a structure provided with any two among the shortening sentence production | generation parts 112a, the division | segmentation part 112b, and the paraphrase part 112c, and you may cascade-connect each part in what order.

[Modification 5]
Only parts different from the first embodiment will be described. The abbreviated sentence generation unit 112a generates a variation of the abbreviated sentence using the method described in Reference 3. In Reference 3, the score of each subtree is defined as follows based on the phrase coverage rate and the phrase connection probability, and the shortened sentence of the sentence whose summary is the subtree rooted at the predicate with the highest score And In this modification, not only the subtree having the predicate with the highest score as the root but also the top N items are obtained as variations of the abbreviated sentence.

Here, X represents a sentence to be summarized, G (X) represents a set of subtrees in the sentence X, W represents a subtree, and node (W) represents the number of clauses constituting the subtree W. P _imp (w _i ) represents the phrase coverage rate,

It is. Here, X represents the entire sentence, w _i and w _j represent phrases, t _k represents a word included in the phrase w _i , and t _l represents a word included in the phrase w _j . By considering the phrase coverage rate, the score of the subtree that includes many phrases including important words that cover the contents of the sentence increases. P _adj (w _i | w _i−1 ) represents the phrase connection probability,

It is. However, f _i represents the main word of the content word of the phrase, and f _i−1 represents the main word of the function word of the phrase. The main word of the content word of a phrase refers to the last word in the content word string existing in the phrase. Moreover, the main word of the function word of a clause refers to the last one excluding symbols such as punctuation marks in a function word string existing in a clause. However, if there is no function word such as adverb clause, it becomes the same word as the main part of the content. The phrase connection probability represents the likelihood that each phrase is adjacent, and is obtained from the maximum likelihood estimation from the corpus.

  By adopting such a configuration, the same effect as in the first embodiment can be obtained.

[Other variations]
The input text data may be one that has not been subjected to morphological analysis, specific expression extraction, or dependency analysis. In that case, the text summarizing device may be provided with a conventional morphological analysis unit, a specific expression extraction unit, a dependency analysis unit, and the like as necessary.

  A feature element does not necessarily have to be a morpheme unit, and may be a unit that can constitute a feature, such as a phrase.

  The content score calculation unit 115 may use a unit different from the feature element as a sentence element (for example, a word or a specific expression). In that case, instead of using the output of the feature element extraction unit 113 as an input, the sentence element extraction unit provided therein extracts a sentence element from the input text data.

  The sentence element score and the weight parameter may be obtained using other methods, or those obtained in advance by another apparatus may be stored in the storage unit 103.

  The important sentence permutation search unit 119 can also devise other ways to reduce redundancy. According to Equation (1) and Equation (4), if the same content word or information extraction result is included in S, they are added to the content score many times. However, in general, it is not preferable that the same information appears in the summary over and over, so the same content word or information extraction result is added to the content score only once, so that the same information is included in the summary. It can be prevented from being included many times. This contrivance reduces the possibility that a plurality of conversion sentences generated from the same input sentence are included in the summary at the same time.

The connection score calculation unit 117 calculates the connection score w ^T Φ (x, y) of a sequence of three or more sentences. The connection score calculation unit 117 calculates only the connection score Connect (sj | si) of two sentences. calculated, and outputs, connecting score sequence of three or more statements in the key sentence permutation search unit ^{119 w T Φ (x, y} ) may be configured to seek.

The connection score calculation unit 117 obtains a connection score w ^T Φ (x, y) of a sequence of three or more sentences from the connection score Connect (sj | si) of two sentences. It is good also as a structure which calculates | requires the connection score from a sentence, and calculates | requires the connection score of the sequence of sentences using the connection score. For example, Connect (sn | si, s (i + 1),..., S (n-1)), and the connection when the sentence sn follows the sentence si, s (i + 1),. Find the score. In this case, a weight parameter or the like is appropriately set in accordance with this.

  When calculating the weight parameter or using the beam search in the important sentence permutation search unit 119, the value of b may be appropriately set in consideration of the calculation performance of the text summarizing device (for example, b = 1). ~ 1000). The number may be changed depending on the time point i. For example, it may be configured to decrease b as i increases. As i increases, the number of hypotheses that can be developed from one hypothesis decreases, and the amount of computation can be adjusted. Further, the value of b is not a constant, and may be changed according to the number Q of sentences included in the text as a summary source, for example, b = Q × 0.1.

  Further, when calculating the weight parameter, the important sentence permutation search unit 119 does not necessarily use the beam search and the dynamic programming. Further, even if only one of them is used, it is possible to improve efficiency. Moreover, you may obtain | require a high score efficiently using another method.

  The summary length may be obtained by a summary length determination unit (not shown) and input to the important sentence permutation search unit 119. The summary length determination unit receives text data as a summary source or its size (kilobytes) via the input unit 131, and determines the summary length according to the size. For example, the summary length may be automatically determined so that the summarized text data has a size that is 5% to 20% or less of the size of the text data that is the summarization source.

<Text Summarization Device 200>
A text summarizing apparatus 200 according to the second embodiment will be described with reference to FIGS. 1 and 21. It differs from the text summarization apparatus 100 in the configuration of the connection score calculation unit 217 and the data stored in the storage unit 203.

<Connection score calculation unit 217>
The concatenated score calculation unit 217 does not include the feature vector generation unit 117a, includes only the calculation unit 217b, and calculates the concatenation score as Mirella Lapata, “Probabilistic Text Structuring: Experiments with Sentence Ordering”, In Proceedings of the 41st Annual Meeting of the 41 Calculation is performed using the method described in Association for Computational Linguistics (ACL), Association for Computational Linguistics, 2003, pp.545-552. In that case, as an example, the connection score can be defined as follows.

  fik is the kth feature element of the sentence si, and fjm is the mth feature element of the sentence sj. This corresponds to the feature elements extracted from the sentences si and sj in FIG. p (fjm | fik) is a probability that the feature element fjm appears in a state where the feature element fik is given. According to Equation (5), the concatenated score calculation unit 217 uses each conditional probability p (fjm | fik) of the direct product set (si, sj) of feature elements of each sentence as the concatenated score of the sentence si and the sentence sj. ) Is obtained by normalizing the total product of) with the concentration of the Cartesian product set. As an example, p (fjm | fik) can be calculated as follows.

Here, C (fik, fjm) is the number of times the feature element fik and the feature element fjm appear in the preceding sentence in the training data τ described above, the feature element fik appears in the previous sentence, and the feature element fjm appears in the subsequent sentence. is there. The denominator is a number in which the feature element fik appears in the training data τ. When using Equation (5) as the connection score, for example, a weight parameter calculation unit (not shown) counts each number using the training data τ, obtains each conditional probability according to Equation (6), and uses it as a weight parameter. And stored in the storage unit 203. FIG. 21 shows an example of the estimated conditional probability. The left side of the feature column corresponds to the feature element fik, the right side corresponds to the feature element fjm, and the conditional probability column corresponds to p (fjm | fik).

  By adopting such a configuration, the same effect as in the first embodiment can be obtained.

100, 200 Text summarization device 103, 203 Storage unit 112 Conversion sentence generation unit 112a Abbreviated sentence generation unit 112b Division unit 112c Paraphrase unit 113 Feature element extraction unit 115 Content score calculation unit 117, 217 Concatenated score calculation unit 119 Important sentence permutation search Part

Claims (9)

A text summarization device that summarizes text data composed of one or more input sentences,
A weighting parameter for a feature that is a combination considering the order of two feature elements, and a storage unit that stores in advance a sentence element score for a sentence element constituting the sentence;
A conversion sentence generation unit that converts input sentences and generates one or more conversion sentences for each input sentence;
A feature element extraction unit that extracts a feature element from the converted sentence;
Using the sentence element score for the sentence element included in each converted sentence, a content score calculating unit for obtaining a content score of each converted sentence;
Using the feature element extracted by the feature element extraction unit and the weight parameter, a connection score calculation unit that obtains a connection score of a converted sentence;
An important sentence permutation search unit that searches for a permutation of a converted sentence that is the maximum value or a sum of the content score and the connection score, or an approximate value of the maximum value;
A text summarization device. The text summarization device according to claim 1,
Further, the storage unit stores the probability of word connection as a language model,
Furthermore, the conversion sentence generation unit includes an abbreviated sentence generation part that generates one or more abbreviated sentences for each input sentence using the input sentence.
A text summarization device. The text summarization device according to claim 1 or 2,
Further, the storage unit stores a division rule for dividing a compound sentence and a heavy sentence into a single sentence,
Further, the conversion sentence generation unit includes a division unit that divides the input sentence into simple sentences when the input sentence is a compound sentence and a heavy sentence with reference to the division rule.
A text summarization device. The text summarization device according to any one of claims 1 to 3,
Further, the storage unit stores a paraphrase table that is a rule for paraphrasing a certain expression into another expression having the same meaning that is shorter.
Furthermore, the conversion sentence generation unit includes a paraphrase part that refers to the paraphrase table and paraphrases the expression of the input sentence into another expression having the same meaning.
A text summarization device. A text summarization method for summarizing text data composed of one or more input sentences,
A weight parameter for a feature that is a combination considering the order of two feature elements, and a sentence element score for a sentence element constituting a sentence are stored in advance,
A conversion statement generation step in which a conversion statement generation unit converts the input statement and generates one or more conversion statements for each input statement;
A feature element extraction unit for extracting a feature element from the conversion sentence;
A content score calculation unit that calculates a content score of each converted sentence using a sentence element score for a sentence element included in each converted sentence;
A concatenation score calculation unit uses the feature element extracted in the feature element extraction step and the weight parameter to calculate a concatenation score of the converted sentence;
The important sentence permutation search unit has an important sentence permutation search step of searching for a permutation of a converted sentence that has a maximum value or an approximate value of the maximum value as a sum of the content score and the connection score.
A text summarization method characterized by that. The text summarization method according to claim 5,
Remember the probability of word connections as a language model,
Further, the conversion sentence generation step includes a short sentence generation step in which the short sentence generation unit generates one or more short sentences for each input sentence using the input sentence.
A text summarization method characterized by that. A text summarization method according to claim 5 or 6, comprising:
A division rule for dividing compound sentences and heavy sentences into simple sentences is stored in advance.
Further, the conversion sentence generation step includes a division step in which the dividing unit divides the input sentence into simple sentences when the input sentence is a compound sentence and a heavy sentence with reference to the division rule.
A text summarization method characterized by that. A text summarization method according to any one of claims 5 to 7, comprising:
Preliminarily store a paraphrase table that is a rule for paraphrasing a certain expression into a shorter other expression having the same meaning,
Further, the conversion sentence generation step includes a paraphrase step in which the paraphrase unit refers to the paraphrase table and paraphrases the expression of the input sentence to another expression having the same meaning.
A text summarization method characterized by that.   A text summarization program for causing a computer to function as the text summarization device according to claim 1.

Images