JP2012113422A - Document processing apparatus, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To support an editor of an article described by a joint support system by automatically extracting, from one of sentences in different languages, new information which is not contained in the other sentence.SOLUTION: An article (each sentence e) in English and an article (each sentence c) in Chinese are translated into the same language, similarity between each sentence e' of the article in English after the translation and each sentence c' of the article in Chinese after the translation is computed for all combinations of e' and c', N pieces of sentences are extracted from among sentences of which the maximum similarity is minimum, in the sentences e', and a flag "+1" indicating a sentence including new information is given to the corresponding English sentence before the translation. A graph is created in which a relationship between nodes for the English and Chinese sentences is represented as a weighted edge corresponding to the similarity. Then, a label 1 is given to a node representing the English sentence with the flag "+1" and 0 is given to the others. A position before or after a Chinese sentence corresponding to a node with a maximum label given through label propagation is determined as an insertion position.

Description

  The present invention relates to a document processing apparatus, method, and program, and in particular, document processing for comparing sentences written in two different languages and extracting new information not included in the other sentence from one sentence. The present invention relates to an apparatus, a method, and a program.

  In recent years, free encyclopedias on the web such as Wikipedia have attracted attention. Wikipedia employs a co-authoring system where articles are described with the help of many editors, and there are multiple language versions of an article, with each language version maintained by a different group of users. Yes. For this reason, these different language versions of the articles do not necessarily have the same content even if they are articles related to the same item. Thus, an editor may obtain information from articles in other language versions in order to ensure that articles in one language version are composed of correct or latest information.

  Several systems have been proposed to assist editors of such co-writing systems. For example, a system for matching different language versions of infoboxes in Wikipedia has been proposed (see Non-Patent Document 1). Also, an algorithm for inserting new information into existing text in consideration of the hierarchical structure of articles has been proposed (see Non-Patent Document 2). In addition, a method for collecting information on the web and automatically generating a simple Wikipedia article has been proposed (see Non-Patent Document 3).

Eytan Adar, Michael Skinner, and Daniel S. Weld. 2009. Information arbitrage across multi-lingual Wikipedia. In Proceedings of the Second ACM international Conference on Web Search and Data Mining, 94-103. Erdong Chen, Benjamin Snyder, and Regina Barzilay. 2007. Incremental Text Structuring with Online Hierarchical Ranking. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Conference on Computational Natural Language Learning 2007, 83-91. Christina Sauper and Regina Barzilay. 2009. Automatically generating Wikipedia articles: a structure-aware approach. In Proceedings of the 47th Annual Meeting of the ACL, 208-216.

  However, the technique described in Non-Patent Document 1 is a technique that handles items of a predetermined format called infoboxes, and has a problem that it cannot be applied to the text of articles.

  Further, the techniques described in Non-Patent Documents 2 and 3 have a problem that only articles in a single language are targeted, and articles written in different languages are not considered.

  The present invention has been made in view of the above problems, by comparing sentences written in different languages, and automatically extracting new information not included in the other sentence from one sentence, It is an object of the present invention to provide a text processing apparatus, method, and program capable of supporting an editor of an article described by a joint support system.

  In order to achieve the above object, a sentence processing apparatus according to the present invention includes a first sentence including one or more sentences written in a first language so that two kinds of sentences written in the same language can be obtained. One or both of the third sentence translated from the first sentence and the fourth sentence translated from the second sentence including one or more sentences described in a second language different from the first language And when the third sentence is obtained by the translation means, each sentence included in the third sentence and each sentence included in the second sentence or obtained by the translation means The degree of similarity with each sentence included in the fourth sentence is determined by comparing each sentence included in the third sentence with each sentence included in the second sentence or each sentence included in the fourth sentence. When all combinations are calculated and the third sentence is not obtained by the translation means The similarity between each sentence included in the first sentence and each sentence included in the fourth sentence indicates the degree of similarity between each sentence included in the first sentence and each sentence included in the fourth sentence. Similarity calculation means for calculating all combinations with the above, and when the third sentence is obtained by the translation means, the similarity calculated by the similarity calculation means for each sentence of the third sentence The maximum similarity is calculated so that the maximum similarity is not more than a predetermined threshold, and the sentence of the first sentence corresponding to the sentence of the third sentence whose maximum similarity is not more than a predetermined threshold is not included in the second sentence When the sentence is extracted as a sentence including new information and the third sentence is not obtained by the translation unit, the maximum similarity is obtained for each sentence of the first sentence, and the maximum similarity is a predetermined threshold value. The sentence of the first sentence that is the following is not included in the second sentence It is configured to include the new information extraction means, the extracting as a statement that contains the new information.

  According to the sentence processing apparatus of the present invention, first, the translation means includes a first sentence including a plurality of sentences described in the first language so that two kinds of sentences described in the same language can be obtained. One or both of the third sentence translated from, and the fourth sentence translated from the second sentence including a plurality of sentences described in a second language different from the first language are obtained. When the similarity calculation unit obtains the third sentence by the translation unit, that is, when the first sentence is translated, each sentence included in the third sentence and the second sentence are included. Similarity between each sentence or each sentence included in the fourth sentence obtained by the translation means is included in each sentence included in the third sentence and each sentence or fourth sentence included in the second sentence Calculate for all combinations with each sentence. Further, when the third sentence is not obtained by the translation means, that is, when the first sentence is not translated, each sentence included in the first sentence, each sentence included in the fourth sentence, Are calculated for all combinations of each sentence included in the first sentence and each sentence included in the fourth sentence.

  Next, when the new information extraction unit obtains the third sentence by the translation unit, the maximum similarity that maximizes the similarity calculated by the similarity calculation unit is obtained for each sentence of the third sentence. Obtaining a sentence of the first sentence corresponding to the sentence of the third sentence having the maximum similarity equal to or less than a predetermined threshold as a sentence including new information not included in the second sentence; If the third sentence is not obtained by the above, the maximum similarity is obtained for each sentence of the first sentence, and the sentence of the first sentence in which the maximum similarity is equal to or less than a predetermined threshold value is obtained. Extracted as a sentence containing new information not included in the sentence.

  In this way, one or both of sentences written in different languages are translated to obtain two kinds of sentences written in the same language, and each of the two kinds of sentences in the same language is obtained. The similarity of each sentence is calculated for all combinations, and the sentence of the first sentence or the sentence of the first sentence corresponding to the third sentence whose maximum similarity is the predetermined threshold value or less. Is automatically extracted as a sentence including new information that is not included in the second sentence, so that the editor of the article described by the joint support system can be supported.

  The sentence processing apparatus of the present invention includes a feature extraction unit that extracts a feature from each sentence of the first sentence, each feature of each sentence of the first sentence extracted by the feature extraction unit, Whether each sentence of the first sentence contains new information based on the feature of each sentence of the first sentence using the extraction result of each sentence of the first sentence by the new information extraction unit Based on the classification result obtained by inputting each sentence of the first sentence to the classifier learned by the classifier learning means, classifier learning means for learning a classifier for identifying whether or not, Re-extracting means for re-extracting a sentence including the new information. Thereby, the extraction precision of the sentence containing new information can be improved.

  Further, the sentence processing apparatus of the present invention includes a node representing each sentence of the first sentence and the second sentence, and an edge to which a relationship between the nodes is weighted according to a similarity between the nodes. And a label based on whether or not the sentence includes the new information is given to a node corresponding to each sentence of the first sentence, and the label obtained by the label propagation method using the graph The second corresponding to the node given the label closest to the label given to the node corresponding to the sentence containing the new information based on the label given to the node corresponding to each sentence of the second sentence It is possible to further comprise determining means for determining before or after the sentence of the sentence as a position to insert the sentence including the new information. Thus, by automatically determining a position suitable for inserting a new sentence, it is possible to assist the editor of the article described by the joint support system.

  Further, the sentence processing method of the present invention translates the first sentence including one or more sentences described in the first language so that two kinds of sentences described in the same language can be obtained. 3 or a fourth sentence translated from a second sentence including one or more sentences described in a second language different from the first language, and the third sentence Is obtained, the degree of similarity between each sentence included in the third sentence and each sentence included in the second sentence or each sentence included in the fourth sentence is expressed as When all the combinations of each sentence included in the third sentence and each sentence included in the second sentence or each sentence included in the fourth sentence are calculated and the third sentence is not obtained Includes the similarity between each sentence included in the first sentence and each sentence included in the fourth sentence, When all the combinations of each sentence included in the first sentence and each sentence included in the fourth sentence are calculated and the third sentence is obtained, each sentence of the third sentence For the above, the maximum similarity that maximizes the similarity is obtained, and the sentence of the first sentence corresponding to the sentence of the third sentence that has the maximum similarity equal to or less than a predetermined threshold is set as the second sentence. When it is extracted as a sentence including new information that is not included and the third sentence is not obtained, the maximum similarity is obtained for each sentence of the first sentence, and the maximum similarity is a predetermined value. In this method, a sentence of the first sentence that is equal to or less than a threshold is extracted as a sentence including new information that is not included in the second sentence.

  The sentence processing method of the present invention further extracts features from each sentence of the first sentence, and extracts each feature of each sentence of the first sentence and each of the first sentence. Identify whether each sentence of the first sentence contains new information based on the characteristics of each sentence of the first sentence using an extraction result indicating whether the sentence contains new information A classifier for learning may be learned, and a sentence including the new information may be re-extracted based on an identification result obtained by inputting each sentence of the first sentence to the learned classifier. .

  Further, the sentence processing method of the present invention further assigns a weight corresponding to the degree of similarity between the nodes and the node representing each sentence of the first sentence and the second sentence. And a label based on whether or not the sentence includes the new information is assigned to a node corresponding to each sentence of the first sentence, and is obtained by a label propagation method using the graph. Based on the label given to the node corresponding to each sentence of the second sentence, the node corresponding to the node assigned the label closest to the label given to the node corresponding to the sentence containing the new information You may make it determine before or after the sentence of a 2nd sentence as a position which inserts the sentence containing the said new information.

  The sentence processing program of the present invention is a program for causing a computer to function as each means constituting the sentence processing apparatus.

  As described above, according to the sentence processing apparatus, method, and program of the present invention, sentences described in different languages are translated into the same language, and each sentence of the first translated sentence and the second translated sentence are translated. Similarity with each sentence is calculated for all combinations, and the sentence of the first sentence corresponding to the sentence of the first translation sentence in which the maximum similarity that maximizes the similarity is equal to or less than a predetermined threshold is By automatically extracting as a sentence including new information that is not included in the second sentence, it is possible to support the editor of the article described by the joint support system.

It is a block diagram which shows the functional structure of the text processing apparatus of 1st Embodiment. It is an image figure for demonstrating the graph preparation for a label propagation method. It is a flowchart which shows the content of the text processing routine in the text processing apparatus of 1st Embodiment. It is a flowchart which shows the content of the new information identification process routine in the text processing apparatus of 1st Embodiment. It is a flowchart which shows the content of the information insertion position search processing routine in the text processing apparatus of 1st Embodiment. It is a block diagram which shows the functional structure of the text processing apparatus of 2nd Embodiment. It is a flowchart which shows the content of the new information identification process routine in the text processing apparatus of 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the first embodiment, an article written in Chinese is compared with an article written in English (first sentence) by comparing an article written in English with an article written in Chinese. A case will be described in which a sentence including new information not included in the second sentence) is extracted and an appropriate position for inserting a sentence including new information at an appropriate position of an article written in Chinese is described.

  The text processing apparatus 10 according to the first embodiment is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a text processing routine described later. Further, an HDD as a storage unit may be included. As shown in FIG. 1, the computer can be functionally represented by a configuration including a data reading unit 12, a new information identification unit 20, and an information insertion position search unit 40.

The data reading unit 12 reads an article written in English and an article written in Chinese from an internal or external storage device. Or you may make it read the article memorize | stored in the external device via a network. The article written in English read here is a sentence composed of L sentences (e ₁ , e ₂ ,..., E _L ), and the article written in Chinese is M articles. It is assumed that the sentence is composed of sentences (c ₁ , c ₂ ,..., C _M ).

  The new information identification unit 20 can be expressed by a configuration including a preprocessing unit 22, a machine translation unit 24, a similarity calculation unit 26, and a flag assignment unit 28.

  The pre-processing unit 22 performs processing for recognizing a section and paragraph, stem processing, etc. for each of an article written in English (first sentence) and an article written in Chinese (second sentence). Pre-processing is performed.

  The machine translation unit 24 translates one or both of the pre-processed article written in English and the article written in Chinese into two kinds of articles written in the same language. obtain. The translated language may be one of the languages of the read article (English or Chinese), or may be a third language different from the language of the read article. For example, when the translated language is English, only the article (second sentence) written in Chinese is translated into English. As a result, the article (first sentence) written in English and the article (fourth sentence) translated from the article written in Chinese become articles written in the same language (English). Similarly, when the translated language is Chinese, only the article (first sentence) written in English is translated into Chinese. Thus, an article (third sentence) translated from an article written in English and an article (second sentence) written in Chinese become articles written in the same language (Chinese). If the translated language is the third language (for example, Japanese), the article written in English (first sentence) and the article written in Chinese (second sentence) Translate both. As a result, an article translated from English (third sentence) and an article translated from Chinese (fourth sentence) are written in the same language (Japanese). Become an article.

In the present embodiment, L articles (e ₁ ′, e ₂ ′, e ₂ ′, e ₂ ′, e. ···, e _L ') and expressed, in the case dealing with articles written in Chinese as articles that were written in the same language, M number of statements _{(c 1', c 2 '} , ··· , referred to as _{c M} '). That is, when an article written in English is not translated, (e ₁ , e ₂ ,..., E _L ) = (e ₁ ′, e ₂ ′,..., E _L ′) If the article written in Chinese is not translated, (c ₁ , c ₂ ,..., C _M ) = (c ₁ ′, c ₂ ′,..., C _M ′) is there. Hereinafter, in this embodiment, an “post-translation” article is an article in which two articles written in different languages are written in the same language regardless of whether or not one of the articles is translated. Refers to the article after becoming.

  For translation in the machine translation unit 24, a known machine translation tool can be used. The machine translation tool only needs to be able to convert an article written in two different languages into the same language, and is not limited to a special machine translation tool.

The similarity calculation unit 26 translates each sentence (e ₁ ′, e ₂ ′,...) Of an article (hereinafter referred to as “translated English article”) obtained by translating an article written in English by the machine translation unit 24. , E _L ′) and articles (c ₁ ′, c ₂ ′) of articles (hereinafter referred to as “translated Chinese articles”) translated by the machine translation unit 24 from articles written in Chinese .., c _M ') and the similarity is calculated. First, each sentence of an English article after translation is represented by a vector composed of word weights, where n is the size of the vocabulary (the number of elements in the union of words appearing in two articles). Specifically, the vector representation of the i-th sentence e _i ′ of the English article after translation is represented by T ^e _i and represented by T ^e _i = (w ₁ , w ₂ ,..., W _n ). Similarly, for translated Chinese articles, the vector representation of the j-th sentence c _j ′ is represented by a vector composed of word weights with T ^c _j . Here, w _i is a word weight obtained by an algorithm for extracting characteristic words in a document, and for example, using a TF-IDF algorithm used in fields such as information retrieval and document summarization. Obtainable.

Next, using the vectors T ^e _i and T ^c _j described above, the similarity between the two sentences e _i ′ and c _j ′ is calculated. The similarity can be calculated, for example, as a cosine similarity in a vector space as shown in the following equation (1). Similar for all combinations (i = 1, 2,..., L, j = 1, 2,..., M) of e _i ′ (T ^e _i ) and c _j ′ (T ^c _j ) Calculate the degree.

  The calculation of similarity is not limited to the above method, and other methods can be used. For example, the similarity can be calculated based on whether two sentences contain the same hyperlink. As for the weight of the word, other methods such as giving a high weight to a word representing a person, place, or event may be used.

Flag addition module 28, based on the calculated degree of similarity by the similarity calculation unit 26, the new information (information that is not included in the Chinese article) each statement e _i article written in English include A flag indicating whether or not the file is present is assigned. Specifically, among all similarities with each sentence c _j 'of the translated Chinese article calculated for each sentence e _i ' of the translated English article, the maximum with the highest similarity _{Find similarities} , extract N sentences (e _min1 ′, e _min2 ′,..., E _minN ′) from sentences with the smallest maximum similarity, and describe in English before translation corresponding to the extracted sentences A flag “+1” indicating that the sentence includes new information is _{added to} the sentence (e _min1 , e _min2 ,..., E _minN ) of the article. Other sentences are given a flag “−1” indicating that no new information is included. This is based on the assumption that English sentences with low similarity to any sentence in Chinese articles are likely to contain new information.

  Here, as an example of extracting sentences whose maximum similarity is equal to or less than a predetermined threshold, a maximum of similarities are arranged in ascending order and up to a predetermined number of sentences are extracted as sentences including new information (a predetermined number of sentences). However, it is also possible to extract a sentence whose maximum similarity is equal to or less than the threshold value as compared to a predetermined threshold value.

  The information insertion position search unit 40 inserts the sentence into an article written in Chinese for an English sentence to which a flag “+1” indicating a sentence including new information is added by the flag assigning unit 28. Search for the most suitable position. To solve this problem, the label propagation method is used.

Specifically, for example, as shown in FIG. 2, each sentence of an article written in English and each sentence of an article written in Chinese, and the relationship between the nodes A graph G = (V, E) having an edge weighted according to the similarity is created. V is a set of nodes, which includes L English sentences (e ₁ , e ₂ ,..., E _L ) and M Chinese sentences (c ₁ , c ₂ ,..., C _M ). To express. E is a set of edges. Between English node and Chinese node exists L × M pieces of edge, the weight of that edge, the statement after the translation sentence e _i English e _{i 'and} the sentence c _j in Chinese The cosine similarity with the translated sentence c _j 'is expressed. An edge is also formed between English nodes or Chinese nodes. For example, if two Chinese sentences c _i and c _j are in the same paragraph, an edge is formed between nodes of c _i and c _j . The weight of this edge is calculated by w _ij = 1 / dist (c _i , c _j ). dist is the distance between c _i and c _j (the number of sentences in between). This graph can represent information about all similar links and document structure.

  Next, in order to initialize the graph, the label “1” is assigned to the node representing the English sentence to which the flag “+1” is assigned, and the label “0” is assigned to the other English nodes. At this stage, the Chinese node label is undecided. Then, according to the label propagation method, the label given to the English node is propagated to the node (Chinese node) whose label is undetermined according to the relationship between the nodes. This can be regarded as executing a Markov chain on the graph created as described above.

  In the label propagation method, a value given to a node whose label is undetermined can be calculated by iterative Markov chain computation or direct eigenvector computation. When direct eigenvalue vector calculation is used, a value to be given is obtained by solving the following equation (2).

Here, f is a vector of (N + M) -dimensional labels. f is limited to “1” or “0” in an English node, and is undecided in a Chinese node. The objective function that if implementing the label propagation by the edge of the weight w _ij is larger if the node pair (i, j) is forced to have a label f _i and f _j similar. f can be calculated by obtaining the eigenvector of the graph Laplacian.

  After propagating the label, each Chinese node is given a numeric label between [0, 1]. As described above, since a similar label is assigned between nodes having a large edge weight, an English sentence corresponding to the node assigned the label “1” and a label having a value closest to the label “1” are displayed. This indicates that the Chinese sentence corresponding to the given node is deeply related. Therefore, the node with the label closest to the label “1”, here the position before or after the Chinese sentence corresponding to the node with the maximum label, the sentence containing the new information. It can be determined as the optimal position to insert.

  Next, a sentence processing routine executed in the sentence processing apparatus 10 according to the first embodiment will be described with reference to FIG.

  In step 100, a new information identification process is performed for extracting a sentence including new information not included in the second sentence from the first sentence.

  Here, the new information identification processing routine will be described with reference to FIG. Here, a case will be described in which a sentence including new information that is not included in an article (second sentence) written in Chinese is extracted from an article (first sentence) written in English.

In step 120, an article written in English and an article written in Chinese are read from an internal or external storage device. Or you may make it read the article memorize | stored in the external device via a network. The article written in English read here is a sentence composed of L sentences (e ₁ , e ₂ ,..., E _L ), and the article written in Chinese is M articles. It is assumed that the sentence is composed of sentences (c ₁ , c ₂ ,..., C _M ).

  Next, in step 122, preprocessing such as processing for recognizing sections and paragraphs and word stem processing is performed on each of the article written in English and the article written in Chinese read in step 120.

Next, in step 124, each of the article written in English and the article written in Chinese pre-processed in step 122 is translated into the same language. Here, articles written in English are L sentences (e ₁ ′, e ₂ ′,..., E _L ′), and articles written in Chinese are M sentences (c ₁ ′). , C ₂ ′,..., C _M ′).

Next, in step 126, the i-th sentence e _i ′ of each sentence (e ₁ ′, e ₂ ′,..., E _L ′) of the translated English article translated in step 124 above, the size of the vocabulary Satoshi n, vector ^T _e i ₌ consists weight _{w i} of the word of each sentence obtained using TF-IDF algorithm or the like _{(w 1, w 2, ···} , w n) is expressed by. Similarly, for a translated Chinese article, the j-th sentence c _j ′ is expressed by a vector T ^c _j . Then, the similarity between the two sentences e _i ′ and c _j ′ is set as, for example, a cosine similarity in a vector space as shown in equation (1), and e _i ′ (T ^e _i ) and c _j ′ (T ^c _j )) for all combinations (i = 1, 2,..., L, j = 1, 2,..., M).

Next, in step 128, among all the similarities with each sentence c _j 'of the translated Chinese article calculated for each sentence e _i ' of the translated English article in step 126 above. Find the maximum similarity with the highest similarity. Then, N sentences (e _min1 ′, e _min2 ′,..., E _minN ′) are extracted from the sentence having the smallest maximum similarity, and described in English before translation corresponding to the extracted sentence. The flag “+1” indicating that the sentence includes new information is _{added to} the sentence (e _min1 , e _min2 ,..., E _minN ) of the published article, and new information is _added to the other English sentences. A flag “−1” indicating that it is not included is assigned and the process returns.

  Returning to the sentence processing routine (FIG. 3), the process proceeds to step 102, and an information insertion position search process for searching for a position for inserting a sentence including the extracted new information is executed.

  Here, the information insertion position search processing routine will be described with reference to FIG.

In step 140, each sentence of the article written in English (e ₁ , e ₂ ,..., E _L ) and each sentence of the article written in Chinese (c ₁ , c ₂ ,..., A graph G = (V, E) having nodes representing c _M ) and edges weighted according to the degree of similarity between the nodes is created.

  Next, in step 142, in order to initialize the graph created in step 140, the label “1” is added to the node representing the English sentence to which the flag “+1” is added in the new information identification process (FIG. 4). And a label “0” is assigned to other English nodes.

  Next, in step 144, the label is propagated from the English node to which the label is assigned to the Chinese node whose label is undetermined. By the label propagation, the Chinese node is given a numeric label between [0, 1].

  Next, in step 146, the position before or after the Chinese sentence corresponding to the node whose label value given to the Chinese node by the label propagation in step 144 is the maximum is included with the new information. Determine the best position to insert the sentence and return.

  Returning to the sentence processing routine (FIG. 3), the process proceeds to step 104, where the data at the position where the sentence including the new information determined by the information insertion position search process is inserted is output, and the process ends. Here, the flag information given in the new information identification process, that is, the sentence data including the new information has been described as being transferred to the subsequent information insertion position search process. However, the sentence data including the new information is described. May be output as a processing result without passing on to the subsequent stage.

  As described above, according to the sentence processing apparatus of the first embodiment, sentences (English articles and Chinese articles) written in different languages are translated into the same language, and the first translated sentence The similarity between each sentence of (translated English article) and each sentence of the second translated sentence (translated Chinese article) is calculated for all combinations, and the maximum similarity is the maximum similarity By automatically extracting the sentence of the first sentence corresponding to the sentence of the first translated sentence whose is less than or equal to a predetermined threshold as a sentence including new information not included in the second sentence, joint support It can assist the editor of the articles described by the system.

  In addition, a graph is created in which each sentence is represented by a node and the relationship between the nodes is represented by a weighted edge corresponding to the degree of similarity, and a sentence (English node) corresponding to each sentence of the first sentence includes new information. A label based on whether or not, a label is propagated from a node corresponding to each sentence of the first sentence, and a node corresponding to each sentence of the second sentence whose label value is undetermined (a Chinese node) ), And automatically determining a suitable position to insert a new sentence based on the label value assigned to the node corresponding to each sentence of the second sentence. It can assist the editor of the articles described by the system.

  Next, a second embodiment will be described. In the second embodiment, a case where a sentence including new information is extracted using the classification result of the classifier will be described. A case will be described in which a sentence including new information that is not included in an article written in Chinese (second sentence) is extracted from an article written in English (first sentence).

  As shown in FIG. 6, the sentence processing apparatus 210 according to the second embodiment is different from the sentence processing apparatus 10 according to the first embodiment in the configuration of the new information identification unit. Hereinafter, differences from the first embodiment will be described.

  The new information identification unit 220 in the text processing device 210 according to the second embodiment is configured by adding a feature extraction unit 30 and a classifier to the configuration of the new information identification unit 20 in the text processing device 10 according to the first embodiment. The learning unit 32 and the flag updating unit 34 are added, and the flag providing unit 28 can be represented by a configuration in which the flag adding unit 228 is replaced.

The flag assigning unit 228 has the highest similarity among all the similarities with each sentence c _j 'of the translated Chinese article calculated for each sentence e _i ' of the translated English article. Find the maximum similarity. Each sentence e _i ′ is arranged in descending order of the obtained maximum similarity, and N sentences (e _min1 ′, e _min2 ′,..., E _minN ′) are extracted from the sentence with the smallest maximum similarity and extracted. The flag “+1” indicating that the sentence includes new information is _{added to} the sentence (e _min1 , e _min2 ,..., E _minN ) of the article described in English corresponding to the sentence before translation. Also, N sentences (e _max1 ′, e _max2 ′,..., E _maxN ′) are extracted from the sentence with the maximum similarity, and the article written in English before translation corresponding to the extracted sentence (E _max1 , e _max2 ,..., E _maxN ) is given a flag “−1” indicating that no new information is included. No flag is given to sentences that are not included in the N sentences from the largest or smallest maximum similarity.

Feature extraction unit 30, for each sentence e _i article written in English, and extracts a feature for use in learning classifier in classifier learning unit 32 will be described later. For example, the following features can be extracted.

Similarity: cosine similarity between each sentence e _i ′ after translation of each sentence e _i of the article written in English and each sentence c _j ′ after translation of each sentence c _j of the article written in Chinese Maximum degree of neighbors Similarity of neighborhood: each of the sentences (e _i-1 ′ and e _{i + 1} ′) after translation of the adjacent (immediately and immediately following) sentences e _i-1 and e _{i + 1} and each article written in Chinese sentence c _j of each sentence c _j of post-translational: and _'cosine similarity of maximum entropy of the sentence e _i after the translation of each statement e _i of the described articles in _English', written in Chinese the similarity between each sentence c _{j 'posttranslational} each sentence c _j article below (3) translation of each sentence c _j of articles written in Chinese calculated by converting the conditional probability by formula Entropy of similarity with each subsequent sentence c _j '

  Note that it is not necessary to use all of the above three features, and at least one or more may be used. It can also be used in combination with other features.

Classifier learning unit 32, the feature of each sentence e _i article written in English extracted by the feature extraction unit 30, which by the flag addition module 28 is assigned to each sentence e _i article written in English flag the given to support vector machine (SVM), based on the characteristics of each sentence e _i article written in the inputted English, a result of identifying whether or not contain new information in each sentence e _i (flag ) Is learned. The learning method may be a conventionally known technique, and is not limited to SVM.

Flag update unit 34, the classifier learned by the classifier learning unit 32, and inputs the feature of each sentence e _i article written in English, based on the identification result, the articles that have been written in English each statement e _i, the flag "+1" indicating that the sentence containing the new information, or to impart a flag "-1" indicating that it does not contain any new information. For a sentence that has already been flagged by the flag assigning unit 228, the flag is updated based on the classification result of the classifier.

  Next, a text processing routine executed in the text processing apparatus 210 according to the second embodiment will be described. Since the text processing routine and the new information identification process in the first embodiment are only different, the new information identification process routine in the second embodiment will be described with reference to FIG. In addition, about the process same as the new information identification process of 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

Through steps 120 to 128, an article written in English composed of _L sentences (e ₁ , e ₂ ,..., E _L ), and M sentences (c ₁ , c ₂ ,... - reads the articles that are written in Chinese, which is composed of c _M), in each sentence e _i of the articles that have been written in English, including the new information that has not been included in the article that was written in Chinese Or not flag.

Next, at step 200, for each sentence e _i article written in English, and it extracts a feature for use in learning classifier.

Next, in step 202, the feature of each sentence e _i of the article described in English extracted in step 200 and the flag of each sentence e _i of the article described in English given in step 128 are set to SVM. given to learning based on the characteristics of each sentence e _i article written in the inputted English, a result of identifying whether or not contain new information in each sentence e _i return classifier (the flag) Let

Next, in step 204, the classifier learned in step 202, and inputs the feature of each sentence e _i article written in English that was extracted in step 200, based on the identification result, English A flag “+1” indicating that the sentence includes new information or a flag “−1” indicating that no new information is included is assigned to each sentence e _{i of} the article described in (1). For the sentence to which the flag has already been assigned in the above step 128, the flag is updated with the classifier identification result obtained in this step, and the process returns.

  As described above, according to the sentence processing apparatus of the second embodiment, the feature extracted from each sentence of the first sentence (English article) and the flag indicating whether or not new information is included. Since the classifier is learned using and the flag is updated using the learned classification result of the classifier, it is possible to improve the accuracy of extracting a sentence including new information.

  In the first embodiment and the second embodiment described above, the case where an article written in English and an article written in Chinese are used as articles written in two different languages has been described. However, it can also be applied to articles written in other languages.

  Next, in order to explain the effects of the above embodiment, the following experimental results will be described.

(Experiment 1) Evaluation for Extracting Sentences Containing New Information Nine (A to I) English articles from Wikipedia (selected by Wikipedia editors who are well-thought, mature, and have comprehensive content) Articles) and corresponding Chinese articles were collected.

  Next, the English version of each article was manually compared with the Chinese version, and sentences containing new information were identified from the English articles. In the Chinese version, a flag was added to a sentence that contains the same information as the sentence in an English article.

  Next, in addition to the first embodiment and the second embodiment, the following two methods were compared.

Positive flag: A method of learning a classifier as in the second embodiment and extracting a sentence including new information from the classification result of the classifier, and includes new information when learning the classifier. When SVM is trained using only data with the correct flag of whether or not Random: When a flag indicating whether or not new information is included in an English sentence is randomly assigned Here, AUC (area under the precision- The performance of each method was evaluated by the recall curve (area under the curve). In this evaluation method, it is not necessary to specify the similarity threshold value. Basically, the higher the AUC value at the same recall level, the higher the accuracy. The evaluation results are shown in Table 1 below.

  The first and second embodiments achieved high AUC values of 70 to 95 in most cases. Moreover, the AUC value of the second embodiment is higher than that of the first embodiment as a whole. From this result, it can be seen that new information can be acquired almost automatically.

(Experiment 2) Evaluation for Determination of Insertion Position of Sentence Containing New Information Next, an experiment was performed to evaluate a method for determining the most appropriate insertion position based on the label propagation method. First, for each article, a sentence that matches the Chinese sentence is randomly selected from the English sentence, and these English sentences are evaluated using the fact that the correct position in the Chinese article is known. Went.

As a comparison target, a method using manual association was used. First, assuming that some sentences in an English article are manually associated with sentences in an Chinese article, a corresponding sentence in Chinese containing the same information as an English sentence was found. When a sentence of English, including the new information and e _i, the most appropriate insertion position is thought that after the e _i-1 and match the sentence c _j of the Chinese. If there is no Chinese sentence matching e _i-1 , e _i-2 and e _i-3 were examined. This method requires manual work and is not completely automatic. This method is called a manual alignment-based method.

  The following two variations were used for the label propagation method. The following two variations differ in the method of forming an edge when creating a graph used in the label propagation method.

Paragraph base edge formation method: the same method as the information insertion position search unit 40 of the first and second embodiments Section base edge formation method: information insertion position search unit 40 of the first and second embodiments And a method of forming an edge between two sentences (nodes) in the same section (another example of the first and second embodiments)
Here, in order to evaluate the performance of each method, the following three indicators were used.

Average Distance: Distance between the predicted position and the correct insertion position Section Accuracy: An indicator that reflects whether it was inserted in the correct section Paragraph Accuracy: Reflected whether it was inserted in the correct paragraph Indicators to be tested We selected 30% and 50% sentences from English articles. The results are shown in Table 2 below.

  It was confirmed that the methods (paragraph base edge forming method, section base edge forming method) of the first and second embodiments are superior to the manual alignment method.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10,210 Text processing apparatus 12 Data reading part 20,220 New information identification part 22 Preprocessing part 24 Machine translation part 26 Similarity calculation part 28,228 Flag assignment part 30 Feature extraction part 32 Classifier learning part 34 Flag update part 40 Information insertion position search unit

Claims (7)

A third sentence obtained by translating a first sentence including one or more sentences written in the first language so that two kinds of sentences written in the same language are obtained; and the first language Translation means for obtaining one or both of a fourth sentence obtained by translating a second sentence including one or a plurality of sentences described in a second language different from
When the third sentence is obtained by the translation means, each sentence included in the third sentence, and each sentence included in the second sentence or included in the fourth sentence obtained by the translation means Similarity with each sentence is calculated for all combinations of each sentence included in the third sentence and each sentence included in the second sentence or each sentence included in the fourth sentence. When the third sentence is not obtained by the translating means, the similarity between each sentence included in the first sentence and each sentence included in the fourth sentence is determined as the first sentence. Similarity calculation means for calculating all combinations of each sentence included in the sentence and each sentence included in the fourth sentence;
When the third sentence is obtained by the translating means, a maximum similarity that maximizes the similarity calculated by the similarity calculating means is obtained for each sentence of the third sentence, and the maximum similarity is obtained. The sentence of the first sentence corresponding to the sentence of the third sentence that is less than or equal to a predetermined threshold is extracted as a sentence containing new information not included in the second sentence, and the translation means When the sentence of 3 is not obtained, the maximum similarity is obtained for each sentence of the first sentence, and the sentence of the first sentence in which the maximum similarity is equal to or less than a predetermined threshold value is determined. A new information extracting means for extracting as a sentence including new information not included in the sentence of 2,
A sentence processing apparatus including: Feature extraction means for extracting features from each sentence of the first sentence;
Using each of the features of each sentence of the first sentence extracted by the feature extraction means and the extraction result by the new information extraction means of each sentence of the first sentence, Classifier learning means for learning a classifier for identifying whether each sentence of the first sentence contains new information based on the characteristics of each sentence;
Re-extracting means for re-extracting a sentence including the new information based on an identification result obtained by inputting each sentence of the first sentence to the classifier learned by the classifier learning means;
The sentence processing apparatus according to claim 1, comprising:   Generating a graph having a node representing each sentence of the first sentence and the second sentence, and an edge weighted according to a similarity between the nodes and a relationship between the nodes; The node corresponding to each sentence of one sentence is given a label based on whether or not the sentence includes the new information, and corresponds to each sentence of the second sentence obtained by the label propagation method using the graph. Based on the label given to the node, before or after the sentence of the second sentence corresponding to the node given the label closest to the label given to the node corresponding to the sentence containing the new information, The sentence processing apparatus according to claim 1, further comprising a determining unit that determines a position at which a sentence including the new information is to be inserted. A third sentence obtained by translating a first sentence including one or more sentences written in the first language so that two kinds of sentences written in the same language are obtained; and the first language Obtaining one or both of a fourth sentence translated from a second sentence containing one or more sentences written in a second language different from
When the third sentence is obtained, the similarity between each sentence included in the third sentence and each sentence included in the second sentence or each sentence included in the fourth sentence is calculated. , Calculating all combinations of each sentence included in the third sentence and each sentence included in the second sentence or each sentence included in the fourth sentence to obtain the third sentence If not, the similarity between each sentence included in the first sentence and each sentence included in the fourth sentence is determined based on the similarity between each sentence included in the first sentence and the fourth sentence. Calculate all combinations with each sentence included in
When the third sentence is obtained, for each sentence of the third sentence, the maximum similarity that maximizes the similarity is obtained, and the third sentence that has the maximum similarity equal to or less than a predetermined threshold. If the sentence of the first sentence corresponding to the sentence is extracted as a sentence including new information not included in the second sentence, and the third sentence is not obtained, the first sentence For each sentence of the sentence, the maximum similarity is obtained, and the sentence of the first sentence in which the maximum similarity is not more than a predetermined threshold is extracted as a sentence including new information not included in the second sentence. Sentence processing method. Extracting features from each sentence of the first sentence;
Each sentence of the first sentence using each extracted feature of each sentence of the first sentence and an extraction result indicating whether each sentence of the first sentence includes new information Learning a classifier for identifying whether each sentence of the first sentence contains new information based on the features of
The sentence processing method according to claim 4, wherein the sentence including the new information is re-extracted based on an identification result obtained by inputting each sentence of the first sentence to the learned classifier.   Generating a graph having a node representing each sentence of the first sentence and the second sentence, and an edge weighted according to a similarity between the nodes and a relationship between the nodes; The node corresponding to each sentence of one sentence is given a label based on whether or not the sentence includes the new information, and corresponds to each sentence of the second sentence obtained by the label propagation method using the graph. Based on the label given to the node, before or after the sentence of the second sentence corresponding to the node given the label closest to the label given to the node corresponding to the sentence containing the new information, 6. The sentence processing method according to claim 4, wherein the sentence is determined as a position where a sentence including the new information is inserted.   A sentence processing program for causing a computer to function as each means constituting the sentence processing apparatus according to any one of claims 1 to 3.

Images