JP2014154102A - Sentence correspondence determination device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To determine sentence correspondence with good accuracy even when sentence correspondences are intersecting.SOLUTION: By a partial document pair score calculation processing unit 21, correspondence of sentences between partial documents is determined so as to optimize a correspondence score about each of all pairs of partial documents in two documents. By sentence correspondence calculation unit 22, the pair of the partial documents to be corresponded is determined by solving an integer linear programming problem, and correspondence of the determined sentences for each of the pairs of the corresponded partial documents is determined as the correspondence of the sentences between the two documents.

Description

  The present invention relates to a sentence association determination apparatus, method, and program, and more particularly, to a sentence association determination apparatus, method, and program for determining sentence association between two documents.

  As a method for determining the association between existing sentence relationships, Non-Patent Document 1 discloses a method based on DP matching of sequences. DP matching is a method for obtaining a correspondence between elements of two sequences at a high speed by dynamic programming when two sequences are given. In addition, although not used in sentence association, the problem of finding the correspondence between words, which is a similar problem, can be formulated and solved as an assignment problem, which is a kind of discrete optimization problem. The association problem can be solved similarly (Non-Patent Document 2).

Robert C. Moore, “Fast and Accurate Sentence Alignment of Bilingual Corpora”, in proceedings of AMTA’02, pp 135-144, 2002. Evgenv Matusov, Richard Zens, and Hermann Ney, “Symmetric Word Alignments for Statistical Machine Translation” In Proceedings of the 20th international conference on Computational Linguistics, page 219. Association for Computational Linguistics.

In the existing DP matching method, there is a restriction that reversal of the order is not allowed in correspondence between possible sentences. That is, now two documents are F and E, and m sentences f ₁ , f ₂ ,. . . , _{F m} and n pieces of sentence _{e 1, e} 2,. . . , And it consists of _{e n.} The problem of obtaining sentence associations is that a set of associations A = {(f _{ρ (1)} , e _{η (1)} ), (f _{ρ (2)} , e _{η ( 2)} ),. . . , (F _{ρ (G),} e _{η (G)} )}. Here, G represents the number of associations, and G ≦ m, n. Further, ρ (i) represents a subscript of f in the i-th association, and η (i) represents a subscript of e. The method using DP matching has a problem that it is impossible to find a correspondence that allows crossover. Here, the associations cross over each other when the association (f _i , e _j ) is made in A and the associations (f _{i ′} , e) such that i ′ <i and j ′> j are satisfied. _{j ′} ) is also included in A. This restriction is often appropriate when E is a parallel translation of F. For example, when the order of paragraphs in a document is changed, there is a possibility that a correct correspondence cannot be obtained. Is expensive.

  Such a problem does not occur when sentence association is performed as an assignment problem, but on the other hand, since correspondence is performed without considering the connection before and after in the document, a good association is obtained. There is a problem that cannot be done.

  The present invention has been made in view of the above circumstances, and a sentence association determination apparatus, method, and method capable of accurately determining sentence association even when sentence associations are crossed. And to provide a program.

  In order to achieve the above object, a sentence association determining apparatus according to the present invention determines a sentence association between a first document including a plurality of sentences and a second document including a plurality of sentences. An association determination device that creates a first partial document composed of one or more consecutive sentences in the first document and includes one or more consecutive sentences in the second document. Create a second partial document, and for each of all pairs of the first partial document and the second partial document, correspondence of the sentence between the first partial document and the second partial document of the pair The partial document pair score calculating means for determining the correspondence of the sentence of the pair so as to optimize the matching score calculated according to the attachment, and calculated by the partial document pair score calculating means for optimization Based on each matching score of the pair The first partial document of the pair of the first partial document and the second partial document to be associated does not overlap with the first partial document of another pair, and is associated with the first partial document The second partial document in a pair of the second partial document does not overlap with the second partial document of another pair, and is a pair of the first partial document and the second partial document that are associated with each other At least one pair of the first partial document and the second partial document to be matched is determined so that the sum of the correspondence scores calculated so as to optimize each of the first partial document and the second partial document is maximized. A sentence that determines the association of the determined sentence for each pair of the first partial document and the second partial document that is determined as a sentence association between the first document and the second document The correspondence determining means is configured to include That.

  The sentence association determination method according to the present invention includes a sentence association in a sentence association determination apparatus that determines sentence association between a first document including a plurality of sentences and a second document including a plurality of sentences. In the determination method, the partial document pair score calculation unit creates each of the first partial documents including one or more continuous sentences of the first document, and the continuous document of the second documents. A second partial document composed of one or more sentences, and for each of all pairs of the first partial document and the second partial document, the first partial document and the second partial document of the pair; The sentence correspondence between the pair is determined so as to optimize the correspondence score calculated according to the sentence correspondence between the two, and optimized by the partial document pair score calculating means by the sentence correspondence determining means Calculated to be said A first partial document of a pair of the first partial document and the second partial document to be associated with each other based on the association score of each of the In addition, the second partial document of the pair of the first partial document and the second partial document to be associated with each other does not overlap with the second partial document of another pair, and is associated with the first partial document The first partial document and the second partial document that are associated with each other so that the sum of the correspondence scores calculated to optimize each pair of the document and the second partial document is maximized. At least one pair is determined, and the determined sentence correspondence between each pair of the first partial document and the second partial document associated with each other is determined between the first document and the second document. It is decided as the correspondence of the sentence in.

  The program according to the present invention is a program for causing a computer to function as each unit of the above sentence association determination apparatus.

As described above, according to the sentence association determination apparatus, method, and program of the present invention,
For each of the partial document pairs, sentence association is determined so as to optimize the association score between the partial documents, and the paired partial documents do not overlap with other pairs of partial documents, and , Determine the pair of partial documents to be matched so that the sum of the optimal matching scores of the pair of partial documents to be maximized, and associate the sentence determined for the pair of partial documents Is determined as the sentence correspondence between documents, so that the sentence correspondence can be accurately determined even when the sentence associations are crossed.

It is the schematic which shows the structure of the sentence matching determination apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the sentence matching determination processing routine in the sentence matching determination apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<System configuration>
The sentence association determination apparatus 100 according to the embodiment of the present invention receives two corresponding documents and determines sentence association between the two documents. The sentence association determination apparatus 100 is composed of a computer including a CPU, a RAM, and a ROM that stores a program for executing a sentence association determination processing routine to be described later. It is configured. As shown in FIG. 1, the sentence association determination apparatus 100 includes a document input unit 10, a calculation unit 20, and an association result output unit 30.

  The document input unit 10 receives input of two documents that are targets of association. The two input documents are a translation source language document and a translation destination language document obtained by translating the document.

  The calculation unit 20 includes a partial document pair score calculation processing unit 21 and an association calculation unit 22.

  The partial document pair score calculation processing unit 21 creates all the continuous groups of one or more sentences (called partial documents) included in each of the two documents received by the document input unit 10, For all pairs, calculate the score as follows:

Here, two documents are F and E, and m sentences f ₁ , f ₂ ,. . . , _{F m} and n pieces of sentence _{e 1, e} 2,. . . , And it consists of _{e n.} In the following, it is assumed that f _{i, j} represents a set of sentences (partial documents) including the i-th sentence to the j-th sentence of F. E is expressed similarly.

A partial document pair is a pair of two sentences, f _{i, j} and e _{k, l} . Since f _{i, j} is (n + 1) n / 2 ways and _{ek, l} is (m + 1) m / 2 ways, the total number of possible pairs is (n + 1) (m + 1) nm / 4 ways.

The score of a pair of partial documents is calculated by performing DP matching between the partial documents using the above method. In DP matching, an optimal score by DP matching between partial documents is calculated based on a score score (f _i , e _j ) for a pair of sentences. As score (f _i , e _j ), for example, one described in Non-Patent Document 1 can be used, and is represented by the following equation (1).

Further, the DP matching, in accordance with the following equation _{(2), f i, j} and _{e k,} best scores _score for a pair of _{l (f i, j, e} k, l) together with calculating the partial document The optimal correspondence between sentences can be obtained. The calculated optimal score score (fi _{, j} , e _{k, l} ) is stored in a memory (not shown). The score score (fi _{, j} , e _{k, l} ) is an example of the association score. The matching score score (f _ij , e _{k, l} ) between the sub-sentence sets f _{i, j} and e _{k, l} can be obtained by a recursive equation such as the following equation (2).

Here, score (f _j ) and score (e _l ) represent scores when the sentences are not associated with each other. score (f _{i, j} , e _{k, l} ) can be efficiently obtained using DP matching. That is,

As a result, the score can be obtained by reusing previously calculated results.

  The association calculation unit 22 uses the scores of all the pairs calculated by the partial document pair score calculation processing unit 21 to calculate the association of the partial documents having the maximum score. The integer linear programming problem processing apparatus 150 is an apparatus for solving an integer linear programming problem that is external to the sentence association determination apparatus 100 and is called from the association calculation unit 22 and used. As the integer linear programming problem processing apparatus 150, any commercially available integer linear programming solver can be used.

  Below, the optimal matching calculation by the matching calculation part 22 is demonstrated. This calculation is performed by solving the following optimization problem in the integer linear programming problem processing apparatus 150.

Here, w _ijkl is a binary variable representing a pair (f _{i, j} , e _{k, l} ) of a group of sentences associated with each other, and f _{i, j} , e _{k, l} is a binary representing a group of each sentence. Is a variable. W _ijkl takes 1 when the finally obtained correspondence includes a sentence group pair (f _{i, j} , e _{k, l} ), and takes 0 otherwise. λ is a cost for adding a pair, and by setting a large λ for score (fi _{, j} , e _{k, l} ), it becomes easy to take correspondence between groups of sentences as large as possible. Note that λ = 0 may be used.

By solving the above optimization problem (integer linear programming problem), the first partial document of the pair of the first partial document and the second partial document to be associated with the first partial document of the other pair to be associated with A second partial document that does not overlap and is associated with a second partial document that is a pair of the first partial document and the second partial document that are associated with each other and that does not overlap with a second partial document that is associated with another pair. The first partial document to be matched such that the sum of the values obtained by subtracting λ from the matching score score (fi _{, j} , e _{k, l} ) of the pair of the one partial document and the second partial document is maximized At least one pair with the second partial document is determined.

  Then, the association calculation unit 22 uses the document input unit 10 to associate the sentence determined by the partial document pair score calculation processing unit 21 for each pair of the first partial document and the second partial document associated with each other. It is determined as the optimum sentence correspondence between the two received documents.

  The association result output unit 30 outputs the optimum sentence association obtained by the sentence association determination apparatus 100.

<Operation of sentence association determination device>
Next, the operation of the sentence association determination apparatus 100 according to the embodiment of the present invention will be described. First, when two documents to be sentence association targets are input to the sentence association determination apparatus 100, the sentence association determination apparatus 100 executes a sentence association determination processing routine shown in FIG.

  First, in step S01, two input documents are received.

In step S02, all partial documents are generated for each of the two documents received in step S01, and for each of all pairs formed by the partial documents of the two documents, DP matching between the partial documents is performed to calculate an optimal score score (fi _{, j} , e _{k, l} ) for the pair.

In the next step S03, the integer linear programming problem processing apparatus 150 is used based on the optimal scores score (fi _{, j} , e _{k, l} ) of all the pairs calculated in step S02. By solving the optimization problem (integer linear programming problem) shown in equation (3), the correspondence between the two documents is determined.

  In step S04, for each of the partial document pairs associated in step S03, the sentence correspondence between the two documents is determined using the sentence association between the paired partial documents determined in step S02. The association is determined and output by the association result output unit 30, and the sentence association determination processing routine is terminated.

As described above, according to the sentence association determination apparatus according to the present embodiment, for all pairs of partial documents in two documents, the score score (f) between the partial documents f _{i, j} , e _{k, l.} sentence correspondence is determined so as to optimize _{i, j} , e _{k, l} ), the paired partial documents do not overlap with other paired partial documents, and are matched A pair of partial documents to be matched is determined so that the sum of values obtained by subtracting λ from the optimal score score (fi _{, j} , e _{k, l} ) of the pair of partial documents to be maximized This is a case where sentence correspondence between two documents is crossed by determining sentence correspondence determined to be optimized for a pair of partial documents as sentence correspondence between two documents. However, the sentence mapping is accurate Can be determined.

  In addition, as an assignment problem having a partial document that is a group of several consecutive sentences as a constituent element, a sentence association problem is formulated and solved. By solving the partial document as a constituent element in this way, it is possible to perform association in consideration of the connection before and after the sentence, which could not be considered when solving as an assignment problem with one sentence as a constituent element. In addition, it is possible to perform association even when there is a sentence crossing, which was impossible with the DP matching method.

  Also, in the problem of associating sentences in two documents, appropriate association should be performed even when the positions of corresponding sentence groups (for example, paragraphs) intersect between documents. As a result, it is possible to improve the accuracy of sentence correspondence between documents.

  In addition, by using the sentence correspondence described in the present embodiment when generating parallel translation data, it is possible to accurately associate sentences in the parallel translation data, improving the accuracy of statistical machine translation, etc. Can be expected. The sentence association described in this embodiment can also be applied to the problem of finding an illegal copy of a document.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Document input part 20 Calculation part 21 Partial document pair score calculation process part 22 Calculation part 30 Matching result output part 100 Sentence matching determination apparatus 150 Integer linear programming problem processing apparatus

Claims (7)

A sentence association determination device that determines sentence association between a first document including a plurality of sentences and a second document including a plurality of sentences,
A first partial document composed of one or more continuous sentences in the first document is created, and a second partial document composed of one or more continuous sentences in the second document is created. Correspondence calculated for each pair of the first partial document and the second partial document in accordance with sentence correspondence between the first partial document and the second partial document of the pair A partial document pair score calculating means for determining the correspondence of the sentence of the pair so as to optimize the attaching score;
The first part of the pair of the first partial document and the second partial document associated with each other based on the correspondence score of each of the pairs calculated to be optimized by the partial document pair score calculation unit The document does not overlap with the first partial document of another pair, and the second partial document of the pair of the first partial document and the second partial document associated with each other is the second partial document of the other pair. The sum of the correspondence scores calculated so as to optimize each pair of the first partial document and the second partial document to be matched does not overlap with a two-part document so as to be maximized. , Determining at least one pair of the first partial document and the second partial document to be associated, and determining the sentence for each of the pair of the first partial document and the second partial document associated with each other Mapping And sentence correspondence determining means for determining a correspondence between statements in between the first document a second document,
A sentence association determination apparatus including   The sentence association determining unit is configured so that the first partial document of a pair of the first partial document and the second partial document to be associated with each other does not overlap with the first partial document of another pair. The second partial document of the pair of the first partial document and the second partial document to be attached does not overlap the second partial document of another pair and is associated with the first partial document and the second partial document The first partial document and the second partial document that are associated with each other so that the sum of values obtained by subtracting a predetermined value from the association score calculated to optimize each pair with the second partial document is maximized. The sentence association determining apparatus according to claim 1, wherein at least one pair with a partial document is determined.   The sentence association determining unit is configured so that the first partial document of a pair of the first partial document and the second partial document to be associated with each other does not overlap with the first partial document of another pair. The second partial document of the pair of the first partial document and the second partial document to be attached does not overlap the second partial document of another pair and is associated with the first partial document and the second partial document By solving the linear programming problem so that the sum of the correspondence scores calculated to optimize each pair with the second partial document is maximized, the first partial document associated with the first partial document and the first partial document are correlated with each other. 3. The sentence association determination apparatus according to claim 1, wherein at least one pair with a two-part document is determined. A sentence association determination method in a sentence association determination apparatus that determines sentence association between a first document including a plurality of sentences and a second document including a plurality of sentences,
The partial document pair score calculation means respectively creates a first partial document composed of one or more continuous sentences in the first document, and consists of one or more continuous sentences in the second document. Create a second partial document, and for each of all pairs of the first partial document and the second partial document, correspondence of the sentence between the first partial document and the second partial document of the pair Determine the correspondence of the pair of sentences so as to optimize the correspondence score calculated according to the attachment,
The first partial document and the second partial document that are associated with each other based on the association score of each of the pairs calculated to be optimized by the partial document pair score calculation unit by the sentence association determination unit The first partial document of the pair of the second partial document does not overlap with the first partial document of the other pair and the paired first partial document and the second partial document are associated with each other. The correspondence score calculated so as to optimize each pair of the first partial document and the second partial document that do not overlap with the second partial document of another pair At least one pair of the first partial document and the second partial document associated with each other is determined so that the sum is maximized, and the pair of the first partial document and the second partial document associated with each other is determined. For each The correspondence of statements with the determined sentence correspondence determination method for determining a correlation statement between the first document and the second document.   Determining at least one pair of the first partial document and the second partial document to be associated by the sentence association determination means is performed between the first partial document and the second partial document to be associated with each other. The first partial document of a pair does not overlap the first partial document of another pair, and the second partial document of the pair of the first partial document and the second partial document to be associated with each other is Predetermined from the correspondence score calculated so as to optimize each pair of the first partial document and the second partial document that do not overlap with the second partial document of another pair 5. The sentence association determination method according to claim 4, wherein at least one pair of the first partial document and the second partial document to be associated is determined so that a sum of values obtained by subtracting the values is maximized.   Determining at least one pair of the first partial document and the second partial document to be associated by the sentence association determination means is performed between the first partial document and the second partial document to be associated with each other. The first partial document of a pair does not overlap the first partial document of another pair, and the second partial document of the pair of the first partial document and the second partial document to be associated with each other is The sum of the correspondence scores calculated so as to optimize each pair of the first partial document and the second partial document that do not overlap with the second partial document of another pair The sentence association determination method according to claim 4 or 5, wherein at least one pair of the first partial document and the second partial document to be associated is determined by solving a linear programming problem so that the maximum is obtained. .   The program for functioning a computer as each means of the sentence matching determination apparatus of any one of Claims 1-3.

Images