JP2017142757A - Information processing method, device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a phrase table with the number of phrase couples reduced for performing translation with high accuracy.SOLUTION: A selection processing part 22 selects a subset X from a phrase table so as to optimize an object function with a deteriorated modular function, represented by using the number of times when each of the k-th words eof a sentence eof an original language is covered with a phrase couple zincluded in the subset X of the phrase table being a set of phrase couples zabout each of a pair (e, f) of training bilingual corpuses and the number of times when each of the k-th words fof a sentence fof a target language is covered with a phrase couple zincluded in the subset X, as an object function.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing method, apparatus, and program.

  A statistical machine translation method is a document written in one language (hereinafter referred to as a source language) using a probability / statistical technique and written in another language (hereinafter referred to as a target language). It is a method of automatically translating into There are various statistical machine translation techniques, but in statistical machine translation based on one of the phrases, the source language sentence is expressed as a sequence of phrases consisting of a series of words, Translation is performed by converting the corresponding phrases in the target language.

  In order to perform statistical machine translation based on a phrase, it is necessary to prepare a table called a phrase table that indicates a phrase in a target language into which a phrase in the source language is translated. Let S be the phrase table. The component of S is a phrase pair (p, q). Here, p is a phrase in the source language, and q is a phrase in the target language. Since the types of phrase pairs included in the phrase table define the vocabulary that can be translated by the translation system, the total number of phrase pairs included in the phrase table is generally enormous.

  When translation is performed by a phrase-based statistical machine translation system, it is necessary to repeatedly access a phrase table stored in a storage device of a computer. If the number of phrase pairs included in the phrase table is enormous, the number of options that can be taken when generating a translation increases, and as a result, it takes time to generate the translation.

  In general, the phrase pairs included in the phrase table are automatically obtained based on the word alignment result of the sentence pairs of the source language and the target language that are in a parallel translation relationship. There are many poor-quality phrase pairs that are not translated in parallel. Poor-quality phrase pairs become noise at the time of translation generation, leading to deterioration in the quality of the generated translation. For these reasons, a technique for creating a smaller phrase table by removing a poor quality phrase from a given phrase table has been studied (for example, Non-Patent Document 1).

Zens, Richard and Stanton, Daisy and Xu, Peng, “A Systematic Comparison of Phrase Table Pruning Techniques”, In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 2012.

  Non-Patent Document 1 proposes a method of creating a small phrase table by scoring a phrase pair using entropy and deleting unnecessary phrase pairs based on the score.

However, in this method, a score is independently assigned to each phrase pair, and a process having a high score is taken out. Therefore, it is possible to directly evaluate the properties of the phrase table configured as a collection of phrase pairs having a high score. Therefore, when the statistical machine translation system is constructed using a table in which the number of phrase pairs is reduced, the translation accuracy may be lowered.
In addition, in order to reduce the number of phrase pairs using entropy, it is necessary to calculate the exponent time for the length of the phrase included in each phrase pair, so there are many phrase pairs consisting of long phrases. However, there was a problem that efficient calculation was difficult.

  The present invention has been made in view of the above circumstances, and provides an information processing method, apparatus, and program capable of obtaining a phrase table with a reduced number of phrase pairs for accurate translation. With the goal.

The information processing method according to the present invention in order to achieve the above object, includes a selection processing means, the pair of the sentence f _i in the target language is a translation of the sentence e _i of the original said the sentence e _i of the source language Language A phrase p _i that is a partial character string of a sentence e _{i in} the source language, which is generated in advance from a training parallel translation corpus that is a set of (e _i , f _i ), and the phrase p _i , and the purpose An information processing method in an information processing apparatus for selecting a subset from a phrase table that is a set of phrase pairs z _i with a phrase q _i that is a partial character string of a sentence f _{i in} a language, wherein the selection processing means includes: For each of the pairs (e _i , f _i ) of the training bilingual corpus, each of the kth words e _ik of the source language sentence e _i is a set of phrase pairs z _j Before being included in subset X The number of times that is covered by the phrase pair z _j, each of the k-th word f _ik sentence f _i of the target language, with the number of times that is covered by the phrase pair z _j included in the subset X A step of selecting a subset X from the phrase table so as to optimize the objective function using the expressed submodular function as an objective function.

The information processing apparatus according to the present invention, sentence f _i and the pair (e _{i, f i)} of the target language is a translation of the sentence e _i of the source language with sentence e _i of the source language translation for a set of training generated in advance from the corpus, the phrase p _i is a substring of the text e _i of the source language, a translation of the phrase p _i, and phrase q _i is a substring of the sentence f _i in the target language Is an information processing apparatus that selects a subset from a phrase table that is a set of phrase pairs z _i with respect to each of the pairs (e _i , f _i ) of the training bilingual corpus each k-th word e _ik statement e _i is the number of times that is covered by the phrase pair z _j included in the subset X a phrase table which is a set of the phrase pair z _j, sentence f of the target language _i of the k-th word _{f ik} of Portion s is submodularity function expressed by using the number of times it is covered by the phrase pair z _j included in the subset X as an objective function, so as to optimize the objective function, from the phrase table Selection processing means for selecting the set X is included.

  The step of selecting the subset X from the phrase table by the selection processing means may select the subset X from the phrase table so as to optimize the objective function using a greedy method.

  The step of selecting the subset X from the phrase table by the selection processing means selects the subset X from the phrase table so as to optimize the objective function g (X) represented by the following equation: be able to.

ただし、Ｋは、部分集合のサイズを表す予め定められた値である。Ｅは、前記訓練用対訳コーパスにおける原言語の文の集合｛ｅ_１,...,ｅ_Ｎ｝を表し、Ｆは、前記訓練用対訳コーパスにおける目的言語の文の集合｛ｆ_１,...,ｆ_Ｎ｝を表す。ｃ（Ｘ，ｆ_ｉｋ）は、前記単語ｆ_ｉｋが前記部分集合Ｘに含まれる前記フレーズ対ｚ_ｊにより被覆される回数を表し、ｃ（Ｘ，ｅ_ｉｋ）は、前記単語ｅ_ｉｋが前記部分集合Ｘに含まれる前記フレーズ対ｚ_ｊにより被覆される回数を表す。

Here, K is a predetermined value representing the size of the subset. E represents a set of source language sentences {e ₁ ,..., E _N } in the training bilingual corpus, and F represents a set of target language sentences {f ₁ ,. ., f _N }. c (X, f _ik ) represents the number of times the word f _ik is covered by the phrase pair z _j included in the subset X, and c (X, e _ik ) represents the word e _ik is the part This represents the number of times covered by the phrase pair z _j included in the set X.

  The step of selecting the subset X from the phrase table by the selection processing means selects the subset X from the phrase table so as to optimize the objective function h (X) represented by the following equation: be able to.

ただし、Ｅは、前記訓練用対訳コーパスにおける原言語の文の集合｛ｅ_１,...,ｅ_Ｎ｝を表し、Ｆは、前記訓練用対訳コーパスにおける目的言語の文の集合｛ｆ_１,...,ｆ_Ｎ｝を表す。ｃ（Ｘ，ｆ_ｉｋ）は、前記単語ｆ_ｉｋが前記部分集合Ｘに含まれる前記フレーズ対ｚ_ｊにより被覆される回数を表し、ｃ（Ｘ，ｅ_ｉｋ）は、前記単語ｅ_ｉｋが前記部分集合Ｘに含まれる前記フレーズ対ｚ_ｊにより被覆される回数を表す。λは０≦λ≦１であるようなパラメーである。Ｉ（・）は、引数が真のときに１、引数が偽のときに０を返すような関数である。

Where E represents a set of source language sentences {e ₁ ,..., E _N } in the training bilingual corpus, and F represents a set of target language sentences {f ₁ ,. ..., f _N }. c (X, f _ik ) represents the number of times the word f _ik is covered by the phrase pair z _j included in the subset X, and c (X, e _ik ) represents the word e _ik is the part This represents the number of times covered by the phrase pair z _j included in the set X. λ is a parameter such that 0 ≦ λ ≦ 1. I (•) is a function that returns 1 when the argument is true and 0 when the argument is false.

  The program according to the present invention is a program for causing a computer to execute each step of the information processing method of the present invention.

As described above, according to the information processing method, apparatus, and program of the present invention, the kth word of the sentence e _{i in} the source language for each pair (e _i , f _i ) of the training parallel translation corpus each e _ik is the number of times that is covered by the phrase pair z _j included in the subset X a phrase table is a set of phrase pair z _j, each of the k-th word f _ik sentence f _i in the target language The subset X is selected from the phrase table so as to optimize the objective function using the submodular function expressed by the number of times covered by the phrase pair z _j included in the subset X as an objective function. By this, the effect that the phrase table for which the number of phrase pairs for reducing a translation accurately can be obtained can be acquired.

It is the schematic which shows the structure of the information processing apparatus which concerns on embodiment of this invention. It is a figure which shows the algorithm which concerns on embodiment of this invention. It is a flowchart which shows the content of the selection process routine in the information processing apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline of Embodiment of the Present Invention>
The embodiment of the present invention is to reduce the number of phrase pairs stored in a phrase table that is necessary for statistical machine translation based on phrases. In the present embodiment, the problem of extracting a set of phrase pairs from the phrase table is formulated as an optimization problem that maximizes the submodular function, and the phrase pairs are extracted by solving this by a greedy method.

<System configuration>
The information processing apparatus 100 according to the embodiment of the present invention selects the subset X from the phrase table S generated in advance from the training parallel translation corpus. The information processing apparatus 100 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a selection processing routine described later, and is functionally configured as follows. . As illustrated in FIG. 1, the information processing apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 30.

The input unit 10 receives as input a phrase table S and a training parallel translation corpus that includes a set E of source language sentences and a set F of target language sentences. In the present embodiment, the phrase table S is S = {z ₁ ,..., Z _M }. The phrase table S is a set of phrase pairs z _i .

The phrase table S is generated in advance from the training parallel translation corpus. The training bilingual corpus is a set of pairs (e _i , f _i ) of a source language sentence e _i and a target language sentence f _i that is a translation of the source language sentence e _i . Further, the phrase table S is a bilingual phrase p _i and the phrase p _i is a substring of the text e _i of the source language of the training corpus, and a partial character string of a sentence f _i in the target language it is a set of phrase pair _{x i} with a certain phrase _{q i.}

Further, M represents the total number of phrase pairs, and x _i = (p _i , q _i ) is a pair of a source language phrase p _i and a target language phrase q _i . Each phrase is a sequence of words,

And E is a set of sentences in the source language, F is a set of sentences in the target language, and e _i and f _i are sentences in the source language and the target language, respectively. l _i is the number of words of _{p i, k i} is the number of words of _{q i.} E = {e ₁ ,. . . , E _N }, F = {f ₁ ,. . . , F _N }, and sentence e _i and sentence f _i are in a bilingual relationship. Each sentence is expressed as a sequence of words,

And e _ij is a source language word, and n _i is the number of words in e _i . As well

And m _i is the number of words of _{f i.}

For a phrase pair z _j = (p _j , q _j ) and a sentence pair (e _i , f _i ) in a parallel corpus, z _j is included in the pair, where p _j is part of e _i It is defined that it matches a column and q _j matches a substring with f _i . That is,

To meet

Is defined to exist. At this time, words included in the partial string matching p _j, the q _j is defined as being covered by the z _j.

  The computing unit 20 selects a subset from the phrase table S based on the training parallel translation corpus and the phrase table S received by the input unit 10. The calculation unit 20 includes a selection processing unit 22.

  Based on the training parallel translation corpus and the phrase table S received by the input unit 10, the selection processing unit 22 uses the submodular function as an objective function and optimizes the objective function from the subset X (phrase). Select table X). In this embodiment, the subset X is selected from the phrase table S by solving the optimization problem so as to maximize the objective function using the greedy method.

In the present embodiment, for each training bilingual corpus pair (e _i , f _i ), each k-th word e _ik of the source language sentence e _i is a set of phrase pairs z _j. The number of times covered by the phrase pair z _j included in the subset X of the table and each of the kth words f _ik of the sentence f _{i in} the target language are covered by the phrase pair z _j included in the subset X A submodular function expressed by the number of times is used as an objective function.

  Specifically, the selection processing unit 22 selects the subset X from the phrase table by solving the optimization problem shown in the following formula (1).

  The variable g (X) is an objective function for evaluating the goodness of the phrase pair subset X, as shown in the following equation (2).

Here, K is a predetermined value representing the size of the subset. E represents a set of source language sentences {e ₁ ,..., E _N } in the training bilingual corpus, and F represents a set of target language sentences {f ₁ ,. f _N }. c (X, f _ik ) represents the number of times the word f _ik is covered by the phrase pair z _j included in the subset X, and c (X, e _ik ) includes the word e _ik in the subset X. It represents the number of times that is covered by the phrase pair z _j.

Also, g (X) gives a high score to a phrase table that covers more words appearing in E and F, but the number of times c (X, f _ik ) or c ( Taking the logarithm of X, e _ik ) and adding it to the score gives a higher score for a phrase table that covers many different words.

  FIG. 2 shows a greedy algorithm for obtaining an approximate solution of the above optimization problem. As shown in FIG. 2, in this embodiment, first, the subset X is initialized as an empty set, and then the difference g (X∪ {z}) − g (X) of the function representing the score is maximized. By selecting zεS in order and adding them to the subset X, a subset X that is a phrase table of size K is generated. The selection processing unit 22 obtains a subset X that is a phrase table such that X⊆S by solving the above optimization problem.

  The output unit 30 outputs the subset X selected by the selection processing unit 22 as a phrase table in which the number of phrase pairs is reduced.

<Operation of information processing device>
Next, the operation of the information processing apparatus 100 according to the embodiment of the present invention will be described. First, when the training parallel translation corpus and the phrase table S are input to the information processing apparatus 100, the information processing apparatus 100 executes a selection processing routine shown in FIG.

  First, in step S <b> 100, a training parallel translation corpus and a phrase table S are received by the input unit 10.

  In step S102, the selection processing unit 22 uses the greedy method based on the training parallel translation corpus and the phrase table S received in step S100, according to the above formulas (1) and (2). A subset X is selected from

  In step S104, the subset X selected in step S102 is output as a result, and the selection processing routine is terminated.

As described above, according to the information processing apparatus according to the present embodiment, the k-th word e _ik of the sentence e _{i in} the source language for each pair (e _i , f _i ) of the parallel translation corpus for training. each, the number of times that is covered by the phrase pair z _j included in the subset X a phrase table is a set of phrase pair z _j, each of the k-th word f _ik sentence f _i of the target language, part of the By selecting the subset X from the phrase table so as to optimize the objective function using the submodular function represented by the number of times covered by the phrase pair z _j included in the set X as the objective function. Therefore, it is possible to obtain a phrase table for performing translation with high accuracy and having reduced phrase pairs. Moreover, a phrase table with a small number of phrase pairs can be obtained.

  As a result, it is possible to speed up the translation document generation process and improve translation accuracy by reducing unnecessary phrase pairs.

  In addition, according to the present embodiment, unlike the existing method using entropy, the goodness of the phrase table can be directly evaluated, so that the number of phrases in the phrase table can be reduced. In addition, the solution by the greedy method operates in polynomial time with respect to the number of phrases, and does not take exponentially time according to the phrase length. Therefore, even when a phrase table including a huge number of phrase pairs is given as an input, it operates at high speed.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the above-described embodiment, the case where g (X) shown in the above formula (2) is used as an example of the submodular function has been described as an example. However, the present invention is not limited to this. For example, h (X) shown in the following formula (3) may be used as the objective function.

However, as in the above equation (2), E represents a set of sentences in the source language {e ₁ ,..., E _N } in the training bilingual corpus, and F represents the target language in the training bilingual corpus. Represents a set of sentences {f ₁ ,..., F _N }. c (X, f _ik ) represents the number of times the word f _ik is covered by the phrase pair z _j included in the subset X, and c (X, e _ik ) includes the word e _ik in the subset X. It represents the number of times that is covered by the phrase pair z _j. Also, λ is a parameter such that 0 ≦ λ ≦ 1. I (•) is a function that returns 1 when the argument is true and 0 when the argument is false.

  When h (X) in the above equation (3) is used as λ <1, for a subset X that covers more different words than g (X) in the above equation (2), A high score is set.

  The information processing apparatus 100 described above has a computer system inside, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Input part 20 Calculation part 22 Selection process part 30 Output part 100 Information processing apparatus

Claims (6)

From a training bilingual corpus that includes a selection processing means and is a set of pairs (e _i , f _i ) of a source language sentence e _i and a target language sentence f _i that is a parallel translation of the source language sentence e _i phrases generated, and phrase p _i is a substring of the text e _i of the source language, a translation of the phrase p _i, and the phrase q _i is a substring of the sentence f _i in the target language An information processing method in an information processing apparatus for selecting a subset from a phrase table that is a set of pairs z _i ,
For each of the pairs (e _i , f _i ) of the training bilingual corpus, the selection processing means determines that each of the kth words e _ik of the source language sentence e _i of the phrase pair z _j The number of times covered by the phrase pair z _j included in the subset X of the phrase table that is a set, and each of the kth words f _ik of the sentence f _i of the target language are included in the subset X And selecting a subset X from the phrase table so as to optimize the objective function using a submodular function represented by the number of times covered by the phrase pair z _j as an objective function. . The information according to claim 1, wherein the step of selecting the subset X from the phrase table by the selection processing unit selects the subset X from the phrase table so as to optimize the objective function using a greedy method. Processing method. The step of selecting the subset X from the phrase table by the selection processing means selects the subset X from the phrase table so as to optimize the objective function g (X) shown in the following equation. Or the information processing method of Claim 2.

Here, K is a predetermined value representing the size of the subset. E represents a set of source language sentences {e ₁ ,..., E _N } in the training bilingual corpus, and F represents a set of target language sentences {f ₁ ,. ., f _N }. c (X, f _ik ) represents the number of times the word f _ik is covered by the phrase pair z _j included in the subset X, and c (X, e _ik ) represents the word e _ik is the part This represents the number of times covered by the phrase pair z _j included in the set X. The step of selecting the subset X from the phrase table by the selection processing means selects the subset X from the phrase table so as to optimize the objective function h (X) expressed by the following equation. Information processing method described in 1.

Where E represents a set of source language sentences {e ₁ ,..., E _N } in the training bilingual corpus, and F represents a set of target language sentences {f ₁ ,. ..., f _N }. c (X, f _ik ) represents the number of times the word f _ik is covered by the phrase pair z _j included in the subset X, and c (X, e _ik ) represents the word e _ik is the part This represents the number of times covered by the phrase pair z _j included in the set X. λ is a parameter such that 0 ≦ λ ≦ 1. I (•) is a function that returns 1 when the argument is true and 0 when the argument is false. A source language generated in advance from a training bilingual corpus that is a set of pairs (e _i , f _i ) of a source language sentence e _i and a target language sentence f _i that is a translation of the source language sentence e _i and phrase p _i is a substring of the text e _i of a bilingual the phrase p _i, and a set of phrase pair z _i of the phrase q _i is a substring of the sentence f _i in the target language An information processing apparatus that selects a subset from a phrase table,
For each of the pairs (e _i , f _i ) of the training bilingual corpus, each of the kth words e _ik of the source language sentence e _i is a set of phrase pairs z _j The number of times covered by the phrase pair z _j included in the subset X and each of the kth words f _ik of the sentence f _{i in} the target language are covered by the phrase pair z _j included in the subset X And a selection processing means for selecting the subset X from the phrase table so as to optimize the objective function using the submodular function represented by   The program for making a computer perform each step of the information processing method of any one of Claims 1-4.

Images