JP6482073B2 - Information processing method, apparatus, and program - Google Patents

Description

  The present invention relates to an information processing method, apparatus, and program.

  The statistical machine translation method is a document written in a language (hereinafter referred to as an original language) using a probability / statistical technique and written in another language (hereinafter referred to as a target language). It is a method of automatically translating into There are various statistical machine translation methods, but in statistical machine translation based on one of the phrases, a sentence in the original language is expressed as an array of phrases consisting of a series of words, Translation is performed by converting the corresponding phrases in the target language.

  In order to perform statistical machine translation based on a phrase, it is necessary to prepare a table called a phrase table, which indicates a phrase in a target language into which a phrase in the original language is translated. Let S be the phrase table. The component of S is a phrase pair (p, q). Here, p is a phrase in the original language, and q is a phrase in the target language. Since the types of phrase pairs included in the phrase table define the vocabulary that can be translated by the translation system, the total number of phrase pairs included in the phrase table is generally enormous.

  When translation is performed by a phrase-based statistical machine translation system, it is necessary to repeatedly access a phrase table stored in a storage device of a computer. If the number of phrase pairs included in the phrase table is enormous, the number of options that can be taken when generating a translation increases, and as a result, it takes time to generate the translation.

  In general, the phrase pairs included in the phrase table are automatically obtained based on the word alignment results of the sentence pairs of the source language and the target language that are in a parallel translation relationship. There are many poor-quality phrase pairs that are not translated in parallel. Poor-quality phrase pairs become noise at the time of translation generation, leading to deterioration in the quality of the generated translation. For these reasons, a technique for creating a smaller phrase table by removing a poor quality phrase from a given phrase table has been studied (for example, Non-Patent Document 1).

Zens, Richard and Stanton, Daisy and Xu, Peng, “A Systematic Comparison of Phrase Table Pruning Techniques”, In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 2012.

  Non-Patent Document 1 proposes a method of creating a small phrase table by scoring a phrase pair using frequency and entropy and deleting unnecessary phrase pairs based on the score.

  However, in these methods, the threshold value θ is determined for the score of the phrase pair, or the phrase table is created in the form of taking out the top K scores, so the parameter θ or K that determines the size of the phrase table Need to be determined by some standard. Since the size of an appropriate phrase table varies depending on the translation system, selecting an appropriate parameter has been a difficult problem involving trial and error.

  In addition, it is important to ensure that the translation system correctly translates the sentences included in the training bilingual corpus, but the existing phrase table reduction method is necessary to correctly translate the sentences in the training bilingual corpus. There was a problem that it was not possible to guarantee that a pair of phrases would be selected.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an information processing method, apparatus, and program capable of obtaining a phrase table with reduced phrase pairs.

The information processing method according to the present invention in order to achieve the above object, includes a selection processing means, the pair of the sentence f _j in the target language is a translation of the sentence e _j of the source language with sentence e _j of the original language (e _{j, f j)} are previously generated from training corpus is a collection of a bilingual phrase p _i and the phrase p _i is a substring of the sentence e _j of the original language and the target language An information processing method in an information processing apparatus for selecting a subset from a phrase table that is a set of phrase pairs x _i with a phrase q _i that is a partial character string of a sentence f _j of the pair _(e j, _{f j)} of the training corpus for each of said pairs _(e j, _{f j)} each of r th word _{e jr} sentence _{e j} of the original language, the phrase pair x _i is a set of phrase table A phrase p _i of any one of the phrase pair x _i minute set, and the included phrase p _i is a substring of the sentence e _j of the original language, the sentence f _j of the target language each of r th word f _jr is a said phrase q _i of any one of the phrase pair x _i of a subset of the phrase table which is a set of the phrase pair x _i, and sentences of the target language f includes a step of selecting a subset from the phrase table so as to be included in the phrase q _i which is a partial character string of _j .

The information processing apparatus according to the present invention, the pair (e _{j, f j)} of the sentence f _j in the target language is a translation of the sentence e _j of the source language with sentence e _j of source language translation for a set of training generated in advance from the corpus, a bilingual phrase p _i and the phrase p _i is a substring of the sentence e _j of the original language and a phrase q _i is a substring of the sentence f _j in the target language from the phrase table is a set of phrase pair x _i of an information processing apparatus for selecting a subset, the pair (e _{j, f j)} of the training corpus for each of said pairs (e _j, each of r th word e _jr sentence e _j of the original language of the f _j) is a phrase p _i of any one of the phrase pair x _i of a subset of the phrase table which is a set of the phrase pair x _i And the sentence in the original language _j included in the partial character strings in a phrase p _i of each of the r-th word f _jr sentence f _j of the target language, any subset of the set is a phrase table of the phrase pair x _i 1 One of the phrases a said phrase q _i pair x _i, and the like contained in the phrase q _i is a substring of the sentence f _j in the target language, selection for selecting a subset from the phrase table The processing means is included.

The step of selecting a subset from the phrase table by the selection processing means may select a subset from the phrase table according to the following equation.
However, the variable y _i (i = 1,..., M) is a binary variable indicating whether or not the phrase pair x _i is included in the subset, and a _ijr is the original language according to the phrase pair x _i . a parameter r th word e _jr the sentence e _j represents whether it is covered, when it is covered is 0 and becomes parameter when a _{IJR = 1,} not covered. b _ijr is a parameter indicating whether or not the r-th word f _jr of the sentence f _j of the target language is covered by the phrase pair x _i , and b _ijr = 1 when covered, 0 when not covered It is a parameter.

  The program according to the present invention is a program for causing a computer to execute each step of the information processing method of the present invention.

As described above, the information processing method of the present invention, apparatus, and according to the program, training corpus pair (e _{j, f j)} for each of the original language of the pair (e _{j, f j)} Each of the r-th words e _jr of the sentence e _j is a phrase p _i of any one phrase pair x _i of a subset of the phrase table that is a set of phrase pairs x _i , and included in the phrase p _i is a substring of the sentence e _j, each of the r-th word f _jr sentence f _j of the target language is any subset of the phrase table is a set of phrase pair x _i 1 a phrase q _i of One phrase pair x _i, and, to be included in the phrase q _i is a substring of the sentence f _j in the target language, by selecting a subset from the phrase table, the phrase pairs Reduced The effect that a raise table can be obtained is obtained.

It is the schematic which shows the structure of the information processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the selection process routine in the information processing apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline of Embodiment of the Present Invention>
The embodiment of the present invention is to reduce the number of phrase pairs stored in a phrase table that is necessary for statistical machine translation based on phrases. In this embodiment, the problem of extracting a subset of phrase pairs from the phrase table is formulated as a set partitioning problem that is a kind of combination optimization problem, and a subset of phrase pairs is extracted by solving the set partitioning problem.

<System configuration>
The information processing apparatus 100 according to the embodiment of the present invention selects a subset T from the phrase table S generated in advance from the training parallel corpus. The information processing apparatus 100 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a selection processing routine described later, and is functionally configured as follows. . As illustrated in FIG. 1, the information processing apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 30.

The input unit 10 receives as input a phrase table S and a training bilingual corpus that includes an original language sentence set E and a target language sentence set F. In the present embodiment, the phrase table S is represented by S = {x ₁ ,. . . , X _M }. Phrase table S is a set of phrase pair x _i.

The phrase table S is generated in advance from the training parallel translation corpus. Training corpus is a collection of pairs of sentences f _j in the target language is a translation of the sentence e _j and the source language sentence e _j of the source language (e _{j, f j).} Further, the phrase table S is a bilingual phrase p _i and the phrase p _i is a substring of the sentence e _j of the original language of the training corpus, and a partial character string of a sentence f _j in the target language it is a set of phrase pair _{x i} with a certain phrase _{q i.}

M represents the total number of phrase pairs, and x _i = (p _i , q _i ) is a pair of the original language phrase p _i and the target language phrase q _i . Each phrase is a sequence of words,

And E is a set of sentences in the original language, F is a set of sentences in the target language, and e _i and f _i are sentences in the original language and the target language, respectively. E = {e ₁ ,. . . , E _N }, F = {f ₁ ,. . . , F _N }, and sentence e _i and sentence f _i are in a bilingual relationship. Each sentence is expressed as a sequence of words,

And e _ij is a word in the original language, and n _i is the number of words in e _i . As well

And m _i is the number of words of _{f i.}

Part there phrase pair _{x j} and sentences in parallel corpus pair _(e i, _{f i)} with respect _to, x _{j =} (p _{j, q} j) and is included in the pair, with _{p j} is the _{e i} It is defined that it matches a column and q _j matches a substring with f _i . That is,

To meet

Is defined to exist. At this time, words included in the partial string matching p _j, the q _j is defined as being covered by x _j.

  The computing unit 20 selects a subset from the phrase table S based on the training parallel translation corpus and the phrase table S received by the input unit 10. The calculation unit 20 includes a selection processing unit 22.

For each pair of training bilingual corpora (e _j , f _j ) received by the input unit 10, the selection processing unit 22 selects the r th word of the sentence e _{j in} the original language of the pair (e _j , f _j ). Each of e _jr is a phrase p _i of any one phrase pair x _i of a subset of the phrase table that is a set of phrase pairs x _i and is a partial character string of a sentence e _{j in} the original language A phrase q _i of any one phrase pair x _{i in} a subset of the phrase table in which each of the r-th words f _jr of the sentence f _{j in} the target language is a set of phrase pairs x _{i included in} the phrase p _i The subset T is selected from the phrase table S so as to be included in the phrase q _i which is a partial character string of the sentence f _{j in} the target language. The selection processing unit 22 obtains a subset T (phrase table T) such that T⊆S by solving the set partitioning problem.

  Specifically, the selection processing unit 22 selects the subset T from the phrase table S by solving the set partitioning problem according to the following formula. Here, the set partitioning problem to be solved is

It is formulated as an integer programming problem for the variable y _i .

The variable y _i (i = 1,..., M) is a binary variable, and y _i = 1 when the phrase pair x _i is included in the subset T, and y _i = 0 otherwise. It is. Further, a _IJR is a parameter indicating whether r th word e _jr sentence e _j of the original language by the phrase pair x _i is covered, and 0 when a _{IJR = 1,} otherwise when it is covered It is a parameter. Similarly, b _ijr is a parameter indicating whether or not the r-th word f _jr of the sentence f _j of the target language is covered by the phrase pair x _i , and is 1 when covered, otherwise This parameter is zero.

The set partitioning problem is a problem of selecting a set of phrases in which each e _jr and f _jr is covered only once.

The output unit 30 outputs the subset T of the phrase pair x _i selected by the selection processing unit 22 as a phrase table in which the phrase pairs are reduced.

<Operation of information processing device>
Next, the operation of the information processing apparatus 100 according to the embodiment of the present invention will be described. First, when the training parallel translation corpus and the phrase table S are input to the information processing apparatus 100, the information processing apparatus 100 executes a selection processing routine shown in FIG.

  First, in step S <b> 100, a training parallel translation corpus and a phrase table S are received by the input unit 10.

In step S102, the selection processing unit 22 determines, for each pair of training bilingual corpora (e _j , f _j ), based on the training bilingual corpus and phrase table S received in step S100. e _{j, f j)} of each of the r-th word _{e jr} sentence _{e j} of the original language, phrase pair _{x i} phrase _{p i} of any one phrase pair _{x i} of a subset of the phrase table is a set of a is and, included in the phrase p _i is a substring of the sentence e _j of the original language, each of the r-th word f _jr sentence f _j of the target language is, is a set of phrase pair x _i a phrase q _i of any one phrase pair x _i of a subset of the phrase table, and, to be included in the phrase q _i is a substring of the sentence f _j in the target language, Furezute A subset T is selected from the table S.

  In step S104, the subset T selected in step S102 is output as a result, and the selection processing routine is terminated.

As described above, according to the information processing apparatus according to the present embodiment, training corpus pair (e _{j, f j)} for each of the original language sentence of the pair (e _{j, f j)} each of the _jth r-th words e _jr is a phrase p _i of any one phrase pair x _i of a subset of the phrase table that is a set of phrase pairs x _i , and a sentence e in the original language _j included in the phrase p _i is a substring of any one phrase of a subset of each of the r-th word f _jr sentence f _j of the target language is a phrase table is a set of phrase pair x _i a phrase q _i pair x _i, and, to be included in the phrase q _i is a substring of the sentence f _j in the target language, by selecting a subset from the phrase table, the training corpus Included phrases A phrase table including pairs and having reduced phrase pairs can be obtained, and a phrase table having a small number of phrase pairs can be obtained.

  Further, as a result of obtaining a phrase table with a small number of phrase pairs, it is possible to speed up the translation document generation process and improve translation accuracy by reducing unnecessary phrase pairs. In addition, by formulating using the concept of division, it is possible to obtain a phrase table that guarantees the translatability of the sentences included in the training bilingual corpus without manually setting parameters relating to the size of the phrase table. it can.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  The information processing apparatus 100 described above has a computer system inside, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Input part 20 Calculation part 22 Selection process part 30 Output part 100 Information processing apparatus

Claims (3)

Wherein the selection processing means, in advance from a set a is training corpus of pairs of sentences f _j in the target language is a translation of the sentence e _j of the source language with sentence e _j of source language (e _{j, f j)} It produced a bilingual phrase p _i and the phrase p _i is a substring of the sentence e _j of the original language and a phrase pair of phrases q _i is a substring of the sentence f _j in the target language An information processing method in an information processing apparatus that selects a subset from a phrase table that is a set of _xi ,
It said selection processing means, the pair _(e j, _{f j)} of the training corpus for each of said pairs _(e j, _{f j)} the sentence _{e j} of the original language of the r-th word _{e jr} of each, a phrase p _i of any one of the phrase pair x _i of a subset of the phrase table which is a set of the phrase pair x _i, and is the partial character string of a sentence e _j of the source language Each of the r-th word f _jr of the sentence f _{j in} the target language sentence f _j is included in the phrase p _i , and the phrase pair x _i of any one of the subsets of the phrase table that is a set of the phrase pairs x _i a the phrase q _i, and said to be included in the target language the phrase q _i is a substring of the sentence f _j of, met information processing method comprising the steps of selecting a subset from the phrase table The
The step of selecting a subset from the phrase table by the selection processing means selects an subset from the phrase table according to the following expression.

However, the variable y _i (i = 1,..., M) is a binary variable indicating whether or not the phrase pair x _i is included in the subset, and a _ijr is the original language according to the phrase pair x _i . a parameter r th word e _jr the sentence e _j represents whether it is covered, when it is covered is 0 and becomes parameter when a _{IJR = 1,} not covered. b _ijr is a parameter indicating whether or not the r-th word f _jr of the sentence f _j of the target language is covered by the phrase pair x _i , and b _ijr = 1 when covered, 0 when not covered It is a parameter. Sentence f _j and a pair (e _{j, f j)} of the target language is a translation of the sentence e _j of the source language with sentence e _j of the original language are previously generated from training corpus is a collection of source language A phrase p _i that is a partial character string of the sentence e _j and a phrase x _i that is a parallel translation of the phrase p _i and a phrase q _i that is a partial character string of the sentence f _{j in} the target language. An information processing apparatus that selects a subset from a phrase table,
Wherein said pair of training corpus _(e j, _{f j)} for each of said pairs _(e j, _{f j)} each r-th word _{e jr} sentence _{e j} of the original language of the phrase pair a x _i phrase p _i of any one of the phrase pair x _i of a subset of the phrase table is a set of, and included in the phrase p _i said a substring of the sentence e _j of the original language , each r-th word f _jr sentence f _j of the target language, encounters the phrase q _i of any one of the phrase pair x _i of a subset of the phrase table which is a set of the phrase pair x _i And a selection processing means for selecting a subset from the phrase table so as to be included in the phrase q _i that is a partial character string of the sentence f _j of the target language ,
The information processing method, wherein the selection processing means selects a subset from the phrase table according to the following expression.

However, the variable y _i (i = 1,..., M) is a binary variable indicating whether or not the phrase pair x _i is included in the subset, and a _ijr is the original language according to the phrase pair x _i . a parameter r th word e _jr the sentence e _j represents whether it is covered, when it is covered is 0 and becomes parameter when a _{IJR = 1,} not covered. b _ijr is a parameter indicating whether or not the r-th word f _jr of the sentence f _j of the target language is covered by the phrase pair x _i , and b _ijr = 1 when covered, 0 when not covered It is a parameter. A program for causing a computer to execute each step of the information processing method according to claim 1 .