JP2018072979A - Parallel translation sentence extraction device, parallel translation sentence extraction method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a parallel translation corpus of which quality is higher than the quality in association of two sentences by using a general dictionary.SOLUTION: A parallel translation sentence acquisition part of a parallel translation sentence extraction device acquires a parallel translation sentence by matching a sentence in a first language and a sentence in a second language structuring a parallel translation document by using a parallel translation dictionary. A word alignment score calculation part calculates a word alignment score showing the probability that each word in the first language structuring the acquired parallel translation sentence matches one word in the second language structuring the parallel translation sentence. A parallel translation word extraction part adds a pair of a word whose calculated word alignment score is higher than a threshold or the greatest among the words in the first language and the one word in the second language to the parallel translation dictionary as a parallel translation word. A parallel translation sentence acquisition part acquires a parallel translation sentence by matching the sentence in the first language and the sentence in the second language structuring the parallel translation document by using the parallel translation dictionary to which the parallel translation word is added.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for creating a bilingual corpus.

  In recent years, the importance of creating a large quantity of high-quality parallel corpora for use in statistical machine translation and text mining has been recognized. In general, it takes a great deal of cost to create a bilingual corpus, and its efficient creation method is a technical problem. As a method of creating a bilingual corpus, for example, a document of one language constituting a bilingual document is converted into a dictionary group and converted into a word group of the other language, and the number of matching words between sentences of the other language is calculated. A method of associating sentences by measuring is known (see Non-Patent Document 1).

Tatsuya Ishizaka, Masao Uchiyama, Eiichiro Sumida, Kazuhide Yamamoto, "Construction of a large-scale open source Japanese-English bilingual corpus", Information Processing Society of Japan, 2009-NL-191, p.1-6, May 2009

  However, in the conventional corpus creation method, the number of matching words is measured using a general dictionary, and as a result of associating sentences, a bilingual corpus based on a bilingual document in which technical terms that are not included in the general dictionary are described. When creating a sentence, there are cases where sentences that are not translated are associated with each other when viewed as a whole sentence.

  The present invention has been made in view of such circumstances, and measures the number of matching words using a general dictionary, and creates a higher-quality parallel corpus than when matching sentences. With the goal.

  In order to solve the above-described problems, the present invention provides a bilingual document obtaining unit that obtains a bilingual document in a first language and a second language, a sentence in the first language that constitutes the obtained bilingual document, and the second language. Matching a sentence of a language using the bilingual dictionary of the first language and the second language to obtain a bilingual sentence of the first language and the second language, and the acquired bilingual sentence A word alignment score calculation unit for calculating a word alignment score indicating a probability corresponding to one word of the second language constituting the acquired parallel translation sentence for each word of the first language constituting the sentence; Of each word of the first language for which the alignment is calculated, a pair of a word whose calculated word alignment score is higher than or equal to the threshold and one word of the second language is used as a bilingual word In the bilingual dictionary A bilingual word extraction unit, and the bilingual sentence acquisition unit adds the bilingual word to the bilingual dictionary by the bilingual word extraction unit, and A bilingual sentence extracting apparatus that matches a sentence with a sentence in the second language using a bilingual dictionary to which the bilingual word is added, and acquires a bilingual sentence in the first language and the second language. I will provide a.

  In a preferred aspect, the threshold value is preset based on the number of parallel translation words to be added to the parallel translation dictionary.

  The present invention is also a bilingual sentence extraction method executed by one or more computers, the step of obtaining a bilingual document in a first language and a second language, and the first bilingual document constituting the obtained bilingual document. Matching a sentence of a language and a sentence of the second language using a bilingual dictionary of the first language and the second language to obtain a bilingual sentence of the first language and the second language; Calculating a word alignment score indicating a probability corresponding to one word of the second language constituting the acquired bilingual sentence for each word of the first language constituting the acquired bilingual sentence; Of each word of the first language for which the alignment is calculated, a pair of a word whose calculated word alignment score is higher than or equal to the threshold and one word of the second language is used as a bilingual word Said A step of adding to the translation dictionary; and after the bilingual word is added to the bilingual dictionary, the bilingual word adds the sentence of the first language and the sentence of the second language constituting the acquired bilingual document There is provided a bilingual sentence extraction method comprising a step of performing matching using the translated bilingual dictionary and obtaining a bilingual sentence in the first language and the second language.

  According to another aspect of the present invention, a computer acquires a bilingual document in a first language and a second language, a sentence in the first language and a sentence in the second language that constitute the acquired bilingual document, Matching using a bilingual dictionary of the first language and the second language to obtain one or more bilingual sentences of the first language and the second language, and the first constituting the acquired bilingual sentence For each word in one language, a step of calculating a word alignment score indicating a probability corresponding to one word of the second language constituting the acquired parallel translation sentence; and a step of calculating the word alignment of the first language Adding a pair of each word that has a calculated word alignment score higher than or equal to a threshold value and one word in the second language to the bilingual dictionary as a bilingual word; versus After a word is added to the bilingual dictionary, the sentence of the first language and the sentence of the second language constituting the acquired bilingual document are matched using the bilingual dictionary to which the bilingual word is added. , And a program for executing the step of obtaining the parallel translation of the first language and the second language.

  According to the present invention, it is possible to create a bilingual corpus having higher quality than a case where the number of matching words is measured using a general dictionary and sentences are associated with each other.

It is a block diagram which shows an example of a structure of the bilingual sentence extraction apparatus. It is a flowchart which shows an example of a bilingual sentence extraction process. It is a figure which shows an example of a bilingual document. It is a figure which shows an example of the data of the parallel translation memory | storage part. It is a figure which shows an example of the data of the parallel translation memory | storage part. It is a figure explaining an example of matching of a Japanese sentence and an English sentence.

1. Embodiment 1-1. Configuration FIG. 1 is a block diagram illustrating an example of a configuration of a parallel translation extraction apparatus 1 according to the present embodiment. The bilingual sentence extraction device 1 is a computer including an arithmetic processing device such as a CPU and a storage device such as an HDD. The bilingual sentence extraction apparatus 1 includes a bilingual document storage unit 101, a bilingual document acquisition unit 102, a word division unit 103, a bilingual dictionary storage unit 104, a bilingual sentence acquisition unit 105, a bilingual sentence storage unit 106, a word It has functions of an alignment score calculation unit 107, a parallel word storage unit 108, and a parallel word extraction unit 109. Among these functions, the functions of the bilingual document storage unit 101, the bilingual dictionary storage unit 104, the bilingual sentence storage unit 106, and the bilingual word storage unit 108 are realized by a storage device. Other functions are realized by the arithmetic processing device executing a program stored in the storage device.

  The bilingual document storage unit 101 stores bilingual documents in the first language and the second language. Here, the first language is Japanese and the second language is English. A bilingual document is a pair of a Japanese document and an English document created by translating the document into English. The bilingual document is, for example, a pair of a Japanese patent application patent publication and a US patent application patent publication belonging to the same patent family. Or it is a pair of a Japanese newspaper article and an English version of the newspaper article. Alternatively, it is a pair of an English manual of open source software and a Japanese translation of the manual.

  The parallel translation document acquisition unit 102 acquires a parallel translation document from the parallel translation document storage unit 101.

  The word division unit 103 divides the bilingual document acquired by the bilingual document acquisition unit 102 into sentences and divides each sentence into words. For Japanese documents, morphological analysis is performed to divide the sentence into sentences with clues as clues, and each sentence into words. At that time, the utilization word may be converted into a basic form. For an English document, it is divided into sentences using a period as a clue, and each sentence is divided into words using a space as a clue. At that time, the ending word may be analyzed to convert the utilization word into a basic form. Further, upper case letters may be converted to lower case letters, and plural forms may be converted to singular forms. In another embodiment, the word dividing unit 103 divides the bilingual document acquired by the bilingual document acquiring unit 102 into sentences and divides each sentence into words using another known method. Also good.

  The bilingual dictionary storage unit 104 stores a bilingual dictionary. Here, the bilingual dictionary is a set of pairs of Japanese words and English words having the same meaning as the words.

  The bilingual sentence acquisition unit 105 matches the Japanese sentence and the English sentence cut out by the word dividing unit 103 using a bilingual dictionary stored in the bilingual dictionary storage unit 104, and thereby obtains one or more of Japanese and English Get the translation of Specifically, the bilingual sentence acquisition unit 105 converts the English sentence extracted by the word dividing part 103 into a dictionary and converts it into a Japanese word group, and converts the entire pair of Japanese sentences and English sentences. Acquire a parallel translation of a Japanese sentence and an English sentence that maximizes similarity. Here, the similarity is a value calculated on the basis of the number of matching words between a Japanese word group obtained by lexicographically writing an English sentence and a Japanese sentence. More specifically, it is a value expressed by the ratio of the number of independent words that match between the Japanese word group and the number of all independent words included in the Japanese sentence. For example, the bilingual sentence acquisition unit 105 acquires bilingual sentences using DP (Dynamic Programming) matching as in the bilingual corpus creation method described in the above non-patent document. As another example, the bilingual sentence acquisition unit 105 uses the DP matching as described in Takehito Utsuro, et al. “Bilingual Text Matching using Bilingual Dictionary and Statistics,” COLING, p.1076-1082, 1994. May be obtained. Here, the bilingual sentence is a pair of a Japanese sentence and an English sentence created by translating the sentence into English. In other words, it is a pair of a Japanese sentence and an English sentence having the same meaning as the sentence.

  In addition, the bilingual sentence acquisition unit 105 adds the bilingual word to the bilingual dictionary by the bilingual word extraction unit 109, and then uses the bilingual dictionary to which the bilingual word is added. Match one sentence to another sentence in English to obtain one or more parallel translations in Japanese and English. The number of times the parallel translation acquisition unit 105 repeats the parallel translation acquisition processing is determined by, for example, the user of the parallel translation extraction device 1.

  The parallel translation storage unit 106 stores the parallel translation acquired by the parallel translation acquisition unit 105 (in other words, the parallel corpus). At that time, the parallel translation storage unit 106 stores each parallel translation in association with a parallel translation ID that identifies the parallel translation.

  The word alignment score calculation unit 107 calculates a word alignment score indicating the probability that a Japanese word corresponds to an English word for the bilingual sentence stored in the bilingual sentence storage unit 106. In that case, the word alignment score calculation part 107 calculates the word alignment score which shows the probability corresponding to one English word which comprises a bilingual sentence about each Japanese word which comprises a bilingual sentence. In addition, the word alignment score calculation unit 107 similarly calculates a word alignment score indicating the probability corresponding to each Japanese word constituting the bilingual sentence for each of all other English words constituting the bilingual sentence. To do. Here, the probability that the Japanese word corresponds to the English word is, in other words, the probability that one word is a translation of the other word. The word is specifically an independent word, and more specifically, a noun, a verb, an adjective, and an adverb.

  The word alignment score calculation unit 107 uses an IBM (registered trademark) model when calculating the word alignment score. See, for example, Peter F. Brown, et al. “The Mathematics of Statistical Machine Translation: Parameter Estimation, Computational Linguistics”, 19 (2): 263-311 (1993) for the IBM® model. Specifically, the word alignment score calculation unit 107 uses GIZA ++ (http://www.fjoch.com/GIZA++.html). For GIZA ++, see Franz Josef Och, Hermann Ney. "A Systematic Comparison of Various Statistical Alignment Models", Computational Linguistics, volume 29, number 1, pp. 19-51 March 2003.

  The parallel word storage unit 108 stores the word alignment score calculated by the word alignment score calculation unit 107 and the Japanese and English parallel words for which the score is calculated in association with each other. At that time, the parallel word storage unit 108 stores each parallel word in association with a parallel word ID for identifying the parallel word.

  The bilingual word extraction unit 109 adds a bilingual word whose calculated word alignment score is higher than the threshold among the bilingual words newly stored in the bilingual word storage unit 108 to the bilingual dictionary stored in the bilingual dictionary storage unit 104. to add. In other words, the bilingual word extraction unit 109 calculates a word whose calculated word alignment score is higher than a threshold among Japanese words whose word alignment is calculated with respect to one English word, and the English One word pair is added as a bilingual word to the bilingual dictionary stored in the bilingual dictionary storage unit 104. Here, the threshold value is a constant value.

1-2. Operation The operation of the bilingual sentence extraction device 1 will be described. FIG. 2 is a flowchart showing an example of the bilingual sentence extraction process executed by the bilingual sentence extracting apparatus 1.

  In step S <b> 1 of the parallel translation extraction process, the parallel translation document acquisition unit 102 of the parallel translation extraction apparatus 1 acquires a parallel translation document from the parallel translation document storage unit 101. FIG. 3 is a diagram illustrating an example of a bilingual document.

  When the bilingual document is acquired by the bilingual document acquisition unit 102, the word dividing unit 103 divides the acquired bilingual document into sentences and divides each sentence into words (step S2). For Japanese documents, morphological analysis is performed to divide the sentence into sentences with clues as clues, and each sentence into words. For an English document, it is divided into sentences using a period as a clue, and each sentence is divided into words using a space as a clue.

  When the bilingual document is divided into sentences by the word dividing unit 103 and each sentence is divided into words, the acquired bilingual sentence acquiring unit 105 sets an initial value “1” in the variable i (step S3), The Japanese sentence and the English sentence cut out by the word dividing unit 103 are matched using the bilingual dictionary stored in the bilingual dictionary storage unit 104 to obtain a Japanese and English bilingual sentence (step S4). . When the parallel translation is acquired, the parallel translation acquisition unit 105 stores each parallel translation in association with the parallel translation ID in the parallel translation storage unit 106 (step S5). FIG. 4 is a diagram illustrating an example of data in the parallel translation storage unit 106 in which the parallel translation is stored by the parallel translation acquisition unit 105.

  When the bilingual sentence is stored in the bilingual sentence storage unit 106 by the bilingual sentence acquisition unit 105, the word alignment score calculation unit 107 calculates a Japanese word and an English word for the bilingual sentence stored in the bilingual sentence storage unit 106. A word alignment score indicating the probability corresponding to is calculated (step S6). When the word alignment is calculated, the word alignment score calculation unit 107 associates each calculated word alignment score, the Japanese and English translation words for which the score is calculated, and the bilingual word ID of the bilingual word in parallel. It memorize | stores in the word memory | storage part 108 (step S7). FIG. 5 is a diagram illustrating an example of data in the parallel word storage unit 108 in which the word alignment score is stored by the word alignment score calculation unit 107.

  When the word alignment score calculation unit 107 stores the word alignment score or the like in the parallel word storage unit 108, the parallel word extraction unit 109 calculates the calculated word from the parallel words newly stored in the parallel word storage unit 108. A bilingual word whose alignment score is higher than the threshold value is added to the bilingual dictionary stored in the bilingual dictionary storage unit 104 (step S8). For example, when the threshold is set to “0.3”, in the example illustrated in FIG. 5, the parallel translations with IDs “030” and “043” are added to the parallel translation dictionary.

When the bilingual word is added to the bilingual dictionary by the bilingual word extraction unit 109, the bilingual sentence acquisition unit 105 increments the value of the variable i (step S9), and whether or not the value of the variable i is larger than the closing price n. Whether or not is determined (step S10). Here, the closing price n indicates the number of times step 4 is executed. If the result of this determination is that the value of the variable i is less than or equal to the closing price n (step S10: NO), the bilingual sentence acquisition unit 105 returns to step S4 and uses the bilingual dictionary with the newly added bilingual word. The step is executed. As a result of this determination, when the value of the variable i is larger than the closing price n (step S10: YES), the bilingual sentence extraction process ends.
The above is the description of the bilingual sentence extraction process.

  According to the bilingual sentence extraction device 1 described above, after a bilingual sentence is acquired from a bilingual document by DP matching using a bilingual dictionary, a bilingual word is extracted from the acquired bilingual sentence by word alignment and added to the bilingual dictionary. Then, a bilingual sentence is acquired again from the bilingual document by DP matching using the bilingual dictionary to which the bilingual word is newly added. In this way, in this bilingual sentence extraction apparatus 1, the bilingual sentence is acquired from the bilingual document using the bilingual dictionary updated based on the bilingual sentence acquired from the bilingual document, and therefore the word matching degree is calculated. As a result, a higher quality bilingual corpus can be created.

  For example, referring to the example shown in FIG. 6, if the term “monoclonal” is not registered in the bilingual dictionary, the English sentence A is converted into the Japanese sentence A in step S4 of the bilingual sentence extracting process. It is impossible to determine which of B and B should be associated. This is because the English word “monoclonal” cannot be translated into “monoclonal”, and as a result, any Japanese sentence between the English sentence A and “protein”, “specific”, “recognition”, “antibody” This is because a total of five words “refining” and “purification” match (that is, the number of matching words is “5”). However, assuming that a bilingual sentence is obtained from the bilingual document and word translation is performed on the bilingual sentence, the bilingual words “monoclonal” and “monoclonal” are newly added to the bilingual dictionary. When DP matching is performed on a bilingual document using the bilingual dictionary added to, the English word “monoclonal” is translated into “monoclonal”. It becomes “6”, and both are easily associated as parallel translations.

2. Modifications The above embodiments may be modified as described below. One or more modifications described below may be combined with each other.

2-1. Modification 1
The bilingual sentence extraction apparatus 1 may be a computer system including a plurality of computers. The storage device included in the bilingual sentence extraction device 1 according to the above embodiment may be connected to the bilingual sentence extraction device 1 via a communication line such as the Internet.

2-2. Modification 2
In the above embodiment, the first language may be English and the second language may be Japanese. The combination of the first language and the second language may be arbitrarily selected from natural languages such as German, French, Chinese, Korean, etc. in addition to Japanese and English.

2-3. Modification 3
The bilingual sentence acquisition unit 105 converts the Japanese sentence extracted by the word dividing unit 103 into a dictionary, converts it into an English word group, calculates the similarity to each English sentence, and calculates the similarity A pair of an English sentence and a Japanese sentence having the maximum may be acquired as a parallel translation.

2-4. Modification 4
Said word alignment score calculation part 107 may calculate a word alignment score using word alignment tools other than GIZA ++. For example, using Berkeley Aligner (https://code.google.com/archive/p/berkeleyaligner/) or PostCAT (http://www.seas.upenn.edu/~strctlrn/CAT/CAT.html) Also good.

  Further, the word alignment score calculation unit 107 may calculate the word alignment score using a model other than the IBM (registered trademark) model. For example, heuristic models based on heuristics such as dice coefficients and log-likelihood ratios, or supervised word alignment may be used.

2-5. Modification 5
The threshold value referred to by the parallel word extraction unit 109 may be automatically set based on the number of parallel words to be newly added to the parallel dictionary. For example, for each bilingual word extraction (step S8), when it is desired to newly add 10% of the bilingual words already stored in the bilingual dictionary, the corresponding number of bilingual words are extracted. A threshold value may be set as described above.

  Alternatively, the bilingual word extraction unit 109 calculates the word having the maximum calculated word alignment score from the Japanese words for which word alignment is calculated with respect to one English word, and the English one A word pair may be added to the bilingual dictionary as a bilingual word.

2-6. Modification 6
If no parallel translation word is added to the parallel translation dictionary as a result of step S8 of the parallel translation extraction process, the parallel translation acquisition unit 105 determines that the value of the variable i is equal to or less than the closing price n in the determination of step S10. Even so, the bilingual sentence extraction process may be terminated.

2-7. Modification 7
A program for realizing each function of the bilingual sentence extraction apparatus 1 according to the above-described embodiment or modification may be provided via a recording medium readable by a computer apparatus. Here, the recording medium is, for example, a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, or a semiconductor memory. In addition, this program may be provided via a network such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Bilingual sentence extraction apparatus, 101 ... Bilingual document memory | storage part, 102 ... Bilingual document acquisition part, 103 ... Word division part, 104 ... Bilingual dictionary memory | storage part, 105 ... Bilingual sentence acquisition part, 106 ... Bilingual sentence memory | storage part, 107 ... Word alignment score calculation unit, 108 ... parallel word storage unit, 109 ... parallel word extraction unit

Claims (4)

A bilingual document acquisition unit for acquiring bilingual documents in the first language and the second language;
The first language sentence and the second language sentence constituting the acquired bilingual document are matched using the first language and the second language bilingual dictionary, and the first language and the second language sentence are matched. A bilingual acquisition unit for acquiring bilingual bilingual sentences;
A word alignment score that calculates a word alignment score indicating a probability corresponding to one word of the second language constituting the acquired parallel translation sentence for each word of the first language constituting the acquired parallel translation sentence A calculation unit;
Of each word in the first language for which word alignment has been calculated, a pair of a word whose calculated word alignment score is higher than or equal to a threshold and one word in the second language is a bilingual word A bilingual word extraction unit to be added to the bilingual dictionary as
The bilingual sentence acquisition unit, after the bilingual word is added to the bilingual dictionary by the bilingual word extraction unit, the sentence of the first language and the sentence of the second language that constitute the acquired bilingual document, The bilingual sentence extraction device, wherein matching is performed using the bilingual dictionary to which the bilingual word is added, and the bilingual sentence of the first language and the second language is acquired.   The bilingual sentence extraction apparatus according to claim 1, wherein the threshold value is preset based on the number of bilingual words to be added to the bilingual dictionary. A bilingual sentence extraction method executed by one or more computers,
Obtaining a bilingual document in a first language and a second language;
The first language sentence and the second language sentence constituting the acquired bilingual document are matched using the first language and the second language bilingual dictionary, and the first language and the second language sentence are matched. Obtaining bilingual translations;
Calculating a word alignment score indicating a probability corresponding to one word of the second language constituting the acquired bilingual sentence for each word of the first language constituting the acquired bilingual sentence;
Of each word in the first language for which word alignment has been calculated, a pair of a word whose calculated word alignment score is higher than or equal to a threshold and one word in the second language is a bilingual word Adding to the bilingual dictionary as
After the bilingual word is added to the bilingual dictionary, the sentence in the first language and the sentence in the second language that constitute the acquired bilingual document are matched using the bilingual dictionary to which the bilingual word is added. And obtaining a parallel translation sentence of the first language and the second language. On the computer,
Obtaining a bilingual document in a first language and a second language;
The first language sentence and the second language sentence constituting the acquired bilingual document are matched using the first language and the second language bilingual dictionary, and the first language and the second language sentence are matched. Obtaining one or more parallel translations in two languages;
Calculating a word alignment score indicating a probability corresponding to one word of the second language constituting the acquired bilingual sentence for each word of the first language constituting the acquired bilingual sentence;
Of each word in the first language for which word alignment has been calculated, a pair of a word whose calculated word alignment score is higher than or equal to a threshold and one word in the second language is a bilingual word Adding to the bilingual dictionary as
After the bilingual word is added to the bilingual dictionary, the sentence in the first language and the sentence in the second language that constitute the acquired bilingual document are matched using the bilingual dictionary to which the bilingual word is added. And the program for performing the step which acquires the translation of the said 1st language and the said 2nd language.

Images