JP5565827B2 - A sentence separator training device for language independent word segmentation for statistical machine translation, a computer program therefor and a computer readable medium. - Google Patents

Description

  The present invention relates to preprocessing of natural language processing (NLP), and more particularly to segmenting sentences in a corpus for training a model used for SMT (Statistical Machine Translation) or natural language understanding.

  Word segmentation work, i.e. identifying word boundaries in continuous text, is one of the basic preprocessing steps in data-driven NLP applications such as natural language understanding, information extraction and machine translation. Unlike Indo-European languages such as English, many Asian languages such as Chinese and Japanese do not use white space to distinguish meaningful word units.

  The word segmentation of these languages has the following problems.

  (1) Ambiguity. For example, in Chinese, a single character can be a component in one context and a single word alone in another.

  (2) An unknown word. In other words, combining existing words can result in a new word such as a proper noun such as “White House”.

  Prior art purely dictionary-based approaches address these challenges with maximum matching heuristics, but their accuracy is highly dependent on the coverage of the dictionary used. Recent research on unsupervised word segmentation has focused on approaches based on probabilistic methods. For example, Non-Patent Document 1 proposes a probabilistic segmentation model based on a unigram word distribution, and Non-Patent Document 2 uses a standard n-gram (1 ≦ n ≦ 3) language model. Non-patent document 3 introduces a nonparametric Bayesian inference approach based on the Dirichlet process, which incorporates unigram and bigram word dependencies.

M.M. Brent. An efficient and stochastic algorithm for segmentation and word discovery. Machine learning, 34: 71-105, 1999. (M. Brent. An efficient, probabilistically sound algorithm for segmentation and word discovery. Machine Learning, 34: 71-105, 1999.) A. Ben Kataraman. A statistical model for word discovery in transcription utterances. Computer language, 27 (3): 351-372, 2001. (A. Venkataraman. A statistical model for word discovery in transcribed speech. Computational Linguistics, 27 (3): 351-372, 2001.) S. Gold water. Context dependency in unsupervised word segmentation. ACL Proceedings, 673-680, Sydney, Australia, 2006. (Contextual Dependencies in Unsupervised Word Segmentation. In Proc. Of the ACL, pages 673-680, Sydney, Australia, 2006.) J. et al. Shu, J. Gao, K. Toutanova and H. Ney. Bayes semi-supervised Chinese word segmentation for SMT. COLING Proceedings, pp. 1017-1024, Manchester, UK, 2008. (J. Xu, J. Gao, K. Toutanova, and H. Ney. Bayesian Semi-Supervised Chinese Word Segmentation for SMT. In Proc. Of the COLING, pages 1017-1024, Manchester, UK, 2008.) A. Ratnaparki. Maximum entropy model for part-of-speech tagging. EMNLP Proceedings, Pennsylvania, USA, 1996. (A. Ratnaparkhi. A Maximum Entropy Model for Part-Of-Speech Tagging. In Proc. Of the EMNLP, Pennsylvania, USA, 1996.)

  However, little attention has been paid to learning word segmentation that is optimal for machine translation work in languages that do not use delimiters. For small translation units, such as single Chinese or Japanese characters, such tokens are often found in training corpora, so these tokens can be translated with SMT, but these The context information given by the token may not be enough to get a good translation.

  For example, a Japanese-English SMT would translate two consecutive characters “white” (“white”) and “bird” (“bird”) to “white bird”, but for humans these consecutive characters “Swan” would be translated as “swan”. Therefore, the longer the translation unit, the more context can be used to find meaningful translations. On the other hand, the longer the translation unit, the less likely that such tokens will occur in the training data due to the sparseness of the language resources used to train the statistical translation model.

  Therefore, the word segmentation that is “optimal” for machine translation is small enough to make sense in the context of a given input sentence, and is symmetric with the complexity of the translation work of a statistical model to optimize the quality of the translation. It can be said that the translation unit that achieves the trade-off with the scope is specified. Using a monolingual probabilistic model does not necessarily produce good machine translation performance. However, improvements have been made from several approaches that consider not only monolingual but also bilingual information, and what has been proposed to derive word segmentation suitable for SMT. In view of the availability of language resources, recent research has focused on the optimization of Chinese word segmentation (CWS) for SMT from Chinese to English. For example, Non-Patent Document 4 proposes a Bayes semi-supervised approach for CWS based on Non-Patent Document 3. This generative model first segments Chinese text using a commercially available separator to learn new word types and distributions suitable for SMT. Furthermore, segmentation consistency and translation unit granularity are also important to improve CWS.

  However, the system disclosed in Non-Patent Document 3 requires a linguistically derived word segmentation tool to obtain the initial segmentation.

  Accordingly, one object of the present invention is to provide a sentence separator training device that does not require a linguistically guided word segmentation tool.

  Another object of the present invention is to provide a sentence separator training device that does not require a linguistically derived word segmentation tool and automatically generates an optimal sentence separator.

  According to a first aspect of the present invention, a sentence separator training device for training a preselected first language sentence separator includes means for storing a bilingual corpus of translation pairs, Each pair includes a source language sentence of the first language and a target language sentence of a second language, and further for segmenting the source language sentence in the bilingual corpus into character strings by a predetermined delimiter A first training means for training SMT means using a bilingual training corpus including a first language-based separator and a translation pair of sentences of the first language and the second language; The SMT means associates each of the translation pairs in the bilingual training corpus during training, Training the second separator of the source language sentence of the bilingual corpus using the evaluation means for evaluating the performance of the SMT means trained by the means and the result of the association by the SMT means Second training means and the source language sentence in the bilingual corpus are separated by the predetermined delimiter using the second separator trained by the second training means. The segmentation means for separating into segment rows, the first training means, the evaluation means, the second training means, and the second separator have predetermined termination conditions relating to the performance of the SMT. Repetitive control means for controlling repetitive operation until satisfied, and the repetitive control In the first iteration of the means, select one of the bilingual corpora containing source language sentences segmented by the character-based first separator and then segmented by the second separator in subsequent iterations Selecting the bilingual corpus having the source language sentence, and selecting means for causing the first training means to train the SMT means using the selected bilingual corpus as the bilingual training corpus. In addition.

  The first separator segments the source language sentences in the bilingual corpus into character strings that are segmented at a predetermined delimiter. In the first iteration, the selection means selects a bilingual corpus that includes source language sentences segmented by the first character-based separator. The first training means trains the SMT means using the selected corpus as a bilingual training corpus. During training, the SMT means associates each translation pair of the bilingual training corpus. The second training means trains the second separator using the result of association by the SMT means. The segmenting means segments the source language sentences in the bilingual corpus into segment strings using the second separator trained by the second training means. In the second iteration, the selection means selects the bilingual corpus segmented by the segmentation means. The first training means trains the SMT means using the selected corpus as a bilingual training corpus. At the end of each iteration of SMT training, the evaluation means evaluates the performance of the SMT means trained by the training means. The repetition control means stops the repetition when a predetermined end condition is satisfied. For example, if the evaluation deteriorates, the repetition is stopped.

  Since the first iteration uses a bilingual corpus with source language sentences segmented into strings, segmentation does not require any language specific separator. The separator used after the next iteration can be trained according to the result of the correspondence obtained during the training of the SMT means. Thus, a sentence separator training device is provided that is language independent and does not require a linguistically derived word segmentation tool.

  In a second aspect based on the first aspect, the second training means annotates each character in the source language sentence of the bilingual corpus using the result of the association by the SMT means, Means for giving each character an annotation indicating whether or not each character is the end of a word; and means for extracting a predetermined feature amount set of each character in the source language sentence of the bilingual corpus And the predetermined feature amount set includes a context of a target character of the source language sentence and a context of a word associated with the target character in the target language sentence paired with the source language sentence. Means for reflecting and further training a probability model used in the second separator, wherein the probability model is extracted by the extraction means By the characteristic quantity of the set of statistical analysis was used to characters in the source language sentence is to estimate whether the probability or is the end of a word.

  At the end of the training of the SMT instrument, the bilingual corpus source language sentence is annotated. A predetermined feature set is extracted from the annotated bilingual corpus, and the probability model used in the second separator determines the feature set and the existing statistical machine translation scheme. Can be used for statistical training.

  Accordingly, a sentence separator training device is provided that automatically generates a statistically optimized sentence separator without the need for linguistically derived word segmentation tools.

  In a third aspect based on the second aspect, the probability model includes a maximum entropy (ME) model.

  The result of mapping during SMT training often includes several mapping errors. However, training the ME model using correspondence can reduce the likelihood of errors, and therefore does not require a linguistically derived word segmentation tool and is a relatively accurate sentence. A sentence separator training device is provided that automatically generates a separator.

  According to a fourth aspect based on the third aspect, the iterative control means includes the first training means, the evaluation means, the second training means, and the second separator according to the evaluation means. Control is performed so that the evaluation is repeated until the evaluation becomes worse than the evaluation of the evaluation means in the preceding repetition. Thus, a sentence separator training device is provided that automatically generates an optimal sentence separator without the need for linguistically derived word segmentation tools. Since optimization is performed using SMT and automatic training of the separator, the resulting separator differs from that trained using a human-segmented bilingual corpus, and is segmented Will be more suitable for SMT.

  A fifth aspect of the present invention relates to a computer program that, when executed on a computer, causes the computer to function as an apparatus based on any of the first to fourth aspects.

  According to a sixth aspect of the present invention, a computer readable medium stores a computer program according to the fifth aspect.

FIG. 3 is a diagram illustrating a method of repeated bootstrap processing according to one embodiment of the present invention. It is a figure which shows the translation pair of a source language and a target language. It is a flowchart of the program which implement | achieves 1st Example of this invention on a computer. It is a figure which shows the example of the original translation pair and the translation pair by which the source language sentence was segmented on the character base. It is a figure which shows the segmentation example of the source language sentence in the various repetition of the repeated bootstrap method. It is a figure which shows the segmentation example of the source language sentence in the various repetition of the repeated bootstrap method. It is a figure which shows the change of the system performance of an experimental system. It is a figure which shows the change of the system performance of an experimental system. It is a figure which shows the change of the vocabulary size and length of an experimental system. It is a figure which shows the change of the vocabulary size and length of an experimental system. It is a figure which shows the change of the out-of-vocabulary (OOV) size of an experimental system. 2 is a front view of a computer system 320. FIG. 2 is a block diagram of a computer system 320. FIG.

[First embodiment]
(Overview)
In contrast to previous approaches, the present invention proposes a language-independent approach that does not require the presence of a linguistically derived word segmentation tool to obtain the initial segmentation. The proposed method uses a parallel corpus to associate source language sentences that are character strings with word units separated by blank spaces in the target language. It becomes a larger source language unit in which consecutive characters associated with the same target word are integrated. Therefore, the granularity of translation units is defined in a given bilingual corpus context. Translation quality of an SMT system trained on a re-segmented bilingual corpus using the Maximum-Entropy (ME) algorithm to minimize the side effects of miscorrespondence and to keep the segmentation consistent Learning of source language word segmentation is performed.

  Modern SMT systems have embedded token-word association subsystems such as GIZA ++. Such a subsystem is known to output the most probable association between the tokens of the source language sentence and the words of the target language sentence, but the accuracy of the association is sometimes questionable.

  An experiment was conducted in which the proposed segmentation method was applied to translation from five Asian languages (Japanese, Korean, Thai, Chinese (standard Chinese, Taiwanese)) into English. As a result of experiments, the proposed method performs better than a baseline system that translates textual source language sentences and produces similar translation results as a SMT module trained in a bilingual corpus segmented with linguistic tools. I knew I would get it.

(Word segmentation)
The proposed word segmentation method consists of two steps. In the first step, a standard SMT model is trained on a parallel text corpus consisting of unigram segmented source language strings and target language words separated by white space. Using the results of the character-word association of the SMT training procedure, each source's bilingual corpus identifies consecutive source language characters associated with the same target language word, and these characters are integrated to create a larger A translation unit.

  In the second step, the word segmentation task is treated as character tagging, where only two tags are used. That is, “WB” (word boundary) when a given source language character is the last character of an integrated character string associated with a word in the target language, and “NB” (no) in other cases (borderary: non-boundary). Using the association-based word boundary annotation, the ME method is applied to learn the optimal source language word segmentation.

(1) Maximum entropy tagging model The ME model provides a general-purpose machine learning technique for classification and prediction. These are versatile tools that can handle many features and have proved very effective in a wide range of NLP tasks including sentence boundary detection or part-of-speech tagging. The maximum entropy classifier is an exponential model and consists of a number of binary feature functions and their weights. The model is trained by adjusting the weights to maximize the entropy of the stochastic model due to constraints imposed by the training data. In the experiment, a conditional maximum entropy model is used, and the conditional probability of the result for a given feature set is modeled in Non-Patent Document 5. The model has the following form:

ここで、
ｔは予測されるタグであり、
ｃはｔのコンテキストであり、
ｘは正規化係数であり、
Ｋはモデル内の特徴量の数であり、
ｆ_ｋは二値特徴量関数であり、
α_ｋは特徴量関数ｆ_ｋの重みであり、
ｐ_０はデフォルトモデルである。

here,
t is the expected tag,
c is the context of t,
x is a normalization factor,
K is the number of features in the model,
f _k is a binary feature quantity function;
α _k is a weight of the feature quantity function f _k ,
p ₀ is the default model.

Table 1 shows a set of feature amounts. The context feature by the dictionary includes a source language character string annotated (tagged) with a tag t. c ₀ indicates a tagged context unit (eg character or word) and c ₋₂ ,... c ₊₂ indicate surrounding context units. t ₀ indicates the current tag, t ₋₁ indicates the preceding tag, and so on. The tag context feature quantity supplies information related to the context of the preceding tag string. This conditional model can be used as a classifier. The model was iteratively trained and an improved iterative scaling algorithm (IIS) was used for the experiments.

（２）繰返しブートストラップ法
ＳＭＴの最適単語セグメント化学習のための、本発明により提案される繰返しブートストラップ法を図１にまとめた。図１を参照して、ターゲット言語文３２とソース言語文３４とを含むバイリンガルコーパス３０を準備する。ターゲット言語文３２の各々がその翻訳のソース言語文と対になっている。

(2) Iterative Bootstrap Method The iterative bootstrap method proposed by the present invention for optimal word segmentation learning of SMT is summarized in FIG. Referring to FIG. 1, a bilingual corpus 30 including a target language sentence 32 and a source language sentence 34 is prepared. Each target language sentence 32 is paired with the source language sentence of the translation.

  Referring to FIG. 2, translation pair 110 includes a source language sentence 112 and a target language sentence 114 that is a translation of the source language sentence 112.

  Referring to FIG. 1 again, in the first iteration (0th iteration), each of the source language sentences 34 is divided into unigram segmented source language sentences 38 for each character by the unigram separator 36. Unigram separator 36 simply inserts a space between each adjacent character in source language sentence 34.

SMT 40 is trained using a bilingual corpus that includes target language sentence 32 and unigram segmented source language sentence 38. Since this is the first iteration, this SMT 40 is called “SMT ₀ ”. During the training of SMT ₀ 40, each of the sentence pairs of the target language sentence 32 and the unigram segmented source language sentence 38 is associated.

Before the next iteration begins, SMT ₀ 40 was evaluated by decoding a source language sentence development set (not shown) into a target language sentence, and the decoded result was further proposed by BLEU (proposed by K. Papineni, BLEU: Automatic Evaluation Method for Machine Translation, 40th ACL Proceedings, pages 311-318, Philadelphia, US, 2002, K. Papineni, “BLEU: a Method for Automatic Evaluation of Machine Translation”, in Proceedings of the 40th ACL, pages 311-318, Philadelphia, US, 2002)) or METEOR (S. Benergi et al., “METEOR: Automated Scale for MT Evaluation” ACL Proceedings, pages 65-72, Ann Arbor, US , 2005 (S. Banerjee et al., “METEOR: An Automatic Metric for MT Evaluation” in Proceedings of the ACL, pages 65-72, Ann Arbor, US, 2005.) Therefore, evaluate. The score of the evaluation result 42 is stored. During the training of SMT ₀ 40, a token-word association result 44 is extracted. Results 44 can be used to annotate each token in unigram segmented source language sentence 38. A feature set for each token in the annotated source language sentence 38 is extracted and used to train the ME classifier 46 (ME ₁ ) capable of handling longer translation units.

After learning the ME classifier 46 from the first letter-word association of each bilingual corpus 30, the resulting ME classifier 46 is applied to re-segment the source language sentence 34 of the unsegmented parallel corpus As a result, a segmented bilingual corpus including the target language sentence 32 and the source language sentence 48 is obtained instead, and this can be used to retrain and re-evaluate another SMT (SMT ₁ ) 50. This can be expected to achieve better translation performance than the first SMT (SMT ₀ ).

The unsupervised ME tagging method can also be applied to token-word associations extracted during SMT ₁ training, resulting in a ME classifier 56 (ME ₂ ) capable of handling longer translation units. be able to.

In this example, the unigram segmented source language sentence 38 is annotated with the result of the association 44 by SMT ₀ 40. For example, in the training of SMT ₀ 40, if it is determined that a certain character is the end of a word, the word is labeled “WE” (Word End), otherwise “NE”. (Not End: non-terminal). Train ME classifiers using annotated source language sentences. In this example, for each annotated character of the unigram segmented source language sentence 38, a set of context features as shown in Table 1 is derived. The ME classifier 46 is trained to maximize the entropy of the probability model under the given constraints imposed by the training data. The ME model is statistically trained with a set of features. In this embodiment, as described above, a conditional maximum entropy model is used for the ME classifier 46.

The next iteration is called the “first” iteration. In the first iteration, the source language sentence 34 is segmented by the ME classifier 46, resulting in a segmented source language sentence 48. The SMT 50 (SMT ₁ ) is trained using a bilingual corpus that includes the target language sentence 32 and the segmented source language sentence 48. During training, each segment of segmented source language sentence 48 is associated with a corresponding word in target language sentence 32. The matching result 54 is extracted from SMT ₁ 50 and is used to annotate the segmented source language sentence 48. The next iterative ME classifier 56 (ME ₂ classifier) is trained using the annotated segmented source language sentence 48.

On the other hand, the performance of SMT ₁ 50 is evaluated by decoding the source language development set statement. The evaluation result 52 is compared with the stored evaluation result 42 of the first iteration. If result 52 is better than result 42, the iteration continues. Otherwise, the iteration is stopped at this stage and the ME classifier 46 is output as the optimal classifier for segmenting the source language sentence.

If the result 52 is better than the saved result 42, the evaluation result is saved and the source language sentence 34 is segmented by the ME classifier 56 resulting in a segmented source language sentence 58. SMT 60 (SMT ₂ ) is trained using a bilingual corpus including target language sentence 32 and segmented source language sentence 58. Source language sentence matching results (not shown) during SMT ₂ 60 training are extracted. The performance of SMT ₂ 60 is evaluated with an automatic evaluator. If the SMT ₂ 60 evaluation result 62 is worse than the stored result 52, the iteration is terminated and the ME classifier 46 is output as the optimal classifier. If the evaluation result 62 is better than the stored result 52, the next iteration is performed.

  Training of ME classifiers, segmentation of source language sentences 34 using ME classifiers, training of SMT with a bilingual corpus including segmented source language sentences, and evaluation of SMT performance are thus preceded by evaluation results. Iterate until it becomes worse than the evaluation result.

That is, with reference to FIG. 1, it is assumed that the ME classifier 76 is trained using the bilingual corpus association in the (J-2) th SMT training in the (J-1) th iteration. In the (J-1) th iteration, the source language sentence 34 is segmented by the ME classifier 76. The resulting segmented text 78 is used to train SMT 80 (SMT _J-1 ) along with the target language sentence 32. The performance of SMT _J-1 80 is evaluated. If the evaluation result 82 is better than the previous result, the result 82 is saved and the result of the association in the training of SMT _J-1 80 is extracted. The ME classifier 86 is trained using the matching result 84. Source language sentence 34 is segmented into segmented source language sentence 88. SMT 90 (SMT _J ) is trained using a bilingual corpus that includes target language sentence 32 and segmented source language sentence 88. The performance of the SMT _J 90 is evaluated by an automatic evaluator and the evaluation result 92 is compared with the previous evaluation result 82. Here, it is assumed that the result 92 is worse than the result 82. Here, the iteration is stopped, and the classifier 76 obtained in the preceding iteration is identified and stored as the optimum classifier.

Such a bootstrap method repeatedly generates a series of SMT _i , each time with less translation complexity. This is because larger chunks can be translated in one step without error in word order or word clarification. However, at some point, overfitting occurs due to an increase in the length of the translation unit learned from the training corpus, and the translation performance when translating a sentence that has not been encountered at the time of learning is reduced. Thus, if the Jth re-segmentation of the training corpus resulted in a lower auto-evaluation score for the test set that was not encountered than the previous iteration, the bootstrap method is aborted. The ME classifier 76 (ME _J-1 ) that achieves the highest automatic translation score is selected and output as the final word separator of the proposed method.

(Realization by computer)
Referring to FIG. 3, the computer program that implements this token classifier training apparatus begins at step 140 where the source language sentence 34 of the bilingual corpus 30 is segmented into unigrams to obtain a unigram segmented source language sentence 48; Thereafter, training SMT ₀ 40 using a bilingual corpus including target language sentence 32 and segmented source language sentence 48 is included.

  Referring to FIG. 4, the bilingual corpus 30 includes a number of translation pairs (sentence pairs), such as a sentence pair 240 that includes a source language sentence and a corresponding target language sentence. 4A shows a pair 240 containing manually segmented source sentences, and FIG. 4B shows a pair 242 containing unigram segmented source language sentences. Here, “unigram segmentation” means “segmented into characters”.

  The program further includes the step (144) of evaluating the performance of the SMT using an automatic evaluator such as BLEU or METEOR, obtaining the result of the evaluation and determining whether this is the first iteration (146). )including. If the determination in step 146 is yes, control proceeds to step 150. Otherwise, control proceeds to step 148. In step 148, it is determined whether the evaluation result calculated in step 144 is worse than the preceding result. If the determination is yes, control proceeds to step 164 where the ME classifier obtained in the preceding iteration is output as the optimal classifier and control ends the series of programs. If the determination at step 148 is no, control proceeds to step 150.

  At step 150, the result calculated at step 144 is stored in a memory location.

  The program further includes: storing the ME classifier obtained immediately before in a memory location (152), extracting a matching result from a preceding SMT training step (154), and using the matching result Annotating the source language sentence (156), extracting a feature quantity set for each of the segmented source language sentence tokens (158), and using the extracted feature quantity set Training an iterative ME classifier (160), segmenting a source language sentence with the ME classifier obtained in step 160 (162), and returning control to step 142.

  In the first iteration, a bilingual corpus containing source language sentences segmented into unigrams is selected and used for SMT training. In subsequent iterations, a segmented bilingual corpus using the ME classifier trained in step 160 is selected and used for SMT training. Since the unigram segmentation is character based, the segmentation in step 140 is language independent. Thus, no linguistically derived word segmentation tool is required.

The association in the SMT of the training there is a well-known tool, but the association of the results, it is not a few, including some of the correlation error. Applying the mapping results directly to bilingual corpus segmentation will result in many errors in the results. However, by statistically training the ME classifier using the result of the association of SMT training, the segmentation result of the ME classifier is relatively less error- prone . The resulting ME classifier at the end of the above iteration will be optimal in the sense that the resulting SMT performance is the best among the SMT obtained during the iteration.

  Those skilled in the art will understand that the program configured as described above realizes the operation shown in FIG. 1 when executed by a computer.

(Experiment)
The effect of using various word segmentations was investigated using a multilingual Basic Travel Expressions Corpus (BTEC). This is a collection of texts that bilingual travel specialists may find useful for people traveling to and from abroad. For the purpose of word segmentation experiments, five Asian languages that do not naturally separate word units into separate parts: Japanese (ja), Korean (ko), Thai (th), and two Chinese languages Standard Chinese (zh) and Taiwanese (tw) were selected.

  Table 2 summarizes the characteristics of the BTEC corpus data set used for SMT model training, model weight tuning (dev), and translation quality evaluation (eval). In addition to the number of sentences (sen) and vocabulary (voc), the length of the sentence (len) is also given as the average number of words per sentence.

  For all source languages, given statistics were obtained using linguistic segmentation tools available for each language. Segmentation was not available for Taiwanese.

ＳＭＴモデルのトレーニングについては、Ｆ．オチ及びＨ．ネイ、「統計的対応付けモデルの系統的比較」、コンピュータ言語、２９（１）：１９−５１ページ，２００３年（F. Och and H. Ney, “A Systematic Comparison of Statistical Alignment Models” in Computational Linguistics, 29(1):19-51, 2003）で提案された標準的単語対応付けと、Ａ．ストックル、「ＳＲＩＬＭ、拡張可能ＬＭツールキット」ＩＣＳＬＰ予稿集、第９０１−９０４ページ、デンバー、ＵＳ．２００２年（A. Stolcke, “SRILM an extensible LM toolkit”, in Proceedings of ICSLP, pages 901-904, Denver, US, 2002）によって提案された言語モデル化ツールを使用した。デコーダのパラメータのチューニングには最小誤り率トレーニング（Ｍｉｎｉｍｕｍ ｅｒｒｏｒ ｒａｔｅ ｔｒａｉｎｉｎｇ： ＭＥＲＴ）を用い、オチ及びネイの提案する技術を用いてｄｅｖセットについて行った。翻訳には、Ａ．フィンチら、「ＮＩＣＴ／ＡＴＲ音声翻訳システム」、ＩＷＳＬＴ予稿集、第１０３−１１０ページ、トレント、イタリア、２００７年（A. Finch et al, “The NICT/ATR Speech Translation System” in Proceedings of the IWSLT, pages 103-110, Trento, Italy, 2007）の提案するマルチスタック句ベースデコーダであってオープンソースツールキットＭＯＳＥＳに相当するものを用いた。翻訳品質の評価には、標準的自動評価尺度、すなわちＢＬＥＵ及びＭＥＴＥＯＲを用いた。この実施例の実験結果では、所与のスコアはパーセントの数字でリストされている。

For training on the SMT model, see F.C. Ochi and H. Ney, “Systematic Comparison of Statistical Alignment Models,” Computer Language, 29 (1): 19-51, 2003 (F. Och and H. Ney, “A Systematic Comparison of Statistical Alignment Models” in Computational Linguistics. , 29 (1): 19-51, 2003), and A. Stockl, “SRILM, Extensible LM Toolkit” ICSLP Proceedings, 901-904, Denver, US. The language modeling tool proposed by 2002 (A. Stolcke, “SRILM an extensible LM toolkit”, in Proceedings of ICSLP, pages 901-904, Denver, US, 2002) was used. Decoder parameters were tuned using minimum error rate training (MERT) and dev sets using techniques proposed by Ochi and Nei. For translation, A. Finch et al., “NICT / ATR Speech Translation System”, IWSLT Proceedings, pages 103-110, Trento, Italy, 2007 (A. Finch et al, “The NICT / ATR Speech Translation System” in Proceedings of the IWSLT, pages 103-110, Trento, Italy, 2007), which is a multi-stack phrase base decoder proposed by MOSES equivalent to the open source toolkit. Standard automatic rating scales, namely BLEU and METEOR, were used for translation quality assessment. In the experimental results of this example, a given score is listed as a percentage number.

  Section (1) describes the proposed method of this embodiment as a baseline system for translating source-separated source sentences and an SMT trained by language-dependent linguistically guided word segmentation. Compared. In addition, the effect of the repeated bootstrap method is summarized in section (2).

  Section (1) Effects of word segmentation

種々にセグメント化されたソース言語リソースでトレーニングされたＳＭＴの自動評価スコアを表３に示す。ここで
「文字」は翻訳にユニグラムセグメント化ソーステキストを用いるベースラインシステムを指す。
「本件」はここで提案される最適ＭＥタグ付けモデルによってセグメント化されたコーパスでトレーニングされたＳＭＴである。
「言語学的」は、ｔｗを除き、調査された言語の各々について利用可能な言語依存の言語学的に導かれた単語セグメント化ツールを用いたものである。

Table 3 shows the automatic evaluation scores for SMT trained with various segmented source language resources. Here "character" refers to a baseline system that uses unigram segmented source text for translation.
“This case” is an SMT trained on a corpus segmented by the optimal ME tagging model proposed here.
“Linguistic” is a language-dependent linguistically derived word segmentation tool that is available for each of the investigated languages, except tw.

  From this result, the method according to this embodiment is superior to the baseline system for each of the target languages, and the evaluation scale is consistently 0.8 to 2.0 points gain for BLEU and METEOR. It is shown that a gain of 0.5 to 3.0 points has been achieved. However, improvements depend on each source language and its characteristics. For example, the smallest gain was in Chinese, because a single letter often constitutes a word on its own, so consecutive hiragana or katakana characters are more meaningful This is because the ambiguity is higher than the Japanese language that composes a unit.

  When comparing the method of this example with a linguistically derived separator, the result was a slightly lower word segmentation rating score that was automatically learned for most of the source languages. The results of the method were very close to or better than linguistic segmentation tools, for example for Korean-English tasks. However, for Thai, which has significantly different character types compared to the other character types investigated, there was a big gap between the method and linguistic segmentation.

Features that affect the segmentation method in this case and therefore should be considered when generating the initial corpus segmentation include: (1) Thai character types are a segmental notation system based on consonants, The vowel notation is compulsory, and the character separation of the baseline system affects the consonant dependency. (2) This uses a tone marker placed on the consonant, but is treated as a single character in this approach. (3) Since the vowels that are pronounced after the consonant are discontinuous and can occur before, after, above, or below the consonant, the number of word word form changes in the training corpus increases and the accuracy of the learned ME classifier Reduced.

  Japanese is written using multiple character types (kanji, hiragana, katakana), so adding additional features related to character types to a given token may help improve the performance of the ME classifier.

  Finally, for Taiwanese where linguistic tools were not available for segmenting text using traditional Chinese character types, translations in any language where linguistically derived segmentation tools are not available Also demonstrated that the method of this example can be applied successfully.

Section (2) Effect of Repeated Bootstrap To see the robustness of the method of this example, the changes that appear in the system performance of each source language during the repeated bootstrap method are shown in FIGS. The BLEU results in FIG. 7 and the METEOR results in FIG. 8 reach the best performance in the first iteration for all languages, but then slightly worse but consistently worse with each iteration. It is shown that. The reason for this is the effect of overfitting caused by the concatenation of source tokens that are associated with longer target language phrases, resulting in segmentation of longer translation units.

  Changes in vocabulary size and word length are summarized in FIGS. Referring to FIGS. 9 and 10, the amount of words extracted by this method is considerably larger than that of the baseline system, 10 times for Mandarin Chinese and Taiwanese, 30 times for Japanese and Korean, In words, the vocabulary size increased 100 times. In addition, all the languages studied were 1.5 to 2.5 times larger in size than the vocabulary obtained with linguistic rules. The average vocabulary length also increased with each iteration, and the length of translation units learned after 10 iterations was approximately twice the average word length of the first iteration.

  The problem of overfitting of the method according to this example is that there is an increase in out-of-vocabulary (OOV) words, ie source language words that are included in a data set that has not been encountered, in each SMT. It is shown in the form of an increase in things that cannot be translated. The results given in FIG. 11 show a large increase in OOV words for the first three iterations, which results in reduced translation quality, as listed in FIGS.

  FIGS. 5 and 6 show several translation examples using various segmentation schemes for Japanese-English translation work. In the figure, “REF” indicates manual segmentation and corresponding translation. The repetition that achieves the best translation is marked with an “*”. In the first example shown in FIG. 5, the “already midnight” concatenation according to the method of this example becomes an OOV word, so that only partial translation can be achieved. In the second example shown at the top of FIG. 6, the best translation is done using word segmentation in the first iteration that correctly handles sentence structure, while this information is omitted in the baseline system output. Has been. However, further concatenation of the first sentence causes the same OOV problem. In the third example shown in the lower part of FIG. 6, it is shown that longer tokens reduce translation complexity and thus allow better translation compared to other segmentations that cause more ambiguity. Show.

  Referring to FIG. 12, computer system 320 includes a computer 340, a monitor 342, a keyboard 346 and a mouse 348, all connected to computer 340. Further, the computer 340 includes a DVD (Digital Versatile Disc) drive 350 and a semiconductor memory port 352.

  Referring to FIG. 13, the computer 340 further includes a bus 366 connected to the DVD drive 350 and the semiconductor memory port 352, a CPU (Central Processing Unit) 356 that executes a computer program for realizing the above-described device, and a computer 340. ROM (Read Only Memory) 358 for storing a boot-up program of the computer, a RAM (Random Access Memory) 360 for providing a work area used by the CPU 356 and a storage area for a program executed by the CPU 356, and a bilingual corpus 30 (FIG. And a hard disk drive (HDD) 364 for storing other data.

  When the computer 340 is used as a translation training device, the HDD 364 further stores a program for the SMT module, and stores a bilingual corpus and a test set.

  Computer 340 further includes a network interface (I / F) 380 that is connected to bus 366 and connects computer 340 to network 382.

  Software for realizing the system of the above-described embodiment may be distributed in the form of an object code recorded on a recording medium such as a DVD 368 or a semiconductor memory 370, and the computer 340 is read by a reading device such as a DVD drive 350 or a semiconductor memory port 352. And may be stored in the HDD 364. When the CPU 356 executes the program, the program is read from the HDD 364 and stored in the RAM 360. An instruction is fetched and executed by the CPU 356 from an address designated by a program counter (not shown) in the CPU 356. The CPU 356 reads data to be processed from the registers in the CPU 356, RAM 360, or HDD 364, and stores the processing result in the registers in the CPU 356, RAM 360, or HDD 364 again.

  Since the general operation of computer system 320 is well known, its details are not described here.

  The software distribution method is not necessarily fixed to the storage medium. For example, software may be transmitted from another computer to computer 340 over network 382. A part of the software may be stored in the HDD 364, and the remaining part of the software may be taken into the HDD 364 from the network and integrated at the time of execution.

  Typically, modern computers utilize the functions provided by a computer operating system (OS) and perform these functions in a controlled manner according to the desired purpose. Therefore, a program that does not include these functions, is provided by the OS or by a third party, and only specifies a combination of execution order of general functions is also a control that achieves a desired purpose as a whole. Any structure is included in the scope of the present invention.

  In the example described above, the iteration stops at step 148 (see FIG. 3) where the evaluation result is worse than the preceding evaluation result. However, the present invention is not limited to such an embodiment. For example, the repetition may be stopped when the evaluation is not higher than the preceding evaluation result, or a moving average of a predetermined repetition evaluation result may be used instead of the evaluation result of one repetition. Good.

  Further, the ME classifier for tagging the characters of the source language sentence of the bilingual corpus may use a statistical model instead of the ME. An SVM (Support Vector Machine) or a decision tree may be used instead of the ME.

(Conclusion)
This example provides a new language-independent method for unsupervised segmentation of sentences that do not use white space to separate meaningful word units to improve the performance of current SMT systems. suggest. The proposed method does not require any linguistic information about the source language, which is important in building an SMT system for translation of relatively minor languages, where morphological analysis tools are often not available. In addition, the development costs are only for the generation of a bilingual corpus, which is much less than the development of linguistic word segmentation tools, or the costs paid to people to manually segment a data set.

  The effect of the proposed method was investigated by translating five Asian languages (Japanese, Korean, Thai and Chinese (Mandarin Chinese, Taiwanese)) into English in the field of travel conversation. Automatic evaluation of translation results showed a consistent improvement of up to 2.0 points for BLEU and up to 3.0 points for METEOR compared to a baseline system that translates character-separated input sentences. In addition, the proposed method uses only a given bilingual corpus to train the ME-based segmentation model without any external information, and is bilingual segmented with linguistically derived tools. Similar translation results were achieved with the SMT model trained on the corpus.

  By handling features specific to the source language character type during the initial word segmentation generation, and identifying additional features of the ME-based tagging model, the performance of the system could be further improved. Furthermore, the word segmentation schemes learned by the iterative bootstrap method are very different from each other. Therefore, combining different schemes with the SMT decoding process has the potential to overcome the overfitting problem of word segmentation schemes that use long translation units and improve system performance.

30 bilingual corpus 32 target language sentence 34 source language sentence 36 unigram separator 40, 50, 60, 80, 90 SMT
42, 52, 63, 82, 92 Evaluation result 44, 54, 84 Token-word matching result 46, 56, 76, 86 ME classifier 48, 58, 78, 80 Segmented source language sentence

Claims (6)

A sentence separator training device for training a preselected first language sentence separator,
Means for storing a bilingual corpus of translation pairs, each of the translation pairs including a source language sentence of the first language and a target language sentence of a second language, each of the source language sentences The target language sentence is composed of non-delimited character strings, each word of the target language sentence is separated from each other by a space character, and the sentence separator training device further defines the source language sentence in the bilingual corpus in advance. A character-based first separator for separating and segmenting each character by a delimiter;
First statistical training means for training statistical machine translation means using a bilingual training corpus including translation pairs of sentences of the first language and the second language, the statistical machine translation means for each of the translation pairs in the bilingual training corpus during the training, the source language sentence, each segment separated by the delimiters, Ru correspondence to one of the words of the target language sentence It has a function
An evaluation means for evaluating the performance of the statistical machine translation means trained by the training means;
Using the result of the association by the statistical machine translation means , for each translation pair of the bilingual corpus, a plurality of consecutive characters in the source language sentence that are associated with the same word in the target language sentence A second training means for training a second separator of the source language sentence to separate the source language sentence into segments while integrating the characters into one string ;
Separating the source language sentence in the bilingual corpus into segments using the second separator trained by the second training means, and inserting the predetermined delimiter at a segment boundary Segmentation means of
The first training means, said evaluating means, wherein the second training means and the second separator, repetitive control for controlling so as to repeatedly operate until the a no observed improvement in the evaluation by the evaluation unit Means,
In the first iteration of the iteration control means, select the bilingual corpus that includes the source language sentence segmented by the character-based first separator, and in subsequent iterations segment by the second separator Selecting the bilingual corpus having the source language sentence that has been converted, and using the selected bilingual corpus as the bilingual training corpus to cause the first training means to train the statistical machine translation means further look at containing and means,
A sentence separator that outputs, as a trained sentence separator, the second separator obtained by the evaluation by the evaluation means among the second separators obtained by the time when the repetition by the control of the repetition control means ends . Training device. The second training means includes
Annotating each character in the source language sentence of the bilingual corpus using the result of the association by the statistical machine translation means, and an annotation indicating whether each character is the end of a word Means for granting to,
Means for extracting a predetermined feature amount set of each character in the source language sentence of the bilingual corpus, wherein the predetermined feature amount set includes the annotation of the target character of the source language sentence. reflecting the context containing, by further wherein the probability models used in the second separator comprises a means for training the probabilistic model statistical analysis of the set of the feature amount extracted by said extraction means, The sentence separator training device according to claim 1, wherein the sentence separator training apparatus is used to estimate a probability that a character in a source language sentence is an end of a word. The sentence separator training device according to claim 2, wherein the probability model includes a maximum entropy model. The iterative control means is configured to evaluate the first training means, the evaluation means, the second training means, and the second separator based on the evaluation of the evaluation means in the iteration preceded by the evaluation by the evaluation means. The sentence separator training apparatus according to any one of claims 1 to 3, wherein the sentence separator training apparatus is controlled so as to repeatedly operate until it becomes worse. A computer program that, when executed by a computer, causes the computer to function as the device according to any one of claims 1 to 4. A computer readable medium recording the computer program according to claim 5.

Images