JP2009059123A - Unit and method for predicting human assessment of translation quality - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a unit for stably predicting human assessment of machine translation quality. <P>SOLUTION: For stably predicting human assessments of machine translation quality, application module 32 includes: feature extraction module 76 for calculating a feature set 78 of translated text 74; binary classifiers 54 each for classifying translated text 74 into one of a predefined binary classes in accordance with selected features; a coding matrix 56 where each of the grades is associated with a row of classifying results of binary classifiers 54; and comparing module 84 for determining a grade of translated text 74 according to a distance between a binary vector 82 of binary decisions 80 by binary classifiers 54 and rows of coding matrix 56. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to machine translation evaluation, and more particularly, to a method and apparatus for predicting machine translation (MT) quality human ratings.

  Evaluation by MT quality personnel is costly and time consuming. Various automatic evaluation measures have been proposed to evaluate MT output more cheaply and quickly. Recent Newswire MT assessment (NIST (National Institute of Standards and Technology), http://www.nist.gov/spec/test/mt) and travel data assessment (IWSLT (the International pound). //Www.slc.art.jp/IWSLT2006) is investigating how well the indicators for these evaluations correlate with human judgment. As a result, it was shown that when the output of the MT system was ranked at the document level, some indicators were highly correlated with human judgment. However, each automatic indicator focuses on a different aspect of the translation output, and the correlation with human judgment depends on the type of human rating (eg fluency or sufficiency). Furthermore, none of the automatic indicators proved satisfactory for predicting the translation quality of a single translation.

  Various strategies have been proposed for how to assess translation quality. Most of them focus on human assessment of translation quality for fluency, sufficiency and acceptability of translation. Fluency refers to how natural the evaluation segment sounds to people whose native language is English. For sufficiency, the evaluator is given a “standard translation” in addition to the original language input, and how much information from the original translation is expressed in the translation. Must be judged. Acceptability determines how easy it is to understand the translation. The judgment of fluency, sufficiency and acceptability consists of any of the grades listed in Table 1 below.

このような人による評価指標のコストが高いことから、機械翻訳のための自動評価指標の開発に多くの関心が寄せられることとなっている。テーブル２はＭＴ研究の分野で広く用いられているいくつかの指標を紹介するものである。

Due to the high cost of such evaluation indexes by humans, much attention has been paid to the development of automatic evaluation indexes for machine translation. Table 2 introduces some indicators widely used in the field of MT research.

翻訳品質の人による評定を予測するための、以前に提案された方策のほとんどは、決定木(ｄｅｃｉｓｉｏｎ ｔｒｅｅｓ：ＤＴ）、サポートベクトルマシン（ｓｕｐｐｏｒｔ ｖｅｃｔｏｒ ｍａｃｈｉｎｅｓ：ＳＶＭ）、又はパーセプトロン等の教師あり学習法を利用して、人の品質判断に近くなりうる判断モデルを学習している。このような分類器は人が評価したＭＴシステム出力から抽出した特徴量の組でトレーニングすることができる。

Most of the previously proposed strategies for predicting translation quality human ratings are supervised learning methods such as decision trees (DT), support vector machines (SVM), or perceptrons. Is used to learn a judgment model that can be close to human quality judgment. Such a classifier can be trained on a set of features extracted from human-evaluated MT system output.

  In the research described in Non-Patent Document 3, a statistical measure for estimating reliability at the word / phrase level is used to collect system-specific features of the translation process itself and train a binary classifier. Distinguish between good and bad translations by using an empirical threshold for automatic assessment scores. Non-Patent Document 3 also examines the availability of various learning strategies for multiclass classification problems for very small data sets in the domain of technical literature.

  Non-Patent Document 1 uses a DT classifier trained with a large number of edit distance features, and here, the combination of vocabulary (stem, word, part of speech) and semantics (thesaurus-based semantic class) is matched. Used to compare MT system output and reference translation to directly approximate human acceptability.

Non-Patent Document 5 is based on automatic scoring features to distinguish between “wire-made” translations and “machine-generated” translations of Newswire instead of directly predicting human judgment. Training the value SVM classifier.
Yasuhiro Akiba, Kenji Imamura, and Eichiro Sumita, 2001. Use multiple edit distances for automatic ranking of machine translation output. MT Summit VIII Proceedings, 15-20 pages. (Yasuhiro Akiba, Kenji Imamura, and Eiichiro Sumita. 2001. Using multiple edit distances to automatically rank machine translation output. In Proc. Of MT Summit VIII, pages 15-20.) Erin Allway, Robert Shapia and Yoram Singer, 2000. Multi-class to binary: A unified approach to margin classifiers. Machine Learning Research Journal, 1: 113-141. (Erin Allwein, Robert Schapire, and Yoram Singer. 2000. Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of Machine Learning Research, 1: 113-141.) Christopher B. Kirk, 2004. Sentence level machine translation confidence measure training. 4th International Conference on Language Resources and Evaluation (LREC), pages 825-828, Portugal. (Christopher B. Quirk. 2004. Training a sentence-level machine translation confidence measure. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), pages 825-828, Portugal.) Rule quest. 2004. Data mining tool c5.0. http://rulequest.com/see5-info.html. (Rulequest. 2004. Data mining tool c5.0. http://rulequest.com/see5-info.html.) Alex Cresza and Stuart M. Shiva. 2004. A learning approach to improve sentence-level MT evaluation. TMI04 Proceedings, USA. (Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proc. Of the TMI04, USA.)

  Previously proposed strategies use supervised learning to predict human ratings of translation quality. However, such multi-class classifiers are primarily trained with a number of features that are internal to the MT system and language dependent, which must be adjusted each time the MT engine or language changes. In addition, previous strategies have attempted to directly predict human judgment (multi-class tasks). Such direct classification tasks tend to require large amounts of training data and, even if trained, are unstable or less precise in predicting translation quality.

  Accordingly, one of the objects of the present invention is to provide a method and apparatus for stably predicting a human translation quality rating.

  Another object of the present invention is to provide a method and an apparatus for stably predicting a human translation rating with high accuracy.

  It is a further object of the present invention to provide a system and language independent method and apparatus for stably predicting human rating of machine translation quality.

  The present invention relates to an apparatus and a method for predicting or estimating a human translation quality rating based on a combination of binary classifiers using an encoding matrix. The multi-class categorization problem is reduced to a set of binary problems, and these are solved with a standard classification learning algorithm trained on the results of many automatic evaluation indices. The binary classifier is trained with features of multiple automatic evaluation indexes such as BLEU and METEOR. The learned judgment model is applied to the MT output for each sentence, and a binary index of translation quality at the sentence level is generated. The multi-class classification problem is then solved by combining the binary classifier results using an encoding matrix.

  In particular, the first aspect of the present invention relates to an apparatus for estimating a human translation quality rating. Human ratings are given by a pre-defined grade. The apparatus predetermines a given translation according to a means for calculating a predetermined set of features for a given translation, each according to a feature selected in the set of features A set of binary classifiers for classifying into one of the binary classes, and means for storing an encoding matrix in which each of the grades is associated with a row of classification results for the set of binary classifiers; Means for determining the grade of a given translation according to the result of the binary classification by the binary classifier and the encoding matrix.

  Preferably, the output of the set of binary classifiers defines a binary vector, each of its elements being a first value or a second value different from the first value. Each row of the encoding matrix defines a ternary vector, each of which may be a first value, a second value, or a third value different from the first and second values. The first and second values indicate that a given translation should be classified into the first and second classes, respectively, by the corresponding pair of binary classifiers. The third value indicates that a given translation is not classified by the corresponding one of the set of binary classifiers. Means for determining means for calculating a distance between the binary vector and each of the ternary vectors, means for finding the row of the encoding matrix closest to the binary vector in the distance, and the binary vector; And means for selecting the grade corresponding to the closest row as an estimated human rating for a given translation quality.

  More preferably, the means for calculating the distance includes means for calculating a Hamming distance between the binary vector and each ternary vector.

  The means for calculating the set of predetermined features is a plurality of preselected automatic multiclasses, each for automatically evaluating a human rating of machine translation quality by a set of grades. Evaluation means may be included.

  Preferably, the means for calculating a predetermined feature amount set further includes an automatic evaluation means for calculating a predetermined internal index feature value.

  A second aspect of the invention relates to a computerized method for estimating human translation grades of machine translation quality. Human ratings are given by a pre-defined grade. The method includes the steps of calculating a predetermined feature set for a given translation and according to a feature selected in the feature set, by each of a set of binary classifiers. A step of classifying the translation into one of the predefined binary classes; a step of storing in the storage unit an encoding matrix associated with a row of the classification result performed in the step of classifying each of the grades; and a step of classifying Determining a grade for a given translation according to the set of binary classification results and the encoding matrix.

  A third aspect of the invention, when executed on a computer, causes the computer to calculate a predetermined feature set for a given translation, each of which is included in the feature set. A set of binary classifiers for classifying a given translation into one of the predefined binary classes according to the feature quantity selected in, and the classification results of each class of binary classifiers Means for storing an encoding matrix associated with a row of the data, and means for determining a grade of a given translation according to the result of the binary classification by the binary classifier and the encoding matrix A computer program.

  Experimental results using a large-scale annotated evaluation corpus showed that decomposition into a binary classifier achieves higher classification accuracy than multiclass categorization problems. In addition, the proposed method achieves a higher correlation with human judgment at the sentence level than the standard rating scale.

[First Embodiment]
Summary An apparatus for predicting translation quality assessment by a person according to a first embodiment of the present invention will be described below. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description will not be repeated.

  The system of this embodiment uses supervised learning to predict a rating by a person with translation quality, but differs from the prior art system in the following two aspects.

(1) Reduction of classification perplexity The decomposition of multi-class classification tasks is reduced to a set of binary classifications. This reduces the complexity of the learning task, resulting in higher classification accuracy.

(2) Set of feature quantities The binary classifier is trained on the results of a number of automatic evaluation indices (see Table 2), and therefore considers various aspects of translation quality handled by each of the indices. This method is independent of the specific MT system or target language. This can be applied to any translation or target language without modification, as long as a reference translation is available.

  The prediction method according to this embodiment is divided into three stages. (1) A learning stage in which a binary classifier is trained by a set of feature values extracted from a database of MT system output evaluated by a person and a machine. (2) A recombination step on the development set. A decomposition stage that selects an optimal set of binary classifiers that maximizes the classification accuracy of (3), applying the binary classifier to a sentence that has never seen the binary classifier, and optimizing the result of the binary classifier An application step of predicting a human score in combination using the encoded matrices.

-Learning stage Decision models for multi-class and binary classification problems are obtained using standard learning algorithms. The method according to the proposal is not limited to a specific classification learning method. For the experiment described below, a standard implementation example of a decision tree (Non-Patent Document 4) was used.

  The set of feature amounts is composed of scores of seven types of automatic evaluation indexes listed in Table 2. All automatic evaluation indices are applied to the input data set consisting of English MT output, and the translation quality is manually evaluated by the person using the indices introduced in Table 1. In addition to the index score, an internal feature quantity of the index, for example, an n-gram accuracy score, a length ratio between the reference and the MT output, and the like are also used, resulting in 54 training feature quantities.

-Decomposition stage There are many ways to decompose a multi-class problem into several binary classification problems. The best known strategies are one-against-all and all-pairs methods. In one-value vs. other strategies, to train a class of classifiers, all training examples belonging to that class are used as positive examples, otherwise they are negative examples. In the all-pair strategy, the classifier is trained for each class pair, and any training examples that do not belong to any of the classes in question are ignored.

  Such decomposition of the multiclass problem is represented by an encoding matrix M in which the class c of the multiclass problem is associated with the row of the binary classifier b. If k is the number of classes and l is the number of binary classification problems, this encoding matrix is defined as follows:

ここでｋはクラスの数であり、ｌは２値分類問題の数である。クラスｃに属するトレーニング例が２値分類器ｂに対し肯定的な例であると考えられる場合、ｍ_ｃ，ｂ＝＋１である。同様に、もしｍ_ｃ，ｂ＝−１であれば、クラスｃのトレーニング例はｂのトレーニングに対し否定的な例として用いられる。ｍ_ｃ，ｂ＝０は、それぞれのトレーニング例が分類器ｂのトレーニングには用いられないことを示す。

Here, k is the number of classes, and l is the number of binary classification problems. If a training example belonging to class c is considered a positive example for binary classifier b, then m _{c, b} = + 1. Similarly, if m _{c, b} = −1, the class c training example is used as a negative example for b training. m _{c, b} = 0 indicates that each training example is not used for training of the classifier b.

  This embodiment utilizes a binary classifier of one value vs. other and all pairs. In addition, train boundary classifiers throughout the training set. In this case, all training examples classified into classes higher than the class in question are used as positive examples, and all other training examples are negative examples. Table 3 lists the 17 binary classification problems that are used to resolve the rating problem by a person introduced in the background section.

それぞれのタスクについて最適な符号化マトリクスを特定するために、２値分類器をまず開発セットに対する分類精度に従って順序付けする。第２のステップとして、マルチクラスの性能を繰返し評価し、繰返しのたびに、性能が最悪の２値分類器を符号化マトリクスから除外する。最後に、マルチクラスタスクについて最良の分類精度を達成する符号化マトリクスを用いてテストセットを評価する。最適化された符号化マトリクスは、標準バイアス−分散トレードオフを反映し、区別能力と２値分類器の組合せの信頼性とのバランスをとる。

In order to identify the optimal encoding matrix for each task, the binary classifier is first ordered according to the classification accuracy for the development set. As a second step, the multi-class performance is evaluated repeatedly, and the binary classifier with the worst performance is excluded from the encoding matrix for each iteration. Finally, the test set is evaluated using an encoding matrix that achieves the best classification accuracy for the multiclass task. The optimized coding matrix reflects the standard bias-dispersion tradeoff and balances the discriminating ability and the reliability of the binary classifier combination.

Application stage Given an input example, all binary classifiers are applied once for each column of the coding matrix, resulting in a vector v of l binary classification results. The multi-class label is predicted as label c, which represents that the corresponding row r of the encoding matrix M is “closest”.

  In Non-Patent Document 2, the distance between r and v is considered (a) a generalized Hamming distance that counts the number of different positions between corresponding vectors, and (b) the size of the binary classifier score. And by decoding with loss (decoding). Both are effective, and this embodiment uses a Hamming distance strategy.

Structure for Predicting Acceptability FIG. 1 shows the overall structure of a translation quality prediction system 30 of this embodiment using a corpus 40 that implements the above-described method for predicting translation quality. Referring to FIG. 1, system 30 predicts human ratings of translated text 74 translated from source text 70 by machine translation 72, and application unit 32 for outputting its rating 86, corpus 40 And a classifier preparation unit 34 for generating a set of an encoding matrix 56 and a binary classifier 54 used in the application unit 32. The corpus 40 includes a number of learning translation sets. Each learning translation set includes a set of source text, its machine translation text, and feature parameter. Details of the corpus 40, the binary classifier 54, and the encoding matrix 56 will be described later.

  The application unit 32 of this embodiment predicts the human acceptability score of the translated text 74 with the 5 grades shown in Table 1 (5, 4, 3, 2, 1). As will become apparent from the following description, a system that predicts fluency or sufficiency can be implemented with a structure similar to that shown in FIG.

  The classifier preparation unit 34 includes a learning module 44 for training the binary classifier 46 and a binary classifier 54 that optimizes the binary classifier 46 so that the prediction accuracy of the development set 52 is maximized. And an optimization module 50 for optimizing.

  As a result of the decomposition stage 42, an encoding matrix 48 is prepared. As will be described later, each row of the encoding matrix 48 forms a ternary vector. In optimizing the binary classifier 46, the optimization module 50 deletes some of the binary classifiers 46 and optimizes the encoding matrix 48 to the optimized encoding matrix 56 accordingly.

  The application unit 32 is connected to receive a predetermined part of the feature quantity set 78 and a feature quantity extraction module 76 for extracting a predetermined feature quantity set 78 from the translated text 74, and outputs a binary decision value 80. A binary classifier 54. The binary decision value 80 forms a binary vector 82.

  The application unit 32 further includes a comparison module 84 that compares the binary vector 82 with each row of the encoding matrix 56 to determine which row is closest to the binary vector 82 at the Hamming distance. The class corresponding to the closest line is selected as the multi-class evaluation 86 of the translated text 74.

  FIG. 2 shows details of the corpus 40. Referring to FIG. 2, the corpus 40 includes a number of learning translation sets 100, each of which includes a source text 110, a machine translation 112 of the source text 110, and a feature set 114.

  The feature amount set 114 includes a human score (one of scores 1 to 5) 120, a set of automatic evaluation scores 122, and an internal index feature amount 124.

  The automatic evaluation score 122 includes a plurality of automatic evaluation scores 130, 132,. The automatic evaluation score used in this embodiment includes seven types of indices shown in Table 2.

  The internal index feature value 124 includes an n-gram accuracy score 150, a length ratio between the reference and the MT output, and the like. Any feature quantity known to indicate the quality of translation can be used as an element of the internal index feature quantity 124.

  FIG. 3 shows how the learning translation set 100 is prepared. Referring to FIG. 3, source text 110 is collected from some source. Machine translation 112 is obtained by supplying source text 110 to some type of translation machine.

  The human score 120 is prepared by a human rating 170 of the machine translation 112. If human ratings are to be presented by multiple people, the average of those scores is used as the human score 120.

  The automatic evaluation score 122 and the internal index feature value 124 are prepared by the respective automatic evaluation systems 172, 174,.

  FIG. 4 shows the decomposition mechanism of the binary classifier used in this embodiment. Referring to FIG. 4, the binary classifier 46 is classified into three types. One-value pairs, all other pairs, and boundaries as described above.

  One-value vs. other classifiers include 5 vs. Other, 4 vs. Other, 3 vs. Other, 2 vs. Other and 1 vs. Other.

  For one-value vs. other strategies, for each class, in the training of a class of classifiers, all training examples belonging to that class are used as positive examples in training and all other examples are negative examples It is said. For example, a 5-to-other classifier is trained with a training example in which all training examples belonging to class “5” are used as positive examples and all others are used as negative examples. As a result, when the translated text 74 is indicated to belong to the class “5” by the feature value set 78, 5 vs. others output +1, and otherwise outputs -1.

  In the all-pair strategy, the classifier is trained for each pair of classes, but all training examples that do not belong to any of the problematic classes are ignored. The all-pair classifier of this embodiment includes 5_4, 5_3, 5_2, 5_1, 4_3, 4_2, 4_1, 3_2, 3_1, and 2_1 classifiers.

  For example, in 5_4 classifier training, all training examples belonging to class “5” are used as positive examples, examples belonging to class “4” are used as negative examples, and other training examples are ignored. .

  In the boundary approach, the classifier is trained on the entire training set. In this case, all training examples with a good class comment above the class in question are used as positive examples and all other training examples are negative examples. The boundary classifier of this embodiment includes 54_321 and 543_21 classifiers.

  For example, in the 54_321 classifier training, all examples of classes “5” and “4” are used as positive examples, and all other examples of class “3”, “2” or “1” are used. Used as a negative example.

  FIG. 5A shows how an example 202 for training 5 vs. other classifiers is generated. Referring to FIG. 5A, all data in corpus 40 is used for training data generation 200. The generated examples 202 each include a binary label 204 (+1 if the human score is “5”, −1 otherwise) and a feature amount 206. As shown in FIG. 3, the feature amount 206 includes an automatic evaluation score 122 and an internal index feature amount 124.

  FIG. 5B shows how an example 212 for training a three-to-other classifier is generated. With reference to FIG. 5B, all data in the corpus 40 is used for the training data generation 210. Each of the generated examples 212 includes a binary label 214 (+1 if the score by a person is “3”, −1 otherwise) and a feature quantity 216. The feature quantity 216 includes an automatic evaluation score 122 and an internal index feature quantity 124.

  FIG. 6A shows how an example 224 for training a 5_4 classifier is generated. With reference to FIG. 6A, an example where the human score is “5” or “4” is extracted by the data extraction process 220, and then the extracted data is used for training data generation 222. Each of the generated examples 224 includes a binary label 226 (+1 if the human score is “5”, −1 if the human score is “4”), and a feature quantity 228. The feature quantity 228 includes an automatic evaluation score 122 and an internal index feature quantity 124.

  FIG. 6B shows how an example 234 for training a 5_2 classifier is generated. With reference to FIG. 6B, an example in which a human score is “5” or “2” is extracted by the data extraction process 230. The extracted data is used for training data generation 232. Each of the generated examples 232 includes a binary label 236 (+1 if the human score is “5”, −1 if the human score is “2”), and a feature amount 238. The feature quantity 238 includes an automatic evaluation score 122 and an internal index feature quantity 124.

  FIG. 7A shows how an example 244 for training a 3_2 classifier is generated. With reference to FIG. 7A, an example where the human score is “3” or “2” is extracted by the data extraction process 240, and then the extracted data is used for training data generation 242. Each of the generated examples 244 includes a binary label 246 (+1 if the human score is “3”, −1 if the human score is “2”), and a feature amount 248. The feature quantity 248 includes an automatic evaluation score 122 and an internal index feature quantity 124.

  FIG. 7B shows how an example 254 for training the 3_1 classifier is generated. With reference to FIG. 7B, an example in which a human score is “3” or “1” is extracted by the data extraction process 250. The extracted data is used for training data generation 252. Each of the generated examples 254 includes a binary label 256 (+1 if the human score is “3”, −1 if the human score is “1”), and a feature amount 258. The feature amount 258 includes an automatic evaluation score 122 and an internal index feature amount 124.

  FIG. 8A shows how an example 262 for training the 54_321 boundary classifier is generated. With reference to FIG. 8A, all examples in the corpus 40 are used for training data generation 260. Each of the generated examples 262 includes a binary label 264 (+1 if the human score is “5” or “4”, −1 otherwise), and a feature quantity 266. The feature quantity 266 includes an automatic evaluation score 122 and an internal index feature quantity 124.

  FIG. 8B shows how an example 272 for training the 543_21 boundary classifier is generated. With reference to FIG. 8B, all examples in the corpus 40 are used for training data generation 270. Each of the generated examples 272 includes a binary label 274 (+1 if the human score is “5”, “4” or “3”, −1 otherwise), and a feature amount 276. The feature quantity 276 includes an automatic evaluation score 122 and an internal index feature quantity 124.

  FIG. 9 shows an encoding matrix 48 used in this embodiment. Referring to FIG. 9, classes (classes “1” to “5”) are arranged in the leftmost column, and the binary classifier is arranged in the top row. The second to sixth columns ("5" to "1") at the left end indicate 5, 4, 3, 2, and 1 to other classifiers, respectively.

  As can be seen from FIG. 9, each row of the encoding matrix 48 forms a ternary vector. The elements of these vectors are +1, -1 or 0. If a training example belonging to a particular class is considered a positive example of a binary classifier, the element corresponding to the combination of that class and that classifier is indicated by “+1”. Similarly, if the element is “−1”, the training example for that class is used as a negative example for training the classifier. The element “0” indicates that each training example is not used for training the corresponding classifier.

  FIG. 10 is a flow of the optimization module 50 realized in the form of a computer program. As can be seen from FIG. 1, as a result of the decomposition stage 42, a binary classifier 46 and an encoding matrix 48 are prepared. The following process is performed on the development set 52 shown in FIG.

  Referring to FIG. 10, the program includes a step 290 for calculating the accuracy of each of the binary classifiers 46, and a step 292 following step 290 for ordering the binary classifiers 46 according to their classification accuracy. .

  The program further includes a step 294 of evaluating multi-class accuracy when the remaining binary classifier is used as the binary classifier 54 in the application unit 32, followed by the best performance among the binary classifiers. The step 296 of excluding the poor classifier binary classifier and whether the accuracy of the “after” encoding matrix excluding the worst performing binary classifier is lower than that of the excluding “previous” encoding matrix , And branching according to the result. If the result of step 298 is NO, control returns to step 294, otherwise control proceeds to the next step.

  The program further includes a step 300 of selecting the best performing set of binary classifiers among the sets appearing between iteration steps 294 to 298, which is performed if the determination in step 298 is YES; Reconstructing an encoding matrix according to the set of binary classifiers selected in step 300.

  FIG. 11 illustrates the result of the optimization process described above. Referring to FIG. 11, binary classifiers 46 are ordered at the bottom of the horizontal axis, and the worst-performing group (all) is located on the leftmost side. The name of the classifier on the horizontal axis indicates what was excluded during the iteration. The vertical axis indicates the multi-class accuracy evaluated in the repeated step 294.

  For example, using all binary classifiers 46, the accuracy is about 56%. If 3_2, 2_1, 4_1, 4_3 and 4_2, which showed the worst performance in step 292, are excluded in this order, the accuracy will be only about 56%. However, when 5_4 is excluded, the accuracy increases to 62%, and even if the classifier is further excluded, the result is not improved. Therefore, in this example, 1 to other, 543_21, 3_1, 54_321, 5 to other, 4 to other, 5_2, 5_1, 5_3, 2 to other and 3 to other classifiers are the best classifier sets. You can conclude. In FIG. 11, 3 vs. other classifiers are not excluded from the set of classifiers and therefore do not appear in the figure.

  Details of the optimized binary classifier 54 are shown in FIG. Referring to FIG. 12, the binary classifier 54 includes 5, 4, 3, 2, 1 value pairs, etc., and respective classifiers 320, 322, 324, 326, and 328, and 5_3, 5_2, 5_1, and 3_1 pairs. Classifiers 342, 344, 346, and 382, and 54_321 and 543_21 boundary classifiers 400 and 402 are included. The other classifiers 340, 360, 362, 364, 380 and 390 are not used in this example.

  Note that the accuracy of the classifier depends on the corpus 40 and the decomposition stage 42. Similarly, the overall multi-class accuracy also depends on the corpus 40 and the decomposition stage 42. Therefore, the configuration of the optimized binary classifier 54 will differ from that shown in FIG. 12 for different corpus or decomposition stages.

  The configuration of the optimized encoding matrix 56 is shown in FIG. As can be seen from FIG. 13, in the encoding matrix 56, the 5_4, 4_3, 4_2, 4_1, 3_2, and 2_1 classifiers are excluded, and the 5, 4, 3, 2, and 1 to other classifiers and 5_3, 5_2. The 5_1, 3_1, 54_321, and 543_21 classifiers remain. Since the number of columns is small, the calculation cost for the binary vector 82 is reduced.

  FIG. 14 shows a scheme of feature quantity extraction performed by the feature quantity extraction module 76 shown in FIG. Referring to FIG. 14, when the source text and the translated text 74 are given, the automatic evaluator 410 evaluates the quality of the translated text 74, and the feature quantity extracting unit 412 for the internal index feature quantity respectively. The feature amount of is calculated. A set 78 of feature amounts is an internal index feature amount in which outputs of each of the features 430, 432,... Field 440, 442,. The feature quantity set 78 is given to the binary classifier 54 shown in FIG.

Operation The system 30 of this embodiment operates as follows. Initially, the binary classifier 46 is trained by the learning module 44. In decomposing the multi-class classification into a binary classifier 46, an encoding matrix 48 is also provided.

  The optimization module 50 optimizes the binary classifier 46 as follows. First, the accuracy of the binary classifier 46 in the development set 52 is calculated and the binary classifier 46 is ordered according to their accuracy.

  Starting with a complete coding matrix (ALL), a multiclass evaluation is performed and the worst performing classification is excluded in the next generation. Repeat this process. The dashed box 310 in FIG. 11 shows the subset of binary classifiers selected for the coding matrix utilized in the example development set evaluation. This subset is shown in FIG. 1 as a binary classifier 54 and is incorporated in the application unit 32. At the same time, the optimization module 50 optimizes the encoding matrix 48 according to the optimized binary classifier 54. The optimized encoding matrix 56 is used by the comparison module 84 of the application unit 32.

  At the application stage, the source text 70 and the translated text 74 which is a translation of the source text 70 by the machine translation 72 are given to the feature quantity extraction module 76. The feature extraction module 76 calculates or extracts a feature set from the source text 70 and the translated text 74 and supplies the resulting feature set 78 to the binary classifier 54.

  A binary classifier 54, which is an optimized version of the binary classifier 46, outputs a binary decision 80, which forms a binary vector 82.

  The comparison module 84 compares each of the ternary vectors comprising the rows of the encoding matrix 56 with the binary vector and determines which ternary vector (row) is closest to the binary vector 82 at the Hamming distance. The comparison module 84 outputs the class identifier corresponding to the closest row as an evaluation 86. The resulting output indicates the estimated or predicted grade of the translated text 74.

Fluency and Sufficiency The above-described embodiments of the invention are applicable to other human ratings such as fluency or sufficiency. FIG. 15 shows an example of system performance (multi-class accuracy) during the optimization process in experiments with fluency (FIG. 15A) and sufficiency (FIG. 15B).

  FIG. 15 is a summary of repeated evaluations of binary classification combinations using the exemplary development set 52 for fluency (FIG. 15A) and sufficiency (FIG. 15B). Starting with a complete coding matrix (all), the next iteration will eliminate the worst performing binary classification. Dashed squares 450 and 452 represent the binary classifier subsets selected for the encoding matrix utilized in comparing the fluency and sufficiency test sets, respectively.

Evaluation The evaluation of this embodiment was performed using the basic travel expression corpus (Basic Travel Expression Corpus (BTEC)) integrated as the corpus 40 by the applicant. The BTEC includes travel-related sentences similar to those commonly found in foreign travel idioms. In total, 3,524 Japanese input sentences were translated by various types of MT systems, and 82,406 English translations were generated. 54,302 translations were annotated with human score for acceptability and 36,302 translations were annotated with human score for sufficiency / fluency. The distribution of human scores for a given translation is summarized in FIG. If a single translation output was judged by many people, the median score for each person was used in this experiment.

  The annotated corpus was divided into three data sets. (1) Training set consisting of 25,988 translations for sufficiency / fluency and 49,516 MT outputs for acceptability, (2) 2,024 sentences (506 for all three indicators) (3) IWSLT evaluation campaign (CSTAR03 data set, 506 input sentence. “CSTAR” is Consortium for Speech Translation Research: Speech Translation Advanced Research Consortium) Is a test set taken from. For fluency and sufficiency, 7,590 test sentences and 15 MT outputs for each were available. For acceptability, 3,036 sentences and 6 MT outputs for each were used in the evaluation.

-Optimization of the encoding matrix The encoding matrix is first created in the decomposition stage and then optimized by the optimization module 50 reflecting the optimization of the binary classifier.

Classification accuracy The baseline of a multi-class classification task is defined as the most frequently occurring class in the training data set. Table 4 summarizes the baseline performance for all three subjective metrics.

マルチクラスタスクの分類精度、すなわちトレーニングセットから直接学習したマルチクラス分類器と、２値分類器の性能とを図１７にまとめる。

FIG. 17 summarizes the classification accuracy of the multiclass task, that is, the performance of the multiclass classifier directly learned from the training set and the binary classifier.

  The classification accuracy of this embodiment was 55.2% for fluency, 62.6% for sufficiency, and 62.3% for acceptability. Thus, this embodiment provides better performance than baseline and multi-class classification classes for all subjective metrics and 22.7% / fluency compared to baseline / multi-class performance. Gains of 6.0%, 31.5% / 6.6% for sufficiency and 19.3% / 1.2% for acceptability were achieved.

  Furthermore, the performance of the binary classifier varies greatly depending on the classification task and the evaluation index. An accuracy of 80% to 90% is achieved for the all-to-one classifier, but 75% to 81% for the boundary classifier and 55% to 91% for the all-pair classifier.

-Correlation with rating by person In order to examine the correlation of the index according to this embodiment to the judgment of the person at the sentence level, the Spearman rank correlation coefficient was calculated for the obtained results. In addition, the test sentences were ranked using the automatic evaluation index and the multi-class classifier listed in Table 2, and their Spareman rank correlation with respect to human ratings was calculated. The correlation coefficients are summarized in FIG.

  The results show that this embodiment performed better than all other indicators, and the correlation coefficients for fluency / sufficiency / acceptability were 0.632 / 0.759 / 0. 769.

Realization by Computer The above-described embodiment can be realized by a computer system and a computer program executed on the system. FIG. 19 shows an appearance of a computer system 650 used in this embodiment, and FIG. 20 is a block diagram of the computer system 650. The computer system 650 shown here is merely exemplary and various other configurations are available.

  Referring to FIG. 19, a computer system 650 includes a computer 660, a monitor 662, a keyboard 666, a mouse 668, a speaker 692, and a microphone 690, all connected to the computer 660. Further, the computer 660 includes a DVD (Digital Versatile Disc) drive 670 and a semiconductor memory port 672.

  Referring to FIG. 20, a computer 660 further includes a bus 686 connected to DVD 670 and semiconductor memory port 672, a CPU (Central Processing Unit) 676 for executing a computer program for realizing the above-described device, and computer 660. ROM (Read-Only Memory) 678 for storing the boot-up program of RAM, RAM (Random Access Memory) 680 for providing a work area used by CPU 676 and a storage area for programs executed by CPU 676, and corpus 40 (FIG. 1), source text 70, translated text 74, machine translation program for translating source text 70, and other machines used in feature quantity extraction module 76 A hard disk drive (HDD) 674 for storing a translation program, all data required for machine translation, a binary classifier 54 and an encoding matrix 56. All these elements are connected to the CPU 676 via the bus 686.

  When computer 660 is used as classifier preparation unit 34, HDD 674 further includes learning module 44 and optimization module 50, binary classifiers 46 and 54, encoding matrices 48 and 56, binary classifier and code. And a development set 52 used to optimize the optimization matrix.

  Computer 660 further includes a network interface (I / F) 696 connected to bus 686 to provide a connection of computer 660 to network 652.

  Software that implements the system of the above-described embodiment is distributed in the form of an object code stored in a storage medium such as the DVD 682 or the semiconductor memory 684, and the computer 660 via a reading device such as the DVD drive 670 or the semiconductor memory port 672. And stored in the HDD 674. When CPU 676 executes a program, the program is read from HDD 674 and stored in RAM 680. An instruction is fetched to the CPU 676 from an address designated by a program counter (not shown) of the CPU 676, and the instruction is executed. The CPU 676 reads data to be processed from the register in the CPU 676, the RAM 680, or the HDD 674, and stores the processing result in the register in the CPU 676, the RAM 680, or the HDD 674.

  The general operation of computer system 650 is well known and will not be described in detail here.

  Regarding the software distribution method, the software does not necessarily have to be fixed on a storage medium. For example, the software may be transmitted to computer 660 from another computer connected to network 652. A part of the software may be stored in the HDD 674, and the remaining part of the software may be taken into the HDD 674 via the network and integrated at the time of execution.

  Typically, modern computers utilize common functions provided by a computer operating system (OS) and perform these functions in a controlled manner according to the desired purpose. Therefore, even if the program does not include a general function that can be provided by the OS or a third party and specifies only a combination of execution order of the general function, the program achieves a desired purpose as a whole. As long as it has a structure, it is clear that the program is included in the scope of the present invention.

Possible Modifications The embodiment described above relates to the translation quality from Japanese to English, but the present invention is not limited to this. As long as a corpus that can be used as the corpus 40 is available, the present invention is applicable to any combination of languages.

  In the embodiment described above, the optimization module 50 optimizes the binary classifier 46 to the binary classifier 54 by first using all classifiers and removing the worst performing classifiers. To do. The present invention is not limited to such an optimization scheme. Any optimization scheme may be used as long as the classifier 54 obtained as a result of optimization exhibits better performance than other combinations of classifiers. For example, all possible combinations of binary classifiers may be examined and the combination that yields the best performance may be selected as the optimized set of binary classifiers. Instead, the performance of the binary classifier 46 in the development set 52 is calculated first, and only the binary classifier with an accuracy higher than a predetermined threshold is used as the optimized binary classifier 54. Also good.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the whole structure of the system 30 of embodiment of this invention. 3 is a diagram showing details of the corpus 40. FIG. It is a figure which shows how the translation set for learning 100 is prepared. It is a figure which shows the decomposition | disassembly scheme of the binary classifier used by this embodiment. FIG. 7 shows how to generate an example for training one value versus another classifier. FIG. 6 shows how to generate an example for training a 5_4 and 5_2 classifier. FIG. 6 shows how to generate an example for training 3_2 and 3_1 classifiers. FIG. 6 shows how to generate an example for training a 5_4 and 5_2 classifier. It is a figure which shows the encoding matrix 48 used in 1st Embodiment. FIG. 5 is a flow diagram of an optimization module implemented in the form of a computer program. It is a figure which shows the example of a result of an optimization process. It is a figure which shows the detail of the optimized binary classifier 54. FIG. It is a figure which shows the structure of the encoding matrix 56 optimized. It is a figure which shows the scheme of the feature-value extraction performed with the feature-value extraction module 76. FIG. It is a figure which shows the example of the system performance (multiclass precision) in the optimization process of the experiment about fluency and sufficiency. It is the figure which put together the distribution of the person's score with respect to the given translation in experiment. It is the figure which put together the classification accuracy of a multiclass task and binary classifier performance. It is the figure which put together the correlation coefficient of a different classifier. 2 is a front view of a computer system 650. FIG. FIG. 7 is a block diagram illustrating a computer system 650.

Explanation of symbols

30 translation quality prediction system 32 application unit 34 classifier preparation unit 40 corpus 44 learning module 46, 54 binary classifier 48, 50 encoding matrix 50 optimization module 52 development set 70 source text 74 translated text 76 feature extraction module 78 Feature value set 80 Binary decision value 82 Binary vector 84 Comparison module

Claims (7)

A device for estimating a machine translation quality rating by a person, the rating being given by a pre-defined grade;
Means for calculating a predetermined set of features for a given translation;
A set of binary classifiers, each for classifying the given translation into one of the predefined binary classes according to a feature selected in the set of features;
Means for storing an encoding matrix, wherein each of the classes is associated with a row of classification results of the set of binary classifiers;
Means for determining a grade of the given translation according to the result of binary classification by the binary classifier and the encoding matrix. The output of the set of binary classifiers defines a binary vector, each of its elements being a first value or a second value different from the first value;
Each row of the encoding matrix defines a ternary vector, each of which is the first value, the second value, or a third value different from the first and second values;
The first and second values indicate that the given translation should be classified into first and second classes, respectively, by the corresponding one of the set of binary classifiers;
The third value indicates that the given translation is not classified by the corresponding one of the set of binary classifiers;
The means for determining is
Means for calculating a distance between the binary vector and each of the rows;
Means for finding a row of the encoding matrix closest to a binary vector at the distance;
Means for selecting a grade corresponding to a row closest to a binary vector as an estimated human rating for the quality of the given translation. The apparatus according to claim 1 or 2, wherein the means for calculating the distance includes means for calculating a Hamming distance between the binary vector and each row. The means for calculating the predetermined set of features is a plurality of preselected automatic, each for automatically evaluating a human rating of machine translation quality by the set of grades. The apparatus according to claim 1, comprising multi-class evaluation means. The apparatus according to claim 4, wherein the means for calculating the predetermined feature amount set further includes an automatic evaluation means for calculating a predetermined internal index feature value. A computerized method for estimating a human rating of machine translation quality, wherein the human rating is given by a predefined grade, the method comprising:
Calculating a predetermined set of features for a given translation;
Classifying the given translation into one of a predefined binary class by each of a set of binary classifiers according to a feature selected in the set of features;
Storing in the storage unit an encoding matrix associated with each classification result row performed in the classifying step of each of the classes;
Determining the grade of the given translation according to the set of binary classification results of the classifying step and the encoding matrix. When executed on a computer, the computer is
Means for calculating a predetermined set of features for a given translation;
A set of binary classifiers, each for classifying the given translation into one of the predefined binary classes according to a feature selected in the set of features;
Means for storing an encoding matrix, wherein each of the classes is associated with a row of classification results of the set of binary classifiers;
A computer program that functions as means for determining a grade of the given translation according to a result of binary classification by the binary classifier and the encoding matrix.

Images