CN116822534A - Interpretation method, interpreter model and computer-readable storage medium for machine translation evaluation indicators based on fine-grained features - Google Patents

Abstract

The invention belongs to the technical field of machine translation and provides an interpretation method, an interpreter model and a computer-readable storage medium for machine translation evaluation indicators based on fine-grained features. The present invention solves the problem of insufficient interpretability of the machine translation quality evaluation model through the interpretation method/interpreter model of the machine translation evaluation index with fine-grained features, and provides fine-grained quality evaluation scores suitable for the machine translation scenario. Interpretability features, these features are expected to be applied to the manual PE stage of machine translation and improve the efficiency of manual PE.

Description

Interpretation methods, interpreter models and machine translation evaluation indicators based on fine-grained features computer readable storage media

Technical field

The invention belongs to the technical field of machine translation. Specifically, it relates to an interpretation method of machine translation evaluation indicators based on fine-grained features, an interpreter model and a computer-readable storage medium.

Background technique

In traditional manual evaluation methods, evaluators usually evaluate the quality of machine translations from three aspects: translation adequacy, translation fluency, and translation readability. In order to obtain faster evaluation, some researchers have proposed BLEU, etc. Rule-based evaluation metrics. The BLEU index is evaluated based on the co-occurrence rate of words or N-grams, and contains the adequacy index in manual evaluation. In recent years, in order to better evaluate the quality of machine translation, another group of researchers have proposed evaluation indicators based on pre-training models such as BertScore, MoverScore and COMET. These indicators can evaluate the quality of machine translation and take into account the adequacy of the translation. and fluency, which is a more advanced evaluation index.

However, at this stage, more advanced machine translation evaluation indicators such as BertScore have not been used on a large scale to evaluate machine translation. The reason is that these evaluation indicators based on pre-training models lack interpretability. The evaluation index of the model is a black box structure, making it difficult to form a reasonable explanation.

In order to solve this problem, researchers have tried to use interpreter models such as LIME and SHAP in machine learning to explain the results of the machine translation evaluation model. However, these methods usually only provide explanations from the word position level, and do not provide other explanation types that are more suitable for machine translation scenarios, such as error types and other information; and the interpreter model based on a simple linear model cannot express machine translation. Fine-grained features in translation quality assessment scenarios.

Contents of the invention

The purpose of the present invention is to provide a method for interpreting machine translation evaluation indicators based on fine-grained features to solve the technical problems existing in the existing technology.

In order to achieve the above objects, the technical solutions adopted by the present invention are as follows:

An interpretation method for machine translation evaluation indicators based on fine-grained features, including the following steps:

Step S1: Combine the machine-translated original text sequence S = (s ₁ , s ₂ ,..., s _n ), the machine-translated text sequence H = (h ₁ , h ₂ ,..., h _m ), and the quality assessment score Q, use tags to splice them together;

Step S2: Input the data obtained in step S1 into the pre-training model, and calculate the encoding sequence:

T=Encoder(S;H;Q)

Among them, Encoder is the pre-trained model, and T is the context label calculated by the pre-trained model;

Step S3: Segment the encoding sequence according to the input position to obtain the context representation T _text of the original text and the translation, and the context representation T _score of the quality assessment score;

Step S4: Use the attention mechanism to calculate the context representation T _text of the original text and the translation, and the attention weight matrix of the context representation T _score of the quality assessment score:

attention=softrmax(T _text ⊙T _score );

Step S5: Perform mean pooling on the attention weight matrix to obtain the one-dimensional relative importance score State:

State=MeanPooling(attention).

Step S6: Use the labels of the sequence annotation to mark the starting and ending positions of each position related to the quality score, and mark the error type for each position to obtain fine-grained interpretable feature labels;

Step S7: Based on the relative importance score State and the fine-grained interpretability feature label, use CRF to calculate the sequence label Y = (y ₁ , y ₂ , ..., y _n+m ):

Y＝CRF(State)

In one embodiment, SEP tag splicing is used in step S1.

In one embodiment, the attention mechanism in step S4 adopts one of dot product attention mechanism, connection attention mechanism and bilinear attention mechanism.

In one embodiment, BIO tags are used in step S6.

In one embodiment, the fine-grained explainability feature labels in step S6 include:

B-miss misses the starting position of translation, I-miss misses the middle position of translation;

B-misT misinterprets the starting position, I-misT misinterprets the middle position;

B-over translation starting position, I-over translation middle position.

In one embodiment, the machine-translated original text sequence S=(s ₁ , s ₂ ,..., _sn ) in step S1 and the machine-translated text sequence H=(h ₁ , h ₂ ,... , h _m ), and the quality assessment score Q are obtained through inference of the machine translation quality assessment model.

In one embodiment, the method for discovering the error type is as follows: first, infer the machine original text sequence S=(s ₁ , s ₂ ,..., s _n ) through the attention mechanism, and the machine translation sequence H=(h ₁ , h ₂ ,..., h _m ), and the error fragments of the quality evaluation score Q, and are expressed as signals with high and low weights in the attention matrix; then, the signals with high and low weights are mapped to 0-n Transfer status signal, where n represents the number of error types.

In order to achieve the above objectives, the present invention also provides an interpreter model of machine translation evaluation indicators based on fine-grained features, including:

Splicing module: combine the machine-translated original text sequence S = (s ₁ , s ₂ ,..., s _n ), the machine-translated text sequence H = (h ₁ , h ₂ ,..., h _m ), and the quality assessment score Q, use tags to splice them together;

Coding sequence module: Input the obtained data into the pre-training model and calculate the coding sequence:

T=Encoder(S;H;Q)

Among them, Encoder is the pre-trained model, and T is the context label calculated by the pre-trained model;

Context representation module: segment the coding sequence according to the input position to obtain the context representation T _text of the original text and the translation, and the context representation T _score of the quality assessment score;

Attention weight module: Use the attention mechanism to calculate the attention weight matrix of the context representation T _text of the original text and the translation, and the context representation T _score of the quality assessment score:

attention=softmax(T _text ⊙T _score );

Relative importance score module: perform mean pooling on the attention weight matrix to obtain a one-dimensional relative importance score State:

State=MeanPooling(attention);

Fine-grained interpretable feature label module: Use sequence annotation labels to mark the starting and ending positions of each position related to the quality score, and at the same time mark the error type for each position to obtain fine-grained interpretable feature labels;

Sequence label module: Based on the relative importance score State and the fine-grained interpretability feature label, use CRF to calculate the sequence label Y=(y ₁ , y ₂ ,..., y _n+m ):

Y=CRF(State).

In order to achieve the above object, the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by the processor to realize the interpretation of the machine translation evaluation index based on fine-grained features. method.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention solves the problem of insufficient interpretability of the machine translation quality evaluation model through the interpretation method/interpreter model of the machine translation evaluation index with fine-grained features, and provides fine-grained quality evaluation scores suitable for the machine translation scenario. Interpretability features, these features are expected to be applied to the manual PE stage of machine translation and improve the efficiency of manual PE.

Description of the drawings

Figure 1 is a schematic flow chart of Embodiment 1 of the present invention.

Figure 2 is a schematic diagram of the model structure of the interpreter model in Embodiment 2 of the present invention.

Figure 3 is a functional block diagram of the interpreter model in Embodiment 2 of the present invention.

Detailed ways

In order to enable those skilled in the art to have a clearer understanding of the present invention, the present invention will be further described in detail below with reference to examples. It should be known that the specific embodiments described below are only used to explain the present invention and facilitate understanding. The technical solutions provided by the present invention are not limited to the technical solutions provided by the following examples. The technical solutions provided by the embodiments Nor should it limit the scope of the present invention.

It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show the components related to the present invention and do not follow the actual implementation of the component numbers, shapes and components. Dimension drawing, in actual implementation, the type, quantity and proportion of each component can be arbitrarily changed, and the component layout type may also be more complex.

Example 1

As shown in Figure 1, this embodiment provides a method for interpreting machine translation evaluation indicators based on fine-grained features. This method implements the interpretation of machine translation evaluation indicators based on fine-grained features. It specifically includes the following implementation steps:

Step S1: Combine the machine-translated original text sequence S = (s ₁ , s ₂ ,..., s _n ), the machine-translated text sequence H = (h ₁ , h ₂ ,..., h _m ), and the quality assessment score Q, use SEP tags to splice them together, where the machine-translated original text sequence S = (s ₁ , s ₂ ,..., s _n ), and the machine-translated text sequence H = (h ₁ , h ₂ ,..., h _m ), and the quality assessment score Q, are both obtained through inference of the machine translation quality assessment model. Example: The original data is the original sentence: I like NBA, the translated sentence: I like basketball, HTER score: 0.32, then preprocessing goes to step S1 (including word segmentation and data splicing operations): [CLS] I like NBA, [SEP] I like Basketball, [SEP] 0.32.

Step S2: Input the data obtained in step S1 into the pre-training model, and calculate the encoding sequence:

T=Encoder(S;H;Q)

Among them, Encoder is a pre-trained model, such as: The pre-trained model is used to infer the error fragments of the original text sequence, the machine-translated text sequence and the quality assessment score, and the error types are discovered through the error fragments.

Step S3: Segment the encoding sequence according to the input position to obtain the context representation T _text of the original text and the translation, and the context representation T _score of the quality assessment score;

Step S4: Use one of the dot product attention mechanism, the connection attention mechanism and the bilinear attention mechanism to calculate the context representation T _text of the original text and the translation, and the attention weight matrix of the context representation T _score of the quality assessment score. :

attention=softmax(T _text ⊙T _score );

Combining the above examples, the hidden layer representation is generated through the pre-training model, and the attention weight is calculated through the attention layer. The model can already pay attention to the corresponding anomalies between "NBA" and "basketball";

Step S5: Perform mean pooling on the attention weight matrix to obtain the one-dimensional relative importance score State:

State=MeanPooling(attention).

Step S6: Use the BIO tags of sequence annotation to mark the starting and ending positions of each position related to the quality score, and mark the error type for each position to obtain fine-grained interpretable feature tags. The tags include three pairs in total, as follows (1) The starting position of B-miss missing translation, the middle position of I-miss missing translation; (2) The starting position of B-misT mistranslation, the middle position of I-misT mistranslation; (3) The starting position of B-over over-translation, I-over over-translation is in the middle position; among them, the error type discovery method is as follows: In steps S4 and S5, the attention mechanism is used to infer the error fragments related to the machine-translated original text sequence, the machine-translated text sequence and the quality assessment score, and are expressed as For the signals with high and low weights in the attention matrix, steps S6 and S7 use the subsequent neural network model to map the above-mentioned signals with high and low weights into 0-n transition state signals, where n represents the number of error types. This is achieved by Discover error types from error fragments.

Step S7: Based on the relative importance score State and fine-grained interpretability feature labels, use CRF (Conditional Random Field, conditional random field) to calculate the sequence label Y = (y ₁ , y ₂ ,..., y _n+m ):

Y=CRF(State);

Combined with the above example, through the processing of the above steps, the corresponding error location label is obtained: 0 0 0 B-misT 00 0 0 B-misT 0 0.

In the prior art, the interpretability evaluation results do not calculate (discover) the error types of the error fragments. The technical solution provided by the present application not only marks the error fragments, but also continues to calculate the error types corresponding to the error fragments.

Example 2

As shown in Figures 2 and 3, this embodiment provides an interpreter model for machine translation evaluation indicators based on fine-grained features. The interpreter model includes:

Splicing module: combine the machine-translated original text sequence S = (s ₁ , s ₂ ,..., s _n ), the machine-translated text sequence H = (h ₁ , h ₂ ,..., h _m ), and the quality assessment score Q, use SEP tags to splice them together; among them, the machine-translated original text sequence S = (s ₁ , s ₂ ,..., s _n ), and the machine-translated text sequence H = (h ₁ , h ₂ ,..., h _m ), and the quality assessment score Q, are both obtained through inference of the machine translation quality assessment model;

Coding sequence module: Input the obtained data into the pre-training model and calculate the coding sequence:

T=Encoder(S;H;Q)

Among them, Encoder is the pre-trained model, and T is the context label calculated by the pre-trained model;

Context representation module: Segment the encoding sequence according to the input position to obtain the context representation T _text of the original text and the translation, and the context representation T _score of the quality assessment score;

Attention weight module: Use dot product attention mechanism, connection attention mechanism or bilinear attention mechanism to calculate the attention weight matrix of the context representation T _text of the original text and the translation, and the context representation T _score of the quality assessment score:

attention=softmax(T _text ⊙T _score );

Relative importance score module: perform mean pooling on the attention weight matrix to obtain a one-dimensional relative importance score State:

State=MeanPooling(attention).

Fine-grained interpretable feature label module: Use the BIO label of sequence annotation to mark the starting and ending positions of each position related to the quality score, and mark the error type for each position to obtain a fine-grained interpretable feature label: B-miss The starting position of missed translation, the middle position of I-miss missing translation; the starting position of B-misT mistranslation, the middle position of I-misT mistranslation; the starting position of B-over over-translation, the middle position of I-over over-translation;

Sequence label module: Based on the relative importance score State and fine-grained interpretability feature labels, the sequence label Y=(y ₁ , y ₂ ,..., y _n+m ) is calculated using CRF:

Y=CRF(State).

It should be noted that the structure and/or principle of each of the above modules corresponds one-to-one to the content in the method for interpreting machine translation evaluation indicators based on fine-grained features described in Embodiment 1, and therefore will not be described again here.

It should be noted that the division of each module of the above system is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or it can be physically separated, and these modules can all be implemented in software. It can be implemented in the form of calling processing elements; it can also be all implemented in the form of hardware; some modules can also be implemented in the form of calling software through processing elements, and some modules can be implemented in the form of hardware. For example, a certain module can be a separate processing element, or can be integrated into a chip of the device. In addition, it can also be stored in the memory of the device in the form of program code, and can be called and implemented by a certain processing element of the device. Execute the function of a certain module, and the implementation of other modules is similar. In addition, all or part of these modules can be integrated together or implemented independently. The processing element described here may be an integrated circuit with signal processing capabilities. During the implementation process, each step of the above method or each of the above modules can be completed by instructions in the form of hardware integrated logic circuits or software in the processor element.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more specific integrated circuits, or one or more microprocessors, or one or more field programmable circuits. Gate array etc. For another example, when one of the above modules is implemented in the form of a processing element scheduling program code, the processing element can be a general processor, such as a central processing unit or other processor that can call the program code. For another example, these modules can be integrated together and implemented in the form of a system-on-chip.

Example 3

This embodiment provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the method for interpreting machine translation evaluation indicators based on fine-grained features provided in Embodiment 1. Those skilled in the art can understand that all or part of the steps for implementing the method provided in Embodiment 1 can be completed by hardware related to a computer program. The above computer program can be stored in a computer-readable storage medium. When the program is executed, Execute steps including the method provided in Embodiment 1; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

The above is the preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the design principles and technical solutions of the present invention, and these improvements should also be regarded as the protection scope of the present invention.

Claims (9)

1. The interpretation method of the turning evaluation index based on the fine granularity characteristics is characterized by comprising the following steps of:

step S1: turning original text sequence S= (S) ₁ ，s ₂ ，...，s _n ) Machine translated text sequence h= (H) ₁ ，h ₂ ，...，h _m ) And a quality assessment score Q, spliced together using labels;

step S2: inputting the data obtained in the step S1 into a pre-training model, and calculating to obtain a coding sequence:

T＝Encoder(S；H；Q)

wherein, the Encoder is a pre-training model, and T is a context label obtained by calculation through the pre-training model;

step S3: dividing the coding sequence according to the input position to obtain context expression T of the original text and the translated text _text And the context of the quality assessment score represents T _score ；

Step S4: calculating contextual representations T of originals and translations using an attention mechanism _text And the context of the quality assessment score represents T _score Is a matrix of attention weights:

attention＝softmax(T _text ⊙T _score )；

step S5: carrying out mean value pooling on the attention weight matrix to obtain a one-dimensional relative importance score State:

State＝MeanPooling(attention)；

step S6: marking the start and stop positions of each position related to the quality score by using a sequence marked tag, and marking the error type for each position to obtain a fine-granularity interpretable feature tag;

step S7: based onThe relative importance score State and the fine granularity interpretable feature tag are calculated using CRF to obtain a sequence tag Y= (Y) ₁ ，y ₂ ，...，y _n+m )：

Y＝CRF(State)。

2. The interpretation method of the turning evaluation index based on the fine granularity characteristics according to claim 1, wherein the interpretation method comprises the following steps: in the step S1, SEP label stitching is used.

3. The method according to claim 2, wherein the attention mechanism in step S4 is one of a point-by-point attention mechanism, a connection attention mechanism, and a bilinear attention mechanism.

4. The method for interpreting a turning evaluation index based on fine-grained characteristics according to claim 3, wherein a BIO tag is used in said step S6.

5. The method for interpreting a machine-turning evaluation index based on fine-grained feature according to claim 4, wherein the fine-grained interpretable feature tag in step S6 comprises:

b-miss translation starting position and I-miss translation intermediate position;

B-misT transliteration start position, I-misT transliteration intermediate position;

b-over-translation start position, I-over-translation intermediate position.

6. The method for interpreting a machine-turning evaluation index based on fine granularity features as claimed in claim 5, wherein the machine-turning original text sequence s= (S) in step S1 ₁ ，s ₂ ，...，s _n ) Machine translated text sequence h= (H) ₁ ，h ₂ ，...，h _m ) And the quality evaluation score Q is obtained through machine turning quality evaluation model reasoning.

7. The interpretation method of the turning evaluation index based on the fine granularity feature according to claim 6, characterized in that the error type mining method is as follows; first, the machine-turned original sequence s= (S) is inferred by the attention mechanism ₁ ，s ₂ ，...，s _n ) Machine translated text sequence h= (H) ₁ ，h ₂ ，...，h _m ) And an error segment of the quality assessment score Q, and is represented as a signal of high or low weight in the attention matrix; then, the signals with high and low weight values are mapped into transition state signals of 0-n, wherein n represents the number of error types.

8. An interpreter model of a machine turning evaluation index based on fine granularity characteristics, which is characterized by comprising:

and (3) splicing modules: turning original text sequence S= (S) ₁ ，s ₂ ，...，s _n ) Machine translated text sequence h= (H) ₁ ，h ₂ ，...，h _m ) And a quality assessment score Q, spliced together using labels;

coding sequence module: inputting the obtained data into a pre-training model, and calculating to obtain a coding sequence:

T＝Encoder(S；H；Q)

wherein, the Encoder is a pre-training model, and T is a context label obtained by calculation through the pre-training model;

the context representation module: dividing the coding sequence according to the input position to obtain context expression T of the original text and the translated text _text And the context of the quality assessment score represents T _score ；

Note weight module: calculating contextual representations T of originals and translations using an attention mechanism _text And the context of the quality assessment score represents T _score Is a matrix of attention weights:

attention＝softrmax(T _text ⊙T _score )；

a relative importance score module: carrying out mean value pooling on the attention weight matrix to obtain a one-dimensional relative importance score State:

State＝MeanPooling(attention)；

fine granularity interpretability feature tag module: marking the start and stop positions of each position related to the quality score by using a sequence marked tag, and marking the error type for each position to obtain a fine-granularity interpretable feature tag;

sequence tag module: based on the relative importance score State and the fine granularity interpretable feature tag, using CRF calculation to obtain a sequence tag y= (Y) ₁ ，y ₂ ，...，y _n+m )：

Y＝CRF(State)。

9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program is executed by a processor to implement the interpretation method of the machine turning evaluation index based on fine-grained feature as set forth in any one of claims 1 to 7.