CN113705253A

CN113705253A - Machine translation model performance detection method and related equipment

Info

Publication number: CN113705253A
Application number: CN202110219173.2A
Authority: CN
Inventors: 刘乐茂; 李冠林; 朱聪慧
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-02-26
Filing date: 2021-02-26
Publication date: 2021-11-26

Abstract

The embodiment of the application discloses a method for detecting the performance of a machine translation model and related equipment, and the method comprises the steps of obtaining a source text of a first language type and obtaining a target text of a second language type corresponding to the source text; replacing second target words in the source text with the plurality of first target words of the first language type respectively to obtain a plurality of candidate texts; translating the candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text; translating the source text through a machine translation model to obtain a second translation text of the second language type; and detecting a translation performance index of the machine translation model according to a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text. The accuracy of detecting the performance of the machine translation model is improved.

Description

Machine translation model performance detection method and related equipment

Technical Field

The application relates to the technical field of natural language processing, in particular to a method for detecting the performance of a machine translation model and related equipment.

Background

Machine translation techniques refer to techniques for translating, using a computer device, an original text in one natural language (generally referred to as a source language or source end) into a translated text in another natural language (generally referred to as a target language or target end). Since machine translation can be done autonomously by a computer device, a large amount of translation work can be processed in a relatively short time compared to manual translation.

In the prior art, Machine Translation can be implemented by a Machine Translation model, for example, a Statistical Machine Translation model (SMT), which generally relies on a word or Phrase alignment Table (Phrase Table) to translate a source language into a target language, and when judging the performance of the machine translation model, the target language translated by the machine translation model is compared with the standard language corresponding to the source language globally, the architecture, the number of parameters, and the amount of training data of the machine translation model are becoming more complex and large-scale, therefore, the source of the performance bottleneck cannot be fully recognized through the internal structure of the machine translation model, and the admissible degree of the prediction of the machine translation model after given arbitrary input relative to a user cannot be effectively controlled, so that the accuracy of the judgment of the performance of the machine translation model is low through the global comparison mode.

Disclosure of Invention

The embodiment of the application provides a method for detecting the performance of a machine translation model and related equipment, wherein the related equipment can comprise a device for detecting the performance of the machine translation model, computer equipment, a storage medium and the like, and the accuracy of detecting the performance of the machine translation model can be improved.

In order to solve the above technical problem, an embodiment of the present application provides the following technical solutions:

the embodiment of the application provides a method for detecting the performance of a machine translation model, which comprises the following steps:

acquiring a source text of a first language type, and acquiring a target text of a second language type corresponding to the source text;

replacing second target words in the source text with the plurality of first target words of the first language type respectively to obtain a plurality of candidate texts;

translating the candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text;

translating the source text through the machine translation model to obtain a second translation text of the second language type;

and detecting a translation performance index of the machine translation model according to a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text.

According to an aspect of the present application, there is also provided a device for detecting performance of a machine translation model, including:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a source text of a first language type and acquiring a target text of a second language type corresponding to the source text;

the replacing unit is used for replacing second target words in the source text by using the first target words of the first language type respectively to obtain a plurality of candidate texts;

the first translation unit is used for translating the candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text;

the second translation unit is used for translating the source text through the machine translation model to obtain a second translation text of the second language type;

and the detection unit is used for detecting the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text.

According to an aspect of the present application, there is also provided a computer device, including a processor and a memory, where the memory stores a computer program, and the processor executes any one of the methods for detecting the performance of the machine translation model provided in the embodiments of the present application when calling the computer program in the memory.

According to an aspect of the present application, there is also provided a storage medium for storing a computer program, which is loaded by a processor to execute any one of the methods for detecting the performance of the machine translation model provided in the embodiments of the present application.

The method and the device for processing the source text can acquire the source text of the first language type, acquire the target text of the second language type corresponding to the source text, and respectively replace the second target words in the source text with a plurality of first target words of the first language type to obtain a plurality of candidate texts; then, a plurality of candidate texts can be translated through the machine translation model to obtain a first translation text of a second language type corresponding to each candidate text, and a source text is translated through the machine translation model to obtain a second translation text of the second language type; at this time, the translation performance index of the machine translation model may be detected according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text. According to the scheme, the candidate text and the source text obtained after word replacement are translated through the machine translation model, and statistical analysis is carried out on the second similarity between the second translation text and the target text based on the first similarity between the first translation text and the target text, so that the translation performance index of the machine translation model is accurately detected, and the accuracy of detecting the performance of the machine translation model is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic view of a scenario in which a method for detecting performance of a machine translation model according to an embodiment of the present application is applied;

FIG. 2 is a schematic flow chart of a method for detecting performance of a machine translation model according to an embodiment of the present application;

FIG. 3 is a diagram illustrating the effect of different replacement strategies on translation performance indicators of a machine translation model according to an embodiment of the present application;

FIG. 4 is another schematic diagram of the effect of different replacement strategies on the translation performance indicators of the machine translation model provided by the embodiment of the present application;

FIG. 5 is a flowchart illustrating a method for detecting performance of a machine translation model according to an embodiment of the present disclosure;

fig. 6 is a schematic diagram of similarity distribution between a translated text and a target text of a candidate text obtained after a second target word in a source text is replaced according to the embodiment of the present application;

fig. 7 is another schematic diagram of similarity distribution between a translated text and a target text of a candidate text obtained after a second target word in a source text is replaced according to the embodiment of the present application;

fig. 8 is another schematic diagram of similarity distribution between a translated text and a target text of a candidate text obtained after a second target word in a source text is replaced according to the embodiment of the present application;

fig. 9 is another schematic diagram of similarity distribution between a translated text and a target text of a candidate text obtained after a second target word in a source text is replaced according to the embodiment of the present application;

fig. 10 is another schematic diagram of similarity distribution between a translated text and a target text of a candidate text obtained after replacement of a second target word in a source text according to the embodiment of the present application;

fig. 11 is another schematic diagram of similarity distribution between a translated text and a target text of a candidate text obtained after a second target word in a source text is replaced according to the embodiment of the present application;

FIG. 12 is a schematic diagram of a device for detecting the performance of a machine translation model according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a computer device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The embodiment of the application provides a method for detecting the performance of a machine translation model and related equipment, wherein the related equipment can comprise a device for detecting the performance of the machine translation model, computer equipment, a storage medium and the like.

Referring to fig. 1, fig. 1 is a schematic view of a scene of an application of a machine translation model performance detection method provided in an embodiment of the present application, where the application of the machine translation model performance detection method may include a machine translation model performance detection device, and the machine translation model performance detection device may be specifically integrated in a server or a terminal or other computer equipment, where the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data platform, and an artificial intelligence platform, but is not limited thereto. The terminal can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a vehicle-mounted device or a wearable device. The server and the terminal may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.

The computer device may be configured to obtain a source text of a first language type, obtain a target text of a second language type corresponding to the source text, and replace a second target word in the source text with a plurality of first target words of the first language type, to obtain a plurality of candidate texts; then, a plurality of candidate texts can be translated through the machine translation model to obtain a first translation text of a second language type corresponding to each candidate text, and a source text is translated through the machine translation model to obtain a second translation text of the second language type; at the moment, the translation performance index of the machine translation model can be detected according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text, so that the accuracy of detecting the performance of the machine translation model is improved.

It should be noted that the scenario diagram of the application of the machine translation model performance detection method shown in fig. 1 is merely an example, and the application and the scenario of the machine translation model performance detection method described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not form a limitation on the technical solution provided in the embodiment of the present application.

The following are detailed below. It should be noted that the following description of the embodiments is not intended to limit the preferred order of the embodiments.

The method for detecting the performance of the machine translation model provided by the embodiment of the application may relate to technologies such as a machine learning technology in artificial intelligence, for example, a text may be translated through a machine translation model including the machine learning technology in artificial intelligence, and the artificial intelligence technology and the machine learning technology are explained first below.

Artificial intelligence is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. Artificial intelligence infrastructures generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operating/interactive systems, and mechatronics. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Machine Learning (ML) is a multi-domain cross discipline, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal learning.

In this embodiment, a description will be given of a machine translation model performance detection apparatus, which may be specifically integrated in a computer device such as a server.

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a method for detecting performance of a machine translation model according to an embodiment of the present application. The method for detecting the performance of the machine translation model can comprise the following steps:

s101, obtaining a source text of a first language type, and obtaining a target text of a second language type corresponding to the source text.

The first language type and the second language type can be flexibly set according to actual needs, for example, the first language type can be Chinese, the source text can be Chinese text, the second language type can be English, and the target text can be English text; or, the first language type may be english, the source text may be english text, the second language type may be chinese, and the target text may be chinese text; alternatively, the first language type may be chinese, the source text may be chinese text, the second language type may be japanese, and the target text may be japanese text; alternatively, the first language type may be Chinese, the source text may be Chinese text, the second language type may be German, and the target text may be German text; or, the first language type may be chinese, the source text may be chinese text, the second language type may be french, and the target text may be french text; and so on.

The lengths, the numbers, the specific contents, and the like of the source text and the target text may be flexibly set according to actual needs, and are not limited herein, for example, the source text and the target text may be a sentence, a segment of text, or an article, and the source text may include a plurality of source texts, and each source text may correspond to one target text.

The source text of the first language type may be retrieved from a local database of the computer device, or may be downloaded from a server, or may receive user input of the source text of the first language type, or may collect and convert voice information entered by a user into the source text of the first language type, and so on. And acquiring a target text of a second language type corresponding to the source text through the trained translation model, or receiving a target text of a second language type corresponding to the source text translated by the user, or acquiring a target text of a second language type matched with the source text from a database in which the second language type text is stored in advance, and the like. Of course, the obtaining mode of the source text and the target text may also be flexibly set according to actual needs, and is not limited herein.

S102, replacing second target words in the source text with the first target words of the first language type respectively to obtain a plurality of candidate texts.

The number and specific content of the first target words may be flexibly set according to actual needs, for example, the first target words may be words or phrases composed of one or two words corresponding to the first language type, and a plurality of words may be screened from a vocabulary of the first language type as the first target words, where the vocabulary stores a plurality of different words. After the source text is obtained, word segmentation can be performed on the source text to obtain n words that constitute the source text X: x ═ X1, X2, X3.., xn ], then a second target word (i.e., a replaced word) is screened out from the n words, and at this time, the second target word in the source text can be replaced by the first target words respectively, so as to obtain a plurality of candidate texts. For example, if the first target word includes "you", "i", "he", and "we", and the second target word is x3, then "you" may replace the second target word x3 in the source text to obtain candidate text 1, "me" may replace the second target word x3 in the source text to obtain candidate text 2, "he" may replace the second target word x3 in the source text to obtain candidate text 3, and "we" may replace the second target word x3 in the source text to obtain candidate text 4.

In one embodiment, replacing the second target words in the source text with the plurality of first target words in the first language type, respectively, and obtaining the plurality of candidate texts may include: calculating a gradient value corresponding to a second target word in the source text through a loss function of a machine translation model based on the source text and the target text; sampling a plurality of first target words from a word list of a first language type according to the gradient value; and respectively replacing the second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

The type, structure, and the like of the Machine Translation model may be flexibly set according to actual needs, for example, the Machine Translation model may be a Neural Machine Translation model (NMT), a Statistical Machine Translation model (SMT), or the like.

In order to improve the reliability of the replacement, the second target word in the source text may be replaced by using a gradient sampling replacement strategy, and specifically, a gradient value corresponding to the second target word in the source text may be calculated by a loss function L (X, Y; theta) of the machine translation model based on the source text and the target text, for example, a gradient value of the loss function on a word vector of the second target word may be calculated. Where X may represent source text, Y may represent target text, and theta may represent parameters of the machine translation model.

Then, the plurality of first target words may be sampled from the vocabulary of the first language type according to the gradient values, and in one embodiment, the sampling the plurality of first target words from the vocabulary of the first language type according to the gradient values may include: updating the word vector of the second target word in the source text according to the gradient value and a preset learning rate to obtain an updated word vector; performing dot product operation on the updated word vector and a word vector matrix corresponding to the word list of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; and sampling a plurality of first target words with similarity greater than a preset similarity threshold value with a second target word in the source text from the word list of the first language type according to the similarity vector.

For example, in order to improve the reliability of the first target word screening, the word vector of the second target word in the source text may be updated according to the gradient value at a preset learning rate, so as to obtain an updated word vector, for example, the updated word vector embedding is the word vector of the second target word — the preset learning rate gradient value, where a specific value of the preset learning rate may be flexibly set according to actual needs, and for example, the preset learning rate may be 1.0.

The updated word vector and the word vector matrix corresponding to the vocabulary of the first language type may be subjected to a dot product operation to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix, where the similarity vector may be a vector of a size dimension of the vocabulary, and the word vector matrix may be a vector matrix formed by word vectors of each word in the vocabulary. At this time, a plurality of first target words whose similarity to a second target word in the source text is greater than a preset similarity threshold may be sampled (e.g., sampled without playback) from the vocabulary of the first language type according to the similarity vector. For example, a word with a larger similarity may be sampled from the word list as the first target word by using a polynomial distribution. After the first target words are obtained, the plurality of first target words may replace second target words in the source text, respectively, to obtain a plurality of candidate texts.

In one embodiment, replacing the second target words in the source text with the plurality of first target words in the first language type, respectively, and obtaining the plurality of candidate texts may include: screening a plurality of candidate words from a word list of a first language type; respectively replacing a second target word in the source text with the candidate words to obtain a plurality of replacement texts; calculating a loss value between the replacement text and the target text through a loss function of the machine translation model; screening out a preset number of candidate words corresponding to the minimum loss value to obtain a plurality of first target words; and respectively replacing the second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In order to improve the accuracy of the replacement, a hierarchical sampling replacement strategy can be used for replacing the second target word in the source text, specifically, a plurality of candidate words can be screened out randomly from a word list of the first language type or according to a preset screening strategy, and the plurality of candidate words are respectively replaced with the second target word in the source text to obtain a plurality of replacement texts. For example, if the candidate word includes word 1, word 2, and word 3, and the second target word is xi, the word 1 may be used to replace the second target word xi in the source text to obtain a replaced text 1, the word 2 may be used to replace the second target word xi in the source text to obtain a replaced text 2, and the word 3 may be used to replace the second target word xi in the source text to obtain a replaced text 3. A loss value between the replacement text and the target text may then be calculated by a loss function L (X ', Y; theta) of the machine translation model, where X' may represent the replacement text, Y may represent the target text, and theta may represent a parameter of the machine translation model. At this time, a preset number of replacement texts corresponding to the minimum loss value can be screened out, candidate words corresponding to the replacement texts with the preset number are used as a plurality of first target words, and the plurality of first target words are respectively replaced with second target words in the source text to obtain a plurality of candidate texts.

In one embodiment, replacing the second target words in the source text with the plurality of first target words in the first language type, respectively, and obtaining the plurality of candidate texts may include: determining a replacement location in the source text; randomly screening a plurality of first target words from a word list of a first language type; and respectively replacing the second target words corresponding to the replacement positions in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In order to improve the convenience of replacement, a random sampling replacement strategy can be used for replacing a second target word in a source text, specifically, the source text can be subjected to position division to obtain a plurality of replacement positions, the replacement positions to be replaced are screened out from the plurality of replacement positions, a plurality of first target words are randomly screened out from a word list of a first language type, and the plurality of first target words are respectively replaced by the second target word corresponding to the replacement positions in the source text to obtain a plurality of candidate texts.

In one embodiment, replacing the second target words in the source text with the plurality of first target words in the first language type, respectively, and obtaining the plurality of candidate texts may include: determining attribute information of a second target word in the source text; screening a plurality of first target words from a word list of the first language type according to the attribute information; and respectively replacing the second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In order to improve the flexibility of the replacement, an attribute sampling replacement strategy may be used to replace a second target word in the source text, where the attribute information may include part-of-speech or semantic information of the second target word, and specifically, the source text may be subjected to word segmentation to obtain a plurality of words, a second target word is screened out from the plurality of words at random or according to a preset screening strategy, and the attribute information of the second target word is determined, for example, the part-of-speech of the second target word may be analyzed or the semantic analysis may be performed on the second target word to obtain the attribute information of the second target word. Then, a plurality of first target words (for example, first target words with the same part of speech or similar semantic information) matched with the attribute information of the second target word may be screened out from the word list of the first language type according to the attribute information of the second target word, and the second target word in the source text is replaced by the plurality of first target words, so as to obtain a plurality of candidate texts.

S103, translating the candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text.

After a plurality of candidate texts are obtained, each candidate text can be translated through a machine translation model respectively to obtain a first translation text of a second language type corresponding to each candidate text, and a first similarity between the first translation text and a target text can be calculated (for example, a higher numerical value indicates a better effect according to a machine translation evaluation standard of a sub-level). For example, the first translated text and the target text may be compared word by word to detect the same portion existing between the first translated text and the target text, and the percentage of the same portion may be calculated as the first similarity. For another example, the word-by-word comparison and semantic comparison may be performed on the first translated text and the target text, and the first similarity between the first translated text and the target text may be calculated according to the comparison result. The higher the first similarity, the more similar the first translated text is to the target text, whereas the lower the first similarity, the less similar the first translated text is to the target text.

And S104, translating the source text through the machine translation model to obtain a second translation text of the second language type.

After the source text is obtained, the source text can be translated through a machine translation model to obtain a second translation text of a second language type corresponding to the source text, and a second similarity between the second translation text and the target text can be calculated. For example, the second translated text may be compared word by word with the target text to detect the same portion existing between the second translated text and the target text, and the percentage of the same portion may be calculated as the second similarity. For another example, the second translated text and the target text may be subjected to word-by-word comparison and semantic comparison, and a second similarity between the second translated text and the target text may be calculated according to the comparison result; and so on.

In one embodiment, obtaining a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text may include: acquiring a first initial similarity between the first translation text and the target text and a second initial similarity between the second translation text and the target text through a machine translation model; calculating a first translation score of the candidate text and a second translation score of the source text by using a reordering algorithm; and adjusting the second initial similarity according to the second translation score to obtain a second similarity between the second translation text and the target text.

For example, for a source text a, under the condition that a target text corresponding to the source text a is unknown, a translation score that the source text a can be translated into the target text can be calculated, where a higher translation score indicates that a translation result of the source text a is closer to the target text, and conversely, a lower translation score indicates that a translation result of the source text a is less close to the target text. In order to improve the accuracy of similarity calculation, the similarity obtained by the machine translation model may be finely adjusted based on the translation score calculated by the reordering algorithm, for example, a first initial similarity between the first translated text and the target text and a second initial similarity between the second translated text and the target text may be obtained by the machine translation model, the first translation score of the candidate text may be calculated by the reordering algorithm, and the second translation score of the source text may be calculated by the reordering algorithm. Then, the first initial similarity is adjusted according to the first translation score to obtain a first similarity between the first translation text and the target text, and the second initial similarity is adjusted according to the second translation score to obtain a second similarity between the second translation text and the target text. For example, the first translation score may be taken as a first similarity between the first translated text and the target text, and the second translation score may be taken as a second similarity between the second translated text and the target text, or a mean value of the first translation score and the first initial similarity may be taken as a first similarity between the first translated text and the target text, a mean value of the second translation score and the second initial similarity may be taken as a second similarity between the second translated text and the target text, and so on.

And S105, detecting the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text.

The translation performance indexes of the machine translation model can include translation accuracy, translation stability and the like, and after a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text are obtained, the translation performance indexes of the machine translation model can be detected according to the first similarity and the second similarity.

For example, the source text and the candidate text may be used as test samples, and a generalization barrier word of each test sample on the machine translation model may be obtained, where the generalization barrier word may be a word that affects accurate translation of the machine translation model, and a part of speech or a semantic of the word may be analyzed to indicate that the word of the part of speech or the semantic affects accurate translation of the machine translation model, and indicate whether the word prevents successful generalization of the machine translation model on the test sample. The generalized barrier word detection algorithm may be used to detect a set of words in the input at the sample level that result in a model that is poorly generalized over the sample.

It should be noted that, in this embodiment, detection of the translation performance index of the machine translation model may be referred to as fine-grained Analysis, and fine-grained representation may be how to judge a specific representation of the machine translation model on an input given the input of the machine translation model, and Analysis at the sample level (Instance-level Analysis) may be referred to as fine-grained Analysis. The fine-grained analysis can convey richer error information, so that the behavior of the machine translation model can be more effectively understood, the energy and the energy of the machine translation model can be known, and an effective solution can be more specifically provided.

It should be noted that, in addition to the detection in the presence of the target text, a scenario without the target text may be used instead, for example, a quality assessment model is used as a substitute for the standard loss to detect the translation performance index of the machine translation model according to the standard loss.

In one embodiment, detecting the translation performance index of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text may include: acquiring a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text; screening candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to the candidate texts to obtain target candidate texts; analyzing the part-of-speech distribution of a second target word in the source text according to the target candidate text; and determining the translation performance index of the machine translation model according to the part of speech distribution.

In order to improve the accuracy of detecting the translation performance index of the machine translation model, the translation performance index of the machine translation model can be determined according to the part-of-speech distribution of a second target word in a source text, and specifically, a candidate text with a first similarity greater than a second similarity can be screened from first translation texts corresponding to a plurality of candidate texts to obtain a target candidate text; for example, the first similarity between the second translated text Yo corresponding to the source text and the target text Y is b0, there are b first similarities between the first translated texts corresponding to the b candidate texts and the target text Y, at this time, a value greater than b0 is to be screened out from the b first similarities, and the candidate text corresponding to the first similarity greater than b0 is taken as the target candidate text. Then, a source text corresponding to the target candidate text may be obtained, and a part-of-speech of a second target word in the source text corresponding to the target candidate text may be determined, and a part-of-speech distribution may be formed for the part-of-speech of the second target word in the one or more source texts, at this time, a translation performance index of the machine translation model may be determined according to the part-of-speech distribution, for example, a part-of-speech affecting accurate translation by the machine translation model may be analyzed to obtain a generalized barrier word of the machine translation model. The part of speech may include nouns, verbs, adjectives, pronouns, prepositions, adverbs, conjunctions, and the like.

In one embodiment, analyzing the part-of-speech distribution of the second target word in the source text according to the target candidate text may include: dividing the target candidate texts into a plurality of groups of candidate texts according to the replacement position of a second target word in the source text; calculating the mean value of the similarity corresponding to each group of candidate texts; and determining the part-of-speech distribution of the second target words in the source text according to the front preset group of candidate texts with the maximum mean value.

In analyzing the part-of-speech distribution of the second target word in the source text, the target candidate texts may be divided into multiple groups of candidate texts according to the replacement position of the second target word in the source text, for example, the group of candidate texts corresponding to the source text a may include: and respectively replacing a second target word at a replacement position a in the source text A with a first target word 1, a first target word 2, a first target word 3, a first target word 4, a first target word 5 and the like to obtain a candidate text group 1, respectively replacing a second target word at a replacement position b in the source text A with a first target word 1, a first target word 6, a first target word 7, a first target word 8 and the like to obtain a candidate text group 2, respectively replacing a second target word at a replacement position c in the source text A with a first target word 9, a first target word 10, a first target word 11, a first target word 12, a first target word 13 and the like to obtain a candidate text group 3, and the like. Then, a mean value of the similarity corresponding to each group of candidate texts may be calculated, for example, a mean value (rounded mean, tm) of the first similarities corresponding to 5 candidate texts in the candidate text group 1 is calculated, a candidate text in a pre-set group with a maximum mean value (which may also be referred to as tm value) is selected from a plurality of candidate text groups corresponding to the same source text (for example, k% candidate text with a maximum tm value is selected), and a part-of-speech distribution of a second target word in the source text corresponding to the candidate text in the pre-set group with the maximum mean value is counted. Wherein, the specific values of the front preset group, k% and the like can be flexibly set according to the actual requirement,

as will be illustrated below, based on generalized barrier word detection, a second target word in the modified source text may be used to obtain a better translation candidate for reordering (re-ranking), and thus to select a better translation prediction. And analyzing the part-of-speech distribution of the generalized barrier words: based on the generalized barrier word detection algorithm, tm values corresponding to second target words in the source text X can be obtained, second target words with higher tm values are obtained through sorting, the second target words with top-k (k is 10%, 20% or 30%) are taken as generalized barrier words on the source text X, and the characteristics of the second target words are captured through analyzing the part-of-speech distribution of the second target words. The part-of-speech distributions (i.e., the part-of-speech distribution of the second target word) of the generalized barrier words in the chinese-english direction and the english-chinese direction are shown in the following tables 1 and 2, respectively. For reference, a natural distribution of parts of speech of all words in the source text (i.e. a distribution of parts of speech of words contained in the source text) is given. Whether a word with a certain part of speech is more likely to be a generalized barrier word or not is judged by comparing the change (increase and decrease) of the barrier word proportion compared with the natural distribution word proportion.

The conclusions that can be drawn by observing tables 1 and 2 include: 1. the generalized barrier words exist widely in all part-of-speech categories and are close to the natural distribution of parts-of-speech; 2. prepositions (Prep.) and punctuation (Punc.) tend to be generalized barrier words; 3. words that express content or reality are less prone to be generalized barrier words.

The value of the above conclusions is: 1. the sensitivity of the machine translation model to the context is reflected, namely, the more the words are related to the context, the more the translation of other words is influenced; 2. the detection classifier can be used as a characteristic for training the generalized barrier word; 3. the system designer can be guided to pay more attention to the influence of certain category words on the whole sentence semantics of the source text (also called a source end), so that a potential method for blocking the context semantic interference is provided.

Table 1: part-of-speech distribution of generalized barrier words in Chinese in the direction of Chinese-to-English translation

Part of speech categories	k＝10％	k＝20％	k＝30％	Natural distribution of
					Byte pair encoding BPE	9.80％-	10.74％-	11.26％-	12.00％
Noun Noun	22.17％-	22.43％-	21.85％-	24.07％
					Pronouns Pron.	1.94％-	2.18％+	2.25％+	2.15％
The Verb Verb.	11.57％+	11.28％+	11.00％-	11.26％
					Adjective Adj	6.74％-	7.19％-	7.26％-	8.19％
Adverb Adv.	3.24％+	3.07％+	2.83％-	2.93％
					Preposition Prep.	12.94％+	13.05％+	13.39％+	11.88％
Punctuation mark Punc.	16.04％+	13.98％+	13.30％+	10.41％
					The qualifier Det.	8.11％-	8.84％-	9.42％+	9.05％
Conjunction C&C	1.94％-	2.06％-	2.05％-	2.20％

Table 2: part-of-speech distribution of generalized barrier words in English in the English-Chinese translation direction

Part of speech categories	k＝10％	k＝20％	k＝30％	Natural distribution of
					Byte pair encoding BPE	14.32％-	15.10％-	15.28％-	15.33％
Noun Noun	16.52％-	16.23％-	15.83％-	17.63％
					Proprietary name prop.n.	6.56％-	6.75％-	6.37％-	7.44％
Pronouns Pron.	1.75％-	1.91％-	2.32％-	2.35％
					The Verb Verb.	18.37％+	18.33％-	18.56％+	18.36％
Adjective Adj	2.50％-	2.56％-	2.60％-	3.19％
					Adverb Adv.	4.30％+	4.27％+	4.14％+	4.07％
Preposition Prep.	4.70％+	4.65％+	4.58％+	3.83％
					Punctuation mark Punc.	16.65％+	14.49％+	14.40％+	11.44％
Number word Q&M	3.95％-	4.49％-	4.59％-	4.87％
					Conjunction C&C	1.84％-	1.79％-	1.99％-	2.23％

In table 1 and table 2, k ═ 10% may indicate that the top 10% candidate text group with the largest average value (i.e., tm value) is selected from a plurality of candidate text groups corresponding to the source text, so as to count the part-of-speech distribution of the second target word in the source text corresponding to the 10% candidate text group. Nature may respectively represent the distribution of parts of speech of each word in the source text, "-" in "9.80% -" may represent that 9.80% is smaller than the natural distribution of 12.00%, "+" in "11.57% +" may represent that 11.57% + is larger than the natural distribution of 11.26%, and the percentage corresponding to each part of speech may represent the proportion of each part of speech to the word in the source text.

In one embodiment, replacing the second target words in the source text with the plurality of first target words in the first language type, respectively, and obtaining the plurality of candidate texts may include: and respectively replacing words in the source text by using a plurality of first target words of the first language type according to different replacement strategies to obtain a plurality of candidate texts corresponding to each replacement strategy. According to a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text, detecting a translation performance index of the machine translation model may include: respectively screening out candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to a plurality of candidate texts obtained by replacement based on each replacement strategy to obtain target candidate texts corresponding to each replacement strategy; dividing the target candidate texts corresponding to each replacement strategy into a plurality of groups of candidate texts according to the replacement positions of second target words in the source text, and calculating the mean value of the similarity corresponding to each group of candidate texts; sequencing the candidate texts corresponding to each replacement strategy according to the sequence of the mean value from high to low, and analyzing the contact ratio of the source texts corresponding to each replacement strategy according to the sequencing result; and determining the translation performance index of the machine translation model according to the contact ratio.

In order to improve the reliability of the detection of the translation performance index of the machine translation model, the translation performance index of the machine translation model may be determined according to the contact ratio corresponding to different replacement strategies, where the replacement strategies may include the above gradient sampling replacement strategy, hierarchical sampling replacement strategy, random sampling replacement strategy, and the like. Specifically, according to different replacement strategies such as a gradient sampling replacement strategy, a hierarchical sampling replacement strategy, a random sampling replacement strategy, and the like, the words in the source text are replaced by the multiple first target words of the first language type, so as to obtain multiple candidate texts corresponding to each replacement strategy. In the process of detecting the translation performance index of the machine translation model, candidate texts with the first similarity larger than the second similarity can be screened out from first translation texts corresponding to a plurality of candidate texts obtained based on each replacement strategy through replacement, and target candidate texts corresponding to each replacement strategy are obtained; dividing the target candidate texts corresponding to each replacement strategy into a plurality of groups of candidate texts according to the replacement positions of second target words in the source texts, and calculating the mean value of the similarity corresponding to each group of candidate texts; and sequencing the candidate texts corresponding to each replacement strategy according to the sequence of the mean value from high to low, analyzing the contact ratio of the source texts corresponding to each replacement strategy according to the sequencing result, and determining the translation performance index of the machine translation model according to the contact ratio.

As will be illustrated below, for example, as shown in fig. 3, the influence of different replacement strategies on the accuracy of the machine translation model in generating the candidate texts is reflected, and it can be seen that, under different candidate texts b, as the number of the candidate texts b increases, the accuracy is higher, and the accuracy corresponding to each replacement strategy is almost equivalent. In fig. 3, the abscissa may represent the number of candidate texts b obtained by replacing the second target word in the source text, and the ordinate may represent the overlap ratio (e.g., whether k% words are the same or not) of the source text corresponding to different replacement strategies.

Analysis of overlap with other source text word categories: the second target word in the source text of top-k obtained by the generalized barrier word recognition algorithm may be compared with some other categories, and the degree of overlap may be as shown in table 3. It can be seen that the coincidence degree of the low-Frequency word (Frequency) and the high-translation-Entropy (entry) word and the error word (Exception) with the barrier word is equivalent to the coincidence degree of the randomly-screened word and the barrier word, which indicates that the generalized barrier word is substantially different from the low-Frequency word, the high-translation-Entropy word and the error word. One reason may be that the words corresponding to the three categories are all global, while the barrier word is a product under the Analysis Instance-level Analysis at the sample level, and there is a great possibility of great difference; another reason may be that the generalization barrier word may itself be translated correctly, but affects the translation of other words, while the low frequency words, the high entropy words and the wrong words are mostly words that themselves are not easily translated correctly.

Table 3: overlap ratio relationship between generalized barrier words and existing other words in English-to-Chinese, Chinese-to-English translation direction

In one embodiment, the source text includes a plurality of texts, and detecting the translation performance index of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text may include: respectively screening candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to a plurality of candidate texts corresponding to each source text to obtain a target candidate text corresponding to each source text; dividing the target candidate texts corresponding to each source text into a plurality of groups of candidate texts according to the replacement positions of second target words in the source text, and calculating the mean value of the similarity corresponding to each group of candidate texts; calculating the variance value of the similarity corresponding to the candidate texts in the candidate text group with the maximum average value; and determining the translation performance index of the machine translation model according to the variance value.

In order to accurately detect the stability of the translation performance of the machine translation model, the translation performance index of the machine translation model may be determined according to the variance value of the similarity corresponding to the candidate text, for example, as shown in fig. 4, in terms of stability, the variance of a hierarchical sampling replacement strategy (which may be simply referred to as hierarchical sampling) is minimum, and the variances based on gradient sampling and random sampling are similar, so that the tm value may be preferably calculated by using hierarchical sampling. In fig. 4, the abscissa may represent the number of candidate texts b obtained by replacing the second target word in the source text, the ordinate may represent the ranking correlation, whether the rearrangement between every two candidate texts in k% is correlated or not, and the smaller the variance, the higher the stability.

In an embodiment, after detecting the translation performance index of the machine translation model according to the first similarity between the first translated text and the target text and the second similarity between the second translated text and the target text, the method for detecting the performance of the machine translation model may further include: adjusting the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model; and translating the text to be translated through the adjusted machine translation model.

After the translation performance index of the machine translation model is obtained through detection, parameters of the machine translation model can be adjusted according to the translation performance index of the machine translation model, and the adjusted machine translation model is obtained. When the subsequent text is translated, the text to be translated can be accurately translated through the adjusted machine translation model.

It should be noted that the importance of unsupervised detection of the generalized barrier word and the feasibility of modifying the second target word in the source text to generate a reordering candidate are reflected by analyzing the gain of the reordering related index brought by modifying the second target word in the source text. To approximate the true re-ranking task, each second target word in the source text may be modified to generate a translation file (which may be referred to as a translation candidate) corresponding to the candidate text, rather than modifying the candidate text if the target text is known. The candidate texts are compared with top-m candidates generated by a standard beam search by measuring similarity values (larger is better), differences (smaller is better) and recall rates (larger is better) of the target texts. It can be found that each index is a candidate obtained by modifying the barrier word, i.e. the probability of obtaining a better translation candidate by modifying the second target word (also referred to as modifying the source end word) in the source text is verified.

Table 4: in the English-to-Chinese and Chinese-to-English translation direction, the translation candidate obtained by modifying the source end word is compared with the translation candidate obtained by beam search

The method described in the above embodiments is further illustrated in detail by way of example.

In this embodiment, for example, the device for detecting the performance of the machine translation model is integrated in a computer device, and for example, the source text is chinese and the target text is english, please refer to fig. 5, and fig. 5 is a schematic flow chart of the method for detecting the performance of the machine translation model according to the embodiment of the present application. The method flow can comprise the following steps:

s201, a source text of Chinese is obtained, and an English target text corresponding to the source text is obtained.

For example, the computer device may retrieve source text from a local database, or may receive source text input by a user, or may collect voice information entered by a user and convert the voice information to source text in chinese, and so on. And the computer device can acquire the target text corresponding to the source text through the trained translation model, or can receive the target text translated by the user, and the like.

S202, replacing second target words in the source text with the plurality of first target words in the Chinese language respectively according to a replacement strategy to obtain a plurality of candidate texts.

The replacement strategy may include a gradient sampling replacement strategy, a hierarchical sampling replacement strategy, a random sampling replacement strategy, and the like, and the computer device may replace a second target word in the source text with a plurality of first target words in chinese according to one or more replacement strategies, to obtain a plurality of candidate texts corresponding to each replacement strategy.

For example, replacing the second target word in the source text with a gradient-sampling replacement policy may include: calculating a gradient value corresponding to a second target word in the source text through a loss function of a machine translation model based on the source text and the target text, and updating a word vector of the second target word in the source text according to the gradient value and a preset learning rate to obtain an updated word vector; performing dot product operation on the updated word vector and a word vector matrix corresponding to the Chinese word list to obtain a similarity vector between the word vector of a second target word in the source text and each word vector in the word vector matrix; and sampling a plurality of first target words with similarity greater than a preset similarity threshold with a second target word in the source text from a word list of the Chinese according to the similarity vector, and replacing the second target words in the source text with the first target words respectively to obtain a plurality of candidate texts.

For another example, replacing the second target word in the source text with the hierarchical sampling replacement policy may include: screening a plurality of candidate words from a Chinese word list, and respectively replacing second target words in the source text with the candidate words to obtain a plurality of replacement texts; calculating a loss value between the replacement text and the target text through a loss function of the machine translation model, screening out a preset number of candidate words corresponding to the minimum loss value to obtain a plurality of first target words, and replacing second target words in the source text with the plurality of first target words respectively to obtain a plurality of candidate texts.

For another example, replacing the second target word in the source text with a randomly sampled replacement policy may include: determining a replacement position in the source text, randomly screening a plurality of first target words from a word list of the first language type, and replacing a second target word corresponding to the replacement position in the source text with the plurality of first target words to obtain a plurality of candidate texts.

S203, translating the candidate texts through a machine translation model to obtain English first translation texts corresponding to the candidate texts.

And S204, translating the source text through the machine translation model to obtain a second English translation text corresponding to the source text.

S205, acquiring a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text.

After a plurality of candidate texts are obtained, each candidate text can be translated through a machine translation model respectively to obtain an English first translation text corresponding to each candidate text, and a first similarity between the first translation text and a target text can be calculated. After the source text is obtained, the source text can be translated through a machine translation model to obtain an English second translation text corresponding to the source text, and a second similarity between the second translation text and the target text can be calculated.

S206, screening candidate texts with the first similarity larger than the second similarity from the first translation texts corresponding to the candidate texts to obtain target candidate texts.

And S207, analyzing the part-of-speech distribution of the second target words in the source text according to the target candidate text.

For example, the target candidate texts may be divided into multiple groups of candidate texts according to the replacement position of the second target word in the source text, a mean value of the similarity corresponding to each group of candidate texts is calculated, and the part-of-speech distribution of the second target word in the source text is determined according to a pre-set group of candidate texts with the largest mean value.

And S208, determining the translation performance index of the machine translation model according to the part of speech distribution.

For example, a generalization barrier word affecting the accurate translation performance of the machine translation model may be determined according to the part-of-speech distribution, and the generalization barrier word detection may determine the effect of the modified word on the generalization performance of the machine translation model on the example by using a similar counterfactual example, i.e., performing fine-tuning on the input (modifying only one word), and observing the average performance improvement of the example (i.e., the source text) caused by a large number of modification examples (i.e., candidate texts). The strong interaction (namely context correlation) of words in the source text can be fully considered, and specific barrier words existing in a specific input sample can be obtained, so that the method is more consistent with the internal logic of sample-level Analysis. That is, detecting the generalized barrier words at the sample level with the target text known, and using the average quality of the better translation output generated by the counterfactual to measure the risk that the modified part of the original input source text becomes the generalized barrier. Due to the fact that the word list size of a certain word of a source text needs to be modified for many times, the approximate sampling method is provided, complexity is greatly reduced, and the risk estimation variance of a determinacy reduction detection algorithm in hierarchical sampling is fully utilized. The detection algorithm can be used to analyze the machine translation model for defects, and specifically to consider which kind of words are modeled. Differences in detection of generalized barrier words in the same input by two different machine translation models may also be counted.

S209, adjusting the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model, and translating the text to be translated through the adjusted machine translation model.

For example, the Source text Source for Chinese is: the goals of the genetic scientist are: diagnostic tools are provided to discover disease-causing defective genes and ultimately provide therapies that prevent these genes from producing disorders.

The English target text Reference corresponding to the source text is: the high of genetics is to product diagnostic tools to identify functional genes such as this at least two ways of describing each of the preceding illustrative genes from the functional functions.

The translated text Original perspective obtained by translating the source text through the machine translation model is as follows: the high of genetic scientists is to product diagnostic tools to products of the diseases and in the end, to product protocols which can be expressed the product of the genes.

The better translated text (better translation result after replacement) obtained by translating the candidate text through the machine translation model is as follows: the coarse of gene scientists to product diagnostic tools to detect genes that are said patents, and even product languages that can be used to detect the genes from the product.

The second target words in the source text may be replaced with a plurality of first target words in chinese, respectively, to obtain a plurality of candidate texts, for example, the "diagnosis", "tool", "may", "defect", "end" or "and" etc. in the source text may be replaced with a plurality of first target words.

As shown in fig. 6, fig. 6 is a histogram of similarity distribution obtained by replacing "diagnosis" in a source text with a plurality of first target words, wherein the abscissa may represent a tm value and the ordinate may represent the number of candidate texts. Fig. 7 is a histogram of similarity distribution obtained by replacing a "tool" in a source text with a plurality of first target words, fig. 8 is a histogram of similarity distribution obtained by replacing a "available" in a source text with a plurality of first target words, fig. 9 is a histogram of similarity distribution obtained by replacing a "defect" in a source text with a plurality of first target words, fig. 10 is a histogram of similarity distribution obtained by replacing a "final" in a source text with a plurality of first target words, and fig. 11 is a histogram of similarity distribution obtained by replacing a "defect" in a source text with a plurality of first target words. It can be seen that the generated counterfactual sample similarity sensor BLEU distribution histogram, after changing the second target word "Defect" in the source text to "according" the machine translation model can produce better prediction results, in other words, the presence of the word "Defect" leads to the translation of "pathogenic genes" and "therapies that produced the disorder".

In order to measure each replacement strategy, the degree of coincidence of the words with tm values positioned at the top k can be calculated through the sorted input source text X obtained by accurate tm values of a plurality of (for example, 50) source texts, so as to measure the accuracy of the replacement strategy; and the variance of different replacement strategies can be reflected by the relevance rank correlation of two sorted input source texts X obtained after two times of different sampling.

In the above embodiments, the descriptions of the embodiments have respective emphasis, and a part which is not described in detail in a certain embodiment may refer to the above detailed description of the method for detecting the performance of the machine translation model, and is not described herein again.

According to the method and the device, a source text of Chinese can be obtained, an English target text corresponding to the source text is obtained, and a plurality of first target words of Chinese are used for replacing second target words in the source text respectively according to a replacement strategy to obtain a plurality of candidate texts; then, a plurality of candidate texts can be translated through the machine translation model to obtain an English first translation text corresponding to each candidate text, and a source text is translated through the machine translation model to obtain an English second translation text; at this time, the translation performance index of the machine translation model can be detected according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text, so that the machine translation model can be adjusted according to the translation performance index, the adjusted machine translation model can accurately translate the text, and the accuracy of detecting the performance of the machine translation model is improved.

In order to better implement the method for detecting the performance of the machine translation model provided by the embodiment of the present application, the embodiment of the present application further provides a device based on the method for detecting the performance of the machine translation model. The meaning of the noun is the same as that in the above method for detecting the performance of the machine translation model, and specific implementation details can refer to the description in the method embodiment.

Referring to fig. 12, fig. 12 is a schematic structural diagram of a machine translation model performance detection apparatus according to an embodiment of the present disclosure, where the machine translation model performance detection apparatus may include an obtaining unit 301, a replacing unit 302, a first translating unit 303, a second translating unit 304, a detecting unit 305, and the like.

The acquiring unit 301 is configured to acquire a source text of a first language type and acquire a target text of a second language type corresponding to the source text.

A replacing unit 302, configured to replace a second target word in the source text with the multiple first target words of the first language type, respectively, so as to obtain multiple candidate texts.

The first translation unit 303 is configured to translate the multiple candidate texts through a machine translation model to obtain a first translated text of the second language type corresponding to each candidate text.

The second translation unit 304 is configured to translate the source text through the machine translation model to obtain a second translation text of the second language type.

The detecting unit 305 is configured to detect a translation performance index of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text.

In an embodiment, the replacing unit 302 may specifically be configured to: calculating a gradient value corresponding to a second target word in the source text through a loss function of a machine translation model based on the source text and the target text; sampling a plurality of first target words from a word list of a first language type according to the gradient value; and respectively replacing the second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In an embodiment, the replacing unit 302 may specifically be configured to: updating the word vector of the second target word in the source text according to the gradient value and a preset learning rate to obtain an updated word vector; performing dot product operation on the updated word vector and a word vector matrix corresponding to the word list of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; and sampling a plurality of first target words with similarity greater than a preset similarity threshold value with a second target word in the source text from the word list of the first language type according to the similarity vector.

In an embodiment, the replacing unit 302 may specifically be configured to: screening a plurality of candidate words from a word list of a first language type; respectively replacing a second target word in the source text with the candidate words to obtain a plurality of replacement texts; calculating a loss value between the replacement text and the target text through a loss function of the machine translation model; screening out a preset number of candidate words corresponding to the minimum loss value to obtain a plurality of first target words; and respectively replacing the second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In an embodiment, the replacing unit 302 may specifically be configured to: determining a replacement location in the source text; randomly screening a plurality of first target words from a word list of a first language type; and respectively replacing the second target words corresponding to the replacement positions in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In an embodiment, the replacing unit 302 may specifically be configured to: determining attribute information of a second target word in the source text; screening a plurality of first target words from a word list of the first language type according to the attribute information; and respectively replacing the second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

In an embodiment, the detecting unit 305 may specifically be configured to: acquiring a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text; screening candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to the candidate texts to obtain target candidate texts; analyzing the part-of-speech distribution of a second target word in the source text according to the target candidate text; and determining the translation performance index of the machine translation model according to the part of speech distribution.

In an embodiment, the detecting unit 305 may specifically be configured to: dividing the target candidate texts into a plurality of groups of candidate texts according to the replacement position of a second target word in the source text; calculating the mean value of the similarity corresponding to each group of candidate texts; and determining the part-of-speech distribution of the second target words in the source text according to the front preset group of candidate texts with the maximum mean value.

In an embodiment, the detecting unit 305 may specifically be configured to: acquiring a first initial similarity between the first translation text and the target text and a second initial similarity between the second translation text and the target text through a machine translation model; calculating a first translation score of the candidate text and a second translation score of the source text by using a reordering algorithm; and adjusting the second initial similarity according to the second translation score to obtain a second similarity between the second translation text and the target text.

In an embodiment, the replacing unit 302 may specifically be configured to: respectively replacing words in the source text by using a plurality of first target words of a first language type according to different replacement strategies to obtain a plurality of candidate texts corresponding to each replacement strategy; the detection unit 305 may specifically be configured to: respectively screening out candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to a plurality of candidate texts obtained by replacement based on each replacement strategy to obtain target candidate texts corresponding to each replacement strategy; dividing the target candidate texts corresponding to each replacement strategy into a plurality of groups of candidate texts according to the replacement positions of second target words in the source text, and calculating the mean value of the similarity corresponding to each group of candidate texts; sequencing the candidate texts corresponding to each replacement strategy according to the sequence of the mean value from high to low, and analyzing the contact ratio of the source texts corresponding to each replacement strategy according to the sequencing result; and determining the translation performance index of the machine translation model according to the contact ratio.

In an embodiment, the source text includes a plurality of source texts, and the detection unit 305 may specifically be configured to: respectively screening candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to a plurality of candidate texts corresponding to each source text to obtain a target candidate text corresponding to each source text; dividing the target candidate texts corresponding to each source text into a plurality of groups of candidate texts according to the replacement positions of second target words in the source text, and calculating the mean value of the similarity corresponding to each group of candidate texts; calculating the variance value of the similarity corresponding to the candidate texts in the candidate text group with the maximum average value; and determining the translation performance index of the machine translation model according to the variance value.

In one embodiment, the apparatus for detecting the performance of the machine translation model may further include:

the adjusting unit is used for adjusting the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model;

and the translation unit is used for translating the text to be translated through the adjusted machine translation model.

In the embodiment of the application, the obtaining unit 301 may obtain a source text of a first language type, obtain a target text of a second language type corresponding to the source text, and the replacing unit 302 may replace a second target word in the source text with a plurality of first target words of the first language type, respectively, to obtain a plurality of candidate texts; then, the first translation unit 303 may translate the multiple candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text, and the second translation unit 304 may translate the source text through the machine translation model to obtain a second translation text of the second language type; at this time, the translation performance index of the machine translation model may be detected by the detection unit 305 according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text. According to the scheme, the candidate text and the source text obtained after word replacement are translated through the machine translation model, and statistical analysis is carried out on the second similarity between the second translation text and the target text based on the first similarity between the first translation text and the target text, so that the translation performance index of the machine translation model is accurately detected, and the accuracy of detecting the performance of the machine translation model is improved.

An embodiment of the present application further provides a computer device, where the computer device may be a computer device, as shown in fig. 13, which shows a schematic structural diagram of the computer device according to the embodiment of the present application, specifically:

the computer device may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the computer device configuration illustrated in FIG. 13 does not constitute a limitation of computer devices, and may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components. Wherein:

the processor 401 is a control center of the computer device, connects various parts of the entire computer device using various interfaces and lines, and performs various functions of the computer device and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the computer device as a whole. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.

The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the computer device, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.

The computer device further comprises a power supply 403 for supplying power to the various components, and preferably, the power supply 403 is logically connected to the processor 401 via a power management system, so that functions of managing charging, discharging, and power consumption are implemented via the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The computer device may also include an input unit 404, the input unit 404 being operable to receive input numeric or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the computer device may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 401 in the computer device loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application programs stored in the memory 402, thereby implementing various functions as follows:

acquiring a source text of a first language type, and acquiring a target text of a second language type corresponding to the source text; replacing second target words in the source text with a plurality of first target words of a first language type respectively to obtain a plurality of candidate texts; translating the candidate texts through a machine translation model to obtain a first translation text of a second language type corresponding to each candidate text; translating the source text through a machine translation model to obtain a second translation text of a second language type; and detecting the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations of the above embodiments.

It will be understood by those skilled in the art that all or part of the steps of the methods of the embodiments described above may be performed by computer instructions, or by computer instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, the present application provides a storage medium (i.e., a computer-readable storage medium) in which a computer program is stored, where the computer program may include computer instructions, and the computer program can be loaded by a processor to execute any one of the methods for detecting the performance of the machine translation model provided by the present application.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

Since the instructions stored in the storage medium may execute the steps in any one of the methods for detecting the performance of the machine translation model provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the methods for detecting the performance of the machine translation model provided in the embodiments of the present application may be achieved, which are detailed in the foregoing embodiments and will not be described herein again.

The above detailed description is given to a method for detecting performance of a machine translation model and related devices provided in the embodiments of the present application, and specific examples are applied in the present application to explain the principles and embodiments of the present application, and the description of the above embodiments is only used to help understanding the method and core ideas of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method for detecting the performance of a machine translation model is characterized by comprising the following steps:

2. The method of claim 1, wherein the replacing a second target word in the source text with a plurality of first target words of the first language type respectively to obtain a plurality of candidate texts comprises:

calculating a gradient value corresponding to a second target word in the source text through a loss function of the machine translation model based on the source text and the target text;

sampling a plurality of first target words from the word list of the first language type according to the gradient value;

and respectively replacing second target words in the source text with the plurality of first target words to obtain a plurality of candidate texts.

3. The method of claim 2, wherein sampling a plurality of first target words from a vocabulary of the first language type based on the gradient values comprises:

updating the word vector of the second target word in the source text according to the gradient value and a preset learning rate to obtain an updated word vector;

performing dot product operation on the updated word vector and a word vector matrix corresponding to the vocabulary of the first language type to obtain a similarity vector between a word vector of a second target word in the source text and each word vector in the word vector matrix;

and sampling a plurality of first target words with similarity greater than a preset similarity threshold value with a second target word in the source text from the word list of the first language type according to the similarity vector.

4. The method of claim 1, wherein the replacing a second target word in the source text with a plurality of first target words of the first language type respectively to obtain a plurality of candidate texts comprises:

screening a plurality of candidate words from the word list of the first language type;

replacing second target words in the source text with the candidate words respectively to obtain a plurality of replacement texts;

calculating a loss value between the replacement text and the target text through a loss function of the machine translation model;

screening out a preset number of candidate words corresponding to the minimum loss value to obtain a plurality of first target words;

5. The method of claim 1, wherein the replacing a second target word in the source text with a plurality of first target words of the first language type respectively to obtain a plurality of candidate texts comprises:

determining an alternative location in the source text;

randomly screening a plurality of first target words from the word list of the first language type;

and respectively replacing the second target words corresponding to the replacement positions in the source text with the plurality of first target words to obtain a plurality of candidate texts.

6. The method of claim 1, wherein the replacing a second target word in the source text with a plurality of first target words of the first language type respectively to obtain a plurality of candidate texts comprises:

determining attribute information of a second target word in the source text;

screening a plurality of first target words from the word list of the first language type according to the attribute information;

7. The method of claim 1, wherein detecting the translation performance indicator of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text comprises:

acquiring a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text;

screening candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to the candidate texts to obtain target candidate texts;

analyzing part-of-speech distribution of a second target word in the source text according to the target candidate text;

and determining the translation performance index of the machine translation model according to the part of speech distribution.

8. The method of claim 7, wherein analyzing the part-of-speech distribution of the second target word in the source text according to the target candidate text comprises:

dividing the target candidate texts into a plurality of groups of candidate texts according to the replacement positions of second target words in the source texts;

calculating the mean value of the similarity corresponding to each group of candidate texts;

and determining the part-of-speech distribution of the second target words in the source text according to the candidate text of the previous preset group with the largest average value.

9. The method of claim 7, wherein the obtaining a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text comprises:

acquiring a first initial similarity between the first translation text and the target text and a second initial similarity between the second translation text and the target text through the machine translation model;

calculating a first translation score of the candidate text and a second translation score of the source text using a reordering algorithm;

and adjusting the first initial similarity according to the first translation score to obtain a first similarity between the first translation text and the target text, and adjusting the second initial similarity according to the second translation score to obtain a second similarity between the second translation text and the target text.

10. The method of claim 1, wherein the replacing a second target word in the source text with a plurality of first target words of the first language type respectively to obtain a plurality of candidate texts comprises:

replacing words in the source text by using a plurality of first target words of the first language type according to different replacement strategies to obtain a plurality of candidate texts corresponding to each replacement strategy;

the detecting a translation performance index of the machine translation model according to a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text comprises:

respectively screening out candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to a plurality of candidate texts obtained by replacement based on each replacement strategy to obtain target candidate texts corresponding to each replacement strategy;

dividing the target candidate texts corresponding to each replacement strategy into a plurality of groups of candidate texts according to the replacement positions of second target words in the source texts, and calculating the mean value of the similarity corresponding to each group of candidate texts;

sequencing the candidate texts corresponding to each replacement strategy according to the sequence of the mean value from high to low, and analyzing the contact ratio of the source texts corresponding to each replacement strategy according to the sequencing result;

and determining the translation performance index of the machine translation model according to the contact ratio.

11. The method of claim 1, wherein the source text comprises a plurality of source texts, and the detecting the translation performance indicator of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text comprises:

respectively screening candidate texts with the first similarity larger than the second similarity from first translation texts corresponding to a plurality of candidate texts corresponding to each source text to obtain a target candidate text corresponding to each source text;

dividing the target candidate texts corresponding to each source text into a plurality of groups of candidate texts according to the replacement positions of second target words in the source text, and calculating the mean value of the similarity corresponding to each group of candidate texts;

calculating the variance value of the similarity corresponding to the candidate texts in the candidate text group with the maximum average value;

and determining the translation performance index of the machine translation model according to the variance value.

12. The method according to any one of claims 1 to 11, wherein after detecting the translation performance indicator of the machine translation model according to the first similarity between the first translated text and the target text and the second similarity between the second translated text and the target text, the method further comprises:

adjusting the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model;

and translating the text to be translated through the adjusted machine translation model.

13. A machine translation model performance detection apparatus, comprising:

14. A computer device comprising a processor and a memory, the memory having stored therein a computer program, the processor when calling the computer program in the memory performing the method of detecting machine translation model performance of any of claims 1 to 12.

15. A storage medium for storing a computer program which is loaded by a processor to perform the method of detecting the performance of a machine translation model according to any one of claims 1 to 12.