CN111797639B - Machine translation quality assessment method and system - Google Patents
Machine translation quality assessment method and system Download PDFInfo
- Publication number
- CN111797639B CN111797639B CN202010601578.8A CN202010601578A CN111797639B CN 111797639 B CN111797639 B CN 111797639B CN 202010601578 A CN202010601578 A CN 202010601578A CN 111797639 B CN111797639 B CN 111797639B
- Authority
- CN
- China
- Prior art keywords
- word
- translation
- vectors
- candidate
- word vector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013519 translation Methods 0.000 title claims abstract description 367
- 238000001303 quality assessment method Methods 0.000 title claims abstract description 6
- 239000013598 vector Substances 0.000 claims abstract description 376
- 230000014616 translation Effects 0.000 claims abstract description 366
- 238000011156 evaluation Methods 0.000 claims abstract description 44
- 238000000034 method Methods 0.000 claims abstract description 22
- 230000011218 segmentation Effects 0.000 claims description 30
- 238000012545 processing Methods 0.000 claims description 23
- 238000013507 mapping Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 9
- 238000013441 quality evaluation Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 5
- 230000014509 gene expression Effects 0.000 abstract description 4
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 238000003058 natural language processing Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241000408659 Darpa Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/51—Translation evaluation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
Abstract
The embodiment of the invention provides a machine translation quality assessment method and a system, wherein the method comprises the following steps: dividing the candidate translations into words, obtaining a candidate translation word list, and carrying out word embedding to obtain a candidate translation word vector list; dividing the reference translation into words, obtaining a reference translation word list, and embedding the words to obtain a reference translation word vector list; acquiring cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting a maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table; obtaining a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and obtaining a final evaluation score according to the length penalty factor and the average cosine value; the method can consider the conditions of synonyms or similar expressions in evaluation, can well process sentences with rich forms, and improves the evaluation standard of translation quality.
Description
Technical Field
The present invention relates to the field of natural language processing, and in particular, to a method and system for evaluating machine translation quality.
Background
In recent years, in the field of natural language research, the evaluation problem is becoming more and more important, and it can be said that the evaluation is the most core and key part in the whole natural language field. The machine translation evaluation has great significance to the research and development of machine translation, the developer of the machine translation system can be improved continuously by evaluating the problems existing in the system, the user can select products meeting the requirements according to the evaluation report, and the evaluation can provide the most reliable basis for the technical development direction of the researchers of the machine translation.
As early as the 90 s, the international language engineering standards (ISLE) program sponsored by the national natural foundation committee and the european union was exclusively set up for the EWG (Evaluation Working Group) machine translation evaluation working group. Between 1992 and 1994, the advanced research planning agency (DARPA) of the united states department of defense specially organized a group of experts performed extensive evaluations of the machine translation systems of the current law, solar and west in terms of fidelity, fluency and information content of the translated version.
The current popular automatic evaluation method is the BLEU algorithm, BLEU (bilingual evaluation understudy), proposed by IBM, which is simply the idea that the closer the translation of the machine translation is to the result of the manual translation, the higher the translation quality thereof. The evaluation algorithm defines how the similarity between the machine translated version and the reference version is defined.
However, since the BLEU algorithm essentially calculates co-occurrence word frequencies of two sentences, the degree of word consistency in the two sentences is too pursued in the calculation process, so that the situation that synonyms or similar expressions are not considered in the evaluation is not considered, the sentences with rich morphology cannot be processed very well, and some reasonable translations are negated.
Disclosure of Invention
The embodiment of the invention provides a machine translation quality evaluation method and a machine translation quality evaluation system, which are used for solving the defects that synonyms or similar expressions are not considered in the prior art, sentences with rich morphology cannot be well received, and some reasonable translations are negated.
In a first aspect, an embodiment of the present invention provides a machine translation quality assessment method, including:
performing word segmentation on the candidate translations to obtain candidate translation word lists, and performing word embedding on the candidate translation word lists through word mapping to obtain candidate translation word vector lists;
performing word segmentation on the reference translation to obtain a reference translation word list, and performing word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list;
respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting the maximum value in the cosine values as the maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table;
and acquiring a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and acquiring a final evaluation score according to the length penalty factor and the average cosine value.
Optionally, the acquiring cosine values of included angles between each word vector in the candidate translated word vector table and each word vector in the reference translated word vector table respectively, and selecting the cosine value of the phrase with the maximum value being the maximum value specifically includes:
obtaining the sum vectors of the adjacent candidate translation word vectors less than or equal to the preset number N in the candidate translation word vector table, and obtaining the sum vectors of the adjacent reference translation word vectors less than or equal to the preset number N in the reference translation word vector table;
and calculating cosine values of included angles between the sum vectors of the candidate translation word vectors and the sum vectors of the reference translation word vectors, and selecting the maximum value in the cosine values as the cosine value of the maximum word phrase, wherein the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors.
Optionally, the calculating the length penalty factor according to the lengths of the candidate translated word vector table and the reference translated word vector table according to a preset algorithm specifically includes:
when the length of the candidate translation word list is larger than that of the reference translation word list, the length penalty factor is 1;
when the length of the reference translation vocabulary is greater than or equal to the length of the candidate translation vocabulary, the length penalty factor is an exponential function based on e, wherein the exponential term of the exponential function is the quotient of the difference of the length value of the candidate translation vocabulary minus the length value of the reference translation vocabulary divided by the length value of the candidate translation vocabulary.
Optionally, the obtaining the length penalty factor specifically includes:
wherein BP is a length penalty factor, len (cand) is the length of the candidate translation vocabulary, and len (ref) is the length of the reference translation vocabulary.
Optionally, the obtaining a final evaluation score according to the length penalty factor and the average cosine value specifically includes:
and multiplying the length penalty factor by the product of all average cosine values to obtain a final evaluation score.
In a second aspect, an embodiment of the present invention provides a machine translation quality evaluation system, including:
the first processing module is used for carrying out word segmentation on the candidate translations through splitting to obtain candidate translation word lists, carrying out word embedding on the candidate translation word lists through word mapping to obtain candidate translation word vector lists;
the second processing module is used for carrying out word segmentation on the reference translation through splitting to obtain a reference translation word list, carrying out word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list;
the third processing module is used for respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting a maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table;
and the evaluation module is used for calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and obtaining a final evaluation score according to the length penalty factor and the average cosine value.
Optionally, the third processing module is specifically configured to:
respectively calculating the sum vectors of the adjacent candidate translation word vectors smaller than or equal to the preset number N in the candidate translation word vector table and the sum vectors of the adjacent reference translation word vectors smaller than or equal to the preset number N in the reference translation word vector table;
calculating cosine values of included angles between sum vectors of the candidate translation word vectors and sum vectors of the reference translation word vectors, and selecting a maximum word phrase cosine value, wherein the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors;
and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table.
Optionally, the evaluation module is specifically configured to:
calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and multiplying the length penalty factor by the product of all average cosine values to obtain a final evaluation score;
the length penalty factor specifically includes:
wherein BP is a length penalty factor, len (cand) is the length of the candidate translation vocabulary, and len (ref) is the length of the reference translation vocabulary.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
the device comprises a memory and a processor, wherein the processor and the memory are communicated with each other through a bus; the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of the first aspects.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the non-transitory computer readable storage medium stores computer instructions, which when executed by a processor, implement a method according to any of the first aspects.
According to the machine translation quality assessment method and system provided by the embodiment of the invention, the maximum word phrase cosine value of the included angle between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table is calculated, the average cosine value is obtained according to the number of word vectors in the candidate translation word vector table, the length penalty factor and the final assessment score are further obtained, the problem that synonyms or similar expressions are not considered in the assessment in the prior art is solved, sentences with rich forms can be well processed, and the assessment standard of translation quality is perfected.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a machine translation quality evaluation method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a machine translation quality evaluation system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, a machine translation quality evaluation method provided by an embodiment of the present invention includes:
performing word segmentation on the candidate translations to obtain candidate translation word lists, and performing word embedding on the candidate translation word lists through word mapping to obtain candidate translation word vector lists;
performing word segmentation on the reference translation to obtain a reference translation word list, and performing word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list;
respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting the maximum value in the cosine values as the maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table;
and acquiring a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and acquiring a final evaluation score according to the length penalty factor and the average cosine value.
Specifically, the original text refers to a sentence to be translated, the candidate translation refers to a translated sentence translated by a machine, and the reference translation refers to a sentence for comparison with the machine translation, and is generally a high-quality manual translation. Firstly, candidate translations are subjected to word segmentation processing, a candidate translation word list is obtained, word segmentation refers to the fact that sentences are split into words, and words are basic units of NLP processing. The word segmentation can adopt the prior art, for example, english word segmentation uses NLTK tool set, chinese word segmentation uses jieba and the like. Then, word embedding is carried out on the candidate translation word list to obtain a candidate translation word vector list, wherein word embedding refers to the technology of mapping words into vectors on a real number domain, and the technology of word2vec or multi-space word vector in the prior art can be adopted. And similarly, performing word segmentation on the reference translation to obtain a reference translation word list, and then performing word embedding on the reference translation to obtain a reference translation word vector list. For example, the candidate translation and the reference translation are subjected to word segmentation to obtain the word list of the candidate translation as [ cand_word1, cand_word2, cand_word3, ], and the word list of the reference translation as [ ref_word1, ref_word2, ref_word3, ]. And respectively embedding words into the word list of the reference translation and the word list of the candidate translation to obtain an orientation list, wherein a vector a represents the word vector list of the candidate translation, and a vector b represents the word vector list of the reference translation, so that the word vector list of the candidate translation is obtained as [ a1, a2, a3, the vector list of the candidate translation is [ b1, b2, b3, the vector list of the reference translation is [ b1, b2, b3, the vector list of the candidate translation ].
Further, calculating the cosine value of the maximum word phrase of the included angle between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, and calculating the average cosine value according to the number of word vectors in the candidate translation word vector table. The maximum word phrase cosine value is obtained by calculating the maximum value of the cosine values of the included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, namely calculating the cosine values of the included angles between each word vector in all the candidate translation word vector tables and each word vector in each reference translation word vector table, and selecting the maximum cosine value from the maximum cosine values as the maximum word phrase cosine value. And then, summing all the obtained maximum word phrase cosine values and dividing the sum by the number of word vectors in the candidate translation word vector table to obtain an average cosine value.
And finally, according to a preset formula for obtaining the length penalty factor, according to a preset algorithm, obtaining the length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, obtaining a final evaluation score according to the length penalty factor and the average cosine value, and evaluating the translation quality of the machine translation by using the final evaluation score.
As an embodiment of the present invention, the respectively obtaining cosine values of angles between each word vector in the candidate translated word vector table and each word vector in the reference translated word vector table, and selecting the cosine value of the phrase with the maximum value being the maximum value specifically includes:
obtaining the sum vectors of the adjacent candidate translation word vectors less than or equal to the preset number N in the candidate translation word vector table, and obtaining the sum vectors of the adjacent reference translation word vectors less than or equal to the preset number N in the reference translation word vector table;
and calculating cosine values of included angles between the sum vectors of the candidate translation word vectors and the sum vectors of the reference translation word vectors, and selecting the maximum value in the cosine values as the cosine value of the maximum word phrase, wherein the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors.
Specifically, calculating the maximum word phrase cosine value of an included angle between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table is by calculating the maximum word phrase cosine value of an included angle between the sum vector of adjacent candidate translation word vectors smaller than or equal to a preset number N in the candidate translation word vector table and the sum vector of adjacent reference translation word vectors smaller than or equal to the preset number N in the reference translation word vector table respectively, wherein N is the preset number, then calculating the maximum word phrase cosine value between the sum vector of the adjacent candidate translation word vectors smaller than or equal to N and the sum vector of the reference translation word vectors, and the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors.
For example, when N is equal to 4, calculating the maximum word phrase cosine value of the included angle between the sum vector of the candidate translated word vectors, which are adjacent to each other in the candidate translated word vector table and are smaller than or equal to the preset number 4, and the sum vector of the reference translated word vectors, which are adjacent to each other in the reference translated word vector table and are smaller than or equal to the preset number 4, wherein the specific calculation method is as follows:
representing a word vector table of candidate translations with vector a, a= [ a1, a2, a 3. ]; vector b represents a reference translation word vector table b= [ b1, b2, b3 > ]. And respectively calculating cosine values of included angles between the sum vectors of the candidate translation word vectors and the corresponding sum vectors of the reference translation word vectors when the sum vectors of the candidate translation word vectors are 1-4 word vectors, and solving the cosine values and average cosine values of the maximum word phrases:
the calculation process when the sum vector of the candidate translation word vectors is 1 word vector component:
N 1 ＝Max(Cos<a 1 ,b 1 >,Cos<a 1 ,b 2 >,...)
N 2 ＝Max(Cos<a 2 ,b 1 >,Cos<a 2 ,b 2 >,...)
...
wherein N1, N2..represents the maximum value of the cosine value of the angle between the word vector in each candidate translation word vector table and the word vector in the corresponding reference translation word vector table, i.e., the maximum word phrase cosine value; c1 represents the average cosine value when the sum vector of the candidate translation word vectors is 1 word vector component.
The calculation process when the sum vector of the candidate translation word vectors is 1 word vector component:
N 1 ＝Max(Cos<a 1 +a 2 ,b 1 +b 2 >,Cos<a 1 +a 2 ,b 2 +b 3 >,Cos<a 1 +a 2 ,b 3 +b 4 >...)
N 2 ＝Max(Cos<a 2 +a 3 ,b 1 +b 2 >,Cos<a 2 +a 3 ,b 2 +b 3 >,Cos<a 2 +a 3 ,b 3 +b 4 >...)
...
n1 and N2 represent the maximum value of the sum vector of the word vectors in the adjacent 2 candidate translation word vector tables and the sum vector angle cosine value of the word vectors in the corresponding 2 reference translation word vector tables, namely the maximum word phrase cosine value; c2 represents the average cosine value when the sum vector of the candidate translation word vectors is composed of 2 word vectors. Further, if the word length of the sentence is less than 2, the term is 0.
The calculation process when the sum vector of the candidate translation word vectors is composed of 2 word vectors:
N 1 ＝Max(Cos<a 1 +a 2 +a 3 ,b 1 +b 2 +b 3 >,Cos<a 1 +a 2 +a 2 ,b 2 +b 3 +b 4 >,Cos<a 1 +a 2 +a 2 ,b 3 +b 4 +b 5 >...)
N 2 ＝Max(Cos<a 2 +a 3 +a 4 ,b 1 +b 2 +b 3 >,Cos<a 2 +a 3 +a 4 ,b 2 +b 3 +b 4 >,Cos<a 2 +a 3 +a 4 ,b 3 +b 4 +b 5 >...)
...
n1 and N2 represent the maximum value of the sum vector of the word vectors in the adjacent 3 candidate translation word vector tables and the sum vector angle cosine value of the word vectors in the corresponding 3 reference translation word vector tables, namely the maximum word phrase cosine value; c3 represents the average cosine value when the sum vector of the candidate translation word vectors is composed of 3 word vectors. Further, if the word length of the sentence is less than 3, the term is 0.
Similarly, an average cosine value C4 may be calculated when the sum vector of the candidate translation word vectors is composed of 4 word vectors.
As an embodiment of the present invention, the calculating the length penalty factor according to the length of the candidate translated word vector table and the reference translated word vector table according to the preset algorithm specifically includes:
when the length of the candidate translation word list is larger than that of the reference translation word list, the length penalty factor is 1;
when the length of the reference translation vocabulary is greater than or equal to the length of the candidate translation vocabulary, the length penalty factor is an exponential function based on e, wherein the exponential term of the exponential function is the quotient of the difference of the length value of the candidate translation vocabulary minus the length value of the reference translation vocabulary divided by the length value of the candidate translation vocabulary.
Specifically, in order to evaluate the reference translation more easily, the invention also introduces a length penalty factor, wherein the calculation formula of the length penalty factor BP is as follows:
when the length len (cand) of the candidate translation word list is larger than the length len (ref) of the reference translation word list, the length penalty factor is 1;
when the length len (ref) of the reference translation vocabulary is greater than or equal to the length len (cand) of the candidate translation vocabulary, the length penalty factor is an exponential function taking e as a base, and dividing the quotient of the difference of the length value of the candidate translation vocabulary minus the length value of the reference translation vocabulary by the length value of the candidate translation vocabulary.
As an embodiment of the present invention, the length penalty factor specifically includes:
wherein BP is a length penalty factor, len (cand) is the length of the candidate translation vocabulary, and len (ref) is the length of the reference translation vocabulary.
When the length len (ref) of the reference translation vocabulary is greater than or equal to the length len (cand) of the candidate translation vocabulary, the length penalty factor is an exponential function taking e as a base, and dividing the quotient of the difference of the length value of the candidate translation vocabulary minus the length value of the reference translation vocabulary by the length value of the candidate translation vocabulary.
As an embodiment of the present invention, the obtaining a final evaluation score according to the length penalty factor and the average cosine value specifically includes:
and multiplying the length penalty factor by the product of all average cosine values to obtain a final evaluation score.
Specifically, the final evaluation score was:
as shown in fig. 2, a machine translation quality evaluation system according to an embodiment of the present invention includes:
the first processing module is used for carrying out word segmentation on the candidate translations through splitting to obtain candidate translation word lists, carrying out word embedding on the candidate translation word lists through word mapping to obtain candidate translation word vector lists;
the second processing module is used for carrying out word segmentation on the reference translation through splitting to obtain a reference translation word list, carrying out word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list;
the third processing module is used for respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting a maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table;
and the evaluation module is used for calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and obtaining a final evaluation score according to the length penalty factor and the average cosine value.
Specifically, the original text refers to a sentence to be translated, the candidate translation refers to a translated sentence translated by a machine, and the reference translation refers to a sentence for comparison with the machine translation, and is generally a high-quality manual translation.
The first processing module and the second processing module are used for firstly performing word segmentation processing on the candidate translations to obtain a word list of the candidate translations, wherein word segmentation refers to the splitting of sentences into words, and the word is a basic unit of NLP processing. The word segmentation can adopt the prior art, for example, english word segmentation uses NLTK tool set, chinese word segmentation uses jieba and the like. Then, word embedding is carried out on the candidate translation word list to obtain a candidate translation word vector list, wherein word embedding refers to the technology of mapping words into vectors on a real number domain, and the technology of word2vec or multi-space word vector in the prior art can be adopted. And similarly, performing word segmentation on the reference translation to obtain a reference translation word list, and then performing word embedding on the reference translation to obtain a reference translation word vector list. For example, the candidate translation and the reference translation are subjected to word segmentation to obtain the word list of the candidate translation as [ cand_word1, cand_word2, cand_word3, ], and the word list of the reference translation as [ ref_word1, ref_word2, ref_word3, ]. And respectively embedding words into the word list of the reference translation and the word list of the candidate translation to obtain an orientation list, wherein a vector a represents the word vector list of the candidate translation, and a vector b represents the word vector list of the reference translation, so that the word vector list of the candidate translation is obtained as [ a1, a2, a3, the vector list of the candidate translation is [ b1, b2, b3, the vector list of the reference translation is [ b1, b2, b3, the vector list of the candidate translation ].
The third processing module is used for calculating the maximum word phrase cosine value of the included angle between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, and calculating the average cosine value according to the number of word vectors in the candidate translation word vector table. The maximum word phrase cosine value is obtained by calculating the maximum value of the cosine values of the included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, namely calculating the cosine values of the included angles between each word vector in all the candidate translation word vector tables and each word vector in each reference translation word vector table, and selecting the maximum cosine value from the maximum cosine values as the maximum word phrase cosine value. And then, summing all the obtained maximum word phrase cosine values and dividing the sum by the number of word vectors in the candidate translation word vector table to obtain an average cosine value.
And finally, the evaluation module is used for obtaining the length penalty factors according to a preset algorithm and the lengths of the candidate translation word vector table and the reference translation word vector table according to a preset formula for obtaining the length penalty factors, obtaining final evaluation scores according to the length penalty factors and the average cosine values, and evaluating the translation quality of the machine translation by using the final evaluation scores.
As an embodiment of the present invention, the third processing module is specifically configured to:
respectively calculating the sum vectors of the adjacent candidate translation word vectors smaller than or equal to the preset number N in the candidate translation word vector table and the sum vectors of the adjacent reference translation word vectors smaller than or equal to the preset number N in the reference translation word vector table;
calculating cosine values of included angles between sum vectors of the candidate translation word vectors and sum vectors of the reference translation word vectors, and selecting a maximum word phrase cosine value, wherein the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors;
and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table.
Specifically, calculating the maximum word phrase cosine value of an included angle between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table is by calculating the maximum word phrase cosine value of an included angle between the sum vector of adjacent candidate translation word vectors smaller than or equal to a preset number N in the candidate translation word vector table and the sum vector of adjacent reference translation word vectors smaller than or equal to the preset number N in the reference translation word vector table respectively, wherein N is the preset number, then calculating the maximum word phrase cosine value between the sum vector of the adjacent candidate translation word vectors smaller than or equal to N and the sum vector of the reference translation word vectors, and the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors.
As an embodiment of the present invention, the evaluation module is specifically configured to:
calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and multiplying the length penalty factor by the product of all average cosine values to obtain a final evaluation score;
the length penalty factor specifically includes:
wherein BP is a length penalty factor, len (cand) is the length of the candidate translation vocabulary, and len (ref) is the length of the reference translation vocabulary.
Specifically, when the length len (ref) of the reference translation vocabulary is greater than or equal to the length len (cand) of the candidate translation vocabulary, the length penalty factor is an exponential function with e as a base, and the quotient of the difference of the length value of the candidate translation vocabulary minus the length value of the reference translation vocabulary divided by the length value of the candidate translation vocabulary is used as an index.
As shown in fig. 3, an electronic device provided in an embodiment of the present invention may include: processor 301, communication interface (Communications Interface) 302, memory (memory) 303 and communication bus 304, wherein processor 301, communication interface 302, memory 303 accomplish the communication between each other through communication bus 304. The processor 301 may call logic instructions in the memory 303 to perform the following method: splitting the candidate translation to perform word segmentation processing to obtain a candidate translation word list, and performing word embedding on the candidate translation word list through word mapping to obtain a candidate translation word vector list; splitting the reference translation to perform word segmentation processing to obtain a reference translation word list, and performing word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list; respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting a maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table; and calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and obtaining a final evaluation score according to the length penalty factor and the average cosine value.
Further, the logic instructions in the memory 303 may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, embodiments of the present invention further provide a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the transmission method provided in the above embodiments, for example, including: splitting the candidate translation to perform word segmentation processing to obtain a candidate translation word list, and performing word embedding on the candidate translation word list through word mapping to obtain a candidate translation word vector list; splitting the reference translation to perform word segmentation processing to obtain a reference translation word list, and performing word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list; respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting a maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table; and calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and obtaining a final evaluation score according to the length penalty factor and the average cosine value.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A machine translation quality assessment method, comprising:
performing word segmentation on the candidate translations to obtain candidate translation word lists, and performing word embedding on the candidate translation word lists through word mapping to obtain candidate translation word vector lists;
performing word segmentation on the reference translation to obtain a reference translation word list, and performing word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list;
respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting the maximum value in the cosine values as the maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table;
acquiring a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and acquiring a final evaluation score according to the length penalty factor and the average cosine value;
the step of respectively obtaining cosine values of included angles between each word vector in the candidate translated word vector table and each word vector in the reference translated word vector table, and the step of selecting the cosine value of the phrase with the maximum value being the maximum value specifically comprises the following steps:
obtaining the sum vectors of the adjacent candidate translation word vectors less than or equal to the preset number N in the candidate translation word vector table, and obtaining the sum vectors of the adjacent reference translation word vectors less than or equal to the preset number N in the reference translation word vector table;
and calculating cosine values of included angles between the sum vectors of the candidate translation word vectors and the sum vectors of the reference translation word vectors, and selecting the maximum value in the cosine values as the cosine value of the maximum word phrase, wherein the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors.
2. The method of claim 1, wherein obtaining a length penalty factor based on the lengths of the candidate translation word vector table and the reference translation word vector table comprises:
when the length of the candidate translation word list is larger than that of the reference translation word list, the length penalty factor is 1;
when the length of the reference translation vocabulary is greater than or equal to the length of the candidate translation vocabulary, the length penalty factor is an exponential function based on e, wherein the exponential term of the exponential function is the quotient of the difference of the length value of the candidate translation vocabulary minus the length value of the reference translation vocabulary divided by the length value of the candidate translation vocabulary.
3. The method according to claim 2, wherein obtaining the length penalty factor comprises:
wherein BP is a length penalty factor, len (cand) is the length of the candidate translation vocabulary, and len (ref) is the length of the reference translation vocabulary.
4. A method according to claim 3, wherein said obtaining a final evaluation score from said length penalty factor and said average cosine value comprises:
and multiplying the length penalty factor by the product of all average cosine values to obtain a final evaluation score.
5. A machine translation quality evaluation system, comprising:
the first processing module is used for carrying out word segmentation on the candidate translations through splitting to obtain candidate translation word lists, carrying out word embedding on the candidate translation word lists through word mapping to obtain candidate translation word vector lists;
the second processing module is used for carrying out word segmentation on the reference translation through splitting to obtain a reference translation word list, carrying out word embedding on the reference translation word list through word mapping to obtain a reference translation word vector list;
the third processing module is used for respectively obtaining cosine values of included angles between each word vector in the candidate translation word vector table and each word vector in the reference translation word vector table, selecting a maximum word phrase cosine value, and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table;
the evaluation module is used for obtaining a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and obtaining a final evaluation score according to the length penalty factor and the average cosine value;
the third processing module is specifically configured to:
respectively calculating the sum vectors of the adjacent candidate translation word vectors smaller than or equal to the preset number N in the candidate translation word vector table and the sum vectors of the adjacent reference translation word vectors smaller than or equal to the preset number N in the reference translation word vector table;
calculating cosine values of included angles between sum vectors of the candidate translation word vectors and sum vectors of the reference translation word vectors, and selecting a maximum word phrase cosine value, wherein the number of component vectors of the sum vectors of the candidate translation word vectors is equal to the number of component vectors of the sum vectors of the reference translation word vectors;
and calculating an average cosine value according to the maximum word phrase cosine value and the number of word vectors in the candidate translation word vector table.
6. The system of claim 5, wherein the evaluation module is specifically configured to:
calculating a length penalty factor according to the lengths of the candidate translation word vector table and the reference translation word vector table, and multiplying the length penalty factor by the product of all average cosine values to obtain a final evaluation score;
the length penalty factor specifically includes:
wherein BP is a length penalty factor, len (cand) is the length of the candidate translation vocabulary, and len (ref) is the length of the reference translation vocabulary.
7. An electronic device, comprising: the device comprises a memory and a processor, wherein the processor and the memory are communicated with each other through a bus; the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of claims 1-4.
8. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the non-transitory computer readable storage medium stores computer instructions, which when executed by a processor, implement the method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010601578.8A CN111797639B (en) | 2020-06-28 | 2020-06-28 | Machine translation quality assessment method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010601578.8A CN111797639B (en) | 2020-06-28 | 2020-06-28 | Machine translation quality assessment method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111797639A CN111797639A (en) | 2020-10-20 |
| CN111797639B true CN111797639B (en) | 2024-03-26 |
Family
ID=72804711
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010601578.8A Active CN111797639B (en) | 2020-06-28 | 2020-06-28 | Machine translation quality assessment method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111797639B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710948A (en) * | 2018-11-27 | 2019-05-03 | 语联网（武汉）信息技术有限公司 | MT engine recommended method and device |
| CN110263349A (en) * | 2019-03-08 | 2019-09-20 | 腾讯科技（深圳）有限公司 | Corpus assessment models training method, device, storage medium and computer equipment |
| CN110674646A (en) * | 2019-09-06 | 2020-01-10 | 内蒙古工业大学 | Mongolian Chinese machine translation system based on byte pair encoding technology |
| CN111027331A (en) * | 2019-12-05 | 2020-04-17 | 百度在线网络技术（北京）有限公司 | Method and apparatus for evaluating translation quality |
-
2020
- 2020-06-28 CN CN202010601578.8A patent/CN111797639B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710948A (en) * | 2018-11-27 | 2019-05-03 | 语联网（武汉）信息技术有限公司 | MT engine recommended method and device |
| CN110263349A (en) * | 2019-03-08 | 2019-09-20 | 腾讯科技（深圳）有限公司 | Corpus assessment models training method, device, storage medium and computer equipment |
| CN110674646A (en) * | 2019-09-06 | 2020-01-10 | 内蒙古工业大学 | Mongolian Chinese machine translation system based on byte pair encoding technology |
| CN111027331A (en) * | 2019-12-05 | 2020-04-17 | 百度在线网络技术（北京）有限公司 | Method and apparatus for evaluating translation quality |
Non-Patent Citations (1)
| Title |
|---|
| BERTSCORE: EVALUATING TEXT GENERATION WITH BERT;Tianyi Zhang et al.;ICLR 2020;第1-4页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111797639A (en) | 2020-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110555095B (en) | Man-machine conversation method and device | |
| CN110543643B (en) | Training method and device of text translation model | |
| CN106610951A (en) | Improved text similarity solving algorithm based on semantic analysis | |
| CA2971884C (en) | Method and device for general machine translation engine-oriented individualized translation | |
| CN111191468A (en) | Term replacement method and device | |
| CN111859950A (en) | Method for automatically generating lecture notes | |
| CN111797639B (en) | Machine translation quality assessment method and system | |
| CN112287667A (en) | Text generation method and equipment | |
| CN110309513B (en) | Text dependency analysis method and device | |
| CN111178060A (en) | Korean word segmentation reduction method based on language model | |
| WO2022166267A1 (en) | Machine translation post-editing method and system | |
| CN114722153A (en) | Intention classification method and device | |
| CN114936567A (en) | Unsupervised machine translation quality estimation method and device based on knowledge distillation | |
| CN109522563B (en) | Method and device for automatically judging statement translation completion | |
| CN111191469B (en) | Large-scale corpus cleaning and aligning method and device | |
| JP2006004366A (en) | Machine translation system and computer program for it | |
| CN113761874A (en) | Event reality prediction method and device, electronic equipment and storage medium | |
| CN111125350B (en) | Method and device for generating LDA topic model based on bilingual parallel corpus | |
| CN115496079B (en) | Chinese translation method and device | |
| CN117035064B (en) | Combined training method for retrieving enhanced language model and storage medium | |
| de Souza et al. | Mt quality estimation for e-commerce data | |
| CN117312917A (en) | Short text classification method, device, electronic equipment and storage medium | |
| CN117313702A (en) | Short text characterization method, short text characterization device, electronic equipment and storage medium | |
| Pinnis | Terminology Integration in Statistical Machine Translation | |
| CN117875316A (en) | Multi-language long text abstract generation method and device based on large model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |