CN110598221B - Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network - Google Patents

Abstract

A method of constructing Mongolian-Chinese parallel corpus using generative adversarial network to improve the quality of Mongolian-Chinese translation. The generative adversarial network includes a generator and a discriminator. The generator uses a hybrid encoder to encode the source language sentence Mongolian into a vector representation, and uses a bidirectional Transformer-based method. The decoder combines the sparse attention mechanism to convert this representation into the target language sentence Chinese, thereby generating Mongolian sentences that are closer to human translation and more Mongolian-Chinese parallel corpora. In the discriminator, it is judged that the Chinese sentences generated by the generator are similar to The gap between human translations, the generator and the discriminator are trained against each other until the discriminator thinks that the Chinese sentences generated by the generator are very similar to the human translations, and a high-quality Mongolian-Chinese machine translation system and a large number of Mongolian-Chinese parallel data sets are obtained. The Mongolian-Chinese machine translation system is used for Mongolian-Chinese translation. The invention solves the problems of the serious shortage of Mongolian-Chinese parallel data sets and the inability of NMT to guarantee the naturalness, sufficiency and accuracy of translation results.

Description

Using Generative Adversarial Networks to Construct Mongolian-Chinese Parallel Corpus to Improve the Quality of Mongolian-Chinese Translation method

technical field

The invention belongs to the technical field of machine translation, and particularly relates to a method for constructing Mongolian-Chinese parallel corpus by using a generative confrontation network to improve the quality of Mongolian-Chinese translation.

Background technique

Machine translation can use computers to automatically translate one language into another language, and it is one of the most powerful means to solve the problem of language barriers. In recent years, many large search companies and service centers, such as Google and Baidu, have conducted large-scale research on machine translation, which has made important contributions to obtaining high-quality translations for machine translation. Therefore, the translation between large languages is close to that of human beings. Translation level, millions of people use online translation systems and mobile apps to communicate across language barriers. In the wave of deep learning in recent years, machine translation has become a top priority and has become an important part of promoting global communication.

A Seq2Seq-based neural machine translation framework consists of an encoder and a decoder, where the encoder reads an input sequence and outputs a single vector, and the decoder reads this vector to produce an output sequence. Since 2013, the framework has grown rapidly, achieving significant improvements in translation quality relative to statistical machine translation. The principle of sentence-level maximum likelihood estimation, gating units in LSTM and GRU, and the addition of attention mechanism have improved the ability of NMT to translate long sentences. In 2017, Ashish Vaswani et al. proposed the Transformer architecture, an architecture that completely relies on an attention mechanism to map the global dependencies between input and output. The advantage of doing this is to realize parallel computing, effectively reduce the training time of the model, and improve the quality of the machine translation model to a certain extent. It avoids the shortcomings of RNN and its derived networks that are slow and cannot be parallelized.

At present, neural machine translation has been very successful, but the best NMT system is still far from people's expectations, and the translation quality needs to be improved. Because NMT usually uses maximum likelihood estimation to train the model, that is, to maximize the probability of the target real sentence conditioned on the source sentence, that is: the model can generate the best candidate word for the current, but the translation of the entire sentence in the long run will not Not the best translation, which leaves a pitfall for NMT. Even the mighty Transformer is no exception. Such a goal does not guarantee the naturalness, adequacy and accuracy of the translation results compared to real human translations.

In addition, the mutual translation between large languages is relatively mature, but the machine translation between small languages is still unsatisfactory due to various challenges, especially the serious lack of corpora, and the cost of manually constructing parallel corpora is very expensive.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a method for constructing Mongolian-Chinese parallel corpus by using generative adversarial network to improve the quality of Mongolian-Chinese translation. Questions such as the naturalness, adequacy and accuracy of the results, the application of generative adversarial networks in Mongolian-Chinese neural machine translation.

In order to achieve the above object, the technical scheme adopted in the present invention is:

A method of using generative adversarial network to construct Mongolian-Chinese parallel corpus to improve the quality of Mongolian-Chinese translation, using generative adversarial network in Mongolian-Chinese machine translation to alleviate the problem of low quality of Mongolian-Chinese machine translation caused by the lack of Mongolian-Chinese parallel corpus and the minimum The difference between human translation and the translation given by the NMT model, the generative adversarial network mainly includes a generator and a discriminator. The addition of the generator can effectively use the Mongolian-Chinese monolingual data and alleviate the Mongolian-Chinese parallel in the machine translation task. The problem of lack of corpus. In the generator, in order to alleviate the UNK phenomenon in Mongolian-Chinese machine translation, a hybrid encoder is used to encode the source language sentence Mongolian into a vector representation, and a bidirectional Transformer-based decoder combined with a sparse attention mechanism is used to transform the vector representation Become the target language sentence Chinese, so as to generate Mongolian sentences that are closer to human translation and more Mongolian-Chinese parallel corpus, and improve the quality and efficiency of Mongolian-Chinese machine translation. In the discriminator, the difference between the Chinese sentence generated by the generator and the human translation is judged. The goal of the generator is to generate Mongolian sentences that are closer to human translation and to effectively use Mongolian-Chinese monolingual data to generate more Mongolian-Chinese parallel corpus, and the purpose of the discriminator is to calculate the gap between the Chinese sentences generated by the generator and the Chinese sentences translated by humans. The generator and the discriminator are trained against each other until the discriminator thinks that the Chinese sentences generated by the generator are very similar to human translations, that is, when the generator and the discriminator achieve Nash equilibrium, a high-quality Mongolian-Chinese machine translation system and a large number of Mongolian-Chinese parallel dataset, using the Mongolian-Chinese machine translation system for Mongolian-Chinese translation.

The hybrid encoder consists of a sentence encoder and a word encoder. In order to capture the semantic information between sentences and the efficiency of the encoder, the sentence encoder consists of a bidirectional Transformer, and the word encoder uses a bidirectional LSTM to ensure the quality of the encoder. This improves the efficiency of the word encoder. In order to optimize Transformer1, the two-way Transformer firstly adds a gated linear unit on the basis of the original Transformer to effectively obtain important information in the source language sentence and discard redundant information; secondly, a branch structure is added to effectively capture the source language sentence. Finally, a capsule network is added to the branch structure and after the third layer standardization, so that the encoder can capture the exact position of the words in the source language sentence, further strengthen the accuracy of the encoder and improve the In the decoder, in order to optimize Transformer2, the bidirectional Transformer firstly added a branch structure on the basis of the original Transformer; secondly, the capsule network was added; finally, the Swish activation function was added to effectively improve the decoding accuracy of the decoder. .

The word encoder and the sentence encoder successively encode the source language sentences, and then fuse them through a fusion function to obtain a vector representation with context information. The vector representation of the Mongolian sentence of the basic unit, and its model formula is:

h1 _i =Φ(h1 _{i -1} , Wi )

Among them, Φ is the activation function, Wi is the weight, and h1 _{i -1} is the hidden layer state of the i-1th word.

The sentence encoder expresses an entire Mongolian sentence in the form of a vector, and constructs a vector representation with the sentence as the basic unit. The model formula is:

的计算公式如下：Among them, v ^j represents the value of the jth word (Value),

The calculation formula is as follows:

Among them, the calculation of α _i , _j is as follows:

Among them, qi is the query of the ⁱ -th word, k ^j is the key of the j-th word, · represents the dot product operation, and d represents the dimensions of q and k;

The fusion function is as follows:

ψ(h1 _i , h2 _i )=a ₁ h1 _i +a ₂ h2 _i

Among them, ψ is the fusion function, a ₁ , a ₂ represent the corresponding weights of the two encoders through random initialization, and the two kinds of encodings are fused into an encoder containing two kinds of vector information of sentences and words.

In the sentence encoder, the bidirectional Transformer refers to reading the entire text sequence at one time, that is, learning based on both sides of the sentence, rather than reading sequentially from left to right or right to left, so that words in the text can be learned. contextual relationship between them.

In the decoder, the bidirectional Transformer refers to reading the vector representation of the source language sentence at one time, that is, decoding is performed based on both sides of the vector representation of the entire sentence, so as to further improve the decoding accuracy of the decoder.

In order to enhance the discriminator's discriminant ability, the discriminator is a multi-scale discriminator, which can identify the general sentence meaning and detailed information (such as phrases and words, etc.) of the Chinese sentences generated by the generator, so as to assist the generator to generate translations that are closer to the real ones. At the same time, in order to overcome the translation invariance of the convolutional neural network, the multi-scale discriminator is implemented by using a capsule network, which can effectively improve the discriminator's discrimination ability without reducing the training efficiency. Degeneration means: for example, in face recognition, the convolutional neural network considers a face with features such as eyes and a mouth to be a human face, ignoring the specific location of the facial features in the face. If it is used as a discriminator in a generative adversarial network, because of its translation invariance, it will be considered that as long as the Chinese sentence generated by the generator has all the words in the human-translated sentence, it is a human-translated sentence. However, the location information of the missing word is ignored, resulting in a misidentification. Capsule network includes convolution layer, main capsule layer, convolution capsule layer and fully connected capsule; in order to use one network to represent multiple discriminators and improve training efficiency, in the convolution layer, the activation values of different sub-layers represent sentences of different granularity The activation value of the lower layer represents the activation value of the word, and the activation value of the high layer represents the activation value of the entire sentence. Finally, the feature maps of different layers are transformed into the same-scale feature map with a channel number of 1.

Given a sentence pair (x, y), each capsule network first constructs a 2D image-like representation by concatenating the embedding vectors of words in x and y, i.e. for the ith word x _i in the source language sentence x and the target language The jth word y _j in sentence y has the following correspondence:

表示源语言中第i个词x_i和目标语言中第j个词y_j构成的矩阵，即虚拟2D图像表示；Where: x _i ^T represents the transpose of x _i , y _j ^T represents the transpose of y _j ,

Represents the matrix formed by the i-th word x _i in the source language and the j-th word y _j in the target language, that is, a virtual 2D image representation;

Based on the virtual 2D image representation, the convolution layer, main capsule layer, convolution capsule layer, and fully connected capsule layer of the capsule network are sequentially passed to capture the sentence y translated by the generator under the condition of the source language sentence x. ' and the degree of similarity between the human-translated sentence y.

The virtual 2D image representation goes through the convolution layer, main capsule layer, convolution capsule layer and fully connected capsule layer of the capsule network in sequence as follows:

(1) After the convolution layer, a convolution operation with a stride of 1 and a convolution kernel of 9 × 9 is first performed, and the correspondence between the sentences in x and y is captured by the following feature map.

为第一次卷积运算的权重，

表示源语言中第i个词x_i和目标语言中第j个词y_j构成的矩阵，b^(1,f)为第一次卷积运算的偏置；Among them, f is the activation function of the first convolution operation,

is the weight of the first convolution operation,

represents the matrix formed by the i-th word x _i in the source language and the j-th word y _j in the target language, and b ^{(1, f)} is the bias of the first convolution operation;

Then, a convolution operation with a stride of 1 and a convolution kernel of 3 × 3 is performed, and the correspondence between the words in x and y is captured by the following feature map.

为第二次卷积运算的权重，f、

b^(1,f)与第一次卷积算法中的相同；in,

is the weight of the second convolution operation, f,

b ^(1,f) is the same as in the first convolution algorithm;

和

接着对较小的特征图

进行填充使得两个特征图大小相同，然后用对两个相同大小的特征图求平均的方法得到最终的特征图，如下式所示：After two convolution operations, two feature maps of different sizes are obtained respectively.

and

Then for smaller feature maps

Filling is performed to make the two feature maps of the same size, and then the final feature map is obtained by averaging the two feature maps of the same size, as shown in the following formula:

(2) Enter the main capsule layer, that is, the first capsule layer, and perform the following calculation on the output of the convolution layer

p=g(W ^b M+b ₁ )

where g is the nonlinear squashing function through the entire vector, M represents the input of the capsule, b ₁ is the bias of the capsule, and W ^b is the weight; in the main capsule layer, the capsule replaces the scalar output of the convolution operation with vector output;

(3) The maximum pooling is replaced by the dynamic routing algorithm, and the weight is dynamically strengthened or weakened to obtain effective features;

(4) Enter the convolution capsule layer; after this layer, all capsules in the previous layer are changed to a series of capsules, and the weights are further dynamically strengthened or weakened by the dynamic routing algorithm to obtain more effective features;

(5) Enter the fully connected capsule layer, and connect all the extracted features;

(6) Input all the features into the multi-layer perceptron, and use the activation function to obtain the probability that the data set (x, y') generated by the generator is the real data (x, y), that is, the similarity with the real sentence.

The final training objective of the generative adversarial network is:

表示求能使损失函数V(D,G)达到最大的D并且使损失函数达到最小的G，E表示期望，P_data(x,y)表示将平行语料库中的源语言x和目标语言y输入到鉴别器D中，鉴别器认为其为人工翻译的概率，G(y'|x)表示将平行语料库中的源语言x和G生成的目标语言y'输入到鉴别器D中，鉴别器认为其为人工翻译的概率，x表示平行语料库中的蒙古语句子，即源语言句子，y表示平行语料库中的汉语句子，即人工翻译结果，y'表示生成器生成的汉语句子，即生成器的翻译结果。Where: G represents the generator, D represents the discriminator, V(D, G) represents the loss function of the generator G and the discriminator D,

Indicates to find the D that can maximize the loss function V(D, G) and the G that minimizes the loss function, E represents the expectation, and P _data (x, y) represents the source language x and target language y in the parallel corpus input. In the discriminator D, the discriminator considers it as the probability of human translation, G(y'|x) indicates that the source language x in the parallel corpus and the target language y' generated by G are input into the discriminator D, the discriminator considers that It is the probability of human translation, x represents the Mongolian sentence in the parallel corpus, that is, the source language sentence, y represents the Chinese sentence in the parallel corpus, that is, the artificial translation result, y' represents the Chinese sentence generated by the generator, that is, the generator's translation result.

During the process:

The additional components of the Mongolian corpus are processed as follows:

Remove the control characters and the additional elements of the case together in Mongolian sentences, leaving only the stem part;

The Mongolian corpus is divided into different granularities, and the Chinese is word-segmented to alleviate the UNK phenomenon in the Mongolian-Chinese machine translation, and the quality of the Mongolian-Chinese machine translation is further improved by processing the additional components of the Mongolian style.

The segmentation method is as follows:

(1) First, the corpus to be preprocessed is divided into the smallest unit. For Mongolian, the smallest unit is the Mongolian alphabet.

(2) Then count the number of occurrences of all adjacent smallest unit combinations in the corpus and sort them, find out the combinations with the highest frequency, and add these combinations to the dictionary and delete the words with the lowest frequency in the dictionary to make the dictionary more frequent. The size remains the same.

(3) Repeat steps (1) and (2) until the frequency of the words in the dictionary appearing in their own corpus is higher than the set value;

Neural machine translation (NMT) has achieved good translation results in an end-to-end framework, but the best NMT systems are far from people's expectations. Compared with this, the present invention can minimize the difference between human translation and the translation given by the NMT model, alleviate the data sparse problem in Mongolian-Chinese machine translation, and alleviate the UNK phenomenon in Mongolian-Chinese machine translation, and not only obtain high-quality The Mongolian-Chinese machine translation system has also obtained a large number of Mongolian-Chinese parallel datasets.

Description of drawings

Figure 1 shows the optimized Transformer1 architecture.

Figure 2 shows the optimized Transformer2 architecture.

Figure 3 shows the generator framework.

Figure 4 shows the discriminator framework.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

The present invention utilizes the generative confrontation network to construct the Mongolian-Chinese parallel corpus to improve the Mongolian-Chinese translation quality, which mainly includes the construction of the encoder and the decoder and the construction of the discriminator model.

Figure 1 shows the optimized Transformer1 architecture. On the basis of the original Transformer, a gated linear unit is firstly added to effectively obtain important information in the source language sentences and discard redundant information; secondly, a branch structure is added, which can effectively capture the diverse semantic information between source language sentences ; Finally, a capsule network is added to the branch structure and after the third layer is normalized, so that the encoder can capture the exact position of the words in the source language sentence, further enhancing the accuracy of the encoder.

Figure 2 shows the optimized Transformer2 architecture. On the basis of the original Transformer, the branch structure is firstly added; secondly, the capsule network is added; finally, the Swish activation function is added. The addition of this activation function can effectively improve the decoding accuracy of the decoder.

首先基于双向优化Transformer将整个句子进行双向编码，同时基于双向LSTM的编码器对其中的单词进行编码，然后使用一个融合函数将两种编码器的表示进行融合，生成源语言的编码表示。接着解码器结合稀疏注意力机制将源语言编码表示解码为目标语言汉语句子“明天要下雨”。Figure 3 shows the generator framework. It is mainly composed of three parts: hybrid encoder, sparse attention and decoder. The encoder accepts input Mongolian sentences such as:

First, the entire sentence is encoded bidirectionally based on the bidirectional optimization Transformer, while the bidirectional LSTM-based encoder encodes the words in it, and then a fusion function is used to fuse the representations of the two encoders to generate the encoded representation of the source language. Then the decoder combines the sparse attention mechanism to decode the source language encoded representation into the target language Chinese sentence "it will rain tomorrow".

Figure 4 shows the framework of the capsule network, including the convolution layer, the main capsule layer, the convolution capsule layer, the fully connected capsule layer, and so on. The convolutional layer includes two layers, one layer captures sentence-level features, and the other layer captures word-level features to realize the multi-scale discrimination function.

Mitigation of data sparsity in Mongolian-Chinese machine translation:

The addition of the generator in the generative adversarial network can effectively alleviate the current data sparse problem in Mongolian-Chinese machine translation. Specifically, firstly, the generator is pre-trained through the Mongolian-Chinese aligned corpus, and after the pre-training model is obtained, the model is used to use the Mongolian monolingual data generates Mongolian-Chinese pseudo-bilingual data, and then, with the assistance of the discriminator, generates Chinese sentences that are closer to human translation, forming a Mongolian-Chinese aligned corpus.

The accuracy and naturalness of Mongolian-Chinese machine translation translation:

The Chinese sentences generated by the generator are often blunt and unnatural. In this invention, the discriminator is equivalent to the generator's teacher, assisting the generator to generate more natural and accurate Chinese sentences. The multi-scale discriminator indicates that the teacher has the ability to judge from many aspects whether the sentences generated by the generator are similar to the human translation.

Decision variable: Mongolian sentence x is input on the encoder side of the generator, and the corresponding machine-translated Chinese sentence y is output on the decoder side of the generator. The Mongolian sentence x, the corresponding human-translated Chinese sentence y, and the corresponding generator-translated Chinese sentence y' are input at the input of the discriminator

The present invention includes the following parts:

1. Mongolian-Chinese neural machine translation system model based on generative adversarial network, including the following parts:

A. The description of the hybrid encoder in the generator of the Mongolian-Chinese neural machine translation system based on the generative adversarial network: the hybrid encoder is composed of a sentence encoder and a word encoder. The word encoder and the sentence encoder successively encode the source language sentences. , where the word encoder represents each word in the form of a vector, and constructs a vector representation of Mongolian sentences with words as the basic unit. The model formula is:

h1 _i =Φ(h1 _{i -1} , Wi )

Among them, Φ is the activation function, Wi is the weight, and h1 _{i -1} is the hidden layer state of the i-1th word.

The sentence encoder expresses a whole Mongolian sentence in the form of a vector, and constructs a vector representation with the sentence as the basic unit. In the sentence encoder, the two-way optimization Transformer1 reads the entire text sequence at one time, that is, learning based on both sides of the sentence, so that the context relationship between sentences can be learned and parallelized. Its model formula is:

的计算公式如下：Among them, v ^j represents the Value of the jth word,

The calculation formula is as follows:

Among them, the calculation of α _i,j is as follows:

Among them, qi is the query (query) of the ⁱ -th word, k ^j is the key of the j-th word, · represents the dot product operation, and d represents the dimensions of q and k.

Finally, the fusion function is used to obtain a vector representation encoded with context information. The fusion function is as follows:

ψ(h1 _i , h2 _i )=a ₁ h1 _i +a ₂ h2 _i

Among them, ψ is the fusion function, a ₁ , a ₂ represent the corresponding weights of the two encoders through random initialization, and are fused into an encoder containing two kinds of vector information of sentences and words through two kinds of granularity encoding.

B. The description of the decoder in the generator of the Mongolian-Chinese neural machine translation system based on the generative adversarial network: The decoder consists of a bidirectionally optimized Transformer2. The optimized Transformer2 is basically similar to the optimized Transformer1 in the encoder. The difference is that this part of the optimized Transformer Added Swish activation function. In the decoding process, the decoder combines the sparse attention mechanism to decode the vector representation of the source language sentence into the target language sentence.

C. Description of the discriminator of Mongolian-Chinese neural machine translation system based on generative adversarial network: the discriminator is a multi-scale discriminator, and this structure can not only identify the general sentence meaning of the Chinese sentences generated by the generator, but also identify the The detailed information of Chinese sentences (such as phrases and words, etc.) can assist the generator to generate sentences that are closer to the real translation. The multi-scale discriminator is implemented using a capsule network, which includes a convolutional layer, a main capsule layer, a convolutional capsule layer, and a fully connected capsule. In order to use one network to represent multiple discriminators and improve training efficiency, in the convolutional layer, the activation values of different sub-layers represent the activation values of sentences of different granularities, the activation values of the lower layers represent the activation values of words, and the activation values of the upper layers represent the whole The activation value of the sentence, and finally transform the feature maps of different layers to the same-scale feature map with a channel number of 1. Specifically, given a sentence pair (x, y), each capsule network first constructs a 2D image-like representation by concatenating the embedding vectors of words in x and y, that is, for the ith word x _i in the source language sentence x There is the following correspondence with the jth word y _j in the target language sentence y:

表示源语言中第i个词x_i和目标语言中第j个词y_j构成的矩阵，即虚拟2D图像表示。Where: x _i ^T represents the transpose of x _i , y _j ^T represents the transpose of y _j ,

Represents the matrix formed by the i-th word x _i in the source language and the j-th word y _j in the target language, that is, a virtual 2D image representation.

Based on such a virtual 2D image representation, through the convolutional layer, main capsule layer, convolutional capsule layer, and fully connected capsule layer of the capsule network in order to capture the sentence y' translated by the generator under the condition of the source language sentence x Similarity to human-translated sentence y. The specific process is:

(1) After the convolution layer, a convolution operation with a stride of 1 and a convolution kernel of 9 × 9 is first performed, and the correspondence between the sentences in x and y is captured by the following feature map.

为第一次卷积运算的权重，

表示源语言中第i个词x_i和目标语言中第j个词y_j构成的矩阵，b^(1,f)为第一次卷积运算的偏置。Among them, f is the activation function of the first convolution operation,

is the weight of the first convolution operation,

Represents the matrix formed by the i-th word x _i in the source language and the j-th word y _j in the target language, and b ^{(1, f)} is the bias of the first convolution operation.

Then, a convolution operation with a stride of 1 and a convolution kernel of 3 × 3 is performed, and the correspondence between the words in x and y is captured by the following feature map.

为第二次卷积运算的权重，f、

b^(1,f)与第一次卷积算法中的相同。in,

is the weight of the second convolution operation, f,

b ^(1,f) is the same as in the first convolution algorithm.

和

接着对较小的特征图

进行填充使得两个特征图大小相同，然后用对两个相同大小的特征图求平均的方法得到最终的特征图，如下式所示：After two convolution operations, two feature maps of different sizes are obtained respectively.

and

Then for smaller feature maps

Filling is performed to make the two feature maps of the same size, and then the final feature map is obtained by averaging the two feature maps of the same size, as shown in the following formula:

(2) Enter the main capsule layer, and perform the following calculations on the output of the convolutional layer.

p=g(W ^b M+b ₁ )

where g is the non-linear squeeze function squash through the whole vector, M is the input of the capsule, b ₁ is the bias of the capsule, and W ^b is the weight.

This is the first capsule layer where the capsule replaces the scalar output of the convolution operation with a vector output.

(3) The maximum pooling is replaced by a dynamic routing algorithm, and the weight is dynamically strengthened or weakened to obtain effective features.

(4) Enter the convolution capsule layer. After this layer, all the capsules in the previous layer are changed into a series of capsules, and the weights are further dynamically strengthened or weakened through the dynamic routing algorithm to obtain more effective features.

(5) Enter the fully connected capsule layer and connect all the extracted features.

(6) Input all the features into the multi-layer perceptron, and use the activation function to obtain the probability that the data set (x, y') generated by the generator is the real data (x, y).

2. The optimized Mongolian-Chinese machine translation model includes the following parts:

A. BPE processing of Mongolian

The current Mongolian script is a pure phonetic script, which is no different from the main phonetic script in Western Europe and around the world in terms of the phonetic method. BPE technology is an algorithm for segmenting the phonetic script by counting the frequency of adjacent characters. Consecutive characters that appear more frequently are considered a combination. In general, various root affixes in Mongolian are Mongolian character combinations with high frequency, so the present invention applies the BPE algorithm to the segmentation of Mongolian. The specific algorithm is described as follows:

(1) First, the corpus to be preprocessed is divided into the smallest unit. For Mongolian, the smallest unit is the Mongolian alphabet.

(2) Then count the number of occurrences of all adjacent smallest unit combinations in the corpus and sort them, find out the combinations with the highest frequency, and add these combinations to the dictionary and delete the words with the lowest frequency in the dictionary to make the dictionary more frequent. The size remains the same.

(3) Steps (1) and (2) are repeated until the words in the dictionary appear frequently in their own corpus.

B. Remove the additional elements of the Mongolian case

In the Mongolian corpus, the components between Mongolian spaces and ordinary spaces are marked as additional components of the case. Mongolian words need additional elements of the case, and the additional elements of the case have only grammatical meaning but no semantics. The Mongolian language can only become smooth after adding the additional element of case. In Mongolian-Chinese machine translation, if the additional elements of the case are not processed, the machine translation model will recognize Mongolian spaces as ordinary spaces for processing, so that a Mongolian word is easily split from the middle and recognized as two words or even multiple words. This will significantly increase the length of Mongolian sentences and affect the translation quality and the final BLEU evaluation. Therefore, the present invention removes the control characters and the additional elements of the case together in the Mongolian sentence, leaving only the stem part.

C. Perform word segmentation on Chinese

Chinese belongs to the Sino-Tibetan language family, and each sentence is only composed of a single word and punctuation, so the computer can only treat the whole sentence as a unit when processing it, which is not conducive to the calculation and processing of the computer. Therefore, in the Mongolian-Chinese machine Before the training of the translation model, the Chinese corpus needs to be separated and processed, and the present invention adopts the method of dividing Chinese characters.

The overall flow of the present invention is as follows:

(1) Building a hybrid encoder of generators in generative adversarial networks

(2) Build the decoder of the generator in the generative adversarial network

(3) Building a discriminator in a generative adversarial network

(4) Case processing of Mongolian corpus

(5) Segment Mongolian corpus with different granularities

(6) characterize Chinese

(7) Training the generator

(8) Generate negative data through the trained generator model

(9) Training the discriminator

(10) Conduct adversarial training

(11) Test the BLEU value of the obtained Mongolian-Chinese machine translation model.

Claims (9)

1. a method that utilizes generative confrontation network to construct Mongolian-Chinese parallel corpus to improve the quality of Mongolian-Chinese translation, described generative confrontation network mainly comprises generator and discriminator, in described generator, use hybrid encoder to convert source language sentence Mongolian The language is encoded into a vector representation, and the vector representation is converted into the target language sentence Chinese using a bidirectional Transformer-based decoder combined with a sparse attention mechanism, thereby generating Mongolian sentences and Mongolian-Chinese parallel corpora that are close to human translation. In the discriminator , judge the gap between the Chinese sentence generated by the generator and the human translation, and train the generator and the discriminator against each other until the discriminator thinks that the Chinese sentence generated by the generator is very similar to the human translation, that is, the generator and the discriminator achieve Nash equilibrium When the Mongolian-Chinese machine translation system and the Mongolian-Chinese parallel data set are obtained, the Mongolian-Chinese machine translation system is used to perform the Mongolian-Chinese translation. General sentence meaning and detailed information to assist the generator to generate sentences that are close to the real translation; the multi-scale discriminator is implemented using a capsule network, which includes a convolutional layer, a main capsule layer, a convolutional capsule layer, and a fully connected capsule layer ; In the convolutional layer, the activation values of different sub-layers represent the activation values of sentences of different granularities, the activation values of the lower layers represent the activation values of words, the activation values of the upper layers represent the activation values of the whole sentence, and finally the feature maps of different layers are transformed. to the same-scale feature map with channel number 1. 2. according to claim 1, utilize Generative Adversarial Network to construct the method for Mongolian-Chinese parallel corpus to improve Mongolian-Chinese translation quality, it is characterized in that, described hybrid encoder is made up of sentence encoder and word encoder, and sentence encoder is made up of two-way Transformer The word encoder uses a bidirectional LSTM. The bidirectional Transformer optimizes Transformer1. On the basis of the original Transformer, a gated linear unit is first added to obtain important information in the source language sentence and discard redundant information; secondly, a branch structure is added, In order to capture the diverse semantic information between sentences in the source language, the branch structure contains 2 capsule networks and 2 activation functions, and the output of each capsule network corresponds to an activation function; finally, on the branch structure and the third layer After standardization, a capsule network is added, so that the encoder can capture the exact position of words in the source language sentence; in the decoder, the bidirectional Transformer optimizes Transformer2, and firstly adds a branch structure on the basis of the original Transformer. The branch structure contains 2 Multi-head attention mechanism, 2 capsule networks and 1 activation function, 2 multi-head attention mechanisms are located between the first layer normalization and the second layer normalization, 2 capsule networks and 1 activation function are located in the second layer between normalization and the third layer normalization, where the output of 1 capsule network is connected to 1 activation function, and 1 capsule network is directly located between the second layer normalization and the third layer normalization; followed by the connection of the third layer Capsule network with layer normalized output and fourth layer normalized input; Swish activation function added at the end. 3. according to claim 2, utilize Generative Adversarial Network to construct the method for Mongolian-Chinese parallel corpus to improve Mongolian-Chinese translation quality, it is characterized in that, described word encoder and sentence encoder successively encode source language sentence, then by fusion function Fusion is performed to obtain a vector representation with contextual information. The word encoder represents each word in the form of a vector, and constructs a vector representation of Mongolian sentences with words as the basic unit. The model formula is: h1 _i =Φ(h1 _{i -1} , Wi ) Among them, Φ is the activation function, Wi is the weight, and h1 _{i -1} is the hidden layer state of the i-1th word; The sentence encoder expresses an entire Mongolian sentence in the form of a vector, and constructs a vector representation with the sentence as the basic unit. The model formula is:

Among them, v ^j represents the value of the jth word,

The calculation formula is as follows:

Among them, the calculation of α _i,j is as follows:

Among them, qi is the query of the ^ith word, k ^j is the key of the jth word, · represents the dot product operation, and d represents the dimensions of q and k; The fusion function is as follows: ψ(h1 _i , h2 _i )=a ₁ h1 _i +a ₂ h2 _i Among them, ψ is the fusion function, a ₁ , a ₂ represent the corresponding weights of the two encoders through random initialization, and the two kinds of encodings are fused into an encoder containing two kinds of vector information of sentences and words. 4. according to claim 2, utilize Generative Adversarial Network to construct the method for Mongolian-Chinese parallel corpus to improve Mongolian-Chinese translation quality, it is characterized in that, in described sentence encoder, two-way Transformer refers to reading the whole text sequence at one time, that is, based on Both sides of the sentence are learned, enabling the learning of contextual relationships between words in the text. 5. according to claim 2, utilize Generative Adversarial Network to construct the method for Mongolian-Chinese parallel corpus to improve Mongolian-Chinese translation quality, it is characterized in that, in described decoder, two-way Transformer refers to the vector representation of one-time reading source language sentence, that is, Decoding based on both sides of the entire sentence vector representation. 6. The method for constructing Mongolian-Chinese parallel corpus using generative adversarial network to improve Mongolian-Chinese translation quality according to claim 1, it is characterized in that, given sentence pair (x, y), each capsule network first connects x and y by The embedding vector of the word is used to construct the representation of the 2D image, that is, for the i-th word x _i in the source language sentence x and the j-th word y _j in the target language sentence y, there is the following correspondence:

Where: x _i ^T represents the transpose of x _i , y _j ^T represents the transpose of y _j ,

Represents the matrix formed by the i-th word x _i in the source language and the j-th word y _j in the target language, that is, a virtual 2D image representation; Based on the virtual 2D image representation, the convolution layer, main capsule layer, convolution capsule layer, and fully connected capsule layer of the capsule network are sequentially passed to capture the sentence y translated by the generator under the condition of the source language sentence x. ' and the degree of similarity between the human-translated sentence y. 7. The method for constructing Mongolian-Chinese parallel corpus using Generative Adversarial Networks to improve Mongolian-Chinese translation quality according to claim 6, wherein the virtual 2D image representation is successively passed through the convolution layer and the main capsule layer of the capsule network. , the specific process of the convolution capsule layer and the fully connected capsule layer are: (1) After the convolution layer, first perform a convolution operation with a stride of 1 and a convolution kernel of 9 × 9, and capture the correspondence between the sentences in x and y through the following feature map;

Among them, f is the activation function of the first convolution operation,

is the weight of the first convolution operation,

represents the matrix formed by the i-th word x _i in the source language and the j-th word y _j in the target language, and b ^{(1, f)} is the bias of the first convolution operation; Then perform a convolution operation with a stride of 1 and a convolution kernel of 3×3, and capture the correspondence between words in x and y through the following feature map:

in,

is the weight of the second convolution operation, f,

b ^(1,f) is the same as in the first convolution operation; After two convolution operations, two feature maps of different sizes are obtained respectively.

and

Then for smaller feature maps

Filling is performed to make the two feature maps of the same size, and then the final feature map is obtained by averaging the two feature maps of the same size, as shown in the following formula:

(2) Enter the main capsule layer, that is, the first capsule layer, and perform the following calculation on the output of the convolution layer p=g(W ^b M+b ₁ ) Among them, g is the nonlinear squash function squash through the whole vector, M indicates that the input of the capsule is also the output of the convolution layer, b ₁ is the bias of the capsule, and W ^b is the weight; in the main capsule layer, the capsule will convolution The scalar output of the operation is replaced by the vector output; (3) The maximum pooling is replaced by the dynamic routing algorithm, and the weight is dynamically strengthened or weakened to obtain effective features; (4) Enter the convolution capsule layer; after this layer, all capsules in the previous layer are changed to a series of capsules, and the weight is further dynamically strengthened or weakened by the dynamic routing algorithm to obtain effective features; (5) Enter the fully connected capsule layer, and connect all the extracted features; (6) Input all the features into the multi-layer perceptron, and use the activation function to obtain the probability that the data set (x, y') generated by the generator is the real data (x, y), that is, the similarity with the real sentence. 8. according to claim 1, utilize Generative Adversarial Network to construct the method for Mongolian-Chinese parallel corpus to improve Mongolian-Chinese translation quality, it is characterized in that, the final training target of described Generative Adversarial Network is:

Where: G represents the generator, D represents the discriminator, V(D, G) represents the loss function of the generator G and the discriminator D,

Indicates to find the D that can maximize the loss function V(D, G) and the G that minimizes the loss function, E represents the expectation, and P _data (x, y) represents the source language x and target language y in the parallel corpus input. In the discriminator D, the discriminator considers it as the probability of human translation, G(y'|x) indicates that the source language x in the parallel corpus and the target language y' generated by G are input into the discriminator D, the discriminator considers that It is the probability of human translation, x represents the Mongolian sentence in the parallel corpus, that is, the source language sentence, y represents the Chinese sentence in the parallel corpus, that is, the artificial translation result, y' represents the Chinese sentence generated by the generator, that is, the generator's translation result. 9. according to claim 1, utilize Generative Adversarial Network to construct the method for Mongolian-Chinese parallel corpus to improve Mongolian-Chinese translation quality, it is characterized in that, in process: The additional components of the Mongolian corpus are processed as follows: Remove the control characters and the additional elements of the case together in Mongolian sentences, leaving only the stem part; The Mongolian corpus is segmented with different granularities, and the methods are as follows: (1) First, the corpus to be preprocessed is divided into the smallest unit, and for Mongolian, the smallest unit is the Mongolian alphabet; (2) Then count the number of occurrences of all adjacent smallest unit combinations in the corpus and sort them, find out the combinations with the highest frequency, and add these combinations to the dictionary and delete the words with the lowest frequency in the dictionary to make the dictionary more frequent. the size remains the same; (3) Repeat steps (1) and (2) until the frequency of the words in the dictionary appearing in their own corpus is higher than the set value; Characterize Chinese.

Images