CN110598221B - Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network - Google Patents
Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network Download PDFInfo
- Publication number
- CN110598221B CN110598221B CN201910807617.7A CN201910807617A CN110598221B CN 110598221 B CN110598221 B CN 110598221B CN 201910807617 A CN201910807617 A CN 201910807617A CN 110598221 B CN110598221 B CN 110598221B
- Authority
- CN
- China
- Prior art keywords
- mongolian
- sentence
- layer
- chinese
- capsule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013519 translation Methods 0.000 title claims abstract description 97
- 238000000034 method Methods 0.000 title claims abstract description 33
- 239000013598 vector Substances 0.000 claims abstract description 39
- 241000282414 Homo sapiens Species 0.000 claims abstract description 19
- 230000002457 bidirectional effect Effects 0.000 claims abstract description 19
- 230000007246 mechanism Effects 0.000 claims abstract description 8
- 239000002775 capsule Substances 0.000 claims description 78
- 230000006870 function Effects 0.000 claims description 39
- 230000004913 activation Effects 0.000 claims description 37
- 230000004927 fusion Effects 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 11
- 235000019580 granularity Nutrition 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 239000000463 material Substances 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 3
- 238000005755 formation reaction Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000005096 rolling process Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims 1
- 230000002950 deficient Effects 0.000 abstract description 2
- 230000014616 translation Effects 0.000 description 74
- 230000001537 neural effect Effects 0.000 description 7
- 238000005457 optimization Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000007476 Maximum Likelihood Methods 0.000 description 2
- 230000003042 antagnostic effect Effects 0.000 description 2
- 230000004888 barrier function Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000001143 conditioned effect Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000116 mitigating effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 230000003313 weakening effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Machine Translation (AREA)
- Document Processing Apparatus (AREA)
Abstract
A method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel linguistic data through a generation countermeasure network comprises a generator and a discriminator, wherein the generator uses a hybrid encoder to encode Mongolian of a source language sentence into vector representation, a decoder based on a bidirectional transducer is combined with a sparse attention mechanism to convert the representation into Chinese of a target language sentence, and accordingly Mongolian sentences closer to human translation and more Mongolian Chinese parallel linguistic data are generated. The method solves the problems that the Mongolian parallel data set is seriously deficient, and the NMT can not ensure the naturalness, the sufficiency and the accuracy of the translation result.
Description
Technical Field
The invention belongs to the technical field of machine translation, and particularly relates to a method for improving the translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using a generated confrontation network.
Background
Machine translation is one of the most powerful means to solve the problem of language barrier, which can automatically translate one language into another language by using a computer. In recent years, many large-scale search enterprises and service centers such as google, hundredth, etc. have conducted extensive research on machine translation, and have made an important contribution to obtaining high-quality translations for machine translation, so that translations between major languages have approached the human translation level, and millions of people have achieved communication across language barriers using online translation systems and mobile applications. In the recent wave of deep learning, machine translation has become an important part and has become an important component for promoting global communication.
The Seq2 Seq-based neural-machine translation framework consists of an encoder that reads an input sequence and outputs a single vector, and a decoder that reads the vector to produce an output sequence. Since 2013, the framework has rapidly evolved, achieving a significant improvement in translation quality over statistical machine translation. The addition of sentence-level maximum likelihood estimation principles, gating cells in LSTM and GRU, and attention mechanisms improves the ability of NMT to translate long sentences. AshishVaswani et al proposed a Transformer architecture in 2017, an architecture that completely relies on an attention mechanism to draw global dependencies between inputs and outputs. The method has the advantages of realizing parallelization calculation, effectively reducing the training time of the model and improving the quality of the machine translation model to a certain extent. The defects that the RNN and the derivative network thereof are slow and can not realize parallelization and the like are avoided.
At present, neural machine translation has been successful, but the best NMT system is far from any one of human expectations and translation quality needs to be improved. Because NMT usually trains the model using maximum likelihood estimation, i.e. maximizes the probability of the target real sentence conditioned on the source sentence, i.e.: the model can generate the best candidate word for the current time, but in the long run the translation of the entire sentence is not the best translation, which leaves a hidden danger to NMT. Even powerful transformers are no exception. Compared with the real translation of human beings, the naturalness, the sufficiency and the accuracy of the translation result cannot be guaranteed by the target.
In addition, the mutual translation between large languages is relatively mature, but the machine translation between small languages is very expensive to construct parallel corpora manually due to various challenges, especially the severe shortage of corpora, and thus the translation effect is still unsatisfactory.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a method for improving the translation quality of Mongolian Chinese by constructing a Mongolian Chinese parallel corpus by using a generated confrontation network, which mainly aims at the problems that the Mongolian Chinese parallel data set is seriously deficient and the NMT cannot ensure the naturalness, the sufficiency, the accuracy and the like of a translation result, and applies the generated confrontation network to the Mongolian Chinese neural machine translation.
In order to achieve the purpose, the invention adopts the technical scheme that:
the method for improving the translation quality of the Mongolian Chinese by constructing the Mongolian Chinese parallel corpus through the generation countermeasure network is characterized in that the generation countermeasure network is used in the Mongolian Chinese machine translation to relieve the problem of low translation quality of the Mongolian Chinese machine caused by the shortage of the Mongolian Chinese parallel corpus and minimize the difference between human translation and translation given by an NMT model, the generation countermeasure network mainly comprises a generator and a discriminator, and the generator can effectively utilize the Mongolian Chinese monolingual data to relieve the problem of the shortage of the Mongolian Chinese parallel corpus in a machine translation task. In the generator, in order to relieve the UNK phenomenon in Mongolian Chinese machine translation, a hybrid encoder is used for encoding Mongolian of a source language sentence into vector representation, a bidirectional Transformer-based decoder is combined with a sparse attention mechanism to convert the vector representation into Chinese of a target language sentence, and therefore Mongolian sentences closer to human translation and more Mongolian parallel linguistic data are generated, and the quality and the efficiency of Mongolian Chinese machine translation are improved. In the discriminator, the difference between the Chinese sentence generated by the generator and the human translation is judged, the generator mainly aims to generate Mongolian sentences which are closer to the human translation and effectively generate more Mongolian parallel linguistic data by using Mongolian single-language data, and the discriminator aims to calculate the difference between the Chinese sentence generated by the generator and the Chinese sentence translated by the human translation. And (3) performing countertraining on the generator and the discriminator until the discriminator considers that the Chinese sentence generated by the generator is very similar to the human translation, namely the generator and the discriminator realize Nash balance, obtaining a high-quality Mongolian Chinese machine translation system and a large number of Mongolian Chinese parallel data sets, and performing Mongolian Chinese translation by using the Mongolian Chinese machine translation system.
The hybrid encoder is composed of a sentence encoder and a word encoder, in order to capture semantic information between sentences and encoder efficiency, the sentence encoder is composed of a bidirectional Transformer, and the word encoder uses bidirectional LSTM, thereby improving the efficiency of the word encoder under the condition of ensuring the quality of the encoder. The bidirectional Transformer is an optimized Transformer1, a gate control linear unit is added on the basis of the original Transformer firstly, so as to effectively obtain important information in a source language sentence and discard redundant information; secondly, a branch structure is added to effectively capture diversified semantic information between source language sentences; finally, a capsule network is added on the branch structure and after the third layer of standardization, so that an encoder can capture the accurate position of a word in a source language sentence, the accuracy of the encoder is further enhanced, and the encoding quality is improved; in the decoder, a bidirectional Transformer is optimized Transformer2, and a branch structure is added firstly on the basis of the original Transformer; secondly, a capsule network is added; finally, Swish activation function is added to effectively improve the decoding accuracy of the decoder.
The word encoder and the sentence encoder encode source language sentences in sequence, and then the source language sentences are fused through a fusion function to obtain vector representation with context information, wherein each word is represented into a vector form by the word encoder, vector representation of Mongolian sentences with the words as basic units is constructed, and the model formula is as follows:
h1i＝Φ(h1i-1，Wi)
where Φ is the activation function, WiAs a weight, h1i-1For the (i-1) th wordHidden layer state.
The sentence encoder represents a whole Mongolian sentence in a vector form, and constructs a vector representation taking the sentence as a basic unit, wherein a model formula is as follows:
wherein v isjA Value (Value) representing the jth word,
the calculation formula of (a) is as follows:
wherein, αi,jIs calculated as follows:
wherein q isiQuery (query) for ith word, kjA key (key) for the jth word,. representing a dot product operation, and d representing the dimensions of q and k;
the fusion function is shown as follows:
ψ(h1i,h2i)＝a1h1i+a2h2i
where ψ is a fusion function, a1，a2The encoder is characterized in that the two encoders are fused into vector information containing sentences and words through the two encodings by corresponding weights initialized randomly.
In the sentence encoder, the bidirectional Transformer means that the whole text sequence is read at one time, namely, learning is based on two sides of a sentence, and the two sides are not read sequentially from left to right or from right to left, so that the context relationship between words in the text can be learned.
In the decoder, the bidirectional Transformer reads the vector representation of the source language sentence at one time, namely decoding is carried out based on two sides of the vector representation of the whole sentence, so as to further improve the decoding accuracy of the decoder.
In order to enhance the discrimination capability of the discriminator, the discriminator is a multi-scale discriminator which can discriminate the general sentence meaning and the detail information (such as phrases, words and the like) of the Chinese sentences generated by the generator so as to assist the generator to generate the sentences which are closer to the real translation; meanwhile, in order to overcome the translation invariance of the convolutional neural network, the multi-scale discriminator is realized by using a capsule network, and the discrimination capability of the discriminator can be effectively improved under the condition of not reducing the training efficiency, wherein the translation invariance refers to: for example, in face recognition, the convolutional neural network considers that a face with eyes and mouth and other features is a human face, and specific positions of five sense organs in the face are ignored. If it is used in the generation of a countermeasure network as a discriminator, it is considered that only sentences having words in all of the manually translated sentences in the chinese sentences generated by the generator are manually translated sentences because of its translational invariance. And neglects the position information of the words, thereby causing a misdiscrimination. The capsule network comprises a convolution layer, a main capsule layer, a convolution capsule layer and a full-connection capsule; in order to use a network to represent a plurality of discriminators and improve training efficiency, in a convolutional layer, activation values of different sub-layers represent activation values of sentences with different granularities, activation values of a lower layer represent activation values of words, activation values of a higher layer represent activation values of the whole sentence, and finally feature maps of different layers are transformed into feature maps with the same scale and with the channel number of 1.
Given a sentence pair (x, y), each capsule network first constructs a 2D-image-like representation by concatenating the embedded vectors of the words in x and y, i.e., for the ith word x in the source language sentence xiAnd the jth word y in the target language sentence yjThe following correspondence relationship is provided:
wherein: x is the number ofi TDenotes xiTranspose of (y)j TDenotes yjThe transpose of (a) is performed,
representing the ith word x in the Source languageiAnd the jth word y in the target languagejThe constructed matrix, i.e. the virtual 2D image representation;
based on the virtual 2D image representation, the similarity between the sentence y' translated by the generator and the sentence y translated by the human under the condition of the source language sentence x is captured through the convolution layer, the main capsule layer, the convolution capsule layer and the fully-connected capsule layer of the capsule network in sequence.
The virtual 2D image represents that the specific process of sequentially passing through the coiling layer, the main capsule layer, the coiling capsule layer and the full-connection capsule layer of the capsule network is as follows:
(1) after the convolutional layer, first, a convolution operation with a convolutional kernel of 9 × 9 is performed with a step size of 1, and the correspondence between sentences in x and y is captured by the following feature mapping.
Wherein f is the activation function of the first convolution operation,
is the weight of the first convolution operation,
representing the ith word x in the Source languageiAnd the jth word y in the target languagejA matrix of formations, b(1,f)An offset for the first convolution operation;
then, a convolution operation with a step size of 1 and a convolution kernel of 3 × 3 is performed to capture the correspondence between words in x and y by the following feature mapping.
Wherein,
the weight of the second convolution operation, f,
b(1,f)The same as in the first convolution algorithm;
two feature maps with different sizes are respectively obtained after two times of convolution operation
And
then for smaller feature maps
Filling is carried out to enable the two feature maps to be the same in size, and then the final feature map is obtained by averaging the two feature maps with the same size, as shown in the following formula:
(2) entering the first capsule layer, the output of the convolution layer is calculated as follows
p＝g(WbM+b1)
Where g is the nonlinear squeeze function square through the entire vector, M represents the input to the capsule, b1Is the bias of the capsule, WbIs a weight; in the main capsule layer, the capsule replaces the scalar output of the convolution operation with a vector output;
(3) the dynamic routing algorithm replaces the maximum pooling, and the weight is dynamically strengthened or weakened to obtain effective characteristics;
(4) entering a rolling capsule layer; after the layer, all the capsules in the previous layer are changed into a series of capsules, and the weights are further dynamically strengthened or weakened through a dynamic routing algorithm, so that more effective characteristics are obtained;
(5) entering a full-connection capsule layer, and connecting all extracted features;
(6) all the features are input into the multilayer perceptron, and the activation function is used to obtain the probability that the data set (x, y') generated by the generator is real data (x, y), namely the similarity degree with a real sentence.
The final training goals for generating the countermeasure network are:
wherein: g denotes a generator, D denotes a discriminator, V (D, G) denotes a loss function of the generator G and the discriminator D,
representing D for maximizing the loss function V (D, G) and minimizing the loss function, E representing the expectation, Pdata(x, y) represents the probability that the source language x and the target language y in the parallel corpus are input into the discriminator D, which the discriminator considers to be manually translated, G (y ' | x) represents the probability that the target language y ' generated by the source language x and G in the parallel corpus is input into the discriminator D, which the discriminator considers to be manually translated, x represents the mongolian sentence in the parallel corpus, i.e., the source language sentence, y represents the chinese sentence in the parallel corpus, i.e., the manual translation result, and y ' represents the chinese sentence generated by the generator, i.e., the translation result of the generator.
In the process:
processing the additional components of the Mongolian language material grid, wherein the method comprises the following steps:
removing the control symbols in Mongolian sentences and the additional components of the lattices together, and only leaving the stem parts;
the Mongolian language materials are segmented at different granularities, Chinese is subjected to word segmentation processing, the UNK phenomenon in Mongolian language machine translation is relieved, and the Mongolian language machine translation quality is further improved by processing additional components of Mongolian lattices.
The cutting method comprises the following steps:
(1) the corpus to be preprocessed is first cut into the smallest units, which are the Mongolian letters for Mongolian.
(2) Then, the times of occurrence of all adjacent minimum unit combinations in the corpus are counted and ranked, the combination with the highest occurrence frequency is found out, and the combinations are added into the dictionary, and the word with the lowest frequency in the dictionary is deleted at the same time, so that the size of the dictionary is kept unchanged.
(3) Repeating the steps (1) and (2) until the occurrence frequency of the words in the dictionary in the corpus is higher than a set value;
neural Machine Translation (NMT) achieves better translation in an end-to-end framework, but the best NMT systems have a larger gap from human expectations. Compared with the method, the method can minimize the difference between human translation and the translation given by the NMT model, relieve the data sparsity problem in Mongolian Chinese machine translation and relieve the UNK phenomenon in the Mongolian Chinese machine translation, thereby not only obtaining a high-quality Mongolian Chinese machine translation system, but also obtaining a large number of Mongolian Chinese parallel data sets.
Drawings
FIG. 1 is an optimized Transformer1 architecture.
FIG. 2 is an optimized Transformer2 architecture.
Fig. 3 is a generator frame.
Fig. 4 is a discriminator framework.
Detailed Description
The embodiments of the present invention will be described in detail below with reference to the drawings and examples.
The invention relates to a method for improving the translation quality of Mongolian Chinese by constructing a Mongolian Chinese parallel corpus by using a generating confrontation network, which mainly comprises the construction of an encoder and a decoder and the construction of a discriminator model.
FIG. 1 shows an optimized Transformer1 architecture. Firstly adding a gate control linear unit on the basis of an original Transformer, effectively acquiring important information in a source language sentence and discarding redundant information; secondly, a branch structure is added, and the structure can effectively capture diversified semantic information between source language sentences; finally, a capsule network is added on the branch structure and after the third layer of standardization, so that an encoder can capture the accurate position of a word in a source language sentence, and the accuracy of the encoder is further enhanced.
FIG. 2 shows an optimized Transformer2 architecture. Adding a branch structure on the basis of the original Transformer; secondly, a capsule network is added; and finally, a Swish activation function is added, and the addition of the activation function can effectively improve the decoding accuracy of the decoder.
Fig. 3 shows a generator frame. Mainly composed of a hybrid encoder, a sparse attention and decoder 3, the encoder accepts input Mongolian sentences such as:
the method comprises the steps of firstly carrying out bidirectional coding on the whole sentence based on bidirectional optimization Transformer, simultaneously carrying out coding on words in the sentence based on a bidirectional LSTM-based coder, and then fusing the representations of the two coders by using a fusion function to generate a coded representation of a source language. The decoder then decodes the source language encoded representation into the target language chinese sentence "rainy tomorrow" in conjunction with a sparse attention mechanism.
Fig. 4 shows a frame of a capsule network, including a convolutional layer, a main capsule layer, a convolutional capsule layer, a fully-connected capsule layer, etc. The convolutional layer comprises two layers, wherein one layer captures characteristics at the sentence level, and the other layer captures characteristics at the word level, so that the multi-scale identification function is realized.
Mitigation of data sparseness problem in Mongolian machine translation:
the generator in the generation countermeasure network can effectively solve the existing data sparseness problem in Mongolian Chinese machine translation, specifically, the generator is pre-trained through Mongolian Chinese alignment corpus to obtain a pre-training model, Mongolian monolingual data is used for generating Mongolian Chinese pseudo bilingual data with the help of the model, and then Chinese sentences which are closer to manual translation are generated with the help of the discriminator to form the Mongolian Chinese alignment corpus.
The accuracy and naturalness of the Mongolian Chinese machine translation are improved:
the Chinese sentences generated by the generator are often relatively hard and unnatural, and in the invention, the identifier is equivalent to a teacher of the generator and assists the generator to generate more natural and accurate Chinese sentences. The multi-scale discriminator means that the teacher has the ability to determine from multiple aspects whether the sentence generated by the generator is similar to the artificial translation.
Decision variables: the Mongolian sentence x is input at the encoder end of the generator, and the corresponding machine-translated Chinese sentence y is output at the decoder end of the generator. Mongolian sentence x, corresponding manually translated Chinese sentence y and corresponding generator translated Chinese sentence y 'input at the input of the discriminator'
The invention comprises the following parts:
1. the Mongolian Chinese neural machine translation system model based on the generation of the countermeasure network comprises the following parts:
A. generating a mixed encoder description in a Mongolian neural machine translation system generator based on an antagonistic network: the hybrid encoder is formed by fusing a sentence encoder and a word encoder, the word encoder and the sentence encoder encode source language sentences in sequence, wherein the word encoder represents each word into a vector form, vector representation of Mongolian sentences taking the word as a basic unit is constructed, and a model formula is as follows:
h1i＝Φ(h1i-1，Wi)
where Φ is the activation function, WiAs a weight, h1i-1Is the hidden layer state of the (i-1) th word.
The sentence encoder represents an entire Mongolian sentence in vector form, constructing a vector representation with the sentence as a basic unit. In the sentence encoder, the bidirectional optimization Transformer1 reads the whole text sequence at one time, i.e. learning based on both sides of the sentence, so that the context relationship between the sentences can be learned and parallelization can be realized. The model formula is as follows:
wherein,vjvalue representing the jth word,
the calculation formula of (a) is as follows:
wherein, αi,jIs calculated as follows:
wherein q isiIs the query of the ith word, kjThe key (key) for the jth word,. represents a dot product operation, and d represents the dimensions of q and k.
And finally, fusing through a fusion function to obtain vector representation with context information. The fusion function is shown as follows:
ψ(h1i,h2i)＝a1h1i+a2h2i
where ψ is a fusion function, a1，a2The encoder which contains the vector information of sentences and words is fused by two kinds of granularity encoding through corresponding weights initialized randomly.
B. Decoder description in generator of Mongolian neural machine translation system based on generation of countermeasure network: the decoder consists of a bi-directional optimization Transformer2, the optimized Transformer2 is basically similar to the optimized Transformer1 in the encoder, except that a Swish activation function is added to the optimized Transformer in this section. The decoder decodes the vector representation of the source language sentence into the target language sentence in combination with a sparse attention mechanism during decoding.
C. Mongolian neural machine translation system discriminator description based on generation of an antagonistic network: the discriminator is a multi-scale discriminator which can not only discriminate the general sentence meaning of the Chinese sentence generated by the generator, but also discriminate the general sentence meaning of the Chinese sentence generated by the generatorThe detailed information (e.g., phrases and words, etc.) may assist the generator in generating sentences that more closely approximate the real translation. The multi-scale discriminator is implemented using a capsule network comprising a convolutional layer, a main capsule layer, a convolutional capsule layer, and a fully-connected capsule. In order to use a network to represent a plurality of discriminators and improve training efficiency, in a convolutional layer, activation values of different sub-layers represent activation values of sentences with different granularities, activation values of a lower layer represent activation values of words, activation values of a higher layer represent activation values of the whole sentence, and finally feature maps of different layers are transformed into feature maps with the same scale and with the channel number of 1. In particular, given a sentence pair (x, y), each capsule network first constructs a 2D-image-like representation by concatenating the embedded vectors of the words in x and y, i.e., for the ith word x in the source language sentence xiAnd the jth word y in the target language sentence yjThe following correspondence relationship is provided:
wherein: x is the number ofi TDenotes xiTranspose of (y)j TDenotes yjThe transpose of (a) is performed,
representing the ith word x in the Source languageiAnd the jth word y in the target languagejThe constructed matrix, i.e. the virtual 2D image representation.
Based on such virtual 2D image representation, the similarity between the sentence y' translated by the generator and the sentence y translated by the human under the condition of the source language sentence x is captured sequentially through the convolution layer, the main capsule layer, the convolution capsule layer, and the fully connected capsule layer of the capsule network. The specific process is as follows:
(1) after the convolutional layer, first, a convolution operation with a convolutional kernel of 9 × 9 is performed with a step size of 1, and the correspondence between sentences in x and y is captured by the following feature mapping.
Wherein f is the activation function of the first convolution operation,
is the weight of the first convolution operation,
representing the ith word x in the Source languageiAnd the jth word y in the target languagejA matrix of formations, b(1,f)Is the offset of the first convolution operation.
Then, a convolution operation with a step size of 1 and a convolution kernel of 3 × 3 is performed to capture the correspondence between words in x and y by the following feature mapping.
Wherein,
the weight of the second convolution operation, f,
b(1,f)As in the first convolution algorithm.
Two feature maps with different sizes are respectively obtained after two times of convolution operation
And
then for smaller feature maps
Filling is carried out to enable the two feature maps to be the same in size, and then the final feature map is obtained by averaging the two feature maps with the same size, as shown in the following formula:
(2) the output of the convolutional layer is calculated as follows.
p＝g(WbM+b1)
Where g is the nonlinear squeeze function square through the entire vector, M represents the input to the capsule, b1Is the bias of the capsule, WbAre weights.
This is the first capsule layer in which the capsule replaces the scalar output of the convolution operation with the vector output.
(3) The dynamic strengthening or weakening of the weight to obtain the effective characteristics through the dynamic routing algorithm instead of the maximum pooling.
(4) Entering the rolling capsule layer. After this layer, all the capsules in the previous layer are changed into a series of capsules, and the weights are further dynamically increased or decreased through a dynamic routing algorithm, so that more effective characteristics are obtained.
(5) And (4) entering a full-connection capsule layer, and connecting all the extracted features.
(6) All features are input into the multi-layer perceptron and the activation function is used to derive the probability that the data set (x, y') generated by the generator is the true data (x, y).
2. An optimized Mongolian Chinese machine translation model, comprising the following parts:
A. BPE processing of Mongolian
The existing Mongolian is a pure pinyin character which is not different from the main pinyin characters in Western Europe and all over the world in the pinyin method, the BPE technology is an algorithm for segmenting the pinyin characters by counting the occurrence frequency of adjacent characters, and continuous characters with high occurrence frequency are considered as a combination. Generally, various root affixes in the Mongolian are Mongolian character combinations with high occurrence frequency, so that the BPE algorithm is applied to the segmentation of the Mongolian. The specific algorithm is described as follows:
(1) the corpus to be preprocessed is first cut into the smallest units, which are the Mongolian letters for Mongolian.
(2) Then, the times of occurrence of all adjacent minimum unit combinations in the corpus are counted and ranked, the combination with the highest occurrence frequency is found out, and the combinations are added into the dictionary, and the word with the lowest frequency in the dictionary is deleted at the same time, so that the size of the dictionary is kept unchanged.
(3) And (3) repeating the steps (1) and (2) until the occurrence frequency of the words in the dictionary in the corpus is higher.
B. Removing additional components of lattice in Mongolian
In the Mongolian corpus, the components between Mongolian spaces and ordinary spaces are labeled as additional components of the lattice. Additional components of the lattice are needed among the words of the Mongolian, and the additional components of the lattice only have grammatical significance and have no semantics. After the additional components of the Mongolian plus the lattice are added, the sentence can become smooth. In Mongolian machine translation, if the additional components of the lattice are not processed, the machine translation model recognizes Mongolian blank spaces as common blank spaces to be processed, so that one Mongolian word is easily segmented from the middle to be recognized as two words or even a plurality of words. This can cause a significant increase in the length of the Mongolian sentence, which affects the translation quality and the final BLEU evaluation. Thus, the present invention removes the tokens from the Mongolian sentence along with the lattice's additional components, leaving only the stem portion.
C. Dividing Chinese characters
The Chinese language belongs to the Tibetan language system, each sentence is only composed of a single character and punctuation marks, so that the whole sentence can be only regarded as a unit when the computer processes the Chinese language, and the calculation and the processing of the computer are not facilitated, therefore, the Chinese language materials need to be separated before the Mongolian Chinese machine translation model is trained.
The whole process of the invention is as follows:
(1) hybrid encoder for building generators in a generative confrontational network
(2) Building a decoder to generate generators in a countermeasure network
(3) Setting up a discriminator in a generative confrontation network
(4) Processing lattice for Mongolian language corpus
(5) Carrying out segmentation of different granularities on Mongolian linguistic data
(6) Dividing Chinese characters
(7) Training generator
(8) Generating negative data through a trained generator model
(9) Training discriminator
(10) Performing confrontation training
(11) The BLEU value of the resulting montmorillohman machine translation model was tested.
Claims (9)
1. A method for improving Mongolian translation quality by constructing Mongolian parallel linguistic data by using a generating confrontation network mainly comprises a generator and a discriminator, wherein in the generator, a mixed encoder is used for encoding Mongolian of a source language sentence into vector representation, a decoder based on a bidirectional Transformer is used for converting the vector representation into Chinese of a target language sentence by combining a sparse attention machine system, so that Mongolian sentences and Mongolian parallel linguistic data close to human translation are generated, in the discriminator, the difference between the Chinese sentences generated by the generator and the human translation is judged, the generator and the discriminator are subjected to confrontation training until the discriminator judges that the Chinese translation sentences generated by the generator are very similar to the human translation, namely the generator and the discriminator realize Nash balance, a Mongolian machine translation system and a Mongolian parallel data set are obtained, and the Mongolian machine translation system is used for Mongolian Chinese translation, the system is characterized in that the discriminator is a multi-scale discriminator and can discriminate general sentence meanings and detail information of Chinese sentences generated by the generator so as to assist the generator to generate sentences close to real translation; the multi-scale discriminator is implemented using a capsule network comprising a convolution layer, a main capsule layer, a convolution capsule layer, and a fully connected capsule layer; in the convolutional layer, the activation values of different sub-layers represent the activation values of sentences with different granularities, the activation values of the lower layers represent the activation values of words, the activation values of the upper layers represent the activation values of the whole sentence, and finally the feature maps of different layers are transformed into the feature map with the same scale and the number of channels being 1.
2. The method for improving the quality of Mongolian translation by using the generative countermeasure network to construct Mongolian parallel corpus according to claim 1, wherein the hybrid encoder comprises a sentence encoder and a word encoder, the sentence encoder comprises a bidirectional Transformer, the word encoder uses bidirectional LSTM, the bidirectional Transformer is used for optimizing the Transformer1, a gated linear unit is firstly added on the basis of the original Transformer to obtain important information in the source language sentence and discard redundant information; secondly, a branch structure is added to capture diversified semantic information between source language sentences, the branch structure comprises 2 capsule networks and 2 activation functions, and the output of each capsule network is correspondingly connected with 1 activation function; finally, a capsule network is added on the branch structure and after the third layer of standardization, so that an encoder can capture the accurate position of a word in a source language sentence; in the decoder, a bidirectional Transformer is used as an optimized Transformer2, a branch structure is firstly added on the basis of an original Transformer, the branch structure comprises 2 multi-head attention mechanisms, 2 capsule networks and 1 activation function, the 2 multi-head attention mechanisms are positioned between a first layer standardization and a second layer standardization, the 2 capsule networks and the 1 activation function are positioned between the second layer standardization and a third layer standardization, wherein the output of the 1 capsule network is connected with the 1 activation function, and the 1 capsule network is directly positioned between the second layer standardization and the third layer standardization; secondly, a capsule network connecting the third layer of standardized output and the fourth layer of standardized input is added; finally the Swish activation function is added.
3. The method for improving the translation quality of Mongolian Chinese by using the generated countermeasure network to construct Mongolian parallel corpus according to claim 2, wherein the word encoder and the sentence encoder sequentially encode source language sentences, and then perform fusion through a fusion function to obtain vector representation with context information, wherein the word encoder represents each word in a vector form to construct the vector representation of Mongolian sentences with words as basic units, and the model formula is as follows:
h1i＝Φ(h1i-1，Wi)
where Φ is the activation function, WiAs a weight, h1i-1Hidden layer state of i-1 word;
the sentence encoder represents a whole Mongolian sentence in a vector form, and constructs a vector representation taking the sentence as a basic unit, wherein a model formula is as follows:
wherein v isjThe value representing the j-th word,
the calculation formula of (a) is as follows:
wherein, αi,jIs calculated as follows:
wherein q isiFor the query of the ith word, kjThe key of the jth word represents a dot product operation, and d represents the dimensions of q and k;
the fusion function is shown as follows:
ψ(h1i,h2i)＝a1h1i+a2h2i
where ψ is a fusion function, a1，a2The encoder is characterized in that the two encoders are fused into vector information containing sentences and words through the two encodings by corresponding weights initialized randomly.
4. The method for improving the quality of Mongolian Chinese translation by using the generated countermeasure network to construct the Mongolian Chinese parallel corpus according to claim 2, wherein in the sentence coder, the bidirectional Transformer means to read the whole text sequence at one time, i.e. to learn based on both sides of the sentence, so as to learn the context relationship between words in the text.
5. The method for improving the quality of Mongolian translation by using the generative countermeasure network to construct Mongolian parallel corpus according to claim 2, wherein in the decoder, the bidirectional Transformer means to read the vector representation of the source language sentence at a time, i.e. decoding based on both sides of the vector representation of the whole sentence.
6. The method for improving the translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus according to claim 1, wherein given sentence pair (x, y), each capsule network first constructs a representation of the 2D image by connecting the embedded vectors of the words in x and y, i.e. x for the ith word in source language sentence xiAnd the jth word y in the target language sentence yjThe following correspondence relationship is provided:
wherein: x is the number ofi TDenotes xiTranspose of (y)j TDenotes yjThe transpose of (a) is performed,
representing the ith word x in the Source languageiAnd the jth word y in the target languagejThe constructed matrix, i.e. the virtual 2D image representation;
based on the virtual 2D image representation, the similarity between the sentence y' translated by the generator and the sentence y translated by the human under the condition of the source language sentence x is captured through the convolution layer, the main capsule layer, the convolution capsule layer and the fully-connected capsule layer of the capsule network in sequence.
7. The method for improving the translation quality of the Mongolian Chinese by constructing the Mongolian Chinese parallel corpus according to the claim 6, wherein the virtual 2D image representation sequentially passes through the convolution layer, the main capsule layer, the convolution capsule layer and the full-connection capsule layer of the capsule network in the following specific processes:
(1) after the convolutional layer, firstly, carrying out convolution operation with step length of 1 and convolution kernel of 9 × 9, and capturing the corresponding relation between sentences in x and y through the following feature mapping;
wherein f is the activation function of the first convolution operation,
is the weight of the first convolution operation,
representing the ith word x in the Source languageiAnd the jth word y in the target languagejA matrix of formations, b(1,f)An offset for the first convolution operation;
then, a convolution operation with step size 1 and convolution kernel 3 × 3 is performed to capture the correspondence between words in x and y by the following feature mapping:
wherein,
the weight of the second convolution operation, f,
b(1,f)The same as in the first convolution operation;
two feature maps with different sizes are respectively obtained after two times of convolution operation
And
then for smaller feature maps
Filling is carried out to enable the two feature maps to be the same in size, and then the final feature map is obtained by averaging the two feature maps with the same size, as shown in the following formula:
(2) entering the first capsule layer, the output of the convolution layer is calculated as follows
p＝g(WbM+b1)
Where g is the nonlinear squeeze function square through the entire vector, M denotes that the input to the capsule is also the output of the convolutional layer, b1Is the bias of the capsule, WbIs a weight; in the main capsule layer, the capsule replaces the scalar output of the convolution operation with a vector output;
(3) the dynamic routing algorithm replaces the maximum pooling, and the weight is dynamically strengthened or weakened to obtain effective characteristics;
(4) entering a rolling capsule layer; after the layer, all the capsules in the previous layer are changed into a series of capsules, and the weights are further dynamically strengthened or weakened through a dynamic routing algorithm, so that effective characteristics are obtained;
(5) entering a full-connection capsule layer, and connecting all extracted features;
(6) all the features are input into the multilayer perceptron, and the activation function is used to obtain the probability that the data set (x, y') generated by the generator is real data (x, y), namely the similarity degree with a real sentence.
8. The method for improving the Mongolian Chinese translation quality by constructing the Mongolian Chinese parallel corpus according to the claim 1, wherein the final training targets of the generation of the countermeasure network are:
wherein: g denotes a generator, D denotes a discriminator, V (D, G) denotes a loss function of the generator G and the discriminator D,
representing D for maximizing the loss function V (D, G) and minimizing the loss function, E representing the expectation, Pdata(x, y) represents the probability that the source language x and the target language y in the parallel corpus are input into the discriminator D, which the discriminator considers to be manually translated, G (y ' | x) represents the probability that the target language y ' generated by the source language x and G in the parallel corpus is input into the discriminator D, which the discriminator considers to be manually translated, x represents the mongolian sentence in the parallel corpus, i.e., the source language sentence, y represents the chinese sentence in the parallel corpus, i.e., the manual translation result, and y ' represents the chinese sentence generated by the generator, i.e., the translation result of the generator.
9. The method for improving the translation quality of Mongolian Chinese by constructing the Mongolian Chinese parallel corpus according to the claim 1, wherein the process comprises the following steps:
processing the additional components of the Mongolian language material grid, wherein the method comprises the following steps:
removing the control symbols in Mongolian sentences and the additional components of the lattices together, and only leaving the stem parts;
the Mongolian linguistic data is segmented with different granularities, and the method comprises the following steps:
(1) firstly, separating the linguistic data required to be preprocessed by a minimum unit, wherein the minimum unit is Mongolian letters for Mongolian;
(2) then counting and sequencing the occurrence times of all adjacent minimum unit combinations in the corpus, finding out the combination with the highest occurrence frequency, adding the combinations into a dictionary, and deleting the words with the lowest frequency in the dictionary to keep the size of the dictionary unchanged;
(3) repeating the steps (1) and (2) until the occurrence frequency of the words in the dictionary in the corpus is higher than a set value;
the Chinese character is divided into words.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910807617.7A CN110598221B (en) | 2019-08-29 | 2019-08-29 | Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910807617.7A CN110598221B (en) | 2019-08-29 | 2019-08-29 | Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110598221A CN110598221A (en) | 2019-12-20 |
| CN110598221B true CN110598221B (en) | 2020-07-07 |
Family
ID=68856234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910807617.7A Active CN110598221B (en) | 2019-08-29 | 2019-08-29 | Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110598221B (en) |
Families Citing this family (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111310480B (en) * | 2020-01-20 | 2021-12-28 | 昆明理工大学 | Weakly supervised Hanyue bilingual dictionary construction method based on English pivot |
| CN111326157B (en) * | 2020-01-20 | 2023-09-08 | 抖音视界有限公司 | Text generation method, apparatus, electronic device and computer readable medium |
| CN111414770B (en) * | 2020-02-24 | 2022-03-18 | 内蒙古工业大学 | Semi-supervised Mongolian neural machine translation method based on collaborative training |
| CN111310411B (en) * | 2020-03-09 | 2022-07-12 | 重庆邮电大学 | Text relevance determination method, device and equipment based on diversity model |
| JP7547077B2 (en) | 2020-05-13 | 2024-09-09 | 株式会社Ｎｔｔドコモ | Text translation device and translation model |
| CN111528832B (en) * | 2020-05-28 | 2023-04-18 | 四川大学华西医院 | Arrhythmia classification method and validity verification method thereof |
| CN111861925B (en) * | 2020-07-24 | 2023-09-29 | 南京信息工程大学滨江学院 | Image rain removing method based on attention mechanism and door control circulation unit |
| CN111862294B (en) * | 2020-07-31 | 2024-03-01 | 天津大学 | Hand-painted 3D building automatic coloring network device and method based on ArcGAN network |
| CN112215017B (en) * | 2020-10-22 | 2022-04-29 | 内蒙古工业大学 | Mongolian Chinese machine translation method based on pseudo parallel corpus construction |
| CN112507733B (en) * | 2020-11-06 | 2023-04-18 | 昆明理工大学 | Dependency graph network-based Hanyue neural machine translation method |
| CN112329760B (en) * | 2020-11-17 | 2021-12-21 | 内蒙古工业大学 | Method for recognizing and translating Mongolian in printed form from end to end based on space transformation network |
| CN112613326B (en) * | 2020-12-18 | 2022-11-08 | 北京理工大学 | Tibetan language neural machine translation method fusing syntactic structure |
| CN112633018B (en) * | 2020-12-28 | 2022-04-15 | 内蒙古工业大学 | Mongolian Chinese neural machine translation method based on data enhancement |
| CN112989845B (en) * | 2021-03-02 | 2023-01-10 | 北京理工大学 | Chapter-level neural machine translation method and system based on routing algorithm |
| CN113065432A (en) * | 2021-03-23 | 2021-07-02 | 内蒙古工业大学 | Handwritten Mongolian recognition method based on data enhancement and ECA-Net |
| CN113343672B (en) * | 2021-06-21 | 2022-12-16 | 哈尔滨工业大学 | Unsupervised bilingual dictionary construction method based on corpus merging |
| CN113642341A (en) * | 2021-06-30 | 2021-11-12 | 深译信息科技(横琴)有限公司 | Deep confrontation generation method for solving scarcity of medical text data |
| CN113657124B (en) * | 2021-07-14 | 2023-06-30 | 内蒙古工业大学 | Multi-mode Mongolian translation method based on cyclic common attention transducer |
| CN113505775B (en) * | 2021-07-15 | 2024-05-14 | 大连民族大学 | Character positioning-based full-text word recognition method |
| CN113538506A (en) * | 2021-07-23 | 2021-10-22 | 陕西师范大学 | Pedestrian trajectory prediction method based on global dynamic scene information depth modeling |
| CN113611293B (en) * | 2021-08-19 | 2022-10-11 | 内蒙古工业大学 | Mongolian data set expansion method |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107368475B (en) * | 2017-07-18 | 2021-06-04 | 中译语通科技股份有限公司 | Machine translation method and system based on generation of antagonistic neural network |
| CN107967262B (en) * | 2017-11-02 | 2018-10-30 | 内蒙古工业大学 | A kind of neural network illiteracy Chinese machine translation method |
| US10776581B2 (en) * | 2018-02-09 | 2020-09-15 | Salesforce.Com, Inc. | Multitask learning as question answering |
| CN108563640A (en) * | 2018-04-24 | 2018-09-21 | 中译语通科技股份有限公司 | A kind of multilingual pair of neural network machine interpretation method and system |
| CN108897740A (en) * | 2018-05-07 | 2018-11-27 | 内蒙古工业大学 | A kind of illiteracy Chinese machine translation method based on confrontation neural network |
| CN108932232A (en) * | 2018-05-07 | 2018-12-04 | 内蒙古工业大学 | A kind of illiteracy Chinese inter-translation method based on LSTM neural network |
| CN109492232A (en) * | 2018-10-22 | 2019-03-19 | 内蒙古工业大学 | A kind of illiteracy Chinese machine translation method of the enhancing semantic feature information based on Transformer |
| CN109783827B (en) * | 2019-01-31 | 2023-02-10 | 沈阳雅译网络技术有限公司 | Deep neural machine translation method based on dynamic linear polymerization |
-
2019
- 2019-08-29 CN CN201910807617.7A patent/CN110598221B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110598221A (en) | 2019-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110598221B (en) | Method for improving translation quality of Mongolian Chinese by constructing Mongolian Chinese parallel corpus by using generated confrontation network | |
| CN110334361B (en) | Neural machine translation method for Chinese language | |
| CN110633683B (en) | Chinese sentence-level lip language recognition method combining DenseNet and resBi-LSTM | |
| Liu et al. | A recursive recurrent neural network for statistical machine translation | |
| CN107967262A (en) | A kind of neutral net covers Chinese machine translation method | |
| CN111160050A (en) | Chapter-level neural machine translation method based on context memory network | |
| CN107357789A (en) | Merge the neural machine translation method of multi-lingual coding information | |
| CN113158665A (en) | Method for generating text abstract and generating bidirectional corpus-based improved dialog text | |
| CN111858932A (en) | Multiple-feature Chinese and English emotion classification method and system based on Transformer | |
| CN111401079A (en) | Training method and device of neural network machine translation model and storage medium | |
| CN108920472A (en) | A kind of emerging system and method for the machine translation system based on deep learning | |
| CN110033008A (en) | A kind of iamge description generation method concluded based on modal transformation and text | |
| CN115471851A (en) | Burma language image text recognition method and device fused with double attention mechanism | |
| CN114627162A (en) | Multimodal dense video description method based on video context information fusion | |
| CN111444730A (en) | Data enhancement Weihan machine translation system training method and device based on Transformer model | |
| CN114238649A (en) | Common sense concept enhanced language model pre-training method | |
| Liu | Neural question generation based on Seq2Seq | |
| Xiu et al. | A handwritten Chinese text recognizer applying multi-level multimodal fusion network | |
| CN115719072A (en) | Chapter-level neural machine translation method and system based on mask mechanism | |
| CN116628174A (en) | End-to-end relation extraction method and system for fusing entity and relation information | |
| CN113887251B (en) | Mongolian machine translation method combining Meta-KD framework and fine granularity compression | |
| CN110502759A (en) | The Chinese for incorporating classified dictionary gets over the outer word treatment method of hybrid network nerve machine translation set | |
| CN114595700A (en) | Zero-pronoun and chapter information fused Hanyue neural machine translation method | |
| CN116662924A (en) | Aspect-level multi-mode emotion analysis method based on dual-channel and attention mechanism | |
| CN110825869A (en) | Text abstract generating method of variation generation decoder based on copying mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |