CN110688860A - A Weight Allocation Method Based on Transformer Multiple Attention Mechanisms - Google Patents

Abstract

The invention discloses a weight distribution method based on multiple attention mechanisms of a transformer; the method comprises the following steps: the input to the attention mechanism is the word vectors in the target and source languages, and the output is an alignment tensor. Multiple alignment tensor outputs may be output using multiple attention mechanism functions, and each output is different due to random parametric variations in the calculation process. All attention mechanism models are put into operation, and various attention mechanism outputs are subjected to regularization calculation to approach to the optimal output. The regularization calculation method determines that the obtained value does not deviate from the optimal value too far, the optimality of each attention model is also preserved, and if the experimental effect of one attention model is excellent, the weight function of the model is increased to increase the influence of the model on final output, so that the translation effect is improved.

Description

A Weight Allocation Method Based on Transformer Multiple Attention Mechanisms

technical field

The present invention relates to the related field of neural machine translation, in particular to a weight distribution method based on multiple attention mechanisms of transformers.

Background technique

Neural network machine translation is a machine translation method proposed in recent years. Compared with traditional statistical machine translation, neural network machine translation can train a neural network that can map from one sequence to another, and the output can be a variable-length sequence, which is useful in translation, dialogue and text generalization. can get very good performance. Neural network machine translation is actually an encoding-decoding system. The encoding encodes the source language sequence, extracts the information in the source language, and then converts this information into another language, the target language, through decoding. Translation of languages.

When the model generates output, it will generate an attention range to indicate which parts of the input sequence to focus on when outputting next, and then generate the next output according to the area of interest, and so on. There are certain similarities between the attention mechanism and some behavioral characteristics of people. When people read a paragraph, they usually only pay attention to words with information, not all words, that is, people will give attention weight to each word. different. Although the attention mechanism model increases the training difficulty of the model, it improves the effect of text generation. In this patent, we are making improvements in the attention mechanism function.

Since the neural machine translation system was proposed in 2013, with the rapid development of computer computing power, neural machine translation has also developed rapidly, and successively proposed the seq-seq model, the transformer model, etc. In 2013, Nal Kalchbrenner and Phil Blunsom proposed a novel end-to-end encoder-decoder structure for machine translation [4]. The model can use a convolutional neural network (CNN) to encode a given piece of source text into a continuous vector, and then use a recurrent neural network (RNN) as a decoder to convert this state vector into the target language. In 2017, Google released a new machine learning model, Transformer, which far outperformed existing algorithms in machine translation and other language understanding tasks.

The traditional technology has the following technical problems:

In the process of attention mechanism function alignment, the existing framework first calculates the similarity of the two input sentence word vectors, and then performs a series of calculations to obtain the alignment function. Each alignment function will output once during calculation, and then use the output of this time as the next input for calculation. The calculation of such a single thread is likely to lead to the accumulation of errors. We introduce the weight distribution of multiple attention mechanisms in order to find the optimal solution in multiple computing processes. achieve the best translation effect.

SUMMARY OF THE INVENTION

Therefore, in order to solve the above deficiencies, the present invention provides a weight distribution method based on multiple attention mechanisms of the transformer; it is applied to the transformer frame model based on the attention mechanism. Including: the input of the attention mechanism is the target language of the target language and the word vector of the source language, and the output is an alignment tensor. Using multiple attention mechanism functions can output multiple aligned tensor outputs, and each output is different due to random parameter changes during computation. Many attention mechanism models have been proposed, such as self-attention mechanism, multi-head attention mechanism, full attention mechanism, local attention mechanism, etc. Each different attention mechanism has different outputs and characteristics, we will All attention mechanism models are put into operation, and the outputs of various attention mechanisms are regularized to approximate the optimal output.

The present invention is implemented in this way, constructs a weight distribution method based on multiple attention mechanisms of transformers, and applies it to the transformer model based on attention mechanisms, and is characterized in that it includes the following steps:

Step 1: In the transformer model, select the best model output for the application scenario.

Step 2: Initialize the value of the weight sequence δ, the weight sequence δ is a random number in the first calculation, and δ ₁ +δ ₂ +....+δ _i =1;

Step 3: Regularize the output of each model and calculate the center point of each output (the closest point to all values), through the calculation formula fin_out=δ ₁ O ₁ +δ ₂ O ₂ +δ ₃ O ₃ .. .....+δ _i O _i calculates the optimal matching value as the final output; where δ ₁ +δ ₂ +....+δ _i =1 and δ _i is the weight parameter we set; O _i is The output of various attention models;

Step 4: Substitute the final output into the subsequent operation, and calculate the difference between the loss function and the previous training. If the loss function decreases, increase the proportion of the sequence near the center point in the delta; if the loss function increases, increase the sequence in the delta sequence. The proportion of the sequence farthest from the center point, the whole process strictly follows the rule of δ ₁ +δ ₂ +....+δ _i =1;

Step 5: Iterative calculation in multiple loops, and finally determine the optimal weight sequence δ.

The invention has the following advantages: the invention discloses a weight distribution method based on multiple attention mechanisms of the transformer. Applied to the transformer framework model based on the attention mechanism. Including: the input of the attention mechanism is the target language of the target language and the word vector of the source language, and the output is an alignment tensor. Using multiple attention mechanism functions can output multiple aligned tensor outputs, and each output is different due to random parameter changes during computation. Many attention mechanism models have been proposed, such as self-attention mechanism, multi-head attention mechanism, full attention mechanism, local attention mechanism, etc. Each different attention mechanism has different outputs and characteristics, we will All attention mechanism models are put into operation, and the outputs of various attention mechanisms are regularized to approximate the optimal output. Using the formula: fin_out=δ ₁ O ₁ +δ ₂ O ₂ +δ ₃ O ₃ .......+δ _i O _i where δ ₁ +δ ₂ +....+δ _i =1 and δ _i is the weight parameter we set. O _i is the output of various attention models. This regularization calculation method determines that the obtained value will not deviate too far from the optimal value, and also preserves the optimality of each attention model. If it is an experiment of an attention model If the effect is excellent, then increase the weight function of the model to increase the influence of the model on the final output, thereby improving the translation effect.

Detailed ways

The present invention will be described in detail below, and the technical solutions in the embodiments of the present invention will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present invention provides a weight distribution method based on multiple attention mechanisms of transformers by improving. Applied to the transformer framework model based on the attention mechanism.

Introduction to the transformer framework:

Encoder: consists of 6 identical layers, each layer contains two sub-layers. The first sub-layer is a multi-head attention layer and then a simple fully connected layer. Residual connections and normalization are added to each sub-layer).

Decoder: It consists of 6 identical layers, but the layer here is different from the encoder. The layer here contains three sub-layers, including a self-attention layer, and the encoder-decoder attentionlayer is finally a fully connected layer. The first two sub-layers are based on the multi-head attention layer. A special point here is masking. The function of masking is to prevent future output words from being used during training.

Attention Model:

Although the encoder-decoder model is very classic, it is also very limited. The bigger limitation is that the connection between encoding and decoding is a fixed-length semantic vector C. That is, the encoder compresses the entire sequence of information into a fixed-length vector. However, there are two drawbacks in this way. First, the semantic vector cannot fully represent the information of the entire sequence. Second, the information carried by the first input content will be diluted by the later input information. The longer the input sequence, the more severe this phenomenon is. This makes it impossible to obtain enough information of the input sequence at the beginning of decoding, so the accuracy of decoding will be discounted to a certain extent.

To solve the above problems, attention model was proposed one year after Seq2Seq appeared. When the model generates output, it will generate an attention range to indicate which parts of the input sequence to focus on when outputting next, and then generate the next output according to the area of interest, and so on. Attention has certain similarities with some behavioral characteristics of people. When people read a paragraph, they usually only pay attention to words with information, not all words, that is, people will give each word a different attention weight. . Although the attention model increases the training difficulty of the model, it improves the effect of text generation.

The first step is to generate the semantic vector of the moment:

s _t =tanh(W[s _t-1 , y _t-1 ])

The second step is to pass the hidden layer information and predict:

Many attention mechanism models have been proposed, such as self-attention mechanism, multi-head attention mechanism, total attention mechanism, local attention mechanism, etc. Each different attention mechanism has different outputs and characteristics.

The improvement here is to modify the attention function.

Here, all attention mechanism models are put into operation, and the outputs of various attention mechanisms are regularized to approximate the optimal output. Using the formula: fin_out=δ ₁ O ₁ +δ ₂ O ₂ +δ ₃ O ₃ .......+δ _i O _i where δ ₁ +δ ₂ +....+δ _i =1 and δ _i is the weight parameter we set. O _i is the output of various attention models. This regularization calculation method determines that the obtained values will not deviate too far from the optimal value, and also preserves the optimality of each attention model. The specific implementation steps of the present invention are as follows;

Step 1: In the transformer model, select the best model output for the application scenario.

Step 2: Initialize the value of the weight sequence δ, the weight sequence δ is a random number in the first calculation, and δ ₁ +δ ₂ +....+δ _i =1;

Step 3: Regularize the output of each model and calculate the center point of each output (the closest point to all values), through the calculation formula fin_out=δ ₁ O ₁ +δ ₂ O ₂ +δ ₃ O ₃ .. .....+δ _i O _i to calculate the optimal matching value as the final output.

Step 4: Substitute the final output into the subsequent operation, and calculate the difference between the loss function and the previous training. If the loss function decreases, increase the proportion of the sequence near the center point in the delta; if the loss function increases, increase the sequence in the delta sequence. The proportion of the sequence farthest from the center point, the whole process strictly obeys the rule of δ ₁ +δ ₂ +....+δ _i =1.

Step 5: Iterative calculation in multiple loops, and finally determine the optimal weight sequence δ.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (1)

1. a weight distribution method based on multiple attention mechanisms of transformer, applied in the transformer model based on attention mechanism, is characterized in that; Comprise the following steps: Step 1: In the transformer model, select the best model output for the application scenario. Step 2: Initialize the value of the weight sequence δ, the weight sequence δ is a random number in the first calculation, and δ ₁ +δ ₂ +....+δ _i =1; Step 3: Regularize the output of each model and calculate the center point of each output (the closest point to all values), through the calculation formula fin_out=δ ₁ O ₁ +δ ₂ O ₂ +δ ₃ O ₃ .. .....+δ _i O _i calculates the optimal matching value as the final output; where δ ₁ +δ ₂ +....+δ _i =1 and δ _i is the weight parameter we set; O _i is The output of various attention models; Step 4: Substitute the final output into the subsequent operation, and calculate the difference between the loss function and the previous training. If the loss function decreases, increase the proportion of the sequence near the center point in the delta; if the loss function increases, increase the sequence in the delta sequence. The proportion of the sequence farthest from the center point, the whole process strictly follows the rule of δ ₁ +δ ₂ +....+δ _i =1; Step 5: Iterative calculation in multiple loops, and finally determine the optimal weight sequence δ.