KR20190053028A - Neural machine translation apparatus and method of operation thereof based on neural network learning using constraint strength control layer - Google Patents

Abstract

Disclosed is an operation method of a neural network machine translation apparatus. The operation method of a neural network machine translation apparatus using a constraint influence control layer comprises the steps of: generating a first conceptual density vector for an original text; generating a second conceptual density vector for a translated text corresponding to the original text; determining a distance between the generated first conceptual density vector and the generated second conceptual density vector; determining a predicted text for the translated text based on the original text and determining a cross entropy of the determined predicted text and the translated text; and training a neural network based on a loss function obtained using the determined distance and the determined cross entropy, wherein the first conceptual density vector is obtained by performing affine transformation on the original text and the second conceptual density vector is obtained using the number of unique lexical tokens of the translated text and an embedding vector length.

Description

TECHNICAL FIELD [0001] The present invention relates to a neural network machine translation apparatus based on neural network learning using a constraint influence control layer, and to a neural network machine translation apparatus using the constraint influence control layer,

The present invention relates to a neural network learning method using a constraint influence control layer in a neural network machine translation apparatus, and more particularly, to a neural network learning method using a constraint influence control layer in a neural network machine translation apparatus, To a neural network learning method of a neural network machine translation apparatus.

Cyclic neural network is a type of in-depth neural network that consists of hidden state, input data, and a network to which result nodes are connected. Structured memories are passed through the hidden state, It is useful for understanding dynamic patterns and characteristics inherent in time series data. However, since the cyclic neural network affects the concealed state of the current time in the learning process, the cumulative value increases exponentially as the learning progresses, or converges to zero rapidly. Therefore, Based short-term memory RNN (RNN-LSTM) has been proposed. This RNN-LSTM architecture can prevent divergence or convergence problems by filtering three hidden gates and memory for storing dynamic information, acquiring only important information, and calculating the output value.

Neural machine translation (NMT) refers to the field of research for creating models for machine translation based on various artificial neural networks. RNN-LSTM is a useful technology in the field of neural network machine translation. And so on. For example, a technique for learning a translation sentence in RNN-LSTM is an encoder-decoder mechanism structure that generally receives an input sentence and generates an output sentence. Based on this, an attention model, bidirectional model model or an input feeding model have been proposed.

The neural network machine translation model based on RNN-LSTM generally includes an encoder for expressing an input sentence as a single vector and a decoder for generating an output sentence using an input sentence. Specifically, the encoder compresses a single input sentence vector using the input original sentence input sentence. Then, the sentence vector generated by the encoder is transmitted to the decoder, and the decoder generates elements such as words and characters constituting the band sentence, one at each time point. That is, the decoder receives the abstract information of the sentence composed of the original language by the encoder as a condition variable (for example, a conditional language model), and obtains the band language based on the sentence structure of the learned band language And selects the element having the highest probability among the candidates satisfying the input condition.

1 is a diagram showing the operation of an encoder and a decoder in a short-term memory based cyclic neural network (RNN-LSTM) according to an embodiment. Referring to FIG. 1, a decoder includes a sentence end reservation word The word is generated until the word <eos> appears, so that words generated by the decoder have a probability of occurrence according to a context vector, which means conditions that are passed from a previous point in time.

Therefore, the RNN-LSTM-based neural network machine translation model can accept all the information for generating the correct output sentence when it is ideally learned, but in reality, when the size of the model becomes very large, The efficiency may be reduced. In particular, after the input information of the original text is aggregated into a single input sentence vector, an output vocabulary with the highest probability is determined as greedy, and information about the relation between the input vocabulary column and the output vocabulary of the original text is directly determined There are not many ways to deliver it, so there is a problem that no models are learned unless there are separate models to judge the final generated sentence.

SUMMARY OF THE INVENTION The present invention provides a neural network machine translation apparatus and an operation method thereof for improving learning efficiency and accuracy of translation of a neural network machine translation.

The technical objects to be achieved by the present disclosure are not limited to the above-mentioned technical subjects, and other technical subjects which are not mentioned are to be clearly understood from the following description to those skilled in the art It will be possible.

A method of operating in a neural network machine translation apparatus based on neural network learning using a contract condition influence control layer, the method comprising: generating a first conceptual density vector for a text; Generating a second conceptual density vector for a band sentence corresponding to the original text; Determining a distance between the generated first conceptual density vector and the generated second conceptual density vector; Determining a prediction statement for the bandwidth statement based on the original statement, and determining a cross-entropy of the determined prediction statement and the bandwidth query; And learning the neural network based on the determined distance and the loss function obtained using the determined cross entropy, wherein the first concept density vector is obtained by performing an affine transformation on the original text, And the second concept density vector is obtained by using the number of lexical tokens and the embedding vector length of the bandwidth query.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present disclosure, a neural network machine translation apparatus and method for improving learning efficiency and improving translation accuracy of a neural network machine translation can be provided.

Further, according to the present disclosure, it is possible to reduce the possibility of generating unnecessary output sentences due to additional constraints in neural network learning, and by constraining the constraint vector of the output sentence, constraint conditions are always prevented from being satisfied, The feedback required for learning can be transmitted more strongly.

In addition, according to the present disclosure, it is possible to control learning influence of constraints by adjusting the size of a network added in an RNN-LSTM-based translation model, and further constraints on input and output RNN-LSTM model with more general constraints can be constructed.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below will be.

1 is a diagram illustrating operations of an encoder and a decoder in a short-term memory based cyclic neural network according to an embodiment.
2 and 3 are block diagrams showing a configuration of a neural network machine translation apparatus using a constraint influence control layer according to an embodiment.
4 is a block diagram illustrating a configuration of a neural network machine translation apparatus using a constraint influence control layer according to another embodiment.
5 is a flowchart illustrating a method of operating a neural network machine translation apparatus using a constraint influence control layer according to an embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily understood by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. Parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.

In the present disclosure, when an element is referred to as being "connected", "coupled", or "connected" to another element, it is understood that not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising " or " having " another element, it is meant to include not only excluding another element but also another element .

In the present disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements, etc. unless specifically stated otherwise. Thus, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly a second component in one embodiment may be referred to as a first component .

In the present disclosure, the components that are distinguished from each other are intended to clearly illustrate each feature and do not necessarily mean that components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of this disclosure.

In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Thus, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Also, embodiments that include other elements in addition to the elements described in the various embodiments are also included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

2 and 3 are block diagrams showing a configuration of a neural network machine translation apparatus using a constraint influence control layer according to an embodiment.

2, a neural network machine translation apparatus 200 according to an embodiment may include an input unit 210, a control unit 220, and a storage unit 230. It should be noted, however, that this shows only some components necessary for explaining the present embodiment, and the elements included in the neural network machine translation apparatus 200 are not limited to the above-described examples.

For example, referring to FIG. 3, the neural network machine translation apparatus 300 may further include an encoder 310, a decoder 320, a constraint influence control layer 330, and the like. The neural network machine translation apparatus 300 of FIG. 3 may correspond to the neural network machine translation apparatus 200 of FIG.

The deep neural network is an artificial neural network composed of several hidden layers between the input layer and the output layer. It can model complex nonlinear relations through many hidden layers. By increasing the number of layers in this way, The structure is called deep learning. Deep learning learns a very large amount of data, and when new data is input, it chooses the highest answer with probability based on the learning result, so it can adaptively operate even if the environment of the image changes. Since the characteristic parameters are automatically detected in the course of learning, there is an increasing tendency to utilize them in the field of machinery recently. Representative examples of deep learning include convolutional neural network, recurrent neural network (RNN), autoencoder, and the like.

The neural network machine translation apparatus 200 can learn the neural network by analyzing the distance between the concept density vector for the original text and the concept density vector for the sentence.

According to one embodiment, a neural network machine translation apparatus 200 based on a short-term memory RNN (RNN-LSTM) using a short-term memory generates a first conceptual density vector for the original text, Determining a distance between the generated first conceptual density vector and the generated second conceptual density vector, determining a predictive statement for a band sentence based on the original text, It is possible to determine the cross entropy of the door and the bandwidth inquiry and to learn the neural network based on the obtained loss function using the determined distance and the determined cross entropy.

Here, the concept density vector is a vector for expressing the amount of abstract information possessed by the sentence vector. The concept density vector expresses the amount of information included in the original text input to the encoder as a concept density vector, and a correct concept density The word generation is induced to express the vector so that various possibilities that are unnecessarily considered in word selection can be excluded. The first conceptual density vector according to an embodiment is obtained by performing an affine transformation on the original text, and the second conceptual density vector according to an embodiment is obtained by multiplying the number of the unique vocabulary tokens of the band statement and the embedding vector length &Lt; / RTI &gt;

Therefore, the neural network machine translation apparatus based on the cyclic neural network using the memory of the present disclosure can add a network that can control the application strength of the constraint condition in the learning process, and can also remove the network related to the constraint condition in the prediction sentence generation process The structure of a general neural network machine translation device can be utilized.

2, an input unit 210 receives text, an image sequence, audio (for example, voice, music, etc.) and additional information (for example, speech) from the outside of the neural network machine translation apparatus 200 under the control of the control unit 220 For example, EPG, etc.).

The input unit 110 according to an exemplary embodiment may receive a sentence sequence for learning the automatic neural network translation system of the encoder-decoder mechanism.

The control unit 220 controls the overall operation of the neural network machine translation apparatus 200 and the signal flow between the internal components of the neural network machine translation apparatus 200 and performs processing of data. The control unit 220 may use various data stored in the storage unit 230 and may execute various applications when the input unit of the user is satisfied or predetermined conditions are satisfied. 2 may further include the encoder 310, the decoder 320, and the constraint influence control layer 330 shown in FIG. 3. However, the controller 220 of FIG. 2 may include only the components necessary for explaining the present embodiment The components included in the control unit 220 are not limited to the above-described examples.

The controller 220 generates a first conceptual density vector for the original text, generates a second conceptual density vector for a band sentence corresponding to the original text, 2 determine the distance of the concept density vector, determine the predictive statement for the band sentence based on the original text, determine the determined cross-entropy of the predictive sentence and the band statement, and determine the loss function obtained using the determined distance and the determined cross entropy To learn neural networks.

The storage unit 230 may store various data, programs or applications for driving and controlling the neural network machine translation apparatus 200 under the control of the control unit 220. [

The storage unit 230 may store data such as a first conceptual density vector, a second conceptual density vector, a distance between the first conceptual density vector and a second conceptual density vector, a cross entropy, or a loss function.

The operation of the neural network machine translation apparatus based on the neural network learning using the constraint influence control layer will be described with reference to FIG.

Referring to FIG. 3, the neural network machine translation apparatus 200 according to an embodiment of the present invention performs neural network learning based on an encoder-decoder mechanism of a short-term memory RNN (RNN-LSTM) It is a structure that adds a layer to control.

The constraint strength control layer 330 according to an exemplary embodiment of the present invention includes a structure of a linear regression model that is mainly used in a deep layer neural network, For the purpose of learning, a concept density vector (Vs) for the original text will be referred to as a first concept density vector for convenience, and a target concept density vector ), The distance between the first concept density vector and the second concept density vector, and the predicted sentence and the cross entropy of the query are used.

Specifically, the first conceptual density vector according to an embodiment receives the sentence vector Cs output from the encoder 310 as input, and uses a weight matrix (We) and a bias vector (be) Can be obtained by performing an affine transformation (Vs = Ve? Cs + be). Here, the sentence vector Cs can be generated by combining the output of the top layer of the encoder 31 by 1 to t, which is the length of the input lexical length t.

In addition, the second concept density vector according to an embodiment can be obtained by using the number of unique vocabulary tokens of the bandwidth query and the length of the embedding vector, for example, as follows. In a general neural network automatic translation learning method, each lexical token must be converted into a one-dimensional vector of a floating point format. The portion performing this operation is called an embedding matrix. The embedded matrix has a size corresponding to the number of unique vocabulary tokens of the query word x the length of the embedding vector. In the learning phase, for example, if there is a band word 'I have an apple.', The number of unique vocabulary tokens is 5, The output embedding vector transformed by an embedding matrix of length 6 is as follows.

I - > 0.1 0.5 0.23? 0.12? 0.01 0.89

have -> -0.1? 0.21 0.17 0.15 1.13 0.68

an -> -0.87? 0.12? 1.3 1.5 0.12 1.0

apple -> -1.52 1.89 0.25 0.99 0.13? 1.21

. - > 0.001 0.52? 0.02? 0.81? 1.0 0.23

If the embedded vector is replaced with an embedded vector based on the respective lexical column, the second conceptual density vector may be obtained (340) by adding the sum of the vector elements. Therefore, the second concept density vector for the band phrase 'I have an apple.'

The second concept density vector for the bandwidth statement = -2.389 2.58? 0.67 1.71 0.37 1.59

The distance 350 between the concept density vector for the original text and the conceptual density vector for the band text according to an exemplary embodiment may be an Euclidean distance of two vectors, but is not limited thereto. May include mathematical methods that can be used. Using the Euclidean distance, the distance between the concept density vector (Vs) for the original text and the concept density vector (Vt) for the band sentence can be expressed by Equation (1).

If the concept density vectors for the original sentences and the band sentences are respectively determined, the cross entropy c (?,?) Between the predicted sentences finally output from the decoder 320 and the band sentences, D) can be determined.

In the conventional neural network automatic translation learning step, the neural network can be learned in a direction in which the cross entropy between the predicted sentence and the band sentence is minimized. However, the neural network machine translation apparatus of the present disclosure is capable of learning the cross entropy between the predicted sentence and the band sentence, The neural network can be learned (360) such that the loss function (?, D) combining the distance between the concept density vectors is minimized. The loss function according to one embodiment can be expressed by Equation (2).

According to one embodiment, the loss function can be defined as a linear combination of the distance between the cross entropy and the two concept density vectors as in equation (2), but is not limited to this, Can be changed.

Therefore, by updating the neural network using the loss function, it is possible to narrow the distance between the concept density vectors for the original sentence and the band sentence, which brings the effect of bringing closer meaning to each language.

4 is a block diagram illustrating a configuration of a neural network machine translation apparatus using a constraint influence control layer according to another embodiment.

According to one embodiment, when automatic learning is predicted (or performed) by using a neural network model whose learning is completed and completed, the neural network machine translation apparatus of the present disclosure does not perform a concept density vector generation process in a learning process, 4, the neural network machine translation apparatus 400 does not include the constraint influence control layer or the like in FIG. 3 .

5 is a flowchart illustrating a method of operating a neural network machine translation apparatus using a constraint influence control layer according to an embodiment.

In step 500, the neural network machine translation device may generate a first conceptual density vector for the original text. According to an embodiment, the first concept density vector is obtained by performing an affine transformation using a weight matrix We and a bias vector be by receiving a sentence vector by an encoder Can

In operation 510, the neural network machine translation apparatus may generate a second concept density vector for a band sentence corresponding to the original text. According to an embodiment, the second concept density vector may be obtained using the number of unique lexical tokens of the query string and the length of the embedding vector.

In step 520, the neural network machine translation apparatus can determine the distance between the generated first concept density vector and the generated second concept density vector. According to one embodiment, the distance between the first conceptual density vector and the second conceptual density vector can be determined using the Euclidean distance.

In operation 530, the neural network machine translation apparatus can determine a predicted sentence for the band sentence based on the original sentence, and determine the determined crossed entropy of the predicted sentence and the band sentence.

In operation 540, the neural network machine translation apparatus can learn the neural network based on the determined distance and the loss function obtained using the determined cross entropy. According to one embodiment, the loss function can be obtained by linear combination of the distance between the cross entropy and the two concept density vectors.

1 to 5, a neural network learning method using a constraint influence control layer in a neural network machine translation apparatus according to an embodiment of the present disclosure has been described.

According to the present disclosure, a neural network machine translation apparatus and an operation method thereof for improving learning efficiency and accuracy of translation of a neural network machine translation can be provided.

Further, according to the present disclosure, it is possible to reduce the possibility of generating unnecessary output sentences due to additional constraints in neural network learning, and by constraining the constraint vector of the output sentence, constraint conditions are always prevented from being satisfied, The feedback required for learning can be transmitted more strongly.

In addition, according to the present disclosure, it is possible to control learning influence of constraints by adjusting the size of a network added in an RNN-LSTM-based translation model, and further constraints on input and output RNN-LSTM model with more general constraints can be constructed.

On the other hand, according to one aspect of the present disclosure, software or a computer-readable medium having executable instructions for performing the method of operation of the neural network machine translation apparatus may be provided. The executable instructions comprising the steps of generating a first conceptual density vector for the original text, generating a second conceptual density vector for the band sentence corresponding to the original text, generating a second conceptual density vector for the generated second conceptual density vector, Determining a distance to the density vector, determining a predictive statement for the band sentence based on the original text, determining the determined cross-entropy of the predictive statement and the band statement, and determining a loss function obtained using the determined distance and the determined cross entropy Wherein the first concept density vector is obtained by performing an affine transformation on the original text and the second concept density vector is obtained using the number of unique lexical tokens of the query string and the embedding vector length. can do.

Although the exemplary methods of this disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the disclosure are not intended to be all-inclusive and are intended to be illustrative of the typical aspects of the disclosure, and the features described in the various embodiments may be applied independently or in a combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is to be accorded the broadest interpretation as understanding of the principles of the invention, as well as software or machine-executable instructions (e.g., operating system, applications, firmware, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

300: Neural network machine translation device
310: encoder
320: decoder
330 constraint influence control layer

Claims (1)

A method of operating a neural network machine translation apparatus based on neural network learning using a contract condition influence control layer,
Generating a first conceptual density vector for the original text;
Generating a second conceptual density vector for a band sentence corresponding to the original text;
Determining a distance between the generated first conceptual density vector and the generated second conceptual density vector;
Determining a prediction statement for the bandwidth statement based on the original statement, and determining a cross-entropy of the determined prediction statement and the bandwidth query; And
Learning the neural network based on the determined distance and the loss function obtained using the determined cross entropy,
Wherein the first conceptual density vector is obtained by performing an affine transformation on the original text and the second conceptual density vector is obtained using the number of unique lexical tokens of the query and the embedding vector length. A method of operating a neural network machine translation device.

Images