JP7262231B2 - learning devices and programs - Google Patents

Description

The present invention relates to a translation device, a learning device, and a program.

In machine translation using machine learning by neural networks, the more learning data there is, the more the translation accuracy improves. In order to improve translation accuracy, it is desirable to prepare as much learning data as possible, but this requires high costs. In order to increase the amount of learning data, it is conceivable to combine data of different domains for learning.

In the technique described in Non-Patent Document 1, machine learning is performed by combining learning data belonging to a plurality of domains, and then re-learning is performed using only data belonging to a target domain.

In the technique described in Non-Patent Document 2, part of a model trained with data of different domains is shared in the classification problem.

Patent Document 1 describes a feature weight optimization device for automatic translation. In this feature weight optimization device, a feature weight optimization unit 278 uses a plurality of domain development sets 212, uses features obtained from a plurality of domain-specific statistical models 272 and a general statistical model 274, or uses linear interpolation of their logarithms to Optimizing each feature weight for natural language translation. The feature weight optimization unit 278 has a domain-specific feature storage area provided for each domain development set 212 . Each of which contains a first area for storing features of a generic statistical model, a plurality of second areas for storing features from multiple domain development sets, and loss function values used for weight optimization. and a third area for storing.

JP 2017-151804 A

Rico Sennrich, Barry Haddow, Alexandra Birch, Improving neural machine translation models with monolingual data, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, p.86-96, Berlin, Germany, August 7-12, 2016. Young-Bum Kim, Karl Stratos, Ruhi Sarikaya, Frustratingly Easy Neural Domain Adaptation, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 387-396, Osaka, Japan, December 11-17 2016.

Conventional techniques have the problem that knowledge cannot be shared between translation models of multiple domains. In the technology described in Non-Patent Document 1, learning data with different properties (for example, learning data in different domains such as sentences in the travel conversation field and sentences in patent documents) are combined for learning, so the translation accuracy is high. There were times when I fell. In Non-Patent Document 2, a part of the model trained with data of different domains is shared in the classification problem, but there is a problem that it cannot be applied to generation problems such as translation processing.

In machine translation, it is known that the more learning data, the higher the accuracy, and it is inefficient to learn by separating the learning data for each domain. The present invention has been made in view of the above circumstances, and provides a translation device for sharing knowledge between translation models of a plurality of domains without completely separating learning data of different domains. , a learning device, and a program.

[1] In order to solve the above problems, a learning device according to an aspect of the present invention converts an input sentence belonging to the first domain into a source language based on encoding processing parameters in the first domain. a first encoder unit that performs an encoding process; a second encoder unit that performs an encoding process on an input sentence belonging to the second domain and in a source language based on parameters for the encoding process in the second domain; a shared encoder unit that encodes the input sentence belonging to either the first domain or the second domain based on encoding parameters shared between the first domain and the second domain; Based on a first semantic vector output as a result of encoding processing in one encoder unit, a common semantic vector output as a result of encoding processing in the shared encoder unit, and parameters of decoding processing in the first domain, A first decoder unit that generates an output sentence corresponding to the input sentence, a second semantic vector that is output as a result of encoding processing in the second encoder unit, and a second semantic vector that is output as a result of encoding processing in the shared encoder unit a second decoder unit that generates an output sentence corresponding to the input sentence based on a common semantic vector and parameters of the decoding process in the second domain; When a sentence pair, which is a pair of sentences in a language, is input as training data, an output sentence is generated based on the input sentence in the source language by the processing of the first encoder unit, the shared encoder unit, and the first decoder unit. is generated, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, parameters for encoding processing in the first encoder unit, parameters for encoding processing in the shared encoder unit, and the first Decoding parameters in the decoder unit are updated, and when a sentence pair that is a pair of sentences in the source language and the target language belonging to the second domain is input as learning data, based on the input sentence in the source language An output sentence is generated by processing of the second encoder unit, the shared encoder unit, and the second decoder unit, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, the second encoder unit section, the encoding process parameter in the shared encoder section, and the decoding process parameter in the second decoder section are updated.

[2] Further, according to one aspect of the present invention, in the above learning device, the element of the first semantic vector output as a result of encoding processing by the first encoder unit and the result of encoding processing by the shared encoder unit are The concatenated vector obtained by arranging the elements of the common semantic vector to be output is reduced in dimension based on the parameters of the dimension reduction processing in the first domain, and the first reduced dimension vector that is the result of the reduction in dimension is output. The first decoder unit is based on the first reduced-order vector output by the first order reduction unit and the parameters of the decoding process in the first domain to generate an output sentence corresponding to the input sentence, and based on the difference between the sentence in the target language of the sentence pair and the output sentence from the first decoder unit, It also updates the parameters of the dimensioning process.

[3] In addition, in the above learning device, an element of a second semantic vector output as a result of encoding processing by the second encoder unit and an element of the second semantic vector output as a result of encoding processing by the shared encoder unit are The concatenated vector obtained by arranging the elements of the common semantic vector to be output is reduced in dimension based on the parameters of the dimension reduction processing in the second domain, and the second reduced dimension vector that is the result of the reduction in dimension is output. and the second decoder unit is configured based on the second reduced-order vector output by the second order reduction unit and the parameters of the decoding process in the second domain. to generate an output sentence corresponding to the input sentence, and based on the difference between the sentence in the target language of the sentence pair and the output sentence from the second decoder unit, It also updates the parameters of the dimensioning process.

[4] Further, according to one aspect of the present invention, in the above learning device, a first semantic vector output as a result of encoding processing by the first encoder unit and a first semantic vector output as a result of encoding processing by the shared encoder unit a first orthogonal error calculation unit that calculates a first orthogonal error that is an orthogonal error with the common semantic vector, a second semantic vector output as a result of encoding processing by the second encoder unit, and a common encoder unit a second orthogonal error calculator that calculates a second orthogonal error that is an orthogonal error between the common semantic vector output as a result of the encoding process, and the sentence in the target language of the sentence pair and the first decoder unit Based on the first orthogonal error calculated by the first orthogonal error calculation unit as well as the difference from the output sentence output from, the encoding processing parameters in the first encoder unit and the encoding in the shared encoder unit updating parameters of processing and parameters of decoding processing in the first decoder unit, and updating the difference between the sentence in the target language of the sentence pair and the output sentence output from the second decoder unit, together with the second orthogonal Also based on the second orthogonal error calculated by the error calculation unit, parameters for encoding processing in the second encoder unit, parameters for encoding processing in the shared encoder unit, and parameters for decoding processing in the second decoder unit. to update the .

[5] A translation device according to an aspect of the present invention includes: a first encoder unit that encodes an input sentence in a source language based on encoding processing parameters in the first domain; A shared encoder unit that performs encoding processing on the input sentence based on shared encoding processing parameters, a first semantic vector that is output as a result of the encoding processing in the first encoder unit, and encoding in the shared encoder unit A first decoder section for generating an output sentence corresponding to the input sentence based on a common semantic vector output as a result of processing and parameters of the decoding process in the first domain.

[6] Further, in the translation device described above, an element of the first semantic vector output as a result of the encoding process by the first encoder unit and an element of the first semantic vector output as a result of the encoding process by the shared encoder unit are The concatenated vector obtained by arranging the elements of the common semantic vector to be output is reduced in dimension based on the parameters of the dimension reduction processing in the first domain, and the first reduced dimension vector that is the result of the reduction in dimension is output. The first decoder unit is based on the first reduced-order vector output by the first order reduction unit and the parameters of the decoding process in the first domain to generate an output sentence corresponding to the input sentence.

[7] Further, according to one aspect of the present invention, in the above translation device, the number of the other domains is one or more.

[8] Further, according to one aspect of the present invention, in the translation device described above, parameters for encoding processing in the first encoder unit, parameters for encoding processing in the shared encoder unit, parameters for decoding processing in the first decoder unit, A parameter is obtained by the processing of the learning device described in any one of [1] to [4] above.

[9] Further, according to one aspect of the present invention, a computer performs encoding processing of an input sentence belonging to the first domain and in a source language based on encoding processing parameters in the first domain. an encoder unit; a second encoder unit that encodes an input sentence belonging to the second domain and in a source language based on encoding processing parameters in the second domain; a shared encoder unit that performs encoding processing of the input sentence belonging to either the first domain or the second domain based on encoding processing parameters shared by the two domains; and encoding processing in the first encoder unit. corresponding to the input sentence based on a first semantic vector output as a result of, a common semantic vector output as a result of encoding processing in the shared encoder unit, and parameters of decoding processing in the first domain a first decoder unit that generates an output sentence; a second semantic vector that is output as a result of encoding processing in the second encoder unit; a common semantic vector that is output as a result of encoding processing in the shared encoder unit; and a second decoder unit that generates an output sentence corresponding to the input sentence based on parameters of the decoding process in the second domain, and a pair of source language and target language sentences belonging to the first domain When a certain sentence pair is input as learning data, an output sentence is generated based on the input sentence in the source language through processing by the first encoder unit, the shared encoder unit, and the first decoder unit, and the sentence is parameters for encoding processing in the first encoder unit, parameters for encoding processing in the shared encoder unit, and parameters for decoding processing in the first decoder unit based on the difference between the sentence in the paired target language and the output sentence. parameters, and when a sentence pair, which is a pair of sentences in the source language and the target language belonging to the second domain, is input as learning data, the second encoder unit and the An output sentence is generated by the processing of the shared encoder unit and the second decoder unit, and parameters for encoding processing in the second encoder unit based on the difference between the sentence in the target language of the sentence pair and the output sentence and a program for functioning as a learning device that updates encoding processing parameters in the shared encoder section and decoding processing parameters in the second decoder section.

[10] Further, according to one aspect of the present invention, a computer includes a first encoder unit that performs encoding processing of an input sentence in a source language based on encoding processing parameters in the first domain, the first domain and other A shared encoder unit that performs encoding processing of the input sentence based on encoding processing parameters shared by the domain, a first semantic vector that is output as a result of the encoding processing in the first encoder unit, and the shared encoder unit. and a first decoder unit that generates an output sentence corresponding to the input sentence based on a common semantic vector output as a result of the encoding process in and parameters of the decoding process in the first domain. It is a program for functioning as

According to the present invention, learning processing for sharing knowledge between different domains can be performed. In addition, translation processing can be performed based on knowledge (model) shared between different domains. By realizing such a mechanism for sharing knowledge, the amount of learning data can be increased, so that translation accuracy can be improved.

1 is a block diagram showing a schematic functional configuration of a translation device (learning device) according to a first embodiment of the present invention; FIG. FIG. 4 is a schematic diagram showing a model of encoding processing in each of the first encoder section, the second encoder section, and the shared encoder section according to the same embodiment; FIG. 5 is a schematic diagram showing a model of decoding processing in each of the first decoder section and the second decoder section according to the same embodiment; FIG. 5 is a schematic diagram showing the order reduction processing in each of a first order reduction unit and a second order reduction unit according to the same embodiment; 4 is a flow chart showing the procedure of learning processing of the translation device according to the same embodiment. 4 is a flow chart showing the procedure of translation processing of the translation device according to the same embodiment. FIG. 11 is a block diagram showing a schematic functional configuration of a translation device (learning device) according to a second embodiment; FIG. 11 is a block diagram showing a schematic functional configuration of a translation device (learning device) according to a third embodiment; FIG. 11 is a block diagram showing a schematic functional configuration of a translation device (learning device) according to a fourth embodiment;

[First embodiment]
Next, one embodiment of the present invention will be described with reference to the drawings. In this embodiment, the neural machine translation model is divided into a part shared between domains and a part used only within the domain. Parts shared between domains are shared by multiple machine translation systems for learning.

FIG. 1 is a block diagram showing a schematic functional configuration of a translation device according to this embodiment. The illustrated translation device 1 can also be regarded as a learning device for learning a translation model. As illustrated, the translation apparatus 1 includes a first input unit 11, a first encoder unit 12, a first dimension reduction unit 13, a first decoder unit 14, a first output unit 15, a second input It includes a unit 21 , a second encoder unit 22 , a second order reduction unit 23 , a second decoder unit 24 , a second output unit 25 and a shared encoder unit 31 . Each of these functional units can be realized by, for example, a computer and a program. In addition, each functional unit has storage means as required. The storage means is, for example, variables on the program or memory allocated by executing the program. Also, if necessary, non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used. Also, at least part of the function of each functional unit may be realized as a dedicated electronic circuit instead of a program.

Hereinafter, each of the first encoder section 12, the second encoder section 22, and the shared encoder section 31 may be simply referred to as an "encoder". Moreover, each of the first decoder section 14 and the second decoder section 24 may be simply referred to as a "decoder".

The first encoder section 12 , the first order reduction section 13 , the first decoder section 14 , and the shared encoder section 31 may be collectively called a first translation model section 17 . Also, the second encoder section 22 , the second order reduction section 23 , the second decoder section 24 , and the shared encoder section 31 may be collectively called a second translation model section 27 . The first translation model section 17 and the second translation model section 27 function as translation models for domains different from each other.

Here, a domain is a field to which a sentence to be translated belongs. For example, a domain can be a grouping of travel conversation sentences, patent sentences, broadcast caption sentences, newspaper sentences, news announcement sentences, and the like. Note that the domains listed above are only examples, and general domains are not limited to these examples. For convenience, the domain targeted by the first translation model unit 17 is called the first domain, and the domain targeted by the second translation model unit 27 is called the second domain. That is, translation apparatus 1 has a configuration in which translation models for two domains, the first domain and the second domain, are integrated.

The translation apparatus 1 outputs the result of translating an input sentence as an output sentence when performing translation processing. The translation device 1 can also function as a learning device that acquires a set of pairs of input sentences and output sentences as learning data and performs machine learning of a translation model.

The first input unit 11 acquires a sentence belonging to the first domain from the outside and passes it to the first encoder unit 12 and the shared encoder unit 31 . The first input unit 11 acquires a sentence pair of learning data during the learning process, and passes a sentence in the original language out of the sentence pair to the first encoder unit 12 and the shared encoder unit 31 . The first input unit 11 acquires a sentence in the original language to be translated during translation processing, and passes the input sentence to the first encoder unit 12 and the shared encoder unit 31 . Note that the first input unit 11 may perform morphological analysis processing, etc. of the input sentence for processing by a subsequent encoder. The first input unit 11 may acquire data of an input sentence expressed as word string data already divided into words.

The first encoder unit 12 encodes an input sentence belonging to the first domain and written in the source language, based on parameters for the encoding process in the first domain. The parameters of the encoding process in the first domain can be referenced in the translation process, which will be described later, and updated by a method such as error back propagation in the learning process, which will be described later. Since the first encoder unit 12 encodes an input sentence belonging to the first domain, it has a function of accumulating knowledge unique to the first domain in the learning process described later. That is, the first encoder unit 12 extracts the features of the first domain as knowledge.

The first dimensionality reduction unit 13 combines the information of the semantic vector output from the first encoder unit 12 and the information of the semantic vector output from the shared encoder unit 31, and then reduces the dimensionality of the information. do. Specifically, the first dimensionality reduction unit 13 concatenates the semantic vector output from the first encoder unit 12 and the semantic vector output from the shared encoder unit 31, and linearly transforms the concatenated vector to Reduce dimensionality. As a result, the first dimensionality reduction unit 13 extracts only the useful part for translation from the information obtained from the first encoder unit 12 and the information obtained from the shared encoder unit 31, and extracts the other parts has the effect of eliminating The first dimensionality reduction unit 13 reduces the dimensionality of the vector obtained from the previous stage to a degree sufficient for a normal machine translation system to operate. For example, if the vector output from the first encoder unit 12 and the vector output from the shared encoder unit 31 have the same number of dimensions, simply concatenating them doubles the number of dimensions. The one-dimensional reduction unit 13 reduces the number of dimensions by half. That is, the first dimensionality reduction unit 13 performs dimensionality reduction to approximately the number of dimensions of the semantic vectors output by the first encoder unit 12 and the shared encoder unit 31 . By deleting a part of the information output from the two encoders by the first dimensionality reduction unit 13, it is possible to increase the speed and efficiency of the calculation processing in the latter stage. That is, the first dimensionality reduction unit 13 creates data to be input to the first decoder unit 14 based on the output from the first encoder unit 12 and the output from the shared encoder unit 31 .

The first decoder unit 14 outputs a first semantic vector output as a result of encoding processing in the first encoder unit, a common semantic vector output as a result of encoding processing in the shared encoder unit, and a decoding process in the first domain. Generates an output sentence corresponding to the input sentence based on the parameters. The output sentence generated by the first decoder unit 14 is a translated sentence (sentence in the target language) of the input sentence (sentence in the source language). That is, the first decoder unit 14 receives the output from the first dimensionality reduction unit 13 as input, and outputs a sentence in the target language to be translated.

The first output unit 15 outputs the sentence output by the first decoder unit 14 to the outside.

The second input unit 21 performs the same processing as the first input unit 11 described above with respect to the second domain. Since the details and actions of the processing have already been described in the description of the first input unit 11, description thereof will be omitted here.

The second encoder section 22 performs the same processing as the above-described first encoder section 12 with respect to the second domain. Since the details and effects of the processing have already been described in the description of the first encoder unit 12, description thereof will be omitted here.

The second dimension reduction unit 23 performs the same processing as the first dimension reduction unit 13 described above with respect to the second domain. The details and actions of the process have already been described in the description of the first dimension reduction unit 13, and therefore description thereof is omitted here. That is, the second dimension reduction unit 23 outputs the second dimension reduction vector that is the result of the dimension reduction.

The second decoder section 24 performs the same processing as the first decoder section 14 described above with respect to the second domain. Since the details and actions of the processing have already been described in the description of the first decoder unit 14, description thereof will be omitted here.

The second output unit 25 performs the same processing as the first output unit 15 described above with respect to the second domain. Since the details and actions of the processing have already been described in the description of the first output unit 15, description thereof will be omitted here.

The shared encoder unit 31 encodes an input sentence belonging to either the first domain or the second domain based on encoding parameters shared between the first domain and the second domain. The parameters of the encoding process in the shared encoder unit 31 can be referred to in the translation process, which will be described later, and can be updated in the learning process, which will be described later, by a method such as error backpropagation. Since the shared encoder unit 31 encodes input sentences belonging to either the first domain or the second domain (input sentences of both domains), in the learning process described later, knowledge common to both domains is accumulated. have the effect of That is, the shared encoder unit 31 extracts features common to domains as knowledge.

That is, in the configuration of the translation device 1, when looking at the translation model of one domain (first domain or second domain), the encoder portion is divided into two. Encoders dedicated to that domain and encoders shared with other domains. Therefore, the first encoder unit 12, the first order reduction unit 13, and the first decoder unit 14 are learned only in the learning process using the learning data of the first domain. Also, the second encoder unit 22, the second order reduction unit 23, and the second decoder unit 24 are learned only in the learning process using the learning data of the second domain. On the other hand, the shared encoder unit 31 is trained using learning data of either the first domain or the second domain.

The encoder and decoder models in this embodiment are based on the structure of a recurrent neural network (RNN). Alternatively, the decoder model may be based on an LSTM (Long Short-Term Memory) type neural network structure, which is a type of RNN. The recurrent neural network itself is based on existing technology. A recurrent neural network is one type of general neural network. A feature of recurrent neural networks is that they can process time-series data. This time-series data may be a fixed-length series or a variable-length series. For example, the recursive neural network can process sentences using each element of the time-series data as a word or the like. For example, the following documents describe recurrent neural networks.

References: Recurrent Neural Networks: Introduction to RNN, @kiminaka, updated on February 12, 2017, URL: https://qiita.com/kiminaka/items/87afd4a433dc655d8cfd
Reference: Natural Language Processing Programming Study Group 8 Recurrent Neural Net, Graham Neubig, Nara Institute of Science and Technology, URL: http://www.phontron.com/slides/nlp-programming-ja-08-rnn.pdf

Next, the outline of the processing in this embodiment will be described with reference to FIGS. 2, 3, and 4. FIG.

FIG. 2 is a schematic diagram showing a model of encoding processing in each of the first encoder section 12, the second encoder section 22, and the shared encoder section 31 of this embodiment. As described above, each of the first encoder section 12, the second encoder section 22, and the shared encoder section 31 is implemented using, for example, a recursive neural network.

In the figure, h ₁ , h ₂ , . . . , h _M are time-series data corresponding to the input sentence. _Each of h ₁ , h ₂ , . In the illustrated example, an input sentence of "I/ha/Kyoto/ni/go/." (forward slashes denote word breaks) is input to the encoder. As in this example, punctuation marks and the like are treated as one word. W ₁ and W ₂ are parameters in the recurrent neural network, respectively. That is, _W1 is a matrix for transforming vector h _i into vector e _i . W ₂ is a matrix for transforming vector e _i into vector e _i+1 (1≦i≦M−1). The matrix _W2 is also used when converting the vector _eM into the content vector _c1 . The value of each element of W ₁ and W ₂ can be stored, for example, in updatable memory or the like and updated by machine learning processing.

The encoder sequentially processes the time-series input and finally outputs the content vector _c1 corresponding to the input sentence. That is, the encoder generates vector e ₁ based on the initial input h ₁ and parameter W ₁ . The encoder then generates _vector e2 based on the next input _h2 and parameter _W1 and the above vector _e1 and parameter _W2 . Similarly thereafter, the encoder generates vector e i+1 based on input h _i+1 and parameter W ₁ and already generated vector e _i and parameter _{W 2 (} where 1≦i≦(M−1)). . The encoder then outputs a content vector _c1 generated based on the generated vector e _i+1 and the parameter _W2 .

The content vector c ₁ contains information on the time-series data h ₁ , h ₂ , . . . , h _M corresponding to the input sentence. The content vector _c1 is, for example, a vector of about 250 dimensions. However, the number of dimensions of the content vector _c1 may be determined as appropriate, such as 250, 500, 1000, 2000, for example.

FIG. 3 is a schematic diagram showing a model of decoding processing in each of the first decoder section 14 and the second decoder section 24 of this embodiment. As described above, each of the first decoder section 14 and the second decoder section 24 is implemented using, for example, a recursive neural network.

In the figure, _c2 is the content vector input to the decoder. y ₁ , y ₂ , . . . , y _L1 are time-series data output from the decoder. y ₁ , y _{2 ,} . W ₃ and W ₄ are parameters in the recurrent neural network, respectively. That is, W ₃ is a matrix for transforming vector d _i into vector d _i+1 (1≦i≦L1−1). _W3 is also used when converting vector _c2 input to the decoder into vector _d1 . _W4 is a matrix for transforming vector d _i into vector y _i (1≦i≦L1). The values of the elements of W ₃ and W ₄ may also be stored in memory or the like and updated by machine learning processing.

The decoder generates and outputs series data Y ₁ , Y ₂ , . . . , _YL1 based on the input content vector _c2 . That is, the decoder first generates vector _d1 based on the input content vector _c2 and parameter _W3 . Then the decoder generates the first output data _y1 based on vector _d1 and parameter _W4 . The decoder then generates vector d2 based on vector d1 _, parameter _W3 , and _{the generated output data y1 .} Similarly, the decoder generates vector d i based on vector d _i−1 , parameter W ₃ and output data y _i−1 , and generates output _{data y i based} on vector d _i and parameter W ₄ . (where 2≤i≤L1).

The time-series data y ₁ , y ₂ , . . . , y _L1 output by the decoder are data obtained by decoding the input content vector c ₂ . As an example _, the time-series data y ₁ , y ₂ , . Here, a period indicating the end of a sentence can also be treated as one word.

FIG. 4 is a schematic diagram showing the order reduction processing in each of the first order reduction unit 13 and the second order reduction unit 23 of the present embodiment. As described below, the first order reduction unit 13 and the second order reduction unit 23 are implemented using neural networks.

In the figure, vector h _enc is the content vector output from the first encoder section 12 or the second encoder section 22 . A vector h _senc is a content vector output from the shared encoder unit 31 . First, the first order reduction unit 13 or the second order reduction unit 23 simply concatenates the vector h _enc and the vector h _senc , and generates the concatenated vector h _conc . Next, the first order reduction unit 13 or the second order reduction unit 23 generates a reduced order vector h _lowd based on the concatenated vector h _conc and the parameter W _lowd . This dimensionality reduction process is performed using, for example, linear transformation. This parameter W _lowd is a matrix, and the elements of the matrix are values that can be updated by the learning process. That is, the vector h _lowd is obtained by multiplying the vector h _conc by the matrix W _lowd and transposing the vector accordingly.

Specifically, the first dimension reduction unit 13 concatenates the vector h _enc output from the first encoder unit 12 and the vector h _senc output from the shared encoder unit 31, and further reduces the dimension by , produces the reduced vector _{h_lowd} . The first dimension reduction unit 13 passes the generated vector h _low to the first decoder unit 14 . Further, the second dimension reduction unit 23 concatenates the vector _{h_enc} output from the second encoder unit 22 and the vector _{h_senc} output from the shared encoder unit 31, and further reduces the dimension, thereby reducing the dimension generates a reduced vector _{h_lowd} . The second dimension reduction unit 23 passes the generated vector h _low to the second decoder unit 24 .

When the number of dimensions of vector h _enc and vector h _senc is H, the number of dimensions of vector h _conc , which is the result of concatenation, is 2H. Also, the number of dimensions of the vector h _lowd , which is the result of the dimension reduction process, is H, for example. In this way, each of the first order reduction unit 13 and the second order reduction unit 23 generates a domain-specific encoding process result (vector h _enc ) and a common encoding process result (vector h _senc ) between domains. output a low-dimensional vector h _lowd that contains the information of and has its redundancy reduced. By having the first order reduction unit 13 and the second order reduction unit 23, the translation device 1 realizes a translation model using reasonable computational resources.

Next, the processing procedures of the translation apparatus 1 during learning processing and translation processing will be described with reference to flowcharts.

FIG. 5 is a flow chart showing the procedure of learning processing of the translation device according to the present embodiment. As a prerequisite for this learning process, a large amount of learning data is given from the outside. The learning data is a set of pairs of input sentences in the source language and output sentences (correct sentences) in the target language. Each sentence pair included in the learning data belongs to either the first domain or the second domain, and it is known to which domain it belongs. The procedure of the learning process will be described below along this flow chart.

In step S1, the translation apparatus 1 selects one unprocessed sentence pair from the externally supplied learning data.

At step S2, the translation apparatus 1 determines whether or not the sentence pair selected at step S1 belongs to the first domain, and branches the processing according to the determination result. Information indicating whether a certain sentence pair belongs to the first domain or the second domain is given as part of the learning data. If the sentence pair belongs to the first domain (step S2: YES), the first input unit 11 processes the sentence pair, and then proceeds to step S3. If the sentence pair does not belong to the first domain, that is, if the sentence pair belongs to the second domain (step S2: NO), the second input unit 21 processes the sentence pair, and then step Proceed to S7. When proceeding to step S3, the process proceeds to step S11 after sequentially performing the processes from steps S3 to S6. When proceeding to step S7, the process proceeds to step S11 after sequentially performing the processes from steps S7 to S10.

In step S<b>3 , the first input unit 11 divides the input sentence included in the sentence pair into individual words, and converts it into time-series vector data as appropriate. The first input unit 11 passes the time-series vector data to the first encoder unit 12 and the shared encoder unit 31 . Each of the first encoder unit 12 and the shared encoder unit 31 processes the time-series data passed from the first input unit 11 and outputs a content vector (vector c ₁ in FIG. 2). These content vectors correspond to vector h _enc and vector h _senc shown in FIG. 4, respectively.

In step S4, the first order reduction unit 13 acquires the vector h _enc and the vector h _senc from the first encoder unit 12 and the shared encoder unit 31, respectively. As shown in FIG. 4, the first dimension reduction unit 13 connects the vector h _enc and the vector h _senc to generate the vector h _conc . Then, the first dimension reduction unit 13 reduces the dimension of the vector h _conc based on the parameter W _lowd and outputs the vector h _lowd .

At step S<b>5 , the translation apparatus 1 inputs the vector h _{low output} from the first dimension reduction unit 13 to the first decoder unit 14 . The first decoder unit 14 performs processing for decoding the vector h _low . The first decoder unit 14 outputs time-series data as a result of the decoding process. The output time-series data may be converted into a string of words as needed.

In step S6, the translation device 1 calculates the error between the data output by the first decoder section 14 in step S5 and the correct data included in the original input sentence pair. Further, based on the calculated error, the translation device 1 uses back propagation (error backpropagation method) to share the Adjust the parameters in the encoder section 31 . That is, the translation device 1 updates the values of these parameters by back propagation. Specifically, the parameters to be updated are W ₁ and W ₂ in the first encoder unit 12 (see FIG. 2), W _lowd in the first order reduction unit 13 (see FIG. 4), and W ₃ and W ₄ (see FIG. 3), and W ₁ and W ₂ in the shared encoder section 31 (see FIG. 2). That is, the machine learning process updates the parameters in the first translation model unit 17 .

In steps S7 to S10, the second input unit 21, second encoder unit 22, second order reduction unit 23, second decoder unit 24, and shared encoder unit 31 of the translation device 1 perform A process similar to the process described above is performed. However, while the processing from steps S3 to S6 is for the first domain, the processing from steps S7 to S10 is for the second domain. As a result of this series of processing, the parameters in the second translation model unit 27 are updated by machine learning processing based on the learning data belonging to the second domain. Specifically, the parameters to be updated are W ₁ and W ₂ in the second encoder unit 22 (see FIG. 2), W _lowd in the second order reduction unit 23 (see FIG. 4), and W ₃ and W ₄ (see FIG. 3), and W ₁ and W ₂ in the shared encoder section 31 (see FIG. 2).

After completing either step S6 or step S10, the process proceeds to step S11.

In step S11, it is determined whether or not the processing of all the learning data has been completed. If all learning data processing is completed (step S11: YES), the entire processing of this flowchart is completed. If all the processing of the learning data has not been completed, that is, if one or more sentence pairs remain unprocessed learning data (step S11: NO), step S1 is performed to process the next sentence pair. back to

As described above, during the learning process, the translation device 1 receives a large amount of learning data (for example, a set of pairs of Japanese sentences and English sentences obtained by translating the Japanese sentences into English). are used to modify the parameters in the encoder, decoder and reducer.

As an example, the sentence pair of the training data is Japanese "I go to Kyoto." and English "I go to Kyoto." (output) is "I went to Tokyo." (output sentence), the processing is as follows. The correct sentence, the output sentence, and the difference are as follows. First, "went" is output in the output sentence corresponding to "go" in the correct sentence. Second, "Tokyo" is output in the output sentence corresponding to "Kyoto" in the correct sentence. Based on these differences, the translation device 1 calculates the value of the loss function, for example by cross-cross entropy. That is, the translation device 1 calculates the error. Then, translation apparatus 1 learns parameter values so as to reduce the error.

Here, as an example, a sentence pair included in the learning data is a Japanese sentence (input side) and an English sentence (output side), but the languages of the input side and the output side may be reversed. Also, the learning data may include sentences in languages other than Japanese and English.

As described above, in the learning process, the translation device 1 converts the first encoder unit 12, the first dimensionality reduction unit 13 into , update the parameters in the first decoder unit 14 . Further, the translation apparatus 1 uses the sentence features of the second domain based on the learning data belonging to the first domain to determine the parameters in the second encoder section 22, the second dimension reduction section 23, and the second decoder section 24. Update. Also, the translation device 1 updates the parameters of the shared encoder unit 31 using the features of both the sentences in the first domain and the sentences in the second domain. In other words, the value of each parameter comes to represent the feature of the first domain or the second domain through the learning process.

That is, the learning process is as follows. In the learning process, when a sentence pair, which is a pair of sentences in the source language and the target language belonging to the first domain, is input as learning data, the first encoder unit 12 and the shared encoder unit 31 are processed based on the input sentence in the source language. and the first decoder unit 14 to generate an output sentence, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, the encoding processing parameters in the first encoder unit 12 and the shared encoder unit 31 and the decoding parameters in the first decoder unit 14 are updated. In this case, the learning process further updates the parameters of the first order reduction unit 13 . In the learning process, when a sentence pair, which is a pair of sentences in the source language and the target language belonging to the second domain, is input as learning data, the second encoder unit 22 and the shared encoder are processed based on the input sentence in the source language. An output sentence is generated by the processing of the unit 31 and the second decoder unit 24, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, the parameters of the encoding process in the second encoder unit 22 and the shared The parameters for the encoding process in the encoder section 31 and the parameters for the decoding process in the second decoder section 24 are updated. In this case, the learning process further updates the parameters of the second order reduction unit 23 .

FIG. 6 is a flow chart showing the procedure of translation processing of the translation device according to the present embodiment. As a premise of this translation processing, the model in the translation apparatus 1 has already been learned. Also, the translation apparatus 1 is provided with a sentence to be translated in the original language from the outside. A sentence to be translated belongs to either the first domain or the second domain, and which domain it belongs to is known. The procedure of translation processing will be described below along this flow chart.

At step S21, the translation device 1 acquires an input sentence given from the outside. This input sentence is a sentence to be translated written in a source language (for example, Japanese).

At step S22, the translation apparatus 1 determines whether or not the input sentence obtained at step S21 belongs to the first domain, and branches the processing according to the determination result. Information indicating whether the input sentence belongs to the first domain or the second domain is provided together with the input sentence. If the input sentence belongs to the first domain (step S22: YES), the first input unit 11 processes the input sentence, and then proceeds to step S23. If the input sentence does not belong to the first domain, that is, if the input sentence belongs to the second domain (step S22: NO), the second input unit 21 processes the input sentence, and then step Proceed to S27. When proceeding to step S23, the translation apparatus 1 sequentially performs the processing from steps S23 to S26. When proceeding to step S7, the translation apparatus 1 sequentially performs the processes from steps S27 to S30.

In step S<b>23 , the first input unit 11 divides the input sentence into words and converts them into time-series vector data as appropriate. The first input unit 11 passes the time-series vector data to the first encoder unit 12 and the shared encoder unit 31 . Each of the first encoder unit 12 and the shared encoder unit 31 processes the time-series data passed from the first input unit 11 and outputs a content vector (vector c ₁ in FIG. 2). These content vectors correspond to vector h _enc and vector h _senc shown in FIG. 4, respectively.

In step S24, the first order reduction unit 13 acquires the vector h _enc and the vector h _senc from the first encoder unit 12 and the shared encoder unit 31, respectively. As shown in FIG. 4, the first dimension reduction unit 13 connects the vector h _enc and the vector h _senc to generate the vector h _conc . Then, the first dimension reduction unit 13 reduces the dimension of the vector h _conc based on the parameter W _lowd and outputs the vector h _lowd .

In step S<b>25 , the translation device 1 inputs the vector h _low output from the first dimension reduction unit 13 to the first decoder unit 14 . The first decoder unit 14 performs processing for decoding the vector h _low . The first decoder unit 14 outputs time-series data as a result of the decoding process. The output time-series data is converted into a string of words.

In step S26, the first output unit 15 creates an output sentence based on the string of words output by the first decoder unit 14 in step S25. The first output unit 15 outputs this output sentence as a translation result.

In steps S27 to S30, the second input unit 21, the second encoder unit 22, the second order reduction unit 23, the second decoder unit 24, the second output unit 25, and the shared encoder unit 31 of the translation apparatus 1 , the same processing as that described in steps S23 to S26 is performed. However, while the processing from steps S23 to S26 is for the first domain, the processing from steps S27 to S30 is for the second domain. As a result of this series of processing, the second output unit 25 outputs the output sentence as a translation result.

When the process of either step S26 or step S30 ends, translation device 1 ends the process of the entire flowchart.

As an example, when the input sentence is "He went to Kyoto." Translation apparatus 1 having a second domain translation model obtains content vectors h _enc and h _senc in second encoder section 22 and shared encoder section 31, respectively. The second dimension reduction unit 23 performs dimension reduction processing based on these content vectors h _enc and h _senc , and acquires the vector h _lowd . The second decoder section 24 outputs the output word string "He/went/to/Kyoto/." based on the vector h _lowd . Based on the output of the second decoder unit 24, the second output unit 25 outputs the output sentence "He went to Kyoto."

As described above, in this embodiment, the translation device 1 has a plurality (two) of translation models corresponding to each domain. In each translation model, the translation apparatus 1 also includes a portion unique to each domain (first encoder section 12, first order reduction section 13, first decoder section 14, second encoder section 22, second order reduction part 23, second decoder part 24) and a part shared by multiple domains (shared encoder part 31). When the translation apparatus 1 performs learning processing using learning data belonging to a certain domain, the translation apparatus 1 learns the portion unique to the domain and the above-mentioned shared portion, and updates the parameters of the model. With such a configuration, domain-specific knowledge is accumulated in the domain-specific portion described above. In addition, common knowledge across domains is accumulated in the shared portion. In other words, a translation model of a certain domain can learn not only a model of a unique part by learning data belonging to the domain, but also a model of a common part by learning data belonging to another domain. In other words, learning data of other domains can also be used to train the model of the domain. That is, the amount of learning processing can be increased with respect to the amount of learning data to be prepared. That is, learning data can be made more efficient. Preparing learning data is often a costly task, but in this embodiment, the cost required for learning (the cost of preparing learning data) can be reduced.

As an example, if the first domain is news announcement text and the second domain is television program closed caption text, the present embodiment contributes the following. It is expensive to prepare the news announcement text for the first domain. As for the subtitle texts of the television programs in the second domain, a large amount of existing resources can be used as learning data at low cost. The learning data of the first domain contributes to learning expressions of sentences including expressions peculiar to news. The learning data of the second domain is not the news manuscript sentences (spoken language) to be announced, but for learning to enrich sentence expressions (vocabulary, etc.) in various fields such as politics, economics, sports, entertainment, etc. To contribute. The shared encoder unit 31 of the present embodiment accumulates knowledge obtained not only from the learning data of the first domain but also from the learning data of the second domain. The first decoder unit 14 outputs an output sentence that is a result of reflecting the knowledge dedicated to the first domain and the knowledge common to the domains accumulated in the model of the shared encoder unit 31 .

The first dimension reduction unit 13 performs processing to reduce the number of dimensions of the vector by linear transformation. The first dimensionality reduction unit 13 reduces the dimensionality of the input vector to a sufficient number of dimensions for the first decoder unit 14 to perform an appropriate output. In other words, redundant information included in the outputs of the first encoder unit 12 and the shared encoder unit 31 can be deleted by the processing of the first dimensionality reduction unit 13 . This enables faster and more efficient calculation.

Although the translation model for the first domain has been mainly described above, the first domain and the second domain have a symmetrical relationship in the translation apparatus 1, and the above description can also be applied to the translation model for the second domain. be.

[Second embodiment]
Next, a second embodiment of the invention will be described. In addition, description may be abbreviate|omitted below about the matter already demonstrated in the previous embodiment. Here, the description will focus on matters specific to this embodiment.

FIG. 7 is a block diagram showing a schematic functional configuration of the translation device (learning device) according to this embodiment. As illustrated, the translation device 2 includes a first input section 11, a first encoder section 12, a first decoder section 14, a first output section 15, a second input section 21, and a second encoder section 22. , a second decoder unit 24 , a second output unit 25 , and a shared encoder unit 31 . In this embodiment, the first encoder section 12, the first decoder section 14, and the shared encoder section 31 may be collectively referred to as the first translation model section 217. Also, the second encoder section 22 , the second decoder section 24 and the shared encoder section 31 may be collectively called a second translation model section 227 . The first translation model section 217 and the second translation model section 227 function as translation models for domains different from each other.

That is, unlike the translation device 1 of the first embodiment, the translation device 2 of this embodiment does not have the first order reduction unit and the second order reduction unit. In other words, the translation device 2 connects the semantic vector output from the first encoder unit 12 and the semantic vector output from the shared encoder unit 31, but does not reduce the dimensionality of the concatenated vectors. A vector obtained by concatenating the semantic vector output from the second encoder unit 22 and the semantic vector output from the shared encoder unit 31 is input to the first decoder unit 14 as it is without being reduced in dimension. used as Likewise, the semantic vector output from the second encoder unit 22 and the semantic vector output from the shared encoder unit 31 are not reduced in dimension after being concatenated, and are sent to the second decoder unit 24 as they are. Used as input.

According to the above configuration of the present embodiment, it is possible to perform translation model learning processing or perform machine translation processing using a trained translation model without using dimensionality reduction processing. .

[Third embodiment]
Next, a third embodiment of the invention will be described. In addition, description may be abbreviate|omitted below about the matter already demonstrated by the previous embodiment. Here, the description will focus on matters specific to this embodiment.

FIG. 8 is a block diagram showing a schematic functional configuration of a translation device (learning device) according to this embodiment. As illustrated, the translation device 3 performs translation processing and learning processing for each of n (n≧3) domains. Specifically, the translation device 3 includes translation models from the first domain to the n-th domain. The translation model for the i-th domain (1≤i≤n) in the translation device 3 includes the i-th input section i-1, the i-th encoder section i-2, the i-th order reduction section i-3, It includes an i-th decoder unit i-4, an i-th output unit i-5, and a shared encoder unit 331. FIG. The shared encoder section 331 is shared by multiple translation models from the first domain to the n-th domain.

In this embodiment, the translation device 3 acquires sentence pairs (sentences written in the source language and the target language) belonging to any one of the first to n-th domains during the learning process. The translation device 3 learns the parameters (including the parameters of the shared encoder section 331) included in the translation model of the i-th domain (1≦in) using sentence pairs belonging to the i-th domain. The translation device 3 acquires an input sentence (original language sentence) belonging to any one of the first domain to the n-th domain during translation processing. The translation device 3 uses the i-th domain translation model to output a translation sentence (target language sentence) corresponding to the input sentence.

According to this embodiment, knowledge for translation can be shared not only in two domains but also in three or more domains. In other words, since the shared encoder unit 331 of the translation device 3 is trained using sentence pairs belonging to three or more domains, it has parameters based on sentence pairs of all these domains.

In the configuration shown in FIG. 8, the translation model of the i-th domain has the i-th reduction part i-3. This i-th order reduction unit i-3 may be omitted. The details of the configuration omitting the function of reducing the dimensionality of the content vectors are already described in the second embodiment. That is, the third embodiment and the second embodiment may be combined for implementation.

[Fourth embodiment]
Next, a fourth embodiment of the invention will be described. In addition, description may be abbreviate|omitted below about the matter already demonstrated by the previous embodiment. Here, the description will focus on matters specific to this embodiment.

FIG. 9 is a block diagram showing a schematic functional configuration of a translation device (learning device) according to this embodiment. As illustrated, the translation device 4 includes a first input unit 11, a first encoder unit 12, a first dimension reduction unit 13, a first decoder unit 14, a first output unit 15, a second input section 21, second encoder section 22, second order reduction section 23, second decoder section 24, second output section 25, shared encoder section 31, first orthogonal error calculation section 419, and 2 orthogonal error calculator 429 . a first input unit 11, a first encoder unit 12, a first dimension reduction unit 13, a first decoder unit 14, a first output unit 15, a second input unit 21, and a second encoder unit 22; , the second order reduction unit 23, the second decoder unit 24, the second output unit 25, and the shared encoder unit 31 have already been described in the previous embodiments. A feature of this embodiment is that the translation apparatus 4 has a first orthogonal error calculator 419 and a second orthogonal error calculator 429 .

The first orthogonal error calculation unit 419 calculates the orthogonal error between the semantic vectors respectively output when the first encoder unit 12 and the shared encoder unit 31 encode the same input sentence. Let this orthogonal error be L _diff . The orthogonal error L _diff is calculated as a positive value. When the semantic vector output from the first encoder unit 12 and the semantic vector output from the shared encoder unit 31 are completely orthogonal, the value of the orthogonal error L _diff is zero. The lower the degree of orthogonality between both semantic vectors, the larger the value of the orthogonality error L _diff .

The second orthogonal error calculator 429 calculates the orthogonal error L _diff between the semantic vectors respectively output when the second encoder 22 and the shared encoder 31 encode the same input sentence. The value of the orthogonal error calculated by the second orthogonal error calculator 429 is the same as the value of the orthogonal error calculated by the first orthogonal error calculator 419, and has the same meaning.

During the learning process, the translation device 4 adjusts the parameter values based also on the orthogonal error L _diff calculated by the first orthogonal error calculator 419 or the second orthogonal error calculator 429 .

Regarding the translation model of the first domain, using a sentence pair belonging to the first domain, an error based on the difference between the output sentence output by the first decoder unit 14 and the correct data (object sentence) included in the sentence pair The use of the backpropagation method has already been explained in the first embodiment and the like. In this embodiment, the translation apparatus 4 not only calculates the difference between the output sentence output by the first decoder unit 14 and the correct data (object sentence) included in the sentence pair, but also the difference calculated by the first orthogonal error calculation unit 419. The orthogonal error L _diff is also used to update the parameters by error backpropagation. Assuming that the error between the output sentence and the correct answer data is L _output , the total error L used by the translation apparatus 4 for updating parameters is expressed by the following equation. Here, α and β are weight values that are appropriately set.

L=α·L _output +β·L _diff

Note that parameter update of the translation model for the second domain based on the orthogonal error L _diff calculated by the second orthogonal error calculator 429 is the same as that for the translation model for the first domain described above.

According to the present embodiment, the first quadrature error calculator 419 or the second quadrature error calculator 429, respectively, the meaning output from the first encoder unit 12 or the second encoder unit 22 (dedicated encoder specific to each domain) An orthogonal error L _diff between the vector and the semantic vector output from the shared encoder unit 31 is calculated. Moreover, the translation device 4 adjusts the parameters by the error propagation method based also on this orthogonal error L _diff during the learning process. This learning process works to increase the orthogonality between the vector output from the first encoder section 12 or the second encoder section 22 and the vector output from the shared encoder section 31 . In other words, by performing learning using a sufficient amount of learning data, the orthogonality between the vector output from the first encoder unit 12 or the second encoder unit 22 and the vector output from the shared encoder unit 31 increases. This ensures that the models of dedicated encoders specific to each domain do not overlap with those of shared encoders. That is, the redundancy between the model of the dedicated encoder and the model of the shared encoder is reduced, and the translation model can be learned efficiently.

In the configuration shown in FIG. 9, the translation models of the first domain and the second domain are provided with the first order reduction section 13 and the second order reduction section 23, respectively. The first dimension reduction unit 13 and the second dimension reduction unit 23 may be omitted. The details of the configuration omitting the function of reducing the dimensionality of the content vectors are already described in the second embodiment. That is, the fourth embodiment and the second embodiment may be combined for implementation.

Also, in the configuration shown in FIG. 9, the number of domains was two. The fourth embodiment may be implemented with three or more domains. That is, the fourth embodiment and the second embodiment may be combined for implementation. Furthermore, the fourth embodiment, the third embodiment, and the second embodiment may be combined for implementation.

A plurality of embodiments have been described above, but the following modifications may be implemented. Moreover, when combinations are possible, a plurality of modified examples may be combined and implemented.

[First Modification] In the above-described embodiment, when the translation device (learning device) has a dimensionality reduction unit, all domains have the dimensionality reduction unit. As a modification, only some of the domain translation models among all the domain translation models may have the dimensionality reduction portion. For example, only one of the two domains may have a dimension reduction part. Also, for example, only one or more arbitrary domains among three or more domains may have a dimensionality reduction unit.

[Second Modification] Although domains such as news announcement sentences and broadcast subtitle sentences have been described as examples of domains, the above embodiments may be applied to other domains.

[Third Modification] In each of the above-described embodiments, a single device serves as both a learning device and a translation device. As a modification, the learning device and the translation device may be separate devices. In this case, by copying the knowledge (numerical values of parameters, etc.) obtained as a result of the learning process in the learning device to the translation device side as, for example, a data file, the translation device performs translation processing using the learning result. can be done.

At least part of the functions of the translation device (learning device) in the above-described embodiments (including modifications) can be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. In addition, “computer-readable recording media” refers to portable media such as flexible discs, magneto-optical discs, ROMs, CD-ROMs, DVD-ROMs, USB memories, and storage devices such as hard disks built into computer systems. Say things. In addition, "computer-readable recording medium" means a medium that temporarily and dynamically retains a program, such as a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line. , it may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing part of the functions described above, or may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system.

The features of the embodiment and modifications described above are summarized as follows.

In the case of having at least one of the first order reduction section and the second order reduction section, it is as follows. The first dimensionality reduction unit 13 converts the elements of the first semantic vector output as a result of the encoding process by the first encoder unit 12 and the elements of the common semantic vector output as a result of the encoding process by the shared encoder unit 31. is reduced in dimension based on the parameters of the dimension reduction processing in the first domain, and a first reduced-order vector that is the result of the reduction in dimension is output. At this time, the first decoder unit 14 generates an output sentence corresponding to the input sentence based on the first reduced dimension vector output by the first reduced dimension unit 13 and the parameters of the decoding process in the first domain. do. As the learning process in this case, the dimensionality reduction processing in the first dimensionality reduction unit 13 is based on the difference between the sentence in the target language of the sentence pair included in the learning data and the output sentence from the first decoder unit 14. also update the parameters of The second dimension reduction unit 23 is similar to the first dimension reduction unit 13 described here.

When performing learning based on the quadrature error between encoder outputs, it is as follows. That is, the learning device includes a first orthogonal error calculator 419 and a second orthogonal error calculator 429 . The first orthogonal error calculator 419 calculates the orthogonal error between the first semantic vector output as a result of encoding processing by the first encoder unit 12 and the common semantic vector output as a result of encoding processing by the shared encoder unit 31. A second orthogonal error calculation unit 429 that calculates a certain first orthogonal error outputs a second semantic vector output as a result of encoding processing by the second encoder unit 22 and a result of encoding processing by the shared encoder unit 31. A second orthogonal error, which is an orthogonal error with the common semantic vector, is calculated. In the adjustment of the parameters by the back propagation method or the like, the difference between the sentence in the target language of the sentence pair of the learning data and the output sentence output from the first decoder unit 14 is Also based on the one orthogonal error, the parameters for the encoding process in the first encoder section 12, the parameters for the encoding process in the shared encoder section 31, and the parameters for the decoding process in the first decoder section 14 are updated. Furthermore, if the order reduction unit is included, the parameters for the order reduction process are also updated. Also, based on the second orthogonal error calculated by the second orthogonal error calculating unit 429 as well as the difference between the sentence in the target language of the sentence pair of the learning data and the output sentence output from the second decoder unit 24, the second The parameters for the encoding process in the encoder section 22, the parameters for the encoding process in the shared encoder section 31, and the parameters for the decoding process in the second decoder section 24 are updated. Furthermore, if the order reduction unit is included, the parameters for the order reduction process are also updated. It should be noted that the parameter adjustment based on the above orthogonal error may be performed for only some domains among all domains.

At least the parameters for the encoding process in the first encoder section 12, the parameters for the encoding process in the shared encoder section 31, and the parameters for the decoding process in the first decoder section 14 in the translation device are obtained by the learning process in the above embodiment. It can be considered as The same applies to the parameters for order reduction processing. The same is true for parameters of other domains.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

[Demonstration experiment]
The results of the demonstration experiment are described below. In the demonstration experiment, the original language was English and the target language was Japanese. A patent corpus (NTCIR, NII Testbeds and Community for Information access Research) was used as the language resource of the first domain. Scientific and technical papers (ASPEC) were used as the second domain. A test set of the first domain patent corpus (NTCIR) was used for the evaluation results. The first evaluation target method is based on a translation model that uses a first encoder section (12) that extracts the features of the first domain as knowledge and a shared encoder section (31) that extracts features common to the domains as knowledge. .
In addition to the configuration of the first evaluation method, the second evaluation method adds a constraint on the orthogonality between the outputs from the first encoder unit and the shared encoder unit when learning the parameters of the first encoder unit and the shared encoder unit. It is a thing (4th Embodiment). The first method for comparison is to independently learn the first domain and the second domain. The second method to be compared is one of the latest methods in previous research, which adds a domain tag (eg <NTCIR>) to the beginning of the corpus. A second comparative approach is described in the literature below.

The literature on the second comparative method: Chenhui Chu, Raj Dabre, and Sadao Kurohashi. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of ACL, 2017.

Accuracy was measured using a commonly used evaluation scale BLEU for machine translation. BLEU is judged to be closer to the correct reference translation as its value increases. In the first comparative approach, the BLEU score was 44.2. In the second comparative approach, the BLEU score was 46.0. In the first evaluated approach, the BLEU score was 48.1. In the second evaluated approach, the BLEU score was 49.76. That is, the scores for the first and second evaluated approaches, which are the embodiments described above, are higher than the scores for the first and second comparative approaches. In other words, good evaluations were obtained for the first and second evaluation target methods.

Translation examples in the demonstration experiment are as follows. The sentence in the original language is "A data space is established in the memory." The reference translation (sentence in the target language) for this is "A data space is formed in memory." The translation result by the method for the first comparison is "a data space is provided in this memory". The result of translation by the second method for comparison is "a data space is formed in the memory." The translation result by the first evaluation target method is "a data space is formed in the memory." The translation result by the second evaluation target method is "a data space is formed in the memory". The output of the translation result by the second evaluation target method is the same as the reference translation, and it can be seen that it is the best result.

INDUSTRIAL APPLICABILITY The present invention can be used for machine translation technology. Machine translation processing using the present invention can also be used, for example, in the media industry such as the broadcasting business. However, the scope of application of the present invention is not limited to those exemplified here.

1, 2, 3, 4 translation device (learning device)
11 First input unit 12 First encoder unit 13 First order reduction unit 14 First decoder unit 15 First output unit 17 First translation model unit 21 Second input unit 22 Second encoder unit 23 Second order reduction unit 24 second decoder unit 25 second output unit 27 second translation model unit 31 shared encoder unit 217 first translation model unit 227 second translation model unit 331 shared encoder unit 419 first orthogonal error calculator 429 second orthogonal error calculator

Claims (4)

a first encoder unit that encodes an input sentence belonging to the first domain and in a source language based on parameters for encoding in the first domain;
a second encoder unit that encodes an input sentence belonging to the second domain in the source language based on parameters for encoding in the second domain;
a shared encoder unit that encodes the input sentence belonging to either the first domain or the second domain based on encoding parameters shared between the first domain and the second domain;
Based on a first semantic vector output as a result of encoding processing in the first encoder unit, a common semantic vector output as a result of encoding processing in the shared encoder unit, and parameters of decoding processing in the first domain a first decoder unit for generating an output sentence corresponding to the input sentence;
Based on a second semantic vector output as a result of encoding processing in the second encoder unit, a common semantic vector output as a result of encoding processing in the shared encoder unit, and parameters of decoding processing in the second domain a second decoder unit for generating an output sentence corresponding to the input sentence;
calculating a first orthogonal error that is an orthogonal error between a first semantic vector output as a result of encoding processing by the first encoder unit and a common semantic vector output as a result of encoding processing by the shared encoder unit; 1 orthogonal error calculator;
calculating a second orthogonal error that is an orthogonal error between a second semantic vector output as a result of encoding processing by the second encoder unit and a common semantic vector output as a result of encoding processing by the shared encoder unit; a two-orthogonal error calculator;
and
When a sentence pair, which is a pair of sentences in the source language and the target language belonging to the first domain, is input as learning data, based on the input sentence in the source language, the first encoder unit, the shared encoder unit, and the An output sentence is generated by processing with the first decoder unit, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, parameters for encoding processing in the first encoder unit and the shared encoder unit updating the parameters of the encoding process in and the parameters of the decoding process in the first decoder unit,
When a sentence pair, which is a pair of sentences in the source language and the target language belonging to the second domain, is input as learning data, based on the input sentence in the source language, the second encoder unit, the shared encoder unit, and the An output sentence is generated by processing with the second decoder unit, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, parameters for encoding processing in the second encoder unit and the shared encoder unit updating the parameters of the encoding process in and the parameters of the decoding process in the second decoder unit,
Based on the first orthogonal error calculated by the first orthogonal error calculating unit as well as the difference between the sentence in the target language of the sentence pair and the output sentence output from the first decoder unit, the first encoder updating parameters for encoding processing in the unit, parameters for encoding processing in the shared encoder unit, and parameters for decoding processing in the first decoder unit;
Based on the second orthogonal error calculated by the second orthogonal error calculating unit as well as the difference between the sentence in the target language of the sentence pair and the output sentence output from the second decoder unit, the second encoder updating parameters of encoding processing in the unit, parameters of encoding processing in the shared encoder unit, and parameters of decoding processing in the second decoder unit;
learning device. The concatenated vector obtained by arranging the elements of the first semantic vector output as a result of the encoding process by the first encoder unit and the elements of the common semantic vector output as a result of the encoding process by the shared encoder unit, A first order reduction unit that reduces the order based on the parameters of the order reduction process in the first domain and outputs a first reduced order vector that is the result of the order reduction;
and
The first decoder unit generates an output sentence corresponding to the input sentence based on the first reduced-dimensional vector output by the first reduced-order unit and parameters of decoding processing in the first domain. death,
Also updating parameters for order reduction processing in the first order reduction unit based on the difference between the sentence in the target language of the sentence pair and the output sentence from the first decoder unit;
A learning device according to claim 1. The concatenated vector obtained by arranging the elements of the second semantic vector output as a result of the encoding process by the second encoder unit and the elements of the common semantic vector output as a result of the encoding process by the shared encoder unit, A second order reduction unit that reduces the order based on the parameter of the order reduction process in the second domain and outputs a second order reduction vector that is the result of the order reduction,
and
The second decoder unit generates an output sentence corresponding to the input sentence based on the second reduced-dimensional vector output by the second reduced-order unit and parameters of decoding processing in the second domain. death,
Also updating the parameter of the order reduction processing in the second order reduction unit based on the difference between the sentence in the target language of the sentence pair and the output sentence from the second decoder unit;
3. The learning device according to claim 1 or 2. the computer,
a first encoder unit that encodes an input sentence belonging to the first domain and in a source language based on parameters for encoding in the first domain;
a second encoder unit that encodes an input sentence belonging to the second domain in the source language based on parameters for encoding in the second domain;
a shared encoder unit that encodes the input sentence belonging to either the first domain or the second domain based on encoding parameters shared between the first domain and the second domain;
Based on a first semantic vector output as a result of encoding processing in the first encoder unit, a common semantic vector output as a result of encoding processing in the shared encoder unit, and parameters of decoding processing in the first domain a first decoder unit for generating an output sentence corresponding to the input sentence;
Based on a second semantic vector output as a result of encoding processing in the second encoder unit, a common semantic vector output as a result of encoding processing in the shared encoder unit, and parameters of decoding processing in the second domain a second decoder unit for generating an output sentence corresponding to the input sentence;
calculating a first orthogonal error that is an orthogonal error between a first semantic vector output as a result of encoding processing by the first encoder unit and a common semantic vector output as a result of encoding processing by the shared encoder unit; 1 orthogonal error calculator;
calculating a second orthogonal error that is an orthogonal error between a second semantic vector output as a result of encoding processing by the second encoder unit and a common semantic vector output as a result of encoding processing by the shared encoder unit; a two-orthogonal error calculator;
and
When a sentence pair, which is a pair of sentences in the source language and the target language belonging to the first domain, is input as learning data, based on the input sentence in the source language, the first encoder unit, the shared encoder unit, and the An output sentence is generated by processing with the first decoder unit, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, parameters for encoding processing in the first encoder unit and the shared encoder unit updating the parameters of the encoding process in and the parameters of the decoding process in the first decoder unit,
When a sentence pair, which is a pair of sentences in the source language and the target language belonging to the second domain, is input as learning data, based on the input sentence in the source language, the second encoder unit, the shared encoder unit, and the An output sentence is generated by processing with the second decoder unit, and based on the difference between the sentence in the target language of the sentence pair and the output sentence, parameters for encoding processing in the second encoder unit and the shared encoder unit updating the parameters of the encoding process in and the parameters of the decoding process in the second decoder unit;
Based on the first orthogonal error calculated by the first orthogonal error calculating unit as well as the difference between the sentence in the target language of the sentence pair and the output sentence output from the first decoder unit, the first encoder updating parameters for encoding processing in the unit, parameters for encoding processing in the shared encoder unit, and parameters for decoding processing in the first decoder unit;
Based on the second orthogonal error calculated by the second orthogonal error calculating unit as well as the difference between the sentence in the target language of the sentence pair and the output sentence output from the second decoder unit, the second encoder updating parameters of encoding processing in the unit, parameters of encoding processing in the shared encoder unit, and parameters of decoding processing in the second decoder unit;
A program to function as a learning device.

Images