JP2020129209A - Optimization method, optimization program, inference method, and inference program - Google Patents

Abstract

To provide a neural network capable of corresponding to a change in usage and a request specification, and its optimization method.SOLUTION: An optimization method includes: a step for preparing training data in which an input signal is associated with a correct answer output signal; a step for inputting the input signal to a neural network, calculating an output signal outputted from the deepest layer included in the neural network, and calculating an output signal outputted from each of one or more layers including the deepest layer; a step for respectively calculating errors of respective output signals that are calculated to the correct answer output signal associated with the input signal; and a step for optimizing the parameter of each layer included in the neural network on the basis of the respective calculated errors.SELECTED DRAWING: Figure 3

Description

The present technology relates to an optimization method for deep learning and a method for using an optimized model obtained by the optimization method.

Methods based on deep learning are showing higher performance than other machine learning methods in each field of artificial intelligence including natural language processing.

In deep learning, a neural network (hereinafter, also simply referred to as “model”) that obtains an output signal by performing a plurality of nonlinear transformations on an input signal is assumed. The nonlinear transformation (that is, the coefficient of the linear transformation matrix and the value of the bias term) in the neural network is optimized based on the error between the output signal of the model and the correct output signal given in advance. With such an optimization method, the optimized model can be determined according to the task. An end-to-end optimization method for optimizing a model based on only a set of an input signal and a correct output signal given by a human divides a complicated process like a human being into fine processes. It has been used for many tasks in recent years because it does not need to be implemented.

In general, increasing the number of nonlinear transformations (that is, the number of layers in the neural network) makes it possible to express a more complicated function, and it may be possible to solve a problem with a complicated correspondence between input and output. Get higher It has been reported that the performance can be improved by increasing the number of layers in various tasks.

A problem called gradient vanishing may occur in the optimization method in deep learning. A technique called a skip structure (residual connection) is often used to deal with the problem of disappearing gradients (see Non-Patent Document 1, for example). This method has a constraint that the number of dimensions of the input signal and the number of dimensions of the output signal must be equal, but the neural network can be stably optimized.

As the number of layers increases, the amount of space calculation (that is, the number of parameters) and the amount of time calculation (that is, the number of times the matrix is multiplied, etc.) increase. As a result, in order to execute the process (inference process) of calculating the output signal of the model for the input signal, more memory is required and the processing speed may be reduced. An approach has been proposed in which parameters in the same layer are recursively used in a neural network for the purpose of reducing the required memory (see Non-Patent Document 2).

As another approach for reducing the space calculation amount and the time calculation amount, there is a method called knowledge distillation (see Non-Patent Document 3). The distillation method is to optimize a relatively simple model by referring to the output signal of the optimized model after first optimizing the complicated model. It has been reported that the distillation method has realized, for example, memory saving of a model in neural machine translation (see Non-Patent Document 4).

As yet another approach to reduce the space calculation amount and the time calculation amount, a method of reducing the representation precision of a real number (using 16-bit representation instead of 32-bit representation) (see Non-Patent Document 5), a parameter in an optimized model There is a method of optimizing so that most of the data can be limited to binary values of +1/-1 (see Non-Patent Document 6), a method of binary encoding a part of the target vocabulary (see Non-Patent Document 7), and the like. Proposed.

The optimization of deep learning is performed only based on the information of the deepest layer of a given neural network. Therefore, when the output signal is calculated using only some layers of the optimized model, the performance may be extremely deteriorated. That is, as an approach to reduce the amount of space calculation and the amount of time calculation, it is not possible to use different layers or network structures between the model targeted for optimization and the model used in inference processing.

Therefore, when the usage or required specification of the model changes, it is necessary to re-optimize the new model. In this respect, the same applies to any of the approaches for reducing the amount of space calculation and the amount of time calculation described above.

The present technology aims to provide an optimization method of a neural network capable of coping with changes in applications and required specifications.

According to one aspect of the present technology, an optimization method for optimizing parameters of a neural network having a plurality of identical or different layers is provided. The optimization method includes a step of preparing training data in which an input signal and a correct output signal are associated with each other, and inputting the input signal to a neural network to calculate an output signal output from the deepest layer included in the neural network. And calculating the output signal output from each of the one or more layers including the deepest layer, and calculating the error of each calculated output signal with respect to the correct output signal associated with the input signal. And a step of optimizing the parameters of each layer included in the neural network based on the calculated respective errors.

The optimizing step may include a step of integrating the calculated respective errors.

The step of integrating the errors may include a step of integrating the calculated errors and calculating error information for back propagation from the deepest layer.

The step of calculating the error information may include a step of calculating an average value of the calculated errors as error information for back propagation from the deepest layer.

The step of optimizing includes the parameter of the target layer based on the error information given by backpropagation to the target layer for which the parameter is optimized and the error calculated for the output signal of the target layer. May be optimized.

The neural network includes an encoder that outputs characteristic information included in the input signal and a decoder that receives the input output signal and the characteristic information included in the input signal and determines the output signal. Good.

According to another aspect of the present technology, an optimization program for causing a computer to execute the optimization method described above is provided.

According to still another aspect of the present technology, an inference method using an optimized model including a neural network having a plurality of same or different layers is provided. The inference method includes the steps of inputting an arbitrary input signal into the optimized model, calculating the output signal in order toward the deepest layer of the optimized model, and applying a plurality of identical or identical signals included in the optimized model. Out of the different layers, the output signal of any layer including the deepest layer determined based on the request is output as an inference result. The optimized model is an output signal output from each of one or more layers including the deepest layer, which is calculated when an input signal included in the training data is input to the neural network, and an input signal included in the training data. It is generated by optimizing the parameters based on the respective errors from the correct answer output signal associated with the signal.

The layer from which the output signal is output as the inference result may be determined based on at least one of the inference performance of the output signal and the time required until the output signal is output.

According to still another aspect of the present technology, an inference program for causing a computer to execute the optimization method described above is provided.

According to the present technology, it is possible to provide a method for optimizing a neural network that can respond to changes in applications and required specifications.

It is a schematic diagram for demonstrating general deep learning. FIG. 7 is a schematic diagram for explaining deep learning according to the present embodiment. 7 is a flowchart showing a main part of a processing procedure according to the present embodiment. It is a schematic diagram which shows an example of the Transformer model which implement|achieves a neural machine translation. FIG. 6 is a schematic diagram for explaining an optimization process according to the first embodiment. 6 is a flowchart showing a main processing procedure of optimization processing according to the first embodiment. It is a graph which shows the evaluation result about the English-Japanese translation task using the multilingual parallel translation corpus for speech translation. It is a graph which shows the evaluation result about the English-German translation task using the parallel translation data of the news field. FIG. 9 is a schematic diagram for explaining an optimization process according to the second embodiment. FIG. 9 is a schematic diagram for explaining an optimization process according to the second embodiment. 9 is a flowchart showing a main processing procedure of optimization processing according to the second embodiment. 16 is a graph showing the evaluation result of the English-Japanese translation task according to the second embodiment. 16 is a graph showing the evaluation result of the English-German translation task in the second embodiment. FIG. 14 is a schematic diagram for explaining an optimization process according to the third embodiment. 17 is a flowchart showing a main processing procedure of optimization processing according to the third embodiment. 16 is a graph showing the evaluation result of the English-Japanese translation task in the third embodiment. It is a schematic diagram which shows an example of the hardware constitutions which implement|achieve the optimization process and inference process according to this Embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated.

[A. Related technology]
First, general deep learning will be described.

FIG. 1 is a schematic diagram for explaining general deep learning. With reference to FIG. 1, in deep learning, a neural network 10 that obtains an output signal by performing a plurality of nonlinear transformations on an input signal is assumed. The neural network 10 typically comprises an input layer 2, one or more hidden layers 4, and an output layer 6. Each of the input layer 2, the hidden layer 4, and the output layer 6 includes a vector indicating a state and an activation function. Adjacent layers are coupled via affine transformation or the like.

For example, a model for obtaining the output signal Y through the 6-layered non-linear transformations L ₁ , L ₂ , L ₃ , L ₄ , L ₅ and L ₆ of the input signal X can be expressed as the following equation (1). it can.

Y=L ₆ (L ₅ (L ₄ (L ₃ (L ₂ (L ₁ (X)))))) (1)
Usually, the input signal X is a finite fixed-dimensional real vector v (εR ⁿ ). Each of the non-linear transformations L _i can be represented by a linear transformation matrix and a bias term. The coefficient of the linear transformation matrix and the value of the bias term are collectively called “parameter”. Parameter optimization refers to the process of adjusting each value of the parameter so as to show higher performance for the target task.

In the optimization of the neural network, it is assumed that the most useful information is obtained in the deepest layer. Based on this assumption, the parameters are optimized by repeating the following three processes (steps S1 to S3).

(1) Calculate the output signal Y for the input signal X (step S1)
(2) The error e between the output signal Y and the correct output signal is calculated (step S2).
(3) Parameters are updated in order from the deepest layer to the shallowest layer based on the error e (error backpropagation) (step S3).
In step S2, the error e is calculated by a method according to the problem.

More specifically, in the case of (i) regression problem (when it is desired to obtain a real number vector v(εR ⁿ ) in a certain range as the output signal Y), the output signal Y is normalized using a sigmoid function or the like. , The error e with respect to the real vector v′ (εR ⁿ ) which is the correct output signal is calculated.

Further, in the case of (ii) the classification problem (when it is desired to obtain the discrete value c(εC) as the output signal Y), the output signal Y is converted into the probability distribution P(c) related to c using a softmax function or the like. The error e with respect to the discrete value c′ (εC) that is the correct output signal is calculated as the cross entropy.

An end-to-end optimization method based on only a pair of an input signal and a correct output signal as shown in FIG. 1 needs to be implemented by dividing a complicated process performed by a human into fine processes. Therefore, it has been used for many tasks in recent years.

By increasing the number of non-linear conversions (that is, the number of layers of the neural network), it becomes possible to express a more complicated function, and the possibility of improving the performance can be increased. On the other hand, as the number of layers increases, there is also a problem that the amount of space calculation (that is, the number of parameters) and the amount of time calculation (that is, the number of times the matrix is multiplied) increase.

[B. Overview]
Next, the outline of the neural network according to the present embodiment will be described.

FIG. 2 is a schematic diagram for explaining deep learning according to the present embodiment. FIG. 2 shows a neural network 1 similar to that shown in FIG. Like the neural network 10 shown in FIG. 1, the neural network 1 performs a plurality of nonlinear conversions on an input signal and outputs an output signal.

Similar to the neural network 10 shown in FIG. 1, the neural network 10 typically includes an input layer 2, one or more identical or different hidden layers 4, and an output layer 6. Each of the input layer 2, the hidden layer 4, and the output layer 6 includes a vector indicating a state and an activation function. Adjacent layers are coupled via affine transformation or the like.

In the neural network 10 shown in FIG. 1, the optimization is executed only based on the deepest layer information (that is, the output signal Y of the output layer 6), whereas in the neural network 1 according to the present embodiment. , The optimization is executed by using the information of other layers in addition to the information of the deepest layer. The neural network 10 adopts a network structure suitable for such optimization.

That is, the neural network 1 according to the present embodiment can extract output signals from layers other than the output layer 6. As a typical example, in the neural network 1 shown in FIG. 2A, output signals Y ₁ , Y ₂ ,..., Y _n−1 from the input layer 2, the hidden layer 4, and the output layer 6, respectively. Y _n can be output.

In the optimization of the neural network 1, information of a plurality of layers including the deepest layer is used. In general deep learning, only the error between the output signal Y of the deepest layer and the correct output signal is used, whereas in the present embodiment, the error between the output signal of each layer and the correct output signal is used. Is calculated, the calculated errors are integrated, and then the parameters are optimized. More specifically, the parameters are optimized by repeating the following processing (steps S11 to S13).

(1) The output signals Y ₁ , Y ₂ ,..., Y _n−1 , Y _n of each layer with respect to the input signal X are calculated (step S11).
(2) Output signal _{Y 1,} Y 2, _{calculated..,} Error _{e 1,} e 2 and each correct answer output signal _{Y n-1, Y n,} ···, and _{e n-1, e n} Yes (step S12)
(3) the error _{e n, e n-1,} ···, e 2, e 1 updates the parameters in order toward the shallow layer from the deepest layer based on (Back Propagation) (step S13)
In the error back propagation in step S13, the errors e _i of the corresponding layers are considered in order. That is, in the kth layer (1≦k<N), not only the error information (gradient) backpropagated from the (k+1)th layer, but also the error e _k calculated in the kth layer After being directly received by the layer, the parameters of the kth layer are updated in consideration of both.

That is, the process of optimizing the parameter is performed on the basis of the error information given to the target layer for which the parameter is optimized by back propagation and the error calculated for the output signal of the target layer. It includes the process of optimizing the parameters of the layers of.

In this way, the parameters of each layer may be updated by back-propagating the error information from the deepest layer to the shallow layer while considering the errors of the corresponding layers in order.

The technical idea according to the present embodiment is not limited to a specific type of neural network, but can be applied to various types of neural networks. For example, it is applicable to neural networks such as CNN (Convolutional Neural Network), Stacked RNN (Recurrent Neural Network), and Transformer (one form of neural machine translation).

Further, as described above, as the output signal, a signal normalized using a sigmoid function or the like is used in the case of a regression problem, and a probability distribution is calculated using a softmax function or the like in the case of a classification problem. The converted signal is used. The above-described optimization method can be applied to any type of output signal.

In the neural network 1 according to the present embodiment, the parameters are optimized so that the output signals Y ₁ , Y ₂ ,..., Y _n−1 , Y _n of each layer have a small error with respect to the correct output signal. To be done. Therefore, in the inference process, not only the information of the deepest layer (that is, the output signal Y _n of the output layer 6) but also the information of other layers (that is, the output signals Y ₁ , Y ₂ ,..., Y _{n−). It} is highly possible that sufficient performance can be exhibited even by using ₁ ). As a result, in the inference processing, the output signal of an arbitrary layer among the plurality of layers can be used as the inference result depending on the required processing speed and performance.

FIG. 2A shows an example in which the output signal of each layer is compared with the correct output signal and the error is calculated in each layer. However, as shown in FIG. 2B, the error calculated in each layer is May be integrated. Hereinafter, the case where the averaging process is adopted will be described as a method of integrating the errors, but an arbitrary method may be adopted.

In the optimization method shown in FIG. 2B, the parameters are optimized by repeating the following processing (steps S11 to S15).

(1) The output signals Y ₁ , Y ₂ ,..., Y _n−1 , Y _n of each layer with respect to the input signal X are calculated (step S11).
(2) Output signal _{Y 1,} Y 2, _{calculated..,} Error _{e 1,} e 2 and each correct answer output signal _{Y n-1, Y n,} ···, and _{e n-1, e n} Yes (step S12)
(3) the error _e 1 of the _layers, e _{2, ···,} by averaging between the _{e n-1, e n} calculates an average error _{e unf} (step S14)
(4) Parameters are updated in order from the deepest layer to the shallowest layer based on the average error e _unf (error back propagation) (step S15).
In this way, the process of optimizing the parameters may employ a process of integrating the calculated errors. The process of integrating the errors includes a process of integrating the calculated errors and calculating error information for back propagation from the deepest layer. Typically, the average value of the calculated respective errors may be calculated as error information for back propagation from the deepest layer.

In this averaging process (step S14), the errors for the output signals of the individual layers are integrated. However, the output signals of the respective layers may be integrated using an arbitrary method, and then the error information may be calculated to optimize the parameters.

By adopting the neural network according to the present embodiment and the optimization method thereof, the spatial calculation amount slightly increases in the optimization phase (during training), but in the inference phase (during usage) Can be reduced. Further, the amount of time calculation increases in the optimization phase (during training), but can be reduced in accordance with the request in the inference phase (during use).

FIG. 3 is a flowchart showing a main part of a processing procedure according to the present embodiment. FIG. 3A shows a processing procedure of the optimization processing according to the present embodiment, and FIG. 3B shows a processing procedure of the inference processing according to the present embodiment.

FIG. 3A shows a processing procedure of an optimization method for optimizing parameters of a neural network (model) having a plurality of layers. The main steps shown in FIG. 3A are typically realized by the processor executing the optimization program.

With reference to FIG. 3(A), first, training data used for the optimization process in which the input signal and the correct output signal are associated with each other is prepared (step S50).

Next, the input signal included in the training data is input to the neural network, and the output signal output from each of the one or more layers including the deepest layer included in the neural network is calculated (step S52).

Then, the error of each calculated output signal with respect to the correct answer output signal associated with the input signal of the training data is calculated (step S54). The parameters of each layer included in the neural network are optimized based on the calculated respective errors (step S56). The process of optimizing the parameters of each layer may include the process of integrating the errors calculated in step S54. The process of integrating the errors is executed in the process of error back propagation, as shown in FIGS. 2A and 2B described above. In the case shown in FIG. 2B, the process of integrating the errors is also executed before the error back propagation.

Usually, the processes of steps S52 to S56 are repeated a preset number of times or until the accuracy of the verification data (development data) prepared separately from the training data converges.

FIG. 3B shows a processing procedure of an inference method using an optimized model including a neural network having a plurality of layers. The main steps shown in FIG. 3B are typically realized by a processor executing an inference program.

Here, the optimized model is generated in accordance with the processing procedure of the optimization method shown in FIG. That is, the optimized model includes an output signal output from each of one or more layers including the deepest layer, which is calculated when an input signal included in the training data is input to the neural network, and an output signal included in the training data. It is generated by optimizing the parameters based on the respective errors from the correct output signal associated with the input signal.

Referring to FIG. 3(B), an arbitrary input signal is input to the optimized model (step S60). Then, the output signal is calculated in order toward the deepest layer of the optimized model (step S62). That is, the nonlinear transformation defined in each layer is sequentially performed on the input signal.

Finally, the output signal of an arbitrary layer including the deepest layer among the plurality of layers included in the optimized model is output as an inference result (step S64). Then, the inference process ends.

The layer in which the output signal is output as the inference result may be determined based on at least one of the inference performance of the output signal and the time required until the output signal is output. This point will be described with reference to specific examples in Embodiments 1 to 3 described later.

According to the neural network and its optimization method according to the present embodiment, it is possible to update the parameters that more directly reflect the error information for all layers, so that the robustness of the model is improved. be able to. Therefore, when using the optimized model (inference phase), even when using only an arbitrary number of layers that is smaller than during training (optimization phase), it is possible to prevent the performance from extremely deteriorating, and eventually The processing speed can be improved. In this way, when the N-layer neural network is optimized, it is possible to realize flexibility of N stages of 1 to N layers during use (inference phase).

[C. Application example]
Next, an application example to which the neural network according to the present embodiment and its optimization method are applied will be described.

As described above, the neural network according to the present embodiment and its optimization method can be applied to all neural networks. In the present specification, a series conversion model, in particular, a neural machine translation is assumed as an example of an application.

Specifically, in Embodiments 1 and 2 described later, a 6-layer Transformer model as shown in Non-Patent Document 8 is adopted, and in Embodiment 3 described later, it is described in Non-Patent Document 2. Such a 6-layer Recurrently Stacked Transformer (RS-Transformer) model was adopted.

FIG. 4 is a schematic diagram showing an example of a Transformer model that realizes neural machine translation. Referring to FIG. 4, in the Transformer model 20, the input signal is a sequence in the first language and the output signal is a sequence in the second language, thereby realizing neural machine translation. Note that neural machine translation can be regarded as a classification problem. More specifically, the Transformer model 20 includes an encoder 30 and a decoder 40.

The encoder 30 outputs characteristic information included in the input signal. The encoder 30 has N hidden layers 32 for extracting characteristic information contained in the input signal. An input layer 36 for converting each word in the sequence (natural language) which is the input signal into a fixed-dimensional vector is arranged in the preceding stage of the encoder 30.

The decoder 40 receives the previously output output signal (already output) and the characteristic information included in the input signal, and determines the output signal. The decoder 40 has M hidden layers 42. An input layer 46 for converting each word in the already output sequence (natural language) into a fixed-dimensional vector is arranged in the preceding stage of the decoder 40.

In the first to third embodiments, the performance of neural machine translation using the Transformer model as shown in FIG. 4 was evaluated.

[D. Embodiment 1]
FIG. 5 is a schematic diagram for explaining the optimization process according to the first embodiment. Referring to FIG. 5, in the first embodiment, error information is generated using respective output signals from M hidden layers 42 (6 layers as an example) of decoder 40. The error information sequentially back propagates along the path 50 from the deepest layer to the shallowest layer of the decoder 40 and from the deepest layer to the shallowest layer of the encoder 30.

FIG. 6 is a flowchart showing a main processing procedure of the optimization processing according to the first embodiment. Referring to FIG. 6, first, training data used for the optimization process is prepared (step S100).

Then, based on the input signal included in the training data, to calculate the tensor X input as an input signal enc ₀ of the encoder 30 of the Transformer Model 20 _(enc 0 = X) (step S102). In addition, the error information loss is initialized to zero (loss=0) (step S104).

Then, the output signal of each layer of the encoder 30 is calculated. That is, the output signal enc _i =L _i ^enc (enc _i−1 ) is calculated for each index i (1≦i≦N) indicating the layer position of the hidden layer 32 included in the encoder 30 (step S110). Here, L _i ^enc represents the nonlinear transformation of the i-th hidden layer 32 included in the encoder 30. The output signal enc _N (characteristic information included in the input signal), which is the output of the deepest layer of the encoder 30, is given to the decoder 40.

Then, the output signal and error of each layer of the decoder 40 are calculated. That is, for the index j (1≦j≦M) indicating the layer position of the hidden layer 42 included in the decoder 40, the output signal dec _j =L _j ^dec (dec) is referenced with reference to the output signal enc _N of the deepest layer of the encoder. _j−1 , enc _N ) are calculated (step S120). Here, L _j ^dec indicates the nonlinear transformation of the j-th hidden layer 32 included in the decoder 40.

In each layer of the decoder 40 calculates an output signal as a probability distribution _{Y ^ j = softmax (dec j} ) ( step S122). Note that due to the restrictions of the electronic filing system, the hat symbol “^” is described after the target character (the same applies hereinafter).

Further, the error between the output signal Y^ _j as a probability distribution and the correct output signal Y as a discrete value is calculated as cross entropy and added to the error information loss (step S124). That is, the error information loss=loss+cross_entropy(Y^ _j , Y) is calculated.

Finally, the average value of the errors calculated in each layer of the decoder 40 is determined as error information used for parameter optimization (step S130). That is, the error information loss=loss/M is calculated. Then, based on the calculated error information loss, the parameters are updated in order from the deepest layer to the shallow layer of the decoder 40, and subsequently, the parameters are updated in order from the deepest layer to the shallow layer of the encoder 30. (Step S132).

Usually, the above-described processing of step S102 and subsequent steps are repeated a plurality of times.
Although not shown in FIGS. 5 and 6 for convenience of description, in practice, processing for avoiding over-learning such as batch normalization (Batch Normalization) and dropout (dropout) is appropriately arranged. You may. Further, any process for speeding up the optimization process may be appropriately arranged (for example, see Non-Patent Documents 5 to 7).

The performance of the optimization process described above was evaluated for the following two translation tasks.
As the first translation task, an English-Japanese translation task using a multilingual parallel translation corpus for speech translation developed by National Institute of Information and Communications Technology (NICT) (see Non-Patent Documents 9 and 10) was set. From the multilingual parallel corpus, about 400,000 sentence pairs were set as training data, and about 2000 sentence pairs were set as evaluation data.

As the second translation task, an English-German translation task using parallel translation data in the news field (see Non-Patent Document 11 and Non-Patent Document 12) was set. From the bilingual data in the news field, about 5.6 million sentence pairs were set as training data, and about 3000 sentence pairs were set as evaluation data.

For each translation task, translation performance was evaluated by BLEU score (see Non-Patent Document 13) and translation speed.

FIG. 7 is a graph showing an evaluation result of an English-Japanese translation task using a multilingual parallel translation corpus for speech translation. FIG. 8 is a graph showing an evaluation result of an English-German translation task using parallel translation data in the news field.

“BLUE: 1×6” shown in FIGS. 7 and 8 is obtained by the optimization process according to the procedure shown in FIG. 6 after the encoder 30 has 6 layers and the decoder 40 has 6 layers. The translation performance of the optimized model is shown. “BLUE: 6-k” is an optimization process according to a related technique for six types of models in which the number of layers of the encoder 30 is 6 and the number of layers of the decoder 40 is different from 1 to 6 (only for error information of the deepest layer). The translation performance of each optimized model (6 types) obtained by the optimization processing based on FIG. “BLUE:6-6” indicates the translation performance of the optimized model obtained by the optimization processing according to the related technique, with the number of layers of the encoder 30 set to 6 and the number of layers of the decoder 40 set to 6. The translation performance shown in "BLUE:6-6" is that the number of layers of the decoder 40 used in the inference phase is different from 1 to 6 for the same optimized model.

It should be noted that execution of the optimization process according to the procedure shown in FIG. 6 took about 2.0 times the time taken to execute the optimization process according to the related art.

The horizontal axes “1” to “6” in FIGS. 7 and 8 indicate the layers of the decoder 40 used in the inference phase (when used). For example, in the position where the horizontal axis is "3", performance is shown when the output signal from the third layer of the decoder 40 is used as the inference result.

As shown in FIG. 7 and FIG. 8, in the optimized model (BLUE:6-6) obtained by the optimization process according to the related art, the number of layers of the decoder 40 to be used is determined from the optimization phase (during training). It can be seen that the translation performance (BLUE score) is extremely deteriorated when is also reduced.

Further, regarding the model having the same number of layers of decoder 40 as that used in the inference phase, in the optimized model (BLUE: 6-k) obtained by the optimization processing according to the related art, the number of layers of decoder 40 is It can be seen that between 2 and 6, the translation performance is almost the same as BLUE:6-6.

On the other hand, according to the optimization processing according to the first embodiment, six models 6-1 to 6-6 are simultaneously optimized based on the output signals of all layers of the decoder 40 (BLUE 1×6), it can be seen that even if the number of layers of the decoder 40 used is reduced from that in the optimization phase (during training), the translation performance is slightly degraded.

Further, the “translation time [sec]” shown in FIGS. 7 and 8 uses the optimized model (BLUE: 1×6) obtained by the optimization processing according to the first embodiment to evaluate data (English). The processing time (including model loading time and input sentence encoding time) required for translation of about 2000 sentences for the Japanese translation task and about 3000 sentences for the English-German translation task is shown. According to the graph of the processing time, it can be seen that a significant speedup can be realized by reducing the number of layers.

Specifically, in the first translation task (English-Japanese translation task), the processing time can be reduced by about 40% by using the two layers of the decoder 40, and by using the three layers of the decoder 40, It can be seen that the processing time can be reduced by about 30%. Further, in the second translation task (English-German translation task), the processing time can be reduced by about 57% by using the two layers of the decoder 40, and the processing time can be reduced by using the three layers of the decoder 40. It can be seen that the reduction can be about 36%.

[E. Embodiment 2]
In the first embodiment, the optimization process using the error information calculated based on the output signals of the respective layers of the decoder 40 has been described, but in the second embodiment, the output signals of the respective layers of the encoder 30 and the decoder 40 are used. The optimization process using the error information calculated based on this will be described.

9 and 10 are schematic diagrams for explaining the optimization process according to the second embodiment. Referring to FIGS. 9 and 10, in the second embodiment, output signals from each of N hidden layers 32 (6 layers as an example) of encoder 30 and M hidden layers of decoder 40. Error information is generated using the respective output signals from 42 (for example, 6 layers). The generated error information back propagates along the path 50 in order from the deepest layer to the shallowest layer of the decoder 40 and from the deepest layer to the shallowest layer of the encoder 30.

FIG. 9 shows, as an example, a case where the output signal enc _N from the deepest layer (N-th hidden layer 32) of the encoder 30 is input to the decoder 40, and FIG. The case where the output signal enc _i from the th layer (i-th hidden layer 32) is input to the decoder 40 is shown. As shown in FIG. 9 and FIG. 10, in the second embodiment, the number of combinations (N×M) of N kinds of output signals from each layer of encoder 30 and M kinds of output signals from each layer of decoder 40. There may be error information for each.

FIG. 11 is a flowchart showing the main processing procedure of the optimization processing according to the second embodiment. Referring to FIG. 11, first, training data used for the optimization processing is prepared (step S200).

Then, the tensor X input as the input signal enc ₀ of the encoder 30 of the Transformer model 20 is calculated based on the input signal included in the training data (enc ₀ =X) (step S202). Further, the error information loss is initialized to zero (loss=0) (step S204).

Then, the output signal and error of each layer of the encoder 30 are calculated. That is, the processes of steps S210 to S216 are repeated for the index i (1≦i≦N) indicating the layer position of the hidden layer 32 included in the encoder 30.

More specifically, the output signals enc _i =L _i ^enc (enc _i-1 ) are calculated (step S210). Here, L _i ^enc represents the nonlinear transformation of the hidden layer 32 included in the encoder 30. At this point, the output signal enc _i output from the encoder 30 is given to the decoder 40.

Further, for each index i, the output signal and error of each layer of the decoder 40 are calculated. That is, the processes of steps S212 to S216 are repeated for the index j (1≦j≦M) indicating the layer position of the hidden layer 42 included in the decoder 40.

More specifically, the output signals dec _j =L _j ^dec (dec _j-1 , enc _i ) are calculated (step S212). Here, L _j ^dec indicates the nonlinear transformation of the j-th hidden layer 32 included in the decoder 40.

Then, the output signal Y^ _i,j =softmax(dec _j ) as the probability distribution is calculated (step S214). Furthermore, the error between the output signal Y^ _i,j as the probability distribution and the correct output signal Y as the discrete value is calculated as the cross entropy and added to the error information loss (step S216). That is, the error information loss=loss+cross_entropy(Y^ _i,j ,Y) is calculated.

Finally, the average value of the errors calculated for the combination of each layer (N layer) of the encoder 30 and each layer (M layer) of the decoder 40 is determined as the error information used for parameter optimization (step S220). ). That is, the error information loss=loss/(N×M) is calculated. Then, based on the calculated error information loss, the parameters are updated in order from the deepest layer to the shallow layer of the decoder 40, and subsequently, the parameters are updated in order from the deepest layer to the shallow layer of the encoder 30. (Step S222).

Usually, the above-described processing of step S202 and subsequent steps are repeated a plurality of times.
For convenience of explanation, although not shown in FIGS. 9 to 11, in practice, processing for avoiding over-learning such as batch normalization and dropout may be appropriately arranged. Further, any process for speeding up the optimization process may be appropriately arranged (for example, see Non-Patent Documents 5 to 7).

The performance of the above-described optimization processing is applied to the first translation task (same as the English-Japanese translation task described in the first embodiment) and the second translation task (same as the English-German translation task described in the first embodiment). evaluated. Similar to the first embodiment, the translation performance was evaluated by the BLEU score and the translation speed, respectively.

FIG. 12 is a graph showing the evaluation result of the English-Japanese translation task according to the second embodiment. FIG. 12A shows the evaluation result of the BLUE score for the optimized model obtained by the optimization processing according to the procedure shown in FIG. 11, and FIG. 12B shows the optimum result according to the procedure shown in FIG. The evaluation result of the processing time about the optimized model obtained by the optimization processing is shown. The processing time represents the processing time required for translating the evaluation data (about 2000 sentences) (including the model loading time and the time required for encoding the input sentence).

FIG. 13 is a graph showing the evaluation result of the English-German translation task according to the second embodiment. FIG. 13A shows the evaluation result of the BLUE score for the optimized model obtained by the optimization processing according to the procedure shown in FIG. 11, and FIG. 13B shows the optimum result according to the procedure shown in FIG. The evaluation result of the processing time about the optimized model obtained by the optimization processing is shown. The processing time represents the processing time required for translating the evaluation data (about 3000 sentences) (including the model loading time and the time required for input sentence encoding).

In FIGS. 12A, 12B, 13A, and 13B, the horizontal axes “1” to “6” indicate the number of layers of the encoder 30 used in the inference phase (when used). Indicates. Further, the vertical axes “1” to “6” indicate the number of layers of the decoder 40 used in the inference phase (when used). For example, at the position where the horizontal axis is “3” and the horizontal axis is “3”, the output signal (characteristic information included in the input signal) from the third layer of the encoder 30 is input to the decoder 40, and the decoder 40 The performance is shown when the output signal from the third layer of 40 is used as the inference result.

It should be noted that execution of the optimization process according to the procedure shown in FIG. 11 took about 9.5 times the time taken to execute the optimization process according to the related art.

As shown in FIG. 12(A), by using four or more layers of the encoder 30 and three or more layers of the decoder 40, the translation performance by the optimized model obtained by the optimization processing according to the related art ( It can be seen that the translation performance (26.09 to 26.53 points) equivalent to 27.09 points) can be exhibited.

As shown in FIG. 12B, when four or more layers of the encoder 30 and three layers of the decoder 40 are used, the optimized model obtained by the optimization process according to the related art is used. It can be seen that the processing time can be reduced by about 30% (57.25 to 59.49 unit hours) as compared with the processing time (85.20 unit hours).

Further, as shown in FIG. 13A, by using the four layers of the encoder 30 and the four layers of the decoder 40, the translation performance by the optimized model obtained by the optimization processing according to the related art (32. It can be seen that the translation performance (31.44 to 32.03 points) equivalent to 31 points) can be exhibited.

As shown in FIG. 13B, when four or more layers of the encoder 30 and four layers of the decoder 40 are used, the optimized model obtained by the optimization process according to the related art is used. It can be seen that the treatment time can be reduced by about 35% as compared with the treatment time (251.69 unit hours) of (137.88-161.44 unit hours).

According to the evaluation results of FIG. 12B and FIG. 13B, reducing the number of layers of the encoder 30 used is not very effective for speeding up the processing, while the decoder 40 used It can be seen that reducing the number of layers is more effective for speeding up the processing.

[F. Third Embodiment]
In Embodiments 1 and 2, a model in which encoder 30 and decoder 40 have a plurality of different layers has been illustrated. By using the same layer recursively instead of the encoder 30 and the decoder 40 having such a plurality of different layers, it is possible to realize the non-linear conversion equivalent to that of a plurality of layers while suppressing the memory usage ( See Non-Patent Document 2, etc.). In the third embodiment, an optimization process for a model that recursively uses the same layer will be described.

FIG. 14 is a schematic diagram for explaining the optimization process according to the third embodiment. Referring to FIG. 14, the RS-Transformer model 20A includes an encoder 30A having a hidden layer 32 recursively enabled and a decoder 40A having a hidden layer 42 recursively enabled. The output signal of encoder 30A (characteristic information included in the input signal) is output to decoder 40A.

By using the hidden layer 32 recursively N times, a non-linear conversion corresponding to N layers can be realized, and by using the hidden layer 42 recursively M times, a non-linear conversion corresponding to M layers can be realized. it can. On the other hand, since the hidden layer 32 and the hidden layer 42 exist only for one layer, the number of parameters defining the RS-Transformer model 20A can be reduced as compared with the Transformer model 20.

In the third embodiment, for simplification, as in the first embodiment, an error signal calculated based on only the output signal of the deepest layer is used for the encoder 30 in the same manner as in the first embodiment, and the decoder 40A has the same structure. The optimization process using the error information calculated based on the output signal of each layer (that is, the output signal in each recursive process) is executed. However, similar to the second embodiment, the error signal is calculated based on the output signal of each layer of the encoder 30A and the output signal of each layer of the decoder 40A (that is, the output signal in each recursive process). The optimization process using the error information may be adopted.

In the optimization process for the RS-Transformer model 20A shown in FIG. 14, the error information is calculated along the path 50 from the deepest layer to the shallowest layer of the decoder 40A and from the deepest layer to the shallowest layer of the encoder 30A. Propagate back in sequence. Also in the process of back-propagating this error information, the recursive process is executed a predetermined number of times. That is, the parameters are optimized by back-propagating the error information multiple times to the same hidden layer.

FIG. 15 is a flowchart showing the main processing procedure of the optimization processing according to the third embodiment. Referring to FIG. 15, first, training data used for the optimization process is prepared (step S300).

Then, based on the input signal included in the training data, to calculate the tensor X input as an input signal enc ₀ of the encoder 30A of the RS-Transformer Model 20A _(enc 0 = X) (step S302). The error information loss is initialized to zero (loss=0) (step S304).

Then, the output signal of each layer of the encoder 30A is calculated. That is, the output signal enc _i =L ^enc (enc _i−1 ) is calculated for each index i (1≦i≦N) indicating the number of times the recursive process is performed on the hidden layer 32 of the encoder 30A (step S310). Here, L ^enc represents the nonlinear transformation of the hidden layer 32 included in the encoder 30A. The output signal enc _N (corresponding to the output signal of the deepest layer) which is the output of the encoder 30A obtained by the recursive processing N times is given to the decoder 40A.

Then, the output signal and error of each layer of the decoder 40A are calculated. That is, the output signal dec _j =L ^dec (dec _j−1 , enc _N ) is calculated for each index j (1≦j≦M) that indicates the number of times the recursive process is performed on the hidden layer 42 of the decoder 40A (step S320). ). Then, the output signal Y^ _j =softmax(dec _j ) as the probability distribution is calculated (step S322). Further, the error between the output signal Y^ _j as the probability distribution and the correct output signal Y as the discrete value is calculated as the cross entropy and added to the error information loss (step S324). That is, the error information loss=loss+cross_entropy(Y^ _j , Y) is calculated.

Finally, the average value of the errors calculated in each layer of the decoder 40A is determined as error information used for parameter optimization (step S330). That is, the error information loss=loss/N is calculated. Then, based on the calculated error information loss, the parameters are updated in order from the deepest layer to the shallow layer of the decoder 40A, and subsequently the parameters are updated in order from the deepest layer to the shallow layer of the encoder 30A. (Step S332).

Normally, the above-described processing of step S302 and thereafter is repeated a plurality of times.
Note that, for convenience of description, although not shown in FIGS. 14 and 15, in practice, processing for avoiding over-learning such as batch normalization and dropout may be appropriately arranged. Further, any process for speeding up the optimization process may be appropriately arranged (for example, see Non-Patent Documents 5 to 7).

The performance of the optimization process described above was evaluated for the first translation task (same as the English-Japanese translation task described in the first embodiment). Similar to the first embodiment, the translation performance was evaluated by the BLEU score. In addition, the number of times of recursive processing of the encoder 30A and the decoder 40A is the same (that is, N=M in FIG. 15).

The optimized model of the RS-Transformer model 20A shown in FIG. 14 has a data size reduced to 47% as compared with the optimized model of the Transformer model 20 according to the first embodiment.

FIG. 16 is a graph showing the evaluation result of the English-Japanese translation task according to the third embodiment. FIG. 16(A) shows the evaluation result of the BLUE score for the optimized model obtained by the optimization process according to the related technique shown in Non-Patent Document 2, and FIG. The evaluation result of the BLUE score about the optimized model obtained by the optimization process according to the procedure shown is shown.

In FIG. 16A and FIG. 16B, the horizontal axes “1” to “6” indicate the number of times of recursive processing in the optimization phase (during training). Further, the vertical axes “1” to “6” indicate the number of times of recursive processing of the decoder 40A used in the inference phase (when used). The number of recursive processes of the encoder 30A used in the inference phase is the same as that in the optimization phase.

In the graphs shown in FIGS. 16A and 16B, the values on the diagonal line from the upper left to the lower right indicate the same number of times of recursive processing as the optimization phase (during training) in the inference phase (during use). The result when executed is shown. The lower left part of the graphs shown in FIGS. 16A and 16B means a case where the number of times of recursive processing of the decoder 40A is increased more than in the optimization phase (during training). In principle, this kind of processing is also possible, but since it is a conflict from the original purpose of speeding up the processing, it has not been actually evaluated (shown as a value of "0.00"). ing).

As shown in FIGS. 16A and 16B, it can be seen that the translation performance is improved as the number of times of recursive processing is increased (toward the right side of the drawing). However, as shown in FIG. 16A, in the optimized model obtained by the optimization processing according to the related art, when the number of times of the recursive processing of the decoder 40A is reduced from the optimization phase (during training). Shows that the translation performance (BLUE score) is extremely deteriorated.

On the other hand, as shown in FIG. 16B, according to the optimized model obtained by the optimization processing according to the procedure shown in FIG. 15, the number of times of the recursive processing of the decoder 40A is set to the optimization phase (during training). ), the translation performance is only slightly deteriorated (BLEU score is 0.5 points at maximum), and the translation performance equivalent to that of the optimized model shown in Non-Patent Document 2 can be maintained. I know that

[G. Hardware configuration]
Next, an example of a hardware configuration for implementing the optimization process and the inference process according to the present embodiment will be described.

FIG. 17 is a schematic diagram showing an example of a hardware configuration for realizing the optimization process and the inference process according to the present embodiment. The optimization process and the inference process according to the present embodiment are typically realized using information processing device 100 which is an example of a computer.

With reference to FIG. 17, the information processing apparatus 100 has a CPU (central processing unit) 102, a GPU (graphics processing unit) 104, a main memory 106, a display 108, and a network interface (main interface) as main hardware components. The I/F (interface) 110, the secondary storage device 112, the input device 122, and the optical drive 124 are included. These components are connected to each other via an internal bus 128.

CPU 102 and/or GPU 104 is a processor that realizes the optimization process and the inference process according to the present embodiment by executing various programs to be described later. A plurality of CPUs 102 and GPUs 104 may be arranged, or a plurality of cores may be provided.

The main memory 106 is a storage area for temporarily storing (or caching) program codes, work data, etc. when the processor (CPU 102 and/or GPU 104) executes processing, and for example, DRAM (dynamic random access memory). ) And SRAM (static random access memory).

The display 108 is a display unit that outputs a user interface related to processing, a processing result, and the like, and is configured by, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display.

The network interface 110 exchanges data with an arbitrary information processing device on the Internet or an intranet. As the network interface 110, for example, an arbitrary communication method such as Ethernet (registered trademark), wireless LAN (local area network), and Bluetooth (registered trademark) can be adopted.

The input device 122 is a device that receives instructions and operations from the user, and is configured with, for example, a keyboard, a mouse, a touch panel, a pen, and the like. Further, the input device 122 may include a sound collecting device for collecting a sound signal necessary for learning and decoding, and includes an interface for receiving an input of the sound signal collected by the sound collecting device. You can leave.

The optical drive 124 reads information stored in an optical disc 126 such as a CD-ROM (compact disc read only memory) or a DVD (digital versatile disc) and outputs it to other components via an internal bus 128. The optical disk 126 is an example of a non-transitory recording medium, and is distributed in a state in which an arbitrary program is stored in a nonvolatile manner. The optical drive 124 reads the program from the optical disk 126 and installs it in the secondary storage device 112 or the like, so that the computer functions as the information processing device 100. Therefore, the subject matter of the present invention may be a program itself installed in the secondary storage device 112 or the like, or a recording medium such as an optical disk 126 storing a program for implementing the functions and processes according to the present embodiment. ..

FIG. 17 shows an optical recording medium such as the optical disk 126 as an example of a non-transitory recording medium, but the present invention is not limited to this, and a semiconductor recording medium such as a flash memory or a magnetic recording medium such as a hard disk or a storage tape. , MO (magneto-optical disk) or the like may be used.

The secondary storage device 112 stores programs and data necessary for causing a computer to function as the information processing device 100. For example, it is configured by a non-volatile storage device such as a hard disk or SSD (solid state drive).

More specifically, the secondary storage device 112 includes, in addition to an OS (operating system) not shown, typically an optimization program 114 for realizing optimization processing and an inference for realizing inference processing. A program 116, parameters 118 defining an optimized model, and training data 120 are stored.

The optimization program 114 is executed by the processor (CPU 102 and/or GPU 104) to realize the parameter optimization process shown in FIG. The inference program 116 is executed by the processor (CPU 102 and/or GPU 104) to implement the inference processing shown in FIG.

A part of the library or function module required when the processor (CPU 102 and/or GPU 104) executes the program may be replaced by the library or function module provided as standard by the OS. In this case, the program alone does not include all the program modules required to realize the corresponding functions, but the target processing can be realized by installing the program module under the OS execution environment. Even a program that does not include such a part of the library or the functional module can be included in the technical scope of the present invention.

Also, these programs may be distributed not only by being stored in any of the above-described recording media and being distributed, but also by being downloaded from a server device or the like via the Internet or an intranet.

FIG. 17 shows an example in which the information processing apparatus 100 is configured using a single computer, but the present invention is not limited to this, and a plurality of computers connected via a computer network cooperate explicitly or implicitly. The information processing apparatus 100 and a system including the information processing apparatus 100 may be realized.

All or part of the functions realized by the processor (CPU 102 and/or GPU 104) executing the program may be realized using a hard-wired circuit such as an integrated circuit. For example, it may be realized by using an ASIC (application specific integrated circuit) or an FPGA (field-programmable gate array).

Those skilled in the art will be able to realize the information processing apparatus 100 according to the present embodiment by appropriately using the technology according to the time when the present invention is implemented.

For convenience of explanation, the same information processing apparatus 100 is used to execute the optimization process and the inference process, but the optimization process and the inference process may be implemented using different hardware.

[H. Summary]
According to the optimization method according to the present embodiment, the parameters of the neural network are optimized based on the error information obtained by comparing the output signals of the plurality of layers including the deepest layer of the neural network and the correct output signal. To do. As a result, even when the output signal internally output from the hidden layer of the neural network is used, it is possible to avoid a situation where the performance is extremely deteriorated with respect to the output signal of the deepest layer.

As a result, in the inference process, it is practically possible to use the output signal of the layer shallower than the deepest layer as the inference result, without using the output signal of the deepest layer as the inference result.

According to the optimization method according to the present embodiment, it is possible to generate an optimized model that can obtain an output signal with relatively high performance from each layer, so that the required specifications (for example, the inference performance of the output signal and the output signal are output). The output signal of any layer can be used as the inference result according to the time required for the process), so that not only the processing speed can be increased, but also the flexibility can be improved.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description of the embodiments but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

1,10 Neural network, 2,36,46 input layer, 4,32,42 hidden layer, 6 output layer, 20,20A model, 30,30A encoder, 40,40A decoder, 50 path, 100 information processing device, 102 CPU, 104 GPU, 106 main memory, 108 display, 110 network interface, 112 secondary storage device, 114 optimization program, 116 inference program, 118 parameters, 120 training data, 122 input device, 124 optical drive, 126 optical disk, 128 internal bus.

Claims (6)

An optimization method for optimizing parameters of a neural network having a plurality of same or different layers, comprising:
Preparing training data in which the input signal and the correct output signal are associated with each other, and
The input signal is input to the neural network to calculate an output signal output from the deepest layer included in the neural network, and an output signal output from each of one or more layers including the deepest layer is calculated. A step of calculating,
Calculating the error of each of the calculated output signals with respect to the correct output signal associated with the input signal,
Optimizing the parameters of each layer included in the neural network based on the calculated respective errors. The optimization method according to claim 1, wherein the optimizing step includes a step of integrating the calculated respective errors. The step of optimizing, based on the error information given by backpropagation to the target layer for which the parameter is optimized, and the error calculated for the output signal of the target layer, The optimization method according to claim 1, comprising the step of optimizing parameters. An optimization program for causing a computer to execute the optimization method according to claim 1. An inference method using an optimized model consisting of a neural network having a plurality of the same or different layers,
Inputting any input signal into the optimized model;
Calculating the output signal in order towards the deepest layer of the optimized model,
Out of the plurality of the same or different layers included in the optimized model, the output signal of any layer including the deepest layer, which is determined based on a request, is output as an inference result.
The optimized model includes output signals output from each of one or more layers including the deepest layer, which are calculated when an input signal included in training data is input to the neural network, and the training data. The inference method generated by optimizing parameters based on the respective errors from the correct answer output signal associated with the input signal included in. An inference program for causing a computer to execute the optimization method according to claim 5.

Images