JP6633999B2 - Encoder learning device, conversion device, method, and program - Google Patents

Description

  The present invention relates to an encoder learning device, a conversion device, a method, and a program, and particularly to an encoder learning device, a conversion device, a method, and a program for solving a problem of transforming a discrete structure.

  The background art will be described using the problem in the field of natural language processing as a theme. The technique of processing a natural language by a computer can be said to be a technique of processing a set of discrete symbols such as superficial characters and words appearing in a sentence. For example, consider an automatic translation system that inputs a sentence in one language and outputs a sentence in another language. In this system, processing is performed by regarding input and output sentences as word strings (character strings). Therefore, it can be considered that a computer in the system is performing a process of converting a discrete structure (symbol structure) into another discrete structure. Also, a system such as a document summarization system, a dialogue system, and a sentence construction system that uses a language as an input / output can be said to be constituted by a process of converting a discrete structure into another discrete structure, like the above-described translation system. As described above, even in a natural language processing system other than the above-described natural language processing system, in a system that handles natural languages, what kind of conversion from input to output is performed because a target to be processed is a discrete structure such as a word, a sentence, and a document. Despite the difference in the definition, the processing framework is the same, and it can be reduced to a discrete structure transformation problem from a discrete structure. FIG. 1 shows various examples of a conversion problem in natural language processing.

  In recent years, a character string-character string conversion method based on a neural network has attracted attention. For example, Non-Patent Document 1 and Non-Patent Document 2 use a recurrent neural network framework to encode a discrete structure into a real-valued vector, and decode the discrete structure from the real-valued vector, thereby forming a discrete-structure to discrete-structure conversion problem. Has been realized.

  For example, Non-Patent Document 1 discloses that an encoder and a decoder are configured as shown in FIG.

Here, a function group and an operator for explanation are shown in FIG. As shown in FIG. 2, the sigmoid function is σ ₁ , the tanh function is σ ₂ , the softmax function is σ ₃ , and the relu function is σ ₄ , as shown in FIG. Each function is a function that receives a vector as an input and returns a vector having the same size (number of dimensions) as the input vector. Each function performs a predetermined calculation for each element of the input vector, and stores the result in the same element number (position) as the input vector. The same applies to a case where a vector is replaced with a matrix.

  The decoder described in Non-Patent Document 1 predicts words in a sequence by sequentially predicting words from the output of the decoding unit according to the following equation.

  In the example of FIG. 2, the predicted word is

And the first word is “It”.

  Here, the principle of the construction of the encoder will be described. The encoder constitutes the entire encoder by connecting the encoding units. Various configurations are conceivable inside the coding unit. Here, as an example, a case where the coding unit is configured by a recurrent neural network (RNN) and a case where the coding unit is configured by a long and short term storage memory (LSTM) will be described.

  The following shows equations (1) and (2) when calculating an encoding unit using RNN and when constructing an encoding unit using LSTM.

  The information input to each coding unit is a vector x of a normal input label and an intermediate (code) state z of the connected coding unit.

Non-Patent Document 2 discloses that an encoder and a decoder are configured as shown in FIG. In FIG. 4, as processing when the first word is generated, c _j calculated from the hidden layer vector obtained in each encoding unit in the encoder, and the first hidden layer vector h _{j of the} decoding unit. ^t (where t = 1) is used as an input to the word generation unit of the decoder.

Here, c _j is calculated according to the following equation (3).

... (3)

  The learning of the encoder and the decoder will be described. The learning of the encoder and decoder consists of the following parameters that minimize the loss function Ψ

Can be formulated as a minimization problem that searches for

  Parameter

Is the set of all parameters in the neural network. The function Ψ corresponds to the negative cross entropy between the correct word y _{n, t} and the current system prediction result on _{, t} . n represents a sentence number, and the number of correct answer data for learning is the upper limit. Let t represent the word number in the sentence.

Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V. Sequence to Sequence Learning with Neural Networks, Advances in Neural Information Processing Systems 27, pp. 3104-3112, 2014 Dzmitry Bahdanau and Kyunghyun Cho and Yoshua Bengio.Neural Machine Translation by Jointly Learning to Align and Translate ICLR-2015

  In general, it can be said that the performance of the conventional technique (such as a neural network) as described above has been greatly improved compared to the previous methods, but it is still far from texts created by humans. It is.

  Normally, in a method using a sequence encoder and a sequence decoder based on a neural network, a process from when an input sentence is given to when an output sentence is generated is represented by one network.

  In this case, the model is represented by a single network, so it is easy to understand.On the other hand, since it is a method to represent the complex main problem of converting sentences into sentences by one network model, the accuracy of machine learning problem is There is an aspect that learning a model to convert well is a very difficult problem.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an encoder learning device, a method, and a program that can accurately learn parameters for converting a discrete structure.

  It is another object of the present invention to provide a conversion device, method, and program for converting a discrete structure with high accuracy.

  In order to achieve the above object, an encoder learning apparatus according to a first aspect of the present invention provides a coder learning device which converts a discrete structure into a predetermined auxiliary problem based on correct data of the main problem. An encoder that connects an auxiliary problem generation unit that generates correct answer data and an encoding unit that encodes the element corresponding to each element of the input discrete structure, and predicts a solution of the auxiliary problem. An encoder including a predictor that performs decoding, a decoder connected to a decoding unit that decodes the elements of the discrete structure, and inputs a code output by the encoder and a solution to the auxiliary problem, and the main problem And a learning unit that learns parameters of the encoding unit, the decoding unit, and the predictor based on the correct data of the sub-problem and the correct data of the auxiliary problem.

  Further, in the encoder learning apparatus according to the first invention, the learning unit includes an encoder construction unit configured to construct the encoder based on the input discrete structure; Based on the initial value of the parameter or the updated parameter, each element of the input discrete structure is input to the corresponding encoding unit, sequentially calculated, and the code of the input discrete structure is calculated. And an encoder calculation unit that calculates the predictor to predict the solution of the auxiliary problem, and inputs the output code and the solution of the auxiliary problem to the decoder, and sequentially calculates A decoder calculation unit that outputs a discrete structure, an objective function calculation that calculates a value of an objective function represented by using the discrete structure output by the decoder calculation unit, and correct data for the main problem Part and notation A parameter updating unit that updates the parameter based on the value of the objective function, and a calculation by the encoder calculation unit, a calculation by the decoder calculation unit, and a calculation by the objective function calculation unit until a predetermined iteration end condition is satisfied. An end determination unit that repeats calculation and updating by the parameter updating unit may be included.

  A conversion device according to a second aspect is a conversion device for solving a main problem of converting an input discrete structure, wherein the element corresponding to each element of the discrete structure is based on the input discrete structure. An encoder connected to an encoding unit that encodes the encoder, and an encoder constructing unit that constructs an encoder including a predictor corresponding to a predetermined auxiliary problem with respect to the main problem; and Based on the encoder, each element of the input discrete structure is input to a corresponding one of the encoding units, sequentially calculated, and outputs the code of the input discrete structure, and the predictor And an encoder calculating section for predicting the solution of the auxiliary problem by calculating the output code and the solution of the auxiliary problem to a decoder connected to a decoding unit for decoding the element of the discrete structure. And calculate sequentially, discrete It is configured to include a decoder calculating section for outputting the granulation, the.

  Further, in the conversion device according to the second invention, the auxiliary problem is a problem of predicting a set of elements included in a converted discrete structure, a problem of predicting the number of elements included in a converted discrete structure, and This may be at least one of the problems of predicting a set of elements included in both the input discrete structure and the transformed discrete structure.

  In the encoder learning method according to a third aspect of the present invention, the auxiliary problem generating unit corrects a predetermined auxiliary problem with respect to the main problem based on correct data of the main problem for transforming the input discrete structure. A step of generating data, wherein the learning unit is connected to a coding unit for coding the element corresponding to each element of the input discrete structure, and predicts a solution of the auxiliary problem. An encoder including a predictor, a code output by the encoder, and a decoder connected to a decoding unit that decodes the elements of the discrete structure, which have the input of the solution of the auxiliary problem and the main problem. Learning the parameters of the encoding unit, the decoding unit, and the predictor based on the correct answer data and the correct answer data of the auxiliary problem. To.

  A conversion method according to a fourth aspect of the present invention is a conversion method for a conversion device for solving a main problem of converting an input discrete structure, wherein an encoder constructing unit determines the discrete structure based on the input discrete structure. Constructing an encoder including a predictor corresponding to a predetermined auxiliary problem with respect to the main problem, the encoder being connected to a coding unit that encodes the element, corresponding to each element of And an encoder calculating unit, based on the constructed encoder, inputs each element of the input discrete structure to the corresponding encoding unit, sequentially calculates, and Outputting a code having a discrete structure and predicting a solution to the auxiliary problem by calculating the predictor; anda decoder calculating section converts the output code and the solution to the auxiliary problem to the discrete structure Decoding to decode to element Enter the decoder connected to the unit, it is sequentially calculated, and executes contain, and outputting the discrete structures.

  A program according to a fifth invention is a program for causing a computer to function as each unit of the encoder learning device according to the first invention.

  A program according to a sixth invention is a program for causing a computer to function as each unit of the conversion device according to the second invention.

  According to the encoder learning apparatus, method, and program of the present invention, based on correct data of a main problem for which a discrete structure is to be converted, correct data of a predetermined auxiliary problem is generated for the main problem and input. A coder including a predictor for predicting a solution of an auxiliary problem, the coder including a coding unit for coding the element corresponding to each element of the divided discrete structure, and a code output by the coder And the input of the solution of the auxiliary problem, a decoder connected to a decoding unit for decoding into elements having a discrete structure, correct data for the main problem, and the correct data for the auxiliary problem, By learning the parameters of the decoding unit and the predictor, the effect is obtained that the parameters for converting the discrete structure can be learned with high accuracy.

  Further, according to the conversion device, method and program of the present invention, based on the input discrete structure, the encoder corresponding to each element of the discrete structure, the encoding unit for encoding the element is connected, Constructing an encoder including a predictor corresponding to a predetermined auxiliary problem with respect to the main problem, and, based on the constructed encoder, converting each element of the input discrete structure into a corresponding encoding unit , Sequentially calculating, outputting the input code of the discrete structure, predicting the solution of the auxiliary problem by calculating the predictor, and outputting the output code and the solution of the auxiliary problem to a decoder. , Sequentially calculating, and outputting the discrete structure, the effect of being able to convert the discrete structure with high accuracy is obtained.

It is a figure showing various examples of a conversion problem in natural language processing. FIG. 9 is a diagram illustrating a configuration of an encoder and a decoder disclosed in Non-Patent Document 1. It is a figure showing the function group and operator for description. FIG. 11 is a diagram illustrating a configuration of an encoder and a decoder disclosed in Non-Patent Document 2. FIG. 6 is a diagram illustrating a technique for automatically creating a summary sentence of a given sentence. 1 is a block diagram illustrating a configuration of an encoder learning device according to an embodiment of the present invention. FIG. 9 is a diagram illustrating an example of a problem of predicting a set of words included in a summary sentence. FIG. 9 is a diagram illustrating an example of a problem of estimating the number of words included in a summary sentence. FIG. 11 is a diagram illustrating an example of a problem of predicting a set of words included in both an input sentence and a converted summary sentence. FIG. 9 is a diagram illustrating an example of a configuration of an encoder including a predictor of the auxiliary problem 1. FIG. 11 is a diagram illustrating an example of a configuration of an encoder including a predictor of Sub-Problem 2. FIG. 14 is a diagram illustrating an example of a prediction result of the auxiliary problem 2. FIG. 14 is a diagram illustrating an example of a configuration of an encoder including a predictor of Sub-Problem 3. FIG. 2 is a block diagram illustrating a configuration of a conversion device according to the embodiment of the present invention. It is a figure which shows an example of the output by the encoder and decoder of the auxiliary problems 1-3. FIG. 9 is a diagram illustrating an example of outputs of the encoder and the decoder of the auxiliary problem 1. It is a figure which shows an example of the output by the encoder and decoder of the auxiliary problems 1 and 3. 5 is a flowchart illustrating an encoder learning processing routine in the encoder learning device according to the embodiment of the present invention. 5 is a flowchart illustrating a conversion processing routine in the conversion device according to the embodiment of the present invention. FIG. 9 is a diagram illustrating an example of a configuration of an encoder and a decoder of the auxiliary problem 1 when a target having a discrete structure is translated. FIG. 9 is a diagram illustrating an example of a configuration of an encoder and a decoder of the auxiliary problems 1 and 2 when a target having a discrete structure is translated. It is a figure which shows an example of the structure of the encoder and decoder of the auxiliary problems 1-3 when the object of a discrete structure is made into translation.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview according to Embodiment of the Present Invention>

  First, an outline of an embodiment of the present invention will be described.

  A description will be given as an encoder learning device and a conversion device for solving a main problem of converting an input sentence by converting it. As shown in FIG. 5, a technique for automatically creating a summary of a given sentence is an extremely effective means for grasping the outline of a sentence in a short time.

<Configuration of Encoder Learning Apparatus According to Embodiment of the Present Invention>

  Next, the configuration of the encoder learning device according to the embodiment of the present invention will be described. As shown in FIG. 6, an encoder learning device 100 according to an embodiment of the present invention includes a CPU, a RAM, a ROM storing a program and various data for executing an encoder learning processing routine described below, And a computer including The encoder learning device 100 functionally includes an input unit 10, an operation unit 20, and a parameter DB 44 as shown in FIG.

  The input unit 10 receives correct answer data on a main problem that converts an input sentence into a summary sentence.

  The calculation unit 20 includes an auxiliary problem generation unit 22 and a learning unit 24.

  The auxiliary problem generation unit 22 generates correct answer data of a predetermined auxiliary problem with respect to the main problem, based on the correct answer data of the main problem for converting the sentence received by the input unit 10 into a summary sentence.

  Here, the auxiliary problem handled in the present embodiment will be described.

  The sub-problem is conditional on being less computationally expensive and simpler than the main problem, and a problem related to solving the main problem. The condition is that correct answer data for learning the auxiliary problem can be constructed from the correct answer data for the main question.

  The auxiliary problem generation unit 22 generates correct data of the following auxiliary problems 1 to 3 based on the correct data of the main problem.

  The auxiliary problem 1 is a problem for predicting a set of words included in a summary sentence as shown in FIG. The auxiliary problem 1 is a simpler problem because the word order does not need to be considered as compared with the main problem. The correct answer data for learning of the auxiliary problem 1 is a set of words obtained by removing the word order from the correct answer data for learning of the main problem.

  The auxiliary problem 2 is a problem for predicting the number of words included in the summary sentence as shown in FIG. The auxiliary question 2 is a question that answers an integer of 1 or more, and is a simple question because it is not necessary to correctly answer the word itself. The correct answer data for learning of the auxiliary problem 2 is the number of output words of the correct answer data for learning of the main problem.

  The auxiliary problem 3 is a problem of predicting a set of words included in both the input sentence and the converted summary sentence as shown in FIG. The auxiliary problem 3 is a problem in which each word of a sentence is given a positive or negative value, and is a simple problem because it is not necessary to correctly answer a paraphrased word. The correct answer data for learning of the auxiliary problem 3 is a set of words that can be obtained by the intersection set of the sentence, which is the correct answer data for learning of the main problem, and the word of the summary sentence.

  The learning unit 24 is an encoder connected to an encoding unit that encodes a word corresponding to each word of an input sentence by processing of each unit described below, and is a prediction unit that predicts a solution of an auxiliary problem. An encoder including a decoder, a decoder connected to a code output by the encoder, and a decoding unit that decodes a word of a summary sentence as an input of the solution of the auxiliary problem, and correct data for the main problem; On the basis of the correct data of the auxiliary problems 1 to 3, the parameters of the coding unit, the decoding unit, and the predictor

To learn.

The encoding unit outputs z _out according to the above equation (1) or (2).

The encoder outputs a vector c _j according to the above equation (3).

  Here, a predictor that predicts the solution of each auxiliary problem will be described.

The encoder including the predictor of the auxiliary problem 1 is configured as shown in FIG. The predictor of the auxiliary problem 1 outputs ^sws as a solution of the auxiliary problem 1 according to the following equation (4).

... (4)

The encoder including the predictor of the auxiliary problem 2 is configured as shown in FIG. The predictor of the auxiliary problem 1 outputs ^swl as a solution of the auxiliary problem 1 according to the following equation (5).

... (5)

The encoder including the predictor of the auxiliary problem 3 is configured as shown in FIG. The predictor of the auxiliary problem 1 outputs ^sow as a solution of the auxiliary problem 1 according to the following equation (6).

... (6)

Next, the decoding unit of the decoder outputs z _out according to the above equation (1) or (2), similarly to the encoding unit, and the decoder outputs the output of the decoding unit and the encoder. Based on the output c _j , a word is predicted according to the following equation (7).

... (7)

  As in the above equation (7), a word is predicted in consideration of the solution of the auxiliary problem.

For example, in the calculation of the word generation unit of the decoder, mainly the solution o _j problems, multiplied by the solution s ^ws a solution s ^ow calculated ~ s using auxiliary problem of the auxiliary problems for each element of the vector - o _j is determined. Further, in the calculation of the word generation unit of the decoder, the calculation is performed by defining the number of words by the solution to the auxiliary problem ~ ^swl .

  The learning unit 24 includes an encoder construction unit 30, an encoder calculation unit 32, a decoder calculation unit 34, a prediction acquisition unit 36, an objective function calculation unit 38, a parameter update unit 40, an end determination unit 42, It is comprised including.

  The encoder construction unit 30 constructs an encoder based on the input text.

  The encoder calculation unit 32 includes the encoder constructed by the encoder construction unit 30 and the parameter

Initial value or updated parameter

Based on the above, each word of the input sentence is input to the corresponding encoding unit, sequentially calculated, and outputs the sign of the input sentence. Predict the solution of 3.

  The decoder calculator 34 inputs the codes output from the encoder calculator 32 and the solutions of the sub-problems 1 to 3 to the decoder, sequentially calculates them, and solves them by the main problem and the sub-problems 1 to 3. Output a summary sentence.

  The prediction acquisition unit 36 acquires, as a prediction result, a summary sentence solved by the main problem and the auxiliary problems 1 to 3 calculated by the decoder calculation unit 34.

  The objective function calculator 38 calculates the value of the objective function represented using the summary sentence acquired by the prediction acquisition unit 36 and the correct answer data for the main problem.

  Here, an objective function for learning predictors for the auxiliary problems 1 to 3 will be described.

The objective function for the auxiliary problem 1, the following equation (8) of the loss function Ψ can be formulated as a minimization problem of searching parameters s ^ws that minimizes.

... (8)

The function Ψ corresponds to a negative cross entropy between the correct answer c ^ws and the current system prediction result ^sws . ^sws is a parameter

Is a value determined depending on.

The objective function for the auxiliary problem 2, the following (9) can be formulated as a minimization problem of searching parameters s ^ws that minimizes the loss function Ψ of Formula.

... (9)

The function Ψ corresponds to the negative cross entropy between the correct answer c ^wl and the prediction result s ^wl of the current system. ^sws is a parameter

Is a value determined depending on. As shown in FIG. 13, if the prediction falls within the range of [c ^wl -0.5, c ^wl +0.5], the loss becomes the minimum value with zero loss.

The objective function for the auxiliary problem 3 can be formulated as a minimization problem that searches for a parameter ^sow that minimizes the loss function の of the following equation (10).

... (10)

The function Ψ corresponds to the negative cross entropy between the correct answer c ^ow and the prediction result s ^ow of the current system. ^sow is a parameter

Is a value determined depending on.

  As shown in the following equation (11), the objective function integrates all the objective functions of the main problem and the auxiliary problems 1 to 3 and performs learning simultaneously.

... (11)

  The learning procedure consists of (1) current parameters

To obtain prediction results for the correct answer data of the main problem and the auxiliary problems 1 to 3 input in (1) (processing from the encoder calculation unit 32 to the prediction acquisition unit 36). (2) The loss function of each of the main problem and each of the auxiliary problems is calculated using the prediction result (the output summary sentence) and the correct answer data of the main problem and the auxiliary problems 1 to 3. (3) Calculate the gradient according to the value of the loss function. (4) Obtain the value of the gradient for each parameter unit according to the chain rule.

The parameter updating unit 40 performs a parameter based on the value of the objective function calculated by the objective function calculating unit 38.

To update. Here, the parameter is determined according to the value obtained in the learning procedure (4) of the objective function calculation unit 38.

To update.

  The end determination unit 42 performs the calculation by the encoder calculation unit 32, the calculation by the decoder calculation unit 34, the acquisition by the prediction acquisition unit 36, the calculation by the objective function calculation unit 38, and the parameters until the predetermined iteration end condition is satisfied. The updating by the updating unit 40 is repeated. Then, the parameter finally updated by the parameter updating unit 40

Is stored in the parameter DB 44.

<Configuration of Conversion Device According to Embodiment of the Present Invention>

  Next, the configuration of the conversion device according to the embodiment of the present invention will be described. As shown in FIG. 14, a conversion device 200 according to an embodiment of the present invention is a computer including a CPU, a RAM, and a ROM storing a program for executing a conversion processing routine described later and various data. Can be configured. The conversion device 200 functionally includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG.

  The input unit 210 receives a sentence having a discrete structure to be summarized.

  The operation unit 20 includes an encoder construction unit 230, an encoder calculation unit 232, a decoder calculation unit 234, and a parameter DB 244.

  The encoder constructing unit 230 is an encoder connected to an encoding unit that encodes a word corresponding to each word of the sentence based on the sentence received by the input unit 210, and is predetermined for the main problem. Construct an encoder that includes predictors corresponding to the given sub-problems 1-3. Here, an encoder may be constructed similarly to the encoder construction unit 30 of the encoder learning apparatus 100.

  The parameter DB 244 includes parameters learned by the encoder learning device 100.

Is stored.

  The encoder calculation unit 232 inputs each word of the input text to the corresponding encoding unit based on the encoder constructed by the encoder construction unit 230, sequentially calculates the words, and sequentially calculates the input text. And outputs the parameter stored in the parameter DB 244 according to the above equations (3) to (5).

Is used to calculate a predictor to predict the solutions of the auxiliary problems 1 to 3.

  The decoder calculation unit 234 inputs the solution of the code and the auxiliary problem output from the encoder calculation unit 232 to the decoder, sequentially calculates the summary and the summary sentence in which the main problem and the auxiliary problems 1 to 3 are solved. Is output to the output unit 250. The specific processing up to the output of the word is the same as the processing described in the learning unit 24 of the encoder learning device 100.

FIG. 15 shows an example of the calculation of the auxiliary problems 1 to 3 using the predictor and the summary sentence output from the decoder. In the calculation of the word generation unit of the decoder, the number of words is defined by the solution ~ s ^wl of the auxiliary problem 2, and ~ o _j obtained by multiplying the solution o _{j of the} main problem by ~ s for each vector element is obtained. Is obtained by multiplying the solution s ^ws of the auxiliary problem 1 and the solution s ^ow of the auxiliary problem 3. When the calculation is performed by defining the number of words using the solution of the auxiliary problem 2 to ^swl , for example, if the specified number of words is 10, a word generation unit for 10 words is calculated and the processing is terminated. . When the calculation is performed using the solution ^sow of the auxiliary problem 3, for example, when a certain word is included in the word list on the output side, the probability of the prediction result of the set of words for the words in the word list is calculated. Fix it. When the input word is not included in the word list on the output side (unknown word) and the input word having the highest probability of attention a _{i, j} is predicted to be the same word when it is determined to be unknown word (UNK), Replace with words.

  FIG. 16 shows a case where the solution of only the auxiliary problem 1 is considered. FIG. 17 shows a case where the solutions to the auxiliary problems 1 and 3 are considered.

<Operation of Encoder Learning Apparatus According to Embodiment of the Present Invention>

  Next, the operation of the encoder learning device 100 according to the embodiment of the present invention will be described. When the input unit 10 receives correct answer data on the main problem for converting the input sentence into a summary sentence, the encoder learning device 100 executes an encoder learning processing routine shown in FIG.

  First, in step S100, based on the correct data of the main problem for converting the sentence received by the input unit 10 into the summary sentence, the correct data of the auxiliary problems 1 to 3 predetermined for the main problem is generated.

  Next, in step S102, based on the input sentence, an encoder connected to an encoding unit that encodes a word corresponding to each word of the input sentence is constructed.

  In step S104, the encoder constructed in step S102 and the parameter

Initial value or updated parameter

Based on the above, each word of the input sentence is input to the corresponding encoding unit, and is sequentially calculated, the sign of the input sentence is output, and the predictor is calculated to solve the auxiliary problem. Predict.

  In step S106, the code and the solution of the auxiliary problem output in step S104 are input to a decoder, and sequentially calculated according to the above equation (7), and a summary sentence solved by the main problem and the auxiliary problem is output. I do.

  In step S108, a summary sentence solved by the main problem and the auxiliary problem calculated in step S106 is obtained as a prediction result.

  In step S110, the value of the objective function represented by using the summary sentence acquired in step S108 and the correct answer data for the main problem is calculated according to the above equation (11).

  In step S112, a parameter is set based on the value of the objective function calculated in step S110.

To update.

  In step S114, it is determined whether the repetition end condition is satisfied. If the repetition end condition is satisfied, the parameter updated in step S112 is determined.

Is stored in the parameter DB 44, and the process is terminated. If the repetition termination condition is not satisfied, the process returns to step S104 to repeat the process. Further, when a plurality of correct answer data on the main problem is received, the processes of steps S100 to S114 may be repeated for each correct answer data on the main problem.

  As described above, according to the encoder learning device according to the embodiment of the present invention, based on correct data of a main problem that summarizes an input sentence, a predetermined auxiliary problem is determined for the main problem. An encoder that generates a correct answer data and that corresponds to each word of the input sentence and is connected to an encoding unit that encodes the word, the encoder including a predictor that predicts a solution of an auxiliary problem, Based on the code output by the encoder, and a decoder connected to a decoding unit that decodes a word, with the solution of the auxiliary problem as input, based on the correct data for the main problem and the correct data for the auxiliary problem, By learning the parameters of the encoding unit, the decoding unit, and the predictor, it is possible to accurately learn the parameters for converting into a summary sentence.

<Operation of Conversion Device According to Embodiment of the Present Invention>

  Next, the operation of the conversion device 200 according to the embodiment of the present invention will be described. When the input unit 210 receives a sentence having a discrete structure to be summarized, the conversion device 200 executes a conversion processing routine shown in FIG.

  First, in step S200, based on the sentence received by the input unit 210, an encoder connected to an encoding unit that encodes a word, corresponding to each word of the sentence, is predetermined for the main problem. We construct an encoder that includes a predictor corresponding to the auxiliary problem.

  Next, in step S202, based on the encoder constructed in step S202, each word of the input sentence is input to the corresponding encoding unit, and is sequentially calculated according to the above equation (7). Outputs the sign of the input sentence, and outputs the parameters stored in the parameter DB 244.

Is used to calculate a predictor to predict the solutions of the auxiliary problems 1 to 3.

  In step S204, the code and the solution of the auxiliary problem output in step S202 are input to the decoder, sequentially calculated, and the summary sentence in which the main problem and the auxiliary problem are solved is output to the output unit 250 for processing. To end.

  As described above, according to the conversion device according to the embodiment of the present invention, based on an input sentence, an encoding unit that encodes a word corresponding to each word of the input sentence is connected. An encoder is constructed, which includes a predictor corresponding to a predetermined auxiliary problem with respect to the main problem, and, based on the constructed encoder, each word of the input sentence is corresponded. Input to the encoding unit, sequentially calculate and output the code of the input sentence, calculate the predictor to predict the solution of the auxiliary problem, and decode the output code and the solution of the auxiliary problem. It is possible to convert the summary sentence with high accuracy by inputting it into a device, calculating sequentially, and outputting the summary sentence.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the encoder learning apparatus 100 according to the above-described embodiment, correct data for the auxiliary questions 1 to 3 is generated, and the parameters used in the encoder and the decoder are used.

However, the present invention is not limited to this. The parameters used for the auxiliary problems 1 to 3 independently generate correct data and are used in the encoder and the decoder.

May be learned. In addition, correct answer data of the auxiliary questions other than the auxiliary questions 1 to 3 is generated, and the parameter

May be learned.

  Further, in the conversion device 200 according to the above-described embodiment, the case where the solutions of the auxiliary problems 1 to 3 are predicted has been described as an example. However, the present invention is not limited to this. You may make it.

  Further, in the above-described embodiment, the case where a sentence is input and a summary sentence is output has been described as an example. However, the present invention is not limited to this, and the method according to the embodiment of the present invention employs a discrete structure. It can be applied to any field as long as it is a problem to be converted. For example, the present invention can be applied to a main problem of translating a sentence and converting it into another language. In the case of applying to a main problem of this translation, an encoder and a decoder should be configured as shown in FIGS. Can be. FIG. 20 illustrates an encoder and a decoder that consider the solution of the auxiliary problem 1, FIG. 21 illustrates an encoder and a decoder that considers the solution of the auxiliary problem 1 and 2, and FIG. Fig. 2 shows an encoder and a decoder that take into account the solutions of the auxiliary problems 1 to 3;

  Further, the present invention can be similarly applied to an object having a discrete structure such as a graph other than the language processing as described above.

10, 210 Input unit 20, 220 Operation unit 22 Auxiliary problem generation unit 24 Learning unit 30, 230 Encoder construction unit 32, 232 Encoder calculation unit 34, 234 Decoder calculation unit 36 Prediction acquisition unit 38 Objective function calculation unit 40 Parameter Update unit 42 End determination unit 100 Encoder learning device 200 Conversion device 250 Output unit

Claims (8)

An auxiliary problem generation unit that generates correct data of a predetermined auxiliary problem with respect to the main problem, based on the correct data of the main problem that converts the input discrete structure;
An encoder connected to an encoding unit for encoding the element corresponding to each element of the input discrete structure, the encoder including a predictor for predicting a solution of the auxiliary problem, and the encoder A decoder connected to a decoding unit that decodes the elements of the discrete structure, and a correct answer data for the main problem, and a correct answer data for the auxiliary problem. A learning unit that learns parameters of the encoding unit, the decoding unit, and the predictor based on
An encoder learning device including: The learning unit includes:
An encoder construction unit that constructs the encoder based on the input discrete structure,
Based on the constructed encoder and the initial value of the parameter or the updated parameter, each element of the input discrete structure is input to the corresponding encoding unit and sequentially calculated. An encoder calculating unit that outputs the code of the input discrete structure, and calculates the predictor to predict the solution of the auxiliary problem.
A decoder calculation unit that inputs the output code and the solution of the auxiliary problem to the decoder, sequentially calculates, and outputs a discrete structure,
An objective function calculation unit that calculates a value of an objective function represented by using the discrete structure output by the decoder calculation unit and correct data for the main problem,
A parameter updating unit that updates the parameter based on the calculated value of the objective function,
Until a predetermined iteration end condition is satisfied, a calculation by the encoder calculation unit, a calculation by the decoder calculation unit, a calculation by the objective function calculation unit, and an end determination unit that repeats the update by the parameter update unit, The encoder learning device according to claim 1, further comprising: A converter for solving a main problem of converting an input discrete structure,
An encoder connected to an encoding unit that encodes the element, corresponding to each element of the discrete structure, based on the input discrete structure, and a predetermined auxiliary problem for the main problem. An encoder construction unit that constructs an encoder including a predictor corresponding to
Based on the constructed encoder, each element of the input discrete structure is input to a corresponding one of the encoding units, sequentially calculated, and outputs the code of the input discrete structure. An encoder calculation unit that calculates the predictor to predict the solution of the auxiliary problem;
A decoder calculation unit that inputs the output code and the solution of the auxiliary problem to a decoder connected to a decoding unit that decodes the elements of the discrete structure, sequentially calculates, and outputs a discrete structure,
A conversion device including:   The auxiliary problem, a problem of predicting a set of elements included in the transformed discrete structure, a problem of predicting the number of elements included in the transformed discrete structure, the input discrete structure, and the transformed discrete structure 4. The conversion according to claim 3, wherein the conversion is at least one of a problem of predicting a set of elements included in any of the above and an auxiliary problem that can be learned using correct data generated from correct data of other main problems. apparatus. Auxiliary problem generator, based on the correct data of the main problem to transform the input discrete structure, generating correct data of a predetermined auxiliary problem for the main problem,
A learning unit, which corresponds to each element of the input discrete structure, is an encoder connected to an encoding unit that encodes the element, and includes an encoder including a predictor that predicts a solution to the auxiliary problem. A decoder connected to a decoding unit that decodes the elements of the discrete structure, having a code output by the encoder, and a solution of the auxiliary problem as inputs, and correct data for the main problem; Learning parameters of the encoding unit and the predictor based on the correct answer data of the problem;
An encoder learning method including: A conversion method in a conversion device for solving a main problem of converting an input discrete structure,
An encoder construction unit, based on the input discrete structure, corresponding to each element of the discrete structure, an encoder connected to an encoding unit that encodes the element, for the main problem Constructing an encoder including a predictor corresponding to a predetermined auxiliary problem;
An encoder calculation unit, based on the constructed encoder, inputs each element of the input discrete structure to the corresponding encoding unit, and sequentially calculates the input discrete structure. And calculating the predictor to predict the solution of the auxiliary problem; and
A decoder calculation unit inputs the output code and the solution of the auxiliary problem to a decoder connected with a decoding unit that decodes the elements into the discrete structure, sequentially calculates and outputs a discrete structure. Steps and
Conversion method including.     A program for causing a computer to function as each unit of the encoder learning device according to claim 1.     A program for causing a computer to function as each unit of the conversion device according to claim 3.