JP7200154B2 - Program, device and method for inferring response sentences to received sentences - Google Patents

Description

The present invention relates to technology of a dialog generation system that infers a response sentence to a received sentence.

Conventionally, there is a technology of a dialog generation system using deep learning (see Non-Patent Document 1, for example). According to this technique, the learning model itself is a black box, and is learned as a sequence-to-sequence conversion model. As applications, we use a dialogue corpus that includes various utterance situations such as microblogs, movie subtitles, and troubleshooting desk questions to realize conversations without a specific purpose (non-task-oriented dialogues) like chats. .

FIG. 1 is a functional configuration diagram of an inference device in the prior art.

According to FIG. 1, it consists of a training phase and an operational phase, each of which is structured as an encoder-decoder model. The encoder and decoder are each based on a long short-term memory (LSTM) that outputs the probability of occurrence of the next word. LSTM is an extension of RNN (Recurrent Neural Network) and is a learning model composed of long term memory and short term memory for sequential data.

<Training stage>
The learning database associates and stores learning received sentences and learning response sentences each including a set of word strings.
The encoder learns to generate a context vector from the learning received sentence, and the decoder generates a learning response sentence from the context vector.
According to FIG. 1, for example, learning received sentences and learning response sentences are associated and learned as follows.
(1) Sentence for learning "Recently, I started learning English conversation"
Learning answer sentence "Can't you speak English?"
(2) Learning received sentence “My hobby is mountain climbing”
Response sentence for learning "Which mountain did you climb?"
<Operation stage>
The encoder generates a context vector from the target received sentence, and the decoder generates a response sentence from the context vector.

Oriol Vinyals, Quoc V. Le, "A Neural Conversational Model", Proceedings of the 31st International Conference on Machine Learning, vol.37, 2017. Ryuichiro Higashinaka, Kotaro Funakoshi, Masahiro Araki, Hiroshi Tsukahara, Yuka Kobayashi, Masahiro Mizukami, “Construction of chat dialogue corpus using text chat and analysis of dialogue failure”, Natural Language Processing, Vol. 23, No. 1, 2016. Star Bookshelf, "Natural Language Processing (NLP)", [online], [searched on December 29, 2019], Internet <URL: http://yagami12.hatenablog.com/entry/2017/12/30 /175113#ID_10-5-1>

According to the technique described in Non-Patent Document 1, only the series conversion model is used, so the context of the dialogue sentence is not learned. As a result, the dialogue sometimes broke down.

Dialogue breakdowns are classified into, for example, the following four cases (see, for example, Non-Patent Document 2).
(Case 1) Broken speech There are cases where the speech itself is broken. For example, there are cases where the syntax is broken and the language is not established as Japanese in the first place.
(Case 2) Corruption of response Although the Japanese is correct, there are cases where the response to the other party's statement is broken. For example, there is a case where a response sentence "When was the last time you traveled?"
(Case 3) Collapse of context Although it is established as a single exchange, there are cases where it does not mesh with what has already been said. For example, although the response text "I like sweets" was answered 10 seconds ago, the response text "I hate sweets" may be immediately returned.
(Case 4) Collapse of the environment Socially (common sense) inappropriate remarks may be made. For example, the artificial intelligence bot “Tay” (registered trademark) published by Microsoft in the United States may suddenly make racist remarks.
Generally, it is recognized that about 50% of bankruptcies in Case 2, about 30% in Case 3, over 10% in Case 1, and about a small number of Case 4 bankruptcies.

Here, the failure of the response in Case 2 is considered to be caused by inappropriate words emphasized by the attention mechanism of the encoder.
Further, the failure of speech in Case 1 may be caused by the lack of accuracy of the decoder.

In recent years, in natural language processing using neural networks, the encoder is equipped with an "attention mechanism" so as not to give too much priority to natural sentences.
The attention mechanism is a new context vector calculated using the ``internal state at the time of word translation of the i-th target to be generated by the decoder'' and the ``hidden layer of each word in the encoder''. It is used when inferring Therefore, in the model including the attention mechanism, when outputting the i-th word, (1) the previous translated word result, (2) the internal state of the decoder, and (3) the attention mechanism calculated given a context vector and use it to infer the ith word.
In this way, since the attention mechanism designates words and phrases to be emphasized, appropriate natural language processing becomes possible.

However, in the case of decoders using LSTM, the attention mechanism may overfit training received sentences and training response sentences. There is a problem of placing importance on

According to FIG. 1, in the operation stage, for example, a target received speech sentence is input to the encoder and a response sentence is output from the decoder as follows.
Target received sentence "Recently, I started mountain climbing"
Response: "Can't you climb mountains?"
Here, although the response sentence is Japanese, there is no problem, but in general, there are limited cases where the response sentence is "cannot climb a mountain", and the context of the response sentence to the received sentence is unnatural and uncomfortable.
This is probably because, for example, in the training phase, the encoder emphasized, for example, "I started ~", and thus the priority of the response sentence "Can't you do ~?"
Thus, according to the example of FIG. 1, the inappropriateness of the word emphasized by the attention mechanism of the encoder (corruption of response in case 2) and the lack of accuracy of the decoder (corruption of speech in case 1) Conceivable.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a program, an apparatus, and a method for inferring a response sentence to a received sentence so as not to fall into a broken response or a broken speech.

According to the present invention, in a program that causes a computer installed in a device for inferring a response sentence to a received sentence to function,
As a training step,
The encoder-decoder model is a parallel encoder that generates a context vector from a first corpus text in a first language, and a second corpus text in a second language that is a parallel translation of the first corpus text from the context vector. Learn the bilingual decoder to output,
As an encoder-decoder model, when a context vector generated by a bilingual encoder from a received training sentence is input, a context vector generated by a bilingual encoder is output from a learning response sentence that is a dialogue of the received training sentence. train the neural network as
As an operational stage,
generating a first context vector from the target received sentence by a bilingual encoder;
generating a second context vector from the first context vector by a neural network;
The computer is operable to infer a response sentence from the second context vector by the parallel decoder.

According to another embodiment of the program of the present invention,
The bilingual encoder has an attention mechanism,
It is also preferred to have the computer function such that the first context vector generated from the bilingual encoder and the second context vector generated from the neural network potentially contain an attention mechanism.

According to another embodiment of the program of the present invention,
The bilingual encoder and the bilingual decoder each have a plurality according to the number of different languages,
A bilingual encoder receives multiple corpus texts in different languages, respectively, and generates a context vector; and/or
The bilingual decoder also preferably causes the computer to be trained to input one context vector and output multiple corpus texts in different languages, respectively.

According to another embodiment of the program of the present invention,
The bilingual encoder and bilingual decoder are based on sequence-to-sequence neural networks,
A bilingual encoder consists of an embedding layer and a recurrence layer,
It is also preferred to have the computer function such that the parallel decoder consists of an embedding layer, a recursion layer and an output layer.

According to the present invention, an inference device for inferring a response sentence to a received sentence:
As a training step,
The encoder-decoder model is a parallel encoder that generates a context vector from a first corpus text in a first language, and a second corpus text in a second language that is a parallel translation of the first corpus text from the context vector. Learn the bilingual decoder to output,
As an encoder-decoder model, when a context vector generated by a bilingual encoder from a received training sentence is input, a context vector generated by a bilingual encoder is output from a learning response sentence that is a dialogue of the received training sentence. train the neural network as
As an operational stage,
generating a first context vector from the target received sentence by a bilingual encoder;
generating a second context vector from the first context vector by a neural network;
A response sentence is inferred by a parallel decoder from the second context vector.

According to the present invention, in an inference method for a device that infers a response sentence to a received sentence,
The device
As a training step,
The encoder-decoder model is a parallel encoder that generates a context vector from a first corpus text in a first language, and a second corpus text in a second language that is a parallel translation of the first corpus text from the context vector. Learn the bilingual decoder to output,
As an encoder-decoder model, when a context vector generated by a bilingual encoder from a received training sentence is input, a context vector generated by a bilingual encoder is output from a learning response sentence that is a dialogue of the received training sentence. train the neural network as
As an operational stage,
generating a first context vector from the target received sentence by a bilingual encoder;
generating a second context vector from the first context vector by a neural network;
It is characterized by performing so as to infer a response sentence by a bilingual decoder from the second context vector.

According to the program, apparatus, and method of the present invention, it is possible to infer a response sentence to a received sentence without falling into a failure of response or failure of utterance.

1 is a functional configuration diagram of an inference device in the prior art; FIG. FIG. 4 is a functional configuration diagram of the training stage of the inference device in the present invention; Fig. 2 is an illustration of a first embodiment representing training of a parallel decoder and a parallel encoder; FIG. 10 is an illustration of a second embodiment representing training of a parallel decoder and a parallel encoder; 3 is a functional configuration diagram of the inference device in the operation stage of the present invention; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 2 is a functional configuration diagram of the training stage of the inference device in the present invention.

According to FIG. 2, the inference device 1 has a parallel corpus database 101, a learning database 102, a parallel encoder 111, a parallel decoder 112, and a neural network 12 as training stages. These functional block diagrams are realized by executing a program that causes a computer installed in the apparatus to function. In addition, the processing flow of these functional components can also be understood as a training stage in the reasoning method of the device.
Here, training is divided into a first training phase and a second training phase. Each training stage is constructed via a context vector as an encoder-decoder model.

<<First training stage>>
In the first training stage, the parallel encoder 111 and the parallel decoder 112 are trained based on the parallel corpus database 101 .

[Parallel corpus database 101]
The bilingual corpus database 101 accumulates corpus texts that are bilingual between different languages. This is a correspondence between the first language corpus text to be input to the bilingual encoder 111 and the second language corpus text to be output by the bilingual decoder 112 .
The corpus text may be, for example, a multilingual parallel corpus in neural machine translation. That is, different languages expressing the same meaning are prepared as parallel translations.
Japanese: "I love you."
English: "I love you."
German: "Ich liebe dich."
Chinese: "I love you"

[Parallel Encoder 111/Parallel Decoder 112]
The bilingual encoder 111 and the bilingual decoder 112 input the corpus text of the first language and the corpus text of the second language as parallel translations in association with each other, and learn as an encoder-decoder model.
The bilingual encoder 111 learns to generate a context vector from the first language corpus text input from the bilingual corpus database 101 . Here, the bilingual encoder has an attention mechanism.
Parallel decoder 112 learns from the context vector to output corpus text in the second language.

The bilingual encoder 111 and the bilingual decoder 112 use, as encoder-decoder models, neural network-based "sequence-to-sequence / seq2seq models" that model the probability of converting from one sequence to the other. )” (see, for example, Non-Patent Document 3). That is, it models the conditional probability P(Y|X) that a certain sequence Y is output when the sequence X is input.
The sequence conversion model is composed of a parallel translation encoder 111 that inputs a sequence X and generates a fixed-length "context vector" and a parallel translation decoder 112 that outputs a sequence Y from the fixed-length context vector.

Here, according to the present invention, the most notable point is to use a bilingual corpus between different languages, even though the received sentence and the response sentence are in the same language.
In general, if the received sentence and the response sentence are in the same language, the same language is used in the training stage and the operation stage. Naturally, if the received sentence and the response sentence are in Japanese, there is no need for a bilingual corpus for other languages. If the encoder-decoder model were to use a corpus of Japanese-Japanese dialogues, it would be nothing more than a simple identity transformation. Therefore, if the received sentence and the response sentence are in the same language, it is not assumed that the bilingual corpus text is used.
In contrast, according to the present invention, learning is performed using a "parallel translation corpus" in which received sentences and response sentences are in different languages. This prevents over-training of the attention mechanism inherent in the context vectors generated by the bilingual encoder. In particular, the greater the number of parallel translation language types, the more it becomes possible to generate a context vector that is not affected by individual language models. It is expected that the finally generated response sentence will be influenced as little as possible by the existing language model.

FIG. 3 is an illustration of a first embodiment representing training of a parallel decoder and a parallel encoder.

According to FIG. 3, the bilingual encoder 111 receives a morpheme sequence based on the corpus text of the first language.
Japanese: "You/to/love/to/be/<EOS>"
A morpheme sequence based on the corpus text of the second language is input to the parallel translation decoder 112 .
English: "<BOS>/I/love/you/<EOS>"
The corpus text of the first language and the corpus text of the second language are synonymous sentences although they are in different languages.

Further, according to FIG. 3, the following Japanese sentence is input to the bilingual encoder 111, for example.
``The shogunate expelled the Portuguese in 1639 and ordered the feudal lords to guard the coast.''
On the other hand, the bilingual decoder 112 learns the bilingual encoder 111 and the bilingual decoder 112 so as to output the following English sentence from the context vector.
"The shogunate banished Portuguese in 1639, ordered Daimyo to guard
the coast."
Similarly, according to FIG. 3, the following Japanese sentence is input to the bilingual encoder 111, for example.
"In 1639 the Portuguese were expelled and the shogunate was ordered by the feudal lords to guard the coast."
On the other hand, the bilingual decoder 112 learns the bilingual encoder 111 and the bilingual decoder 112 so as to output the following English sentence from the context vector.
"In 1639, the Portuguese were expelled, and the shogunate was ordered
to protect the coast from Daimyo."

According to FIG. 3, the bilingual encoder 111 consists of an embedding layer and a recursion layer and learns to output context vectors from the corpus text of the first language.
The embedding layer transforms each word x of the input text X into a distributed representation of an embedding vector.
The recurrent layer then functions as a recurrent neural network, inputting the embedding vector and outputting the context vector.

On the other hand, the decoder 12 consists of an embedding layer, a recursion layer and an output layer, receives a context vector and learns to output a corpus text of the second language.
The embedding layer transforms each word y of the output text Y into a distributed representation of embedding vectors.
Next, the recurrent layer receives the embedding vector and the context vector and functions as a recurrent neural network.
The output layer receives a hidden layer state vector corresponding to the word y in the output sequence Y output from the recursive layer, and outputs text.

FIG. 4 is an illustration of a second embodiment representing the training of the parallel decoder and encoder.

According to FIG. 4, context vectors are generated from two parallel encoders 111 and two parallel decoders 112 while associating corpus texts of four different languages. That is, it is composed of two translation encoders 111 and two translation decoders 112 .
According to FIG. 4, the context vectors output from the parallel encoder 111 for Japanese and the parallel encoder 111 for Chinese are converted into the context vectors output from the parallel decoder 112 for English and the parallel decoder 112 for German. and learning to type. As a result, context vectors common to parallel corpora that are synonymous sentences are generated for four different languages.

Of course, in further embodiments, the parallel encoder 111 and the parallel decoder 112 may be configured differently from 1:2 and 2:1.
For example, context vectors output from the Japanese translation encoder 111 may be input to the English translation decoder 112 and the German translation decoder 112 for learning.
Further, for example, even if context vectors output from the parallel translation encoder 111 corresponding to Japanese and the parallel translation encoder 111 corresponding to Chinese are input to the parallel translation decoder 112 corresponding to English and learning is performed, good.

<<Second training stage>>
Returning to FIG. 2 , in the second training stage, the training database 102 , two parallel encoders 111 and the neural network 12 are trained.

[Learning database 102]
The learning database 102 stores received speech sentences for learning and response sentences for learning in association with each other. This is similar to the learning database in FIG. 1 as the prior art.

[Two Parallel Encoders 111]
The two bilingual encoders 111 use the bilingual encoders 111 learned in the first training stage as they are. On the other hand, the bilingual encoder 111 inputs a learning received sentence to be a dialogue, generates a context vector from the learning received sentence, and inputs the context vector to the neural network 12 . The other bilingual encoder 111 inputs a learning response sentence for dialogue, generates a context vector from the learning response sentence, and inputs the context vector to the output side of the neural network 12 .

For example, similar to FIG. 1 described above, learned received sentences for learning and response sentences for learning are associated and learned as follows.
(1) Sentence for learning "Recently, I started learning English conversation"
Learning answer sentence "Can't you speak English?"
(2) Learning received sentence “My hobby is mountain climbing”
Response sentence for learning "Which mountain did you climb?"

[Neural network 12]
The neural network 12, as an encoder-decoder model, receives a context vector generated by a bilingual encoder from a received learning sentence, and generates a response sentence for learning, which is a dialogue of the received sentence for learning, by the bilingual encoder. It learns to output a context vector.
Neural network 12 is preferably a convolutional neural network, for example.

The point to be noted is that the parallel translation encoder 111 and the parallel translation decoder 112 are not trained.
Also, each context vector generated by the two parallel encoders 111 will potentially contain an attention mechanism.
According to the present invention, by using the bilingual corpus database 101, the attention mechanism of the bilingual encoder 111 is unaffected by the type of language and attaches importance to semantically important words. can be suppressed. Here, by training the bilingual decoder 112 using bilingual corpora of multiple different languages, the performance can be improved and the breakdown of response sentences can be further suppressed.

<<Operation stage>>
FIG. 5 is a functional configuration diagram of the inference device in the operational stage of the present invention.

The inference device 1 infers a response sentence to a target received speech sentence.
According to FIG. 5 , in the operational stage, it is inferred by an encoder-decoder model consisting of a parallel encoder 111 , a neural network 12 and a parallel decoder 112 . Parallel encoder 111 and parallel decoder 112 were trained in a first training phase, and neural network 12 was trained in a second training phase.

The bilingual encoder 111 generates a first context vector from the target received sentence.
Neural network 12 then generates a second context vector from the first context vector.
The bilingual decoder 112 then generates a response sentence from the second context vector.
Here, the first context vector and the second context vector potentially contain the attention mechanism.

According to FIG. 5, during the operation stage, for example, a target received sentence is input to the bilingual encoder 111 and a response sentence is output from the bilingual decoder 112 as follows.
Target received sentence "Recently, I started mountain climbing"
Response: "Which mountain did you climb?"
Here, the response sentence does not have any problem as Japanese, and the context does not cause unnaturalness or discomfort. This is based on the fact that the attention mechanisms inherent in the context vector of the encoder-decoder model are not overfitting. In this point, it is different from the above-described FIG. 1 in the prior art.

According to the present invention, since the bilingual encoder 111 is trained using the bilingual corpus database 101, "I started learning (English conversation)" and "I started (mountain climbing)" are clearly distinguished. learning. Therefore, the attention mechanism of the bilingual encoder 111 places importance on things other than "begin" (eg, "hill climbing"), and eventually infers the correct response sentence.
In particular, "dialogue generation," which is a weak point of deep learning, becomes possible, and the application range of the neural network can be expanded.

As described in detail above, according to the program, apparatus, and method of the present invention, it is possible to infer a response sentence to a received sentence without falling into a failure of response or speech.

For the various embodiments of the present invention described above, various changes, modifications and omissions within the spirit and scope of the present invention can be easily made by those skilled in the art. The foregoing description is exemplary only and is not intended to be limiting. The invention is to be limited only as limited by the claims and the equivalents thereof.

Reference Signs List 1 inference device 101 bilingual corpus database 102 learning database 111 bilingual encoder 112 bilingual decoder 12 neural network

Claims (6)

In a program that operates a computer installed in a device that infers a response sentence to a received sentence,
As a training step,
The encoder-decoder model is a parallel encoder that generates a context vector from a first corpus text in a first language, and a second corpus text in a second language that is a parallel translation of the first corpus text from the context vector. Learn the bilingual decoder to output,
As an encoder-decoder model, when a context vector generated by a bilingual encoder from a received training sentence is input, a context vector generated by a bilingual encoder is output from a learning response sentence that is a dialogue of the received training sentence. train the neural network as
As an operational stage,
generating a first context vector from the target received sentence by a bilingual encoder;
generating a second context vector from the first context vector by a neural network;
A program characterized by causing a computer to infer a response sentence from a second context vector by a parallel decoder. The bilingual encoder has an attention mechanism,
2. The method of claim 1, wherein the first context vector generated from the bilingual encoder and the second context vector generated from the neural network potentially cause a computer to include an attention mechanism. program. The bilingual encoder and the bilingual decoder each have a plurality according to the number of different languages,
A bilingual encoder receives multiple corpus texts in different languages, respectively, and generates a context vector; and/or
3. The program according to claim 1 or 2, wherein the bilingual decoder inputs one context vector and causes the computer to function so as to be trained to output multiple corpus texts in different languages respectively. The bilingual encoder and bilingual decoder are based on sequence-to-sequence neural networks,
A bilingual encoder consists of an embedding layer and a recurrence layer,
4. The program according to any one of claims 1 to 3, wherein the bilingual decoder causes the computer to function so as to consist of an embedding layer, a recurrence layer and an output layer. In an inference device that infers a response sentence to a received sentence,
As a training step,
The encoder-decoder model is a parallel encoder that generates a context vector from a first corpus text in a first language, and a second corpus text in a second language that is a parallel translation of the first corpus text from the context vector. Learn the bilingual decoder to output,
As an encoder-decoder model, when a context vector generated by a bilingual encoder from a received training sentence is input, a context vector generated by a bilingual encoder is output from a learning response sentence that is a dialogue of the received training sentence. train the neural network as
As an operational stage,
generating a first context vector from the target received sentence by a bilingual encoder;
generating a second context vector from the first context vector by a neural network;
An inference device that infers a response sentence from a second context vector by a bilingual decoder. In an inference method for a device that infers a response sentence to a received sentence,
The device
As a training step,
The encoder-decoder model is a parallel encoder that generates a context vector from a first corpus text in a first language, and a second corpus text in a second language that is a parallel translation of the first corpus text from the context vector. Learn the bilingual decoder to output,
As an encoder-decoder model, when a context vector generated by a bilingual encoder from a received training sentence is input, a context vector generated by a bilingual encoder is output from a learning response sentence that is a dialogue of the received training sentence. train the neural network as
As an operational stage,
generating a first context vector from the target received sentence by a bilingual encoder;
generating a second context vector from the first context vector by a neural network;
A method of inference, comprising inferring a response sentence by a parallel decoder from the second context vector.

Images