JP7367609B2 - Response sentence generation device, reinforcement learning device, response sentence generation method, model generation method, program - Google Patents

Description

The present invention relates to a response sentence generation device, a reinforcement learning device, a response sentence generation method, a model generation method, and a program for generating response sentences to a dialogue history.

Non-patent document 1 is known as a technique for selecting a response sentence. Non-Patent Document 1 describes an approach in which entrainment is considered during dialogue and applied to response sentence selection in an example-based dialogue system. Entrainment, also referred to as a tendency to conform or synchrony, refers to a phenomenon in which speakers in a dialogue behave in the same way or in similar manners, such as the tone of their voices.

Masahiro Mizukami, Koichiro Yoshino, Graham Neubig, Satoshi Nakamura, “Evaluation of a response sentence selection model based on entrainment analysis”, Proceedings of the 23rd Annual Conference of the Language Processing Society, pp. 370-373, March 2017. ．．

With the development of dialogue systems, there is a growing demand for dialogue systems to not only return reasonable responses, but also to reproduce more human-like responses and dialogue phenomena unique to humans, and to improve their human-likeness. Many dialogue systems to date have mainly been question-and-answer systems that return appropriate responses to arbitrary input, using data obtained from dialogue between humans as a reference. Non-Patent Document 1 discloses that response sentence candidates are selected, and word frequencies (also called language models) calculated from past dialogue history are used to find the most entraining candidate among the selected response sentence candidates. A technique for creating an appropriate response sentence by selecting the one that occurs is shown. However, Non-Patent Document 1 is also a technique in which response sentence candidates are selected based on the immediately preceding utterance, and response sentence candidates are not generated based on a dialogue history consisting of a plurality of past utterances.

In recent years, a method of generating responses using neural networks (hereinafter referred to as "sentence generation method") has become common in dialogue systems.In the sentence generation method, the entire dialogue is input and the next response is output. Models that can take the context into account are also being proposed. However, research has shown that even if all interactions are input, the model ultimately only focuses on the previous user's utterance. This shows that even if the model is capable of taking context into account, it cannot take context into account sufficiently. Furthermore, no model has been proposed that can take entrainment into account in a sentence generation method that inputs the entire dialogue.

For example, suppose there is the following interaction between the user and the system.
User: Hello.
System: Hello.
User: It's nice weather.
System: Well, I'm glad it's sunny.
User: When the weather is sunny, I feel like going camping or having a picnic.
System: Do you like camping?
User: Yeah, I like it. Do you like camping?
An example of a system response to the above dialogue that takes entrainment into consideration is, "Yeah, I like camping, too." This response sentence is in sync with the user's speaking style. On the other hand, an example of a response sentence in the current system that does not take entrainment into account is, "I like camping." The content is the same, but the responses are simple and unnatural as a dialogue.

An object of the present invention is to generate a response sentence to a dialogue history consisting of a plurality of past utterances based on the degree of entrainment.

The response sentence generation device of the present invention includes a recording section and a response generation section. The recording unit records a response generation model for outputting a response sentence using the dialogue history as input. The response generation model is a reinforcement learning model using an expected reward value based on the degree of entrainment between the dialogue history and response sentences. The response generation unit receives the dialogue history as input, uses the response generation model, and outputs a response sentence to the dialogue history.

The reinforcement learning device of the present invention performs reinforcement learning on a response generation model for outputting a response sentence to an input dialogue history. The reinforcement learning device of the present invention includes a reward calculation model section and a parameter update section. The reward calculation model section receives at least the other person's dialogue history, response sentences generated for the dialogue history, and reference responses for the dialogue history, and calculates reward expectations based on the degree of entrainment between the dialogue history and the response sentences. Calculate the value and output the expected reward value. The parameter updating unit receives the response generation model and the expected reward value as input, updates the parameters of the response generation model using the expected reward value, and outputs the updated parameters.

According to the response sentence generation device of the present invention, a response sentence to a dialogue history consisting of a plurality of past utterances can be generated based on the degree of entrainment. Further, according to the reinforcement learning device of the present invention, the response generation model used in the response sentence generation device of the present invention can be subjected to reinforcement learning.

The figure which shows the example of a structure of a response sentence generation device and a reinforcement learning device. The figure which shows the processing flow of response sentence generation. A diagram showing a processing flow of reinforcement learning. A diagram showing the number of dialogues/number of utterances in the ConvAI2 dataset. The figure which shows the response sentence generation result when each response generation model is used. FIG. 1 is a diagram showing an example of a functional configuration of a computer.

Embodiments of the present invention will be described in detail below. Note that components having the same functions are given the same numbers and redundant explanations will be omitted.

FIG. 1 shows an example of the configuration of a response sentence generation device and a reinforcement learning device. FIG. 2 shows the processing flow of response sentence generation, and FIG. 3 shows the processing flow of reinforcement learning. The response sentence generation device 100 includes a recording section 120 and a response generation section 110. The recording unit 120 records a response generation model for outputting a response sentence by inputting the dialogue history. The response generation model is a reinforcement learning model using an expected reward value based on the degree of entrainment between the dialogue history and response sentences. The response generation unit 110 receives the dialogue history as input (S101), generates a response sentence for the dialogue history using a response generation model (S110), and outputs it (S102). Note that the "dialogue history" consists of a plurality of utterances and does not include other information such as the degree of entrainment.

The reinforcement learning device 200 performs reinforcement learning on a response generation model for outputting a response sentence to the input dialogue history. The reinforcement learning device 200 includes a reward calculation model section 210 and a parameter update section 220. In the reinforcement learning processing flow, a dialogue history is input to the response sentence generation device 100, and a set of the dialogue history and a reference response is input to the reinforcement learning device 200 as training data (S201). The response generation unit 110 generates a response sentence to the dialogue history using the response generation model (S110). The reward calculation model unit 210 receives at least another person's dialogue history, a response sentence generated for the dialogue history, and a reference response for the dialogue history, and calculates a reward based on the degree of entrainment between the dialogue history and the response sentence. The expected value is calculated and the expected reward value is output (S210). Here, "other person" means a person other than the person who is trying to speak (the person who is trying to generate a response sentence). Note that "self" means a person who is trying to speak (a person who is trying to generate a response sentence). For example, in the case of a dialogue between user A and user B, when user A is generating an utterance (response sentence), user A is the user, and user B is the other person. Further, when the user B's utterance (response sentence) is being generated, the user is the user B and the other person is the user A. In this way, "self" and "other" are determined based on whose response sentence is being generated. The parameter updating unit 220 receives the response generation model and the expected reward value, updates the parameters of the response generation model using the expected reward value, and outputs the updated parameters (S221). These processes are repeated (S202). Below, the processing of each component will be explained in detail.

<Response generation unit 110, response generation model>
The response generation unit 110 generates a response sentence R _i = {w _i,1 , w i,2 , ... when the dialogue history H = {H _i -1 , H _{i-2 ,} ..., H _iN } is given. , w _i,t }. Here, i is the turn of the dialogue, N is the history length, t is the word order, H _iN is the i-Nth utterance, and w _i,t is the tth of the i-th utterance (response sentence to be processed). is the word. The response generation unit 110 uses a model consisting of a hierarchical encoder that encodes the input dialogue history into a fixed-length context expression, and a decoder that generates utterances (response sentence generation) using the context expression received from the hierarchical encoder. You can use This technique is described in Reference 1 (Serban, IV, Sordoni, A., Bengio, Y., Courville, AC, and Pineau, J.: Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models., in AAAI , pp. 3776-3784 (2016).). Furthermore, the response generation unit 110 may use an attention mechanism that takes the dialogue history into consideration with respect to the model shown in Reference 1, in order to realize response sentence generation that takes the dialogue history into consideration. In other words, the response generation unit 110 may use a hierarchical encoder-decoder with an attention mechanism. Further, a GRU (Gated Recurrent Unit) may be used for the RNN (Recurrent Neural Network) cell.

The hierarchical encoder encodes each utterance H _i in the dialogue history into a fixed-length utterance expression h _i using the utterance encoder. Here, h _i may be a vector u i, _{t finally obtained by recursively applying the following equation to each word w i, t} of H _i .
u _i,t = GRU(u _i,t-1 , Embedding(w _i,t ))
However, Embedding is a linear transformation function that maps the word w _i,t to a fixed-length dense vector. Then, by recursively applying the following equation to the utterance expression h _i obtained from the utterance encoder in the context encoder, the context expression c _i of the dialogue history can be obtained.
c _i = GRU(c _i-1 , h _i )

In the Decoder, the context representation c _i-1 of the dialogue history H obtained from the hierarchical encoder is used as the initial state h ₀ ', and the intermediate state h _t ' of the Decoder and the word generation probability p _t are calculated as follows: All you have to do is ask.
h _t '= GRU(h _t-1 ', Embedding(w _i,t-1 ))
p _t = softmax(Linear(h _t '))
However, Linear is a linear transformation function that maps h _t ' to a dense vector of vocabulary size dimension. Also, w _i,t is sampled from p _t and used as input for the next step.

応答生成部１１０は、さらに、文脈ベクトルh^-を用いて、ステップｔにおける出力単語の予測を次のように行えばよい。
h_t^= tanh(Linear([h^-,h_t’]))
p_t = softmax(Linear(h_t^)) Since the response generation unit 110 handles a conversation phenomenon called entrainment, it is required to construct a response generation model that takes the conversation history into consideration. Therefore, it is sufficient to introduce an attention mechanism to the above-mentioned Decoder in order to more efficiently handle the information of each utterance in the dialogue history. Specifically, when c _{i-1-N:i-1 is} the sequence of context vectors obtained by the context encoder and h _t ' is the intermediate state of the decoder at step t, each intermediate state is It is sufficient to calculate the alignment weight for that and obtain the context vector h ^- . Note that due to descriptive limitations, it is expressed as "h ^- ", but in this expression, " ^- " means that it is located above "h". Similarly, in the case of "h _t ^", "^" means that it is located above "h".

The response generation unit 110 may further predict the output word in step t using the context vector h ^- as follows.
h _t ^= tanh(Linear([h ^- ,h _t ']))
p _t = softmax(Linear(h _t ^))

<Remuneration calculation model section 210>
The reward calculation model unit 210 calculates the expected reward value used by the parameter update unit 220 (S210). Therefore, we define a reward calculation model for evaluating the degree of entrainment of generated response sentences. For example, as a simple example, the degree of similarity between the previous utterance H _i-1 by another person and the generated response sentence R _i may be defined as the reward r _previous using WDM (Word Mover's Distance). The specific content of WDM can be found in Reference 2 (Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K.: From word embeddings to document distances, in International conference on machine learning, pp. 957) -966 (2015).) etc. However, since the WDM shown in Reference 2 and the like is a similarity index that is not normalized, it is not desirable to use it as is as a reward in reinforcement learning. Therefore, it is sufficient to define a WDM _norm in which WDM is normalized from 0 to 1 as shown below, and calculate the reward r _previous . Note that “e^(-WDM(H _i-1 , R _i ) ² )” means “e” raised to the power “(-WDM(H _i-1 , R _i ) ² )”. .
WDM _norm (H _i-1 , R _i ) = e^(-WDM(H _i-1 , R _i ) ² )
r _previous (H _i-1 , R _i ) = WDM _norm (H _i-1 , R _i )

ただし、R_i ^refは対話履歴Hに対応するレファレンスの応答文（訓練データに含まれている正解の応答文）、H^otherは対話履歴から自身の発話をすべて除外したもの（言い換えると、他者の対話履歴）である。また、U_entrainedは与えた他者の対話履歴H^otherの中でレファレンスの応答文と最も類似している発話を示す。なお、上述したとおり、「自身」と「他者」は、だれの応答文を生成しようとしているときかに基づいて決まる。 Furthermore, entrainment is not necessarily performed only on the other person's previous utterance, but it is preferable to appropriately determine the utterance to be entrained according to the context from the utterance history of the other person. Therefore, the remuneration calculation model unit 210 may also receive information regarding the ideal degree of entrainment and calculate the expected remuneration value based on the value relative to the ideal degree of entrainment. To this end, we define _rLID, which indicates the degree of entrainment for another person's utterances, as follows.

However, R _i ^ref is the response sentence of the reference corresponding to the dialogue history H (the correct response sentence included in the training data), and H other is the response sentence of the reference corresponding to the dialogue history H (correct response sentence included in the training data), and H ^other is the response sentence from which all own utterances have been excluded from the dialogue history (in other words, the response sentence of the reference that corresponds to the dialogue history H) (dialogue history). In addition, U _entrained indicates the utterance most similar to the reference response sentence among the given other's dialogue history H ^other . Note that, as described above, "self" and "other" are determined based on whose response sentence is to be generated.

Ａ％は、例えば、９０％、７０％、５０％などであり、応答文でどの程度のエントレインメントを起こしたいかに応じてエントレインメント度合いが大きいほど大きなパーセンテージとなるように適宜定めればよい。そして、各ステップtからのロールアウトを用いたMonte Carlo Tree Searchによって報酬期待値を求めればよい。なお、Monte Carlo Tree Searchについては、参考文献４（Yu, L., Zhang, W., Wang, J., and Yu, Y.: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient., in AAAI, pp. 2852-2858 (2017).）などに具体的に示されている。 Entrainment degree r In _LID , the reward is calculated by considering the relative value of _{the ideal} entrainment value r so that the generated response sentence is similar to the entrainment target utterance calculated from the reference response sentence. give. This makes it possible to learn a response generation model that indirectly maximizes LID (Local Interpersonal Distance). Note that LID is one of the entrainment evaluation indicators, and is described in Reference 3 (Nasir, M., Chakravarthula, SN, Baucom, BR, Atkins, DC, Georgiou, P., and Narayanan, S.: Modeling Interpersonal Linguistic Coordination in Conversations Using Word Mover's Distance, in Proc. Interspeech 2019, pp. 1423-1427 (2019). The reward calculation model unit 210 calculates the actual degree of entrainment in all response cases in the training data using the following formula as the ideal entrainment value r _ideal , and sets the top A% value as the ideal entrainment value. The model should be trained with the degree of imment r _ideal .

A% is, for example, 90%, 70%, 50%, etc., and may be set as appropriate so that the larger the degree of entrainment, the larger the percentage, depending on the degree of entrainment desired in the response sentence. Then, the expected reward value may be obtained by Monte Carlo Tree Search using the rollout from each step t. Regarding Monte Carlo Tree Search, see Reference 4 (Yu, L., Zhang, W., Wang, J., and Yu, Y.: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient., in AAAI, pp. 2852-2858 (2017).).

<Parameter update unit 220>
The parameter updating unit 220 may use, for example, the REINFORCE algorithm, which is a type of policy gradient reinforcement learning. The REINFORCE algorithm is described in Reference 5 (Williams, RJ: Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, Vol. 8, No. 3-4, pp. 229-256 (1992)). has been done.

When the response generation unit 110 generates a response sentence, when the context expression of the dialogue history H = {H _i-1 , H _i-2 , ..., H _iN } is input, a response sentence R _i = {w _i,1 , w _i,2 , …, w _i,t } is generated. The process of generating such a response sentence can be regarded as a series of actions executed according to a certain policy in the Markov decision process. The parameter updating unit 220 may use the expected reward value output from the reward calculation model unit 210 to update the parameters of the response generation model using the REINFORCE algorithm so that the expected reward value becomes larger.

ステップＳ２２１では、パラメータ更新部２２０は、上記の目的関数J_REINFORCEとその勾配に基づいて応答生成モデルG_θのパラメータθを更新すればよい。 Let G _θ be a response generation model with parameter θ, and p be the probability of generating word w _t . The objective function J _REINFORCE and the gradient can be defined as in the following equation.

In step S221, the parameter updating unit 220 may update the parameter θ of the response generation model G _θ based on the objective function J _REINFORCE and its gradient.

Note that if the parameter θ is updated only based on the objective function J _REINFORCE and its gradient, there is a risk that the response generation model will collapse due to placing too much emphasis on the degree of entrainment. If there is such a risk, the parameter updating unit 220 may update the parameter θ using a loss function as well. Updating using a loss function has existed for a long time, and it updates the parameter θ so that the word prediction result at each step of the decoder and the negative log likelihood J _MLE (θ) of the correct word are small. Good (S222). However, in order to prevent the influence of the log likelihood J _MLE (θ) from becoming dominant, the update may be performed by multiplying by a coefficient λ. The coefficient λ may be set to, for example, 0.1. In this way, the update based on entrainment and the update based on the loss function may be weighted appropriately.

According to the response sentence generation device 100, a response sentence for a dialogue history consisting of a plurality of past utterances can be generated based on the degree of entrainment, without degrading the quality of the response sentence. Further, according to the reinforcement learning device 200, the response generation model used in the response sentence generation device 100 can be reinforced based on the degree of entrainment while not impairing the quality of the response sentences.

<Experiment>
For training and evaluation of the response generation model, we used the PersonaChat dataset provided in Reference 6 (ConvAI2, [retrieved April 27, 2020], Internet <http://convai.io/>.) did. Figure 4 shows the number of interactions/utterances in the ConvAI2 dataset. Since the evaluation data set was not made public, the development data was divided into two parts and used as a development data set and an evaluation data set. FIG. 5 shows the response sentence generation results when using each response generation model.

MLE indicates the case where a response generation model trained by minimizing the negative log likelihood is used. The response generation model indicated as REINFORCE in the training method column is a response generation model that has undergone reinforcement learning based on entrainment. Furthermore, the response generation models used in the evaluation experiment are a sequence-to-sequence model (SEQ2SEQ), a model that introduces an attention mechanism using inner product score calculation to SEQ2SEQ (Attention-SEQ2SEQ), and the above-mentioned hierarchical encoder model. We used four response generation models: -Decoder model (HED) and hierarchical encoder-decoder model with attention mechanism (Attention-HED). The attention mechanism introduced in Attention-SEQ2SEQ can be found in Reference 7 (Luong, M.-T., Pham, H., and Manning, C. D.: Effective approaches to attention-based neural machine translation, arXiv preprint arXiv:1508.04025 (2015).). Note that the row where the model is Human indicates evaluation assuming a human response.

In both response generation models, GRU was used for the RNN, and the dimension of the word embedding layer was set to 300, the dimension of the middle layer was set to 300, and the number of layers was set to 1. In addition, the vocabulary size used was set to 15,000, and unknown words were replaced with the special symbol "UNK". Pre-training of the response generation model was performed using cross-entropy error. Thereafter, the reward calculation model unit 210 is provided with a response generation model (hereinafter referred to as "r _previous ") that has been subjected to reinforcement learning using r _previous , and an r _LID whose top 90% values are determined as the ideal entrainment degree r _ideal . (hereinafter referred to as "r _LID ^90% "), a response generation model that was reinforcement learned using r _LID , where the top 70% value was determined as the ideal entrainment degree r _ideal (hereinafter referred to as “r _LID ^70% ”), and a response generation model that is reinforcement learned using r _LID , which has the top 50% value as the ideal entrainment degree (r _ideal ) (hereinafter referred to as “r _LID ^50% ”) ) was generated. In training the response generation model, the patch size was 64, the learning rate was 1×10 ⁻⁴ , and SGD (Stochastic Gradient Descent) was used as the optimizer.

In the evaluation of response sentence generation, as in the past, evaluation was performed using perplexity (PPL), which is an index for measuring the performance of a language model, and evaluation based on the degree of similarity (relevance) between the generated response and the reference utterance. As evaluation indicators for relevance, evaluation using BLEU and evaluation using the normalized WDM _norm average (×100) were used. The column labeled “WDM ⁻ ” in FIG. 5 shows the evaluation results using the normalized WDM _norm . For PPL, the smaller the value, the higher the evaluation, and for BLEU and WDM ^- , the larger the value, the higher the evaluation. Furthermore, in order to evaluate response sentence generation focusing on entrainment, we used the average (x100) of reward values given to response utterances by each reward calculation model. The columns labeled “r ^- _previous ”, “r ^- _LID ^90% ”, “r ^- _LID ^70% ”, and “r ^- _LID ^50% ” in Figure 5 indicate the evaluation when using each reward calculation model. Showing results. These evaluation results have a maximum value of 100, and the larger the value, the higher the evaluation. Note that the notation "WDM ^- " means a description with "-" added above "WDM". The notation "r ^- " means a description with "-" added above "r". In addition, in the calculation of WDM, the 200-dimensional word distributed expression vector pre-trained using TWITTER (registered trademark) data was normalized so that the norm was 1. Regarding this word distributed representation vector, see Reference 8 (GloVe: Global Vectors for Word Representation, [searched on April 27, 2020], Internet <https://nlp.stanford.edu/projects/glove/>. ) is shown.

The results shown in Figure 5 show that the MLE, which corresponds to the response generation model using the conventional loss function, improves the entrainment-based evaluation results (“r ⁻ _previous ”, “r ⁻ _LID ^90% '', ``r <sup>-</sup> <sub>LID</sub> <sup>70%</sup> '', ``r ^- _LID ^50% '') are significantly inferior. This suggests that existing response generation models lack the ability to perform entrainment, and are also unable to effectively utilize information from the interaction history. On the other hand, the response generation model that applies reinforcement learning using reward expectations based on the degree of entrainment has almost the same PPL as MLE, and the evaluation result based on entrainment (“r ^－previous '', ``r <sup>－</sup> <sub>LID</sub> <sup>90%</sup> '', ``r ^－ _LID ^70% '', <sub>`</sub> `r ^－ _LID ^50% '') are significantly improved. In particular, from the evaluation of the part surrounded by the thick line in FIG. 5, it can be seen that reinforcement learning is performed to achieve the specified ideal entrainment value r _ideal . Regarding the relevance evaluation between the generated response sentence and the reference response sentence, it can be seen that when reinforcement learning is applied, although BLEU tends to decrease, WMD ^- significantly improves. Therefore, in the evaluation of PPL, BLEU, and ^WDM- , it can be seen that reinforcement learning of the response generation model can be performed based on the degree of entrainment while not impairing the quality of response sentences.

[Program, recording medium]
The various processes described above are performed by causing the recording unit 2020 of the computer 2000 shown in FIG. This can be done by letting

A program describing the contents of this process can be recorded on a computer-readable recording medium. The computer-readable recording medium may be of any type, such as a magnetic recording device, an optical disk, a magneto-optical recording medium, or a semiconductor memory.

Further, this program is distributed by, for example, selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, this program may be distributed by storing the program in the storage device of the server computer and transferring the program from the server computer to another computer via a network.

A computer that executes such a program, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing a process, this computer reads a program stored in its own recording medium and executes a process according to the read program. In addition, as another form of execution of this program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and furthermore, the program may be transferred to this computer from the server computer. The process may be executed in accordance with the received program each time. In addition, the above-mentioned processing is executed by a so-called ASP (Application Service Provider) type service, which does not transfer programs from the server computer to this computer, but only realizes processing functions by issuing execution instructions and obtaining results. You can also use it as Note that the program in this embodiment includes information that is used for processing by an electronic computer and that is similar to a program (data that is not a direct command to the computer but has a property that defines the processing of the computer, etc.).

Further, in this embodiment, the present apparatus is configured by executing a predetermined program on a computer, but at least a part of these processing contents may be realized by hardware.

100 Response sentence generation device 110 Response generation section 120 Recording section 200 Reinforcement learning device 210 Reward calculation model section 220 Parameter update section

Claims (8)

a recording unit that records a response generation model for outputting a response sentence using the dialogue history as input;
a response generation unit that receives a dialogue history as input and outputs a response sentence to the dialogue history using the response generation model;
Equipped with
The response generation device is characterized in that the response generation model is a model subjected to reinforcement learning using an expected reward value based on a degree of entrainment between a dialogue history and a response sentence. The response sentence generation device according to claim 1,
A response sentence generation device, wherein the response generation unit uses a hierarchical encoder-decoder with an attention mechanism. A reinforcement learning device for reinforcement learning of a response generation model for outputting a response sentence to an input dialogue history,
At least the other person's dialogue history, a response sentence generated for the dialogue history, and a reference response for the dialogue history are input, and an expected reward value is calculated based on the degree of entrainment between the dialogue history and the response sentence. a remuneration calculation model section that outputs an expected remuneration value;
a parameter updating unit that receives the response generation model and the expected reward value as input, updates parameters of the response generation model using the expected reward value, and outputs the updated parameters;
A reinforcement learning device equipped with The reinforcement learning device according to claim 3,
The reinforcement learning device, wherein the parameter updating unit updates the parameters also using a loss function. The reinforcement learning device according to claim 3 or 4,
The reinforcement learning device is characterized in that the reward calculation model section also receives information regarding an ideal degree of entrainment and calculates the expected reward value based on a value relative to the ideal degree of entrainment. . A response sentence generation method using a response sentence generation device that records a response generation model for outputting a response sentence using dialogue history as input, the method comprising:
a step of inputting interaction history;
a response generation step of generating a response sentence corresponding to the interaction history using the response generation model;
a response sentence output step of outputting the response sentence;
has
The response generation method is characterized in that the response generation model is a model subjected to reinforcement learning using an expected reward value based on the degree of entrainment between the dialogue history and the response sentence. A model generation method that performs reinforcement learning on a response generation model for outputting a response sentence to an input dialogue history, and generates a reinforcement learned response generation model, the method comprising:
At least the other person's dialogue history, a response sentence generated for the dialogue history, and a reference response for the dialogue history are input, and an expected reward value is calculated based on the degree of entrainment between the dialogue history and the response sentence. a remuneration calculation step for outputting an expected remuneration value;
a parameter updating step of inputting the response generation model and the expected reward value, updating parameters of the response generation model using the expected reward value, and outputting the updated parameters;
A model generation method that performs. A program for causing a computer to function as the response sentence generation device according to claim 1 or 2, or the reinforcement learning device according to any one of claims 3 to 5.

Images