JP2018155980A - Dialogue device and dialogue method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly decide whether to perform task-oriented dialog or non-task-oriented dialogue in a dialogue device.SOLUTION: The device includes utterance recognition means for recognizing the content of utterance made by a user, response generating means for generating a response to the utterance according to a response policy which is a rule for generating a response to the utterance, evaluation obtaining means for determining a user evaluation which is a value representing the preference for the response, and updating means that performs reinforcement learning with the user evaluation as a reward and updates the response policy. According to the response policy, the response generating means determines a response mode indicating whether to generate a task-oriented response or a non-task-oriented response, and generates a response according to the response mode.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an apparatus for interacting with a user.

  In recent years, many devices that interact with a user in a natural language have been proposed. In particular, in addition to task-oriented dialogues such as “providing information for questions”, the progress of technology for non-task-oriented dialogues such as chat is remarkable. For example, Patent Literature 1 discloses a method for generating a dialogue strategy with a user by learning.

  With the advance of such technology, devices capable of both task-oriented dialogue and non-task-oriented dialogue have appeared. A task-oriented dialogue is a dialogue requesting a specific task, such as “Tell me the weather for tomorrow”, and a non-task-oriented dialogue is “I feel fine in May.” An interaction that does not require a specific task. By using both of these, you can have a more natural conversation.

JP 2006-72477 A

When both task-oriented dialogue and non-task-oriented dialogue are possible, there is a scene where it is necessary to estimate in which mode the speaker (user) wants the dialogue.
For example, if the user utters "It's a pleasant weather", it can be seen that a non-task-oriented conversation has started, but if the utterance is "Is it fine for tomorrow?" Then, it cannot be clearly determined whether the user wants to know the weather forecast for tomorrow or wants to talk about tomorrow's action schedule.

  The present invention has been made in consideration of the above-described problems, and an object of the present invention is to provide an interactive apparatus that appropriately determines whether to perform a task-oriented conversation or a non-task-oriented conversation.

The interactive apparatus according to the present invention is:
An utterance recognition means for recognizing the content of an utterance made by a user, a response generation means for generating a response to the utterance according to a response policy that is a rule for generating a response to the utterance, and a value representing the preference for the response Evaluation acquisition means for determining a user evaluation, and update means for performing reinforcement learning using the user evaluation as a reward and updating the response policy, wherein the response generation means is task-oriented according to the response policy A response mode indicating whether to generate a type response or a non-task-oriented response is determined, and the response is generated according to the response mode.

The response policy is a policy (rule) that defines what kind of response is returned when the dialog is in what state. In the present invention, it is determined whether to generate a task-oriented response or a non-task-oriented response according to a predetermined response policy.
The response policy is updated by performing reinforcement learning using a reward obtained based on the result of the dialog with the user. For example, it may be started from a state having a default response policy, and learning may be updated through dialogue. Moreover, in this invention, the value (user evaluation) showing how much the response provided to the user was preferable as a reward used for reinforcement learning is determined afterwards.

User ratings (ie, how favorable the response was) may be determined based on the user's condition. For example, if the user's response to the response is favorable, the user evaluation may be increased. Conversely, if the user's response is not favorable, the user evaluation may be decreased. Whether or not the user's reaction is favorable can be determined using, for example, the result of sensing the user's speech or facial expression.
In addition, the user evaluation may be determined based on the content of the dialogue after the response is made. For example, it is possible to use the time during which the dialogue has continued.
According to such a configuration, reinforcement learning is performed through dialogue, and a response policy is more preferable. That is, as learning progresses, it becomes possible to appropriately determine whether to generate a task-oriented response or a non-task-oriented response.

  Further, the dialogue apparatus according to the present invention is a state estimation means for estimating a current dialogue state based on the content of the utterance from a plurality of dialogue states associated with the degree of task-oriented dialogue. The response generation means may perform the reinforcement learning using the plurality of dialogue states.

  The conversation state is a state determined by the user's utterance content. The dialogue state is discretized according to the degree of task orientation. Reinforcement learning using such a dialog state can appropriately determine whether to adopt a task-oriented response mode or a non-task-oriented response mode according to the current dialog state. It becomes like this.

  The reinforcement learning may be Q learning, and the response policy may be a Q value associated with a combination of the dialog state and the response mode.

  The dialogue apparatus according to the present invention can preferably use Q learning which is a kind of reinforcement learning.

  Moreover, the dialogue apparatus according to the present invention further includes a target estimation unit that identifies a target referred to by the user in the dialog, and performs the dialog state estimation and reinforcement learning for each identified target. May be a feature.

  When the subject that the user mentions in the conversation changes (ie, the topic changes), whether the user wants a task-oriented response or a non-task-oriented response may change significantly. is there. Therefore, it is possible to select a more appropriate response mode by performing conversation state estimation and reinforcement learning for each topic.

  The dialogue apparatus according to the present invention may further include speaker estimation means for identifying the user, and the dialogue state estimation and reinforcement learning are performed for each identified user.

  The criteria for determining whether a task-oriented response is preferable or a non-task-oriented response is different for each user. Therefore, a more appropriate response mode can be selected by separately performing dialog state determination and learning for each user.

In addition, the evaluation acquisition unit may determine the user evaluation based on the user's utterance after performing a response according to the response policy.
In addition, the evaluation acquisition unit may determine the user evaluation based on a length of a series of dialogues after a response is made according to the response policy.
In addition, the evaluation acquisition unit may determine the user evaluation based on a time from when a response is made according to the response policy until the user speaks further.

  For example, when a user desires a task-oriented dialog but provides a non-task-oriented response, such as when an inappropriate response is made, it can be assumed that the dialog is interrupted. Therefore, the user evaluation can be determined based on the user's utterance following the response, the duration of the conversation, the time until the user speaks, and the like.

  In addition, this invention can be specified as an interactive apparatus containing at least one part of the said means. It can also be specified as a dialogue method performed by the dialogue device. The above processes and means can be freely combined and implemented as long as no technical contradiction occurs.

  According to the present invention, it is possible to appropriately determine whether a task-oriented dialogue or a non-task-oriented dialogue is performed in the dialogue apparatus.

1 is a system configuration diagram of an interactive apparatus according to a first embodiment. It is a figure explaining the dialog state in this invention. It is an example of the reward table in 1st embodiment. It is a process flowchart figure which the dialogue apparatus which concerns on 1st embodiment performs. It is an example of the dialogue which a user and an apparatus perform. It is a system block diagram of the dialogue apparatus which concerns on 2nd embodiment. It is a system block diagram of the dialogue apparatus which concerns on 3rd embodiment.

(First embodiment)
<System overview>
Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
The dialogue apparatus according to the first embodiment is a system that performs dialogue with the user by acquiring voice uttered by the user, performing voice recognition, and generating a response sentence based on the recognition result.

  The dialogue apparatus according to the present embodiment has a configuration capable of both task-oriented dialogue and non-task-oriented dialogue. A task-oriented dialogue is a dialogue aimed at achieving a specific task (task), and a non-task-oriented dialogue is a dialogue not aimed at achieving a specific task. For example, when an utterance such as “tell me tomorrow's weather” is made, the task of “providing weather information” is a task to be achieved. Further, when an utterance such as “lower the temperature of the air conditioner” is made, the task of lowering the set temperature of the air conditioner is a task to be achieved. On the other hand, in a non-task-oriented dialogue, there is no task to be achieved, so the dialogue is mainly chat.

  As described above, when the user's utterance is a request for a specific task such as “Please do”, it is understood that the task-oriented dialogue has started. However, for example, when an utterance such as "Is it sunny tomorrow?" The interactive apparatus according to the first embodiment has a function of using reinforcement learning to determine whether to perform a task-oriented response or a non-task-oriented response and to respond in an appropriate mode.

<System configuration>
Next, a system configuration of an apparatus for realizing the above-described function will be described.
FIG. 1 is a system configuration diagram of the interactive apparatus according to the first embodiment. The dialogue apparatus 100 according to the present embodiment includes an input / output unit 101, an utterance acquisition unit 102, a response generation unit 103, a state estimation unit 104, a learning unit 105, and an evaluation acquisition unit 106.

  The input / output unit 101 is means for inputting / outputting sound by using a built-in microphone and speaker. The input / output unit 101 has a function of converting input sound into sound data and outputting the input sound data by sound.

The utterance acquisition unit 102 is means for acquiring and recognizing the voice uttered by the user via the input / output unit 101. The utterance acquisition unit 102 performs voice recognition on the acquired voice data using a known technique. For example, the utterance acquisition unit 102 stores an acoustic model and a recognition dictionary, extracts features by comparing the acquired speech data with the acoustic model, and matches the extracted features with the recognition dictionary to generate speech. Recognize. The recognition result is transmitted to the response generation unit 103 and the state estimation unit 104.
Note that the utterance acquisition unit 102 may perform speech recognition using a service outside the apparatus. For example, as shown in FIG. 1, it may be configured to communicate with an external voice recognition server via a network and acquire a recognition result.

The response generation unit 103 is a unit that generates a response sentence to be provided to the user based on the text acquired from the utterance acquisition unit 102. The response sentence to be provided may be based on, for example, a dialogue scenario (dialog dictionary) stored in advance, or may be based on information obtained by searching a database or the web.
The response generation unit 103 generates a task-oriented response or generates a non-task-oriented response based on the conversation state estimated by the state estimation unit 104 described later and the information stored in the learning unit 105. A response mode indicating whether to perform the determination is determined, and different types of responses are generated according to the response mode. A specific method will be described later.

The state estimation unit 104 is a means for estimating a dialog state with the user. The dialogue state in the present specification is a plurality of states associated with the degree of task-oriented dialogue, and is determined based on the content of the utterance performed by the user.
Here, the dialogue state will be described with reference to FIG. In the present specification, the state at a certain point in the dialogue is specified in association with the degree of the task-oriented dialogue (hereinafter referred to as the task degree).
Here, when the set of states that can be taken by the dialogue is S, and the task level is discretized into 10 stages, S is as follows. S ₀ means a state where there is no utterance.
S = {S ₀ , S ₁ , S ₂ ,..., S ₁₀ }

In the case of the example of FIG. 2, the state S ₁ is a state in which the dialogue is most non-task-oriented, and the state S ₁₀
However, dialogue is the most task-oriented state. The conversation state can change (transition) each time a user or device speaks. Although it is possible to freely change between each dialogue state, there are cases where the dialogue state does not change even if there is an utterance.
The state estimation unit 104 determines the task level based on the content of the utterance performed by the user, and estimates the dialog state each time. A known technique can be used to determine the task level.

By the way, there is a case where an appropriate response mode cannot be selected only by determining the task level using a uniform standard. For example, in an utterance such as “Is it clear tomorrow?”, There is a case where the user A has an intention to request a weather forecast and the user B has an intention to have a conversation about tomorrow's schedule.
Therefore, the interactive apparatus according to the present embodiment performs reinforcement learning by means described later, and determines a response mode by using the obtained learning results together.

The learning unit 105 is means for accumulating data for determining the response mode by reinforcement learning. The learning unit 105 learns, by Q-learning, in what response mode the response state is maximized (i.e., can the response satisfying the user be satisfied) in what response mode the response is in? To do.
3A and 3B are examples of tables (reward tables) held by the learning unit 105. FIG. In the present embodiment, the dialogue state (corresponding to the state estimator conversation state 104 is estimated), response mode (A ₁ is meant the response of non-task-oriented, A ₂ means the response of task-oriented) Is stored in association with the expected reward.
The expected reward is a reward expected in reinforcement learning, and is a value representing the appropriateness of response in the present embodiment. The expected reward is updated through learning based on, for example, user satisfaction. The way of learning and the expected reward will be described later.

FIG. 3A shows a reward table in which initial values are stored, and FIG. 3B shows a reward table after learning. For example, in the case of FIG. 3B, it is shown that when the dialogue state is S ₁ , a larger reward can be expected when a non-task-oriented response is made (that is, the user is satisfied). Also, if the conversation state is in S _10, it is shown that can be expected is greater than reward when the response of the task-oriented.

  The learning unit 105 has means for temporarily or permanently storing data used by the apparatus. For example, it is preferable to use a storage medium such as a flash memory that can read and write at high speed.

  The evaluation acquisition unit 106 is a means for acquiring a reward for the learning unit 105 to perform reinforcement learning. In this embodiment, the reward is a value (user evaluation) that represents the preference of the response generated by the response generation unit 103, and is acquired afterward based on the user's reaction.

Here, a reward acquisition method will be described.
When the response generated by the response generation unit 103 is an appropriate response mode (that is, when the response mode matches the response mode desired by the user), it is assumed that the dialogue continues smoothly. In addition, when the response generated by the response generation unit 103 is not in an appropriate response mode, it is assumed that the conversation does not continue smoothly. Therefore, in this embodiment, after providing a response to the user, a reward is calculated based on whether or not a series of conversations has continued.
For example, when the dialogue continues, a value of +1.0 is used as a reward, and when the dialogue is interrupted, a value of -1.0 is used as a reward. The calculated reward is transmitted to the learning unit 105 and used as learning data for evaluating the response generated immediately before.

In this example, the reward is determined based on whether or not the conversation has continued. However, the reward may be calculated by another method as long as it can be evaluated whether or not the response mode is appropriate. For example, the reward may be calculated based on the length of time (time or number of turns) that the conversation has continued. Moreover, you may calculate a reward based on the time until a user reacts.
In addition, the reward may be calculated based on the result of sensing the user. For example, the user's satisfaction may be estimated based on a voice tone or the like and used as a reward. Of course, the reward may be calculated based on the content of the utterance (for example, whether or not an utterance that denies the response mode in the immediately preceding response was made).

The interactive device 100 can be configured as an information processing device having a CPU, a main storage device, and an auxiliary storage device. Functions described below are realized by loading a program stored in the auxiliary storage device into the main storage device and executing it by the CPU. Note that all or part of the illustrated functions may be executed using a circuit designed exclusively (such as a semiconductor integrated circuit).

<Process flowchart>
Next, processing performed by each unit shown in FIG. 1 will be described with reference to FIG. 4 which is a processing flowchart. The flowchart shown in FIG. 4 is started from an initial state where the apparatus is not learning.

First, in step S11, a default value is inserted into the reward table. In the present embodiment, ten dialog states as shown in FIG. 3A are defined, and A ₁ (non-task-oriented response) and A ₂ (task-oriented response) are associated with each. Yes. In addition, an expected reward (Q value) is associated with each response. The expected reward (Q value) is a dimensionless number that indicates which mode is more rewarding (user satisfaction) when responding. In this example, 5 is inserted in all items as a default value.
Further, when the dialogue with the user is started, the process proceeds to step S12.

In step S12, the utterance acquisition unit 102 acquires the utterance performed by the user and performs voice recognition. The text obtained as a result of recognition is transmitted to the response generation unit 103 and the state estimation unit 104.
Next, the state estimation unit 104 estimates the current dialog state based on the content of the utterance performed by the user. For example, if the initial state of the states shown in FIG. 2 is S ₀ (a state meaning no utterance), it is estimated from S ₀ to which of the states S _{1 to} S ₁₀ . Further, when the current state is in any of S _{1 to} S ₁₀ , it is estimated whether the state has further transitioned to another state. Note that the state may not change (transition to the same state as a result of the utterance).

Next, in step S13, the response generation unit 103 selects a response mode with reference to the state estimated by the state estimation unit 104 in the previous step and the reward table stored in the learning unit 105. For example, compensation table stored in the learning unit 105, a state as shown in FIG. 3 (B), the dialogue state, which is estimated in step S12 is assumed to be S _2. In this case, the response mode in which the expected reward is the highest in the conversation state S ₂ is A ₁ (non-task-oriented response). Therefore, in this case, the response generation unit 103 selects a non-task-oriented response as the response mode.

  Next, in step S <b> 14, the response generation unit 103 generates a response according to the selected response mode, and outputs the sound via the input / output unit 101.

  In step S15, whether to perform reinforcement learning is selected based on the user's response to the output response. Reinforcement learning is ON by default, and is continued until a learning end determination is made in step S19 described later.

Steps S16 to S19 are steps for performing reinforcement learning based on the user's response to the output response. In this step, based on the utterance made by the user immediately before, the response uttered by the device in the previous conversation turn is evaluated.
FIG. 5 shows an example of a conversation turn. In this example, at time t1, a user who wants to know tomorrow's weather forecast utters “Is it fine tomorrow?”, And at time t2, the dialog device is a non-task-oriented type that says “I hope it is clear!”. It is assumed that a response (response in a mode not desired by the user) is returned (step S14).
On the other hand, it is assumed that the user rephrased “Tell me the weather tomorrow” at time t4 after a little silence.

  In this case, the process of step S12 is performed at time t1, and the process of step S14 is performed at time t2. However, since there is no previous conversation turn, the processes after step S16 are omitted.

On the other hand, when the user's utterance is acquired at time t4, since there is a previous conversation turn (turn “no utterance”), the process of step S16 is executed, and the evaluation acquisition unit 106 calculates the reward. Is called.
The reward is a reward in reinforcement learning, and is a score for action selection (selection of response mode in this example). That is, in step S16, it is determined based on the next user's utterance whether or not the response mode in the previous conversation turn was correct.

  There are various methods for calculating the reward. For example, the following calculation method can be considered. Of course, as long as the user's satisfaction with the response mode selected in the previous conversation turn can be estimated, methods other than those exemplified may be used.

(1) Time or number of turns that the conversation lasts without interruption For example, it is possible to estimate whether a response has been made in a desired response mode based on the time that the conversation lasts without interruption or the number of turns of the conversation. This is because if the response is made in an inappropriate response mode, it is considered that the dialogue is less likely to be sustained.

(2) Satisfaction (dissatisfaction) of the user obtained as a result of analyzing the voice uttered by the user
For example, it is possible to estimate whether the user is satisfied by analyzing the pitch, tone, and volume of the voice.

(3) Time until the next utterance is obtained from the user For example, when a response is made in an inappropriate response mode, the user may be confused. When this criterion is applied to the example of FIG. 5, since silence occurs at time t3, the device can know that the reward for the response made at time t2 was low at time t4.

Next, in step S17, the learning unit 105 performs reinforcement learning based on the obtained reward, and updates the reward table.
In Q learning, the expected reward when the action a is performed in the state s under the policy π is expressed as Q ^π (s, a). Policy π is a reward table, and state s is the dialog state estimated in step S12. A is A ₁ or A ₂ .
In this example, when the conversation is interrupted as a result of the response from the device (when silence occurs, that is, when the state transitions to S ₀ ), a reward of −1.0 is given and the conversation continues. It is assumed that a reward of +1.0 is given (that is, when the state transitions from S _{1 to} S ₁₀ ).
When such a learning is continued and the reward table is updated, the Q value that maximizes the reward converges to a specific value for each dialog state.

  In step S18, it is determined whether or not the amount of change in the Q value (expected reward) when the reward table is updated is greater than a threshold value. Here, when the fluctuation amount is large to some extent, the process returns to step S12. On the other hand, if the fluctuation amount is sufficiently small, it is considered that the target value has already been converged. Therefore, the process proceeds to step S19, and the flag for performing reinforcement learning is turned OFF. As a result, the process does not shift to step S16 and thereafter.

When the processing described above is repeated, a reward table for selecting an optimal response mode by Q learning is updated by learning. This makes it possible to appropriately determine whether to perform a task-oriented response or a non-task-oriented response.

(Second embodiment)
The second embodiment is an embodiment in which a user who has a conversation is identified and learning is performed for each user. FIG. 6 is a system configuration diagram of the interactive apparatus 100 according to the second embodiment.
The dialogue apparatus 100 according to the second embodiment is different from the dialogue apparatus 100 according to the first embodiment in that the dialogue apparatus 100 includes means (user identification unit 107) for identifying a user who has a dialogue.

The user identification unit 107 is means for identifying a user based on the acquired voice. For example, the user identification unit 107 assigns a unique identifier to the user who has emitted the voice based on the result of analyzing the voice. In addition, although the analysis of the audio | voice was illustrated here, you may identify a user based on a user's face image etc.
In the second embodiment, a plurality of reward tables is stored for each user identifier, and reinforcement learning is possible for each user. As described above, by performing the processing in the first embodiment for each user, a more personalized answer is possible.

  In addition, when the interactive device interacts simultaneously with a plurality of users, the processing may be performed while switching the reward table each time the user currently interacting changes. In addition, when a user who is interacting cannot be identified, a default reward table (a reward table learned so as to maximize the reward for a general user) may be used.

(Third embodiment)
The third embodiment is an embodiment in which an object (that is, a topic) referred to by a user in a conversation is identified and learning is performed for each topic. FIG. 7 is a system configuration diagram of the interactive apparatus 100 according to the third embodiment.
The dialogue apparatus 100 according to the third embodiment is different from the dialogue apparatus 100 according to the first embodiment in that the dialogue apparatus 100 includes means (topic identification unit 108) for identifying an object referred to by the user in the dialogue.

The topic identification unit 108 is a means for identifying a target (target topic) that is referred to by the user in the conversation. The topic in the dialogue can be identified by, for example, performing morphological analysis on the text obtained as a result of speech recognition and analyzing the obtained plurality of words. For example, the topic identification unit 108 assigns a unique identifier (topic identifier) for each topic.
In the third embodiment, a plurality of reward tables is stored for each topic identifier, and reinforcement learning can be performed for each topic. As described above, by executing the processing in the first embodiment for each topic, a more accurate answer can be obtained.

(Modification)
The above embodiment is merely an example, and the present invention can be implemented with appropriate modifications within a range not departing from the gist thereof.
For example, the second embodiment and the third embodiment may be combined to use both the user and the topic. In this case, the reward table may be integrated or used separately. When the reward table is not integrated, after determining the response mode individually, it may be determined which one is adopted based on the obtained Q value.

DESCRIPTION OF SYMBOLS 100 Dialogue device 101 Input / output part 102 Speech acquisition part 103 Response generation 104 State estimation part 105 Learning part 106 Evaluation acquisition part

Claims (10)

An utterance recognition means for recognizing the content of an utterance made by the user;
Response generating means for generating a response to the utterance according to a response policy that is a rule for generating a response to the utterance;
An evaluation acquisition means for determining a user evaluation which is a value representing the preference for the response;
Update means for performing reinforcement learning using the user evaluation as a reward, and updating the response policy;
Have
The response generation means determines a response mode indicating whether to generate a task-oriented response or a non-task-oriented response according to the response policy, and generates a response according to the response mode.
Interactive device. It further includes state estimation means for estimating a current conversation state based on the content of the utterance from a plurality of conversation states associated with the degree of task-oriented conversation.
The response generation means performs the reinforcement learning using the plurality of dialogue states.
The interactive apparatus according to claim 1. The reinforcement learning is Q learning,
The response policy is a Q value associated with the combination of the dialog state and response mode.
The interactive apparatus according to claim 2. Further comprising object estimation means for identifying objects that the user refers to in the dialogue;
For each identified object, the dialog state estimation and reinforcement learning are performed.
The interactive apparatus according to claim 2 or 3. Speaker estimation means for identifying the user;
For each identified user, the dialog state estimation and reinforcement learning are performed.
The interactive apparatus according to any one of claims 2 to 4. The evaluation acquisition means determines the user evaluation based on the user's utterance after responding according to the response policy;
The interactive apparatus according to claim 1. The evaluation acquisition means determines the user evaluation based on a length of a series of dialogues after responding according to the response policy.
The interactive apparatus according to claim 1. The evaluation acquisition means determines the user evaluation based on a time from when the user responds according to the response policy until the user speaks further.
The interactive apparatus according to claim 1. An utterance recognition step for recognizing the content of an utterance made by the user;
A response generation step of generating a response to the utterance according to a response policy which is a rule for generating a response to the utterance;
An evaluation acquisition step of determining a user evaluation, which is a value representing the preference for the response;
An update step of performing reinforcement learning with the user evaluation as a reward and updating the response policy;
Including
In the response generation step, a response mode indicating whether to generate a task-oriented response or a non-task-oriented response is determined according to the response policy, and a response is generated according to the response mode.
How to interact.   A program for causing a computer to execute the interactive method according to claim 9.

Images