JP5150583B2 - Interviewer evaluation device, Interviewer evaluation method, Interviewer evaluation program - Google Patents

Description

  The present invention relates to a dialogue system evaluation technique, and more particularly to a technique for evaluating a listener's dialogue in a dialogue system.

  In developing a dialogue system, performance evaluation is one of the important technologies.

  Conventionally, a task-oriented dialogue system in which a computer automatically responds to achieve the purpose such as ticket purchase and schedule reservation has been actively studied. As a method for evaluating such a dialog system, there is a dialog system evaluation method and system disclosed in Patent Document 1 and the like.

  Recently, attention has been paid to the sociality and entertainment of a system that performs a more flexible dialogue such as chat (more non-task-oriented dialogue, that is, non-task-oriented dialogue).

  In the evaluation of such a non-task-oriented dialog having no purpose to be achieved, it is not appropriate to perform evaluation based on the time until the task is achieved or the achievement rate. The method is not enough.

  As a method for evaluating a non-task-oriented dialog system for solving this problem, there is a dialog system evaluation method disclosed in Patent Document 2. This method creates a hidden Markov model from tagged chat conversations, and evaluates the naturalness of dialog system dialogue based on this model.

JP 2004-272740 A JP 2008-262523 A

Dan Jurafsky, Liz Shriberg, and Debra Biasca, “Switchboard SWBD-DAMSL Shallow-Discovery-Function Anoders Manual”, August 1. Allen E.M. Ivey and Mary Bradford Ivey, “Intentional Interviewing and Counseling: Facility Client Development in a Multicultural Society”, Brooks / ColePub.

  In the dialogue system, a speaker is divided into a listener and a speaker, and the listener hears the talk of the speaker positively.

  Conventionally, there has been a method for evaluating a dialog using a hidden Markov model created from chatting as in the dialog evaluation system of Patent Document 2. However, there was nothing that paid attention to the listener dialogue. While listening volunteers and counseling are regarded as important, it is an important issue to evaluate the level of dialogue among listeners.

  Further, Patent Document 2 evaluates a dialog using only a single hidden Markov model created from chat. However, the act of listening to the other party's story, such as a dialogue with the audience, also appears in the chat. Therefore, if you evaluate the likelihood of a dialogue with a hidden Markov model created only from the data of a dialogue with a dialogue, you will evaluate a mix of chatty and familiarity. In this, it cannot be said that the audibility itself is evaluated.

  Therefore, the present invention creates a hidden Markov model of the interviewer dialogue and chat based on the interviewer conversation data and chat data collected in advance, and compares the output probabilities of the data strings of the dialogue actions to be evaluated from both models. Then, the likelihood of the dialogue of the dialogue act is evaluated.

  As an aspect of the listening dialogue evaluation apparatus of the present invention, there is a listening dialogue evaluation device for evaluating a listening dialogue in a dialogue system, which is a hidden Markov model of a listening dialogue that models a transition state of listening dialogue data recorded in the dialogue system in advance. Model learning means for creating a hidden Markov model of chat that models the transition state of the chat data recorded in advance in the dialogue system, and the dialogue data to be evaluated for the hidden Markov model of the interviewer dialogue and the chat Evaluation means for applying to a hidden Markov model to calculate the output probability of the data string of the dialogue data from each model and comparing the output probabilities of the two to evaluate the likelihood of the dialogue of the dialogue to be evaluated Is provided.

  In addition, as an aspect of the method for evaluating a listener dialogue according to the present invention, a method for evaluating a listener dialogue in a dialogue system, in which a model learning means models a transition state of listener dialogue data recorded in advance in the dialogue system. Creating a hidden Markov model of the interviewer dialogue and creating a hidden Markov model of the chat that models the transition state of the chat data pre-recorded in the dialogue system; Applying to the hidden Markov model and the hidden Markov model of the chat, calculating the output probability of the data string of the dialogue data from each model and comparing the output probabilities of the two, the listener of the dialogue of the evaluation target A step of evaluating the likelihood of interaction.

  The present invention may be an aspect of a program for evaluating a listener's performance that causes a computer to function as each means constituting the listener's dialog evaluation apparatus.

  Therefore, according to the above invention, it is possible to objectively and quantitatively evaluate the level of hearing in the dialogue system. In addition, since the ability to hear in dialogue between people can also be used for evaluation, it can be used for skill evaluation of counselors and listening volunteers, for example.

BRIEF DESCRIPTION OF THE DRAWINGS The schematic block diagram of the listener dialog evaluation apparatus which concerns on embodiment of invention. The flowchart which showed the procedure of the interviewer dialogue evaluation which concerns on embodiment of invention. The figure which shows the example of the dialogue act extracted from the dialogue between a listener and a speaker. The figure which showed an example of the hidden Markov model of the hearer dialog calculated by the Example of the hearer dialog evaluation apparatus which concerns on invention. The figure which showed an example of the hidden Markov model of the chat calculated by the Example of the listener interaction evaluation apparatus which concerns on invention. The listening dialogue evaluation execution result by the Example of the listening dialogue evaluation apparatus which concerns on invention. Listening dialogue evaluation execution result (continuation of FIG. 6). Listening dialogue evaluation execution result (continuation of FIG. 7).

  Hereinafter, embodiments of the invention will be described with reference to the drawings.

  The listener dialogue evaluation apparatus 1 according to the embodiment of the invention shown in FIG. 1 performs a series of processing on a dialogue log given by voice or text input, and determines the likelihood of the dialogue as a listener dialogue. The interviewer dialogue evaluation apparatus 1 tags dialogue data to collected data of interviewer dialogues and chats in advance, and models the interviewer dialogues and chat dialogues using a hidden Markov model (hereinafter referred to as HMM). . Then, by comparing the probabilities (output probabilities) that the tag of the dialogue act to be evaluated is output from each of the HMM created from the conversation and the HMM created from the chat, the likelihood of the hearing dialogue is determined. Since the interviewer dialogue contains elements of chat, it is difficult to determine the likelihood of the interviewer dialogue with the method using only the HMM created from the interviewer dialogue. Can be clearly determined. As a result, the evaluation of the dialogue performance of the objective listener in the dialogue system is realized.

  The listening dialogue evaluation device 1 includes an HMM learning unit 30 and a listening dialogue evaluation unit 60. The HMM learning unit 30 and the listener interaction evaluation unit 60 have at least a calculation function based on HMM main algorithms (Viterbi algorithm, forward algorithm, Baum-Welch algorithm, etc.).

  The HMM learning unit 30 creates a talker dialogue HMM 40 that models the transition state of the interviewer dialogue data 10 recorded in advance in the dialogue system, and creates a chat HMM 50 that models the transition state of the chat data 20 recorded in advance in the dialogue system. To do.

  Further, the HMM learning unit 30 generates a listener dialogue HMM 40 and a chat HMM 50 reflecting the dialogue action by attaching a tag indicating the dialogue action to each data string in the listener dialogue data 10 and the chat data 20.

  In other words, the HMM learning unit 30 performs dialogue for each stage of dialogue that proceeds with one recorded in advance as a listener (hereinafter referred to as “listener dialogue”) and a dialogue that proceeds without dividing the listener / speaker (hereinafter referred to as “chat”). The learning data (listener dialogue data 10 and chat dialogue data 20) labeled with “dialogue action tag” as an action tag is received. Then, a hearing person HMM 40 and a chat HMM 50 in which the probability of transition to the state of each dialogue action is modeled by the HMM are generated. An EM algorithm (Baum-Welch algorithm), which is one of the main algorithms, is applied to the generation of the HMMs 40 and 50.

  When the utterance form is speech, the “listener dialogue” and “chat” are preliminarily converted into text by a well-known automatic speech recognition technique or human transcription.

  Examples of the “dialogue action tag” include “self-disclosure”, “information provision”, “conformity”, “question,“ sympathy ”,“ greeting ”and the like defined below.

  Self-disclosure: Utterances that disclose preferences and emotions. (Example) I like watching movies.

  Information provision: An utterance that conveys objective information. (Example) Tony Leon appears in Redcliffe.

  Aiso: Encourage dialogue partners to speak. (Example) Yeah.

  Question: An utterance that expects the other's response. (Example) What other movies do you like? .

  Empathy: sympathetic utterances, encouragement of others, etc. (Example) I also love Tony Leon.

  Greeting: Social utterances seen at the beginning and end of the dialogue. (Example) Hello.

  The tagging method of “dialogue action tag” for each stage of dialogue may be manual input or automatic. As an automatic method, for example, there is a method of automating labeling from a certain amount of learning data by using a sequence labeling method such as CRF (Conditional Random Fields). The above tagging method may be applied to the addition of a dialogue action tag to the dialogue log data 70 provided to the listener dialogue evaluation unit 60.

  The interviewer dialogue evaluation unit 60 calculates the output probability of the data string of the dialogue log data 70 from the HMMs 40 and 50 by applying the dialogue log data 70 of the evaluated dialogue 90 to be evaluated to the interviewer dialogue HMM 40 and the chat HMM 50. By comparing the output probabilities of the two, the likelihood of the dialogue of the evaluated dialogue 90 is evaluated.

  In addition, the dialogue log data 70 is provided with a tag indicating a dialogue action for each data string in the same manner as the interviewer dialogue data 10 and the chat data 20, thereby evaluating the likelihood of the dialogue of the evaluated dialogue 90 reflecting the dialogue act. Is executed by the listener pair evaluation unit 60. The dialog log data 70 is obtained from the voice or text of the dialog as in the data 10 and 20. When the utterance that is the basis of the data 70 is voice, it is converted into text by the instant messenger function.

That is, the interviewer dialogue evaluation unit 60 applies the series of dialogue action tags of the dialogue log data 70 to be evaluated to the models of the interviewer HMM 40 and the chat HMM 50 generated by the HMM learning unit 30. Then, the probability that the series of dialogue action tags is output from each of the interviewer HMM 40 and the chat HMM 50 is calculated. For the calculation of these probabilities, the Viterbi algorithm and the forward algorithm, which are the main algorithms, are applied. Then, based on the calculated output probability P _k in the hearer dialogue HMM and output probability P _z in the chat HMM, a hearer dialogue score 80 which is a score S representing an index of “hearing likelihood” is set as S = log (P _k ) −log ( _Pz ) is calculated and output.

  With reference to the flowchart of FIG. 2, a hearing dialogue evaluation procedure according to this embodiment and specific examples of each procedure will be described.

  S1: Prior to evaluation of the interviewer dialogue, the interviewer dialogue and chat are recorded, and the interviewer dialogue data 10 and chat data 20 are created in advance.

  In the embodiment of S1, data was recorded according to the following flow. A total of 16 participants in the data collection of this example, 8 were listening and the remaining 8 were speaking. The male-female ratio was halved, and the age was 21-29 years old. Each experiment participant first had two dialogue chats, then two dialogue listener dialogues, for a total of four dialogues. A total of 32 dialogues were collected.

  When recording the chat, the participants are not informed about the roles assigned to them afterwards. After the chat recording, we inform each role. Before and during the recording, the listener and the speaker were recorded in separate rooms so that the participants did not face each other.

  The dialogue shown in Fig. 3 was recorded as an example of the interviewer dialogue. In FIG. 3, the speaker (L) repeatedly asks the speaker (S) questions to encourage self-disclosure of the speaker.

  Prior to the start of the experiment, a favorite topic was selected from a plurality of topics prepared by the experimenter, and each experiment participant chatted for 30 minutes on the favorite topic. Topics included food, travel, movies, music, entertainers, sports, health, housework and childcare, computers and the Internet, animals, fashion, and games.

  Table 1 shows the number of conversations collected, the number of utterances, and the average number of listeners and spoken words per utterance. The utterances in the listener dialogue are longer than the chat utterances. This seems to be because the participants in the experiment described themselves more in detail to help them understand themselves.

  In addition, after each dialogue, the experiment participants were asked to answer a questionnaire, and the satisfaction level of the dialogue and how much they were able to speak to the other party of the dialogue, etc. Measured at. According to the results of this questionnaire, the speaker was able to get a feeling of “being heard” without lowering the level of satisfaction compared to the chat (Welch's pairwise t-test; p = 0.016). This indicates that it is meaningful to analyze the characteristics of the collected interviewer dialogues as well as the role of the participants in the experiment.

  S2: Any of “self-disclosure”, “information provision”, “conformity”, “question”, “sympathy”, and “greeting” defined as dialogue action tags for the dialogue logs of the interviewer dialogue data 10 and chat data 20 collected in S1. Is added to each sentence of the dialogue log. A single sentence is basically a punctuation mark or a break at the end of a single utterance. If the utterance has ended in the middle of a single sentence, or a punctuation mark has not been inserted properly, an annotator (Tag giver) decided the appropriate sentence break.

  For these tags, a DAMSL tag set (Non-patent Document 1) created for tagging a general dialog and a tag set for therapy dialog (Non-patent Document 2) were referred to.

  The annotator added seven tags (six dialog act tags and "other" tag) to each utterance. As a result, 1177 tags were assigned to the listener dialogue and 1312 tags were assigned to the chat. The reason why the number of tags and the number of utterances do not match is that a tag is attached to each sentence because one utterance is long due to the nature of text dialogue.

  S3: The HMM learning unit 30 learns the HMM by using the interviewer dialogue data 10 and the chat data 20 to which the dialogue action tag is assigned in S2, so that the interviewer dialogue HMM 40 is obtained from the interviewer dialogue data 10, and the chat HMM 50 is obtained from the chat data 20. Create

In the embodiment of S3, a series of dialogue action tags is used as learning data, and HMM learning is performed using an EM algorithm as the main algorithm. In order to find the optimum HMM with the fewest number of states, the number of states was varied from 1 to 10, and 100 HMMs in each state number, a total of 1000 HMMs, were learned. This is because the learning result of the HMM changes depending on the initial probability. One optimal HMM is selected from the 1000 HMMs using an MDL (Minimum Description Length) scale. Seven utterance action tags were modeled using a total of 14 observations, considering the listener and the speaker as separate observations. Specifically, the observed values are the following 14 items.
(0) Interviewer_Self Disclosure (1) Interviewer_Information Provision (2) Interviewer_Aoi (3) Interviewer_Question (4) Interviewer_Empathy (5) Interviewer_Greeting (6) Interviewer_Other (7) Talker_ Self-disclosure (8) Talking _ Information provision (9) Talking _ Aoi (10) Talking _ Question (11) Talking _ Empathy (12) Talking _ Greeting (13) Talking _ Other Use two methods HMM was analyzed. That is, only half of the state outputs only the speaker's discourse act tag, and the other half outputs only the listener's discourse act tag, and learning is made that state transition is possible to all states.

  Listening dialogue HMM and chat HMM, which are learning results in S3, are shown in FIGS. 4 and 5, respectively.

  4 and 5, states (1) to (6) indicated by the elliptic model describe the speaker, the speech action thereof, and the observation probability thereof, and “speaker: speech action tag: probability”. It is shown in order. L represents "listener", S represents "speaker", Greeting represents "greeting", Self-disclosure represents "self-disclosure", Question represents "question", and Ack represents "confidence". Moreover, the numerical value attached | subjected to the arrow of FIG. 4, FIG. 5 shows the transition probability in the same state, or the transition probability between two different states. Only the observation probability and transition probability are 0.1 or more. For example, “L: Greeting: 0.888” in the state (1) shown in FIG. 4 indicates that the observation probability of the “greeting” of the listener is 0.888.

  S4: The dialogue 90 to be evaluated is recorded, a tag similar to the dialogue act tag of S2 is assigned to each sentence, and tag series dialogue log data 80 is created. Tagging in the example of S4 was performed manually.

  S5: The interviewer dialogue evaluation unit 60 calculates the probability that the tag sequence of the dialogue log data 70 is output from each of the interviewer dialogue HMM 40 and the chat HMM 50, and uses these calculated probabilities for the calculation according to the following formula. Score 80 is calculated.

S = log (P _k ) −log (P _z )
S: Interviewer score 80
P _k : Output probability of the tag sequence of the dialogue log data 70 calculated from the interviewer dialogue HMM 40 P _z : Output probability of the tag sequence of the dialogue log data 70 calculated from the chat HMM 50 If the calculated interviewer dialogue score 80 is large, The larger it is, the more likely it is to be “listening dialogue”. On the contrary, it is determined that the smaller the value is, the closer to chat.

  FIG. 6 to FIG. 8 show the calculation execution result (listening dialogue identification execution result) which is the embodiment of S5. Here, the output probabilities of 8 interview dialogues and 8 chat dialogues are calculated for the HMM created from 8 interview dialogues and 8 chats, and the interviewer score S is calculated.

  6 to 8, the line from “seq” to “kikiyaku ratio” shows the identification result of one dialogue, and “seq” is the tag from the start to the end of one dialogue. A series is represented, and 14 observation values used at the time of HMM learning are expressed as numerical values. “0” to “13” in the illustrated execution result correspond to the observation values (0) to (13) in step S3, respectively. “Kikiyaku prob” is the output probability from the listener dialogue HMM, and “zatadan prob” is the output probability from the chat HMM. Further, “kikiyaku ratio” represents a hearing score S.

  A value greater than or equal to 0 is calculated for the interviewer dialogue, and a value of 0 or less is calculated for the chat.

  By listening, people are satisfied with their desire to speak and can maintain a good mental state. As is clear from the above description, the listener dialogue evaluation apparatus 1 according to the present invention evaluates that the dialogue that seems to be a hearer can be performed as the hearer dialogue score is high. Can be determined.

  Therefore, an objective evaluation of the interviewer dialogue can be performed. It is also useful for improving the skills of counselors and listening volunteers. Furthermore, when developing a system to be a listener, the evaluation cost can be reduced.

  The functional means 30 and 60 that constitute the above-described hearing aid evaluation apparatus 1 can be constituted by computer hardware resources such as a CPU, a memory (RAM), a hard disk device, and a communication device. That is, the functional units 30 and 60 can be realized by cooperation between the CPU and the program. The data 10, 20, 70, 80 may be temporarily stored in a recording unit exemplified by a hard disk device or a server device. The data output from the function unit 30 and the data output from the function unit 60 are appropriately output and displayed via the display device.

  Further, the present invention can be realized by a computer CPU (MPU) reading and executing the program as a document storage program for causing the computer to realize the functional units 30 and 60 or a computer-readable recording medium recording the program. In this case, the program itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium storing the program, for example, a CD-ROM, DVD-ROM, CD-R, MO, An HDD or the like constitutes the present invention.

DESCRIPTION OF SYMBOLS 1 ... Interviewer evaluation apparatus 10 ... Interviewer data 20 ... Chat data 30 ... HMM (Hidden Markov Model) learning part (model learning means)
40 ... HMM
50 ... Chat HMM
60 ... Interviewer evaluation section (evaluation means)
70 ... Dialogue log data 80 ... Interviewer dialogue score

Claims (9)

A listener dialogue evaluation apparatus for evaluating a listener dialogue in a dialogue system,
Create a hidden Markov model of the interviewer model that models the transition state of the interviewer conversation data recorded in advance in the dialog system and create a hidden Markov model of the chat model that models the transition state of the chat data recorded in the dialog system in advance Model learning means,
The dialogue data to be evaluated is applied to the hidden Markov model of the interviewer dialogue and the hidden Markov model of the chat to calculate the output probability of the data string of the dialogue data from each model and compare the output probabilities of the two And an evaluation means for evaluating the likelihood of the dialog of the dialogue to be evaluated. 2. The apparatus for evaluating a listener interaction according to claim 1, wherein the evaluation unit evaluates the likelihood of the speaker interaction based on a logarithmic difference between the output probabilities of the two. The model learning means creates a hidden Markov model of the interviewer dialogue based on a tag that indicates an interaction act for each data sequence of the interviewer dialogue data, and performs an interaction act for each data sequence of the chat data. The listening dialogue evaluation apparatus according to claim 1 or 2, wherein a hidden Markov model of the chat is created based on a tag to which the tag is indicated. The evaluation means applies a tag that indicates a dialogue action for each data string of dialogue data to be evaluated to the hidden Markov model of the interviewer dialogue and the hidden Markov model of the chat, from each model. The listening dialogue evaluation apparatus according to claim 3, wherein an output probability of a data string of the dialogue data is calculated. A listener dialogue evaluation method for evaluating a listener dialogue in a dialogue system,
Hidden Markov model of conversational conversation modeled by the model learning means modeled transition state of conversational conversation data pre-recorded in the interactive system while creating a hidden Markov model of conversational conversation modeled in advance Creating a model,
The evaluation means applies the dialogue data to be evaluated to the hidden Markov model of the interviewer dialogue and the hidden Markov model of the chat to calculate the output probability of the data string of the dialogue data from each model, and outputs both And evaluating the likelihood of the dialogue of the dialogue to be evaluated by comparing the probabilities. 6. The method for evaluating a listener interaction according to claim 5, wherein, in the step of evaluating, the likelihood of the speaker interaction is evaluated based on a logarithmic difference between the output probabilities of the two. In the step of creating the hidden Markov model of the interviewer dialogue and the hidden Markov model of the chat, the hidden Markov model of the interviewer dialogue is generated based on a tag indicating a dialogue action for each data string of the interviewer dialogue data. 7. The method for evaluating a listener's dialogue according to claim 5 or 6, wherein a hidden Markov model for the chat is created based on the chat data having a tag indicating a dialogue action for each data string of the chat data. . In the evaluation step, each of the data strings of the interaction data to be evaluated is applied with a hidden Markov model of the interviewer dialogue and a hidden Markov model of the chat by applying a tag indicating a dialogue action. 8. The method for evaluating a listener dialogue according to claim 7, wherein an output probability of a data string of the dialogue data is calculated.   A program for evaluating a dialogue with a listener for causing a computer to function as each means constituting the apparatus for evaluating a dialogue with a listener according to any one of claims 1 to 4.

Images