JP2006039484A - Reliability degree calculation processing method for dialog comprehension result - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To increase the estimated accuracy of reliability of comprehension result about a user information request content in a voice dialog system. <P>SOLUTION: By making use of a prototype voice dialog system, a dialog record 10 which includes system utterance information, user utterance information, comprehension result of a user information request content by means of a voice dialog system for each user utterance and information about its correctness is produced, and it is stored in a memory device. The conversation information regarding the exchange between the system and the user is retrieved from the stored dialog record 10, and the conversation characteristics quantity is extracted by a conversation characteristics quantity extracting part 30. A reliability criterion producing part 50 produces a reliability criterion for evaluating the reliability of the comprehension result of the user information request content by making use of a conversation characteristics quantity extracted from the conversation information in addition to a sound/linguistic characteristic quantity of a user's utterance voice by means of identification learning. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a voice dialogue technology in which a computer interacts with a person using voice. In particular, in a voice dialogue system, the reliability of the understanding result of user information request contents is added to the acoustic and linguistic characteristics of user voice, It is related with the reliability calculation processing method of the dialogue understanding result calculated using the discourse information regarding the interaction between the system and the system.

  Every time a user inputs an information request to the system using voice, the user's system interaction history is used to update the understanding result of the user information request content held by the system. A system that responds to the user by voice according to the understanding result is called a voice dialogue system.

  FIG. 2 shows a general configuration of the spoken dialogue system. 1 is an utterance understanding unit, and 2 is an utterance generation unit. In the utterance understanding unit 1, 100 is a speech recognition unit, 101 is a language understanding unit, and 102 is a discourse understanding unit. In the utterance generation unit 2, reference numeral 200 denotes a content generation unit, 201 denotes a surface layer generation unit, and 202 denotes a voice generation unit. 103 is a context understanding rule / knowledge database (DB) in which knowledge is stored; 104 is a syntax / semantic understanding rule / knowledge DB in which knowledge is stored; 105 is a dialog in which a dialog state is stored The state DB 106 is a dialog state response generation rule / knowledge DB in which dialog control response generation rules / knowledge is stored, 107 is a database in which data that the user wants to know is stored, and 108 is a generation rule in which a generation rule template is stored. It is a template DB.

  The user voice used when the user inputs an information request to the utterance understanding unit 1 of the voice dialogue system is called user utterance, and the information request content transmitted to the voice dialogue system by one utterance of the user is called dialogue action. Deriving dialogue actions from user utterances is called language understanding.

  In the voice dialogue, first, the voice recognition unit 100 of the utterance understanding unit 1 receives a user uttered voice and outputs a word string as a recognition result. The language understanding unit 101 receives a word string of the recognition result as input, and performs natural language processing such as keyword extraction, syntax analysis, and semantic analysis using syntax semantic understanding rules / knowledge in the syntax semantic understanding rules / knowledge DB 104, Outputs the understanding result of utterance intention and utterance content called dialogue action.

  The discourse understanding unit 102 receives the dialogue action and the current dialogue state as input, and updates the dialogue state in the dialogue state DB 105 using the context understanding rule / knowledge in the context understanding rule / knowledge DB 103. Dialogue state refers to information about various dialogues that the system maintains internally. The main element is the understanding result of the user information request contents, and also holds the user utterance history, system utterance history, and the like.

  Based on the updated dialog state, the content generation unit 200 of the utterance generation unit 2 uses the dialog state response generation rule / knowledge in the dialog control response generation rule / knowledge DB 106 and the information in the database 107 to create a system. , The surface layer generation unit 201 outputs the utterance character string using the generation rule template in the generation rule template DB 108 based on the dialogue action, and the voice generation unit 202 generates the system utterance. Generate.

  FIG. 3 shows an example of dialogue and an understanding result of information request contents of the voice dialogue system for each user utterance. In the example of FIG. 3, the understanding result of the user information request content is represented by a frame expression. The frame representation is a data structure composed of attribute value pairs called slots, and is used in many systems to represent the understanding result of the user's information request contents. The whole frame of “place, date, info” shown in FIG. 3 is a frame, and each element of “place”, “date”, and “info” is a slot. The spoken dialogue system understands the user's information request contents by filling appropriate values in each slot by user utterance. As a value for filling the slot, for example, a specific word (keyword) included in the speech recognition result can be considered.

  As shown in FIG. 3A, when the values for each of the slots “place”, “date”, and “info” are not filled, for example, the first system utterance (S1) “Please contact us” Suppose that "Please tell me the weather tomorrow" is returned as the first user utterance (U1).

  If the spoken dialogue system misrecognizes "tomorrow" in the first user utterance (U1) as "today", the understanding result of the information request contents of the user after U1 is shown in FIG. Thus, “Today” is filled as the value for the slot “date”, and “Weather” is filled as the value for the slot “info”.

  Next, for example, in response to the utterance “Where is the weather” in the second system utterance (S2), if “It is Tokyo” is returned as the second user utterance (U2), As shown in FIG. 3C, the user's understanding result of the information request content is “Tokyo” as the value for the slot “place”, “today” as the value for the slot “date”, and “value” for the slot “info”. The weather is filled.

  Next, for example, in response to the utterance of “the weather in Tokyo today” in the third system utterance (S3), “No, tomorrow” is returned as the third user utterance (U3). As shown in FIG. 3D, the understanding result of the user's information request contents after U3 is “0” for the slot “date” in the understanding result of the information request contents of the user in FIG. Changed to "Tomorrow". Therefore, the voice dialogue system returns “I will tell you. The weather in Tokyo tomorrow will be fine” as the fourth system utterance (S4).

  In order to convey correct information to the user, the spoken dialogue system needs to correctly understand the user's information request contents. However, since the output of the speech recognizer is not perfect, the user's information request content held by the speech dialogue system as a result of dialogue with the user may contain errors. For this reason, a technique for calculating the reliability of the understanding result of the user's information request content and detecting an error in the information request content has been studied. With this technology, it is possible to smoothly proceed with dialogues, such as requesting correction of incorrect parts and preferential confirmation.

  FIG. 4 shows an example of assigning the reliability of the user information request contents using the reliability of words included in the speech recognition result of the user utterance. In FIG. 4, the reliability is assigned to each word being uttered by the user, and when the word enters the slot, the reliability of the slot value becomes the reliability of the word. That is, for each understanding result of the information request contents of the user, a reliability indicating whether or not the understanding result is correct is calculated and assigned to each slot.

  For example, if the spoken dialogue system misrecognizes the word “tomorrow” in the first user utterance (U1) as “today”, the understanding result of the information request content of the user after U1 is shown in FIG. ), The word “today” is filled as the value for the slot “date”, and “c1” is assigned as the reliability thereof, but “No, it is tomorrow” is input as the third user utterance (U3). In the understanding result of the information request contents of the user after U3, the slot value for the slot “date” is changed from the word “today” to the word “tomorrow” as shown in FIG. As the reliability, “c9”, which is the reliability of the word “tomorrow” in U3, is assigned.

  This reliability is used to reject the understanding result of the information request contents of the user or to confirm by the system utterance such as “Is it weather today?” According to the value of the reliability.

  The conventional methods for calculating the reliability of the understanding result of the user's information request include the method using the acoustic score output by the speech recognizer as the reliability, the method using the recognition accuracy of the speech recognizer investigated in advance, For example, there is a method of creating a scale for obtaining reliability from the acoustic and linguistic features of the recognition result and using the scale.

  The following non-patent document 1 proposes a method of assigning, as reliability, the probability that each element included in the understanding result of the information request content of the user is correct. For example, when the user utters an utterance including the keyword “Tokyo”, the probability of correctly hearing “Tokyo” is obtained in a dialog record created in advance, and the value is used as the reliability.

  Non-Patent Document 2 does not deal directly with the understanding result of the user's information request contents, but for each element (word or sentence) in the user's utterance that can constitute the understanding result of the user's information request contents, We have proposed a method for obtaining reliability with high accuracy from acoustic and linguistic features output by the recognizer.

  Specifically, first, a plurality of user utterances are collected in advance, and speech recognition results and acoustic / linguistic features that are output by the speech recognizer for those uttered speeches are obtained. FIG. 5 shows an example of acoustic and linguistic features related to a word being uttered by a user. The “phoneme reliability” in FIG. 5 is the frame number average frame purity of the difference in score from the maximum likelihood state of the HMM in each acoustic frame (N-best purity is considered for phonemes in each acoustic frame) ). FIG. 6 shows an example of acoustic and linguistic features related to speech. Next, the output (words and sentences) of the speech recognizer is labeled for correctness. Finally, an expression (hereinafter referred to as a reliability measure) is created that determines whether each element of the speech recognition result is “correct” or “incorrect” from the acoustic and linguistic features.

  The speech recognition system can classify a correct answer or an incorrect answer for a word in an unknown user utterance in the future by using the reliability scale created in this way. Moreover, the classification accuracy can be used as the reliability.

Non-Patent Document 3 is similar to Non-Patent Document 2 in that the speech recognition result output from the speech recognizer and the acoustic and linguistic features are used, but for each content of the system utterance immediately before the target user utterance. The idea is to create a reliability scale. In addition, the present invention is similar to the present invention in that it uses the discourse information of the immediately preceding system utterance, but the present invention expresses the discourse information in numerical form, and is different from the prior art in this respect.
"Efficient Spoken Dialogue Control Depending on the Speech Recognition Rate and System's Database," Kohji Dohsaka and Norihito Yasuda and Kiyoaki Aikawa, In Proc. Eurospeech, 2003, pp.657-660 "Recognition confidence scoring and its use in speech understanding systems," Timothy J. Hazen, Stephanie Seneff and Joseph Polifroni, Computer Speech and Language, 2O02, vol.16, pp.49-67 "Estimating Semantic Confidence for Spoken Dialogue systems," Sameer S. Pradhan and Wayne H. Ward, In Proc. ICASSP, 2002, pp.233-236

  In the conventional method, although the dialogue consists of the interaction between the system and the user, the reliability of the understanding result of the user information request content is mainly used only for the acoustic feature and the linguistic feature output by the speech recognition result. Therefore, there is a problem in estimation accuracy.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above-described problems of the prior art and to improve the estimation accuracy of the reliability of the understanding result of user information request contents.

  In order to solve the above-mentioned problems, the present invention maintains a speech dialogue system using a history of exchanges between the user and the speech dialogue system every time the user inputs an information request to the voice dialogue system using a computer. In the spoken dialogue system in which the user information request content understanding result is updated and the user responds by voice according to the updated user information request content understanding result, the reliability of the user information request content understanding result is as follows: Is calculated.

  First, using a prototype spoken dialogue system, create a dialogue record that includes system utterance information, user utterance information, user information request content understanding results and correct / wrong information for each user utterance. And save it in the storage device.

  Next, the discourse information regarding the interaction between the user and the voice interaction system is extracted from the conversation record stored in the storage device, and the number of appearances or the appearance ratio or change of the understanding result value of the user information request content in the discourse information is obtained. Extract the discourse feature amount related to the indicated feature.

As the discourse feature amount, for example, at least one of the following feature amounts is included.
-Appearance ratio of the current value of the user information request content understanding result-Appearance ratio of the most frequently occurring value in the user information request content understanding result-Number of types of appearance values in the user information request content understanding result-User information Number of times the current value of the understanding result of the request content was canceled ・ Number of times the current value of the understanding result of the user information request content was overwritten by other values ・ How many times the current value of the understanding result of the user information request content was Spoken dialogue system shows the value that indicates whether the current value of the user information request content has a different value until the current value of the user information request content understands the value that the user information request content understands Number of times the user transmitted the same value to the spoken dialogue system after confirmation ・ How many times the same value as the current value of the understanding result of user information request appeared in the user's utterance so far The value that indicates the number of times that a different value from the current value of the understanding result of the user information request content has appeared in the user utterance so far The same value as the current value of the understanding result of the user information request content A value that indicates how many times the system utterance has appeared in the past ・ A value that indicates how many times a value different from the current value of the understanding result of the user information request content has appeared in the system utterance so far Next , The understanding result of the user information request content read from the dialogue record stored in the storage device, the information about the correctness, the acoustic and linguistic features of the user uttered speech, and the discourse feature extracted from the discourse information Based on this, a reliability measure is created to evaluate the reliability of the understanding result of user information request contents using discriminative learning.

  This reliability measure includes system utterance information, user speech recognition results including user speech recognition results and their acoustic and linguistic features, and information on the results of understanding the user information requirements of the spoken dialogue system for each user utterance. And the discourse feature value extracted from the discourse information related to the interaction between the user and the spoken dialogue system, and the reliability of the understanding result of the user information request content is output, which corresponds to the calculation formula. Specifically, it is realized by, for example, a functionalized program or data group. Here, this is called reliability evaluation means.

  In the spoken dialogue system used in actual operation, using the created reliability evaluation means, the system utterance information, the user utterance information including the user speech recognition result and its acoustic and linguistic features, and the user utterance The reliability of the understanding result of the user information request content based on the information of the understanding result of the user information request content of the Japanese conversation system and the discourse feature extracted from the discourse information about the interaction between the user and the voice interaction system calculate.

  The present invention makes it possible to calculate the reliability of the understanding result of the user information request contents with high accuracy in the spoken dialogue system. As a result, rejection of understanding results with low reliability and confirmation of things with high reliability can be realized with high accuracy, and smoother dialogue between people and the system can be realized.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a device configuration for realizing the reliability measure calculation method of the present invention. The reliability measure calculation apparatus shown in FIG. 1 includes a dialogue record 10 in which system utterances and user utterances are recorded, a discourse memory 20 in which an understanding result of information request contents for each utterance in the dialogue record 10 is recorded, and in the discourse memory 20. A discourse feature amount extraction unit 30 that extracts discourse feature amounts based on an understanding result of the information request contents, a data memory 40 in which data used for creating a reliability measure described later is accumulated for each utterance, and a reliability measure This is realized by the reliability scale creation unit 50.

  In the present invention, as in Non-Patent Document 2, the reliability of the understanding result of the information request content is obtained through the construction of the reliability scale. FIG. 7 is a diagram showing the flow of reliability scale construction.

  [Step S1]: The dialogue record 10 is recorded using the prototype system configured as shown in FIG. The dialogue record 10 includes a system utterance and user utterance (user speech and speech recognition result and acoustic / linguistic feature) at the time of dialogue recording, an understanding result of information request contents for each user utterance, and an understanding result of information request contents. Correctness is labeled.

  [Step S2]: i = 0 and the speech number i is initialized.

  [Step S3]: The utterance number i is incremented.

  [Steps S4 and S5]: The i-th utterance is extracted from the dialogue record 10. If there is no i-th utterance in the dialog record 10, the process proceeds to step S11.

  [Steps S6 and S7]: If the i-th utterance is a system utterance, the contents of the system utterance are added to the discourse memory 20, and the process returns to step S3.

  [Steps S8, S9]: If the i-th utterance is a user utterance, first, the understanding result of the utterance content and the information request content immediately after the user utterance is added to the discourse memory 20, and the information request content in the discourse memory 20 The discourse feature is extracted based on the understanding result.

[Step S10]: Next,
(A) Understanding result of information request immediately after user utterance,
(B) Correct answer of understanding result of information request contents,
(C) acoustic and linguistic features of user utterances;
(D) a discourse feature obtained from the discourse memory 20 by the discourse feature extraction unit 30;
, And the utterance number i and the above (a), (b), (c), and (d) are added together to the data memory 40, and the process returns to step S3.

  [Step S11]: A reliability measure is created by the reliability measure creation unit 50 from the data included in the data memory 40. As a method for creating a reliability scale, a technique using discriminative learning, which is often mentioned in Non-Patent Document 2, is common. Examples of discriminative learning include linear discriminant methods, Support Vector Machines (SVM), and decision tree learning. Here, a linear discrimination method will be described as an example of discriminative learning.

  FIG. 8 shows an image diagram of linear discriminant analysis by the linear discriminant method. The linear discriminant method is called alias discriminant analysis, and is a method of finding a straight line that divides a sample into two groups. FIG. 8 shows a state in which two types of samples (group A and group B) having feature amount 1 and feature amount 2 are divided into two by a linear discriminant straight line. Similarly, SVM and decision tree learning are techniques for finding straight lines (or planes and hyperplanes) that divide a plurality of samples into two (or more) parts. Even if discriminative learning such as SVM or decision tree learning is used. Similarly, a reliability measure can be created.

  In the method described in Non-Patent Document 2, first, a conversion vector ↑ p for converting a sample feature vector ↑ f to a scalar value r is obtained by the following equation. The initial value of ↑ p can be obtained by a linear discriminant method.

r = ↑ p ^T・ ↑ f
All samples (x _1, ..., x _n) to actually calculate the r _i, the probability distribution of the sample r _Ai in group A (mean and variance), the probability distribution of r _Bj sample group B (average And variance). Here, whether r, which is the result of converting an unknown sample with the conversion vector ↑ p, belongs to the A group or the B group, is obtained by the following equation.

c = log [{p (r | A) P (A)} / {p (r | B) P (B)}]-t
Taking the ratio of the probability of being a group A (p (r | A) P (A)) to the probability of being a group B (p (r | B) P (B)) when having a value of r, a threshold t Adjust the value with. The ratio value expresses the A-likeness of r. If the value of t is increased, the value of c can be decreased, and the group B can be easily obtained for the same value of r. c is a value indicating reliability, and is determined as group A if positive and group B if negative. The conversion vector ↑ p and the threshold value t are adjusted so as to minimize the classification error of the A group and the B group for known samples (see Non-Patent Document 2).

  By using the reliability scale created by the above steps, the reliability of the understanding result of the information request content immediately after the user utterance is obtained from the acoustic / linguistic feature amount and the discourse feature amount of the user utterance voice.

  In the following, embodiments of the present invention will be described using examples in a weather information guidance system.

[Dialogue experiment]
In the embodiment of the present invention, a voice dialogue system having the configuration of FIG. 2 was actually constructed, and a dialogue record was created by a dialogue experiment. Dialogue data collection was conducted for users who had never used a voice dialogue system in the past. Eighteen subjects each had 16 dialogues with the system, for a total of 288 dialogues. The experiment was conducted 6 times by 3 people. The dialogue records made at each time are called SET001, SET002, SET003, SET004, SET005, and SET006. All user utterances in the dialogue record were transcribed, and the correct / incorrect label was attached to the user information request contents after each user utterance.

[Discourse features]
The result of understanding the information request content is expressed in frame representation, and a reliability scale was created to determine the reliability of the slot that is a component of the frame. The value of the slot is assumed to be filled with a keyword included in the user utterance, and a reliability measure for estimating the reliability is created from the feature amount regarding the value when the slot is filled. Although the frame includes a plurality of slots, in the case of a weather information system, the discourse feature amount extraction unit 30 obtains a discourse feature amount described below for each slot of [place, date, info].

  The acoustic and linguistic features are the same as the conventional ones such as the acoustic and linguistic features related to the word in the user utterance shown in FIG. 5 or the acoustic and linguistic features related to the utterance shown in FIG. The following features are introduced as discourse features. FIG. 9 summarizes the discourse feature amounts D1 to D12 related to the slot values described below.

[D1]: Slot purity in context
For each understanding result that has passed through the current understanding result, it counts how many times the value has entered the slot, and indicates the percentage of the current value. For example, if the value changes from Tokyo → Osaka → Kyoto → Osaka, the current value Osaka occupies 2 out of 4 changes, so the value is ½. In the example of the date slot in FIG. 2, the slot value “Tomorrow” is 1/3.

[D2]: Top slot purity in context
For each of the understanding results that have passed through the current understanding result, count the number of times that value has entered the slot, and the proportion of the element with the highest proportion. For example, if the value changes from Tokyo → Osaka → Kyoto → Osaka, Tokyo is 1/4, Osaka is 1/2, Kyoto is 1/4, and the maximum value is 1/2 of Osaka. Becomes 1/2. In the example of the date slot in FIG. 2, “Today” is 2/3 and “Tomorrow” is 1/3, so the maximum value is 2/3.

[D3]: Slot variety
A value that indicates how many values the slot value on the understanding path has ever had. In the case of Tokyo → Osaka → Kyoto → Osaka, the value is 3 because there are three types [Tokyo, Osaka, Kyoto]. In the example of the date slot in FIG. 2, it is 2 because there are two types of “today” and “tomorrow”.

[D4]: Deny count
The number of times the current slot value has been negated in user interactions so far. For example, if there is a change in the slot value of Tokyo → None → Kyoto → Tokyo, the current value of Tokyo is once “None” (cancelled), and the value is 1.

[D5]: Overwrite count
The number of times the current slot value has been overwritten by other values in previous user interactions. For example, if there is a change in the slot value of Tokyo → Osaka → Kyoto → Tokyo, the current value of Tokyo was once overwritten by “Osaka” before, so the value is 1.

[D6]: Continue count
A value that indicates how long the current slot value has the same value on the understanding path. For example, if there is a change in the slot value of no value → Tokyo → Tokyo → Tokyo, the value is 2 because there is “Tokyo” twice immediately before the current value.

[D7]: Different value count
A value indicating how many times the current slot value has a different value on the understanding path. For example, if there is a change in slot value such as Tokyo → Osaka → Kyoto → Tokyo, the value is 2 because there is a value that is not “Tokyo” twice immediately before the current value.

[D8]: Same key pair count
It is considered unnatural from the viewpoint of redundancy as an exchange if the system confirms a certain slot value and the user transmits the same slot value to the system without affirming or denying the value. For example, it is expected that the exchange of “System is the weather in Tokyo?” → User: “It is Tokyo” does not occur very much. Therefore, for the slot value currently held by the system, the system checks the value in the past, and counts the number of times the user has transmitted the same value to the system.

[D9]: Same key count user
A value that indicates how many of the previous user utterances have the same value as the current slot value. For example, when the slot value of the current system is “Tokyo”, the number of times “Tokyo” appears in the user utterance history understood by the system is counted.

[D10]: Different key count user
A value that indicates how many different user utterances are different from the current slot value. For example, the slot value of the current system is “Tokyo”, and the number of times a value different from “Tokyo” appears in the user utterance history understood by the system so far is counted.

[D11]: Same key count system
A value that indicates how many of the previous system utterances have the same value as the current slot value. For example, when the slot value of the current system is “Tokyo”, the number of times “Tokyo” appears in the system utterance history that has been spoken by the system so far is counted.

[D12]: Different key count system
A value that indicates how many different values in the system utterance are different from the current slot value. For example, when the slot value of the current system is “Tokyo”, the number of times that a value different from “Tokyo” appears in the system utterance history that the system has spoken by confirmation or the like is counted.

〔Evaluation results〕
First, the following (1) and (2) were excluded from the slot value data during dialogue recording.

  (1) The feature value in the present invention assumes that the slot value has two or more values. For this reason, the slot value when the value is first obtained from a state where there is no value, and the data of the slot value when the speech recognizer outputs only a specific value are excluded. The reliability of the slot value when it has a value for the first time can be estimated from the acoustic and linguistic features because there is not enough discourse element about the value. Also, if the speech recognizer outputs only a specific value due to repeated misrecognition, it may be possible to determine the reliability from the small fluctuation of the value, but the speech recognizer recognizes the correct word. It is difficult to distinguish from the case of continuing, and it is considered more appropriate to estimate from the acoustic and linguistic features.

  (2) A confirmed slot. During the dialogue, the system may check whether the value held by the system is correct for the slot value. If the user approves the confirmation, the slot value is correct. Therefore, if it is assumed that the system has already confirmed the value of a certain slot, the reliability is roughly correct, so it was excluded from the target.

  In creating the slot reliability scale, the values of 777 slots excluding the slot values meeting the above conditions 1 and 2 were used in all data. Then, one of the six sets was set as test data and the rest as training data. The reliability measure was learned and tested for all sets, and the performance of the reliability measure was examined.

  In the following, the reliability scale generation according to the present invention in comparison with the conventional method will be described. FIG. 10 shows an example of data used for reliability scale generation in the conventional method. The first line “purity” to “num_of_words” is a list of acoustic and linguistic feature names, the second and subsequent lines are data, and the first item “slot-level-label” on the second line is this Indicates that the row is slot data, and the second item (for example, “1” or “0”) is correct (represented by “1”) or incorrect (represented by “0”). It is a label.

  FIG. 11 shows an example of data used for reliability scale generation in the present invention. The data example shown in FIG. 11 is stored in the data memory 40. From “slot_purity_in_context” to “num_of_words” on the first line is a list of discourse feature names and acoustic / linguistic feature names, the second and subsequent lines are data, and the first item “slot-level” on the second line -label "indicates that this line is slot data, and the second item (for example," 1 "or" 0 ") is correct (represented by" 1 ") or incorrect (" 0 "). ”).

  10 and 11, the acoustic and linguistic features in the list on the first row are those shown in FIGS. 5 and 6. In FIG. 11, the discourse feature in the list on the first row is shown in FIG. It is shown in.

  FIG. 12 shows the accuracy of a reliability measure created from only acoustic and linguistic features using the conventional method and a reliability measure created from both acoustic and linguistic features and discourse features using the present invention. It is a figure which shows a comparison. In FIG. 12, False Acceptance Rate is the rate at which errors are classified as correct, and False Rejection Rate is the rate at which correct answers are classified as errors. Each is an index used to evaluate the reliability scale, and is obtained by the following formula.

-------------------------------------------------- --------------
False Acceptance Rate
= Number of false positives / (number of false positives + number of true negatives)
-------------------------------------------------- --------------
False Rejection Rate
= Number of False Negatives / (Number of True Positives + Number of False Negatives)
-------------------------------------------------- --------------
In the above formula, False Positive means that it is mistakenly classified as a correct answer, and True Negative means that it is correctly classified as an incorrect answer. True positive means that it is correctly classified as a correct answer, and False Negative means that it is mistakenly classified as an incorrect answer.

  As shown in FIG. 12, it can be seen that in the present invention, the value of False Acceptance Rate and the value of False Rejection Rate are both lower than the conventional method.

FIG. 13 shows the correct / incorrect classification performance of the present invention compared to the conventional method. For all samples, there were 72 samples when only the present invention was correct and 15 samples when only the conventional method was correct. When a test called McNemar Test described in Reference Document 1 below is used to test whether there is a statistically significant difference, the result is p = 4.26e-10 (<0.01), which is statistically superior. Differences were noted indicating that the present invention was effective.
[Reference 1]
"Some Statistical Issues in the Comparison of Speech Recognition Algorithums," L. Gillick and S. Cox, ICASSP 89, v. 1, pp.532-535.
FIG. 14 is a False Acceptance Rate-False Rejection Rate curve of the reliability measure regarding the slot. This curve is a general curve required for determining whether a sample is correct or incorrect. This graph plots False Acceptance Rate on the vertical axis and False Rejection Rate on the horizontal axis.

  Since the curve of the present invention is always located below the conventional method, it can be said that the reliability scale created by the present invention is higher performance than the reliability scale created using the conventional method.

It is a figure which shows the apparatus structure which implement | achieves the reliability scale calculation method of this invention. It is a figure which shows the general structure of a speech dialogue system. It is a figure which shows the example of a dialog, and the understanding result of the information request | requirement content of the voice dialog system for every user utterance. It is a figure which shows the example of allocation of the reliability of the user information request | requirement content using the reliability of the word contained in the speech recognition result of a user utterance. It is a figure which shows the example of the acoustic and linguistic feature-value regarding a word. It is a figure which shows the example of the acoustic and linguistic feature-value regarding speech. It is a figure which shows the flow of reliability scale construction. It is an image figure of a linear discriminant analysis. It is a figure which shows the discourse feature-value regarding a slot value. It is a figure which shows the example of data used for the reliability scale preparation in a conventional method. It is a figure which shows the example of data used for the reliability scale preparation in a proposal method. It is a figure which shows the accuracy comparison of a reliability scale. It is a figure which shows the correct / false classification performance of this invention compared with the conventional method. It is a figure which shows a False Acceptance Rate-False Rejection Rate curve.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Speech understanding part 2 Speech production | generation part 10 Dialog recording 20 Discourse memory 30 Discourse feature-value extraction part 40 Data memory 50 Reliability scale creation part 100 Speech recognition part 101 Language understanding part 102 Discourse understanding part 103 Context understanding rule and knowledge DB
104 Syntactic Semantic Rules / Knowledge DB
105 Dialogue state DB
106 Dialog control response generation rule / knowledge DB
107 database 108 generation rule template DB
200 content generation unit 201 surface layer generation unit 202 voice generation unit

Claims (3)

Every time a user inputs an information request to a computer-based spoken dialogue system using voice, the user's spoken dialogue system updates the understanding result of the user information request held by the voice dialogue system. , A method for calculating the reliability of the understanding result of the user information request content in the spoken dialogue system that responds to the user by voice according to the understanding result of the updated user information request content,
Using an existing spoken dialogue system, create a dialogue record that includes system utterance information, user utterance information, and the user information request content understanding result and correct / wrong information for each user utterance. A process of storing in a storage device;
A process of extracting discourse information regarding the interaction between the user and the voice interaction system from the conversation record stored in the storage device, and extracting a discourse feature amount regarding a characteristic of an understanding result of the user information request content in the discourse information;
At least the understanding result of the user information request content read from the dialogue record stored in the storage device, the information about the correctness, the acoustic / linguistic feature amount of the user utterance voice, and the discourse feature amount extracted from the discourse information And a process of creating a reliability evaluation means for evaluating the reliability of the understanding result of the user information request content using identification learning,
Information on system utterances in the spoken dialogue system, information on user utterances including user speech recognition results and their acoustic and linguistic features, information on understanding results of user information requirements of the voice dialogue system for each user utterance, A dialogue characterized in that, based on the discourse feature extracted from the discourse information related to the interaction between the user and the spoken dialogue system, the reliability of the understanding result of the user information request content is calculated using the reliability evaluation means. Method for calculating the reliability of understanding results. In the process of extracting the discourse feature,
Appearance ratio of the current value of the understanding result of the user information request content, appearance ratio of the most frequently occurring value in the understanding result of the user information request content, number of types of appearance values in the understanding result of the user information request content, user information request The number of times the current value of the content understanding result was canceled, the number of times the current value of the understanding result of the user information request content was overwritten by another value, and the current value of the understanding result of the user information request content The spoken dialogue system confirms the value indicating whether the values have the same value, the value indicating how many times the current value of the understanding result of the user information request has a different value, and the value of the understanding result of the user information request The number of times the user has transmitted the same value to the spoken dialogue system and the number of times the same value as the current value of the understanding result of the user information request has appeared in the user's utterance so far The value that indicates the number of times that a different value from the current value of the understanding result of the user information request content has appeared in the user utterance so far, and the same value as the current value of the understanding result of the user information request content A value that indicates how many times the system utterance has appeared in the past, or a value that indicates how many times the system utterance has appeared in a different value from the current understanding value of the user information request. The method of claim 1, wherein at least one is extracted as a discourse feature. In the process of creating the reliability evaluation means,
The method for calculating the reliability of a dialogue understanding result according to claim 1 or 2, wherein linear discriminant method, support vector machine or decision tree learning is used as discriminative learning.

Images